Publication number | US7979435 B2 |
Publication type | Grant |
Application number | US 10/528,636 |
PCT number | PCT/FR2003/050051 |
Publication date | Jul 12, 2011 |
Filing date | Sep 11, 2003 |
Priority date | Sep 27, 2002 |
Fee status | Paid |
Also published as | DE60320118D1, DE60320118T2, EP1543443A2, EP1543443B1, US20060106816, WO2004029832A2, WO2004029832A3 |
Publication number | 10528636, 528636, PCT/2003/50051, PCT/FR/2003/050051, PCT/FR/2003/50051, PCT/FR/3/050051, PCT/FR/3/50051, PCT/FR2003/050051, PCT/FR2003/50051, PCT/FR2003050051, PCT/FR200350051, PCT/FR3/050051, PCT/FR3/50051, PCT/FR3050051, PCT/FR350051, US 7979435 B2, US 7979435B2, US-B2-7979435, US7979435 B2, US7979435B2 |
Inventors | Lionel Oisel, Bertrand Chupeau, François Le Clerc |
Original Assignee | Thomson Licensing |
Export Citation | BiBTeX, EndNote, RefMan |
Patent Citations (9), Non-Patent Citations (4), Referenced by (3), Classifications (12), Legal Events (2) | |
External Links: USPTO, USPTO Assignment, Espacenet | |
This application claims the benefit, under 35 U.S.C. §365 of International Application PCT/FR03/50051, filed Sep. 11, 2003, which was published in accordance with PCT Article 21(2) on Apr. 8, 2004 in French and which claims the benefit of French patent application No. 0211952, filed Sep. 27, 2002.
The invention relates to a method and a device for clustering key images by using spatial-temporal attributes.
Clustering is aimed at grouping data by utilizing measurements of distances or similarities between them, data that are not very distant being clustered within one and the same class. An application pertaining to digital video data relates to the automatic construction of video summaries.
In a first step 1, the video sequence is chopped into video shots. A second step 2 extracts characteristic images or key images from the various shots of the sequence. For each key image, this is step 3, a signature is calculated, for example on the basis of attributes of the image such as colour, structure, etc. The next step 4 is aimed at clustering the various shots which resemble one another into clusters of shots. A measurement of similarity is thus performed, on the basis of any calculation of distances between the signatures of the key images associated with each shot. The next step 5 constructs the summary by extracting a key image per cluster of shots.
This method makes it possible to reduce the number of characteristic shots, for example during video sequences with alternating shots, so as to create video summaries, indices, etc.
A known approach tackling the problems of clustering of shots for the construction of video summaries is that proposed by Yeung and Yeo in the document “Segmentation of video by clustering and graph analysis”, Computer Vision and Image Understanding, vol 71, no 1, July, pp 94-109, 1998. With each pair of shots is associated a distance which is a measure regarding the differences between signatures. In addition to the distance between signatures, the procedure proposed contrives not to gather together shots whose temporal distance is greater than a temporal threshold T. The underlying idea rests upon the assumption that shots belonging to one and the same semantic unit cannot be very distant. This assumption also has the advantage of limiting the number of potential clusterings and thus of limiting the calculation cost. The clustering algorithm used operates on successive clusterings of shots commencing with the most similar until the distances are all greater than a threshold.
In this procedure, if two shots are separated by more than T images, the clustering is not possible. In the article cited, T is fixed at a value of the order of a few thousand images. The main problem of this procedure resides in the fact that this threshold is fixed, that it is of significant importance to the final result and that it is therefore difficult to fix a priori. For example, if a dialogue scene lasts more than 3000 images and if this value exceeds the temporal threshold, overchopping occurs. All the shots thus clustered must be pairwise close both visually and temporally. The size of the clusters thus generated is therefore relatively limited. The final result of this clustering algorithm is characterized by the obtaining of clusters that are relatively uniform in terms of number of shots, this number generally being small and in any event limited by the value of the threshold T.
The invention is aimed at alleviating the aforesaid drawbacks. Its subject is a method of clustering images of a video sequence consisting of shots and represented by a graph-like structure, a node of the graph representing a shot or a class of shots defined by key images and the nodes being connected by edges, characterized in that it comprises the following iteration:
According to a particular implementation of the invention, the graph is initialized by assigning a node to each shot and by creating edges from one node to another node if the shots relating to these nodes are separated by a predetermined maximum number T of shots.
According to a particular implementation of the invention, the value of potential associated with the edges and with the nodes is a respectively decreasing and increasing function of temporal distance.
The calculation of similarity of the images takes into account the distance between signatures, for example the difference between histograms, and the temporal distance between key images defining the shots. This temporal constraint is adaptive, two shots that are initially too far apart to be clustered possibly then becoming potentially “clusterable” should intermediate shots be merged.
The calculation algorithm is based on energy minimization by merging relying on a graph structure. Contrary to the previous procedure, in order to belong to a cluster of images, also subsequently referred to as a class of images or class of shots, a candidate image must be visually and temporally close to at least one image of the cluster.
An advantage of the invention is the taking into account of images even if they are temporally far apart, due to merging, while favouring the clustering of temporally close images.
Another advantage resides in the fact that only one image of a cluster has to be similar to the candidate image in order to permit the merge. The clusters thus created can comprise more elements than in the case of the conventional procedure. The scheme proposed makes it possible to generate more heterogeneous clusters, that are less dependent on values of thresholds and hence closer to the data, without thereby rendering the calculations too complex or too lengthy.
Other features and advantages of the invention will become clearly apparent in the following description given by way of nonlimiting example and offered in conjunction with the appended figures which represent:
The procedure proposed relies on a graph structure. A graph consists of a set of nodes and edges. A node n_{i }symbolizes a shot or cluster of video shots that are similar according to the visual and temporal distance chosen, each shot P_{i }being represented by at least one key image. The edges connect the nodes together, two nodes that are not connected by an edge not being potentially mergeable.
A size of temporal window T is chosen. It is given as a maximum number of shots T separating two shots in the video sequence. Upon the initialization of the graph, edges are created on the basis of a given node corresponding to a given shot towards each of the nodes whose shots lie in the temporal window of this given shot.
In a practical manner, the nodes are first connected by edges according to the sequential unfurling of the video sequence. This progression is subsequently taken into account, when adding the complementary edges, for the application of the temporal window. Then, the integer value T chosen for this step of initializing the graph signifies that only the nodes separated at most by T nodes from the node considered are taken into account for the calculation of the initial edges starting from this node. An initial edge a_{ij }is created between node n_{i }and node n_{j }only if the temporal distance d_{T }separating these nodes with which the shots P_{i }and P_{j }are associated is less than a threshold T. This constraint is effective only on initialization of the graph, the later mergings of nodes making it possible to connect nodes that are much further away from the origin, if necessary.
The choice of this threshold on initialization is made with the aim of optimizing the calculation time that is highly dependent on the number of edges.
The initialization of the graph is performed by creating a node per video shot.
The energy of the system is equal to the sum of the potentials of the edges and of the nodes. The aim is to cluster or merge the nodes in such a way as to reduce the overall energy of the system.
To define the overall energy of the system, the concept of potential V_{n }of a node and V_{a }of an edge is introduced. A potential is calculated with the help of a base distance between two images I_{i }and I_{j}, temporal distance and distance between attributes.
The value of the potential associated with an edge a_{ij }is defined with the aid of a decreasing function of distance. The bigger the temporal distances and the distances between attributes, the smaller the potential. The overall objective of the algorithm being to minimize the overall energy of the system, the closer according to the distances defined above are the shots tied to two nodes, the higher will be the potentials of the edges and the more favoured will be the merging and hence the elimination of the edges.
The value of the potential associated with node n_{k }is defined with the help of an increasing function of distance. The smaller the temporal distances or the distances between attributes as calculated between the images belonging to two distinct nodes, the smaller the potential associated with the merge node, hence the more the system will be inclined to merge the two nodes to create a node of small potential.
Let V_{a }(n_{i}, n_{j}) and V_{n }(n_{m}) be respectively the potentials of an edge connecting nodes n_{i }and n_{j }and of a node n_{m}.
Firstly we describe the potential related to the edges. Here we calculate the “inter-image” potentials of the edge for each of the pairs formed of an image from among the images associated with an end node of the edge and an image from among the images associated with another end node of the edge. Stated otherwise, we are dealing with potentials calculated on the basis of the cross distances between the images associated with the first node and those associated with the second node at the ends of the edge.
The potential V_{a }(n_{i}, n_{j}) of an edge may therefore be written as a function of the cross distances between the images I_{k }belonging to n_{i }and the images I_{l }belonging to n_{j}. The following minimum value is calculated:
where:
The set of images that is associated with node n_{m }is also referred to as the class of shots associated with node n_{m}. This class clusters together the set of images of the classes of merged shots.
The distance between attributes may be taken as a classwise difference between histograms characterizing each image. The temporal distance is for its part defined as the number of shots separating the two shots to which the images I_{k }and I_{l }belong.
In the implementation, the decreasing function F_{a }is of the following form:
Values of potential are also calculated for the nodes of the graph. When two nodes n_{i }and n_{j }corresponding to key images I_{i }and I_{j }belonging respectively to the classes of shot P_{i }and P_{j }are merged to create a new node n_{k}, the images I_{i }and I_{j }are integrated into the new class of shots P_{k }associated with node n_{k}. The number of associated images thus increases in tandem with the mergings. In a general manner, the number of images associated with a new node corresponds to the sum of the images associated with the merged nodes generating the new node.
The potential of a node merged from a first and second node is obtained with the help of the potentials calculated over the set of image pairs formed of an image associated with the first node and of an image associated with the second node. This value is dependent on the distance between attributes d_{att }and on the temporal distance d_{T}. There are as many potentials calculated as there are image pairs. The value chosen is that which gives the minimum potential.
The potential Vn(nm) of a node arising from the merging of two existing nodes n_{i }and n_{j }is therefore defined by the following function:
The potential of a merge node is therefore calculated on the basis of the cross distances between the images belonging to each of the nodes that are candidates for the merge.
In the implementation, the increasing function F_{n }is of the following form:
On initializing the graph, the potential of the nodes is zero. There is a key image per node. It is in the course of the merging process that this potential takes a strictly positive value.
The factor K used in the functions F_{a }and F_{n }is a parameter which influences the curvature of the function. The bigger is K, the more the function tends to 0 or to 1 rapidly when moving away from the mean value {tilde over (d)}. In terms of potential, the bigger is K, the more the variations in d around the mean value give rise to big variations in the potential.
In the case of the function tied to the distance between attributes, K_{att }may be taken equal to 2. In the case of the function tied to the temporal distance, K_{T }may be taken equal to 7.
When calculating a potential relating to an image pair (I_{i},I_{j}), the further apart temporally or according to their attributes are the images processed, the lower is the potential of the edge corresponding to these two images, the greater is the potential of the node corresponding to the merged pair of images.
The temporal constraint therefore appears in two forms in our algorithm:
The overall energy of the system is equal to the sum of the potentials associated with each edge and with each node.
An iterative merging process is implemented to obtain a configuration of the system having the lowest possible overall energy. The aim of each iteration is to decrease the overall energy. The calculation of the minimum energy would require all possible combinations to be tried. This number possibly being appreciable, as a function of the number of shots, the energy minimization scheme chosen is a suboptimal scheme. It does not guarantee that the solution found does correspond to the configuration minimizing the total energy of the system. However, it has the benefit of being fast while providing an acceptable energy solution.
The smaller the distance between two images, the larger the potential of the edge relating to these two images and hence the higher the potential of the edge connecting the corresponding nodes. One seeks to reduce the overall energy of the system, hence to eliminate the edges of largest potential, this amounting to eliminating the potential relating to these edges, the potentials of the edges emanating from the merged nodes being recalculated. This corresponds well to wishing to cluster the closest images, both from the attributes point of view and from the temporal point of view.
The potential of a node obtained after merging the two nodes at the ends of the deleted edge is recalculated. The greater the distances between the images associated with the nodes to be merged, the bigger the potentials of the merged node that relate to the image pairs, the bigger the potential of this node which is the minimum value out of these potentials.
In the example of
The iterative energy reduction process implemented is described in
The first step referenced 6 verifies whether all the edges of the graph have been tested.
If they have, the iterations are terminated and the shots or key images are clustered into classes of shots, created in tandem with the mergings of the nodes of the graph.
If they have not, the next step is step 7 which takes into account an untested edge a_{k }of the graph whose ends are the nodes that will be called n_{i }and n_{j}. The edges are for example selected as a function of their energy value in decreasing order of the values of the associated potential. Indeed a deleted edge of high potential gives rise a priori to a big energy drop.
The next step 8 performs a calculation of the energy of the graph by assuming a merging of nodes n_{i }and n_{j}. In addition to the potential of the merged node, the potential of the new edges connecting the new node to the set of nodes adjacent to nodes n_{i }and n_{j }is taken into account for the calculation of the energy of the new graph. If the energy found is greater than the energy before merging, test performed in step 9, we return to step 6 then to step 7 to process the next edge, should all the edges not have been tested.
Should the energy calculated be less than the energy of the graph before merging, the next step is step 10 which validates the merging of nodes n_{i }and n_{j }and the new graph which stems therefrom.
The next step 6 verifies whether all the edges have been tested. This deals with the edges not modified by the merging of the nodes but also new edges calculated during this merge. If they have, the final graph is that validated in the previous step 10.
A simplification consists in using a suboptimal stopping criterion making it possible to avoid the exhaustive traversal of the edges. This simplification consists in stopping the minimization as soon as the potential merging of two nodes gives rise to an increase in the energy. This simplification is rendered acceptable by the intelligent selection of the edges in decreasing order of their value of potential.
The scheme presented is suboptimal overall on account of the irreversible nature of the merging. The switch to minimization of the optimal energy may be envisaged along with high calculation cost.
The choice of the function of energy and potentials that we have just defined allows us to simplify the energy minimization scheme. Specifically, choosing the minimum of the potentials implies that if the potential associated with a node originating from the merging of two nodes is less than the sum of the potentials of these two nodes, then the merging of the two nodes necessarily gives rise to a drop in the potential of the neighbouring edges. A local study of the energy variation is therefore sufficient, while ensuring an overall drop in this energy.
The method described makes it possible to obtain an efficacious merging process despite the calculational limitation. The latter relates to the limited possibilities of initial pairings, the number of edges being limited on initialization. It also relates to the utilization of a reduced window on initialization. However, on account of the merging process, shots that are far apart may be clustered together. The potentials take into account the temporal distances and the distances between attributes. They allow merges if the key images, although distant, greatly resemble one another or else, if, although resembling one another averagely, the key images are temporally very close.
Cited Patent | Filing date | Publication date | Applicant | Title |
---|---|---|---|---|
US5572246 * | Jun 7, 1995 | Nov 5, 1996 | The Arbitron Company | Method and apparatus for producing a signature characterizing an interval of a video signal while compensating for picture edge shift |
US5706503 | May 18, 1994 | Jan 6, 1998 | Etak Inc | Method of clustering multi-dimensional related data in a computer database by combining the two verticles of a graph connected by an edge having the highest score |
US5821945 * | May 15, 1997 | Oct 13, 1998 | The Trustees Of Princeton University | Method and apparatus for video browsing based on content and structure |
US6185314 * | Feb 6, 1998 | Feb 6, 2001 | Ncr Corporation | System and method for matching image information to object model information |
US6549660 * | Mar 17, 1999 | Apr 15, 2003 | Massachusetts Institute Of Technology | Method and apparatus for classifying and identifying images |
US7054367 * | Dec 31, 2001 | May 30, 2006 | Emc Corporation | Edge detection based on variable-length codes of block coded video |
US7212201 * | Sep 20, 2000 | May 1, 2007 | New York University | Method and apparatus for segmenting an image in order to locate a part thereof |
US20020028021 * | Mar 11, 1999 | Mar 7, 2002 | Jonathan T. Foote | Methods and apparatuses for video segmentation, classification, and retrieval using image class statistical models |
US20020069218 * | Jul 23, 2001 | Jun 6, 2002 | Sanghoon Sull | System and method for indexing, searching, identifying, and editing portions of electronic multimedia files |
Reference | ||
---|---|---|
1 | A. Sarkar et al. "A Simple Unsupervised MRF Model Based Image Segmentation Approach", IEEE Transactions on Image Processing, IEEE Inc., New York, US, vol. 9, No. 5, May 2000, pp. 801-812. | |
2 | M. Yeung et al. "Segmentation of Video by Clustering and Graph Analysis", Computer Vision and Image Understanding Academic Press, US, vol. 71, No. 1, Jul. 1, 1998, pp. 94-109. | |
3 | Rui Yong et al.: "Exploring video structure beyond the shots", Multimedia Computing and System, 1998, Proceedings. IEEE International Conference on Austin, TX, USA Jun. 28 through Jul. 1, 1998, Los Alamitos, CA, pp. 237-240. | |
4 | Search Report Dated Apr. 15, 2004, 3 pages. |
Citing Patent | Filing date | Publication date | Applicant | Title |
---|---|---|---|---|
US9171578 * | May 9, 2011 | Oct 27, 2015 | Futurewei Technologies, Inc. | Video skimming methods and systems |
US9336302 | Mar 14, 2013 | May 10, 2016 | Zuci Realty Llc | Insight and algorithmic clustering for automated synthesis |
US20120033949 * | Feb 9, 2012 | Futurewei Technologies, Inc. | Video Skimming Methods and Systems |
U.S. Classification | 707/737, 707/725 |
International Classification | G06F7/00, G06K9/62, G06K9/00, G06F17/30 |
Cooperative Classification | G06K9/6224, G06F17/30843, G06K9/00711 |
European Classification | G06K9/62B1P2, G06K9/00V3, G06F17/30V4S |
Date | Code | Event | Description |
---|---|---|---|
Mar 22, 2005 | AS | Assignment | Owner name: THOMSON LICENSING S.A., FRANCE Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:OISEL, LIONEL;CHUPEAU, BERTRAND;CLERC, FRANCOIS LE;REEL/FRAME:016901/0029 Effective date: 20050215 |
Dec 15, 2014 | FPAY | Fee payment | Year of fee payment: 4 |