|Publication number||USRE42790 E1|
|Application number||US 12/324,481|
|Publication date||Oct 4, 2011|
|Filing date||Nov 26, 2008|
|Priority date||Jan 11, 2003|
|Also published as||US7095786, US7142600|
|Publication number||12324481, 324481, US RE42790 E1, US RE42790E1, US-E1-RE42790, USRE42790 E1, USRE42790E1|
|Inventors||Dan Schonfeld, Karthik Hariharakrishnan, Philippe Raffy, Fathy Yassa|
|Original Assignee||Neomagic Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (26), Non-Patent Citations (7), Referenced by (7), Classifications (15), Legal Events (4)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This application is a continuation-in-part of the co-pending application for Object Tracking Using Adaptive Block-Size Matching along Object Boundary and Frame-Skipping When Object Motion is Low, U.S. Ser. No. 10/248,348, filed Jan. 11, 2003 now U.S. Pat. No. 7,095,786.
This invention relates to image processing, and more particularly to object tracking, contour prediction, and occlusion and disocclusion detection in a video sequence.
With increasing computational power, more complex operations may be performed on images, such as detecting foreground objects. Once the foreground object is located, it can be tracked through frames of a video sequence. Knowing the location of the foreground object in each frame allows video compression to be improved, since more computational resources and transmission bandwidth can be allocated to the foreground object while less is allocated for the background.
Software is desirable that detects foreground or moving objects, and draws bounding boxes or contours around the object in each frame of the video sequence. The parent application described in detail such object tracking using macroblocks. Near the boundary of the object, the macroblocks can be divided into smaller blocks to improve the tracked contour of the object. Such block-based object tracking can be performed more efficiently than other known object-tracking methods. Frames can be skipped when object motion is low, while faster object motion tracking proceeds frame-by-frame. Such tracking modulation was also described in the parent application.
The location of the best-matching block in frame T determines the type of macroblock in frame T+3. There are three types:
blocks inside the object are certain blocks
blocks outside the object are background blocks
blocks that have the object's boundary passing through the block are uncertain blocks.
Each block in frame T+3 is categorized based on what type of block best matches in frame T. Block 15′ in frame T+3 is categorized as a background block since the best-matching block 15 in frame T is outside the initial object contour for object 10. Block 12′ in frame T+3 is categorized as a certain object block since the best-matching block 12 in frame T is inside the initial object contour for object 10 in frame T. Likewise block 14′ in frame T+3 is categorized as a certain object block since the best-matching block 14 in frame T is also inside the initial object contour for object 10.
Blocks 16′, 18′ in frame T+3 is categorized as uncertain blocks since the best-matching blocks 16, 18 in frame T are along the initial object contour for object 10. The boundary of object 10 in frame T passes through blocks 16, 18.
The certain object blocks in frame T+3, such as blocks 12′, 14′, are shown in solid lines, while the uncertain blocks such as 16′, 18′ are shown with dashed lines. As the new object mask is being constructed for frame T+3, the certain blocks such as 12′, 14′ form the beginning or “seed” of the new object mask. The exact location of the boundary of object 10′ is not yet known for frame T+3. However, it is relatively certain that the certain object blocks are part of object 10′.
The actual boundary for object 10′ is somewhere within the dashed uncertain blocks, but the exact pixel locations of this boundary has not yet been determined for frame T+3.
Background blocks 22 often have many errors in their motion vectors, since the background may have little motion or a variety of motions. Also, the background blocks may lack differentiating features. The lack of such variations may result in aliasing, where a background block 22 matches many other blocks. For example, the water may be a relatively uniform blue without distinguishing features. A blue background block may match many other background blocks, resulting in errors in the background motion vectors.
Since uncertain blocks 24 often include some background pixels and some object pixels, finding good matches may be difficult. When the object changes shape due to rotation or z-axis motion, the location of the boundary changes and a match may not be found, or a match found with the wrong block. Thus errors in the motion vectors can occur along the boundary with uncertain blocks 24. More variation in the direction and magnitude of motion vectors is seen for uncertain blocks 24 than for certain blocks 22.
Only certain blocks 22 are used to calculate the average object motion. This reduces errors, since the poorly-matching and changeable uncertain blocks 24 are not included in the average. The motion vectors of certain blocks 22 usually show a lower variance than do the motion vectors of uncertain blocks 24. An average motion that more accurately represents the object's motion is produced.
Once the best-matching macroblock in frame T is found, step 102, the relative displacement between the macroblocks in the two frames is calculated, such as the delta x and delta y values. This displacement is the motion vector for the block, step 104.
The location of the best-matching macroblock in frame T is compared to the object location in frame T. The object contour or object mask is already known for frame T but has not yet been generated for frame T+N. When the best-matching macroblock in frame T is within the object mask, step 106, then the macroblock in frame T+N can be marked as a certain block and can be added to the new object mask being constructed for frame T+N, step 110.
When the best-matching macroblock in frame T is not within the object mask, but is along the boundary of the object, step 108, then the macroblock in frame T+N is marked as an uncertain block, step 112. Uncertain blocks are not considered when calculating the average motion, but can be further processed by adaptive-size block matching.
When the best-matching macroblock in frame T is outside the object and not on the object boundary, it is marked as a background macroblock, step 114. When there are more macroblocks in frame T+N, the process of
When a current macroblock in frame T+N being processed is a certain block, step 120, then the macroblock's motion vector is accumulated into an average, step 126. The next block in frame T+N is selected, step 122, and steps 120, 126 repeated until all macroblocks in frame T+N have been processed.
At the end of processing when step 124 is executed, a final average motion vector is available to be compared to a motion threshold, such as 5 pixels of movement. When the average motion vector of the certain blocks of the object exceed the motion threshold, step 132, then high motion exists, and the modulation parameter is set to a low value such as 1. The block-matching and motion estimation of
When the average motion vector is less than the motion threshold, step 130, then slow or low motion is occurring, step 128, and the modulation parameter can be increased to a higher value such as N=3. The boundary of the object in the new frame T+N may then be refined by adaptive block matching 200.
Various algorithms exist that allow a computer or processor to extract the location of object 10 in frame T. For example, segmentation or watershed analysis can determine the contour or boundary of object 10 by the rapid change in color at the perimeter of object 10, which might be a yellow fish while the background is blue water.
Contour 11 of object 10 can be extracted as points along a line having a maximum gradient or change in color between the fish and the water. Similar contour extractions could be performed for subsequent frames T+1, T+2, and T+3 to generate contours 11′, 11″, and 11′″ of
Unfortunately, coral 13 may be the same color as object 10, such as yellow coral blocking a yellow fish. This may cause part of coral 13 to be added to the object mask. Contours 11′, 11″, and 11′″ then include some of coral 13 within the predicted contour of object 10. Coral parts 13′, 13″, and 13′″ should not be part of fish contours 11′, 11″, and 11′″ but are included by mistake.
Such additions to the object contour are dangerous, since after further processing of additional frames, the object tracked may change. If enough of coral 13 is added to contours 11′, 11″, and 11′″, then in later frames coral 13 may be tracked rather than the fish, object 10.
Occlusion and disocclusion of other parts of the fish also occurs in this video sequence. The fish may be turning away from the viewer, causing its tail to shrink in size and even disappear by frame T+2. In a later frame T+3 the tail may suddenly reappear as the fish twists. The tail then needs to be added back into the object mask. Such additions to an object mask, or disocclusion, must be carefully made.
The inventors have realized that motion can be used to determine when to add or remove regions from an object mask. The inventors realize that regions belonging to an object tend to move together, while regions that are not part of an object tend to move in other directions, often in a more random manner when compared to the object motion. Thus motion can be better than color for generating object masks.
A decision to include the re-emerging tail of the fish in frame T+3 can also be made using motion vectors. The re-emerging (disoccluded) tail has motion vector 15, which is closer to the fish's motion vector 15 than to the coral's motion vector. Thus the tail is added to object contour 11′″ while coral part 13′″ is removed from contour 11′″ when motion vectors are considered.
What is desired is to improve the object tracking methods described in the parent application by adding occlusion and disocclusion detection. It is desired to adjust the predicted object contours by adding disoccluded regions and removing occluded regions from the object mask. It is further desired to use motion information to decide when to include or exclude regions from the object mask.
The present invention relates to an improvement in object tracking with occlusion and disocclusion. The following description is presented to enable one of ordinary skill in the art to make and use the invention as provided in the context of a particular application and its requirements. Various modifications to the preferred embodiment will be apparent to those with skill in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the particular embodiments shown and described, but is to be accorded the widest scope consistent with the principles and novel features herein disclosed.
Embodiments of the present invention include computer-program products comprising a computer-usable medium having computer-readable program code means embodied therein for tracking an object boundary in a video stream.
The initial object mask for frame T is input, step 160. A user can manually draw a contour around the object, or an automated method can be used. This initial contour generation can be performed intermittently or only has to be performed once, for the first frame (T=1) in the video sequence being processed, or for the first frame that the object appears in.
The parameter N is the frame-modulation number, or the number of frames to skip ahead to. Object tracking is performed every N frames. When N=1, object tracking is performed every frame, while when N=3 object tracking is performed every third frame, and two frames are skipped. N is set to 3 when slow motion is detected, but set to one when high motion is detected.
Initially, the frame-modulation parameter N is set to 3, step 162. Backward motion estimation, step 164, is performed between new frame T+N and first frame T. Each macroblock in frame T+N is compared to a range of macroblocks in frame T to find the closest matching macroblock in frame T. A sum-of-absolute differences or least-variation of the YUV or other pixel color can be used to determine how well the blocks match. The displacement between the macroblock in frame T+N and the best-matching macroblock in earlier frame T is the motion vector for the macroblock in frame T+N.
Motion vectors for all macroblocks in frame T+N can be generated in step 164. The search range may be restricted, such as to a range of 32 pixels in any direction, or the entire frame T can be searched.
The location of each best-match block in frame T is compared to the object contour of frame T to determine if the best-matching block is within the object or outside the object or along the contour or boundary itself. Blocks along the boundary are specially processed by adaptive block sizes as described later.
Blocks in frame T+N that match a frame T block that is entirely within the initial object mask or contour and referred to as “certain” object blocks. Blocks in frame T+N that match a block in frame T that is entirely outside the initial object contour are also “certain” blocks, but are background blocks. Blocks in frame T+N that best match a block that includes the object boundary are referred to as “uncertain” blocks. The certain object blocks are marked and their average motion is computed, step 166.
The average motion of the object calculated in step 166 is compared to a threshold motion. When the average object motion exceeds this threshold motion, high motion is said to occur, step 170. Then the modulation parameter N is reset to 1, step 174, and motion estimation and average-motion calculation (steps 164-166) are repeated for the next frame T+1. Thus a finer granularity of frames for motion estimation is used when motion exceeds the threshold.
When the average object motion is below the threshold motion, low motion is occurs, step 170. Skipping frames is acceptable since the object is moving relatively slowly.
The location of the object boundary is more precisely determined using adaptive block matching, step 172. The uncertain blocks lying on the object boundary are sub-divided and matched using adaptive block matching (
While backward motion estimation from frame T+N to frame T was performed in step 164 to generate the initial object mask, forward motion estimation from frame T+N to frame T+2N is performed in step 175. Forward and backward motions are used for occlusion/disocclusion processing.
Using the forward and backward motion vectors, the object mask is refined to remove occluded regions from the object mask, process 800 (
Also using the forward and backward motion vectors, the object mask is again refined to add disoccluded regions back into the object mask, process 700 (
For low motion, the modulation parameter N remains set to 3. The video is advanced and the process repeated. The first frame T in the method is advanced to frame T+N, step 176. Frame T+N becomes frame T, while frame T+2*N becomes frame T+N as the video is advanced by step 176. Motion estimation and average-motion calculation (steps 164-166) are repeated for the new initial or base frame and the new current frame T+N. When N=1, the process flow passes through step 170 to step 176 even when the average motion exceeds the threshold. This is because N=1 is already as slow as possible.
Blocking object 32 is moving toward the upper left, and is also rigid and purely translational, with motion vectors pointing to the upper-left as shown. In
In frame T+N, shown in
Actual objects may not be rigid and may have non-translational motion. These more difficult types of objects may still be tracked by comparing motion vectors for suspected occluded or disoccluded regions to an average motion vector for the object. Non-translational motion usually cannot be described by just one average. Clustering of motion vectors is used for this case. The motion of the object can be better described using cluster centroids. The averaging of the object motion vectors allows for a simpler comparison of vectors even when the object is moving in a non-translational manner or is changing in apparent shape. One average motion vector or centroid for the object can be compared to the average or centroid motion vector for a suspected occluding or disoccluding region.
In this example, detection of a future occlusion occurs as blocking object 32 blocks object 30 in frame T+2N but not in frames T+N and T. Occluded region 34 is removed from the object mask for frame T+N before the occlusion actually occurs to allow the object mask for frame T+N to match the occluded object in future frame T+2N. For displaying the object in frame T+N, the object is displayed without removing the occluded region. For the computation of the object mask in the next frame (T+2N), the object mask with occluded regions removed is used. For display of frame T+N, the full object is shown since occlusion happens at frame T+2N.
Frame T+N is motion compensated with frame T to produce a motion-compensated frame (T+N)′. This motion-compensated frame (T+N)′ from step 802 is subtracted from the original frame T+N to produce the displaced frame difference (DFD), step 804. Differences that are greater than a threshold are considered to be newly covered regions, since regions that disappear cannot be matched and produce large differences. A binary mask can be made of these covered regions within the object mask. These suspect regions with large differences within the object mask may correspond to obscured regions or they may be noise. Suspect regions outside the object mask are discarded, and suspect regions that are not near the border can also be ignored as noise.
Various filtering can optionally be applied, step 806, to remove small regions that may represent noise rather than newly covered regions within the object mask. For example, a 5-by-5 pixel kernel can be used for median filtering the DFD. Then several opening and closing operations can be performed to try to join smaller regions with larger ones. Regions that are still smaller than a threshold can then be removed.
The filtered differences represent regions that are suspected of being covered in frame T+2N. In step 808 these covered regions are removed prior to clustering from the object mask for frame T+N, which is the last frame processed before occlusion occurs (occlusion has not yet occurred in frame T+N, but the object mask is adjusted for comparison to future frames such as T+2N).
These suspected regions may or may not be part of the object. If the region is not part of the object, it cannot be a real occlusion. To determine whether the suspect region was part of the object in prior frames, a motion similarity test is used. The prior motion of each suspect region in frames T and T+N, before the occlusion occurs, is compared to the motion of the tracked object. If the region's motion is similar to the object motion in frames T and T+N, the region is probably part of the object and represents a real occlusion in frame T+2N. The suspect region with similar motion should be removed from the object mask as an occlusion. If the region's motion is not similar to the object motion, the region is probably just noise and not part of the object. The noisy region should not be removed from the object mask but should be ignored.
Rigid objects could be represented by a single motion vector for the whole object. However, many real-world objects are not rigid. Instead, portions of the object can move differently than other portions, such as a person swinging his arms as he walks. The inventors have discovered that better tracking occurs when the object is divided into smaller portions called clusters. Rather than compare each suspect region's motion to an average motion for the entire object, the object is divided into one or more clusters. The average motion vector for each cluster is determined, and is known as the cluster centroid. Motion vectors used are those for motion between frames T and T+N, the backward motion vectors already calculated, step 812.
A variance-minimizing algorithm can be used to determine which blocks in the object are assigned to which clusters. For example, a K-means algorithm can be used where the number of clusters K is adaptively estimated. First, all blocks can be assigned to one cluster, and the variance in motion vectors calculated. Then K can be set to 2 clusters, and each block randomly assigned to one cluster or the other. The blocks can then be re-assigned to the cluster that better fits their motion using one or more iterations. Other numbers of clusters (K=3, 4, 8, etc.) can be tested and iterated in a similar manner. The number of clusters that produces the minimum variance between the cluster centroid motion vector and each block's motion vector can be chosen as the best-fit number of clusters and assignment of blocks to clusters.
Rather than calculate the K-means algorithm to full convergence, a threshold can be tested against for each successively larger value of K. For each number of clusters K, the final difference between the block and centroid motion vectors is calculated for each block. When a large fraction of the blocks have a final difference greater than the threshold, then K is increased and clustering repeated for the larger number of clusters. This can be repeated until a small fraction of the blocks have differences less than the threshold, such as no more than 10% of the blocks. Alternately, when increasing the number of clusters K increases the differences, then the smaller value of K is used as the final cluster assignment. The ideal number of clusters K is typically 2 to 4 for many objects tracked, but can have other values.
During K-means clustering, step 810, only the blocks within the object mask that are not suspected of being obscured are clustered. Suspect regions were already removed from the object mask in step 808. Removal of suspect regions produces better accuracy of object motion since the suspect regions may be noisy or not part of the object.
When a block is not fully within the object, such as for a boundary block, a weighting can be used. The block's contribution to the centroid motion vector is reduced by the fraction of pixels in the block that are outside the object. Blocks are ideally macroblocks that were motion estimated in step 812, but could be other blocks or regions that had motion vectors calculated between frame T and T+N.
In step 812, motion vectors between frames T and T+N are read and averaged for the region, or re-generated for the suspect regions that were removed from the object mask of frame T+N in step 808. These suspect regions are suspected of being obscured. The regions could be blocks such as macroblocks, but then removal of these block-shaped regions could leave blocky staircase edges on the object mask. Instead, the inventors prefer to allow the regions to be irregular, having whatever shape and size remain after filtering out smaller regions in step 806. One backward motion vector average is generated for each suspect region, step 814, such as by averaging motion vectors for blocks or pixels within the region.
Each suspect region's motion vector is compared to the centroid motion vectors for all clusters in the object, step 816. The absolute-value difference between the suspect region's motion vector and the cluster motion vector is calculated for all combinations of suspect regions and object clusters. For each suspect region, the object cluster having a centroid motion vector that has the smallest difference with the region's motion vector is chosen as the best-fit cluster. The best-fit cluster is the object cluster than has a motion that most closely matches the motion of the suspect region.
When the smallest difference is greater than a threshold value, then the differences in motion are too large for the suspect region to be part of the object. However, when a suspect region's smallest difference is below the threshold value, then the motions of the suspect region and best-fit cluster of the object are close enough for the suspect region to be part of the object.
Suspect regions with motions close enough to one of the object clusters are removed from the object mask for frame T+N as occluded regions of the object, step 818. Suspect regions with dissimilar motions that were removed from the object mask in step 808 can be added back into the object mask after occlusion processing is complete, since these regions represent noise rather than actual object occlusions.
The prior motion, before the occlusion occurs, is what is compared to determine if a suspect region is really part of the object or is simply a noisy artifact. For example, region 34 is occluded in frame T+2N. Region 34 corresponds to region 34″ in frame T+N, which is a suspect region detected by the DFD frame difference (step 804). The prior motion of region 34′″ in frame T and region 34″ in frame T+N are compared to the object clusters in these frames T and T+N for the motion similarity test. When prior motions match, the suspect region is part of the object and can be removed as a future occlusion.
In this example detection of a current disocclusion occurs as blocking object 32 uncovers part of object 30 in frame T+N. Disoccluded region 42 is added to the object mask for frame T+N when the disocclusion actually occurs to allow the object mask for frame T+N to match the disoccluded object with region 42′ in future frame T+2N.
The motion-compensated frame from step 702 is subtracted from the original frame to produce the displaced frame difference (DFD), step 704. Differences outside the predicted object mask that are greater than a threshold are considered to be newly uncovered regions, since regions that suddenly appear out of nowhere cannot be matched and produce large differences. A binary mask can be made of these uncovered regions. These new regions may really be part of the object, or they may be noise.
Various filtering can optionally be applied, step 706, to remove small regions that may represent noise rather than newly uncovered regions. For example, a 5-by-5 pixel kernel can be used for median filtering the DFD. Then several opening and closing operations can be performed to try to join smaller regions with larger ones. Regions that are still smaller than a threshold can then be removed from further disocclusion processing.
The filtered differences represent regions that are suspected of being uncovered in frame T+N. These newly uncovered regions are not part of the object mask for frame T+N, which is the first frame processed that disocclusion occurs (disocclusion has not yet occurred in frame T, but the object mask is adjusted for comparison to future frames such as T+2N).
These suspected regions may or may not represent actual disocclusions. To determine whether an actual disocclusion has occurred, a motion similarity test is used. The motion of each suspect region is compared to the motion of the tracked object. If the region's motion is similar to the object motion, the region is included in the object mask (disocclusion). If the region's motion is not similar to the object motion, the region is not included in the object mask (no disocclusion). Dissimilar motions indicate noise.
The forward motion vectors between frames T+N and T+2N were generated for the object in step 802 of
Since adding a suspected uncovered region into the object mask can eventually result in tracking the wrong object if the suspect region is really from a different object, stricter requirements can be used for disocclusion than for occlusion processing. In particular, a special test is included for disocclusion processing. The object mask is enlarged by a certain amount, such as by 30% or 4-5 pixels. Then suspected uncovered regions that lie outside the enlarged object mask are removed from further processing, since they lie too far from the object. Suspected regions within the enlarged object mask are tested using the motion similarity test with the object clusters, step 712.
Each suspect region's motion vector is compared to the centroid motion vectors for all clusters in the object in step 712. The absolute-value difference between the suspect region's motion vector and the cluster motion vector is calculated for all combinations of suspect regions and object clusters. For each suspect region, the object cluster having a centroid motion vector that has the smallest difference with the region's motion vector is chosen as the best-fit cluster. The best-fit cluster is the object cluster than has a motion that most closely matches the motion of the suspected uncovered region.
When the smallest difference is greater than a threshold value, then the differences in motion are too large to include the suspect region in the object mask. However, when a suspect region's smallest difference is below the threshold value, then the motions of the suspect region and best-fit cluster of the object are close enough to include the suspect region in the object mask. These suspected uncovered regions with motions that are close enough to one of the object clusters are added into the object mask for frame T+N, step 714. Suspect regions with dissimilar motions or that are too far from the object are not added to the object mask.
The predicted shape of object 500 changes between frames T+N and T+2N. Since the motion of cluster 504 is slightly more upward than for cluster 502, cluster 504′ grows upward in object 500′. Likewise the motion of cluster 506 is slightly more downward than for cluster 502, so cluster 506′ has a downward extension in object 500′. These changes to the shape of object 500 are predicted by motion vectors of macroblocks in object 500. Such changes in the shape of the object mask are detected before occlusion and disocclusion processing, such as by adaptive block matching (step 172 of
New region 501 of object 500 does not have corresponding macroblocks in object 500 that can be matched during motion compensation. Instead, new region 501 seems to appear out of nowhere, being a newly uncovered region. Such disoccluded regions can occur due to movement away of a blocking object, such as will eventually occur in future frames as blocking object 580 moves past object 500′. However, in this example, new region 501 appears due to non-translational motion of object 500. For example, as a fish swims in the x direction, it waves its tail back and forth in the z direction. The fish's tail may suddenly re-appear due to this twisting and rotational motion of the fish's body.
The object mask in frame T+N, Pobj T+N, is adjusted to remove all suspect regions that may be covered in frame T+2N. A displaced frame difference (DFD) between frames T+N and T+2N produces a large difference for the left-most part of cluster 502, since it matches covered region 582 in frame T+2N. This suspect region is removed from the object mask in frame T+N to produce the new object mask 512, known as Pnew
Backward motion vectors for the object between frames T and T+N are read or generated. The motion vectors for the object within new object mask 512 (without the suspect regions) are then clustered. The optimal grouping of motion vectors produces three clusters 502, 504, 506. The centroid motion vector for cluster 504 is slightly more upward than the centroid motion vector for cluster 502, while the centroid motion vector for cluster 506 is slightly more downward than the centroid for cluster 502.
When motion vector 584 is compared to the centroid motion vector for cluster 502, the magnitude and direction differ by a small amount, less than the threshold. Since this difference with best-match cluster 502 is smaller than the threshold, covered region 582′ is classified as being part of object 500. Since region 582 is later obscured in frame T+2N, it is removed as an obscured region that is not part of object 500′.
For disocclusion processing, the displaced frame difference (DFD) is again performed, but between frames T and T+N rather than T+N and T+2N. This time only regions outside of updated object mask 514, after occlusion processing, are considered.
In one embodiment, dividing of blocks is stopped when the brightness (luminance) or color (chrominance) of a block is relatively uniform. The gradient of YUV or just Y is a measure of the uniformity of color and brightness, respectively. The Y gradient of the block is measured and compared to a gradient threshold, step 144. When the gradient is below the gradient threshold, the block is relatively uniform in brightness. Further sub-dividing of the block is halted. Instead the object contour is copied from the matching block of frame T to the block in frame T+N, step 146. The contour information is copied even when the block is a larger 8×8or 16×16 block.
Halting block dividing when the gradient is small helps to minimize errors. When the block's gradient is small and the color or brightness is uniform, the pixels often can match many other blocks since there is little uniqueness in the block's pattern that can be matched. This lack of a larger gradient and a distinct pattern can cause aliasing errors because the low-gradient block may not produce accurate matches during motion estimation.
Often the edge of an object has a sharp change in color or brightness, while blocks within an object or in the background have a relatively uniform color or brightness. Thus the color or brightness gradient across a block is an indication of whether the object boundary passes through the object. Thus a secondary reason to halt further dividing of a block with a low gradient is because the block may not really contain the object boundary.
When a sufficiently large gradient is found within the block, step 144, the block is divided into smaller sub-blocks, step 148. For example, a 16×16 macroblock can be divided into four 8×8 sub-blocks, while an 8×8 block can be divided into four 4×4 sub-blocks. Dividing into other size blocks or regions such as triangles could also be substituted.
The newly-divided sub-blocks in frame T+N are then each motion estimated. A restricted search range in frame T helps to reduce aliasing errors that can arise from the reduced number of pixels in the smaller sub-block. The best-matching sub-block in frame T+N is found for each of the new sub-blocks, step 150. When the matching sub-block is within the object contour of frame T, the sub-block in frame T+N is added to the object mask being refined for frame T+N, step 152.
Sub-blocks that are uncertain (containing the object boundary) are further processed. When these sub-blocks are already at the minimum block size, such as 4×4, step 156, then the object contour information is copied from the matching sub-block in frame T to the sub-block in frame T+N, step 154. Processing of that sub-block ends and the next block or sub-block can be selected, step 142.
When the sub-block is not at the minimum block size, step 156, then it is checked to see if it is an uncertain sub-block, step 140. The gradient of uncertain sub-blocks can be checked, step 144, and the contour copied when the gradient is too small, step 146. For sub-blocks with a sufficiently large gradient, step 144, the sub-block can be further sub-divided, step 148, and motion estimation repeated on the smaller sub-block, step 150.
Sub-blocks having matches within the object contour are certain sub-blocks and added to the object mask, step 152, while uncertain sub-blocks can be further subdivided if not yet at the minimum block size, step 156. When these sub-blocks are already at the minimum block size, such as 4×4, step 156, then the object contour information is copied from the matching sub-block in frame T to the sub-block in frame T+N, step 154. Processing of that sub-block ends and the next block or sub-block can be selected, step 142. More detail and examples of adaptive-block matching are provided in the parent application.
Several other embodiments are contemplated by the inventors. A block or region can be marked or added to the object mask in a wide variety of ways, such as by setting a bit in a memory, or by adding a pointer, identifier, or address of the block to a list of blocks within the object mask, or by expanding a contour or bound of the object, etc. Object contours can be line segments along the object perimeter, or pixels along the perimeter, or can be defined in other ways. For example, the area within the contour may be stored as an object mask, either including the perimeter or excluding the perimeter, or all pixels within the object's predicted contour can be stored.
The variance minimized by clustering can be a sum of squared differences, absolute values, etc. The variance may not be at a true minimum value when the number of iterations is limited. Nerveless, the minimum obtained may be useful, even though not an absolute minimum but only a minimum of the iterations tested in a limited suite of possibilities.
When very little motion occurs, such as for a stationary object, tracking may be difficult. Problems can also occur when both the object and background have similar motions. These situations may be detected and disocclusion processing disabled to prevent errors.
Macroblock matching can compare differences in all color components such as YUV or RGB, or can just compare one or two components such as luminance Y. Gradients can likewise be calculated using all components YUV or just Y. Different search ranges and methods can be used when searching for the best-matching macroblock. For example, a diamond-shaped search pattern or a 3-point pattern may be more efficient than exhaustively searching a square region. Different search strategies can be used to further speed up the computation.
The gradient of a block can be defined in a variety of ways, such as the difference between the largest Y value and the smallest Y value, or the standard deviation of Y values in a block, or variance of Y values or color values, or other functions such as an energy function of the gradient. The gradient can be calculated for every pixel in the image. The gradient can be calculated along both the row and the column for every pixel. Since this produces a gradient value for every pixel, the average gradient for the block can be computed from the individual pixel gradients. Two averages can be used, such as an average gradient across the row and an average gradient across the column. These two gradient values can then be summed and divided by the number of pixels to give the average gradient for the block. Entropy or randomness measures can also be used as the gradient when deciding when to halt block dividing.
The direction of the video sequence could be reversed, and forward motion estimation or even bi-directional motion estimation could be substituted for backward motion estimation. Some frames may be forward estimated while others backward estimated. Frames that do not have motion vectors already generated could be skipped when the compression is performed before object tracking, or when a compressed video sequence is used as the input.
The methods may be applied to object tracking on an RGB or YUV-pixel video stream prior to compression by a standard such as MPEG-4. The methods may also be applied to content-retrieval applications using standards such as H.26L. Object tracking requires much less computational load since segmentation and watershed computations do not have to be performed on all frames. Only the very first frame in a long sequence of frames may need to be segmented to locate the object or objects to be tracked. Alternately, when very high motion occurs between two consecutive frames, then re-segmentation can be performed. Re-segmentation can also be performed on scene changes.
The occlusion and dis-occlusion routines can be varied and implemented in many ways. Optical flow is computationally expensive. Computational expense can be reduced by using block motion vectors. Adaptive block size minimizes blocking artifacts, which can otherwise limit the use of block-based methods.
Different numbers of frames can be skipped during modulation. For example, the number of frames before then next object mask is generated, N, can be set to values other than 3, such as 2 or 5 or many other values. Multiple thresholds can be used, such as adding a second very-low motion threshold that sets N to 10 while motions above the very-low motion threshold but below the regular threshold set N to 3. The motion-similarity thresholds could be adjusted depending on the motion speed or on the type of video sequence, (bright, dark, cluttered, sparse, interview, TV show, surveillance camera, etc.) or on a test of background or other object motions, or by other means.
Object contours can be line segments along the object perimeter, or pixels along the perimeter, or can be defined in other ways. For example, the area within the contour may be stored as an object mask, either including the perimeter or excluding the perimeter, or all pixels within the object's predicted contour can be stored.
The order of the steps can be varied, and further routines, selections, and categories can be added, such as for certain background and uncertain background, or even several kinds of background or secondary objects. Steps in program or process flows can often be re-arranged in order while still achieving the same or similar results.
For example, three possible modules that could be used for occlusion detection are:
Module 1: Clustering of previous frames results in similar backward prediction motion vectors.
Module 2: Clustering of future frames results in dissimilar backward prediction motion vectors.
Module 3: Energy of forward prediction of current frames is high.
In principle, any two of the modules described above could be used for occlusion detection. The motion vectors of the clusters can be compared to the average motion vectors of the suspect region in the description. However, the motion vectors of the clusters could be compared to each other directly or to a motion vector from a previous frame. Occlusion procedures can be reversed in time and used for disocclusion detection.
It is not necessary to process all macroblocks in frame T+N. For example, only a subset or limited area of each frame could be processed. It may be known in advance that the object only appears in a certain area of the frame, such as a moving car only appearing on the right side of a frame captured by a camera that has a highway on the right but a building on the left. The “frame” may be only a subset of the still image captured by a camera or stored or transmitted.
While the invention has been described in simplified terms as tracking foreground objects, any object may be tracked, whether a foreground or a background object. The background may consist of many objects moving in different directions.
While macroblocks such as 16×16 blocks and 8×8, and 4×4 sub-blocks have been described, other block sizes can be substitutes, such as larger 32×32 blocks, 16×8 blocks, etc. Non-square blocks can be used, and other shapes of regions such as triangles, circles, ellipses, hexagons, etc., can be used as the region or “block”. Adaptive blocks need not be restricted to a predetermined geometrical shape. For example, the sub-blocks could correspond to content-dependent sub-objects within the object. Smaller block sizes can be used for very small objects for motion estimation and generating the average motion.
The abstract of the disclosure is provided to comply with the rules requiring an abstract, which will allow a searcher to quickly ascertain the subject matter of the technical disclosure of any patent issued from this disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. 37 C.F.R. § 1.72(b). Any advantages and benefits described may not apply to all embodiments of the invention. When the word “means” is recited in a claim element, Applicant intends for the claim element to fall under 35 USC § 112, paragraph 6. Often a label of one or more words precedes the word “means”. The word or words preceding the word “means” is a label intended to ease referencing of claims elements and is not intended to convey a structural limitation. Such means-plus-function claims are intended to cover not only the structures described herein for performing the function and their structural equivalents, but also equivalent structures. For example, although a nail and a screw have different structures, they are equivalent structures since they both perform the function of fastening. Claims that do not use the word means are not intended to fall under 35 USC § 112, paragraph 6. Signals are typically electronic signals, but may be optical signals such as can be carried over a fiber optic line.
The foregoing description of the embodiments of the invention has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US5635986||Jun 27, 1996||Jun 3, 1997||Daewoo Electronics Co., Ltd||Method for encoding a contour of an object in a video signal by using a contour motion estimation technique|
|US5936671||Jul 2, 1997||Aug 10, 1999||Sharp Laboratories Of America, Inc.||Object-based video processing using forward-tracking 2-D mesh layers|
|US5940538||Aug 5, 1996||Aug 17, 1999||Spiegel; Ehud||Apparatus and methods for object border tracking|
|US5946043||Dec 31, 1997||Aug 31, 1999||Microsoft Corporation||Video coding using adaptive coding of block parameters for coded/uncoded blocks|
|US6075875||Sep 30, 1996||Jun 13, 2000||Microsoft Corporation||Segmentation of image features using hierarchical analysis of multi-valued image data and weighted averaging of segmentation results|
|US6137913||Oct 8, 1998||Oct 24, 2000||Electronics And Telecommunications Research Institute||Method for segmenting moving picture objects by contour tracking|
|US6169573||Jul 3, 1997||Jan 2, 2001||Hotv, Inc.||Hypervideo system and method with object tracking in a compressed digital video environment|
|US6192156||Apr 3, 1998||Feb 20, 2001||Synapix, Inc.||Feature tracking using a dense feature array|
|US6236680||Feb 16, 2000||May 22, 2001||Samsung Electronics Electronics Co., Ltd.||Encoding and decoding system of motion image containing arbitrary object|
|US6272253||Oct 25, 1996||Aug 7, 2001||Texas Instruments Incorporated||Content-based video compression|
|US6298170||Jan 14, 1997||Oct 2, 2001||Fujitsu Limited||Image tracking apparatus for tracking an image within a local region to continuously track a moving object|
|US6335985||Dec 30, 1998||Jan 1, 2002||Kabushiki Kaisha Toshiba||Object extraction apparatus|
|US6337917||Jun 21, 1999||Jan 8, 2002||Levent Onural||Rule-based moving object segmentation|
|US6389168||Oct 13, 1998||May 14, 2002||Hewlett-Packard Company||Object-based parsing and indexing of compressed video streams|
|US6393054||Apr 20, 1998||May 21, 2002||Hewlett-Packard Company||System and method for automatically detecting shot boundary and key frame from a compressed video data|
|US6400846||Jun 4, 1999||Jun 4, 2002||Mitsubishi Electric Research Laboratories, Inc.||Method for ordering image spaces to search for object surfaces|
|US6424370||Oct 8, 1999||Jul 23, 2002||Texas Instruments Incorporated||Motion based event detection system and method|
|US6466624||Oct 13, 1999||Oct 15, 2002||Pixonics, Llc||Video decoder with bit stream based enhancements|
|US6625333||Aug 6, 1999||Sep 23, 2003||Her Majesty The Queen In Right Of Canada As Represented By The Minister Of Industry Through Communications Research Centre||Method for temporal interpolation of an image sequence using object-based image analysis|
|US6985172 *||Jan 25, 2002||Jan 10, 2006||Southwest Research Institute||Model-based incident detection system with motion classification|
|US7095786||Jan 11, 2003||Aug 22, 2006||Neo Magic Corp.||Object tracking using adaptive block-size matching along object boundary and frame-skipping when object motion is low|
|US7142600||Apr 21, 2003||Nov 28, 2006||Neomagic Corp.||Occlusion/disocclusion detection using K-means clustering near object boundary with comparison of average motion of clusters to object and background motions|
|US7342963 *||Aug 7, 2001||Mar 11, 2008||France Telecom||Method for calculating an image interpolated between two images of a video sequence|
|US20040090523||Jun 26, 2002||May 13, 2004||Tetsujiro Kondo||Image processing apparatus and method and image pickup apparatus|
|US20040091047||Nov 11, 2002||May 13, 2004||Sony Corporation||Method and apparatus for nonlinear multiple motion model and moving boundary extraction|
|US20050213660||May 9, 2005||Sep 29, 2005||Sony Corporation||Method and apparatus for nonlinear multiple motion model and moving boundary extraction|
|1||D. Schonfeld and D. Lelescu "VORTEX: Video Retrieval and Tracking From Compressed Multimedia Databases-Multiple Object Tracking From MPEG-2 Bit Stream"; Journal of Visual Communication and Image Representation 11, pp. 154-182, 2002.|
|2||D. Schonfeld and D. Lelescu "VORTEX: Video Retrieval and Tracking From Compressed Multimedia Databases—Multiple Object Tracking From MPEG-2 Bit Stream"; Journal of Visual Communication and Image Representation 11, pp. 154-182, 2002.|
|3||D. Schonfeld and D. Lelescu, "VORTEX: Video retrieval and tracking from compressed multimedia databases-multiple object tracking from MPEG-2 bit stream"; Journal of Visual Communications and Image Representation, Special Issue on Multimedia Database Management, vol. 11, pp. 154-182, 2000.|
|4||D. Schonfeld and D. Lelescu, "VORTEX: Video retrieval and tracking from compressed multimedia databases—multiple object tracking from MPEG-2 bit stream"; Journal of Visual Communications and Image Representation, Special Issue on Multimedia Database Management, vol. 11, pp. 154-182, 2000.|
|5||Eickeler et al., "Content-Based Indexing of Images and Video using face Detection and Recognition Methods".|
|6||IEEE Int. Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2001 (4pp).|
|7||Journal of Visual Communications and Image Representation, Special Issue on Multimedia Database Management, vol. 11, pp. 154-182, 200 (50pp).|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US8823771 *||Mar 23, 2009||Sep 2, 2014||Samsung Electronics Co., Ltd.||Image processing apparatus and method|
|US8934055 *||Jun 14, 2013||Jan 13, 2015||Pixelworks, Inc.||Clustering based motion layer detection|
|US9436996 *||Jul 5, 2013||Sep 6, 2016||Noritsu Precision Co., Ltd.||Recording medium storing image processing program and image processing apparatus|
|US20100091092 *||Mar 23, 2009||Apr 15, 2010||Samsung Electronics Co., Ltd.||Image processing apparatus and method|
|US20130129314 *||Nov 23, 2012||May 23, 2013||Lg Electronics Inc.||Digital video recorder and method of tracking object using the same|
|US20140016815 *||Jul 5, 2013||Jan 16, 2014||Koji Kita||Recording medium storing image processing program and image processing apparatus|
|US20140368612 *||Aug 29, 2014||Dec 18, 2014||Samsung Electronics Co., Ltd.||Image processing apparatus and method|
|International Classification||H04N7/12, H04N11/02, H04B1/66, H04N11/04|
|Cooperative Classification||H04N19/57, H04N19/61, H04N19/20, H04N19/51, G06T7/2006|
|European Classification||H04N7/26J, H04N7/50, G06T7/20A, H04N7/26M2, H04N7/26M4V|
|Jan 13, 2011||AS||Assignment|
Owner name: NEOMAGIC CORP., CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SCHONFELD, DAN;HARIHARAKRISHNAN, KARTHIK;RAFFE, PHILIPPE;AND OTHERS;SIGNING DATES FROM 20030425 TO 20030625;REEL/FRAME:025636/0854
Effective date: 20080213
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:NEOMAGIC CORPORATION;REEL/FRAME:025637/0001
Owner name: FAUST COMMUNICATIONS HOLDINGS, LLC, DELAWARE
|Jul 22, 2011||AS||Assignment|
Owner name: INTELLECTUAL VENTURES I LLC, DELAWARE
Free format text: MERGER;ASSIGNOR:FAUST COMMUNICATIONS HOLDINGS, LLC;REEL/FRAME:026636/0268
Effective date: 20110718
|Jun 19, 2012||CC||Certificate of correction|
|Apr 24, 2014||FPAY||Fee payment|
Year of fee payment: 8