|Publication number||US7587064 B2|
|Application number||US 11/051,860|
|Publication date||Sep 8, 2009|
|Filing date||Feb 3, 2005|
|Priority date||Feb 3, 2004|
|Also published as||US20050169529|
|Publication number||051860, 11051860, US 7587064 B2, US 7587064B2, US-B2-7587064, US7587064 B2, US7587064B2|
|Inventors||Yuri Owechko, Swarup Medasani, Narayan Srinivasa|
|Original Assignee||Hrl Laboratories, Llc|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (5), Non-Patent Citations (21), Referenced by (43), Classifications (8), Legal Events (2)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This application is a non-provisional application, claiming the benefit of priority to provisional application No. 60/541,554, filed in the United States on Feb. 3, 2004, and titled “Method for Fingerprinting of Objects in Dynamic Environments.”
The present invention relates to an object recognition system and, more particularly, to a system that enables fingerprinting of objects in a dynamic environment.
Surveillance systems typically consist of a camera and monitor. Sensors are often added to notify a user if a person or object has come within the range of view of the sensor. The problem with such a system however, is that it does not identify the object or recognize the object as having been previously identified. Fingerprinting an object is the ability to identify an object based on unique features of the object. Because objects move through environments in a dynamic fashion and because of changing views and occlusion, fingerprinting objects is inherently a dynamic problem.
While there have been methods for tracking and recognizing objects from static images, there have been none that address the problem of tracking and recognizing in dynamic environments where the objects and the sensors that track and sense them are moving simultaneously. Additionally, there are currently no methods for acquiring fingerprint models of objects in an incremental and automated fashion under dynamically varying environments.
Thus, a continuing need exists for a system that allows for object identification, tracking, and fingerprinting in a dynamic environment.
The present invention relates to an active learning system for fingerprinting an object in an image frame. The system comprises an oriented localized filter module configured to extract features from a potential object candidate within the image frame; a learning-feature graph-fingerprinting module configured to receive the features and build a fingerprint of the object for tracking the object; and an object graph prototype module for receiving, storing, and transmitting object fingerprint information to the learning-feature graph-fingerprinting module to be updated as new features are received.
In another aspect, the system further comprises a flow-based object segmentation module for segmenting a potential object candidate from a sequence of video frames taken from a scene, and communicating the segmented potential object candidate to the oriented localized filter module for iconic feature extraction; and an incremental learning module for predicting a general class of the potential object candidate.
In yet another aspect, the system further comprises a fixed-basis function decomposition module using Haar wavelets to extract a relevant feature set from the potential object candidate; a static classifier for initial/temporary classification of the potential object candidate, thereby generating a classification category selected from a group consisting of a positive identification, a negative identification, a false positive identification, and a false negative identification, and where upon classification, the static classifier communicates the classification to the incremental learning module.
In another aspect, the oriented localized filter module is configured to receive an object model of the object candidate and compute points on the object from different views to form a feature vector at each computed point. The feature vector is based on local intensity variations present in an image region around the point as seen from a given view, such that a number of points that form the object model is selected based on local photometric energy at each point. The point is retained as an actual object point if its photometric energy is greater than a threshold, where the threshold is computed based on the photometric energy associated with points within the image region.
In another aspect, the learning-feature graph-fingerprinting module forms a fuzzy graphical model (FGM) of object candidates to fingerprint the object by representing three-dimensional (3D) spatial relationships between salient object points and corresponding two-dimensional (2D) view-dependent feature vectors.
Furthermore, the FGM utilizes an algorithm for sub-graph matching to minimize an objective function, wherein the objective function is defined by:
Additionally, after the fuzzy graph model is formed, the system is configured to initiate visual tracking and track an object candidate through:
In another aspect, the flow-based object segmentation module is configured to:
In yet another aspect, the incremental learning module includes a learning-based cascaded classifier framework for predicting the general class of an object. The cascaded classifier includes a pre-learning cascade and a post-learning cascade, wherein the pre-learning cascade includes a front-end multi-layer back-propagation neural network and an initial back-end incremental learning algorithm, and wherein the post-learning cascade includes the multi-layer back-propagation neural network and an updated incremental learning algorithm.
Additionally, the multi-layer back-propagation neural network is a Non-linear Discriminant Analysis (NDA) classifier, and the back-end incremental learning algorithm is a Modified-Probabilistic Adaptive Resonance Theory (M-PROBART) classifier.
Furthermore, the fixed-basis function decomposition module is configured to learn a relevant feature set for a class of objects based on an example set of images. The example set of images is transformed from image space to wavelet space utilizing an over-complete basis-set of Haar wavelets that respond to local intensity differences at a plurality of orientations and scales, where the wavelet space operates as the relevant feature set used to train the static classifier.
In yet another aspect, the fixed-basis function decomposition module is configured to transform the example set of images from image space to wavelet space through operations of:
Finally, as can be appreciated by one in the art, the present invention also comprises a method and computer program product for performing the operations of the invention described herein.
The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fees.
The objects, features and advantages of the present invention will be apparent from the following detailed descriptions of the various aspects of the invention in conjunction with reference to the following drawings, where:
The present invention provides a learning system for object fingerprinting, which may be tailored to a variety of applications. The following description is presented to enable one of ordinary skill in the art to make and use the invention and to incorporate it in the context of particular applications. Various modifications, as well as a variety of uses in different applications will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to a wide range of embodiments. Thus, the present invention is not intended to be limited to the embodiments presented, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
In the following detailed description, numerous specific details are set forth in order to provide a more thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without necessarily being limited to these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention.
The reader's attention is directed to all papers and documents which are filed concurrently with this specification and which are open to public inspection with this specification, and the contents of all such papers and documents are incorporated herein by reference. All the features disclosed in this specification, (including any accompanying claims, abstract, and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features.
Furthermore, any element in a claim that does not explicitly state “means for” performing a specified function, or “step for” performing a specific function, is not to be interpreted as a “means” or “step” clause as specified in 35 U.S.C. Section 112, Paragraph 6. In particular, the use of “step of” or “act of” in the claims herein is not intended to invoke the provisions of 35 U.S.C. 112, Paragraph 6.
Before describing the invention in detail, first a glossary of terms used in the description and claims is given as a central resource for the reader. Next, a description of various principal aspects of the present invention is provided. Third, an introduction is provided to provide the reader with a general understanding of the present invention. Finally, a detailed description of various aspects of the present invention is provided to give an understanding of the specific details.
Before describing the specific details of the present invention, a glossary is provided in which various terms used herein and in the claims are defined. The glossary provided is intended to provide the reader with a general understanding for the intended meaning of the terms, but is not intended to convey the entire scope of each term. Rather, the glossary is intended to supplement the rest of the specification in more clearly explaining the terms used.
C5 Decision Tree Learning Algorithm—The term “C5 decision tree learning algorithm” refers to an algorithm that learns sets of if-then rules from training data for discovering patterns that delineate categories, assembling them into classifiers, and using them to make predictions.
Cascaded Classifier—The term “cascaded classifier” refers to a sequence of classifiers wherein classifiers later in the sequence are used if the earlier classifier predicts the test object to be of a certain class.
Fuzzy Graph Model—The term “fuzzy graph model” means representing and modeling a directed attributed graph using fuzzy sets. The node attributes and edge attributes are modeled using fuzzy sets and membership functions.
Gaussian Derivative—The term “Gaussian derivative” refers to the Nth-order derivative of an N-D Gaussian. In our invention, we use up to a maximum of three orders.
Haar Wavelet—The term “Haar wavelet” refers to a piecewise constant function. A multiresolution of such functions forms the Haar wavelet basis.
Harris Corner Detector—The term “Harris corner detector” refers to algorithm that uses local image structure to detect corners and interest points in an image. The local structure is first computed from derivatives of the image intensity and then smoothed. A corner magnitude is used to select dominant corners in an image.
Heuristic Models—The term “heuristic models” refers to a problem-solving technique in which the most appropriate solution of several found by alternative methods is selected at successive stages of a program for use in the next act of the program. As applied to the present invention, a heuristic model can be used to estimate the dimensions of an object in an image.
Hough Transform—The term “Hough transform” refers a technique used in image analysis that allows recognition of global patterns in an image space by recognition of local patterns (ideally a point) in a transformed parameter space.
Instruction Means—The term “instruction means” as used with respect to this invention generally indicates a set of operations to be performed on a computer, and may represent pieces of a whole program or individual, separable, software modules. Non-limiting examples of “instruction means” include computer program code (source or object code) and “hard-coded” electronics (i.e. computer operations coded into a computer chip). The “instruction means” may be stored in the memory of a computer or on a computer readable medium such as a floppy disk, a CD-ROM, and a flash drive.
Kullback Information Measure—The term “Kullback Information Measure” refers to a measure of information from statistical point of view involving two probability distributions.
Lucas-Kanade—The term “Lucas-Kanade (LK)” refers to a technique for solving a linear system using information around a good feature point for computing the optical flow in an image.
Modified-Probabilistic Adaptive Resonance Theory (M-PROBART) Classifier—The term “M-PROBART Classifier” refers to a function approximation approach that achieves superior generalization over other adaptive resonant theory (ART) based approaches.
Non-linear Discriminant Analysis (NDA) Classifier—The term “NDA Classifier” refers to an approach of using multilayer perceptrons to extract discriminating parameters between classes of objects. This is achieved by setting the number of nodes in the last hidden layer to a small number thus decreasing input feature redundancy and extracting relevant information for classification.
Pyramidal Lucas-Kanade—The term “Pyramidal Lucas-Kanade (LKP)” refers to a technique that expands upon the LK approach, utilizing a pyramid of the input frames to compute flow at different levels using a fixed size neighborhood, where flow computed at coarser levels is then propagated to finer resolution levels. At each level, the LK algorithm is used to compute the flow field and to propagate the flow value to the next level. This process is repeated for all of the good feature points resulting in the flow field.
Wavelet—The term “wavelet” refers to mathematical functions that cut up data into different frequency components and then study each component with a resolution matched to its scale.
Wavelet Basis Functions—The term “wavelet basis functions” refers to the collection of shifted and dilated wavelet functions of a particular family of wavelets.
Wavelet Filter—The term “wavelet filter” refers to using wavelet basis functions for filtering signals.
(2) Principal Aspects
The present invention has three “principal” aspects. The first is an active learning system. The active learning system is typically in the form of a computer system operating software or in the form of a “hard-coded” instruction set. This system may be incorporated into a wide variety of devices that provide different functionalities. The second principal aspect is a method, typically in the form of software, operated using a data processing system (computer). The third principal aspect is a computer program product. The computer program product generally represents computer readable code stored on a computer readable medium such as an optical storage device, e.g., a compact disc (CD) or digital versatile disc (DVD), or a magnetic storage device such as a floppy disk or magnetic tape. Other, non-limiting examples of computer readable media include hard disks, read-only memory (ROM), and flash-type memories. These aspects will be described in more detail below.
A block diagram depicting the components of an active learning system of the present invention is provided in
An illustrative diagram of a computer program product embodying the present invention is depicted in
The present invention relates to an active learning system for object fingerprinting. The problem of tracking and fingerprinting objects is inherently a dynamic problem because objects move through environments in a dynamic fashion, and because of changing views and occlusion. Fingerprinting an object is the ability to identify an object based on unique features of the object. For example, if the object appears in other environments in familiar views, or in some cases views not perceived in the past, the object can still be recognized (identified) based on its features.
The present invention utilizes an integrated approach to perform tracking and fingerprinting of objects, wherein the features used for tracking are also used for fingerprinting. The system is adaptive and capable of rapidly modifying object models to changing views and false positives that may affect system performance. Using an incremental learning approach and fuzzy graph models, the system provides for robust confirmatory identification and tracking of individual three-dimensional (3D) video objects in a 3D environment. The system can be used in a variety of security, surveillance and tracking systems.
Architecture for the present invention is shown in
Since the space of non-object examples is practically infinite, the static classifier 306 is bound to make mistakes on some examples of either object images or non-images, thereby generating false positives and false negatives. To correct for these mistakes, the present invention optionally utilizes an oracle memory 308 that is capable of providing and verifying the correct class labels for every object candidate. An option for an oracle memory 308 is to insert a human in the loop, where the human compares the image to the classification provided by the static classifier 306 and thereafter labels the classification as either a positive, negative, false-positive, or false-negative classification. Another choice is to use an object tracker module, which contains the track list of current objects, with a priori knowledge that it is extremely accurate under some conditions. For example, a vehicle classifier could be used to identify the vehicle when the vehicle is seen from the front or back at close range. Using the correct class labels from the oracle 308, the system is capable of incrementally reshaping the object class decision boundaries using an incremental learning module 310. In this manner, the system can incrementally learn view-based models of objects. Once the incremental object classification learning module 310 corrects the mistakes, the system then proceeds to extract iconic features from the potential object candidate using an oriented localized filter module 312. Using the iconic features, the system continues to build and refine fingerprint models for the object candidate utilizing a learning-feature graph-fingerprinting module 314. The learning-feature graph-fingerprinting module 314 is configured to use fuzzy graphical models (FGM) of object candidates to fingerprint the objects. Additionally, an object graph prototype module 316 is included for receiving, storing, and transmitting object fingerprint information (object graph prototypes) to the learning-feature graph-fingerprinting module 314 to be updated as new features are received.
As shown in
Visual tracking can be divided into two types of approaches, bottom-up and top-down. Bottom-up approaches generally tend to construct target states by analyzing the content of images. Top-down approaches generate candidate hypotheses of the target state from previous time frames based on some parametric representation of the target. The present invention combines both the bottom-up and top-down approaches to achieve robust tracking.
The tracking approach of the present invention is partitioned into four components: target state representation, observation representation, hypotheses measurement, and hypotheses generation. The target state representation for the approach of the present invention is based on two different feature types. The first feature type is the 3D locations of all the salient points of an object as encapsulated by the corresponding 3D graph model. The second feature type will be a colored iconic representation at each of these 3D locations based on local intensity measurements. Observation representation is also based on the 3D model and consists of the two feature types as discussed above. Hypotheses measurement evaluates the matching between target state hypotheses and their image observations in the current frame. Hypothesis generation produces a new state hypothesis based on the old estimation of the target's representation and old observation, which implies the evolution of the dynamic process. Using the target and observation representation as described above, a two-scale matching strategy is adopted to perform hypothesis measurement. At the finer scale, the present invention matches the individual 3D nodes based on normalized cross-correlation between the colored iconic representation of the target node in the prior frame with the observed state of the 3D node being matched in the current frame. At a coarser scale, the fuzzy graph matching technique is utilized to match a tracked fuzzy sub-graph from the prior frame with nodes matched at the finer scale for the current frame. The coarse scale essentially matches the entire target as a single entity.
This approach provides tracking and fingerprinting capabilities in cluttered environments. Since the objects to be tracked and fingerprinted are moving dynamically, it is imperative that the system be adaptive and capable of rapidly modifying object models to changing views and false positives that may affect system performance. This adaptability is achieved by using an incremental learning framework that reacts to changing environments as well as changing object shapes and views, and adapts the object models that are being tracked and fingerprinted in an on-line fashion. As the objects are tracked, new object views become visible and the incremental learning module is used to add new nodes and views to the FGMs representing each object. Once a 3D FGM is formed for an individual object, new virtual 2D views of the individual object can be easily generated for tracking purposes using video cameras in different locations and views. In addition, if necessary, the FGM representation allows tracking to be done using a single video camera. This approach results in accurate confirmatory identification and 3D tracking of individual objects.
The learned 3D models can also be used to generate virtual views of objects for active learning. Active learning allows the system to adjust to the current environment and illumination conditions, thereby increasing detection rates and reducing false alarms.
(a) Object Segmentation Using Optical Flow
This section describes the robust and computationally efficient approach to object segmentation using the flow-based object segmentation module. The process for object segmentation corresponds to the process shown and described in U.S. patent application Ser. No. 11/004,501, whose entire disclosure is hereby incorporated by this reference as though fully set forth herein. Observing a moving object using a panning camera results in different apparent motions for the background and foreground object, which reduces the effectiveness of conventional photometric statistical background estimation techniques for object segmentation. Optical flow methods can be used in such situations. However, it can be easily appreciated that optical flow vectors arising from moving objects in a real-world scene observed by a moving camera can be very noisy. The present approach involves using stacked histograms of optical flow vectors to robustly estimate object positions, heights, and widths. The flow patterns induced by the background and other objects in a scene can be approximated in the flow histograms by linear segments. Discovering the segments corresponding to objects enables estimation of the object's geometric properties like height and width. This information is transformed in the raw optical flow maps into a stacked optical flow histogram space that is less noisy, easy to compute, and enables fast and accurate estimation of the positions and widths of objects.
Once the apparent flow field is computed, the 2D flow field is converted into a flow histogram image 504. The optical flow field consists of horizontal and vertical components. Because histograms can be computed in two directions (i.e., horizontal and vertical), a total of four projections are generated.
The Hough transform provides the properties of the linear segment corresponding to the background flow in the scene and helps to eliminate the flow induced by the background. To this effect, simple heuristics are used to delete flow vectors in the flow histogram space that are close to the line segment modeling the background flow. Next, median distances are computed between points in the remaining segments to the background line. Points constituting the segment that are farthest from the background flow are then clustered to find the center and the extent of the object in the x and y directions. The process is not trivial since dense flow is not always guaranteed. Also, poor texture in the object might result in the linear segment corresponding to an object being broken into several smaller segments with holes in between. To account for this, heuristic models are used to accurately estimate the widths of the objects 510. In this approach, the estimates of the object width and height are filtered using 1D Gaussians. The mean and standard deviation of the distributions are adapted as estimates of the object widths and heights become available 512. The filtered estimates of the object height and width, and thereby the position, are then used to segment the object from the scene 514.
The segmented object is then passed to the feature extraction routine (described in section (b) below) to learn the specific representation of the objects in the scene. To further improve the stability of the window position estimates, the system fuses the previous estimate of the object flow vector and the estimate of displacement of the selected iconic feature points across frames to restrict the location of the new window position.
(b) Feature Extraction
The present invention performs salient feature extraction utilizing two different types of features: (1) iconic object-based features; and (2) image-based features. The iconic features are generated using oriented local filters formed from Gaussian derivatives. The image-based features are extracted by analyzing area of interests (AOI) using a fixed set of localized basis functions, such as Haar wavelets or similar functions. Each of the localized basis functions is associated with a fixed location, scale, and orientation in the AOI. Unlike the iconic features, these image-based features have fixed spatial relationships. The iconic features are used to form “bottom-up” models of targets that are built up from local object features, whereas the image features form “top down” models from analysis of the AOI using a fixed set of analysis functions.
(i) Computation of Iconic Features Using an Oriented Localized Filter Module
The iconic features are in the form of a high-dimensional feature vector comprised of the responses of different order derivatives of steerable Gaussian filters at a range of orientations and scales.
The object model is be defined by the iconic representation at a set of points on the object. The present invention takes advantage of the 3D model of the object (as described below) to select the relevant set of points from different views and depths. The iconic feature vector at each computed point will be based on the local intensity variations that may be present in the image region around the point of interest as seen from a given view. The number of points that form the object model is selected based on the local photometric energy or spectral power at each point. A point is deemed to be important if its photometric energy is greater than a threshold that is computed based on the mean and standard deviation of the energy associated with points within the object region.
(ii) Computation of Image Features Using the Fixed Basis Decomposition Module
The Haar wavelet-based approach to object detection is capable of learning a relevant feature set for a class of objects based on an example set of images. The relevant feature set can be used to train a classifier that can accurately predict the class of the object. To incorporate high spatial resolution and to efficiently capture global structure, over-complete/redundant wavelet basis is used. The present invention applies this multi-scale feature extraction approach to various application domains, non-limiting examples of which include automotive occupant sensing for smart airbags and detection of faces, detection of pedestrians, and detection of vehicles for collision avoidance and surveillance applications. For example, face and vehicle detection utilizes visible illumination, while near infrared illumination may be used for the automotive occupant sensing application. As another non-limiting example, both visible and infrared illumination may be used in pedestrian detection applications. Haar wavelet features have proven to be equally effective in both visible and infrared wavelengths.
The present invention utilizes an over-complete basis-set of Haar wavelets that respond to local intensity differences at several orientations and scales. A set of labeled training data from the various occupant classes is used to learn an implicit model for each of the classes. The occupant images used for training are transformed from image space to wavelet space and are then used to train a classifier. For a given image, the wavelet transform computes the response of the wavelet filters over the image. Typically three oriented wavelets, vertical, horizontal, and diagonal, are computed at different scales. The multi-scale approach allows the system to represent coarse as well as fine-scale features. The over-complete representation corresponds to a redundant basis wavelet representation, thereby providing an exceptional spatial resolution. This is accomplished by shifting wavelet templates by some portion of the template. An absolute value of the wavelet coefficients is used to eliminate the differences in features when considering a dark object on a white background, and vice-versa. Additionally, the Haar wavelet features can be implemented very efficiently using the concept of an “integral image.” The integral image is the same size as the AOI. Each point in the integral image is calculated as the sum of all pixels whose row and column indices are less than or equal to those of the point. Once the integral image is calculated, the sum of all pixels inside any rectangular area (i.e., rectangular sub-feature) can be calculated using only four table lookups and four addition operations. These rectangular sub-features can be combined to form the final wavelet features as well as other related features that maximally respond to center-surround or line structures.
(c) Creating Object Models
There are two types of object models that can be constructed from the salient features extracted above. The two types are view-based object models and iconic feature-based fingerprinting models. The first type is used to predict the general class of the object in question and the second type is used to obtain the specific 1D of the object.
An incremental learning module including an incremental learning view-based cascaded classifier is used to predict the general class of an object (e.g., tank, car, truck, pedestrian, etc.). The cascaded classifier has two levels, the first level being a pre-learning cascade such as a front-end multi-layer back-propagation neural network and an initial back-end incremental learning algorithm, and the second level being a post-learning cascade such as a multi-layer back-propagation neural network and an updated incremental learning algorithm. As a non-limiting example, the multi-layer back-propagation neural network is a Fast Non-linear Discriminant Analysis (NDA) classifier, and the back-end incremental learning algorithm is a Modified-Probabilistic Adaptive Resonance Theory (M-PROBART) classifier.
In the iconic-feature-based model, 3D object models are constructed from multiple views by 3D fuzzy attributed relational graphs (FARGs). The node attributes of the fuzzy graph representing an object in a frame are comprised of the iconic features extracted from each view. The edge attributes between neighboring nodes represent fuzzy spatial relations of the type “left-of,” “above,” etc. Objects in this scheme are thus represented using 3D FARGs. Comparing the objects is then done in the graph domain and not in the original image domain. As such, fuzzy graph matching methods (as described below) are used for classification and recognition of 3D fuzzy graph-based object models. The distance metric for ascertaining compatibilities between nodes in two graphs is the normalized cross-correlation score between the respective iconic features. Since the object and/or system may be moving and could have changed its view of the object in the subsequent frame or, be viewed from a different perspective by a different camera or, some parts may be occluded, not all features will match between consecutive frames. The fuzzy graph approach described herein can handle inexact matching by exploiting the tolerance of fuzzy graphs to partial matches during classification.
(i) Incremental Learning View-Based Object Models Using the Incremental Learning Module
This section further describes the incremental learning view-based cascaded classifier framework for predicting the general class of an object. For further illustration, below is an example image as input into the cascade. The cascade then computes the pre-specified Haar wavelet and edge symmetry features, and thereafter classifies the window as belonging to one of the pre-defined classes. The following example and results are for demonstrative purposes only and are not intended to limit the use and scope of invention described herein. Rather, these results are intended to assist the reader in obtaining a better qualitative understanding of the concepts presented herein.
In the following non-limiting example, an incremental learning cascaded classifier is used to detect vehicles in scenes. The cascaded classifier includes a front-end fast multi-layer back-propagation neural network such as a Non-linear Discriminant Analysis (NDA) classifier, and a back-end incremental learning algorithm such as an online learning-capable Modified-Probabilistic Adaptive Resonance Theory (MPROBART) classifier. Use of the MPROBART classifier corresponds to the process described in U.S. Pat. No. 6,456,991, whose entire disclosure is hereby incorporated by this reference as though fully set forth herein.
A 10,446 example feature set was collected from a variety of imagery using camcorders, digital cameras, and wide-angle lens cameras. The data set was partitioned into 5,041 training images and 5,045 test images of front and back views of vehicles. The fast NDA classifier was trained on 142 selected Haar wavelet features. These features were sub-selected from a much larger set by using the Kullback information measure and the C5 decision tree learning algorithm. A slow NDA classifier was also trained on a 173 dimensional feature vector that included the 142 Haar wavelet features with an additional 31 edge symmetry features. The edge symmetry features contain information about the symmetry of the object about horizontal and vertical axes. The Fast NDA classifier has approximately a 95.9% average prediction accuracy on both the vehicle and non-vehicle classes. The M-PROBART classifier was trained on the same dataset as the Fast NDA classifier and was found to have a prediction accuracy of 94.9% on both the vehicle and non-vehicle classes.
To quantify the incremental learning results, a diverse set often images was picked to form a training set. The ten images have varying backgrounds and vehicles of different sizes. The goal was to process each of the ten images using the incremental learning cascaded classifier and then select the false-positive and false-negative points from each of the saliency maps. Features were extracted from 2:3 aspect ratio windows around the selected false positive and false negative points, and then fed to the incremental learning module to learn the erroneous classifications. In the ten images, over 700 false positives were manually selected. The M-PROBART classifier of the active learning cascade starts off with 155 rules. Features from selected points in each of the ten images are fed sequentially to the incremental learning module. The M-PROBART classifier finally ends up with 164 rules after it trains through the new features. To test the generalization capability of the active learning cascaded classifier, four images were selected that were captured using the varying input types, including a digital camera, a camcorder, and a wide-angle lens camera, under widely varying conditions. Seven cars were selected from these four images and passed through the pre-learning cascade and the post-learning cascade classifiers. The pre-learning cascade included the fast NDA and the M-PROBART before active learning was performed, while the post-learning cascade included the fast NDA and an updated M-PROBART. A total of 50,312 windows were processed across the four test images with window heights set to 70 pixels or 100 pixels. The pre-learning cascaded classifier provided 157 core vehicle points and 88 false-alarm points while detecting all the selected seven vehicles. Additionally, the post-learning cascaded classifier detected 159 core vehicle points and only 45 false-alarm points. In effect, the active learning phase reduced the false-positives by almost 50% without any significant reduction in the detection rate.
There are two main reasons for the extendibility of the present approach. First, the approach relies on a static classifier that is trained on a generic feature set generated using the Haar wavelets to act as the primary filter. The feature set can be readily generated for any other class of objects much like the vehicle classifier. Second, the fingerprinting is based on fuzzy graph models that are flexible and robust and can model any object class. Finally, the features at the nodes of the fuzzy graph models are based on Gaussian derivative filters that are only dependent on local image properties, and that can model any other class of objects as well.
(ii) Fuzzy Graph Matching for Iconic Feature-Based Fingerprinting Models Using the Learning Feature Graph Fingerprinting Module
The proposed fuzzy FARG matching algorithm uses ideas from relaxation labeling and fuzzy set theory to solve problems associated with sub-graph isomorphism. The algorithm can handle exact as well as inexact sub-graph matching. The fuzzy FARG matching utilizes an algorithm for sub-graph matching to minimize an objective function. The objective function is defined by:
In equation (1), η is a constant that controls the relative influence of the two terms in the minimization process, cij represents the absolute compatibility between nodes i in graph A (i.e., graph A is stored graph representing a candidate object) and j in graph B (i.e., graph B represents an input object that is matched against graph A) (given the fuzzy assignments U), taking into account the attributes of the edges incident on nodes i and j and those of the neighboring nodes of i and j. In other words, C=[cij] is the compatibility matrix. Additionally, uij is an element of an assignment matrix U, representing a degree to which node i in graph A matched nodej in graph B. Furthermore, n and m are the number of nodes in graphs A and B respectively. The function ƒ( ) is similar to a distance function and takes on high values if cy is low and vice versa. As mentioned above, the compatibilities cij depend on U. Similarly, the assignments U depend on the compatibilities C. U and C are updated in an alternating fashion, giving rise to a relaxation process. The update equations for the FGM algorithm are based on the Lagrange method and use the Karush-Kuhn-Tucker conditions.
The computational complexity of the FGM algorithm can be shown to be O(n2m2). Computing compatibilities is the most computationally expensive part of the FGM algorithm. Note that in this model, the graphs are completely connected, even though the strengths of relations between edges may be zero.
(d) Further Experimental Results
The description above provides examples for the various sub-components of the present invention. Following are experimental results that illustrate the fingerprinting aspect of the present invention. As mentioned previously, the following example and results are for demonstrative purposes only and are not intended to limit the use and scope of invention described herein. Rather, these results are intended to assist the reader in obtaining a better qualitative understanding of the concepts presented herein.
To this effect, two example sequences were used of an un-manned aerial vehicle (UAV)-based camera system tracking a tank and a truck in a desert environment. In these sequences, an active camera is mounted on a UAV and the UAV hovers over a tank and a truck capturing video as they drive through the desert. The video sequence is challenging because there are a lot of elements that change dynamically, such as the location of the UAV (and hence the camera) with respect to the object causing changes in scale and view of the object. In addition, the object is moving which results in changing views and scales as well, with a goal of tracking the objects. These sequences were selected to demonstrate the unique fingerprinting approach described herein.
After the object is segmented using the flow segmentation approach discussed above, the Gaussian derivative filter responses are extracted at all the pixels within the window. In order to limit the number of points on the object, the system then computes the photometric energy (computed as the L2 norm of the filter response) of the points within the segmented window. The mean energy is computed and used as a measure to filter points. In particular, points are filtered at energy levels much below the mean energy, thereby eliminating points with very little photometric energy such as image patches with little or no texture. An additional constraint based on distance between high-energy points is applied. This step prevents selecting points in close proximity, thereby reducing the redundancy in the points selected and improving the richness of the information captured. The filtered points are then used to build a fingerprint (i.e., a fuzzy graph model) for the object being tracked. An aspect of this feature point selection process is that it also provides an approach for adaptively recruiting new points for the model over time to adapt to changes such as different views, scale changes of the object of interest etc. By carefully selecting points using the photometric energy criterion instead of performing random sampling, the system can achieve faster learning rates and better recognition performance than traditional example-based passive learning algorithms.
Following are fingerprinting results obtained for the two video sequences.
The fuzzy graph also possesses edge properties. The edge properties define the spatial relationships between the node locations in the image. In this 2D example, there are four attributes (i.e., above, right, below, and left) for each edge of the fuzzy graph. These attributes define the relative spatial location of a node with respect to all the other member nodes of the fuzzy graph. The attributes for the edges of the fuzzy graph for the tank object in the first frame of
The fuzzy graph of the tank in the first frame (i.e., numeral 1400 shown in
The fuzzy graph matching process is illustrated in
There are two notable situations that may occur during the matching process. The first being that the object may have undergone geometric transformations, while the second is potential object occlusion.
During the matching process, the object may have undergone geometric transformations such as a scale change or a view change and, therefore the fuzzy graph may not obtain a match for all of its member nodes in any given view. However, another result of the fuzzy graph approach is that partial or sub-graph matching is allowed and hence can be used to measure the quality of the fingerprint. Sub-graph matching is also used as a cue by the present invention to recruit new points into the graph to account for the geometric transformation.
The second possibility is that the object may become partially occluded. In this case, the object can still be fingerprinted based on partial matching like before. However, unlike before, it would not be ideal to recruit high-energy points on the occluding object. Because the system may encode a few nodes in the process of building a complete fingerprint model for an object, weighting the nodes that result in matches over longer periods of time will suppress the nodes that do not match on a consistent basis over a large number of frames. This can be readily incorporated into the matching process by assigning a frequency or weight term for each node of the fuzzy graph into the objective function (i.e., equation (1)).
The second example illustrates the fingerprinting process for a truck.
Three tests were performed in order to test the quality of the fingerprint models built for the truck and the tank. In the first test, as shown in
Another exemplary test is shown in
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US5034986 *||Feb 12, 1990||Jul 23, 1991||Siemens Aktiengesellschaft||Method for detecting and tracking moving objects in a digital image sequence having a stationary background|
|US5093869 *||Dec 26, 1990||Mar 3, 1992||Hughes Aircraft Company||Pattern recognition apparatus utilizing area linking and region growth techniques|
|US6456991||Sep 1, 1999||Sep 24, 2002||Hrl Laboratories, Llc||Classification method and apparatus based on boosting and pruning of multiple classifiers|
|US6937744 *||Jun 13, 2000||Aug 30, 2005||Microsoft Corporation||System and process for bootstrap initialization of nonparametric color models|
|US20030204384 *||Apr 24, 2002||Oct 30, 2003||Yuri Owechko||High-performance sensor fusion architecture|
|1||*||"Fuzzy Graph Tracking" by Borotschnig et al., Proc. 5th International Symposium on Intelligent Robotic Systems, H.I.Christensen and C.Brautigam and C.Ridderstrom, Ed., Centre for Autonomous Systems, KTH, 1997.|
|2|| S. Gold and A. Rangarajan, A graduated assignment algorithm for graph matching, IEEE Transaction on Pattern Analysis and Machine Intelligence, vol. 18, pp. 377-387, Apr. 1996.|
|3||B. D. Lucas and T. Kanade, An Iterative Image Registration Technique with an Application to Stereo Vision, Intl. Joint Conf. on Artificial Intelligence, 674-679, 1981.|
|4||B. Horn and B. G. Schunk, "Determining Optic Flow," Artificial Intelligence, vol. 17, pp. 185-203, 1981.|
|5||C. Harris and M. Stephens, A combined corner and edge detector, in Proc. of the 4th ALVEY Vision Conference, pp. 147-151, University of Manchester, English, Sep. 1988.|
|6||C. Papageorgiou, T. Evgeniou and T. Poggio, "A Trainable Pedestrian Detection System," in Proc. of IEEE Intelligent Vehicles Symp., pp. 241-246, Oct. 1998.|
|7||D. Rumelhart and J. McClelland, Parallel Distributed Processing, vol. 1, MIT Press, Cambridge, MA, 1986.|
|8||Faugeras, Three-Dimensional Computer Vision- A Geometric Viewpoint, MIT Press, 1993.|
|9||G. Healey and D. Slater, Global Color Constancy: recognition of objects by use of illumination-invariant properties of color distributions, vol. 11, No. 11, pp. 3003-3010, Nov. 1994.|
|10||J. C. Bezdek, Pattern Recognition With Fuzzy Objective Function Algorithms, Plenum Press, New York, 1981.|
|11||J. Mao and A. K. Jain, "Artificial Neural Networks for Feature Extraction and Multivariate Data Projection," IEEE Trans. on Neural Networks, vol. 6, No. 2, Mar. 1995.|
|12||J. Ross Quinlan: C4.5: Programs for Machine Learning. Morgan Kaufmann 1993.|
|13||*||Medasani et al "A fuzzy approach to content-based image retrieval" Fuzzy Systems Conference Proceedings, Aug. 22-25, 1999, pp. 964-968.|
|14||*||Medasani et al. "A fuzzy approach to content-based image retrieval" Fuzzy Systems Conference Proceedings, Aug. 22-25, 1999, pp. 1251-1260.|
|15||N. Srinivasa, Learning and Generalization of Noisy Mappings Using a Modified PROBART Neural Network, IEEE Trans. on Signal Processing, vol. 45, No. 10, pp. 2533-2550, Oct. 1997.|
|16||N. Srinivasa, S. Medasani, Y. Owechko, and D. Khosla, Fuzzy edge-symmetry features for improved intruder detection, The 12th IEEE International Conference on Fuzzy Systems, vol. 2, 920-925, 2003.|
|17||*||Papageorgiou et al. "A General Framework for Object Detection" Sixth International Conference on Computer Vision, Jan. 4-7, 1998 pp. 555-562.|
|18||R. N. Rao and D. H. Ballard, An active Vision Architecture based on Iconic Representations, Artificial Intelligence, vol. 78, 461-505, 1995.|
|19||S. Medasani, R. Krishnapuram, and Y. Choi, Graph Matching by Relaxation of Fuzzy Assignments, IEEE Transactions on Fuzzy Systems, 9(1), 173-183, Feb. 2001.|
|20||*||Srinivasa et al. "Learning and Generalization of Noisy Mappings Using a Modified PROBART Neural Network" IEEE Transactions on Signal Processing, Oct. 1997, vol. 45 No. 10, pp. 2533-2550.|
|21||W. T. Freeman and E. H. Adelson, The design and use of steerable filters, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, No. 9, pp. 891-906, 1991.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US7965867 *||Jul 5, 2007||Jun 21, 2011||Fondazione Bruno Kessler||Method and apparatus for tracking a number of objects or object parts in image sequences|
|US8131477||Aug 8, 2006||Mar 6, 2012||3M Cogent, Inc.||Method and device for image-based biological data quantification|
|US8254669 *||Sep 16, 2008||Aug 28, 2012||Ricoh Company, Ltd.||Data processing apparatus, computer program product, and data processing method for predicting an optimum function based on a case database and image feature values calculated by a feature-value calculating unit|
|US8254728||Jul 9, 2009||Aug 28, 2012||3M Cogent, Inc.||Method and apparatus for two dimensional image processing|
|US8275179||Apr 17, 2008||Sep 25, 2012||3M Cogent, Inc.||Apparatus for capturing a high quality image of a moist finger|
|US8295591 *||Aug 18, 2009||Oct 23, 2012||Behavioral Recognition Systems, Inc.||Adaptive voting experts for incremental segmentation of sequences with prediction in a video surveillance system|
|US8340352||Aug 18, 2009||Dec 25, 2012||Behavioral Recognition Systems, Inc.||Inter-trajectory anomaly detection using adaptive voting experts in a video surveillance system|
|US8379982||Oct 13, 2009||Feb 19, 2013||3M Cogent, Inc.||System and method for fast biometric pattern matching|
|US8385632||Jun 1, 2010||Feb 26, 2013||Mitsubishi Electric Research Laboratories, Inc.||System and method for adapting generic classifiers for object detection in particular scenes using incremental training|
|US8411916||Mar 28, 2008||Apr 2, 2013||3M Cogent, Inc.||Bio-reader device with ticket identification|
|US8442820 *||Dec 1, 2009||May 14, 2013||Hyundai Motor Company||Combined lip reading and voice recognition multimodal interface system|
|US8553992 *||Nov 19, 2008||Oct 8, 2013||Deepinder Singh Thind||Determination of class, attributes, and identity of an occupant|
|US8583379||Jan 25, 2012||Nov 12, 2013||3M Innovative Properties Company||Method and device for image-based biological data quantification|
|US8615105||Aug 31, 2010||Dec 24, 2013||The Boeing Company||Object tracking system|
|US8805002 *||Dec 23, 2010||Aug 12, 2014||Metalo GmbH||Method of determining reference features for use in an optical object initialization tracking process and object initialization tracking method|
|US8805090||Feb 7, 2012||Aug 12, 2014||Google Inc.||Matching based upon rank|
|US8913792 *||Jun 13, 2014||Dec 16, 2014||Metaio Gmbh||Method of determining reference features for use in an optical object initialization tracking process and object initialization tracking method|
|US9053391||Apr 12, 2011||Jun 9, 2015||Sharp Laboratories Of America, Inc.||Supervised and semi-supervised online boosting algorithm in machine learning framework|
|US9122935 *||Jan 28, 2014||Sep 1, 2015||Megachips Corporation||Object detection method, storage medium, integrated circuit, and object detection apparatus|
|US9202130 *||Mar 21, 2013||Dec 1, 2015||Fujitsu Limited||Image processing apparatus and image processing method|
|US9300927 *||Jun 11, 2007||Mar 29, 2016||Koninklijke Philips N.V.||Fingerprint, apparatus, method for identifying and synchronizing video|
|US9449258||Jul 2, 2015||Sep 20, 2016||Agt International Gmbh||Multi-camera vehicle identification system|
|US20070112525 *||Aug 8, 2006||May 17, 2007||Songtao Li||System and device for image-based biological data quantification|
|US20080031492 *||Jul 5, 2007||Feb 7, 2008||Fondazione Bruno Kessler||Method and apparatus for tracking a number of objects or object parts in image sequences|
|US20080273771 *||Apr 17, 2008||Nov 6, 2008||Ming Hsieh||Apparatus for capturing a high quality image of a moist finger|
|US20080304723 *||Mar 28, 2008||Dec 11, 2008||Ming Hsieh||Bio-reader device with ticket identification|
|US20090074288 *||Sep 16, 2008||Mar 19, 2009||Hirobumi Nishida||Data processing apparatus, computer program product, and data processing method|
|US20090167942 *||Jun 11, 2007||Jul 2, 2009||Koninklijke Philips Electronics N.V.||Fingerprint, apparatus, method for identifying and synchronizing video|
|US20090268988 *||Jul 9, 2009||Oct 29, 2009||Cogent Systems, Inc.||Method and apparatus for two dimensional image processing|
|US20100014755 *||Jul 21, 2008||Jan 21, 2010||Charles Lee Wilson||System and method for grid-based image segmentation and matching|
|US20100027852 *||Oct 13, 2009||Feb 4, 2010||Ming Hsieh||System and Method for Fast Biometric Pattern Matching|
|US20100124376 *||Nov 19, 2008||May 20, 2010||Deepinder Singh Thind||Determination Of Class, Attributes, And Identity Of An Occupant|
|US20110044492 *||Aug 18, 2009||Feb 24, 2011||Wesley Kenneth Cobb||Adaptive voting experts for incremental segmentation of sequences with prediction in a video surveillance system|
|US20110044499 *||Aug 18, 2009||Feb 24, 2011||Wesley Kenneth Cobb||Inter-trajectory anomaly detection using adaptive voting experts in a video surveillance system|
|US20110071830 *||Dec 1, 2009||Mar 24, 2011||Hyundai Motor Company||Combined lip reading and voice recognition multimodal interface system|
|US20110161323 *||Dec 3, 2010||Jun 30, 2011||Takehiro Hagiwara||Information Processing Device, Method of Evaluating Degree of Association, and Program|
|US20110194731 *||Dec 23, 2010||Aug 11, 2011||Metaio Gmbh||Method of determining reference features for use in an optical object initialization tracking process and object initialization tracking method|
|US20120027295 *||Apr 14, 2010||Feb 2, 2012||Koninklijke Philips Electronics N.V.||Key frames extraction for video content analysis|
|US20130308867 *||Mar 21, 2013||Nov 21, 2013||Fujitsu Limited||Image processing apparatus and image processing method|
|US20140247987 *||Jan 28, 2014||Sep 4, 2014||Megachips Corporation||Object detection apparatus, storage medium, and integrated circuit|
|US20140321705 *||Jun 13, 2014||Oct 30, 2014||Metaio Gmbh|
|US20150116597 *||Oct 28, 2014||Apr 30, 2015||Nec Laboratories America, Inc.||Trajectory Features and Distance Metrics for Hierarchical Video Segmentation|
|US20150294191 *||Apr 15, 2014||Oct 15, 2015||Xerox Corporation||System and method for predicting iconicity of an image|
|U.S. Classification||382/103, 382/159|
|International Classification||G06K9/62, G06K9/46, G06K9/00|
|Cooperative Classification||G06K9/469, G06K9/6254|
|Oct 5, 2007||AS||Assignment|
Owner name: HRL LABORATORIES, LLC, CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:OWECHKO, YURI;MEDASANI, SWARUP;SRINIVASA, NARAYAN;REEL/FRAME:019951/0727;SIGNING DATES FROM 20070928 TO 20071001
|Feb 28, 2013||FPAY||Fee payment|
Year of fee payment: 4