|Publication number||US7787011 B2|
|Application number||US 11/220,839|
|Publication date||Aug 31, 2010|
|Filing date||Sep 7, 2005|
|Priority date||Sep 7, 2005|
|Also published as||US20070052858|
|Publication number||11220839, 220839, US 7787011 B2, US 7787011B2, US-B2-7787011, US7787011 B2, US7787011B2|
|Inventors||Hanning Zhou, Qiong Liu, Donald Kimber, Lynn Wilcox, Matthew L. Cooper|
|Original Assignee||Fuji Xerox Co., Ltd.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (4), Non-Patent Citations (7), Referenced by (14), Classifications (14), Legal Events (2)|
|External Links: USPTO, USPTO Assignment, Espacenet|
A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
1. Field of the Invention
This invention relates to the field of multi-camera video monitoring, analysis and display.
2. Description of the Related Art
Recent years have witnessed the emergence of numerous video surveillance systems, most of them consist of multiple cameras. The visualization of video streams in such systems has been limited to 2-D, and it costs the security staffs excessive efforts to keep track of the spatial location of the physical scene that each camera is capturing. It is thus desirable to alleviate this difficult problem by bringing the 2-D displays into a 3-D immersive 3-D environment, in which the security staffs can navigate intuitively and inspect any corner of the scene at a click of the mouse.
Various embodiments of the present invention introduces a novel technique to analyze and monitor videos captured from multiple cameras. It renders and displays the videos in an immersive 3-D environment, which uses computer graphics (CG) to convey the spatial distribution of the cameras as well as their fields of view. Given the videos captured from a multi-camera surveillance system, the present invention detects moving objects via Gaussian-Mixture-Model (GMM) based adaptive background subtraction and tracks them by Bayesian filtering. The tracking results can be visualized in real time in two ways: 1) highlighting the foreground; 2) rendering an avatar of the object at its location in the 3-D environment. The spatial arrangement of the displays can be generated by multi-dimensional scaling (MDS) of the amount of simultaneous motion across different displays. When the geometries of the cameras are available, it can be combined with the MDS result by linear interpolation.
The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
Certain embodiments of the present invention will be described in detail on the following figures, wherein:
The invention is illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that references to “an” or “one” or “some” embodiment(s) in this disclosure are not necessarily to the same embodiment, and such references mean at least one.
Compared to the prior approaches, the present invention enjoys at least the following advantages:
In some embodiments, GMM based adaptive background subtraction algorithm can be used by the background subtraction module to detect and segment the moving objects in the video streams. Each pixel in the image is modeled using several (usually 3-5) Gaussian distributions in RGB color space. This allows the system to model a variable background (analogous to trees swaying in the wind) and to keep track of the previous background color distribution even when a pixel color changes due to a foreground object or temporary background changes. The decision to label a pixel as foreground or background is made by scoring each Gaussian based on the total evidence and its variance. It is assumed that background pixels will have more evidence and lower variance than foreground pixels. Mathematically it is equivalent to setting a fixed threshold for the log likelihood, because likelihood is a ratio between the evidence and the variance. The classification is achieved by summing the normalized scores in descending order until a certain percentage is accounted for. All Gaussian kernels that are required to reach this percentage are considered background and all others are labeled as foreground. An exemplary output of the GMM based motion detection method of moving objects is shown in
In some embodiments, a Bayesian framework can be constructed to track the moving objects along a corridor. The locations of the objects are hidden variables in the Bayesian filter. The observations of the filter are the sizes and centroids of the motion blobs detected from the cameras at both sides of the corridor. Intuitively, when a person/object moves from one side to another, the scale of the motion blob in one end will increase continuously and that from the other end will decrease accordingly. In the case of tracking multiple moving objects, discriminative tracking can be used to handle the data association problem. In the case of tracking human, face detection and skin color can be used as additional observations to increase the robustness of the tracker.
Due to factors that include but are not limited to, global lighting changes, dynamic backgrounds, cast shadows and speckle lights, the results from the background subtraction module can be noisy. Morphological operator can be applied to filter out the small regions and filled in the small holes. Even after this post-processing, there will be cases, where part of the foreground or background is mis-segmented.
In some embodiments, the motion detection based alpha blending can be introduced as the blending module to alleviate the effect of such errors and convey the motion detection results to help the user track the object. The alpha values can be directly derived from the statistics from a foreground/background segmentation algorithm, or confidence scores from an object (pedestrian, face) detection algorithm. The alpha channel of a texture map indicates the transparency value for each pixel in the texture. Since the video stream is shown as texture map, different values can be assigned to different regions in the alpha channel based on the foreground mask shown in
In some embodiments, the video rendering module can put processed video streams into an immersive 3-D environment by texture mapping in order to indicate the spatial distribution of the video streams from the various cameras. When the geometries of the cameras are known, the video streams can be rendered as multiple displays at the sensor planes of the corresponding cameras, as shown by the exemplary snapshot of the 3-D layout in
The sensor plane of the camera, however, may not be a good indication of where the camera is focused. For a non-limiting example, when two cameras mounted at each end of a corridor cover the entire hallway, ideally it is more preferable to paste the captured video streams at their corresponding physical location. The idea of content-based display arrangement is that video streams from two cameras looking at the same scene (maybe from different view points) should be rendered closer in space, even though they may be mounted far away from each other. In order to decide the similarity between the fields of view from different cameras, one can collect video streams of daily office activities and detect the amount of motion at each frame. The exemplary result of the motion detection across nine cameras on a short segment of 400 frames is plotted in
In some embodiments, a multidimensional scaling (MDS) method can be used by the content analysis module to find a lower dimensional manifold of these camera views assuming each camera view is a point in a high dimensional space. More specifically, one can find a 2-D layout that best preserves the similarity relationship between the video streams from the different cameras by imposing the manifold to be 2-D. Ideally this layout would be the same as the actual layout of the camera views (notice that this layout is different from the physical location where the camera is mounted). However, the MDS solution can be a flipped or rotated version of the same topology since the orientation is not constrained. An exemplary snapshot of the 3-D layout based on content of the video streams from the cameras via MDS is shown in
In some embodiments, physical locations of the cameras can be combined with content-based scene analysis. The user can tune the combination weights according to their preferences via the 3-D layout generation module, and the layout can seamlessly move from pure content-based to pure geometry-based, as shown by the exemplary 3-D layout based on combination of image plane of the cameras and content of the video streams form the cameras in
In many Virtual Reality (VR) systems, navigating by dragging the mouse (fly-through) is not effective for quickly switching from one view to another, but in surveillance systems, the user constantly needs to zoom in to a particular view for scrutiny of the scenes with suspicious activities. To facilitate with this task, the present invention provides an interface via the rendering module for the user to switch to the best view of a particular video display by double clicking on a polygon where the video display is mapped onto, wherein these views can be preset during the system setup or specified by the user. Two exemplary suggested views for the video display are shown in
In some embodiments, the user can navigate in the immersive environment based on the tracking results reported by the motion tracking module. The user first chooses a target pedestrian for tracking by double-clicking the avatar corresponding to this person. When the target pedestrian moves around, the user's view will follow the person such that the display with the best view of the target person is given the largest screen resolution, while the model and other displays shown at the rest of the screen can still give the user the perception of an immersive environment. An exemplary sequence of views when the target is moving is shown in
In some embodiments, the tracking result of the user can be visualized in at least the following two ways:
Such an environment can provide the user with a bird's eye view of the entire region for general patrolling and enables the user to easily track multiple moving humans/objects across all the cameras. Upon any abnormal event, the user can quickly zoom into the best view of the activity and scrutinize the region at the maximum resolution of their desktop monitor. When a person/object moves out of the view of one camera, the system will search for it in the video streams from the neighboring cameras or follow it by flying through the CG environment.
In some embodiments, the tracker can be initialized by a pedestrian detector at the first sight of a new moving object. A single frame pedestrian detector such as (an AdaBoost-based one) can be adopted that works reasonably well and does not rely on foreground segmentation.
One embodiment may be implemented using a conventional general purpose or a specialized digital computer or microprocessor(s) programmed according to the teachings of the present disclosure, as will be apparent to those skilled in the computer art. Appropriate software coding can readily be prepared by skilled programmers based on the teachings of the present disclosure, as will be apparent to those skilled in the software art. The invention may also be implemented by the preparation of integrated circuits or by interconnecting an appropriate network of conventional component circuits, as will be readily apparent to those skilled in the art.
One embodiment includes a computer program product which is a machine readable medium (media) having instructions stored thereon/in which can be used to program one or more computing devices to perform any of the features presented herein. The machine readable medium can include, but is not limited to, one or more types of disks including floppy disks, optical discs, DVD, CD-ROMs, micro drive, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of media or device suitable for storing instructions and/or data. Stored on any one of the computer readable medium (media), the present invention includes software for controlling both the hardware of the general purpose/specialized computer or microprocessor, and for enabling the computer or microprocessor to interact with a human user or other mechanism utilizing the results of the present invention. Such software may include, but is not limited to, device drivers, operating systems, execution environments/containers, and applications.
The foregoing description of the preferred embodiments of the present invention has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations will be apparent to the practitioner skilled in the art. Particularly, while the concept “interface” is used in the embodiments of the systems and methods described above, it will be evident that such concept can be interchangeably used with equivalent concepts such as, bean, class, method, type, component, object model, and other suitable concepts. Embodiments were chosen and described in order to best describe the principles of the invention and its practical application, thereby enabling others skilled in the art to understand the invention, the various embodiments and with various modifications that are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the following claims and their equivalents.
The entire disclosure of U.S. patent application Ser. No. 11/220,730, entitled SYSTEM AND METHOD FOR USER MONITORING INTERFACE OF 3-D VIDEO STREAMS FROM MULTIPLE CAMERAS by Hanning Zhou et al, filed concurrently including specification, claims, drawings and abstract is incorporated herein by reference in its entirety.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US6954498 *||Oct 24, 2000||Oct 11, 2005||Objectvideo, Inc.||Interactive video manipulation|
|US6987883 *||Dec 31, 2002||Jan 17, 2006||Objectvideo, Inc.||Video scene background maintenance using statistical pixel modeling|
|US7142209 *||Mar 31, 2005||Nov 28, 2006||Microsoft Corporation||Real-time rendering system and process for interactive viewpoint video that was generated using overlapping images of a scene captured from viewpoints forming a grid|
|US7224735 *||May 21, 2003||May 29, 2007||Mitsubishi Electronic Research Laboratories, Inc.||Adaptive background image updating|
|1||Cox, T.F. and Cox, M.A.A., Multidimensional Scaling, Chapman & Hall/CRC Press: Boca Rotan, ISBN 1-58488-094-5, 2000.|
|2||I.O. Sebe, J. Hu, S. You, U. Neumann, 3D Video Surveillance, pp. 107-112, Berkeley, California, (in conjunction with ACM Multimedia 2003), Nov. 7, 2003.|
|3||Kameda, Y., Takemasa, T. and Ohta, Y., Outdoor see-through vision utilizing surveillance cameras, Third IEEE and ACM International Symposium on Mixed and Augmented Reality, Nov. 2-5, 2004 pp. 151-160.|
|4||Michael Isard and Andrew Blake, ICondensation: Unifying low-level and high-level tracking in a stochastic framework, Proc 5th European Conf. Computer Vision, 1998.|
|5||Sawhney, H.S., Arpa, A., Kumar, R., Samarasekera, S., Aggarwal, M., Hsu, S., Nister, D., and Hanna, K. 2002. Video flashlights: real time rendering of multiple videos for immersive model visualization.|
|6||Stauffer C, Grimson W.E.L., Learning patterns of activity using real-time tracking. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2000. 22(8): p. 747-57.|
|7||Viola, P.; Jones, M.J.; Snow, D., "Detecting Pedestrians Using Patterns of Motion and Appearance", IEEE International Conference on Computer Vision (ICCV), vol. 2, pp. 734-741, Oct. 2003.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US7929729 *||Apr 2, 2007||Apr 19, 2011||Industrial Technology Research Institute||Image processing methods|
|US9008361 *||Mar 21, 2011||Apr 14, 2015||Robert Bosch Gmbh||Method for visualizing zones of higher activity in surveillance scenes|
|US9076212||Sep 23, 2013||Jul 7, 2015||The Queen's Medical Center||Motion tracking system for real time adaptive imaging and spectroscopy|
|US9138175||Apr 28, 2015||Sep 22, 2015||The Queen's Medical Center||Motion tracking system for real time adaptive imaging and spectroscopy|
|US9305365||Mar 14, 2013||Apr 5, 2016||Kineticor, Inc.||Systems, devices, and methods for tracking moving targets|
|US9606209||Aug 24, 2012||Mar 28, 2017||Kineticor, Inc.||Methods, systems, and devices for intra-scan motion correction|
|US9607377||Aug 27, 2015||Mar 28, 2017||Kineticor, Inc.||Systems, devices, and methods for tracking moving targets|
|US9717461||Jan 23, 2014||Aug 1, 2017||Kineticor, Inc.||Systems, devices, and methods for tracking and compensating for patient motion during a medical imaging scan|
|US9734589||Jul 22, 2015||Aug 15, 2017||Kineticor, Inc.||Systems, devices, and methods for tracking and compensating for patient motion during a medical imaging scan|
|US20070222565 *||Feb 21, 2007||Sep 27, 2007||Mazda Motor Corporation||Pedestrian detecting device for vehicle|
|US20080240500 *||Apr 2, 2007||Oct 2, 2008||Industrial Technology Research Institute||Image processing methods|
|US20110115920 *||Feb 10, 2010||May 19, 2011||Industrial Technology Research Institute||Multi-state target tracking mehtod and system|
|US20130094703 *||Mar 21, 2011||Apr 18, 2013||Robert Bosch Gmbh||Method for visualizing zones of higher activity in surveillance scenes|
|US20160023111 *||Jul 25, 2014||Jan 28, 2016||Zynga Inc.||In-browser emulation of multiple technologies to create consistent visualization experience|
|U.S. Classification||348/143, 348/159, 348/153, 348/155|
|Cooperative Classification||G06T2207/30241, G06T7/277, G06K9/00771, G06T2207/10021, G06T7/254, G06T2207/10012|
|European Classification||G06T7/20D, G06T7/20K, G06K9/00V4|
|Oct 21, 2005||AS||Assignment|
Owner name: FUJI XEROX CO., LTD., JAPAN
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ZHOU, HANNING;LIU, QIONG;KIMBER, DONALD;AND OTHERS;REEL/FRAME:017131/0433;SIGNING DATES FROM 20051003 TO 20051004
Owner name: FUJI XEROX CO., LTD., JAPAN
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ZHOU, HANNING;LIU, QIONG;KIMBER, DONALD;AND OTHERS;SIGNING DATES FROM 20051003 TO 20051004;REEL/FRAME:017131/0433
|Jan 29, 2014||FPAY||Fee payment|
Year of fee payment: 4