WO2006040687A2

WO2006040687A2 - Automatically expanding the zoom capability of a wide-angle video camera

Info

Publication number: WO2006040687A2
Application number: PCT/IB2005/003504
Authority: WO
Inventors: Bartu Ahiska; Mark Kenneth Davey; Ahmet Enis Cetin
Original assignee: Grandeye, Ltd.
Priority date: 2004-07-19
Filing date: 2005-07-19
Publication date: 2006-04-20
Also published as: US20060056056A1; WO2006040687A3; US8405732B2; US7990422B2; US20120038776A1; GB0701038D0; JP2008507229A; GB2431312A

Abstract

A system for automatically expanding the zoom capability of a wide-angle video camera using images from multiple camera locations. One preferred embodiment achieves this using images from the wide-angle video camera that are analyzed to identify regions of interest (RoI). Pan-Tilt-Zoom (PTZ) controls are then sent to aim slave cameras toward the RoI. Processing circuitry is' then used to replace the RoI from the wide-angle images with the higher-resolution images from one of the slave cameras. In addition, motion-detecting software can be utilized to automatically detect, track, and/or zoom in on moving objects.

Description

PCT APPLICATION

AUTOMATICALLY EXPANDING THE ZOOM CAPABILITY OF A WIDE-ANGLE VIDEO CAMERA

CROSS-REFERENCE TO OTHER APPLICATIONS

This application claims priority from provisional U.S. patent application 60/589,104 filed July 19, 2004, which is hereby incorporated by reference.

BACKGROUND AND SUMMARY OF THE INVENTION

Field of the Invention

The present inventions relate to video monitoring systems, and more specifically, to automatically expanding the zoom capability of a wide-angle video camera.

Background

Real-time video surveillance systems have become increasingly popular in security monitoring applications. A new class of cameras replaces the mechanical Pan-Tilt-Zoom (PTZ) functions with a wide-angle optical system and image processing, as discussed in US patent application 10/837,019 entitled "Method of Simultaneously Displaying Multiple Views for Video Surveillance," which is hereby incorporated by reference. This class of cameras is further discussed in US patent application 10/837,325 entitled "Multiple View Processing in Wide-Angle Video Camera," which is hereby incorporated by reference. This type of camera monitors a wide field of view and selects regions from it to transmit to a base station; in this way it emulates the behavior of a mechanical PTZ camera. The wide-angle optics introduces distortion into the captured image, and processing algorithms are used to correct the distortion and convert it to a view that has the same perspective as a mechanical PTZ camera.

The US patent application 10/837,326 entitled, "Multiple Object Processing in Wide-Angle Video Camera" by Yavuz Ahiska, which is hereby incorporated by reference, describes a way to correct the distorted view captured by a wide-angle camera. This camera, even using this distortion-correction process, only has limited capabilities to zoom into a region of interest. The camera can also be a programmable one as described in US patent application 10/837,325, entitled "Multiple View Processing in Wide-Angle Video Camera," containing programmable embedded microprocessors.

There exists a conflict between a video camera's field of view and the effective resolution of its image. Wide-angle lenses rarely offer any significant optical zoom, and similarly, video cameras with a high zoom capability have restricted fields of view (especially when their magnification is increased). A solution to monitoring a wide-angle area while being able to capture regions at a higher detail is to utilize multiple cameras at differing locations. The US patent 6,724,421, which is hereby incorporated by reference, and the public domain document, "A Master-Slave System to Acquire Biometric Imagery of Humans at Distance," by X. Zhou et al, which is hereby incorporated by reference, describe systems using multiple cameras to monitor a wide-angle area. In these systems, a separate base station unit controls the two cameras monitoring the scene. In addition, these systems do not try to expand the zoom function of the master camera.

The US patent 6,147,709, which is hereby incorporated by reference, describes a method and apparatus for overlaying a high-resolution image onto a hemispherical interactive image captured by a camera by matching at least three points between the high- resolution image and the perspective corrected image. A major drawback with this process is that it makes comparisons in the perspective corrected domain.

Moving regions in a video corresponding to persons or moving objects, together with tracked objects which may no longer be moving, and their local neighborhoods in the video define Regions of Interest (RoI) because persons, moving and/or tracked objects, etc. are important in security monitoring applications. In order to provide real-time alarms for dangerous events, RoI should be tracked and zoomed for closer inspection. Conventional Closed Circuit Television (CCTV) systems, which only capture recorded video for later analysis, cannot provide automatic alarm and event triggers without delay.

A wide field of view camera that can both monitor a wide-angle scene, while also being able to simultaneously and automatically capture regions of interest at a greater magnification is very desirable in surveillance systems. For example, a high-resolution image could make the difference in positively identifying a criminal committing an offense or the detail surrounding an unattended suitcase. Therefore, it is very important to provide a high-resolution view of a person in a surveillance application.

Wide-angle surveillance is necessary in many CCTV applications. Cameras such as dome cameras and cameras with fisheye or peripheral lenses can produce wide-angle video. A major weakness of wide-angle surveillance cameras and systems is that they either do not have the capability to zoom into a RoI or are limited in their zooming capability.

The system can also have a computer program comprising a machine-readable medium having computer executable program instructions thereon for executing the moving object detection and object tracking algorithms fully in the programmable camera device as described in US patent application 10/924,279, entitled "Tracking Moving Objects in Video Using Wavelet Domain Information," by A. E. Cetin and Y. Ahiska, which is hereby incorporated by reference. Automatic moving-object detection and object tracking capability of the wide field of view camera can define a RoI in the wide-angle scene monitored by the camera containing the object in question. As this RoI will be of interest in many security applications, the region can be tracked by the electronic PTZ capability of the master camera. There is a present demand for a system that can both monitor a wide area, while also being able to simultaneously and automatically capture regions of interest at a higher resolution.

Automatically Expanding the Zoom Capability of a Wide- Angle Video Camera

The present innovations include a new approach that achieves the ability to monitor a wide-angle area while being able to capture regions of higher detail.

In one example embodiment, a wide-angle, master camera, such as a dome camera or a camera with a fish-eye or peripheral lens, preferably with substantially no zoom capabilities, is used to capture images and automatically identify RoI, e.g. motion detecting and/or object tracking. In this embodiment, at least one other camera, preferably with expanded zoom capabilities relative to the master camera, can be used to zoom into the identified RoI. The views from the cameras other than the master camera can be used for several purposes including, but not limited to, input into the master camera or output to a base station.

In another example embodiment, control circuitry sends PTZ controls to one or more slave cameras based in at least partial dependence on the wide-angle images captured by the master camera. Among other things, these controls can be used to aim the slave camera towards the RoI and/or zoom the slave camera onto the RoI.

In another class of embodiments, the output of a slave camera is compared to the images captured by the master camera and PTZ controls are sent to one or more slave cameras based in at least partial dependence on the comparison. Output images from the slave cameras can then be used for several purposes including, but not limited to, comparing them to RoI from the master camera, outputting them to a base station, or overlaying them onto other images.

In a sample of this embodiment, after the slave camera has moved in accordance with the PTZ controls, the output from the slave camera can be compared to the images from the master camera to generate a new set of PTZ controls. This process can be, but does not have to be, used to match the output images from the slave camera to the RoI identified in the output images from the master camera. This process can be, but does not have to be, an iterative process that can be repeated to yield any level of desired matching accuracy. There are multiple methods for implementing this synchronization including, but not limited to image-processing techniques to match views, calibration procedures, or position analysis of feedback from the slave camera.

In another example embodiment, the images from the slave camera can be used to replace, correct, inset, or overlay some or all of the images from the master camera. The composite images can be used for several purposes including, but not limited to, recording them, outputting them to a base station, and/or using them to generate PTZ controls. In another embodiment, several slave cameras, preferable monitoring different regions, can be used and the perspective-corrected view of the master camera can be altered in at least partial dependence on the adjusted views of at least one of these slave cameras.

In another embodiment, motion-detecting software can be utilized to define the RoI as moving regions in the video corresponding to, but not limited to, persons, moving objects, tracked objects which may no longer be moving, and/or their local neighborhoods in the video.

These and other embodiments of the present innovations are described more fully below.

Docket No. GRN D.04

BRIEF DESCRIPTION OF THE DRAWINGS The disclosed inventions will be described with reference to the accompanying drawings, which show important sample embodiments of the invention and which are incorporated in the specification hereof by reference, wherein:

Figure IA shows a diagram of a camera system consistent with a preferred embodiment of the present invention.

Figure IB shows a diagram of a camera system consistent with a preferred embodiment of the present invention. Figure 2 shows a diagram of a camera system consistent with a preferred embodiment of the present invention.

Figure 3 shows a diagram of a camera system consistent with a preferred embodiment of the present invention.

Figure 4 shows a diagram of a camera system consistent with a preferred embodiment of the present invention.

Figure 5 shows a diagram of a camera system consistent with a preferred embodiment of the present invention.

Figure 6 shows a diagram of a camera system consistent with a preferred embodiment of the present invention. Figure 7 shows a flowchart implementing process steps consistent with the preferred embodiment of the present invention.

Figure 8 shows a flowchart implementing process steps consistent with the preferred embodiment of the present invention.

Docket No. GRN D.04

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The numerous innovative teachings of the present application will be described with particular reference to the presently preferred embodiment (by way of example, and not of limitation).

Before the present innovations, the systems available only had limited zoom capabilities, but using a slave PTZ camera controlled from the master camera can expand this electronic-zooming capability to get even higher resolution images of the RoI.

Unlike US patent 6,147,709, the method and systems disclosed below which make comparisons do so in the wide-angle distorted domain within the master camera, as opposed to the perspective corrected domain, to generate PTZ commands for controlling a slave PTZ video camera. This can be an iterative process, and yields the desired matching accuracy given enough steps.

The control system within the master camera that performs view matching between a master, wide-angle video camera and a slave PTZ camera allows the master camera to acquire detailed images of required areas from the slave camera. There are multiple methods for implementing this synchronization namely using image-processing techniques to match views and/or by a calibration procedure. Position feedback from the slave camera can also be a useful element of information for accurate view matching. The US patent 6,509,926 entitled "Surveillance Apparatus for Camera Surveillance System," which is hereby incorporated by reference, discusses a system for generating the azimuth and elevation angles of a camera and lens. Carrying out this process in the distorted domain allows the comparison to be made without losing anything in terms of quality. Comparing images or vectors x and y can be measured many different ways, the most well known way is the Euclidian distance, llx-yll, but can also use llg(x)-g(y) Il where g is an appropriate function representing the distortion.

In a preferred embodiment, the master wide-angle camera has the capability of sending PTZ control signals to the slave PTZ camera to zoom into the RoI in an automatic manner by implementing the motion-detection and/or object tracking algorithms on the current wide field of view image. In an example embodiment, the slave is commanded to go to a set angle, which can be described as a PTZ control although it is not the standard widespread PTZ interface. A control system resident in the master camera can perform view matching between the master, wide-angle video camera and the slave PTZ camera. One preferred embodiment uses image-processing techniques to match the views of the master and slave cameras, allowing detailed images of the RoI to be acquired.

In one class of embodiments, the master and slave cameras are calibrated and PTZ controls can be sent to the slave camera based at least partially on these calibrations. If calibration between the master and slave cameras is insufficient alone, image registration or matching can be carried out either using the corrected images of the scene or using the raw wide-angle images captured by the master camera.

s Docket No. GRND.04

The following is one possible example of the calibration process between the master and slave cameras. It provides a switching mode where the master camera's output can be switched to the slave camera's output, the switch can be based on a predefined zoom point where the slave camera's position can then be lined up with the master camera's selected view and, if slave tracking is being used, the slave camera can be used to follow an object being tracked by motion tracking. The master camera's wide-angle view can be divided into smaller regions and, using image processing, these regions can be zoomed in on. By tracking these smaller views, the master camera is acting as a virtual camera, or VCAM. As mentioned earlier, the zoomed in VCAM views have smoothed edges and blurred details. The slave camera views are needed to capture the level of detail required in most surveillance applications.

Manual Mode: The video of Head A (in one preferred embodiment this refers to view of the output from the master camera) is switched to the video output of the slave camera on pressing the enter key. Once switched, keyboard control of the slave camera is provided. The slave camera can then be manually calibrated to aim at the same object as the master camera. Pressing escape returns the video and keyboard control to the master camera. In another example embodiment, there can be two analogue outputs from the master camera, each referred to as a Head. A monitor can view the output from a Head. There can be BNC connectors at the back of the master camera, labeled A and B so that a monitor can be connected to either Head A or Head B.

Zoom Switch Mode:

While controlling any VCAM on Head A, if the field of view goes beyond 25 degrees, the video is switched to the slave camera and it is moved to the same position as the master camera's VCAM. An option is provided for pre-positioning. If this option is turned on, the slave camera will be moved to the position of the master camera's VCAM at a zoom level of 30 degrees and is repositioned until the zoom level reaches 25 degrees at which point the video is switched to the slave camera. Once switched, keyboard control of the slave camera is provided. Pressing escape returns the video and keyboard control to the master camera.

Slave Tracking Mode:

Whenever motion tracking is triggered, Head A is switched to the slave camera and the slave camera is moved to the position of the motion rectangle being tracked and is updated as the motion rectangle moves. If the motion rectangle stops moving or slows down the slave camera is zoomed in and will zoom out again if the motion rectangle moves faster again. If the user moves the joystick, control of the slave camera will be given to them. If after 5 seconds of no activity from the keyboard the camera is still tracking, it will return the slave camera to tracking the motion rectangle. If tracking has ended control and video will be returned to the master camera.

Design Docket No. GRND.04

Three additional classes were required in order to implement the master/slave features; slavecalibration, slavemanager and serial out. The code described here is enabled via the abspos setting within the adome.ini. Slavecalibration:

This class is responsible for calibrating the master camera with the slave camera and translating between the master camera spherical coordinates and slave camera spherical coordinates. The calibration phase is run by positioning the slave camera at a reference point.

Its video output is then displayed on Head A while a VCAM is positioned at the same reference point, (but in the coordinate system of the master camera). The user then has control of positioning of the master camera's VCAM and should line up the VCAM to match the image from the slave camera. Once matched, the user would press enter and the current position of the VCAM would be read. The difference in pan between the reference point and the new position of the VCAM is stored for later use by the translation function. For the calibration phase the slavecalibration class collaborates with the Menus class and it is possible to use more than one reference point if required without changing the Menus class (see GenerateCalibrationPresets function of the slavecalibration class).

The second responsibility of the slavecalibration class is to provide a translation of the master camera's coordinates to the slave camera's coordinates for the slavemanager class. This is done in the TranslateToSlaveCoordinates function by firstly assuming that the point being viewed is a distance of 5 meters away. The spherical coordinates are then translated into Cartesian coordinates. A rotation in the z-axis by the difference in pan that was measured during the calibration phase is then made. A translation in x and z coordinates is then made. This translation is accounting for the physical distance between the two cameras (including their difference in height). The mounting kit will ensure that the distance between the two cameras is constant along the x-axis. As the height of the slave cameras can be different from one another the z-axis translation depends on which slave camera is connected. The final stage is to convert the translated and rotated Cartesian coordinates back into spherical coordinates.

Slavemanager: The slavemanager class is responsible for checking the zoom level for when to switch to the slave camera, switching the video to the slave camera, positioning the slave camera and dealing with timeouts from no keyboard activity.

The ProcessSlaveMode function is called once per frame. If the zoom switch is enabled it will check the zoom level of the active VCAM on Head A and if it is under 25 it will switch to the slave camera and position it by calling the SetSlaveMode function (described below). If prepositioning is enabled it will also position the slave camera, but not switch to it when the zoom level is between 30 and 25. This is done by calling the SwitchSlaveToNearestPreset function (described below). A timer is managed by the class in order to deal with timeouts from slave mode and slave tracking mode. This timer is checked in this function and the appropriate mode entered after a timeout. The timer is Docket No. GRND.04

reset by calling the SlaveMoved function (this is done by serialOut and serial described below).

The SetSlaveMode function switches the video to the slave camera and positions it. The switch to the slave camera video is done by setting a bit of the register controlling the CPLD via an i2c write. The positioning is carried out by reading the current position of the active VCAM, translating the coordinates by calling the TranslateToSlaveCoordinates function of the slavecalibration class and passing it to the output queue for the serialout class to deal with (described below). The SwitchSlaveToNearestPreset function takes the master camera's spherical coordinates, uses the TranslateToSlaveCoordinates function of the slavecalibration class and passing it to the output queue for the serialout class to deal with (described below). This is used by the prepositioning and by the MotionTracker class for slave tracking (described below).

Serialout:

The serialout class is responsible for sending commands to the slave camera via RS485. The serialout class runs a separate thread, which blocks on the output queue until a command is added to the queue. Once a command is added to the queue it calls the appropriate send function on the serial class.

In addition to the new classes described above some changes have been made to existing classes. The key changes are described below:

Serial:

The serial class that deals with keyboard input has the addition of a passthrough mode which is enabled when in slave mode, slave tracking mode or slave calibration mode. In the passthrough mode all commands received are passed out of the second serial port (the one connected to the slave dome). The escape key is captured by the serial class while in this mode and slave mode is disabled when it is received (by calling the

SetSlaveMode function of SlaveManager). While in this mode the SlaveMoved function of Slavemanager is called every time a command is received from the keyboard and passed through. This prevents the slavemanager from timing out of the mode by reseting its timer.

MotionTracker:

The MotionTracker class, that deals with motion tracking, has an additional function, SetSlavePos, that is called when the motion tracking moves the tracking VCAM camera slave tracking is enabled.

The SetSlavePos function takes the raw motion rectangle and the zoom level decided upon by the motion detection and tracking algorithms used for the tracking VCAM. It then attempts to position the slave camera so that it is centered on the top half of the motion rectangle (this is a simple aim high system attempting to aim for the persons head). If the position decided upon for the slave is less than a defined threshold away from the current position of the slave camera, the slave camera is left where it is. This is Docket No. GRND.04

in order to reduce the number of small moves made by the slave camera - when the slave camera is moved it stops quite suddenly so a slight vibration effect can be noticed so the algorithm prefers larger movements rather than lots of small movements in order to reduce the impact of this effect. If the motion rectangle is moving slowly or stops the algorithm will cause the slave camera to zoom in further than the zoom value calculated by the VCAM 's motion tracking algorithm. This is achieved by creating a m_DistanceMoved variable. This variable is set to 100 when tracking begins. During tracking the value is recalculated by taking 80% of its current value and 20% of the distance the slave dome has been moved by (straight line distance) since the last time it was moved. When this value drops below 3 the zoom level is increased before the slave camera is positioned. If the value is greater than or equal to 3 the zoom level is set to the one calculated by the VCAM' s motion tracking algorithm. The above embodiments are only example implementations and are not intended to limit the many possible ways that the present innovations can be implemented.

In one example embodiment, the images are compared in the same domain, e.g., distorted or corrected domain. In order to compare images in the wide-angle image domain, the images of the slave camera go through a reverse transformation into a distorted view similar to that of a portion of the raw wide-angle image of the master camera. The images can then be compared and, in one example embodiment, controls can be sent in at least partial dependence on the results of this comparison. The master camera can adjust the PTZ control signals according to the image matching results so that RoI is in the center of the field of view of the slave camera. Image matching can be implemented in an iterative manner to increase the accuracy of the view matching between the RoI and the field of view of the slave camera or cameras. This image matching can be done in many ways. Some of these methods are described in the public domain textbook, "Fundamentals of Digital Image Processing" by Anil Jain, Prentice-Hall, NJ, 1988. In an example of this embodiment, the master camera can send incremental PTZ control signals in order to achieve any level of desired matching accuracy.

Once the two images are matched or registered, the perspective corrected view of the master camera can be replaced by the revised view from the slave camera to have a higher resolution image of the RoI than would have been possible with the zooming capability of the master camera alone. The slave PTZ camera, having optical zooming capability, produces a sharper image of the RoI compared to the corrected view of the RoI obtained from the master wide-angle camera. This is because some wide-angle cameras zoom into a region by performing numerical interpolation, which may smooth the edges of the objects in the RoI. In this example embodiment, replacing an image from the master camera with the sharper image obtained from the slave camera expands the zooming capability of the master camera.

Examples of preferred embodiments are shown in Figures 1 through 9. An example embodiment of the inventions is shown in Figures 3, 4, and 5 containing programmable embedded microprocessors and circuitry. A preferred embodiment can Docket No. GRN D.04

have the capability of performing all necessary image processing operations to achieve an expanded optical zoom.

Figure IA shows a preferred layout for the system including a preferred embodiment of the master wide-angle camera. The wide-angle optical system, 101 in conjunction with the image sensor 102, captures an image that can be passed to the image processing circuitry 103 for correction. In one example embodiment, the correction can be an image warp that compensates for distortions introduced by the lens. As disclosed in US patent application 10/837,012, entitled, "Correction of Optical Distortion by Image Processing," which is hereby incorporated by reference, the distortion may be arbitrarily complex. The distortion can be corrected through the use of tables that define the necessary warping. The image processing circuitry 103, which has a memory, can be implemented in several ways, including either one or a cascaded set of microprocessors coupled with a high bandwidth bus to increase the available processing capability. The digital sensor data can be sent to the image processing circuitry 103 through a buffer or directly if the circuitry operates at a sufficiently high speed. This circuitry can also be responsible for debayering, color equalization and color balancing of the image. Characteristics of the image sensing 102, such as the exposure and aperture, and the image processing 103 can be controlled by the control circuitry 104. The output circuitry 105 can be used to out put a video signal to the base station.

The image processing circuitry 103 in the master camera can also take the digital video from the slave PTZ camera as another input. The view from the slave camera can be used when an image with a greater optical zoom is desired for improved detailed. The decision for whether the view from the slave is necessary can be dictated by the control circuitry resident in the master camera 104, which acts on a resident software program and from base station control. The control from the base station can be any standard, including RS485 or TCP/IP format. The slave camera is not controlled directly from the base station, but via the control circuitry, preferably in the master camera 104.

In an example embodiment, the control circuitry 104 performs any required view matching between the master, wide-angle video camera and the slave PTZ camera. PTZ controls can be transmitted to the slave camera from the control circuitry in the master camera to achieve an initial approximate matching. This approximation can, for example, PTZ presets in the slave camera. There is an optional position feedback signal from the slave camera, which aids the control circuitry in positioning the slave PTZ camera to the desired location.

In a preferred embodiment, this matching can be assisted using image-processing techniques to match the views. In order to compare images in the wide-angle image domain, the digital video output of the slave PTZ camera may go through a reverse transformation in the image processing circuitry 103 into a distorted slave image similar to that of a portion of the raw wide-angle image of the master camera. The reverse transformation can take a rectangular image from the slave camera and can transform it into a more complex shape with curved sides in the space of the distorted wide-angle image. Docket No. GRND.04

In one example embodiment, the reverse transformation can be considered to take place in two stages in the image processing circuitry 103. Given knowledge of the pan, tilt and zoom of the slave camera, the first stage transforms (x,y) coordinates in the slave image into world coordinates (θ,φ). This is the inverse of the transformation used to generate corrected images within the master camera. If position feedback from the zoom level is unavailable, it can be ensured that the transition between master and slave views is always performed at the same zoom. Once world coordinates have been calculated, the second stage involves a projection from world coordinates to the distorted image using a look-up table. This two-stage projection may be applied to individual pixels, to triangles that tessellate the slave image, or any other shape that tiles the slave image.

In one example embodiment, the required PTZ control adjustments to correct the slave camera's view can be determined by comparing the color and wavelet histograms of the two views. The corresponding image translation vector can be transformed into the perspective-corrected domain and used to generate the PTZ adjustment commands. Using only a proportion of this translation vector can maximize the convergence of the slave camera to the desired status. These PTZ adjustment control signals can be transmitted to the slave camera to obtain a better match between the view from the slave camera and the perspective corrected view for the said RoI from the master camera. The image matching and PTZ control-sending process can be implemented in an iterative manner to achieve the desired matching accuracy, which can be determined by using mean square error, mean absolute difference and histogram comparison, or other means. The master and the slave cameras may have different color settings in practice.

Before performing registration, their color histograms can be equalized so that they both have the same dynamic range, brightness and exposure levels, and possible color offsets are also removed by histogram equalization, which is a widely used image processing technique (see e.g., the text book entitled, Fundamentals of Digital Image Processing by Anil Jain, Prentice-Hall, NJ, 1988, which is hereby incorporated by reference).

The master camera can have the functionality to support privacy regions which obscure user defined regions from being outputted, as described in the US patent application 11/178,232 (Attorney Docket No. GRND-02) entitled "Image Processing of Regions in a Wide Angle Video Camera," which is hereby incorporated by reference. As the view from the slave PTZ camera and a perspective corrected RoI in the master camera are matched, masks representing the shape of the privacy regions defined in the master camera can be applied to blank the appropriate regions in the slave image. In one example embodiment, this is done in the image processing circuitry 103.

In one class of preferred embodiments, once the two images are matched or registered, the control circuitry 104 dictates the desired image that will be composited in the image processing circuitry 103. If greater magnification is required, the relevant perspective corrected view can be replaced by the appropriately matched slave view. The intention of an example of this embodiment is to transition between the master view and the slave view as seamlessly as possible to create the quality of a continuous zoom Docket No. GRND.04

function. Outputs from the image processing circuitry can include a number of perspective corrected views from the wide-angle image, the slave camera's view, or a collage of multiple views including any number of these. The digital output is preferably passed to be formatted and compressed as necessary in the output circuitry 105 before being digitally output to the base station for monitoring. For example, multiple MPEG4 streams are possible. This process, describing the embodiment of Figure 1, is illustrated using a flow chart in Figure 8. The distorted wide-angle video image is captured using wide-angle master camera (Step 802). An RoI in the master camera is then defined (Step 804). The estimated PTZ commands are then transmitted to the salve camera from the master camera to, achieve approximate view matching with RoI (Step 806). The output of the slave camera is then reverse transformed by the master camera (Step 808). The distorted slave image is then compared with the distorted wide-angle image to determine and transmit adjustment PTZ control signals to the slave camera (Step 810) to determine whether the desired matching accuracy ahs been met (Step 812). If the desired matching accuracy has been met, then the process continues on to Step 814. If the desired matching accuracy has not been met, then the process is looped back to Step 806. Once the desired matching accuracy has been met, the perspective corrected master camera view is replaced by adjusted slave camera view to achieve an expanded zoom function (Step 814).

The method and systems can have several slave cameras. Such method and systems can track several moving regions at the same time by assigning each moving object into a different slave camera producing a sharper image of the moving blob. In this case, the perspective corrected view of the master camera can be replaced by the adjusted views of the slave cameras tracking moving blobs. An example embodiment is shown in Figure IB where a master camera controls two slave cameras with optional zoom capabilities.

Another variation embodiment of these inventions can use analog video for the output from the master camera. Conversion from digital-to-analog video and formatting can take place in the output circuitry 105. Another possible embodiment consists of two analog composite video output channels.

Another embodiment can use an analog slave camera. The analog video produced by this camera can be converted into digital video using an analog-to-digital converter (ADC) (206) as shown in Figure 2. The output circuitry 205 can perform the formatting and compression required for digital video output from the master camera. As mentioned for Figure 1, variations of the embodiment in Figure 2 include systems with analog video output from the master camera. Conversion from digital-to-analog video and formatting can take place in the output circuitry 205.

Another embodiment with an analog-switching version of the system is shown in Figure 3. In this embodiment the video from the slave PTZ camera is analog. The optical system 301, in conjunction with the image sensor 302, can be used to capture an image that can then be passed to the image processing circuitry 303 for correction. As mentioned in an example embodiment above, the correction can be an image warp that compensates Docket No. GRND.04

for distortions introduced by the Bens. The image processing circuitry 303, which can have a memory, can be comprised of either one or a cascaded set of microprocessors coupled with a high bandwidth bus to increase the available processing capability. The digital sensor data can be sent to the image processing circuitry 303 through a buffer or directly if the circuitry operates at a sufficiently high speed. This circuitry can also be responsible for debayering, color equalization and color balancing of the image. Characteristics of the image sensing 302, such as the exposure and aperture, and the image processing 303 can be controlled by the control circuitry 304. In one preferred embodiment, the decision for whether the view from the slave is necessary is dictated by the control circuitry resident in the master camera 304, which can act on a resident software program and/or from base station control. The control from the base station can be any standard including RS485 or TCP/IP format. The slave camera, preferable, is not controlled direbtly from the base station, but via the control circuitry in the master camera 304. It may be desirable for the user to be unaware that multiple cameras are in use.

In one example embodiment, the control circuitry 304 can perform approximate view matching between the master, wide-angle video camera and the slave PTZ camera if required. PTZ controls can be transmitted to the slave camera from the control circuitry in the master camera to achieve an approximate matching. In one preferred embodiment, this approximation uses PTZ presets in the slave camera. An alternative embodiment uses a slave camera, which can be commanded to turn to any PTZ state. Another alternative embodiment uses a slave camera in which the camera's position output is predictable and consistent when PTZ commands are issued. In an example of this embodiment, the slave camera could have a base that is controlled using stepper motors that are occasionally calibrated to a known position. Another embodiment can utilize a calibration technique in which the user calibrates a finite number of positions so that both of the cameras are viewing the same calibration object. The slave camera can then give a positional feedback signal, the value of which can b'e stored alongside the master camera view's virtual PTZ coordinates in a table in memory. Linear interpolation can be used to determine intermediate positions between jthese calibrated points. The slave PTZ camera view can thus be sent to a position in approximate matching with the desired RoI. In these analog solutions, it is possible for the slave camera to be commanded to move before a threshold- zoom level is exceeded. This pre-emption can reduce delays due to the transit time of the mechanical slave PTZ camera.

The output from the image processing circuitry 303 can be passed to the analog conversion and formatting circuitry 305 which can produce an analog output from the digital data. The outputs from the analog conversion and formatting circuitry 305 and the slave PTZ camera can be passed to video switching and output circuitry 306. This circuitry can be controlled by the control circuitry 304, which decides which video stream should be output on the appropriate output channel. The output from the video switching circuitry can be one or more analog video channels. Docket No. GRND.04

The master camera can have the functionality to support privacy regions which obscure user defined regions from being outputted, as described in the US patent application 11/178,232 (Attorney Docket No. GRND-02) entitled "Image Processing of Regions in a Wide Angle Video Camera." A mask representing the shape of the privacy regions defined in the master camera can be generated in the image processing circuitry 303. As the view from the slave PTZ camera and a perspective corrected RoI in the master camera are approximately matched, this mask can be applied to blank the appropriate approximate regions in the slave image using the video switching and output circuitry 306. An alternative embodiment uses a slave camera having its own privacy region support that can be calibrated through the master camera.

Figure 4 is a modified version of the embodiment in Figure 3. In this example embodiment, the video from the slave PTZ camera is digital instead of analog. Thus the decision of which outputs to show can be conducted in the digital domain by the 'multiplexing, compression & formatting circuitry' 405. This can include an option of outputting either a single video stream or a combination of streams using multiplexing. The camera system can have a digital output (e.g. MPEG4). The mask representing any privacy regions defined in the master camera can be applied in this circuitry 405 to blank the appropriate approximate regions in the slave image. Apart from this final stage, the rest of the process follows the same steps as the procedure described for Figure 3.

An alternative embodiment is shown in Figure 5 where the slave camera has an analog video output that can be converted into digital video using an analog-to-digital converter (ADC) 506. Otherwise it follows the same process and layout as described in the description for Figure 4.

The embodiment in Figure 6 illustrates a layout in which video switching might not take place. The video output from the slave PTZ camera does not have to be passed to any circuitry in the master camera. The control circuitry 604 in the master camera can still be responsible for moving the slave camera to the desired approximate location to view the selected RoI. The two cameras can have separate video outputs. The output from the slave PTZ camera can be either analog or digital video. The digital output from the image processing circuitry 603 in the master camera can be formatted and compressed as necessary in the output circuitry 605 before being digitally outputted to the base station for monitoring. Multiple MPEG4 outputs are possible. A variation from this embodiment is a system with analog video output from the master camera. Conversion from digital-to- analog and formatting of the digital data from the image processing circuitry 603 can take place in the output circuitry 605. A possible embodiment consists of two analog composite video channels. The output from the slave PTZ can be either analog or digital video. Privacy regions can be implemented by using the slave PTZ camera's individual circuitry and configuration settings. Docket No. GRND.04

Motion Detection in the master camera:

Motion detection in the master camera can be carried out in preferred embodiments by using the well-known background-subtraction method. There are many public domain documents describing background estimation in video. The background of the scene can be estimated in many ways. The background image of the scene can be defined as those pixels belonging to stationary objects in the scene. For example, in the public domain article by R. Collins et al entitled "A System for Video Surveillance and Monitoring," which is hereby incorporated by reference, a recursive background estimation method is proposed based on the equation:

B_n+I (k,l) = αB_n (k,l) +(l-a) I_n if I_n(k,l) is a stationary pixel,

= aB_n (k,l), if I_n(k,l) is a moving pixel. where I_n (k,l) represent a pixel in the n-th image frame I_n of the video captured by the master camera, and the image B_{n+ 1} is the estimated background image at time instant n+1, and a is a parameter between 0 and 1. This recursive equation provides a weighted sum of past image frames. Temporary objects disappear over long time averaging and stationary objects of the scene remain in the background. The background estimation process can be carried out by other means as well. Likewise, motion detection itself can be carried out by other methods, within the scope of the present innovations.

The moving pixels of the current image are preferably estimated by subtracting the current image I_n from the current background image B_n. These pixels are then connected to a moving blob by connected component analysis, which is a well-known image processing technique (see e.g., Fundamentals of Digital Image Processing by Anil Jain, Prentice-Hall,

NJ, 1988). Moving blobs in a video corresponds to persons or moving objects and they together with their local neighborhoods in the video define Regions of Interest because persons, moving objects, left objects in the scene etc. are important in security monitoring applications. Unlike conventional systems, which only capture recorded video for later analysis, real-time surveillance offers the added benefits of alarm and event triggers without delay. Therefore, such regions should be tracked and zoomed for closer inspection.

Tracking in the master camera:

Once a moving blob is detected, it can be tracked by the master camera. One preferred embodiment carries out the tracking in the master camera according to U.S. patent application 10/924,279, entitled "Tracking Moving Objects in Video Using Wavelet Domain Information" by A. E. Cetin and Y. Ahiska in the master camera using the wide- angle image frames of the video. In this patent application moving blobs are characterized by a one-dimensional histogram constructed from color and wavelet domain information of the blob. Blobs in the current image frame of the video and blobs in the previous image frame are compared to each other using the histograms of blobs. Histogram comparison is Docket No. GRND.04

carried out using the mean-absolute difference or the Bhattacharya coefficient. Blobs producing the smallest mean-absolute difference are associated with each other.

In another embodiment, tracking can be initiated by pointing on an object. If the clicked pixel is inside a moving blob then this blob is tracked as above in the plurality of image frames forming the video. If the clicked pixel is not a part of a moving blob then a region-growing algorithm is initiated around the clicked pixel and pixels having similar characteristics are combined to form a blob. The color and wavelet histogram of the estimated region is compared with the histogram of the same region in the next image frame. If the color and wavelet histogram of this region changes over time then this means that the object started moving. This also means that some portions the region are likely to be a part of a moving blob determined by the motion detection algorithm of the camera. Once a decision is made that this stationary object is now a part of a moving object, then it is tracked as described in the above paragraph.

In many surveillance applications it is very important to get a high quality picture of a person or a moving object. The tracking algorithm provides the necessary information to get a closer picture of the moving object. The wide-angle camera described in US patent application 10/837,325, entitled "Multiple View Processing in Wide-Angle Video Camera," has a zooming capability. This capability can be expanded by using a slave PTZ camera taking instructions from the master camera. An RoI encapsulating the center of mass of the tracked blob can be used to pass PTZ controls to the slave PTZ camera resulting in the salve camera zooming into the blob to achieve an expanded zoom capability.

The slave PTZ camera will often produce a sharper image of the blob compared to the corrected view of the blob obtained from the master wide-angle camera. This is because the master wide-angle camera zooms into a region by performing numerical interpolation, which smoothes the edges of the objects due to limited sensor resolution, in many cases leading to smooth pictures. By replacing the smooth picture obtained from the master camera with the sharp picture from the slave camera, the zooming capability of the system is expanded. As persons and moving objects (or objects which have moved in the past) are important in security monitoring applications, object tracking is useful in defining RoI for zooming to obtain a closer inspection.

As discussed earlier, a preferred embodiment can consist of a master camera and two slave cameras with optical zoom capabilities as shown in Figure IB. Alternative embodiments of the method and camera systems shown in Figures 2, 3, 4, and 5 can also consist of a single master and multiple slaves with optical zoom capabilities.

In one class of example embodiments, the method and systems have the flexibility of realizing the moving object tracking in the master camera or in slave cameras provided that slave cameras have built-in tracking capabilities. In one example of these embodiments, the master camera detects moving regions in the scene and assigns each moving region to a different slave camera. Each slave camera can then track a moving region using built-in tracking mechanisms. T/IB2005/003504

18

Docket No. GRND.04

Image Registration in the Master Camera:

The image of the moving blob and its immediate neighborhood captured by the slave PTZ camera and the corresponding view of the blob in the master camera can be registered in the master camera to achieve a high-quality picture of the RoI. Image registration can be implemented in the master camera in two stages. In the first stage some salient points in both a portion of the image of the master camera containing the moving blob and the transformed image of the PTZ camera can be determined by running the same algorithm in the two images. For example, if the master camera has an image from a fisheye, then the image of the slave PTZ camera is transformed into the distorted fisheye image coordinates. This also applies for other means of capturing wide-angle video such as using a peripheral lens. Since both images represent the same blob in the scene, the salient point detection algorithm should produce the same pixels as points of interest. Salient points can be determined using a wavelet domain method. After this stage the salient points of the two images can be matched to each other using the local color histograms around each point.

A flow chart describing an example embodiment of this image registration algorithm implementation for the master camera is illustrated in Figure 7. The current image I_n represents the raw wide-angle image of the master camera and the image J_n represents the transformed image of the slave camera. Using the I_n input, salient points in the image are determined (Step 702). Using the J_n input, salient points in the image are determined (Step 704). These two salient points are matched using local histogram comparison (Step 706). This information is then used to update the pixels of the image I_n of master camera using the pixels of J_n of slave camera (Step 708).

There are many public domain salient point detection algorithms in the literature (see e.g., the text book entitled, Fundamentals of Digital Image Processing by Anil Jain, Prentice-Hall, NJ, 1988). Commonly used ones include the Harris Corner detector and wavelet domain salient corner detectors. Wavelet transforms in two dimensions carry both space and scale (frequency) information. A salient point of an image can be defined as a pixel whose wavelet coefficients have relatively high amplitude values compared to other wavelet coefficients in all or some of high-frequency subband images of the wavelet transform. If a pixel is part of a flat region in the image, then its corresponding wavelet coefficients are ideally zero or very close to zero. If a pixel is on the horizontal (vertical) edge of an object then it produces high-amplitude wavelet coefficients in low-high (high- low) subband image and another set of high amplitude coefficients the high-high subband image obtained after one stage of the wavelet transform. On the other hand, if the pixel is on the corner of an object then it produces high-amplitude wavelet coefficients in low- high, high-low and the high-high subband images. Therefore significant corners of an image can be determined by thresholding high-amplitude wavelet coefficients in all subband images. It turns out that some of the salient points are on the corners and significant edges of the moving blob and its immediate neighborhood in the background part of the image. Docket No. GRND.04

Once the salient points of both images are detected they have to be matched to each other to register the two images coming from the master and slave cameras viewing the same object. The simplest matching process can be carried out by comparing the values of the corresponding pixels. However this may lead to incorrect results because an object may consist of a single color and all salient points may have the same or similar pixel values. Therefore it is better to compare the local neighborhoods around the salient points to achieve robust results. In some embodiments, this matching is performed by comparing the local color histograms around the salient points. A color histogram around a pixel can be determined in any color representation space. The most widely used color representation schemes include Red, Green, and Blue (RGB) and luminance and chrominance representations (YUV or YCrCb).

The normalized color histogram of a local region O around a salient point p in the image I_n of the master camera is expressed as

where s represents the color valued pixel s, O represents a local region around the salient pixel p, δ is the Kronecker-delta function, N is the number of data points in O, q is a quantizer function mapping the color space domain data into a L bit number. L is selected as 12 in this embodiment. The color histogram h_p, which is constructed from the color information around the pixel p characterizes this pixel.

Let h_q(k) is another normalized histogram around the salient pixel q in the image J_n of the slave camera. Histograms h_p(k) and h_q(k) can be compared to each other in many ways. Mean-absolute difference (MAD) gives a measure of comparison:

K- 1 H h_P - IIq II₁ = (IZK) Σ I hp(k) - h_q(k) I k=0 where K = 2¹² is the number of points in the normalized color histogram. If the mean- absolute difference distance between p-th salient point and the q-th salient point are smaller than distance between the other salient points then p-th salient point of the image I_n is assigned to the q-th salient point of the image J_n .

Other color histogram comparison measures include the mean square error, cross correlation, and the Bhattacharya measure:

*>(^Λ,Λ)=Σ * VM*⁾M*⁾

Higher the Bhattacharya measure D of h_p(k) and h_q(k) better the match between the histograms h_p(k) and hq(k). B2005/003504

20

Docket No. GRND.04

Once the salient points of images from the master and the slave cameras are matched the pixels of the image I_n are updated in the master camera using the pixels of J_n according to the matched salient points. Image registration in the system is preferably an iterative process. A typical commercial PTZ camera can take finitely many (e.g. 128) possible physical viewing positions. The initial position information provided by tracking algorithm may not be accurate and as a result the PTZ camera may cover only a part of the region of interest and/or tracked object. In such a case, some of the salient points determined on the image of the master camera may not be matched. This means that additional position information should be transmitted to the PTZ camera to match almost all of the salient points in both images. Also, the initially registered image may not be detailed enough. For example, the tracked blob may be only a small portion of the image returned by the slave PTZ camera. The slave camera should zoom into the scene so that the tracked blob should become a large portion of the image returned by the slave PTZ camera. In this case, additional commands can be transmitted to the slave camera in an iterative manner as well. The iterative process can be terminated after comparing the two images. The comparison of distorted slave image from the PTZ camera with the raw wide-angle video image can be performed using many commonly used image comparison measures including mean square error (MSE), mean absolute difference (MAD), and matching the colour histograms the two images. If the MSE, MAD or color histogram difference between the two images drops below a threshold then the iterative registration process is terminated.

In the above paragraphs, the image registration process is described for a single slave camera. Extension of the above image registration method to multiple slave cameras monitoring different RoI' s is straightforward. Video images produced by slave cameras are placed on the perspective corrected view of the master camera one by one.

Slave cameras can communicate with the master camera via RS485 bus or any other bus capable of carrying positional data information.

4

21

Docket No. GRND.04

Further Information

The following documents can be used for further information in the field of the invention and are hereby incorporated by reference.

References cited:

US Patent Serial No. 6,509,926, entitled "Surveillance Apparatus for Camera Surveillance System," which is hereby incorporated by reference.

US Patent Serial No. 6,724,421, entitled "Video Surveillance System With Pilot and Slave Cameras," which is hereby incorporated by reference.

US Patent Serial No. 6,147,709, entitled "Method and Apparatus for Inserting a High Resolution Image Into a Low Resolution Interactive Image to Produce a Realistic Immersive Experience," which is hereby incorporated by reference.

US Patent Application Serial No. 10/837,326 (Attorney Docket No. GRND- 15), filed 04/30/04, entitled "Multiple Object Processing in Wide-Angle Video Camera," which is hereby incorporated by reference.

US Patent Application Serial No. 10/837,325 (Attorney Docket No. GRND-14), filed 04/30/04, entitled "Multiple View Processing in Wide-Angle Video Camera," which is hereby incorporated by reference. US PatentApplication Serial No. 10/837,019 (Attorney Docket No. GRND-16), filed 04/30/04, entitled "Method of Simultaneously Displaying Multiple Views for Video Surveillance," which is hereby incorporated by reference.

US Patent Application Serial No. 10/924,279 (Attorney Docket No. GRND-05), filed 08/23/04, entitled "Tracking Moving Objects in Video Using Wavelet Domain Information," by A. E. Cetin and Y. Ahiska, which is hereby incorporated by reference. US Patent Application Serial No. 10/837,012 (Attorney Docket No. GRND-13), filed 04/30/04, entitled "Correction of Optical Distortion by Image Processing," which is hereby incorporated by reference.

US Patent Application Serial No. 11/178,232 (Attorney Docket No. GRND-02), filed 07/08/05, entitled "Image Processing of Regions in a Wide Angle Video Camera," which is hereby incorporated by reference.

Public Domain Documents:

[I]- X. Zhou, R. Collins, T. Kanade, and P. Metes, "A Master-Slave System to Acquire Biometric Imagery of Humans at Distance," ACM International Workshop on Video Surveillance, November, 2003, which is hereby incorporated by reference. [2]- R. Collins, Lipton and Kanade, "A System for Video Surveillance and Monitoring," in Proc. American Nuclear Society (ANS) Eighth International Topical Meeting on Robotics and Remote Systems, Pittsburgh, PA, April 25-29, 1999, which is hereby incorporated by reference.

[3]- "Fundamentals of Digital Image Processing" by Anil Jain, Prentice-Hall, NJ, 1988, which is hereby incorporated by reference. Docket No. G RND.04

Modifications and Variations

As will be recognized by those skilled in the art, the innovative concepts described in the present application can be modified and varied over a tremendous range of applications, and accordingly the scope of patented subject matter is not limited by any specific exemplary teachings given.

For example, it is contemplated that the present innovations can be implemented using any number of different structural implementations. An alternative embodiment of the present invention includes, but is not limited to, a single physical structure housing both master and slave cameras and all necessary circuitry, separate housings for the master camera, all slave cameras, and all necessary circuitry or any combination of the above housing distributions. In another class of contemplated embodiments, the present innovations can be implemented by adding a fourth axis of mobility to the slave camera. For example, the slave camera can Rotate as well as Pan, Tilt and Zoom.

Further, these innovative concepts are not intended to be limited to the specific examples and implementations disclosed herein, but are intended to included all equivalent implementations, such as, but not limited to, using different types of cameras for the master and slave cameras. This includes, for example, using PTZ controllable for both the master and slave cameras. This also includes, for example, using cameras with zoom capabilities for both the master and slave cameras, or for neither the master nor slave cameras.

In another class of contemplated embodiments, the present innovations can be implemented using, in addition to motion detection and object tracking, 3d-perspective view comparisons to identify the RoI. For example, if the master camera was aimed at a row of windows, the image processing circuitry could be programmed to ignore unimportant movement, such as leaves falling, and only identify as RoI open windows.

An alternative and less preferred embodiment of the present innovations can be implemented using optical, digital, mechanical, or any of a number of different ways of doing optical zooming.

An alternative embodiment utilizes two master cameras. These can be, but do not have to be, positioned facing in opposite directions. These cameras can be, but do not have to be, fish-eye cameras. The advantage of this embodiment is that a global perspective can be achieved through the use of master cameras that may not have 360- degree viewing capability otherwise. This embodiment does not exclude the use of one, single master camera with a 360-degree field of view, such as a dome camera.

In another class of contemplated embodiments, one or several master cameras can control multiple slave cameras. These master cameras can control the slave cameras each independently, in a hierarchy, or in any of a number of different ways. In one example of Docket No. GRND.04

this class of embodiments, one or several master cameras control one or several intermediate cameras, which control one or several slave cameras. An example implementation of this embodiment is the "daisy chain" the slave cameras so the master camera assigns separate tracking tasks either directly or indirectly through other slave cameras. The advantages of utilizing several slave cameras include, but are not limited to, obtaining different views of a single RoI, capturing several RoI, and/or following RoI as they pass behind physical structures. In an example of this embodiment, the slave cameras

In another embodiment, the slave camera can have built-in tracking capabilities. In this embodiment, the slave camera could take over the tracking job after the master camera had assigned it. The master camera could then assign another tracking task to another slave camera.

In another class of contemplated embodiments, the master and/or slave cameras can be equipped with any of a number of different vision enhancements, including, but not limited to, night vision, infrared vision, or heat-sensing ability. The advantages of thermal sensitivity include, but are not limited to, better detection and tracking of heat producing objects such as cars, people and/or animals. The advantages of utilizing night vision or other low-light vision enhancement include the ability to monitor an unlit area at night.

None of the descriptions in the present application should be read as implying that any particular element, step, or function is an essential element, which must be included in the claim scope: THE SCOPE OF PATENTED SUBJECT MATTER IS DEFINED ONLY BY THE ALLOWED CLAIMS. Moreover, none of these claims are intended to invoke paragraph six of 35 U. S. C. § 112 unless the exact words "means for" are followed by a participle. Moreover, the claims filed with this application are intended to be as comprehensive as possible: EVERY novel and non-obvious disclosed invention is intended to be covered, and NO subject matter is being intentionally abandoned, disclaimed, or dedicated.

Claims

T/IB2005/00350424Docket No. GRND.04CLAIMSWhat is claimed is:

1. A video camera system comprising of: a wide angle master video camera one or more slave video cameras (in close proximity to said master camera) capable of monitoring at least parts of the area covered by said master camera. automatically controlling the field of vision of said slave camera or cameras from said master camera wherein at least some of the control signals to said slave camera or cameras are based on information available to said master camera.

2. A system as recited in claim 1 wherein said control signals to said slave camera or cameras include an adjustment for the relative positions of said master and slave cameras.

3. A system as recited in claim 1 wherein said slave camera or cameras are mechanical PTZ video cameras .

4. A system as recited in claim 1 wherein said control signals to said slave camera or cameras are determined by reference to pre-designated PTZ presets within said slave camera.

5. A system as recited in claim 1 wherein the brightness, exposure level, color balance and other image qualities of said slave camera or cameras' video output are automatically adjusted by comparing the video images from said slave camera or cameras to that from said master camera.

6. A system as recited in claim 5 wherein said image qualities are adjusted by comparing the histograms of the video images from said slave camera or cameras to that from said master camera.

7. A system as recited in claim 1 wherein the wide-angle video camera has a fish eye lens.

8. A system as recited in claim 1 wherein the wide-angle video camera has a purpose designed lens to amplify peripheral vision.

9. A system as recited in claim 1 wherein at least part of said information available to said master camera is derived locally by a software program resident in the master camera by performing image processing on the wide-angle video.

10. A system as recited in claim 9 wherein said image processing is performed on a perspective corrected view generated from said wide angle master camera via a transformation engine within the camera.

11. A system as recited in claim 9 wherein said software program involves motion detection and determines moving regions of the video image.

12. A system as recited in claim 9 wherein said software program involves automatic moving region and object tracking. B2005/003504

25

Docket No. GRND.04

13. A system as recited in claim 11 wherein said software program involves automatic moving region and object tracking.

14. A system as recited in claim 9 wherein said software program controls the pre- designated PTZ presets of said slave camera so that tracked regions and objects are within the field of views of slave cameras.

15. A system as recited in claim 11 wherein said software program controls the pre- designated PTZ presets of said slave camera so that tracked regions and objects are within the field of views of slave cameras.

16. A system as recited in claim 12 wherein said software program controls the pre- designated PTZ presets of said slave camera so that tracked regions and objects are within the field of views of slave cameras.

17. A system as recited in claim 13 wherein said software program controls the pre- designated PTZ presets of said slave camera so that tracked regions and objects are within the field of views of slave cameras.

18. A system as recited in claim 9 wherein said software program assigns each detected moving region to a different slave camera by controlling the field of vision of the said slave cameras.

19. A system as recited in claim 11 wherein said software program assigns each detected moving region to a different slave camera by controlling the field of vision of the said slave cameras.

20. A system as recited in claim 9 wherein the said slave cameras may have automatic moving object tracking capabilities and they can independently track moving regions assigned to them by the master camera.

21. A system as recited in claim 11 wherein the said slave cameras may have automatic moving object tracking capabilities and they can independently track moving regions assigned to them by the master camera.

22. A system as recited in claim 18 wherein the said slave cameras may have automatic moving object tracking capabilities and they can independently track moving regions assigned to them by the master camera.

23. A system as recited in claim 19 wherein the said slave cameras may have automatic moving object tracking capabilities and they can independently track moving regions assigned to them by the master camera.

24. A system as recited in claim 1 wherein at least part of said information available to said master camera is obtained from external commands issued by the user.

25. A method of automatically expanding the zoom capability/resolution of a wide-angle video camera comprising the steps of: capturing a distorted wide-angle video image using a wide-angle master camera. using information available to said master camera to remotely control slave video cameras capable of monitoring at least parts of the area covered by said master camera. Docket No. GRND.04

26. A method as recited in claim 25 wherein video output from said slave cameras are routed through said master camera.

27. A method as recited in claim 26 wherein the output from said master camera is switched between video derived from said wide-angle video image and said routed slave videos depending on information available to the master camera.

28. A method as recited in claim 25 wherein said wide angle master camera is capable of extracting (a plurality of) perspective corrected views via a transformation engine within the camera, and wherein at least one of said perspective corrected views is automatically replaced by the views from said slave cameras depending on information available to the master camera.

29. A method as recited in claim 25 wherein the wide-angle video camera has a fish eye lens.

30. A method as recited in claim 25 wherein the wide-angle video camera has a purpose designed lens to amplify peripheral vision.

31. A method as recited in claim 25 wherein at least part of said information available to said master camera is derived locally by a software program resident in the master camera from the wide-angle video.

32. A method as recited in claim 25 wherein said software program involves motion detection and determines moving regions of the video image.

33. A method as recited in claim 25 wherein said software program involves automatic moving region and object tracking.

34. A method as recited in claim 32 wherein said software program involves automatic moving region and object tracking.

35. A method as recited in claim 25 wherein said software program controls the pre- designated PTZ presets of said slave camera so that tracked regions and objects are within the field of views of slave cameras.

36. A method as recited in claim 32 wherein said software program controls the pre- designated PTZ presets of said slave camera so that tracked regions and objects are within the field of views of slave cameras.

37. A method as recited in claim 33 wherein said software program controls the pre- designated PTZ presets of said slave camera so that tracked regions and objects are within the field of views of slave cameras.

38. A method as recited in claim 34 wherein said software program controls the pre- designated PTZ presets of said slave camera so that tracked regions and objects are within the field of views of slave cameras.

39. A method as recited in claim 25 wherein said software program assigns each detected moving region to a different slave camera by controlling the field of vision of the said slave cameras. 3504

27

Docket No. GRN D.04

40. A method as recited in claim 32 wherein said software program assigns each detected moving region to a different slave camera by controlling the field of vision of the said slave cameras.

41. A system as recited in claim 25 wherein said slave cameras may have automatic moving object tracking capabilities and they can independently track moving regions assigned to them by the master camera.

42. A system as recited in claim 32 wherein said slave cameras may have automatic moving object tracking capabilities and they can independently track moving regions assigned to them by the master camera.

43. A system as recited in claim 39 wherein said slave cameras may have automatic moving object tracking capabilities and they can independently track moving regions assigned to them by the master camera.

44. A system as recited in claim 40 wherein said slave cameras may have automatic moving object tracking capabilities and they can independently track moving regions assigned to them by the master camera.

45. A method as recited in claim 25 wherein at least part of said information available to said master camera is obtained from external commands issued by the user.

46. A method of reducing the response time for a mechanical slave PTZ camera in a multi-camera video system comprising the steps of: capturing a distorted wide-angle video image using a wide-angle master camera which is capable of extracting a perspective corrected view via a transformation engine within the camera. remotely controlling from said master camera mechanical slave PTZ video cameras capable of monitoring at least part of the area covered by said master camera. transmitting PTZ commands to said slave cameras from said master camera to approximately track said perspective corrected view when a predetermined threshold zoom level is exceeded by the zoom level of said perspective corrected view.

47. A method as recited in claim 46 wherein said perspective corrected view of said master camera is replaced by the view from said slave cameras when a remote zoom command for said perspective corrected view results in a zoom level outside the zoom range of said master camera.

48. A method of automatically expanding the zoom capability of a wide-angle video camera comprising the steps of: capturing a distorted wide-angle video image using a wide-angle master camera which is capable of extracting a plurality of perspective corrected views under remote PTZ control via a transformation engine within the camera. remotely controlling a mechanical slave PTZ video camera from said master camera. Docket No. GRND.04

transmitting estimated PTZ commands to said slave camera from the master camera to achieve an approximate match between a perspective corrected region of interest view from the master camera and the view from the slave both in direction and size. routing the digital video output of said slave camera to the said transformation engine in the master camera for reverse transformation into a distorted slave image which is distorted in a similar manner as the distorted wide angle video image. determining adjustments to said PTZ control signals by comparing said distorted slave image with said distorted wide angle image. transmitting said PTZ adjustment control signals to said slave camera to obtain a better match between the view from the slave-camera and the perspective corrected view for the said region of interest from the master camera. further iteratively repeating the previous three steps of this method in order to achieve desired matching accuracy. replacing the perspective corrected view of the master camera by the adjusted view from the slave camera so that a remote zoom command for the said region of interest which is outside the zoom range of the master camera may be routed to said slave camera to achieve an expanded optical zoom function.

49. A method as recited in claim 48 wherein said remote PTZ control signals to said master camera are issued locally by a software program resident in the master camera.

50. A method as recited in claim 49 wherein said software program involves motion detection and determines moving regions of the video image.

51. A method as recited in claim 49 wherein said software program involves automatic moving region and object tracking.

52. A method as recited in claim 50 wherein said software program involves automatic moving region and object tracking.

53. A method as recited in claim 48 wherein said estimated PTZ commands are determined by including an adjustment for the relative physical positions of said master and slave cameras.

54. A method as recited in claim 48 wherein said estimated PTZ commands are determined by reference to the pre-designated PTZ presets within the slave camera.

55. A method as recited in claim 48 wherein said slave PTZ camera has an analog video output which needs to be converted into a digital video signal.

56. A method as recited in claim 48 wherein the comparison of said distorted slave image with said distorted wide-angle video image is performed using an image matching technique based on comparing color and wavelet histograms.

57. A method as recited in claim 48 wherein the comparison of said distorted slave image with said distorted wide-angle video image can be performed using a commonly used image comparison measure. 3504

29

Docket No. GRND.04

58. A method as recited in claim 57 wherein mean square error is used as said image comparison measure.

59. A method as recited in claim 57 wherein mean absolute difference is used as said image comparison measure.

60. A method as recited in claim 57 wherein color histogram differencing is used as said image comparison measure.

61. A method as recited in claim 48 wherein said replaced perspective corrected image is one of the plurality outputs from the master camera displayed as a separate portion of a multi-view output.

62. A method as recited in claim 48 wherein said replacement has been performed by overlaying.

63. A method as recited in claim 48 wherein the brightness, exposure level, colour balance and other image qualities of said slave camera's video output are automatically adjusted by comparing the histograms of the video images from said slave camera to that from said master wide-angle camera.

64. A method of claim 48 wherein the wide-angle camera has a fish eye lens.

65. A method of claim 48 wherein the wide-angle camera has a purpose designed lens to amplify peripheral vision.

66. A method of claim 48 wherein the wide-angle camera has the capability of applying user defined privacy regions to the video output, whether the source video is obtained the camera's wide-angle image sensor or the slave camera or both.

67. A method of claim 48 wherein the zoom capability of said wide-angle video camera can be expanded by several slave cameras within the close proximity of the said master camera.

68. A method of automatically expanding the zoom capability of a wide angle ^vvideo camera comprising the steps of: capturing a distorted wide-angle video image using a wide-angle master camera which is capable of extracting a plurality of perspective corrected views under remote PTZ control via a transformation engine within the camera. remotely controlling a mechanical slave PTZ video camera from said master camera to capture regions of interest at a higher resolution. transmitting estimated PTZ commands to said slave camera from the master camera to achieve an approximate match between a perspective corrected region of interest view from the master camera and the view from the slave both in direction and size. replacing the perspective corrected view of the master camera by the adjusted view from the slave camera so that a remote zoom command for the said region of interest which is outside the zoom range of the master camera may be routed to said slave camera to achieve an expanded optical zoom function. T/IB2005/003504

30

Docket No. GRND.04

69. A method as recited in claim 68 wherein said estimated PTZ commands are calculated by including positional PTZ feedback information from said slave PTZ camera.

70. A method as recited in claim 69 wherein said stage of using positional PTZ feedback information to determine required PTZ commands is repeated iteratively to result in an improved match between said slave PTZ view and a perspective corrected region of interest view from the master camera both in direction and size.

71. A method as recited in claim 68 wherein said estimated PTZ commands are determined by including an adjustment for the relative physical positions of said master and slave cameras.

72. A method as recited in claim 68 wherein said remote PTZ control signals to said master camera is issued locally by a software program resident in the master camera.

73. A method as recited in claim 72 wherein said software program involves motion detection and determines moving regions of the video image.

74. A method as recited in claim 72 wherein said software program involves automatic moving region and object tracking.

75. A method as recited in claim 68 wherein said estimated PTZ commands are determined using a model of the behaviour of said slave PTZ camera when subject to PTZ commands. The said slave PTZ camera is intermittently calibrated to a known fixed position.

76. A method as recited in claim 75 wherein said estimated PTZ commands are determined by including an adjustment for the relative physical positions of said master and slave cameras.

77. A method of claim 68 wherein the wide-angle camera has the capability of applying user defined privacy regions to the video output, whether the source video is primarily obtained from the camera's wide-angle image sensor or the slave camera or both.

78. A method of claim 68 wherein the zoom capability of said wide-angle video camera can be expanded by several slave cameras within the close proximity of the said master camera.

79. A system comprising: a first camera which provides a first stream of output images in a wide-angle image geometry, and which has substantially no zoom capability; a monitor process which analyzes said first stream to identify regions of interest within said images; a second camera which provides a second stream of output images in a second image geometry, and which has expanded zoom capability relative to said first camera; and a position control system which automatically aims said second camera toward a current one of said regions of interest. B2005/003504

31

Docket No. GRND.04

80. A system comprising: a first camera which provides a first stream of output images in a wide-angle image geometry, and which has substantially no zoom capabilities; a monitor process which analyzes said first stream to identify regions of interest within said images; a second camera which provides a second stream of output images in a second image geometry, and which has expanded zoom capability relative to said first camera, and which is physically steerable relative to said first camera; and a position control system which receives from said monitor process at least a partial identification of said regions of interest, and which automatically aims said second camera in dependence thereon.

81. A system comprising: a first camera which provides a first stream of output images in a distorted wide- angle image geometry, and which has substantially no motion capabilities, and which has substantially no zoom capabilities; a monitor process which analyzes said first stream to identify regions of interest within said images; a second camera which provides a second stream of output images in a second image geometry, and which pans, tilts, and zooms in accordance with PTZ control inputs, and which has expanded zoom capabilities relative to said first camera; and a control system which receives from said monitor process at least a partial identification of said regions of interest, and which updates said PTZ control inputs in dependence thereon.

82. A method comprising the actions of: outputting a first stream of images from a first camera in a wide-angle image geometry; analyzing said first stream and identifying regions of interest within said first stream; outputting a second stream of images from a second camera in a second image geometry; aiming said second camera toward an identified region of interest through commands from a position control system at least partially dependent on output from said first camera.

83. A method comprising the actions of: outputting a first stream of images in a wide-angle image geometry from a first camera; B2005/003504

32

Docket No. GRND.04

monitoring said first stream to identify regions of interest within said images; outputting a second stream of images in a second image geometry from a second camera, which is physically steerable relative to said fist camera, and which has a narrower field of view relative to said first camera; controlling the field of view of said second camera based at least partially on said regions of interest using a position control system.

84. A method comprising the actions of: monitoring a stream of output images from a first camera in a distorted first image geometry, and identifying regions of interest within said images by comparison of different images within said stream; providing PTZ control inputs to control pan, tilt, and zoom of a second camera, in at least partial dependence on said regions of interest identified by said monitoring action; and outputting a second stream of output images in a second image geometry from said second camera.

85. A system comprising: a first camera which provides a first stream of output images in a first image geometry, and which has substantially no zoom capabilities; a second camera which provides a second stream of output images in a second image geometry, and which has expanded zoom capabilities relative to said first camera, and which said first and second cameras having at least overlapping fields of views; and a process which combines said first and second streams in real time by transforming one of said streams to match the image geometry of the other of said streams.

86. The system of claim 85, wherein said first camera has a field of view which surrounds said field of view of said second camera.

87. A method comprising the actions of: outputting a first stream of images in a first image geometry from a first camera; outputting a second stream of images in a second image geometry from a second camera having at least an overlapping field of view with said first stream; combining said first and said second streams of images in real time by transforming one of said streams to match the image geometry of the other of said streams. 3504

33

Docket No. GRND.04

88. A surveillance system, comprising: an anamorphic imager which outputs a distorted image sequence corresponding to more than 5 steradians of field of view, and which has limited or no zoom capabilities; and a zoom imager which is automatically controlled in dependence on object and/or motion tracking of said sequence, and which has zoom capabilities relative to said first camera.

89. The system of claim 88, further comprising as least on image processing unit which combines the outputs of both said imagers.

90. A method comprising the actions of: outputting a distorted image sequence corresponding to more than 5 steradians of field of view from an anamorphic imager that has limited optical zoom capability; controlling a zoom imager automatically in dependence on object and/or motion tracking of said outputted sequence.