Search Images Maps Play YouTube News Gmail Drive More »
Sign in
Screen reader users: click this link for accessible mode. Accessible mode has the same essential features but works better with your reader.

Patents

  1. Advanced Patent Search
Publication numberUS20030231769 A1
Publication typeApplication
Application numberUS 10/175,236
Publication dateDec 18, 2003
Filing dateJun 18, 2002
Priority dateJun 18, 2002
Also published asUS8752197
Publication number10175236, 175236, US 2003/0231769 A1, US 2003/231769 A1, US 20030231769 A1, US 20030231769A1, US 2003231769 A1, US 2003231769A1, US-A1-20030231769, US-A1-2003231769, US2003/0231769A1, US2003/231769A1, US20030231769 A1, US20030231769A1, US2003231769 A1, US2003231769A1
InventorsRudolf Bolle, Lisa Brown, Jonathan Connell, Arun Hampapur, Sharatchandra Pankanti, Andrew Senior, Ying-Li Tian
Original AssigneeInternational Business Machines Corporation
Export CitationBiBTeX, EndNote, RefMan
External Links: USPTO, USPTO Assignment, Espacenet
Application independent system, method, and architecture for privacy protection, enhancement, control, and accountability in imaging service systems
US 20030231769 A1
Abstract
The system and method obscures descriptive image information about one or more images. The system comprises a selector for selecting the descriptive image information from one or more of the images, a transformer that transforms the descriptive information into a transformed state, and an authorizer that provides authorization criteria with the image. In a preferred embodiment, the transformed state is the respective image encoded with the descriptive information. The descriptive information can be obscured so that the descriptive information in the transformed state can be decoded only if one or more authorization inputs satisfy the authorization criteria.
Images(16)
Previous page
Next page
Claims(35)
We claim:
1. A system for obscuring descriptive image information about one or more images, the system comprising:
a selector for selecting the descriptive image information from one or more of the images;
a transformer that transforms the descriptive information into a transformed state, the transformed state being the respective image encoded with the descriptive information, where the descriptive information is obscured; and
an authorizer that provides authorization criteria with the image, the descriptive information in the transformed state capable of being decoded only if one or more authorization inputs satisfy the authorization criteria.
2. A system, as in claim 1, where the descriptive information includes any one or more of the following: an image of a person, an image of a face, a biometric of a person, an image of an object, and a descriptive information about the object.
3. A system, as in claim 1, where the obscuring of the descriptive information includes any one or more of the following: a change of a background of the image, a change in sequence of two or more frames of the image, a removal of one or more frames of the image, insertion of one or more additional frames in the image, a change of an identity of one or more objects in the image, a removal of one or more objects in the image, a replacement of one or more objects in the image with a replacement object, a replacement of one or more object in the image with a neutral replacement object, a change in one or more tracks in the image, a replacement of one or more tracks with a replacement track in the image, a deletion of one or more tracks of the image, an insertion of one or more inserted tracks in the image, a change in an action in the image, a removal of an action in the image, a replace of an action in the image with a replacement action, a change of an interaction in the image, a deletion of an interaction in the image, a replacement of an interaction in the image with a replacement interaction, an addition of an additional interaction to the image, a change in a model, an addition of one or more models, a replacement of one or more models, a deletion of one or more models, a change in one or more labels, a replacement of one or more labels, an addition of one or more labels, and a deletion of a label.
4. A system, as in claim 1, where the authorization criteria are any one or more of the following: an authorization to the entire image, an authorization to a portion of the image, an authorization for sensitive images information being images of people, an authorization of an authority, an authorization of a user of whom the sensitive image information was created, an authorization to access statistical information from the descriptive image information, and a biometric of a user from whom the senstive image information was created.
5. A system for encrypting one or more images in an image stream, the system comprising:
an image input device for receiving the images;
an extractor that extracts one or more objects from the image;
an processor that processes the image;
a selector that selects one or more pieces of descriptive image information about the images according to a profile; and
an transformer that obscures one or more of the pieces of descriptive information using an obscuring method selected according to the profile to create a transformed state of the image.
6. A system, as in claim 5, where the processor at least classifies one of more of the objects into an object class and the descriptive information is about the object class.
7. A system, as in claim 5, where the processor at least identifies a relative identity of one or more of the objects in an object class and the descriptive information is about relative identity.
8. A system, as in claim 5, where the processor at least identifies an absolute identity of one or more of the objects in an object class and the descriptive information is about the object class.
9. A system, as in claim 5, where the processor analyzes one or more of the objects to detect parts of the objects and one or more interrelationships among the obects.
10. A system, as in claim 5, where the processor relates identity of one or more of the objects to the objects in one or more previous images in one or more previous frames to form tracks of objects.
11. A system, as in claim 5, where the descriptive information includes one or more of the following: an identity, a location, a track, a time, a region of interest, a foreground region, a model, an action, and an interaction.
12. A system, as in claim 5, where the transformer includes selectively performing one or more of the following: a encoding, a decimation, a de-identification, a removal, a transcription, a distortion, a randomization, an exclusion, a reordering, and an insertion.
13. A system as in claim 5, where the transformer requires an authorization.
14. A system, as in claim 5, where the transformer obscures by a decimation mechanism that includes any one or more of the following: a selective blurring, a selective iconization, and a selective miniaturization.
15. A system, as in claim 5, where the transformer obscures by a de-identification measure that includes any one or more of the following: a psuedo label, a psuedo-appearance, and a masking.
16. A system, as in claim 5, further comprising an authenticator that incorporates an access code with the obscured descriptive information.
17. A system, as in claim 5, further comprising an authenticator that incorporates an access code with the obscured descriptive information in the image.
18. A system, as in claim 16, where the access code includes any one or more of the following: a time, an image resolution, an image background location access, an image background detail, a foreground object, a foreground object identity, a foreground object location, a track access authorization to a foreground object, a track access authorization to a level of detail of a foreground object, a reaction of a foreground object, a level of detail to a reaction of a foreground object, an interaction between two or more foreground objects, a detail of an interaction between two or more foreground objects, a foreground object position, a foreground object component, a foreground object pose, and a foreground object action.
19. A system, as in claim 16, where the access code include any one or more of the following non-image access codes: a token, a key, a password, a biometric, and information that is pre-registered and exclusively known to authorized users.
20. A system, as in claim 16, where the access code is a biometric within the image.
21. A system, as in claim 5, where the extractor extracts any one or more of the following: one or more portions of the image depicting objects of interest; a classification of one or more foreground objects, a relative identity of one or more of the objects, a location of one or more of the objects in the image, an absoulte identity of one or more of the individual objects, an object pose, an object state, an object emotional state, a model for one or more of the foreground objects, a position of one or more of the forground objects, a context, a property, an appearance, a clothing style, a color, a frame-to-frame change in one or more of the objects, a reaction between one or more objects, and an object encoding.
22. A system, as in claim 5, further comprising an authorizer that provides an authorization criteria with the image.
23. A system, as in claim 22, where the authorization criteria exists in levels of authorization.
24. A system, as in claim 23, where the authorization levels permit decoding of a foreground of the image.
25. A system, as in claim 24, where the decoding of the foreground is any one or more of the following: a time, an image resolution, a foreground object, a foreground object identity, a foreground object location, a track access authorization to a foreground object, a track access authorization to a level of detail of a foreground object, a reaction of a foreground object, a level of detail to a reaction of a foreground object, an interaction between two or more foreground objects, a detail of an interaction between two or more foreground objects, a foreground object position, a foreground object component, a foreground object pose, and a foreground object action.
26. A system, as in claim 23, where the authorization levels permit decoding of a background of the image.
27. A system, as in claim 26, where the decoding of the background is any one or more of the following: a time, an image resolution, a background object, an interaction between a foreground object and the bachground.
28. A system, as in claim 23, where the authorization levels are determined by an identity of the user.
29. A system, as in claim 23, where the authorization levels are determined by an authority of the user.
30. A system, as in claim 23, where the authorization levels permit only statistical information about the sensitive information in the image and not to any aspect of any individual identity.
31. A system, as in claim 1, where the images are frames in a video stream.
32. A system, as in claim 3 1, where the video stream is processed any one or more of the following ways: (a) a frame analysis; (b) a transforming of extracted information (c) a selective encoding of information about one or more foreground objects; (d) a selective encoding of information about one or more backgrounds for each of the frames of the video; and (e) a creation of certificates for the encoded descriptions vouching for their authenticity.
33. A system, as in claim 1, where the imaging of next frame is controlled by the camera control inputs provided by an authorized user viewing the decoded video information.
34. A method for selecting descriptive information from one or more images, the method comprising the steps of:
selecting descriptive information from one or more of the images;
transforming the description image information into a transformed state defined by one or more pieces of descriptive information about the sensitive image information; and
providing one or more authorization criteria, the authorization criteria determined from one or more second pieces of descriptive information about the sensitive image information.
35. A method, as in claim 34, further comprising the step of:
decoding the transformed sensitve image information to an untransformed state to be provide to a user inputing the authorization input only if the second set of sensitive image information provided by a user match criteria about the sensitve image information.
Description
    FIELD OF THE INVENTION
  • [0001]
    This invention relates to the field of automatic imaging systems, that are capable of automatically capturing images (still and video) of a scene based on a combination of the specifications provided by the user. More specifically, the invention relates to the control of the information flow within the system and control of the dissemination of the information to the users, designers, and administrators of the systems.
  • BACKGROUND OF THE INVENTION
  • [0002]
    As fraud and violent crime in our society is escalating, video monitoring/surveillance is being increasingly used to either deter the criminals or collect acceptable evidence against the perpetrators. These video-based systems include outdoor surveillance of the violence prone public spaces as well as the indoor surveillance of the automatic teller machines (ATMs) and stores/malls. Some of the indoor monitoring may involve relatively low resolution imagery while other video footage may be highly detailed and intrusive. For instance, the Forward Looking Infrared (FLIR) video technology enables acquiring visual information beyond common physical barriers (e.g., walls) and millimeter radar enables visualization of the naked human body of a fully clothed person.
  • [0003]
    Like collection of any other personal data (e.g., name, telephone numbers, address, credit card information), acquisition of video footage is also raising many concerns among the public about their privacy. The public would like to know who all are collecting their video footage and where this video information is being collected. What level of detailed imagery does the acquired data consists? Who will have access to the video or the processed results? To what purpose the acquired video information will/can be used? Will they know when the video footage is abused (e.g., used for the purpose beyond its intended/publicized purpose)? Can people have control over the video which contains their personal information (e.g., can they demand destruction of such data?).
  • [0004]
    Increased automation can be used for gleaning individual information over extended periods of time and/or providing highly individualized content service to the contracted parties. This disclosure provides a means of reaching agreeable conditions for exchanging (or not-exchanging) sensitive individual information as it relates to the content provided by video/images of people infested environments.
  • PROBLEMS WITH THE PRIOR ART
  • [0005]
    Most of the video privacy literature consists of obliterating the raw video which may potentially contain personal information. For instance, Hudson's privacy ref protection scheme is based on the premise that lowering the information content will automatically obliterate the personal information. Low resolution imaging (footprints) has been used by IBM footprints
  • [0006]
    For obscuring individual identity. People are imaged in overhead low resolution passive infrared band where each pixel captures 2 sq ft. In both the approaches, not only identity is obscured but also the other details which may not be necessarily related to individual privacy.
  • [0007]
    Protection of privacy using different methods is studided by Zhao and Stasko: See Zhao, Qiang Alex and Stasko, John T., “Evaluating Image Filtering Based Techniques in Media Space Applications”, Proceedings of the 1998 Conference on Computer Supported Cooperative Work (CSCW '98), Seattle, Wash., November 1998, pp. 11-18.
  • [0008]
    Zhao and Stasko detect the moving objects in a video either by frame differencing or by background subtraction. In frame differencing, the (n−1)th video frame is pixelwise subtracted from the nth frame. If the pixel difference at location (e.g., ith row, jth column in the nth frame) is signficantly large, it is inferred that there is a moving object at that location in that frame. In background subtraction, a pixelwise model of the background (e.g., static) objects in the scene is initially acquired and subsequently updated. If a pixel at location (e.g., ith row, jth column in the nth frame) is signficantly different from its background model, it is inferred that there is a moving object at that location in that frame. Zhao and Stasko propose that the personal information is located within the pixels which cannot be explained by the background model and/or where the frame difference is large. If these pixels are masked/blanked out, they conjecture that the personal information in the video is obliterated. This method makes no distinction between moving objects, animals and humans. Further, there is no provision for a person to watch the original video which consists of exclusively their own personal data (e.g., was I wearing glasses that day?). Further, there is no provision for a person selectively watching their personal component of the data in a video which may show many people.
  • [0009]
    U.S. Pat. No. 5,615,391 issued to Klees, Mar. 25, 1997 of Eastman Kodak Company (also, EP 0 740 276 A2) discloses a system for an automated image media process station which displays images only while the customer present is detected thereby ensuring privacy. This disclosure detects presence of the customer using a camera installed in the self-development film processing kiosk and displays the photographic images being developed if the customer's presence is detected within field of view. This patent specifically relates to the privacy related to the single person detection and how it is applied to display of the images. It is not obvious how this disclosurecan be extended to a larger set of scenarios where not only person identity is important but also the person's actions and location of the imaging may be need to be selectively displayed.
  • [0010]
    Privacam is a method of sharing the sensitive information as disclosed in the public domain literature (Dutch newspaper, NRC):
  • [0011]
    Big Brother gebreideld, Apr. 28, 2001 NRC HANDELSBLAD—WETENSCHAP Http://www.nrc.nl/W2/Nieuws/2001/04/28/Vp/wo.html
  • [0012]
    The references herein cited are incorporated by reference in their entirety.
  • [0013]
    PrivaCam envisages a method of distributed encoding of the video frames by multiple parties so that viewing of each frame of video requires specific authorization is needed from all parties. This disclosure deals with control of the sensitive data by multiple parties. The scope of the disclsoure does not provide methods for selective viewing of individual locatio, and actions.
  • [0014]
    U.S. Pat. No. 5,828,751 issued to Walker et al. On Oct. 27 1998 titled Method and apparatus for secure measurement certification describes an invention where the measurements obtained from video are used for assuring the authenticity of the video. While this method is crucial to establishing the authenticity of the video evidence against (or in support of) an individual, it is not useful in providing general methods for protection of privacy of individuals depicted in the video.
  • [0015]
    U.S. Pat. No. 6,067,399 issued to Berger on May 23, 2000 titled “Privacy mode for acquisition cameras and camcorders” describes a method of detecting (skin tone) color of the objects being depicted in the video and selectively obscuring those colors so that the racial identity of the individuals depicted in the video is obliterated. The scope of the invention does not provide methods for selective viewing of individual locatio, and actions.
  • [0016]
    European Patent EP 1 081 955 A2 issued to Koji et al. on Jul. 03, 2001 titled “Monitor Camera system and method of displaying picture from monitor thereof” envisages masking out a predetermined portion of the field of view of camera covering “privacy zone”. For example, a public road in view of a camera monitoring a private property may blank out public road traffic from the camera field of view so that only the remaining portion is clearly visible in the video. One of the limitations of this invention is that it does not comperehensively deal with a number of aspects related with the invidual privacy (e.g., actions, individual identity based selective viewing).
  • OBJECTS OF THE INVENTION
  • [0017]
    An object of this invention is to provide individual/collective control of the way the sensitive information in the content of an image is being used.
  • [0018]
    An object of this invention is to segment sensitive information, like images, from an larger image, transform the segmeneted sensitive information, and control access to the transfored information.
  • SUMMARY OF THE INVENTION
  • [0019]
    The present invention is a system and method for obscuring descriptive image information about one or more images. The system comprises a selector for selecting the descriptive image information from one or more of the images, a transformer that transforms the descriptive information into a transformed state, and an authorizer that provides authorization criteria with the image. In a preferred embodiment, the transformed state is the respective image encoded with the descriptive information. The descriptive information can be obscured so that the descriptive information in the transformed state can be decoded only if one or more authorization inputs satisfy the authorization criteria.
  • BRIEF DESCRIPTION OF THE FIGURES
  • [0020]
    [0020]FIG. 1 describes the overall architecture of a typical instantiation of the system.
  • [0021]
    [0021]FIG. 2 presents an overview of a typical instantiation of the encoding system.
  • [0022]
    [0022]FIG. 3 illustrates a typical instantiation of video analysis and information extraction system.
  • [0023]
    [0023]FIG. 3A shows a summary of different components of video information extracted by a typical instantiation of the video analysis subsystem.
  • [0024]
    [0024]FIG. 4 describes an overview of a typical instantiation of transformation method undergone by information extracted from video.
  • [0025]
    [0025]FIG. 4A is a flow chart of a transformation process.
  • [0026]
    [0026]FIG. 5 shows typical different methods of selecting video information to be transformed.
  • [0027]
    [0027]FIG. 5A is a flow chart of a selection process.
  • [0028]
    [0028]FIG. 6 presents different typical obscuration methods used on the selected video information.
  • [0029]
    [0029]FIG. 6A is a flow chart of an obscuration process.
  • [0030]
    [0030]FIG. 7 shows a typical instantiation of encrypting transformed video information.
  • [0031]
    [0031]FIG. 7A shows a flow chart of typical encryption process for one component of video information.
  • [0032]
    [0032]FIG. 7B shows a flow chart of typical encryption process for entire video information.
  • [0033]
    [0033]FIG. 8 presents an overview of a typical instantiation of the decoding system.
  • [0034]
    [0034]FIG. 8A is a flow chart of a selective decryption process.
  • DETAILED DESCRIPTION OF THE INVENTION
  • [0035]
    The invention enables the content in the video to be selectively obscured/encrypted so that only authorized individuals have access to the sensitive information within the video and unauthorized individuals will only be able to access relatively insensitive information. Video generated by a typical video source 110 is fed into an encoding system 120 to generate an encoded video 130. The encoded video may consist of a combination of a means of selectively transforming of the original video information and a means of selectively encrypting different components of the transformed video information. The role of transformation is to any combination of the following functions: hiding sensitive video information, removing sensitive video information, distorting sensitive information. The role of the encryption is to provide selective access to the (transformed) video information to authorized users only. The encoded video 130 is thus a combination of transformed and encrypted video. The encoded video is decoded using a decoding system 170. The role of the decodeing to is regenerate the transformed video information. The authentication module 140 shows a means for verifying the authorizations of the users 160, generating appropriate keys for encoding and decoding the video. A query processor 180 facilitates restriction of user 160 access to only statistical information 191 within the video. Some authorized users based on their authorization can access the decoded video 190. By selective access to decoded video and statistical query processor, access to sensitive individual information can be controlled.
  • [0036]
    Referring to FIG. 2 presents further details of encoding system 120. A video source 210 represents one or more video sources. Each video source 210 could be either be a live camera or video file representing a camera input. Further, the camera could be either a static camera or dynamically controlled (e.g., pan-tilt-zoom) camera. The video consists of a sequences of images captured at successive time instants. Each image is called a frame. A video is typically characterized by the frequency of frame capture (e.g., frames per seconds), the resolution of each frame (e.g., pixels per frame), and the format in which the frames are represented (e.g., NTSC, MPEG2). It is typically assumed that all the video sources in a system are time synchronized (i.e., if frame m in video source 1 and frame n in video source 2 represent events are time t, then all other frames m+i in video source 1 and n+j in video source 2 represent the events at time t′).
  • [0037]
    Encoding system 120 performs analysis of the video from 110 and extracts information from the video. Encoding process results in many useful pieces of information about the content of the video. A preferred instantiation of the video analysis system 120 results in information about the background (static) objects in the scene and about the moving objects (e.g., foreground) objects in the scene. A more preferred instantiation of 120 presents many additional attibutes (e.g., locations, category, actions, interactions) of the foreground objects in the scene at successive times (e.g., frames). Further, 120 classifies each foreground object into various categories, at least one category being humans. Examples of other categories include vehicles, animals.
  • [0038]
    The appearance of the foreground regions are transmitted to an authentication module 140. Authentication module 140 (also see FIG. 3) identifies foreground region (e.g., humans) by comparing (e.g., matching) 250 the foreground region appearance and on the appearance of the known objects in the enrollment database 260. The enrollment database stores the information about the identities, biometrics, authorizations, and authorization codes of the enrolles in a database. That is, for instance, given an identity, the enrollment database can release information about the appearance (biometrics), appropriate authorization codes corresponding to that identity to the authentication module either in encrypted form or in unencrypted form.
  • [0039]
    Authentication modules 140 determines the identity of the objects represented by the foreground regions. The identity can be determined by the visual appearance of the objects in the video and/or a distinctive (may or may not be visual, e.g., radio freqeuency identification) signature emitted by the object. The following references describe specific instantiations of the authentication module operation. See:
  • [0040]
    A. W. Senior. Face and Feature Finding for a Face Recognition System In proceedings of Audio- and Video-based Biometric Person Authentication '99 pp. 154-159. Washington D.C. USA, Mar. 22-24, 1999
  • [0041]
    A. Senior. Recognizing faces in broadcast video, in proceedings of IEEE International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems ICCV 1999.
  • [0042]
    These references herein cited are incorporated by reference in their entirety. Authentication modules 140 may identify the objects based on a prior enrollment of the objects into a enrollment database. For instance, different appearances/signatures of a known object may be previously stored/registered within a database. Individuals objects (e.g., cars, persons) may then be identified by comparing the appearance/signature associated with a foreground region with those in the enrollment database. This method of identifying the objects is referred to as absolute identification.
  • [0043]
    On the other hand, each time a foreground region appears in the video, authentication module 140 may compare it against the appearances of the previously seen object foreground regions enrolled into an enrollment database 260 and determine whether a given foreground region can be explained based on the previously seen foreground regions or whether the given foreground region is associated with an object not seen heretofore. Based on the determination, the system associates the given foreground region with previously seen objects or enrolls the foreground regions as a new object into the enrollment database. This method identification does not absolutely determine the identity of object associated with a foreground region (e.g., William Smith) but only relatively identifies a foreground region associated with an object with respect to previously observed object foreground regions. This method of identification is referred to as relative identification.
  • [0044]
    Authentication modules 140 may use any combination of relative and absolute identification to identify the objects associated with the foreground regions. Authentication modules 140 may either use entire foreground region associated with an object or only portions of the foreground regions associated with an object. For instance, known parts of an object (e.g., face of a human) segmented by 370 (see FIG. 3) may be used by the authentication module 140 to determine the object identity.
  • [0045]
    Authentication module 140 may use any biometric identifier for authenticating the appearance of the humans. One of the preferred methods of authentication of humans is based on their faces.
  • [0046]
    The following reference describes an example the state of the prior art identity authentication system using face recognition. See:
  • [0047]
    A. W. Senior. Face and Feature Finding for a Face Recognition System In proceedings of Audio- and Video-based Biometric Person Authentication '99 pp. 154-159. Washington D.C. USA, Mar. 22-24, 1999
  • [0048]
    A. Senior. Recognizing faces in broadcast video, in proceedings of IEEE International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems ICCV 1999.
  • [0049]
    These references herein cited are incorporated by reference in their entirety.
  • [0050]
    One of the functions of the identity authentication systems is to assist in labeling the foreground objects. The objects recognized as same will be labeled identically. The label identity association is passed on to a key generation block. One of the functions served by the authentication module is to provide identity related information to key generation (280) and the transformation parameter generation (270) processes. Both key generation and transformation parameter generation processes in accordance with the system policies (275) generate appropriate control information in the system to determine the output of the encoding system. For instance, a particular system may want to implement blocking all the identity information of some specific individuals. In such situation, the identities information of all individuals is passed on by the authentication module (140 a) to the transformation parameters generation (270). Given this information to transformation parameter generation and the list of identities to blocked by the system policies module (275), the transformation parameter generation generates “block identity” transformation on that particular individual to the transformation (230).
  • [0051]
    Key generation module (280) facilitates selective encoding of the sensitive information so that only specific individuals can (at the decoding end) be able to access selective video information. For instance, say the system policies dictate that a specific individual (John Smith) be able to watch his own tracks to finest possible details (at the decoding end) and block all other users from watching fine details of John Smith's tracks. Given the identity information from the authhetication module (140 a), the key generation process (280) generates identity specific key to encode finest details of track information. The process of encryption (240) is also influenced by the transformation generation process (270).
  • [0052]
    Once the foreground objects are labeled by authentication module 140, they may undergo any combination of transformations by 230. The transformations 230 are performed on any sensitive information available within the foreground/background appearance. This invention envisages any combination of the following transformations to protect the sensitive information in the video: a change of a background of the image, a change in sequence of two or more frames of the image, a removal of one or more frames of the image, insertion of one or more additional frames in the image, a change of an identity of one or more objects in the image, a removal of one or more objects in the image, a replacement of one or more objects in the image with a replacement object, a replacement of one or more object in the image with a neutral replacement object, a change in one or more tracks in the image, a replacement of one or more tracks with a replacement track in the image, a deletion of one or more tracks of the image, an insertion of one or more inserted tracks in the image, a change in an action in the image, a removal of an action in the image, a replace of an action in the image with a replacement action, a change of an interaction in the image, a deletion of an interaction in the image, a replacement of an interaction in the image with a replacement interaction, an addition of an additional interaction to the image, a change in a model, an addition of one or more models, a replacement of one or more models, a deletion of one or more models, a change in one or more labels, a replacement of one or more labels, an addition of one or more labels, and a deletion of a label.
  • [0053]
    The type of the transformations could be modulated by the transformation profile specified (either dynamically, statically, or statistically/randomly) by 270. The transformation parameters 270 prescribed in the transformation profile are determined by the results of the authentication from the authentication module.
  • [0054]
    The method transforming (obscuring) the descriptive information includes any one or more of the following: a change of a background of the image, a change in sequence of two or more frames of the image, a removal of one or more frames of the image, insertion of one or more additional frames in the image, a change of an identity of one or more objects in the image, a removal of one or more objects in the image, a replacement of one or more objects in the image with a replacement object, a replacement of one or more object in the image with a neutral replacement object, a change in one or more tracks in the image, a replacement of one or more tracks with a replacement track in the image, a deletion of one or more tracks of the image, an insertion of one or more inserted tracks in the image, a change in an action in the image, a removal of an action in the image, a replace of an action in the image with a replacement action, a change of an interaction in the image, a deletion of an interaction in the image, a replacement of an interaction in the image with a replacement interaction, an addition of an additional interaction to the image, a change in a model, an addition of one or more models, a replacement of one or more models, a deletion of one or more models, a change in one or more labels, a replacement of one or more labels, an addition of one or more labels, and a deletion of a label.
  • [0055]
    The transformed objects are encrypted with (e.g., with public) keys specified by the key generation scheme 280. One of the properties of the key generation scheme is that consistently labeled objects are encrypted with consistent encryption keys.
  • [0056]
    [0056]FIG. 3 presents details of a typical instantiation 220 of video analysis and information extraction.
  • [0057]
    A video source 310 represents one or more video sources. Each video source could be either be a live camera or video file representing a camera input. Further, the camera could be either a static camera or dynamically controlled (e.g., pan-tilt-zoom) camera. The video consists of a sequences of images captured at successive time instants. Each image is called a frame. A video is typically characterized by the frequency of frame capture (e.g., frames per seconds), the resolution of each frame (e.g., pixels per frame), and the format in which the frames are represented (e.g., NTSC, MPEG2). It is typically assumed that all the video sources in a system are time synchronized (i.e., if frame m in video source 1 and frame n in video source 2 represent events are time t, then all other frames m+i in video source 1 and n+j in video source 2 represent the events at time t′). f1 600, f2 1000 are examples of video frames from a video source at different times t1 and t2, respectively.
  • [0058]
    A background estimation module 320 is designed to estimate the information about the appaearance relatively static objects in the scene, e.g., furniture, walls, sky etc. One method for background estimation assumes that the background is constant and will never change. This method will estimate the background objects only once (e.g., by using one or more video frames of the scene when no objects are in the field of view of the camera pointing to the scene) and from there onwards, use these static estimates of the background for the rest of the video processing.
  • [0059]
    The following reference describes an example the state of the prior art static background estimation method 320:
  • [0060]
    T. Horprasert, D. Harwood, and L. S. Davis. A statistical approach for real-time robust background subtraction and shadow detection. In IEEE International Conference on Computer Vision ICCV'99 Frame-Rate Workshop, 1999.
  • [0061]
    This reference herein is incorporated by reference in its entirety.
  • [0062]
    It is also possible to use other methods of background estimation 320 and use of such methods are within the scope of this invention. For instance, some background estimation algorithms will frequently update their estimates of the appearance of the background (e.g., every video frame or every few video frames) based on the currently arriving video frame.
  • [0063]
    The following reference describes an example the state of the prior art dynamic background estimation method:
  • [0064]
    K. Toyama, J. Krumm, B. Brumitt and B. Meyers, Wallflower: principles and practice of background maintenance, In IEEE International Conference on Computer Vision ICCV'99, pages 255-261, Volume 1, 1999.
  • [0065]
    This reference herein is incorporated by reference in its entirety.
  • [0066]
    Given a (static or dynamic) estimate of the appearance of the background, 330 compares the currently arriving frame of video with the estimate of the background appearance to determine the foreground (i.e., not background) objects. A background subtraction 330 function is performed on the input video frame. Typically, the background subtraction involves sophisticated image processing operation (and not simply a direct difference of the arriving frame and the background Estimate).
  • [0067]
    The following reference describes an example the state of the prior art static background subtraction method:
  • [0068]
    T. Horprasert, D. Harwood, and L. S. Davis. A statistical approach for real-time robust background subtraction and shadow detection. In IEEE International Conference on Computer Vision ICCV'99 Frame-Rate Workshop, 1999.
  • [0069]
    This reference herein is incorporated by reference in its entirety. Other examples include
  • [0070]
    K. Toyama and J. Krumm and B. Brumitt and B. Meyers, Wallflower: principles and practice of background maintenance, International Conference on computer vision (ICCV99), Pages 255-261, volume 1, 1999.
  • [0071]
    Into the woods: visual surveillance of noncooperative and camouflaged targets in complex outdoor settings Boult, T. E.; Micheals, R. J.; Xiang Gao; Eckmann, M. Proceedings of the IEEE, Volume: 89 Issue: Oct. 10, 2001 Page(s): 1382-1402.
  • [0072]
    This reference herein is incorporated by reference in its entirety.
  • [0073]
    Once all the foreground pixels are determined by the background subtraction 330, the mutually adjacent (e.g., connected) foreground pixels are grouped into single regions by a component analysis module 340. Further, in a preferred embodiment, the foreground regions comprising fewer than a threshold number T number of pixels are discarded (e.g., reclassified as background pixels). This procedure is referred to as connected component analysis 340. As a result of this processing, the foreground pixels identified by 330 are apportioned into foreground regions comprising of a sufficiently large number of pixels.
  • [0074]
    The following reference describes an example the state of the prior art connected component analysis method:
  • [0075]
    D. H. Ballard and Christopher M. Brown, “Computer Vision—Region Growing”, Department of Computer Science, University of Rochester, Rochester, N.Y., Prentice-Hall, Inc., Englewood Cliffs, N.J. 07632, pp. 149-165, 1982
  • [0076]
    This reference herein is incorporated by reference in its entirety.
  • [0077]
    A tracking module, 350 relates the foreground regions of frame n to those from earlier frames. Each foreground region in the current frame may represent an entirely new object, may correspond to foreground region from previous frames depicting moving object. Some of the previously seen objects (e.g., foreground regions from previous video frames) do not appear in the current frame because either these objects may have been occluded by other objects or these objects may have left the field of the camera. Locations of foreground regions corresponding to a single object in the successive video frames defines a track. The process of identifying the relationship among the foreground regions of different video frames (e.g., do two given foreground regions from two video frames depict same object or different object) is called tracking.
  • [0078]
    The basis for the determination of such relationships may include a combination of one or more of the following foreground region information: (a) the color of the foreground region or distribution of color of pixels within the foreground regions may be used to determine which foreground region in one video frame corresponds (or does not correspond) to which other foreground region in another video frame, (b) known/estimated speed/velocity (or their temporal distribution) of a foreground region may be used to determine which foreground region in one video frame corresponds (or does not correspond) to which other foreground region in another video frame (c) the shape/size of the foreground region or temporal distribution of shapes of the regions may be used to determine which foreground region in one video frame corresponds (or does not correspond) to which other foreground region in another video frame, (d) known ideal model of the foreground region may be used to determine which foreground region in one video frame corresponds (or does not correspond) to which other foreground region in another video frame.
  • [0079]
    The tracking module 350 identifies (and creates) a new track representation when a new (unseen till now) foreground region. 350 terminates a track when there are no more foreground regions corresponding to that track (e.g., last appearance of the foreground region corresponding to that object) in the successive video frames. The tracking module 350 continues (and updates) a track when it determines that a foreground region in the current video frame corresponds to a foreground region from a previous frame based on any combinations of the foreground region information described above.
  • [0080]
    It is possible that one or more (e.g., two) different objects come so sufficiently close to each other in the field of camera that they may be detected as a single foreground region in 340. This situation is called a merge. In such situation, the tracking module 350 also detects a merge and correctly identifies a merge and apportions fractions pixels of a single foreground region to different (previously seen) objects based on any combination of the foreground region information mentioned above. Further, the different regions thus split are correctly related to their corresponding previous foreground regions to continue their tracks.
  • [0081]
    On the other hand, it is also possible that a foreground region in the previous video may split into two separate foreground regions in the subsequent video frames. This situation is called a split. In such situation, 350 also detects a split—whether such a split is due to imaging noise/artifact in which case the two regions are treated as if they were single foreground region and the corresponding track is upated accordingly. If 350 determines that the regions are split due to separation of (heretofore detected as) a single object into two independent objects, the corresponding track is split into two tracks and their history/properties (e.g., shape, locations) are retrospectively updated.
  • [0082]
    The determination of tracks (e.g., position, size, shapes of an object at different video frames, i.e., times) enables the tracking module 350 to compute/estimate different additional attributes of the object (e.g., average speed, average velocity, instantaneous speed distributions, 3D orientation, gait). The tracking module 350 not only determines the tracks but also computes one or more these trajectory based attributes of (e.g., position, size, shapes, gross speeds, velocities, of an object at different video frames, i.e., times) each track.
  • [0083]
    Another function the tracking module 350 performs is filtering the tracks based on certain known properties of the tracks and updating the information about the scene accordingly. For instance, if detected foreground regions exhibit any real movement of a real object but only an artifacts of a movement for an extended period of time, it may determine such tracks to be spurious.
  • [0084]
    The following reference describes an example the state of the prior art tracking method:
  • [0085]
    I. Haritaoglu, D. Harwood, and L. S. Davis. W4: Real-time surveillance of people and their activities. IEEE Trans. Pattern Analysis and Machine Intelligence, 22(8):809-830, August 2000.
  • [0086]
    Pfinder: real-time tracking of the human body Wren, C. R.; Azarbayejani, A.; Darrell, T.; Pentland, A. P. Pattern Analysis and Machine Intelligence, IEEE Transactions on , Volume: 19 Issue: Jul. 7, 1997 Page(s): 780-785
  • [0087]
    These references are herein incorporated by reference in its entirety.
  • [0088]
    An object classification module 360 classifies the foreground object associated with each track into different object categoriesbased on their shape, color, size, movement patterns (e.g., gait). Typical classification methods classify moving objects as vehicles, humans, animals.
  • [0089]
    The following reference describes an example the state of the prior art object/track classifier method:
  • [0090]
    I. Haritaoglu, D. Harwood, and L. S. Davis. W4: Real-time surveillance of people and their activities. IEEE Trans. Pattern Analysis and Machine Intelligence, 22(8):809-830, August 2000.
  • [0091]
    Algorithms for cooperative multisensor surveillance Collins, R. T.; Lipton, A. J.; Fujiyoshi, H.; Kanade, T. Proceedings of the IEEE, Volume: 89 Issue: Oct. 10, 2001 Page(s): 1456-1477
  • [0092]
    These references are herein incorporated by reference in its entirety. The foreground regions can be further analyzed. A part identification module 370 decomposes the foreground regions, classified by the tracking module 350 (say, as humans), into regions that represent parts of object (e.g., parts of human body). This decomposition may be based on the shape of the silhouette of the human body, the image texture (appearance), motion information (e.g., portions of the foreground region depicting a human body the current video frame most similar to the portions of the corresponding foreground region in the previous video depicting a known body part). This functionality is often termed as shape (body part) identificition. The extent of the detailed body part decomposition depends upon image resolution, the application at hand, image details available. The foreground regions representing humans can be decomposed into any of the following body parts: (head,torso), (head, torso, left hand, right hand, left foot, right foot), (head, torso, left upper arm, left forearm, right upper arm, right forarm, left upper foot, left lower foot, right upper foot, right lower foot), or a set of body parts that include even more detailed description of the body including fingers, phalanges, so on and so forth. It is not necessary to choose a single body decomposition of the body into body parts and multiple decompositions of the a body shape into multiple body parts sets is also conceivable.
  • [0093]
    Once the body parts are identified, an estimate of the location, size, and pose of the body part can be estimated. One method of obtaining body part location, size, and pose by method of model fitting: each body part is modeled as an idealized shape (e.g., cylinder) and by fitting the idealized shape to the foreground region representing the body part, an estimate of the body part location, size, and pose is obtained. The part identification module 370 not only decomposes the foreground human regions into human body parts but also estimates location, size, pose of each body part.
  • [0094]
    The determination of position, size, shapes of parts of a human body as estimated by the part identification modle 370 at different video frames, i.e., times enables a part tracking module 380 to compute/estimate different additional attributes of the parts of the body (e.g., average speed, average velocity, instantaneous speed distributions, 3D orientation) related to the trajectory of the body parts. Either the absolute trajectories of each body part (i.e., with respect to absolute frame of reference) may be estimated or the relative motions/trajectories of each body part may be computed with respect to frame of reference of determined by the overall body.
  • [0095]
    Given the body part position, size, and pose for each part at different times, the part tracking module 380 determines one or more trajectory based attributes of (e.g., velocity, speed, orientations of human body parts at different video frames, i.e., times) parts of a human body. By collating the tracks of the individual body parts over a period of time and by conjoining the interrelationships among the body parts facilitates inferring the actions (e.g., walking, running, loitering) of the humans seen in the video.
  • [0096]
    Given actions of the individuals in the scene and their body part positions, sizes, and poses at different times, the interactions (e.g., hand shake, fight) between the individuals in the scene can be concluded by the part tracking module 380. Similarly, actions of the individuals and their inter-relationship with other objects reveal information about object-human interactions. The part tracking module 380, by collating the tracks of the individual body parts over a period of time and by conjoining the interrelationships among the body parts and the other objects facilitates inferring the interactions (e.g., picking, grabbing) of the humans with the other objects seen in the video.
  • [0097]
    The following reference describes an example the state of the prior art fitting/articulation/action method:
  • [0098]
    D. M. Gavrila and L. S. Davis, 3-D Model-based tracking of humans in action: a multi-view approach, CVPR96, pages 73-80, 1996
  • [0099]
    C. Bregler and J. Malik,Video Motion Capture, University of California, Berkeley, Http://www.cs.berkeley.edu/˜bregler/digmuy.html, 2001.
  • [0100]
    C. Bregler, Learning and Recognizing Human Dynamics in video sequences, CVPR97, pages 568-574, 1997.
  • [0101]
    These references are herein incorporated by reference in their entirety.
  • [0102]
    In summary, the video analysis and extraction process identifies foreground regions, tracks them, categorizes each object into categories (e.g., people, vehicles), segments each foreground region representing (say) a human into component body parts, tracks the body parts to infer the actions of individuals. Overall, video analysis and information extraction stage 220 provides overall object tracks, in case of (say) humans, it provides body part tracks (possibly at different resolutions) in addition to the overall body tracks. Other possible results of the video analysis processing include object labels (e.g., vehicles, birds, animals, humans), model fit parameters at different levels, sihouettes (same as regions in 340 without any “texture” information but only outlines of the regions), and texture maps (e.g., color of the pixels comprising regions in 340). Further, the objects tracks may contain information about the actions of the objects (381) and their interactions with the other objects (381). In addition, the tracks may contain information about the identity of the regions either in absolute form (e.g., John Smith) or in relative form (e.g., I saw this very same person in frame 385 but I do not know who she is). When absolute identity is required, the individuals objects are required to be enrolled into the database with the identity information (e.g., name) and identity vs measurements (e.g., biometrics) associations.
  • [0103]
    Each track encoding (object pose, overall object region silhouette, overall object appearance, overall object movement, object part movements, object actions, object interactions, object category, object identity) and background encoding (representation describing background scene) contains sufficient information in them to synthesize the object upto the information contained in the video to varying levels of abstraction representation. For instance, representations encoding of silhouette contains less detailed information than those contained in the texture maps (e.g., appearance in terms of color, edges, etc.). Similarly, representation containing detailed (refined) model fits (e.g., person region decomposed into upper arm, lower arm, upper leg, lower leg, abdomen, head) contains more information than that contained in the coarse model fit (e.g., hand, feet, abdomen, head) or object class labels (e.g., human, vehicle). Given the abstract representation about sizes, shapes, orientations of the body parts alongwith the information about the locations of the overall objects, their parts, their actions, and interactions in a video enables the system creation of a synthetic representation of video accentuating certain information in the video at the same time obscuring other information components in the video.
  • [0104]
    The function of the transformation method 230 is to selectively obscure one or more components of the information contained in the video that may potentially convey sensitive information. The transformation functionality is controlled by the transformation profile 270 which can be adjusted by the local system administrator and/or the results of the user authentication 141.
  • [0105]
    The information obscuration is performed in ajoint manner. Some extracted information in video analysis stage (220) is permanently obscured in the transformed methods 230. Further, obscuration of the information occurs in the decoding method 170 which selectively decodes part or full information contained in the video. In other words, it is possible to remove components of detailed representations of the objects/tracks/background from encoded video representation at the source (e.g., 230) so that selective decryption 820 simply cannot obtained that component of detailed information from the encoded representation of video.
  • [0106]
    How 230 and 820 share their functionality to achieve a desired level of privacy protection is a system design issue. A highly conservative system may not trust (for instance) the integrity of the video decoding system and therefore may remove all the sensitive information from the encoded video representation. In such case, no authenticated (ordinary) user will ever be able to access the sensitive information. In other systems, the encoding systems may include all the detailed information (e.g., detailed models of humans) in the video; at the receiving end, the decoding system may not allow any ordinary user to access the raw video/transformed video but only the statistical information in the transformed video through the statistical query processor 890.
  • [0107]
    We will first describe the transformation method 230. The transformation module 230 comprises two primary methods: (de)selection methods of portions of video 410 and obscuration methods 420 as shown in FIG. 4. The selection methods 410 select a portion of the video or extracted video information and then obscuration methods 420 apply the obscuration methods to the selected portion of the video exclusively and produce the transformed video 425.
  • [0108]
    The process of transformation method (230) is further explained in FIG. 4A. As mentioned earlier, the transformation method consists of selection methods (FIG. 5) and obscuration methods (FIG. 6). Similarly, the transformation process consists of selection process (717, FIG. 5A) followed by the obscuration process (719, FIG. 6A). The details of the method as a follows. Refer to FIG. 4A, at each instant of time.
  • [0109]
    [0109]FIG. 4A shows the process of transformation. Determined by the transformation profile 411 and the identities, locations, times etc. Of the video source content, the selection critiria are intialized 413. The process of the initialization could be a one time event or it could be occasional or it could be frequent. It can be triggered by the events in the scenery or it could driven by the administrative policies of the site or it could be remotely governed. Similarly, the obscuration criteria are also initialized (423).
  • [0110]
    At the onset every new video frame 415, the content of the video frame as represented by the encoding system 120 are compared by the selection process (417) against the selection policies prescribed at the time arrival of the video frame. An instance of the comparison could be, does the identity of person is included in the list of the identities to be selected as prescribed by the transformation profile? If the comparison yields an affirmative answer (“yes”), the representation in the frame is tagged to be “true” else the representation in the frame is tagged “false”. The tagged frame representation of the frame is passed on to obscuration process (419). The process of obscuration selects one or more methods of obscuration depending upon the current obscuration parameters. If the current video frame representation is tagged (selected) and if the current obscuration method(s) are meaningful for application to the current tagged frame representation, it applies the obscuration method to the current tagged video frame representation. Else the current video frame representation remains same and no obscuration method is applied to the current frame. The current video frame representation (whether it is obscured or not) is then passed on (421) to the next stage of processing (240).
  • [0111]
    The tagging of the frame is a binary instance. It is limited to “yes” or “no”. Thus a video frame representation is either selected or not selected depending upon the selection criteria. It is within the contemplation of the present invention to have a more richer choice of tagging which conveys the decision of selection process more finely. For instance, the selection criteria may select not the entire video but only a selected portion (object) within the video frame representation and tag that portion (object) for the benefit of obscuration process. In such case, the obscuration process will only limit its operation to the selected portion (objects) within the current frame representation. It is also contemplation of the present invention to have the selection process not be based on frame-by-frame basis (either from efficiency or for richer selection/obscuration process). Such enrichments of the present invention are obvious to those skilled in the art and are entirely within the scope of the present invention.
  • [0112]
    The (de)selection methods as listed in FIG. 5 (410) consists of any combination of the methods described below. By default, the system may consider all portions of video selected, deselected, or any intermediate combination specified by the transformation policies prescribed by the transformation profile (270) and in its turn controlled by the authentication module 140.
  • [0113]
    One of the ways of selecting certain portions of video is based on the location 501. i.e., either to select or deselect locations within video. In this (de)selection operation, one or more regions within a camera view are (de)selected. The regions could be of arbitrary shape and potentially could include entire field of view or none of the field of view. The location can be for example described as a region of interest within a field of view of camera. In the most preferred instantiation, the (de)selected regions are specfied in terms of regions in the video frame. In addition to the regions of interest, it is possible to (de)select the view/resolution of interest. For instance, in systems with pan-tilt-zoom cameras, it is possible to select a view of interest where a high resolution (e.g., zoomed-in) view of scene while deselecting the same scene with less detailed information. In calibrated (multi-)camera scenario and in situations where the correspondence between the actual world coordinates and the image coordinates are known & is invertible (e.g., by triangulation), the (de)selected regions could also be specified in terms of the world coordinates as a frustrums of viewing cones. In such situations, for instance, it is possible to select the objects in the view which are very close to camera and very far frame camera while excluding the objects in the intermediate range. The (de)selected regions can be static (permanently fixed), session static (fixed per session), or dynamic (frequently changing automatically, semi-automatically, and/or manually). When selected regions can be computed from deselected regions by exclusion.
  • [0114]
    One of the ways of selecting certain portions of video is based on the time 503, i.e., either to select or deselect durations within video. In this (de)selection operation, one or more time durations within a video time span are (de)selected. In the most preferred instantiation, the (de)selected durations are specfied in terms of durations in the video frame. In time synchronized (multi-)camera scenario and in situations where the correspondence between the actual real time and the video frames are known & invertible, the (de)selected durations could also be specified in terms of the real time. The units of duration specification may not be a monotonic function of the real time (e.g., number of scene changes, number of sessions, number of temporal events). The (de)selected durations can be static (permanently fixed), session static (fixed per session), or dynamic (frequently changing automatically, semi-automatically, and/or manually). When selected durations can be computed from deselected durations by exclusion.
  • [0115]
    The (de)selection of video of interest can also accomplished by nature of the foreground regions (507). For instance, it is possible to (de)select the video when there are foreground regions with signficant fraction of pixels with flesh tone color. Recall that the foreground regions are determined are determined in 330.
  • [0116]
    The (de)selection of video of interest can also accomplished by nature of the connected components 509 . For instance, it is possible to (de)select the video when the number of connected components in the video equals/exceeds a specified number. Recall that connected components are determined in 340.
  • [0117]
    The (de)selection of video of interest can also accomplished by the nature of tracks 511 . For instance, it is possible to (de)select the video when the length (e.g., time duration) of a track of a person signficantly exceeds a average length of tracks made by other people in the scene in a sensitive area. Recall that the tracks are determined in 350.
  • [0118]
    The (de)selection of video of interest can also accomplished by the nature of object categories . For instance, it is possible to (de)select the video when certain configurations of object categories appear together in the scene (e.g., a person and a knife). Recall that the object categories are determined in 360.
  • [0119]
    The (de)selection of video of interest can also accomplished by the nature of identities 505. For instance, it is possible to (de)select the video when certain identities (e.g., registered visually impaired person) in the watch list appear in the scene in a sensitive area (e.g., swimming pool). Recall that the identities are determined in authentication module 141.
  • [0120]
    The (de)selection of video of interest can also accomplished by actions performed by the people in the scene 515. For instance, it is possible to (de)select the video when there are loitering people in the scene. The actions of the people in the scene are determined in 380.
  • [0121]
    The (de)selection of video of interest can also accomplished by nature of the interactions among humans or interactions between humans and objects in the scene 517. For instance, it is possible to (de)select the video when a person leaves an object behind in a scene or when two people share askance glances. The interactions among objects are determined in 380.
  • [0122]
    The (de)selection of video of interest can also accomplished by nature of the model fit 513. For instance, it is possible to (de)select the video when there are more than two persons presenting themselves in sufficient visual detail as to allow fine level model fit (e.g., head, torso, left upper arm, left forearm, right upper arm, right forarm, left upper foot, left lower foot, right upper foot, right lower foot). The model fitting is determined in 370.
  • [0123]
    [0123]FIG. 5A shows the process of (de)selection. Determined by the transformation profile and the identities, locations, times etc. Of the video source content, the selection critiria are intialized 520. For instance, a top secret location will always select the background location identifying information irrespective of the identities of the individuals within the scene and the times of the video capture. The process of the initialization could be a one time event or it could be occasional or it could be frequent. It can be triggered by the events in the scenery or it could driven by the administrative policies of the site or it could be remotely governed.
  • [0124]
    At the onset every new video frame 527, the content of the video frame as represented by the encoding system 120 are compared (525) against the selection policies prescribed at the time arrival of the video frame. An instance of the comparison could be, does the identity of person is included in the list of the identities to be selected as prescribed by the transformation profile? If the comparison yields an affirmative answer (“yes”) 530, the representation in the frame is tagged to be “true” else the representation in the frame is tagged “false” 535. The tagged representation of the frame is outputted and the process of selection continues to obtain the next frame of the video 527.
  • [0125]
    The function of the obscuration method 420 is to remove a certain (sensitive) information content in the track attributes. It takes a attributes of track extracted by the video analysis (e.g., texture maps, silhouettes, models fits, actions, interactions, object labels, object identities 405) and produces obscured versions of the same attributes (e.g., texture maps, silhouettes, models fits, actions, interactions, object labels, object identities 425). The nature of the obscuration is determined by the selection methods and the transformation parameters.
  • [0126]
    The obscuration component can be any combination of the following obscuration methods 420: location obscuration 603, time obscuration 609, identity obscuration 613, model obscuration 607, action obscuration 611, interaction obscuration 617, track obscuration methods 615 as depicted in the FIG. 6.
  • [0127]
    We first describe location obscuration method 603.
  • [0128]
    The location obscuration specifies whether for each selected region, allow the system to specify whether the transformation is to exclusively obscure the scene location or whether the both the location as well as the events in the locations are to be obscured. In the latter case, further specifications of the events to be selectively obscured can also be specified through a joint (pairwise) specification of the event (e.g., any of the other obscuration methods as described below) and location obscuration method matrix. For instance, whether all the individuals in the selected region to be obscured. Whether specific individuals and specific inviduals within this region of interest need to be obscured, etc.
  • [0129]
    One method of location obscuration is by removal. The background model as estimated by 520 can be completely removed from the scene to avoid dissemination of any information about the location of the scene.
  • [0130]
    One method of location obscuration is by iconization 603. The background model as estimated by 520 can be replaced by a general symbolic description to avoid dissemination of detailed information about the location of the scene.
  • [0131]
    One method of location obscuration is by caricaturization 603. The background model as estimated by 520 can be replaced by sufficiently general (perhaps, distorted) sketch of the scene to avoid dissemination of any detailed information about the location of the scene.
  • [0132]
    One method of location obscuration is by replacement 603. The background model as estimated by 520 can be replaced by a different background to avoid dissemination of information about the location of the scene.
  • [0133]
    One method of location obscuration 603 is by background model estimate rendered with a different resolution and/or with pseudo-color texture. For example, the background model as estimated by 520 can be replaced by a blurred version of the background estimate to avoid dissemination of detailed information (e.g., number of windows) about the location of the scene.
  • [0134]
    One method of location obscuration 603 is by random shuffling of individual (or blocks of) pixels background model estimate around. The random displacement of the pixels will create a meaningless background texture but the statistical analysis 890 may be able to infer some overall properties of the background (e.g., color of the background, is it likely to be outdoor) without actually revealing any particular information about the background location.
  • [0135]
    We now describe time obscuration method 609.
  • [0136]
    The time obscuration method specifies whether for each selected duration, allow the system to specify whether the transformation is to exclusively obscure the scene location or whether the both the location as well as the events in the locations are to be obscured. In the latter case, further specifications of the events to be selectively obscured can also be specified through a joint (pairwise) specification of the event (e.g., any of the other obscuration methods as described below) and time obscuration method matrix. For instance, whether all the individuals in the selected duration to be obscured. Whether specific individuals and specific inviduals within this region of interest need to be obscured, etc.
  • [0137]
    One method of time obscuration is by subsampling (e.g., dropping intervening video frames) the frame rate of the selected video. The subsampling need not be uniform and may be performed, for example, in the units of scene changes.
  • [0138]
    One method of time obscuration is by time warping of the frame rate of the selected video. The time warping typically involves stretching time scale as a continuous function of time (frame rate).
  • [0139]
    One method of time obscuration is by permutation of the video frames the selected video. The resulting video may be meaningless (without spending inordinately compute intensive process to correctly reassemble it) but the statistical analysis 890 may be able to infer some overall properties of the video (e.g., how many distinct individuals were in the video) without actually revealing any particular temporal information about the video..
  • [0140]
    We now describe identification obscuration method 613.
  • [0141]
    One method of identity obscuration is by removal of the presence of the regions representing the identity. For example, the regions extracted by the connected components (340) of a particular person can be synthetically removed.
  • [0142]
    One method of identity obscuration is by iconization, i.e, the regions representing the identity are removed and a generic symbol describing the object replaces the original texture map. For example, the regions extracted by the connected components (340) of a particular person can be replaced by symboic description.
  • [0143]
    One method of identity obscuration 613 is by silhouettes, .i.e, the regions representing the identity are removed and a silhouette of the region replaces the original texture map. For example, the regions extracted by the connected components (340) of a particular person can be replaced by silhouettes (region outlines without revealing any appearance information) the which obscure their true identity.
  • [0144]
    One method of identity obscuration 613 is by selective subsampling/blurring, .i.e, the regions representing the identity are blurred/subsampled to obscure the identity. For instance, any lossy compression schemes for foreground regions could be used for blurring.
  • [0145]
    One method of identity obscuration 613 is by model, i.e, the regions representing the identity are removed and a model fit of the region replaces the original texture map. For example, the regions extracted by the connected components (340) of a particular person can be replaced by skeleton fits (e.g., stick figure) which obscure the true identity of the human beings in the original scene. Any level of model representations could be used for obscuring the identity. The more detailed the model representation, the more distinctive information it can potentially convey about the identity of the person. For instance, very tall persons can be distinguished from the rest of the population simply based on their height.
  • [0146]
    We now describe model obscuration method 607.
  • [0147]
    One method of model obscuration 607 is by model shape substitution, i.e, the regions representing the identity are removed and a model shape parameters of a different identity in the video replaces the actions represented by the original texture map. Any level of model representations could be used for obscuring the identity. In this situation, it is more difficult to infer the true identity based on the shape parameters of the model although the temporal/behavioral biometrics (e.g., gait) may still give away the true identity of the person.
  • [0148]
    One method of model obscuration 607 is by model shape homogenization, i.e, the regions representing the identity are removed and a prototypical (e.g., average) model shape parameters replaces the actions represented by the original texture map. Any level of model representations could be used for obscuring the identity. In this situation, it is difficult to infer the true identity based on the shape parameters of the model although the temporal/behavioral biometrics (e.g., gait) may still give away the true identity of the person.
  • [0149]
    We now describe action obscuration method 611.
  • [0150]
    One method of action obscuration 611 is by model action removal, i.e, the sequence of model part movements representing the actions of an individual are removed and a coarse model shape (e.g., one cylinder) replaces the actions represented by the original texture map.
  • [0151]
    One method of action obscuration 611 is by model action homogenization, i.e, the sequence of model part movements representing the actions of an individual are removed and a prototypical (e.g., average) model action parameters replaces the actions represented by the original texture map.
  • [0152]
    One method of action obscuration 611 is by model action randomization, i.e, the sequence of model part movements representing the actions of an individual are removed and a random (e.g., average) model action parameters replaces the actions represented by the original texture map.
  • [0153]
    One method of action obscuration 611 is by model action iconization, i.e, the sequence of model part movements representing the actions of an individual are removed and a symbolic description of the action replaces the actions represented by the original texture map.
  • [0154]
    We now describe interaction obscuration method 617.
  • [0155]
    One method of interaction obscuration 617 is by model interaction removal, i.e, the sequence of model part movements representing the interactions of an individual are removed and a coarse model shape (e.g., one cylinder) replaces the interactions represented by the original texture map.
  • [0156]
    One method of interaction obscuration 617 is by model interaction homogenization, i.e, the sequence of model part movements representing the interactions of an individual are removed and a prototypical (e.g., average) model interaction parameters replaces the interactions represented by the original texture map.
  • [0157]
    One method of interaction obscuration 617 is by model interaction randomization, i.e, the sequence of model part movements representing the interactions of an individual are removed and a random (e.g., average) model interaction parameters replaces the interactions represented by the original texture map.
  • [0158]
    One method of interaction obscuration 617 is by model interaction iconization, i.e, the sequence of model part movements representing the interactions of an individual are removed and a symbolic description of the interaction replaces the interactions represented by the original texture map.
  • [0159]
    We now describe track obscuration method 615.
  • [0160]
    One method of track obscuration is deletion of the segments of the track. Another method of track obscuration is insertion of spurious segments of the track. Track can also be obscured by assigning incorrect/random labels to the tracks at the intersection of two or more tracks. Entirely spurious tracks can also be added with ficititious track labels.
  • [0161]
    The process of obscuration is illustrated in FIG. 6A. Determined by the transformation profile and the identities, locations, times etc. Of the video source content, the obscuration critiria are intialized 625. For instance, a top secret location will always obscure the background location identity irrespective of the identities of the individuals within the scene and the times of the video capture. The process of the initialization could be a one time event or it could be occasional or it could be frequent. It can be triggered by the events in the scenery or it could driven by the administrative policies of the site or it could be remotely governed.
  • [0162]
    At the onset every new video tagged video frame representation 637 (which is 540, FIG. 5A) the content of the video frame as represented by the encoding system 120 are checked to see if the video frame representation is tagged by the selection process (629). If the video frame representation is indeed not tagged, the video frame is not selected and hence no obscuration methods need to be applied. In such an event, the output video frame representation is identical to the input frame representation (635). If the video frame representation is tagged, it implies that the frame is a selected frame and may undergo obscuration. A further check is made to see if the current obscuration parameters imply any meaningful obscuration to the presentation tagged video representation. For instance, if the present obscuration parameters only imply an identity obscuration and the present video frame does not have any identities, the obscuration method is not relevant. In such an event, the tag in the video representation is removed (633) and the output video frame representation (635) is same as the input video representation. In all other cases, the obscuration method indicated by the present obscuration parameters are applied to the present video frame representation (631), the tag in the present frame removed (633) and the obscured representation is output (635). The process of obscuration then continues to obtain the next frame of the video 637.
  • [0163]
    As mentioned earlier, the transformation methods 230 track related attributes (e.g., texture maps, silhouettes, class labels, coarse model fits, one or more fine model fits, actions, interactions) and transform the attributes to result in transformed attributes of the tracks. Some of the transformation may involve loss of information. When a particular component of the data is not completely specified or different components of the obscuration methods prescribe conflicting transformations, a default transformation or default priority of obscuration methods resolves the ambiguity. These defaults are specified in 270.
  • [0164]
    Encryption method 240 (FIG. 7) takes the transformed attributes of the tracks as inputs and produces an encrypted (e.g., encoded by public keys as in DES) version of the track attributes. In this scheme, a separate key may be used to encode different components of the track information (e.g., texture maps, silhouettes, class labels, coarse model fits, one or more fine model fits, actions, interactions) related to an identity. The keys used for the encryption involve identities of objects (as determined by authentication system 141 decisions) related to one or more tracks. The keys encode a method of exclusive authorization to access the video data related to one or more video tracks. Another function of 240 is to provide digital signatures related to the encoded data so that the authenticity of the video information can be vouched for. The encrypted video information may contain encoded version of the transformed video as well as the original video. Further, the encrypted video may consist information about the estimated background models. The encrypted video information may also contain sufficient information about authentication standards (e.g., how conservative the positive person identification method should be) and authorization provided to each authenticated identities (e.g., Will Smith gets authorization to access only coarse information in the video containing his close-up shots). When tracks of individuals are entwined (e.g., because of interactions of the corresponding individuals), access to the track data may require authorizations from more than one individual. For many reasons (e.g., efficiency, simplicity), it is possible to use same keys for different attributes of a track, different attributes of different tracks, same attrbutes of different tracks.
  • [0165]
    Referring to FIG. 7A, the process of encryption of the transformed video is further explained for one generic video information content. As noted before (See FIG. 3A), there are several information components of the video information. These include, time 1800, background 1891, action 1880, interaction 1890, identity 1895, track 1810, texture map 1870, silhouette 1860, fine level models 1850, coarse level models 1830, 1840, object label 1820, and finally, the raw video information itself. FIG. 7A illustrates process of encrypting one (X) of these components of the information. Give a representation of the current frame of video (710), it extracts a particular information component (X) of the video information from the current frame (720). It additionally obtains key corresponding to information component X (possibly individual specific), from the key generation process (280), and encrypts the video information X using K to produce K(X).
  • [0166]
    [0166]FIG. 7B illustrates the overall encryption process. Give a representation of the current frame of video (750), it uses information specific ( X) encryption process described above (753) to encrypts the video information X using K to obtain the encrypted information produce K(X) for all X. It then collates all the encrypted specific (X) information to produce an overall vector of encrypted information (M, 755). The overall encrypted information is further encrypted with an overall key J to produce encrypted overall encrypted information (757). The encrypted overall encrypted information (757) may possibly include information about the identities, the system operating points, and other information which may be helpful at the decoding process. The overall encryption may also include processes such as compression to improve the bandwidth efficiency. The preferred embodiment of the encryption process uses public encryption method. Other methods of encryption are also within the scope of this invention.
  • [0167]
    Referring to FIG. 8, a preferred instantiation of the decoding system 170 is shown. It comprises a means of authenticating user authorizations 142. The decisions generated within the authentication system An authentication module 142 enable release of appropriate (e.g., private) keys in the key generation method 830. This, in turn, enables a selective decoding of the encoded video 410 (same as in 290) generating decoded raw video 860, selective parameters of the transformed video 821, or both). Note that 860 represents some or all of the information in original video (as in 110, 210). Note also that 821 some or all of the information in the transformed video information as generated in 230. The extent/nature of recovery of original (210, 821) information depends upon appropriate authorizations released from authentication module 142. The extent of the recovery of the information from encoded video depends upon the nature of keys/authorizations released by 830 which in their turn depend upon the decisions of authentication module 142. In other words, the loss of information content from original video (110,210), original transformed video parameters as generated in 230 to decoded raw video 860, decoded transformed video parameters 821 depends upon the key generation/authorization 830 released by the user authentication procedures 142. Given the recovered parameters of the transformed video 821, 870 synthesizes the transformed video 871.
  • [0168]
    The decoding of the video itself does not necessarily imply user access to the results. The system permit direct user access to the selectively synthesized tranformed video (897) and/or selectively decoded raw video (896) depending upon the sets of authorizations released by the user authentication module 142. Alternatively, the system may not allow direct user access to any video footage (either selectively recovered original raw video or selectively synthesized transformed video) at all and only permit user 850 to pose a (statistical) query 880 about the video content to the decoded raw video (860) and/or transform synthesized video (871) through a statistical query processor 890. The authorization to access the video and/or the query results is modulated by the authorization methods 875 as determined by the decisions made by the authentication module 142.
  • [0169]
    For instance, the law enforcement agencies requiring the most detailed viewing of the video will (after appropriate authentication procedures 142) be authorized (875) view the original video (896 which represents part or full of 210). Security personnel (after successful completion of appropriate authentication procedures 142) on the premise will be authorized (875) to see the transformed video (897 which is synthesized from part or full of the parameters of the transformed video information in 330) but may not be able to see the original video. A typical user, however, can neither see the original video (896) nor the transformed video (897). A typical user authentication 142 will authorize 875 passage of the transformed video 871, parameters of the transformed video 821 or the original raw video 860 to a statistical query processor 490 which will only provide query results 895 of queries of statistical nature (880) posed by user (850) and will not give away any sensitive information. Note that the law enforcement and security personnel may have access to the query processor 890 and its results 895.
  • [0170]
    Given the (private) keys released by the authorization module 830, Decryption method 820 (FIG. 9) takes an encrypted (e.g., encoded by public keys as in DES) version of the track attributes, encrypted background and encrypted raw video, generates a subset of transformed attributes of the tracks, background and raw video. What subset of the transformed track attributes are generated depends upon depending upon the number of valid (private) keys received. For instance, due to a limited the authorization of a user, only transformed silhouettes and transformed coarse models of individual tracks of all individuals in the video and fine models of his own track may generated. Default parameters of the decoding method may allow default decoding of the encrypted video information when keys corresponding to particular transformed attributes are not available/valid.
  • [0171]
    Given (a subset of) transformed attributes of the tracks, background and raw video, synthesis method (870, FIG. 10) generates the synthesized transformed video. Depending upon the authorization of the users, they may be able to directly access the video or will be able only access the statistical information in the transformed video through statistical query processor 490.
  • [0172]
    [0172]FIG. 8A presents the flowchart of the process of selective decryption (820). In step 801, the decoding system obtain the next encrypted overall encrypted information (757). It starts with initializing resetting a fresh decoded frame (803). The initialized state of the decoded frame of video may be prescribed by the system polcies at the video decoding end, or they may be determined by the outcome of the decoding process of the last frame. It is also envisaged that the default parameters of the decoding operation will be governed by the status/authorization of the user.
  • [0173]
    In step 805, it initializes the parameters of the video reconstruction (e.g., parameters of the 3D model). Again the default video parameters may be prescribed by the system policies or by the system inertia retained from the last frame. It then fetches the overall (private) key from the key generation process (807). In the next step (809), the system uses the overall key to decrypt the encrypted overall encrypted information of the current frame. If the user is already authenticated (by a system determined distinctive possession, knowledge, or biometrics) and is authorized to access the system, the overall key obtained from the key generation process will be able to decrypt the encrypted overall encrypted information. Based on the output of the decryption process, a test is performed to determine the success of the decryption process (811). If the test is not not successful, the system determines overall authorization failure and the situation is handled as determined by the system policy. If the overall decryption is successful, the decoding system now has access to overall vector of encrypted information (M, 755, FIG. 7A). It then obtains other keys Kx specific to the information content X the user is authorized to access (813) from the key generation process (830). In step 815, these specific keys Kx are then used to decrypt the corresponding specific information content in the contained in overall vector of encrypted information (M, 755, FIG. 7A). Note that access to specific key Kx does not necessarily mean access to information content X as the information X may have been blocked out at the encoding end. In step 817, the decrypted video information is passed on to the transformed video synthesis process (870, FIG. 8) and the decoded raw video process (860, FIG. 8). The decryption process then goes on to decrypt the next video frame (819). The video frame-by-frame decryption process as described here is a preferred embodiment. For efficiency or convenience, the decryption process can also be performed in other ways (e.g., using a segment of video at a time). Such embodiments of this invention are obvious to those skilled in the art and are under scope of this invention.
Patent Citations
Cited PatentFiling datePublication dateApplicantTitle
US5195135 *Aug 12, 1991Mar 16, 1993Palmer Douglas AAutomatic multivariate censorship of audio-video programming by user-selectable obscuration
US5659614 *Nov 28, 1994Aug 19, 1997Bailey, Iii; John E.Method and system for creating and storing a backup copy of file data stored on a computer
US5689718 *Jan 30, 1996Nov 18, 1997Mitsubishi Denki Kabushiki KaishaSystem and method for processing document information using password protected icons that represent document content
US6031575 *Mar 19, 1997Feb 29, 2000Sony CorporationMethod and apparatus for encoding an image signal, method and apparatus for decoding an image signal, and recording medium
US6886098 *Aug 12, 2000Apr 26, 2005Microsoft CorporationSystems and methods for compression of key sets having multiple keys
US6931531 *Sep 1, 1999Aug 16, 2005Matsushita Electric Industrial Co., Ltd.Image object recording, compression, and encryption method and system
US6937730 *Feb 16, 2000Aug 30, 2005Intel CorporationMethod and system for providing content-specific conditional access to digital content
US20020010860 *Jan 29, 2001Jan 24, 2002Chu Chang-NamMethod for operating internet site offering encrypted contents
US20020065678 *Aug 21, 2001May 30, 2002Steven PeliotisiSelect video
US20020087403 *Jan 3, 2001Jul 4, 2002Nokia CorporationStatistical metering and filtering of content via pixel-based metadata
US20020150239 *Apr 16, 2002Oct 17, 2002Vidius Inc.Method for personalized encryption in an un-trusted environment
Referenced by
Citing PatentFiling datePublication dateApplicantTitle
US7468662Jun 16, 2006Dec 23, 2008International Business Machines CorporationMethod for spatio-temporal event detection using composite definitions for camera systems
US7702633Mar 5, 2007Apr 20, 2010Microsoft CorporationPreviews providing viewable regions for protected electronic documents
US7868912 *Jan 11, 2011Objectvideo, Inc.Video surveillance system employing video primitives
US7932923Sep 29, 2009Apr 26, 2011Objectvideo, Inc.Video surveillance system employing video primitives
US7956905 *Feb 17, 2006Jun 7, 2011Fujifilm CorporationTitling apparatus, a titling method, and a machine readable medium storing thereon a computer program for titling
US8134457Jun 4, 2008Mar 13, 2012International Business Machines CorporationMethod and system for spatio-temporal event detection using composite definitions for camera systems
US8219554Dec 16, 2008Jul 10, 2012International Business Machines CorporationGenerating receptivity scores for cohorts
US8301443Nov 21, 2008Oct 30, 2012International Business Machines CorporationIdentifying and generating audio cohorts based on audio data input
US8411167Apr 2, 2013Fujifilm CorporationTitling apparatus, a titling method, and a machine readable medium storing thereon a computer program for titling
US8417035Apr 9, 2013International Business Machines CorporationGenerating cohorts based on attributes of objects identified using video input
US8479009Sep 17, 2010Jul 2, 2013International Business Machines CorporationWearable time-bracketed video authentication
US8493216Dec 16, 2008Jul 23, 2013International Business Machines CorporationGenerating deportment and comportment cohorts
US8493443Jan 4, 2005Jul 23, 2013Hewlett-Packard Development Company, L.P.Methods and apparatus for location determination and asserting and maintaining privacy
US8494159Apr 4, 2008Jul 23, 2013International Business Machines CorporationSystem and practice for surveillance privacy-protection certification and registration
US8548853 *Jun 8, 2005Oct 1, 2013Microsoft CorporationPeer-to-peer advertisement platform
US8553782 *Jan 4, 2008Oct 8, 2013Euclid Discoveries, LlcObject archival systems and methods
US8559764 *Jun 15, 2004Oct 15, 2013At&T Intellectual Property I, L.P.Editing an image representation of a text
US8564661Jul 26, 2007Oct 22, 2013Objectvideo, Inc.Video analytic rule detection system and method
US8565478Jul 15, 2010Oct 22, 2013Nederlandse Organisatie Voor Toegepast-Natuurwetenschappelijk Onderzoek TnoMethod of improving the resolution of a moving object in a digital image sequence
US8601494Jan 14, 2008Dec 3, 2013International Business Machines CorporationMulti-event type monitoring and searching
US8626505Sep 6, 2012Jan 7, 2014International Business Machines CorporationIdentifying and generating audio cohorts based on audio data input
US8639563Jun 27, 2007Jan 28, 2014International Business Machines CorporationGenerating customized marketing messages at a customer level using current events data
US8711217Dec 15, 2005Apr 29, 2014Objectvideo, Inc.Video surveillance system employing video primitives
US8738917 *Oct 14, 2004May 27, 2014Canon Kabushiki KaishaImage data verification
US8749570 *Dec 11, 2008Jun 10, 2014International Business Machines CorporationIdentifying and generating color and texture video cohorts based on video input
US8754901 *Oct 30, 2013Jun 17, 2014International Business Machines CorporationIdentifying and generating color and texture video cohorts based on video input
US8775238Sep 27, 2007Jul 8, 2014International Business Machines CorporationGenerating customized disincentive marketing content for a customer based on customer risk assessment
US8812355Jun 29, 2007Aug 19, 2014International Business Machines CorporationGenerating customized marketing messages for a customer using dynamic customer behavior data
US8831972Sep 27, 2007Sep 9, 2014International Business Machines CorporationGenerating a customer risk assessment using dynamic customer data
US8842154Jul 3, 2012Sep 23, 2014Euclid Discoveries, LlcSystems and methods for providing personal video services
US8902971Feb 20, 2013Dec 2, 2014Euclid Discoveries, LlcVideo compression repository and model reuse
US8908766Jan 4, 2008Dec 9, 2014Euclid Discoveries, LlcComputer method and apparatus for processing image data
US8942283Oct 6, 2009Jan 27, 2015Euclid Discoveries, LlcFeature-based hybrid video codec comparing compression efficiency of encodings
US8953065Feb 28, 2013Feb 10, 2015Facebook, Inc.Titling apparatus, a titling method, and a machine readable medium storing thereon a computer program for titling
US8954433Feb 22, 2012Feb 10, 2015International Business Machines CorporationGenerating a recommendation to add a member to a receptivity cohort
US8958557 *Apr 27, 2010Feb 17, 2015Verint Americas Inc.Systems and methods for protecting information
US8964835Dec 30, 2011Feb 24, 2015Euclid Discoveries, LlcFeature-based video compression
US9031857May 3, 2007May 12, 2015International Business Machines CorporationGenerating customized marketing messages at the customer level based on biometric data
US9031858Sep 26, 2007May 12, 2015International Business Machines CorporationUsing biometric data for a customer to improve upsale ad cross-sale of items
US9092808Sep 27, 2007Jul 28, 2015International Business Machines CorporationPreferred customer marketing delivery based on dynamic data for a customer
US9106977 *Dec 30, 2011Aug 11, 2015Euclid Discoveries, LlcObject archival systems and methods
US9122742Jun 3, 2013Sep 1, 2015International Business Machines CorporationGenerating deportment and comportment cohorts
US9165216Feb 10, 2012Oct 20, 2015International Business Machines CorporationIdentifying and generating biometric cohorts based on biometric sensor input
US9253433Nov 27, 2012Feb 2, 2016International Business Machines CorporationMethod and apparatus for tagging media with identity of creator or scene
US9253434 *Sep 18, 2013Feb 2, 2016International Business Machines CorporationMethod and apparatus for tagging media with identity of creator or scene
US9361623Sep 27, 2007Jun 7, 2016International Business Machines CorporationPreferred customer marketing delivery based on biometric data for a customer
US9378632Mar 10, 2014Jun 28, 2016Avigilon Fortress CorporationVideo surveillance system employing video primitives
US20040174998 *Mar 5, 2003Sep 9, 2004Xsides CorporationSystem and method for data encryption
US20050278627 *Jun 15, 2004Dec 15, 2005Malik Dale WEditing an image representation of a text
US20060020553 *Jul 26, 2004Jan 26, 2006Septon Daven WLicense proxy process to facilitate license sharing between a plurality of applications
US20060104444 *Nov 16, 2004May 18, 2006Arun HampapurSystem and practice for surveillance privacy-protection certification and registration
US20060170767 *Jan 4, 2005Aug 3, 2006Brassil John TMethods and apparatus for asserting and maintaining privacy
US20060192862 *Feb 17, 2006Aug 31, 2006Fuji Photo Film Co., Ltd.Titling apparatus, a titling method, and a machine readable medium storing thereon a computer program for titling
US20060248348 *Oct 14, 2004Nov 2, 2006Canon Kabushiki KaishaImage data verification
US20060282309 *Jun 8, 2005Dec 14, 2006Microsoft CorporationPeer-to-peer advertisement platform
US20070013776 *Jun 28, 2005Jan 18, 2007Objectvideo, Inc.Video surveillance system employing video primitives
US20070201694 *Mar 20, 2007Aug 30, 2007Bolle Rudolf MPrivacy management in imaging system
US20070291117 *Jun 16, 2006Dec 20, 2007Senem VelipasalarMethod and system for spatio-temporal event detection using composite definitions for camera systems
US20070291118 *Jun 16, 2006Dec 20, 2007Shu Chiao-FeIntelligent surveillance system and method for integrated event based surveillance
US20070294207 *Jun 16, 2006Dec 20, 2007Lisa Marie BrownPeople searches by multisensor event correlation
US20080235221 *Mar 5, 2007Sep 25, 2008Microsoft CorporationPreviews providing viewable regions for protected electronic documents
US20080249793 *Sep 27, 2007Oct 9, 2008Robert Lee AngellMethod and apparatus for generating a customer risk assessment using dynamic customer data
US20080249836 *Jun 27, 2007Oct 9, 2008Robert Lee AngellGenerating customized marketing messages at a customer level using current events data
US20080249837 *Sep 27, 2007Oct 9, 2008Robert Lee AngellAutomatically generating an optimal marketing strategy for improving cross sales and upsales of items
US20080249851 *Apr 3, 2007Oct 9, 2008Robert Lee AngellMethod and apparatus for providing customized digital media marketing content directly to a customer
US20080249856 *May 3, 2007Oct 9, 2008Robert Lee AngellMethod and apparatus for generating customized marketing messages at the customer level based on biometric data
US20080249857 *May 31, 2007Oct 9, 2008Robert Lee AngellGenerating customized marketing messages using automatically generated customer identification data
US20080249859 *Jun 29, 2007Oct 9, 2008Robert Lee AngellGenerating customized marketing messages for a customer using dynamic customer behavior data
US20080249867 *Sep 26, 2007Oct 9, 2008Robert Lee AngellMethod and apparatus for using biometric data for a customer to improve upsale and cross-sale of items
US20080249869 *Sep 27, 2007Oct 9, 2008Robert Lee AngellMethod and apparatus for presenting disincentive marketing content to a customer based on a customer risk assessment
US20080252727 *Jun 23, 2008Oct 16, 2008Lisa Marie BrownPeople searches by multisensor event correlation
US20080273088 *Jun 4, 2008Nov 6, 2008Shu Chiao-FeIntelligent surveillance system and method for integrated event based surveillance
US20090002492 *Jun 4, 2008Jan 1, 2009Senem VelipasalarMethod and system for spatio-temporal event detection using composite definitions for camera systems
US20090005650 *Jun 29, 2007Jan 1, 2009Robert Lee AngellMethod and apparatus for implementing digital video modeling to generate a patient risk assessment model
US20090006125 *Jun 29, 2007Jan 1, 2009Robert Lee AngellMethod and apparatus for implementing digital video modeling to generate an optimal healthcare delivery model
US20090006295 *Jun 29, 2007Jan 1, 2009Robert Lee AngellMethod and apparatus for implementing digital video modeling to generate an expected behavior model
US20090070163 *Sep 11, 2007Mar 12, 2009Robert Lee AngellMethod and apparatus for automatically generating labor standards from video data
US20090083121 *Sep 26, 2007Mar 26, 2009Robert Lee AngellMethod and apparatus for determining profitability of customer groups identified from a continuous video stream
US20090089107 *Sep 27, 2007Apr 2, 2009Robert Lee AngellMethod and apparatus for ranking a customer using dynamically generated external data
US20090089108 *Sep 27, 2007Apr 2, 2009Robert Lee AngellMethod and apparatus for automatically identifying potentially unsafe work conditions to predict and prevent the occurrence of workplace accidents
US20090183177 *Jul 16, 2009Brown Lisa MMulti-event type monitoring and searching
US20090310865 *Dec 17, 2009Jenn Hwan TarngVideo Surveillance System, Annotation And De-Annotation Modules Thereof
US20100013926 *Jan 21, 2010Lipton Alan JVideo Surveillance System Employing Video Primitives
US20100086062 *Jan 4, 2008Apr 8, 2010Euclid Discoveries, LlcObject archival systems and methods
US20100131263 *Nov 21, 2008May 27, 2010International Business Machines CorporationIdentifying and Generating Audio Cohorts Based on Audio Data Input
US20100148970 *Dec 16, 2008Jun 17, 2010International Business Machines CorporationGenerating Deportment and Comportment Cohorts
US20100150457 *Dec 11, 2008Jun 17, 2010International Business Machines CorporationIdentifying and Generating Color and Texture Video Cohorts Based on Video Input
US20100150458 *Dec 12, 2008Jun 17, 2010International Business Machines CorporationGenerating Cohorts Based on Attributes of Objects Identified Using Video Input
US20100153133 *Dec 16, 2008Jun 17, 2010International Business Machines CorporationGenerating Never-Event Cohorts from Patient Care Data
US20100153146 *Dec 11, 2008Jun 17, 2010International Business Machines CorporationGenerating Generalized Risk Cohorts
US20100153147 *Dec 12, 2008Jun 17, 2010International Business Machines CorporationGenerating Specific Risk Cohorts
US20100153174 *Dec 12, 2008Jun 17, 2010International Business Machines CorporationGenerating Retail Cohorts From Retail Data
US20100153180 *Dec 16, 2008Jun 17, 2010International Business Machines CorporationGenerating Receptivity Cohorts
US20100153389 *Dec 16, 2008Jun 17, 2010International Business Machines CorporationGenerating Receptivity Scores for Cohorts
US20100153390 *Dec 16, 2008Jun 17, 2010International Business Machines CorporationScoring Deportment and Comportment Cohorts
US20100153597 *Dec 15, 2008Jun 17, 2010International Business Machines CorporationGenerating Furtive Glance Cohorts from Video Data
US20100202611 *Aug 12, 2010Verint Americas Inc.Systems and methods for protecting information
US20100284567 *Nov 11, 2010Arun HampapurSystem and practice for surveillance privacy-protection certification and registration
US20110081046 *Jul 15, 2010Apr 7, 2011Nederlandse Organisatie Voor Toegepast-Natuurwetenschappelijk Onderzoek TnoMethod of improving the resolution of a moving object in a digital image sequence
US20110182352 *Oct 6, 2009Jul 28, 2011Pace Charles PFeature-Based Video Compression
US20110199510 *Aug 18, 2011Shuji OnoTitling apparatus, a titling method, and a machine readable medium storing thereon a computer program for titling
US20110317832 *Dec 22, 2009Dec 29, 2011Visioncare CorporationDynamic Image Reproduction Code Generation Device, Dynamic Image Distribution System, Execution Format Program, Recording Medium Containing The Program, Server, And Dynamic Image Reproduction Code Generation Method
US20120163446 *Jun 28, 2012Euclid Discoveries, LlcObject Archival Systems and Methods
US20140146202 *Sep 18, 2013May 29, 2014International Business Machines CorporationMethod and apparatus for tagging media with identity of creator or scene
US20150106194 *Dec 16, 2013Apr 16, 2015Elwha LlcMethods, systems, and devices for handling inserted data into captured images
US20150106628 *Nov 19, 2013Apr 16, 2015Elwha LlcDevices, methods, and systems for analyzing captured image data and privacy data
EP1615186A2Jul 7, 2005Jan 11, 2006Sony CorporationImage protection apparatus, imaging apparatus, and program
WO2008109218A1 *Feb 5, 2008Sep 12, 2008Microsoft CorporationPreviews providing viewable regions for protected electronic documents
WO2009091259A1 *Jan 19, 2009Jul 23, 2009Nederlandse Organisatie Voor Toegepast-Natuurwetenschappelijk Onderzoek TnoMethod of improving the resolution of a moving object in a digital image sequence
WO2009124819A1 *Mar 11, 2009Oct 15, 2009Siemens AktiengesellschaftMethod for the protection of personality rights when taking pictures of persons
Classifications
U.S. Classification380/210
International ClassificationG06T1/00
Cooperative ClassificationH04L9/0847, G06T1/0021
European ClassificationG06T1/00W
Legal Events
DateCodeEventDescription
Jun 18, 2002ASAssignment
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BOLLE, RUDOLF M.;BROWN, LISA M.;CONNELL, JONATHAN H.;ANDOTHERS;REEL/FRAME:013038/0379;SIGNING DATES FROM 20020603 TO 20020611
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BOLLE, RUDOLF M.;BROWN, LISA M.;CONNELL, JONATHAN H.;ANDOTHERS;SIGNING DATES FROM 20020603 TO 20020611;REEL/FRAME:013038/0379