US20020054083A1 - Media browser using multimodal analysis - Google Patents
Media browser using multimodal analysis Download PDFInfo
- Publication number
- US20020054083A1 US20020054083A1 US09/934,486 US93448601A US2002054083A1 US 20020054083 A1 US20020054083 A1 US 20020054083A1 US 93448601 A US93448601 A US 93448601A US 2002054083 A1 US2002054083 A1 US 2002054083A1
- Authority
- US
- United States
- Prior art keywords
- feature
- media file
- media
- confidence score
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/74—Browsing; Visualisation therefor
- G06F16/745—Browsing; Visualisation therefor the internal structure of a single video sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/7834—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using audio features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/7847—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using low-level visual features of the video content
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99941—Database schema or data structure
- Y10S707/99944—Object-oriented database structure
- Y10S707/99945—Object-oriented database structure processing
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99941—Database schema or data structure
- Y10S707/99948—Application of database or data structure, e.g. distributed, multimedia, or image
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S715/00—Data processing: presentation processing of document, operator interface processing, and screen saver display processing
- Y10S715/961—Operator interface with visual structure or function dictated by intended use
- Y10S715/965—Operator interface with visual structure or function dictated by intended use for process control and configuration
- Y10S715/966—Computer process, e.g. operation of computer
- Y10S715/968—Computer process, e.g. operation of computer interface for database querying and retrieval
Definitions
- This invention relates to a method and apparatus for reviewing an aural and/or visual and/or other representation of a media file. Specifically, the invention relates to using media content features to allow a user to more easily review a media file.
- Text documents often have many cues, such as headings, paragraphs, punctuation, etc., that allow a reader to quickly determine the beginning and end of different sections of the document and to aid the reader in finding areas of interest.
- video and audio browsing systems typically do not provide information to the user regarding simple features, like section beginning and end points, much less more complicated information, like the name of the speaker on a video clip.
- Such browsing systems typically offer only standard “VCR-type” playback control options, like play, stop, rewind, and fast forward. As anyone who has tried to find a specific video clip on a conventional video tape using a standard VCR will understand, it is often difficult to locate portions of interest in a video using the standard playback controls.
- media or media file is used to represent any data stream that contains information regarding video or other image information, audio information, text information and/or other information.
- a feature of a media file is a property of the video, audio and/or text information in the media file, such as video or audio format, or information relating to the content of the media file, such as the identity of a speaker depicted in a video sequence, occurrences of applause, video shot boundaries, or motion depicted in a video sequence.
- video or audio format such as video or audio format
- information relating to the content of the media file such as the identity of a speaker depicted in a video sequence, occurrences of applause, video shot boundaries, or motion depicted in a video sequence.
- Providing feature information in a media browsing system can be very useful for a user when identifying areas of interest in a media file, controlling media playback, editing a media file, or performing other operations with a media file. For example, graphically identifying areas in a media file where a particular speaker is shown on a video clip can allow a user to quickly determine and playback those portions that contain the speaker.
- Providing feature information to the user based on automatically identified features also eliminates the need for a user to manually index or otherwise mark significant portions of the media file for later retrieval.
- the invention can use existing methods for automatically identifying features in a media file to generate and provide feature information to a user to aid the user in browsing the media file.
- the invention provides a media browser that uses media feature information as an aid in navigating, selecting, editing, and/or annotating a media file.
- media features are selected by a user.
- media browsing functions such as play, rewind, stop, fast-forward, index, automatic slide show, and automatic preview, are controlled based on feature information.
- feature information for a selected feature is mapped to a corresponding confidence score.
- the media browser includes a feature indicator that provides information related to a corresponding selected feature based on a corresponding confidence score.
- a feature indicator combines at least two confidence scores and provides information based on the combination.
- a feature indicator provides information related to a confidence score based on a value of another confidence score.
- the invention also provides a method for browsing a media file.
- a feature of the media file being browsed is selected and information related to a confidence score for at least one selected feature is provided.
- the confidence score relates to the existence of a corresponding selected feature in the media file. Based on the information related to the confidence score, a portion of the media file is selected.
- a metadata value representing a time-wise evaluation of a feature in the media file is mapped to a corresponding confidence score.
- mapping of a metadata value to a corresponding confidence score is non-linear.
- mapping of a metadata value to a confidence score is dependent on a user-defined control value or values.
- FIG. 1 is an example screen display for a graphical user interface in accordance with the invention
- FIG. 2 shows example metadata values, corresponding confidence score information, and an example graphical depiction of corresponding feature information
- FIGS. 3 - 9 show different ways of graphically depicting information related to a feature in a media file
- FIG. 10 is an example screen display depicting how features can be combined and corresponding combined feature information can be shown
- FIG. 11 is an example screen display depicting how display of feature information relating to a corresponding feature can be conditioned on another feature
- FIG. 12 is a flow chart explaining basic steps of a preferred method for browsing a media file
- FIG. 13 is a schematic block diagram of a preferred media browser in accordance with invention.
- FIG. 14 is a flow chart of steps of a preferred method for selecting a media file to be browsed
- FIG. 15 is a preferred media selection screen
- FIG. 16 is a flow chart of steps of a preferred method for mapping feature metadata to a corresponding confidence score
- FIG. 17 is a flow chart of steps of a preferred method for processing confidence scores and providing information related to the confidence scores
- FIG. 18 is a flow chart of steps of a preferred method for controlling playback functions based on a selected feature.
- FIG. 19 is a flow chart of steps of a preferred method for reviewing a media file based on selection of a confidence score indication.
- browsing refers to any operation related to reviewing, manipulating, editing, selecting portions of, storing, transmitting, or other operations related to a media file.
- a preferred embodiment of a graphical user interface for a media browser is described below having specific functions. In many cases, various functions of the media browser can be eliminated or altered, as desired. For example, information that is displayed graphically in the embodiment described below can be provided aurally or otherwise, as described more fully below.
- a common video analysis technique is the detection of shot boundaries, which represent significant differences between consecutive video frames.
- the difference between consecutive video frames can be determined based on statistical properties of the pixels in the frame images, histogram differences, compression algorithms, edge differences, or motion detection.
- Application of one of several known techniques for detecting shot boundaries results in a series of time-varying metadata values that each represent a difference between consecutive video frames.
- metadata regarding the shot boundary feature is determined, information can be provided to a user to aid the user in locating portions of interest in the media file, e.g., places in the media file where scene changes occur.
- Metadata can be automatically generated
- metadata can be input by a user or group of users. For example, a user could review a media file and identify sections of the media file where a particular speaker is featured, e.g., the speaker is either shown visually or is speaking. After identifying portions of the media file where a speaker is featured, the user can manually enter metadata that indicates the user's subjective evaluation of whether the speaker is present in the media file at each time-wise position in the media file. The user can enter the metadata continuously after viewing each video frame or listening to each predefined sub-portion of the media file. Alternately, the user could identify onoy those portions of the media file where the speaker is featured, and all other portions of the media file could be assigned metadata values that indicate that the speaker is not present in the media file.
- FIG. 1 shows an example graphical user interface 10 in accordance with the invention.
- the graphical user interface 10 is typically displayed on a video display, such as a computer monitor, but can be presented to a user on any other display device.
- the interface 10 has a video display portion 1 in which a still video frame representation of an image contained in a media file can be presented. Video frames can also be displayed at any desired video frame rate, such as that performed on a standard computer monitor or television set. Audio representations of sound information contained in a media file can be presented to a user by a speaker (not shown). Alternately, visual representations of sound information, such as a waveform, can be presented on the graphical user interface 10 .
- a playback control bar 2 contains playback controls that are used to control how a visual and/or aural representation of a media file is presented to a user.
- the playback control bar 2 can include buttons for controls such as play, stop, rewind, fast forward, and forward and reverse index.
- the playback controls shown in FIG. 1 are not intended to be an exhaustive representation of all of the different types of playback controls that can be provided.
- controls can be provided to alter the effective frame rate at which a video display is generated or the speed at which a sound representation is played back.
- a user activates controls on the playback control bar 2 by clicking a desired button with a mouse cursor.
- Other methods for activating the controls can be used, such as individual keystrokes or combinations of keystrokes on a keyboard, selection of controls using a touch screen or light pen, or other selection device.
- a timeline 3 includes a thumb 31 that indicates a current time-wise position in the media file.
- the timeline 3 shown in FIG. 1 ranges from time 0:00 to time 47:38 and can be optionally zoomed in or out and panned using the timeline controls 32 .
- the thumb 31 can be moved time-wise along the timeline 3 by using the left and right timeline controls 32 .
- a user could select the thumb 31 using a mouse and move the thumb 31 to a desired position along the timeline 3 using the mouse.
- By moving the thumb 31 time-wise along the timeline 3 a corresponding video and/or aural representation of the media file located at the time-wise position of the thumb 31 can be played back.
- video frames or preselected keyframes at corresponding time-wise locations in the media file can be displayed in the video display portion 1 as the thumb 31 is moved.
- video/aural playback could begin from a corresponding position in the media file after the thumb 31 is moved to a desired position and released.
- a user can select a feature in the media file using a media feature selection 4 .
- the feature selection 4 includes a scroll bar for navigating through a list of selectable features.
- features in the feature selection 4 are selected by double clicking a desired feature with a mouse cursor.
- a user could type in a selected feature using a keyboard or otherwise enter a selected feature, such as by speaking a selected feature, which is recognized by the browsing system using voice recognition techniques.
- the features listed in the feature selection 4 are organized in alphabetical order or otherwise grouped logically. For example, video-related features are preferably grouped apart from audio-related features.
- Selected features are displayed in selected feature windows 5 .
- selected feature windows 5 In the example graphical user interface 10 of FIG. 1, three selected feature windows 5 - 1 , 5 - 2 , and 5 - 3 are shown. However, fewer or more selected feature windows 5 could be used.
- Corresponding to each selected feature window 5 is a feature indication window 6 .
- the feature indication window 6 preferably graphically indicates the time-wise presence or absence of a corresponding feature in the media file, but the feature indication window 6 could display other information related to the corresponding feature.
- the feature indication window 6 - 3 graphically displays the time-wise presence of applause in the media file.
- solid vertical lines displayed in a feature indication window 6 indicate a determination that the corresponding feature is likely present in the media file at the corresponding time-wise location in the media file.
- other representations are possible, as discussed more fully below.
- a feature summary bar 7 preferably graphically displays a summary of the selected features.
- the feature summary bar 7 includes graphical icons 8 that each indicate, or represent, specific information.
- the scissors icon 8 indicates the presence of a cut or shot detect in the media file.
- the icons 8 can also indicate information related to two or more features. For example, a rectangular icon 8 containing a circle indicates a portion of the media file that contains an occurrence of applause followed shortly after a shot detect occurrence. This is only one example of the possible icons 8 that can be used to summarize selected features. Additional summary representations are discussed below.
- Key playback buttons 9 are also provided for each of the selected features and for the feature summary bar 7 .
- the key playback buttons 9 allow a user to select a feature that is used to control playback functions of the playback control bar 2 .
- the play and/or index buttons in the playback control bar 2 could be controlled such that whenever the button is selected, portions of the media file containing the speaker MD are “jumped” to and played back. Other portions of the media file are skipped and not presented to the user.
- the index buttons in the playback control bar 2 could be controlled such that whenever an index button is selected, a representation of the media file begins immediately before or after occurrences of applause in the media file.
- the interface 10 also includes a confidence score value manipulation control 11 that allows a user to adjust confidence score values for corresponding features as desired.
- a confidence score relates to a corresponding feature, is preferably determined using metadata for a corresponding feature, and preferably is used to generate the information displayed in the feature indication windows 6 .
- automatically derived metadata is preferably mapped to a corresponding confidence score.
- the confidence score is thus intended to convey to a user, in an understandable form, the probability of a feature's existence in the media file.
- the metadata is mapped to a corresponding time-varying confidence score using some function that is based on prior knowledge about the metadata's reliability.
- mapping of the metadata to a confidence score can be performed in a one-to-one relationship, where individual metadata data values are equal to corresponding confidence score values.
- FIG. 2 shows an example of shot detect metadata derived from a media file containing video information.
- the metadata values are represented graphically, and in this example, relatively larger metadata values represent relatively large differences between consecutive video frames at a corresponding time-wise position in the media file.
- Metadata representing shot boundary values are mapped to a confidence score using a threshold. If a metadata value is above the threshold, the corresponding confidence score is assigned a value of one, indicating a relatively high probability that a video cut is present in the media file. If the metadata value is below the threshold, the corresponding confidence score is assigned a value of zero.
- the threshold is adjusted automatically or by a user based on the reliability of the metadata.
- the confidence scores for the shot detection feature can be depicted in a feature indication window 6 of a graphical user interface 10 as shown in FIG. 2. In this example, dark portions of the feature indication window 6 represent a confidence score of zero and light portions of the feature indication window 6 represent confidence score values of one.
- the metadata values for a media feature can be mapped to a confidence score in many different ways.
- the metadata values are preferably mapped to corresponding confidence score values using a function that is determined based on the feature corresponding to the metadata and the reliability of metadata.
- Linear or non-linear mapping including normalization, thresholding and filtering, can be performed for numerical metadata.
- non-numerical metadata such as text, closed captions (subtitles), or MIDI streams
- metadata can be mapped to confidence score values using statistical or other mathematical methods.
- the time-wise frequency of a word or set of words in a media data stream could be converted to a set of confidence score values where a relatively high confidence score value is assigned to a portion of the media file that contains relatively more occurrences of the word or set of words, and lower confidence scores are assigned to portions containing fewer occurrences of the word or set of words.
- a lower dimensional confidence score can be computed.
- a face recognition algorithm generates five numerical values for each face detected in a video frame: the coordinates of the upper left and lower right corners of the bounding rectangle of the candidate face, and how well the candidate matches the face template.
- a confidence score for the face is preferably determined according to Equation 1:
- CA is the area of the candidate face determined from the upper left and lower right comer coordinates
- MFA is the maximum face area (a constant).
- CA is preferably not larger than or constrained to be not larger than MFA.
- mapping metadata values to confidence score values can be performed according to functions that are “learned” using techniques such as learning Bayesian networks. For example, a confidence score mapping system could “learn” that metadata determined for a particular feature is more or less reliable than originally anticipated, and adjust the mapping function, e.g., a mapping threshold, accordingly.
- a confidence score mapping system could “learn” that metadata determined for a particular feature is more or less reliable than originally anticipated, and adjust the mapping function, e.g., a mapping threshold, accordingly.
- Mapping of metadata values is also preferably controlled based on user control values.
- a user can adjust a threshold value or other control values using the confidence score value manipulation control 11 in the graphical user interface 10 .
- adjusting the sliders on the confidence score value manipulation control 11 can adjust the threshold value or values used to map the metadata to confidence scores.
- the user can adjust the sensitivity of the mapping function to the metadata and customize the feature information displayed in the feature indicator windows 6 .
- only one threshold is used and so only one slider need be displayed on the confidence score value manipulation control 11 .
- more than one threshold can be used to map metadata to confidence scores and so an additional slider can be displayed.
- the confidence score value manipulation control 11 can also be used to adjust mapping control values other than thresholds, such as filter coefficient values, or any other value used to change the way metadata is mapped to confidence score values.
- the confidence score information is preferably provided to a user as an aid in browsing a media file as discussed above.
- FIGS. 3 - 9 show additional examples of how confidence score information can be presented to a user. These examples are not intended to be exhaustive and other methods can be used to provide confidence score information.
- confidence score information can be conveyed to stimulate human senses other than sight, including sound, touch (e.g., by activating a Braille touchpad) or smell.
- FIGS. 3 - 6 involve a selected feature “speaker MD”.
- This feature represents a speaker depicted and/or recorded in a media file where MD represents the speaker's initials.
- metadata related to the speaker's identity can be generated using face recognition techniques, in the case where the speaker is identified “visually” or using speech recognition techniques, where the speaker is identified as having a particular speech pattern, for example.
- the speaker's identity can also be verified manually by a user. For example, a user can review a media file and enter metadata directly into the system indicating whether a particular speaker is featured in the media file. The user can also enter metadata for other feature types, thereby eliminating a need to have the metadata determined automatically.
- FIG. 3 shows an example feature indication window 6 containing icons 8 .
- the icons 8 represent likely portions in a media file where the speaker MD either talks or is represented visually. In this example, portions where the speaker MD either talks or is represented visually are identified as those portions for which the confidence score exceeds a given threshold.
- the icons 8 optionally contain a representation of speaker MD's face or is a thumbnail keyframe extracted from the media file at a corresponding point in the media file.
- a representation of the media file at a point corresponding in time is presented to the user.
- portions of the media file that most likely contain the speaker MD are indicated by shaded portions in the feature indication window 6 .
- the shaded portions can include color.
- speaker MD could be assigned the color green, and all green portions displayed in the feature indication window 6 indicate portions of the media file most likely containing the speaker MD.
- the display is generated based on the corresponding confidence score, which may not always accurately indicate if the corresponding feature is present in the media file.
- the metadata and corresponding confidence score represents a probability or “best guess” regarding whether the feature is present in the media file. This may be true even when a user enters the metadata into the system. For example, a user may not be certain that a particular person is speaking in a media file even though the user is very familiar with the person's voice.
- the confidence score corresponding to speaker MD is displayed in the feature indication window 6 .
- the metadata corresponding to speaker MD can be directly mapped on a one-to-one basis to the confidence score. That is, the metadata values are the same as the corresponding confidence score values.
- the confidence score values may not always indicate that the speaker MD is either positively included or not included in the media file. Rather, the confidence score values can vary indicating an uncertainty regarding whether the corresponding feature is present in the media file. This uncertainty may be due to noise in the media data stream, two or more speakers speaking simultaneously, or that a portion of the speaker's face is partially obscured.
- the confidence score values shown in FIG. 5 have been thresholded to generate the display in the feature indication window 6 .
- any confidence score values that fall below the maximum value are clipped to the lowest possible value.
- a user can vary the threshold values used to process the confidence score values to generate the feature indication using the confidence score value manipulation control 11 , or otherwise vary how the feature indication is presented based on the confidence score values.
- FIGS. 7 - 9 relate to the display of confidence score information for a video cut feature in a media segment.
- the confidence score values are shown without manipulation in the feature indication window 6 .
- increasing confidence score values indicate higher probabilities that a video cut has been detected.
- the video cut feature indication shown in the feature indication window 6 is generated based on integrating the confidence score values over time. This type of indication can be useful for determining the total number of video cuts in a media segment or in determining how much the video images change within the media segment.
- the change in the integrated value is proportional to the fraction of total accumulated feature change in the media file.
- FIG. 8 can vary in color as the curve represents an increase in value.
- the curve could start as a blue color for relatively low values and gradually change to a red color for maximum integration values. That is, the curve could start as a blue color and as the curve increases in value, the curve gradually changes through various shades of purple of increasing red content.
- FIG. 9 shows an alternate representation of the integrated confidence score value where the integrated value is represented only by a corresponding color change in the feature indication window 6 without display of the integration curve itself.
- the change in color represents the total accumulated feature change in the media file.
- the graphical user interface 10 preferably includes an option for combining features and displaying confidence score information based on the feature combination, as shown in FIG. 10.
- the features “speaker MD” and “applause” are combined, and the corresponding feature indication window 6 indicates media portions where the speaker MD is depicted during applause. This type of combination can be useful to locate portions in a media file where a particular speaker begins and/or ends a presentation or has made a particularly interesting comment during a presentation.
- FIG. 10 shows another combination of the features “silence” and “shot detect”.
- the corresponding feature indication window 6 indicates portions in the media file where a shot detect and silence are simultaneously present. This combination can be useful to indicate the beginning or end of a desired portion in the media file.
- the graphical user interface 10 includes a “Combine” button that a user can click to combine two or more features.
- a user can define which features to combine by other methods.
- confidence score information can be conditioned on other features.
- confidence score information related to the feature “speaker MD” is displayed only when confidence score information indicates a relatively high probability that the speaker MD is shown in a video segment of the media file along with or within two minutes of applause in the soundtrack of the media file.
- other feature combinations or conditions e.g., Boolean operators, can be used depending on the type of feature indication information needed for a specific task and the relevant features.
- FIG. 12 is a flowchart of a preferred method for browsing a media file.
- a user inputs a command indicating a desire to browse a media file.
- the desired media file is retrieved.
- the graphical user interface 10 is preferably displayed to the user, although other interfaces can be used.
- the user preferably selects at least one feature of the media file.
- confidence scores corresponding to the selected features are retrieved and/or generated.
- metadata values for all of the selectable features are precomputed and stored with the media file so that metadata values need not be generated during a browsing session. However, the metadata values and corresponding confidence score values can be generated during a browsing session.
- step S 60 feature indications related to the selected features are displayed to the user.
- step S 70 the user enters commands to play back or otherwise manipulate the media file as desired. For example, a user can click on an icon 8 in a feature indication window 7 and media playback begins from a corresponding position in the media file.
- FIG. 13 is a schematic block diagram of a preferred embodiment of a media browser 100 in accordance with the invention.
- the media browser 100 preferably includes a general purpose computer 110 that has a controller 111 .
- the controller 111 can be implemented as a single special purpose integrated circuit (e.g., ASIC) each having a main or central processor section for overall, system-level control, and separate sections dedicated to performing various different specific computations, functions and other processes under the control of the central processor section.
- the controller 111 can also be implemented as a single microprocessor circuit or a plurality of separate dedicated or programmable integrated or other electronic circuits or devices, e.g., hardwired electronic or logic circuits such as discrete element circuits or programmable logic devices.
- the controller 111 and/or the general purpose computer 110 also preferably include other circuitry or components, such as memory devices, relays, mechanical linkages, communications devices, etc., to effect desired control and/or input/output functions.
- the general purpose computer 110 also includes a feature selector 112 , a position indicator 113 , a feature indicator 114 and a mapping module 115 , which are all preferably implemented as software modules that include sets of instructions performed by the general purpose computer 110 and/or the controller 111 .
- these elements like the controller 111 , can be implemented as an ASIC or an array of ASICs, separate dedicated or programmable integrated or other electronic circuits or other devices.
- a media input device 120 preferably inputs a media file in the form of a data stream to the general purpose computer 110 .
- the media input device 120 can include one or more of any of the following devices: a semiconductor memory device, CD-ROM device, magnetic disk drive, other volatile or non-volatile memory devices, communications devices, such as a modem, facsimile transmitter or receiver, local-area or wide-area network devices, telecommunications devices, or other wired or wireless communications devices.
- the media input device 120 can be any device that transmits a data stream representing a media file to the general purpose computer 110 .
- the data stream transmitted from the media input device 120 to the general purpose computer 110 can take any of many different forms.
- the data stream can be analog or digital data that is either compressed or uncompressed, and the data can be provided in parallel or serial form at any desired transmission rate using any desired transmission protocol, if any.
- the controller 111 preferably communicates with a display 130 , such as a CRT or LCD display.
- a display 130 is not limited to these types of displays.
- the display 130 can include any device capable of presenting a representation of a media file to a user, including an audio speaker, visual display device, touch display (such as a Braille device), etc.
- the media browser 100 also preferably includes a user interface 140 , such as a keyboard and/or computer mouse, that permits a user to communicate with the controller 111 .
- a user interface 140 such as a keyboard and/or computer mouse
- Other devices can be used with the user interface 140 , including touch screen devices, light pens, voice activated and/or voice recognition systems, etc.
- the user interface 140 may include a general purpose computer and appropriate communications devices to send and receive signals to a more remotely located general purpose computer 110 , i.e., a remote server.
- the general purpose computer 110 can send display control signals to the user interface 140 , which directly controls the display 130 .
- the user interface 140 can perform some or all of the functions performed by the general purpose computer 110 , and some or all of the modules, such as the position indicator 113 , mapping module 115 , etc., can operate, at least in part, within the user interface 140 .
- the media input 120 can be physically located near the general purpose computer 110 , the user interface 140 (e.g., a CD-ROM drive), or at some other location. Alternately, the media input 120 can be formed of a multitude of media sources, such as magnetic hard drives distributed over a wide network. All that is required is that the media stored in the media input 120 be accessable to the media browser 100 .
- the media browser 100 receives a command from a user via the user interface 140 indicating a desire to browse a media file.
- the desired media file is then either transmitted from the media input device 120 to the general purpose computer 110 or the media file is retrieved from a memory device provided as part of the controller 111 , for example.
- the controller 111 preferably sends a signal to the display 130 to display the graphical user interface 10 to the user.
- a command to indicate a current position of the media file is the beginning of the media file. That is, the thumb 31 on the timeline 3 is displayed at the far left position. Of course, the thumb 31 need not be displayed, or can be displayed at any desired position along the timeline 3 .
- a video frame if present, is displayed in the video display portion 1 corresponding to a current position in the media file.
- the signal provided by the controller 111 to the display 130 regarding the position of the thumb 31 is based on a signal provided by the position indicator 113 , which keeps track of a current position in the media file.
- the user After viewing the graphical user interface 10 , the user preferably selects at least one feature of the media file using the media feature selection 4 and the user interface 140 .
- Feature selection indications are provided to the feature selector 112 , which stores the selected features and provides a signal to the controller 111 to display the selected features in a selected feature window 5 of the interface 10 .
- the feature selector 112 also provides an indication to the controller 111 if a selected feature is not selectable. For example, if a user selects a feature that is not present in a media file, e.g., a user selects the shot detect feature for a media file that does not contain video information, an indication that the selected feature is not selectable is provided to the user.
- the feature selector 112 provides a list of selectable features to the controller 111 so that only selectable features are displayed in the media feature selection 4 .
- the mapping module 115 generates and/or retrieves confidence scores corresponding to the selected features.
- metadata values for all of the selectable features are precomputed and stored with the media file so that metadata values need not be generated during a browsing session.
- the mapping module 115 can automatically generate metadata values and map the metadata values to corresponding confidence score values “on the fly.”
- the feature indicator 114 generates and provides information to the controller 111 to display feature indications related to the selected features.
- a user can select a portion of the media file for review, or perform other actions, such as zooming the timeline 3 , moving the timeline thumb 31 , etc.
- a user can input signals to the media browser 100 to playback or otherwise manipulate the media file. For example, the user can click on an icon 8 in a feature indication window 7 and the controller 111 controls the display 130 to provide a representation of the media file starting from a time-wise position in the media file consistent with the selected icon 8 .
- FIG. 14 is a flowchart that describes in further detail how a user chooses a media file for browsing and provides a corresponding signal to the media browser 100 .
- the controller 111 sends a signal to the display 130 to display a media selection screen.
- the displayed selection screen includes at least the display shown in FIG. 15.
- the selection screen preferably includes several rows 200 , each row 200 displaying information for a corresponding media file.
- Each row 200 preferably includes such information as a keyframe 201 , a timeline 202 , a media file name 203 , identifying information, such as a date, 204 , and a keyword listing 205 .
- the keyframe 201 is preferably selected from a particularly relevant portion of the media file or otherwise visually conveys information related to the overall content of the media file.
- the timeline 202 is similar to the timeline 3 in the graphical user interface 10 .
- the timeline 202 preferably displays information related to a selected media feature, e.g., shot boundaries.
- a user can view keyframes or other video information when a cursor 206 is moved over the timeline 202 .
- keyframes are represented on the timeline 202 by a small icon that preferably changes appearance when the icon is nearest the cursor 206 .
- Keyframes can be automatically selected based on confidence score information or can be previously identified by an operator.
- the media browser 100 could display a predefined summary video sequence or another shortened representation that provides a user additional information to make a selection decision.
- the timeline 202 can also provide other information related to media features in any of the ways described above. For example, icons, color sequences or keyframes could be displayed on the timeline 202 to provide media feature information.
- the media feature information can be displayed based on metadata and confidence score information or based on predefined summary information provided by a user.
- a user can search a database of media files by the name 203 , identifying information 204 and/or keywords 205 to narrow an initial search and generate an initial display of media file rows 200 on the selection screen.
- the user can search using any known search screen, such as those associated with electronic database applications or Internet browser/search engines.
- the media browser 100 can be very useful in browsing media files posted on a large network, such as the Internet, since the media browser 100 allows a user to rapidly and accurately identify and experience desired portions in media file.
- the media displayed on the initial selection screen are preferably sorted, placing those media files most relevant to the search at the top. Sorting can be done based on confidence score information, search results (e.g., number of hits in a media file), or other desired criteria.
- step S 120 the user reviews the media files, either by viewing keyframes, keywords, selected portions of the media files using the timeline 202 features, or by reviewing the media files in any of the ways discussed above.
- step S 130 the user selects one or more of the media files for browsing. Preferably, the user selects a media file by clicking on an associated keyframe 201 , but the user can select media files in other ways, such as by entering a media file name or other identifier by keyboard.
- FIG. 16 is a flowchart that describes in further detail how the mapping module 115 generates and/or retrieves confidence scores.
- step S 510 the mapping module 115 generates and/or retrieves metadata information for the selected features.
- the metadata information is preferably precomputed and stored along with the media file.
- the mapping module 115 can apply any of the above described automatic techniques to the media file to generate metadata information.
- the mapping module 115 can receive manually input metadata information from a user.
- the mapping module 115 optionally processes the metadata information as desired in preparation for mapping the metadata to confidence scores. For example, the mapping module 115 thresholds, combines, filters or otherwise processes the metadata information in preparation for mapping the metadata information to confidence scores.
- step S 530 the mapping module maps the processed metadata information to confidence score values for each selected feature using a desired function, e.g., a one-to-one mapping, other linear or non-linear functions, etc.
- a desired function e.g., a one-to-one mapping, other linear or non-linear functions, etc.
- FIG. 17 is a flowchart describing in further detail how the feature indicator 114 provides feature information for the selected features.
- the feature indicator 114 processes the confidence scores for each selected feature as desired. For example, the feature indicator 114 thresholds, filters, combines, etc., the confidence score values as discussed above in preparation for providing feature information.
- step S 620 the feature indicator 114 generates information to display feature indications for each of the selected features based on the processed confidence scores. For example, the feature indicator 114 generates signals to display feature indications in a feature indication window 6 .
- FIG. 18 is a flowchart describing in further detail how the controller 111 controls media playback as desired.
- a user optionally selects features using the key playback buttons 9 on the graphical user interface 10 .
- the user could select the “speaker MD” feature.
- the controller 111 inputs a playback control signal received from a user and adjusts the media playback based on the features selected in step S 710 .
- the user could select the play button, and the media browser 100 adjusts the replay of the media file based on the “speaker MD” feature.
- the media browser 100 could adjust the playback speed according to Equation 2:
- R is the playback rate
- C is the confidence score for the speaker MD feature
- M is a maximum playback rate, which is preferably set by a user.
- the confidence score C varies between 0 and 1. Therefore, as the confidence score increases (indicating that the speaker MD is speaking or is depicted in the media file), the playback rate approaches the value C. In contrast, as the confidence score decreases (indicating that the speaker MD is not featured in the media file), the playback rate approaches the maximum playback rate M, thereby speeding replay in sections where the speaker MD is not present in the media file.
- the media browser 100 can control the media playback controls based on a selected feature.
- the media browser 100 can also control the stop button to stop playback at a point in the media file where a feature is no longer present (e.g., a speaker leaves the room) or where a feature next occurs (e.g., stop at the next shot boundary).
- the index controls can be controlled to jump to a point in the media file where a feature is present in or absent from the media file.
- the timeline thumb 31 can be controlled based on a selected feature.
- the thumb 31 could change its appearance, e.g., color, when the thumb 31 is moved over a media file portion where a selected feature is present, or the thumb 31 could be “indexed” so that a user can only move the thumb 31 within portions of the media file where a selected feature is present or absent.
- control of the playback controls, including the thumb 31 is based on the confidence score value for a selected feature. Determination of whether a feature is present or not in a media file is preferably based on the confidence score value, e.g., if the confidence score exceeds/is below a desired threshold the media browser 100 determines that the feature is present in/absent from the media file.
- FIG. 19 is a flowchart describing in further detail another way how the controller 111 optionally controls media playback.
- a user selects a confidence score indication in a feature indication window 6 .
- a user can select (e.g., double click) an icon 8 displayed in a feature indication window 6 .
- the controller 111 accesses the media file and controls the display 130 to begin playback at a point in the media file corresponding to the selected confidence score indication. Playback can be of selected portions of the media file of particular interest determined based on confidence score values.
Abstract
Description
- 1. Field of Invention
- This invention relates to a method and apparatus for reviewing an aural and/or visual and/or other representation of a media file. Specifically, the invention relates to using media content features to allow a user to more easily review a media file.
- 2. Description of Related Art
- Text documents often have many cues, such as headings, paragraphs, punctuation, etc., that allow a reader to quickly determine the beginning and end of different sections of the document and to aid the reader in finding areas of interest. However, video and audio browsing systems typically do not provide information to the user regarding simple features, like section beginning and end points, much less more complicated information, like the name of the speaker on a video clip. Such browsing systems typically offer only standard “VCR-type” playback control options, like play, stop, rewind, and fast forward. As anyone who has tried to find a specific video clip on a conventional video tape using a standard VCR will understand, it is often difficult to locate portions of interest in a video using the standard playback controls.
- Many techniques exist for extracting information that represents the feature content of a media file. In this application, the term media or media file is used to represent any data stream that contains information regarding video or other image information, audio information, text information and/or other information. A feature of a media file is a property of the video, audio and/or text information in the media file, such as video or audio format, or information relating to the content of the media file, such as the identity of a speaker depicted in a video sequence, occurrences of applause, video shot boundaries, or motion depicted in a video sequence. For example, Pfeiffer et al., “Automatic Audio Content Analysis” ACM MULTIMEDIA 96, Boston, Mass., 1996, pp. 21-30; Wilcox, et al., “Segmentation of Speech Using Speaker Identification”, Proc. ICASSP 94, vol. S1, April 1994, pp. 161-164; and Foote, “Rapid Speaker ID Using Discrete MMI Feature Quantisation,” Expert Systems with Applications, vol. 13, no. 4, 1997, pp. 293-289, describe various methods for identifying audio features, such as music, human speech, and speaker identity. Regarding video data, Boreczky et al., “Comparison of Video Shot Boundary Detection Techniques” Proc. SPIE Conf. On Storage and Retrieval for Still Image and Video Databases IV, San Jose, Calif, vol. 2670, February 1996, pages 170-179, and Zhang et al., “Automatic Partitioning of Full-motion Video”, Multimedia Systems, vol. 1, no. 1, 1993, pp. 10-28, disclose methods for identifying shot boundaries (radical changes in video content) and motion. As described in these and other similar references, features in a media file can be identified automatically using any of a number of different techniques.
- Providing feature information in a media browsing system can be very useful for a user when identifying areas of interest in a media file, controlling media playback, editing a media file, or performing other operations with a media file. For example, graphically identifying areas in a media file where a particular speaker is shown on a video clip can allow a user to quickly determine and playback those portions that contain the speaker.
- Providing feature information to the user based on automatically identified features also eliminates the need for a user to manually index or otherwise mark significant portions of the media file for later retrieval. Thus, the invention can use existing methods for automatically identifying features in a media file to generate and provide feature information to a user to aid the user in browsing the media file.
- The invention provides a media browser that uses media feature information as an aid in navigating, selecting, editing, and/or annotating a media file.
- In one aspect of the invention, media features are selected by a user.
- In one aspect of the invention, media browsing functions, such as play, rewind, stop, fast-forward, index, automatic slide show, and automatic preview, are controlled based on feature information.
- In one aspect of the invention, feature information for a selected feature is mapped to a corresponding confidence score.
- In one aspect of the invention, the media browser includes a feature indicator that provides information related to a corresponding selected feature based on a corresponding confidence score.
- In one aspect of the invention, a feature indicator combines at least two confidence scores and provides information based on the combination.
- In one aspect of the invention, a feature indicator provides information related to a confidence score based on a value of another confidence score.
- The invention also provides a method for browsing a media file. A feature of the media file being browsed is selected and information related to a confidence score for at least one selected feature is provided. The confidence score relates to the existence of a corresponding selected feature in the media file. Based on the information related to the confidence score, a portion of the media file is selected.
- In one aspect of the invention, a metadata value representing a time-wise evaluation of a feature in the media file is mapped to a corresponding confidence score.
- In one aspect of the invention, mapping of a metadata value to a corresponding confidence score is non-linear.
- In one aspect of the invention, mapping of a metadata value to a confidence score is dependent on a user-defined control value or values.
- The invention will be described in relation to the following drawings in which reference numerals refer to like elements, and wherein:
- FIG. 1 is an example screen display for a graphical user interface in accordance with the invention;
- FIG. 2 shows example metadata values, corresponding confidence score information, and an example graphical depiction of corresponding feature information;
- FIGS.3-9 show different ways of graphically depicting information related to a feature in a media file;
- FIG. 10 is an example screen display depicting how features can be combined and corresponding combined feature information can be shown;
- FIG. 11 is an example screen display depicting how display of feature information relating to a corresponding feature can be conditioned on another feature;
- FIG. 12 is a flow chart explaining basic steps of a preferred method for browsing a media file;
- FIG. 13 is a schematic block diagram of a preferred media browser in accordance with invention;
- FIG. 14 is a flow chart of steps of a preferred method for selecting a media file to be browsed;
- FIG. 15 is a preferred media selection screen;
- FIG. 16 is a flow chart of steps of a preferred method for mapping feature metadata to a corresponding confidence score;
- FIG. 17 is a flow chart of steps of a preferred method for processing confidence scores and providing information related to the confidence scores;
- FIG. 18 is a flow chart of steps of a preferred method for controlling playback functions based on a selected feature; and
- FIG. 19 is a flow chart of steps of a preferred method for reviewing a media file based on selection of a confidence score indication.
- The invention is described in connection with browsing a media file. As used herein, browsing refers to any operation related to reviewing, manipulating, editing, selecting portions of, storing, transmitting, or other operations related to a media file. In addition, a preferred embodiment of a graphical user interface for a media browser is described below having specific functions. In many cases, various functions of the media browser can be eliminated or altered, as desired. For example, information that is displayed graphically in the embodiment described below can be provided aurally or otherwise, as described more fully below.
- As discussed above, several methods exist for automatically deriving information that represents the likelihood of the existence of a feature in a media data stream. Such information, called metadata, can be very useful and provides cues for locating portions of interest in a media file. For example, automatic methods exist for identifying silent portions in audio data. Identifying such portions can be useful in editing the audio data because, if silent portions are identified, an edit region of the audio data can be bounded by the silent portions, thereby insuring that speech in the edit region is not cut mid-word or mid-sentence.
- As another example, a common video analysis technique is the detection of shot boundaries, which represent significant differences between consecutive video frames. As is well known, the difference between consecutive video frames can be determined based on statistical properties of the pixels in the frame images, histogram differences, compression algorithms, edge differences, or motion detection. Application of one of several known techniques for detecting shot boundaries results in a series of time-varying metadata values that each represent a difference between consecutive video frames. Once metadata regarding the shot boundary feature is determined, information can be provided to a user to aid the user in locating portions of interest in the media file, e.g., places in the media file where scene changes occur.
- Although metadata can be automatically generated, metadata can be input by a user or group of users. For example, a user could review a media file and identify sections of the media file where a particular speaker is featured, e.g., the speaker is either shown visually or is speaking. After identifying portions of the media file where a speaker is featured, the user can manually enter metadata that indicates the user's subjective evaluation of whether the speaker is present in the media file at each time-wise position in the media file. The user can enter the metadata continuously after viewing each video frame or listening to each predefined sub-portion of the media file. Alternately, the user could identify onoy those portions of the media file where the speaker is featured, and all other portions of the media file could be assigned metadata values that indicate that the speaker is not present in the media file.
- FIG. 1 shows an example
graphical user interface 10 in accordance with the invention. Thegraphical user interface 10 is typically displayed on a video display, such as a computer monitor, but can be presented to a user on any other display device. Theinterface 10 has avideo display portion 1 in which a still video frame representation of an image contained in a media file can be presented. Video frames can also be displayed at any desired video frame rate, such as that performed on a standard computer monitor or television set. Audio representations of sound information contained in a media file can be presented to a user by a speaker (not shown). Alternately, visual representations of sound information, such as a waveform, can be presented on thegraphical user interface 10. - A
playback control bar 2 contains playback controls that are used to control how a visual and/or aural representation of a media file is presented to a user. For example, theplayback control bar 2 can include buttons for controls such as play, stop, rewind, fast forward, and forward and reverse index. The playback controls shown in FIG. 1 are not intended to be an exhaustive representation of all of the different types of playback controls that can be provided. For example, controls can be provided to alter the effective frame rate at which a video display is generated or the speed at which a sound representation is played back. As with other features of theinterface 10, a user activates controls on theplayback control bar 2 by clicking a desired button with a mouse cursor. Other methods for activating the controls can be used, such as individual keystrokes or combinations of keystrokes on a keyboard, selection of controls using a touch screen or light pen, or other selection device. - A
timeline 3 includes athumb 31 that indicates a current time-wise position in the media file. Thetimeline 3 shown in FIG. 1 ranges from time 0:00 to time 47:38 and can be optionally zoomed in or out and panned using the timeline controls 32. Thethumb 31 can be moved time-wise along thetimeline 3 by using the left and right timeline controls 32. Alternately, a user could select thethumb 31 using a mouse and move thethumb 31 to a desired position along thetimeline 3 using the mouse. By moving thethumb 31 time-wise along thetimeline 3, a corresponding video and/or aural representation of the media file located at the time-wise position of thethumb 31 can be played back. For example, video frames or preselected keyframes at corresponding time-wise locations in the media file can be displayed in thevideo display portion 1 as thethumb 31 is moved. Alternately, video/aural playback could begin from a corresponding position in the media file after thethumb 31 is moved to a desired position and released. - A user can select a feature in the media file using a
media feature selection 4. Thefeature selection 4 includes a scroll bar for navigating through a list of selectable features. Preferably, features in thefeature selection 4 are selected by double clicking a desired feature with a mouse cursor. Alternately, a user could type in a selected feature using a keyboard or otherwise enter a selected feature, such as by speaking a selected feature, which is recognized by the browsing system using voice recognition techniques. Preferably, the features listed in thefeature selection 4 are organized in alphabetical order or otherwise grouped logically. For example, video-related features are preferably grouped apart from audio-related features. - Selected features are displayed in selected
feature windows 5. In the examplegraphical user interface 10 of FIG. 1, three selected feature windows 5-1, 5-2, and 5-3 are shown. However, fewer or moreselected feature windows 5 could be used. Corresponding to each selectedfeature window 5 is afeature indication window 6. Thefeature indication window 6 preferably graphically indicates the time-wise presence or absence of a corresponding feature in the media file, but thefeature indication window 6 could display other information related to the corresponding feature. For example, the feature indication window 6-3 graphically displays the time-wise presence of applause in the media file. In this example, solid vertical lines displayed in afeature indication window 6 indicate a determination that the corresponding feature is likely present in the media file at the corresponding time-wise location in the media file. However, other representations are possible, as discussed more fully below. - A feature summary bar7 preferably graphically displays a summary of the selected features. In this example, the feature summary bar 7 includes
graphical icons 8 that each indicate, or represent, specific information. For example, thescissors icon 8 indicates the presence of a cut or shot detect in the media file. Theicons 8 can also indicate information related to two or more features. For example, arectangular icon 8 containing a circle indicates a portion of the media file that contains an occurrence of applause followed shortly after a shot detect occurrence. This is only one example of thepossible icons 8 that can be used to summarize selected features. Additional summary representations are discussed below. -
Key playback buttons 9 are also provided for each of the selected features and for the feature summary bar 7. Thekey playback buttons 9 allow a user to select a feature that is used to control playback functions of theplayback control bar 2. For example, if thekey playback button 9 for the selected feature “speaker MD” is selected, the play and/or index buttons in theplayback control bar 2 could be controlled such that whenever the button is selected, portions of the media file containing the speaker MD are “jumped” to and played back. Other portions of the media file are skipped and not presented to the user. As another example, if thekey playback button 9 for the selected feature “applause” is selected, the index buttons in theplayback control bar 2 could be controlled such that whenever an index button is selected, a representation of the media file begins immediately before or after occurrences of applause in the media file. - The
interface 10 also includes a confidence scorevalue manipulation control 11 that allows a user to adjust confidence score values for corresponding features as desired. As explained more fully below, a confidence score relates to a corresponding feature, is preferably determined using metadata for a corresponding feature, and preferably is used to generate the information displayed in thefeature indication windows 6. - Because automatic techniques for generating metadata for media features do not always work reliably, the metadata itself is not always easy to interpret, and the metadata rarely positively indicate the absolute presence or absence of a feature in a media file, automatically derived metadata is preferably mapped to a corresponding confidence score. The confidence score is thus intended to convey to a user, in an understandable form, the probability of a feature's existence in the media file. Preferably, the metadata is mapped to a corresponding time-varying confidence score using some function that is based on prior knowledge about the metadata's reliability. However, mapping of the metadata to a confidence score can be performed in a one-to-one relationship, where individual metadata data values are equal to corresponding confidence score values.
- FIG. 2 shows an example of shot detect metadata derived from a media file containing video information. The metadata values are represented graphically, and in this example, relatively larger metadata values represent relatively large differences between consecutive video frames at a corresponding time-wise position in the media file.
- In the example shown in FIG. 2, metadata representing shot boundary values are mapped to a confidence score using a threshold. If a metadata value is above the threshold, the corresponding confidence score is assigned a value of one, indicating a relatively high probability that a video cut is present in the media file. If the metadata value is below the threshold, the corresponding confidence score is assigned a value of zero. Preferably, the threshold is adjusted automatically or by a user based on the reliability of the metadata. The confidence scores for the shot detection feature can be depicted in a
feature indication window 6 of agraphical user interface 10 as shown in FIG. 2. In this example, dark portions of thefeature indication window 6 represent a confidence score of zero and light portions of thefeature indication window 6 represent confidence score values of one. - As will be understood by those of ordinary skill in the art, the metadata values for a media feature can be mapped to a confidence score in many different ways. As discussed above, the metadata values are preferably mapped to corresponding confidence score values using a function that is determined based on the feature corresponding to the metadata and the reliability of metadata. Linear or non-linear mapping, including normalization, thresholding and filtering, can be performed for numerical metadata. For non-numerical metadata, such as text, closed captions (subtitles), or MIDI streams, metadata can be mapped to confidence score values using statistical or other mathematical methods. For example, the time-wise frequency of a word or set of words in a media data stream could be converted to a set of confidence score values where a relatively high confidence score value is assigned to a portion of the media file that contains relatively more occurrences of the word or set of words, and lower confidence scores are assigned to portions containing fewer occurrences of the word or set of words.
- For multi-dimensional data, such as metadata generated by a face recognition algorithm, a lower dimensional confidence score can be computed. In a preferred embodiment, a face recognition algorithm generates five numerical values for each face detected in a video frame: the coordinates of the upper left and lower right corners of the bounding rectangle of the candidate face, and how well the candidate matches the face template. A confidence score for the face is preferably determined according to Equation 1:
- C=MS(CA/MFA) (1)
- where C is the confidence score, MS is the value representing how well the candidate matches the face template, CA is the area of the candidate face determined from the upper left and lower right comer coordinates, and MFA is the maximum face area (a constant). In this example, CA is preferably not larger than or constrained to be not larger than MFA.
- Mapping metadata values to confidence score values can be performed according to functions that are “learned” using techniques such as learning Bayesian networks. For example, a confidence score mapping system could “learn” that metadata determined for a particular feature is more or less reliable than originally anticipated, and adjust the mapping function, e.g., a mapping threshold, accordingly.
- Mapping of metadata values is also preferably controlled based on user control values. For example, a user can adjust a threshold value or other control values using the confidence score
value manipulation control 11 in thegraphical user interface 10. Using the example in FIG. 2, adjusting the sliders on the confidence scorevalue manipulation control 11 can adjust the threshold value or values used to map the metadata to confidence scores. By adjusting the threshold value(s), the user can adjust the sensitivity of the mapping function to the metadata and customize the feature information displayed in thefeature indicator windows 6. In the FIG. 2 example, only one threshold is used and so only one slider need be displayed on the confidence scorevalue manipulation control 11. However, more than one threshold can be used to map metadata to confidence scores and so an additional slider can be displayed. Further, other graphical controls, such as dials, up/down buttons, numerical indicators, etc., in addition to or in place of the sliders can be used. The confidence scorevalue manipulation control 11 can also be used to adjust mapping control values other than thresholds, such as filter coefficient values, or any other value used to change the way metadata is mapped to confidence score values. - After confidence score values are determined for a feature, the confidence score information is preferably provided to a user as an aid in browsing a media file as discussed above. FIGS.3-9 show additional examples of how confidence score information can be presented to a user. These examples are not intended to be exhaustive and other methods can be used to provide confidence score information. For example, confidence score information can be conveyed to stimulate human senses other than sight, including sound, touch (e.g., by activating a Braille touchpad) or smell.
- The examples shown in FIGS.3-6 involve a selected feature “speaker MD”. This feature represents a speaker depicted and/or recorded in a media file where MD represents the speaker's initials. In this example, metadata related to the speaker's identity can be generated using face recognition techniques, in the case where the speaker is identified “visually” or using speech recognition techniques, where the speaker is identified as having a particular speech pattern, for example. The speaker's identity can also be verified manually by a user. For example, a user can review a media file and enter metadata directly into the system indicating whether a particular speaker is featured in the media file. The user can also enter metadata for other feature types, thereby eliminating a need to have the metadata determined automatically.
- FIG. 3 shows an example
feature indication window 6 containingicons 8. Theicons 8 represent likely portions in a media file where the speaker MD either talks or is represented visually. In this example, portions where the speaker MD either talks or is represented visually are identified as those portions for which the confidence score exceeds a given threshold. Theicons 8 optionally contain a representation of speaker MD's face or is a thumbnail keyframe extracted from the media file at a corresponding point in the media file. Preferably, when a user clicks on anicon 8, a representation of the media file at a point corresponding in time is presented to the user. - In FIG. 4, portions of the media file that most likely contain the speaker MD are indicated by shaded portions in the
feature indication window 6. Optionally, the shaded portions can include color. For example, speaker MD could be assigned the color green, and all green portions displayed in thefeature indication window 6 indicate portions of the media file most likely containing the speaker MD. In this example, as in all cases, the display is generated based on the corresponding confidence score, which may not always accurately indicate if the corresponding feature is present in the media file. Instead, as discussed above, the metadata and corresponding confidence score represents a probability or “best guess” regarding whether the feature is present in the media file. This may be true even when a user enters the metadata into the system. For example, a user may not be certain that a particular person is speaking in a media file even though the user is very familiar with the person's voice. - In FIG. 5, the confidence score corresponding to speaker MD, or a normalized version of the confidence score, is displayed in the
feature indication window 6. In this type of display, the metadata corresponding to speaker MD can be directly mapped on a one-to-one basis to the confidence score. That is, the metadata values are the same as the corresponding confidence score values. As can be seen in FIG. 5, the confidence score values may not always indicate that the speaker MD is either positively included or not included in the media file. Rather, the confidence score values can vary indicating an uncertainty regarding whether the corresponding feature is present in the media file. This uncertainty may be due to noise in the media data stream, two or more speakers speaking simultaneously, or that a portion of the speaker's face is partially obscured. - In FIG. 6, the confidence score values shown in FIG. 5 have been thresholded to generate the display in the
feature indication window 6. In this example, any confidence score values that fall below the maximum value are clipped to the lowest possible value. As discussed above, a user can vary the threshold values used to process the confidence score values to generate the feature indication using the confidence scorevalue manipulation control 11, or otherwise vary how the feature indication is presented based on the confidence score values. - FIGS.7-9 relate to the display of confidence score information for a video cut feature in a media segment. In FIG. 7, the confidence score values are shown without manipulation in the
feature indication window 6. In this case, increasing confidence score values indicate higher probabilities that a video cut has been detected. In FIG. 8, the video cut feature indication shown in thefeature indication window 6 is generated based on integrating the confidence score values over time. This type of indication can be useful for determining the total number of video cuts in a media segment or in determining how much the video images change within the media segment. Thus, the change in the integrated value is proportional to the fraction of total accumulated feature change in the media file. Optionally, the curve indicating the integrated value of the confidence score values shown in FIG. 8 can vary in color as the curve represents an increase in value. For example, the curve could start as a blue color for relatively low values and gradually change to a red color for maximum integration values. That is, the curve could start as a blue color and as the curve increases in value, the curve gradually changes through various shades of purple of increasing red content. FIG. 9 shows an alternate representation of the integrated confidence score value where the integrated value is represented only by a corresponding color change in thefeature indication window 6 without display of the integration curve itself. In this example, the change in color represents the total accumulated feature change in the media file. - The
graphical user interface 10 preferably includes an option for combining features and displaying confidence score information based on the feature combination, as shown in FIG. 10. In this example, the features “speaker MD” and “applause” are combined, and the correspondingfeature indication window 6 indicates media portions where the speaker MD is depicted during applause. This type of combination can be useful to locate portions in a media file where a particular speaker begins and/or ends a presentation or has made a particularly interesting comment during a presentation. - FIG. 10 shows another combination of the features “silence” and “shot detect”. The corresponding
feature indication window 6 indicates portions in the media file where a shot detect and silence are simultaneously present. This combination can be useful to indicate the beginning or end of a desired portion in the media file. - As will be appreciated, other feature combinations are possible. For example, a combination of speech detection and motion detection could provide useful information regarding gestures made by a speaker. In addition, more than two features can be combined to generate more sophisticated feature indication information. Preferably, the
graphical user interface 10 includes a “Combine” button that a user can click to combine two or more features. However, a user can define which features to combine by other methods. - As shown in FIG. 11, display of confidence score information can be conditioned on other features. In this example, confidence score information related to the feature “speaker MD” is displayed only when confidence score information indicates a relatively high probability that the speaker MD is shown in a video segment of the media file along with or within two minutes of applause in the soundtrack of the media file. As will be appreciated by those of ordinary skill in the art, other feature combinations or conditions, e.g., Boolean operators, can be used depending on the type of feature indication information needed for a specific task and the relevant features.
- FIG. 12 is a flowchart of a preferred method for browsing a media file. In step S10, a user inputs a command indicating a desire to browse a media file. In step S20, the desired media file is retrieved. In step S30, the
graphical user interface 10 is preferably displayed to the user, although other interfaces can be used. In step S40, the user preferably selects at least one feature of the media file. In step S50, confidence scores corresponding to the selected features are retrieved and/or generated. Preferably, metadata values for all of the selectable features are precomputed and stored with the media file so that metadata values need not be generated during a browsing session. However, the metadata values and corresponding confidence score values can be generated during a browsing session. - In step S60, feature indications related to the selected features are displayed to the user. In step S70, the user enters commands to play back or otherwise manipulate the media file as desired. For example, a user can click on an
icon 8 in a feature indication window 7 and media playback begins from a corresponding position in the media file. - FIG. 13 is a schematic block diagram of a preferred embodiment of a
media browser 100 in accordance with the invention. Themedia browser 100 preferably includes ageneral purpose computer 110 that has acontroller 111. Thecontroller 111 can be implemented as a single special purpose integrated circuit (e.g., ASIC) each having a main or central processor section for overall, system-level control, and separate sections dedicated to performing various different specific computations, functions and other processes under the control of the central processor section. Thecontroller 111 can also be implemented as a single microprocessor circuit or a plurality of separate dedicated or programmable integrated or other electronic circuits or devices, e.g., hardwired electronic or logic circuits such as discrete element circuits or programmable logic devices. Thecontroller 111 and/or thegeneral purpose computer 110 also preferably include other circuitry or components, such as memory devices, relays, mechanical linkages, communications devices, etc., to effect desired control and/or input/output functions. - The
general purpose computer 110 also includes afeature selector 112, aposition indicator 113, afeature indicator 114 and amapping module 115, which are all preferably implemented as software modules that include sets of instructions performed by thegeneral purpose computer 110 and/or thecontroller 111. However, these elements, like thecontroller 111, can be implemented as an ASIC or an array of ASICs, separate dedicated or programmable integrated or other electronic circuits or other devices. - A
media input device 120 preferably inputs a media file in the form of a data stream to thegeneral purpose computer 110. Themedia input device 120 can include one or more of any of the following devices: a semiconductor memory device, CD-ROM device, magnetic disk drive, other volatile or non-volatile memory devices, communications devices, such as a modem, facsimile transmitter or receiver, local-area or wide-area network devices, telecommunications devices, or other wired or wireless communications devices. In short, themedia input device 120 can be any device that transmits a data stream representing a media file to thegeneral purpose computer 110. - The data stream transmitted from the
media input device 120 to thegeneral purpose computer 110 can take any of many different forms. For example, the data stream can be analog or digital data that is either compressed or uncompressed, and the data can be provided in parallel or serial form at any desired transmission rate using any desired transmission protocol, if any. - The
controller 111 preferably communicates with adisplay 130, such as a CRT or LCD display. However, thedisplay 130 is not limited to these types of displays. In short, thedisplay 130 can include any device capable of presenting a representation of a media file to a user, including an audio speaker, visual display device, touch display (such as a Braille device), etc. - The
media browser 100 also preferably includes auser interface 140, such as a keyboard and/or computer mouse, that permits a user to communicate with thecontroller 111. Other devices can be used with theuser interface 140, including touch screen devices, light pens, voice activated and/or voice recognition systems, etc. - Where the invention is used over a more distributed system, such as the Internet, the
user interface 140 may include a general purpose computer and appropriate communications devices to send and receive signals to a more remotely locatedgeneral purpose computer 110, i.e., a remote server. In this case, thegeneral purpose computer 110 can send display control signals to theuser interface 140, which directly controls thedisplay 130. In addition, theuser interface 140 can perform some or all of the functions performed by thegeneral purpose computer 110, and some or all of the modules, such as theposition indicator 113,mapping module 115, etc., can operate, at least in part, within theuser interface 140. Themedia input 120 can be physically located near thegeneral purpose computer 110, the user interface 140 (e.g., a CD-ROM drive), or at some other location. Alternately, themedia input 120 can be formed of a multitude of media sources, such as magnetic hard drives distributed over a wide network. All that is required is that the media stored in themedia input 120 be accessable to themedia browser 100. - In a preferred mode of operation, the
media browser 100 receives a command from a user via theuser interface 140 indicating a desire to browse a media file. The desired media file is then either transmitted from themedia input device 120 to thegeneral purpose computer 110 or the media file is retrieved from a memory device provided as part of thecontroller 111, for example. Thecontroller 111 preferably sends a signal to thedisplay 130 to display thegraphical user interface 10 to the user. Preferably, included in the signal provided to thedisplay 130 is a command to indicate a current position of the media file is the beginning of the media file. That is, thethumb 31 on thetimeline 3 is displayed at the far left position. Of course, thethumb 31 need not be displayed, or can be displayed at any desired position along thetimeline 3. Likewise, a video frame, if present, is displayed in thevideo display portion 1 corresponding to a current position in the media file. The signal provided by thecontroller 111 to thedisplay 130 regarding the position of thethumb 31 is based on a signal provided by theposition indicator 113, which keeps track of a current position in the media file. - After viewing the
graphical user interface 10, the user preferably selects at least one feature of the media file using themedia feature selection 4 and theuser interface 140. Feature selection indications are provided to thefeature selector 112, which stores the selected features and provides a signal to thecontroller 111 to display the selected features in a selectedfeature window 5 of theinterface 10. Thefeature selector 112 also provides an indication to thecontroller 111 if a selected feature is not selectable. For example, if a user selects a feature that is not present in a media file, e.g., a user selects the shot detect feature for a media file that does not contain video information, an indication that the selected feature is not selectable is provided to the user. Preferably, thefeature selector 112 provides a list of selectable features to thecontroller 111 so that only selectable features are displayed in themedia feature selection 4. - Once media features are selected, the
mapping module 115 generates and/or retrieves confidence scores corresponding to the selected features. Preferably, metadata values for all of the selectable features are precomputed and stored with the media file so that metadata values need not be generated during a browsing session. However, themapping module 115 can automatically generate metadata values and map the metadata values to corresponding confidence score values “on the fly.” - Next, the
feature indicator 114 generates and provides information to thecontroller 111 to display feature indications related to the selected features. - Based on the displayed feature indications, a user can select a portion of the media file for review, or perform other actions, such as zooming the
timeline 3, moving thetimeline thumb 31, etc. When the user makes a decision to review a portion of the media file, a user can input signals to themedia browser 100 to playback or otherwise manipulate the media file. For example, the user can click on anicon 8 in a feature indication window 7 and thecontroller 111 controls thedisplay 130 to provide a representation of the media file starting from a time-wise position in the media file consistent with the selectedicon 8. - FIG. 14 is a flowchart that describes in further detail how a user chooses a media file for browsing and provides a corresponding signal to the
media browser 100. In step S110, thecontroller 111 sends a signal to thedisplay 130 to display a media selection screen. Preferably, the displayed selection screen includes at least the display shown in FIG. 15. The selection screen preferably includesseveral rows 200, eachrow 200 displaying information for a corresponding media file. Eachrow 200 preferably includes such information as akeyframe 201, atimeline 202, amedia file name 203, identifying information, such as a date, 204, and akeyword listing 205. Thekeyframe 201 is preferably selected from a particularly relevant portion of the media file or otherwise visually conveys information related to the overall content of the media file. Thetimeline 202 is similar to thetimeline 3 in thegraphical user interface 10. Thetimeline 202 preferably displays information related to a selected media feature, e.g., shot boundaries. Preferably, a user can view keyframes or other video information when acursor 206 is moved over thetimeline 202. For example, when a user moves thecursor 206, a keyframe in the media file that is nearest in time to thecursor 206 position is displayed to the user. Keyframes are represented on thetimeline 202 by a small icon that preferably changes appearance when the icon is nearest thecursor 206. Keyframes can be automatically selected based on confidence score information or can be previously identified by an operator. Alternately, themedia browser 100 could display a predefined summary video sequence or another shortened representation that provides a user additional information to make a selection decision. Thetimeline 202 can also provide other information related to media features in any of the ways described above. For example, icons, color sequences or keyframes could be displayed on thetimeline 202 to provide media feature information. The media feature information can be displayed based on metadata and confidence score information or based on predefined summary information provided by a user. - Preferably, a user can search a database of media files by the
name 203, identifyinginformation 204 and/orkeywords 205 to narrow an initial search and generate an initial display ofmedia file rows 200 on the selection screen. Preferably, the user can search using any known search screen, such as those associated with electronic database applications or Internet browser/search engines. In fact, themedia browser 100 can be very useful in browsing media files posted on a large network, such as the Internet, since themedia browser 100 allows a user to rapidly and accurately identify and experience desired portions in media file. To speed review and selection, the media displayed on the initial selection screen are preferably sorted, placing those media files most relevant to the search at the top. Sorting can be done based on confidence score information, search results (e.g., number of hits in a media file), or other desired criteria. - In step S120, the user reviews the media files, either by viewing keyframes, keywords, selected portions of the media files using the
timeline 202 features, or by reviewing the media files in any of the ways discussed above. In step S130, the user selects one or more of the media files for browsing. Preferably, the user selects a media file by clicking on an associatedkeyframe 201, but the user can select media files in other ways, such as by entering a media file name or other identifier by keyboard. - FIG. 16 is a flowchart that describes in further detail how the
mapping module 115 generates and/or retrieves confidence scores. In step S510, themapping module 115 generates and/or retrieves metadata information for the selected features. As described above, the metadata information is preferably precomputed and stored along with the media file. However, if metadata for a selected feature is not stored with the media file, themapping module 115 can apply any of the above described automatic techniques to the media file to generate metadata information. In addition, themapping module 115 can receive manually input metadata information from a user. - In step S520, the
mapping module 115 optionally processes the metadata information as desired in preparation for mapping the metadata to confidence scores. For example, themapping module 115 thresholds, combines, filters or otherwise processes the metadata information in preparation for mapping the metadata information to confidence scores. - In step S530, the mapping module maps the processed metadata information to confidence score values for each selected feature using a desired function, e.g., a one-to-one mapping, other linear or non-linear functions, etc.
- FIG. 17 is a flowchart describing in further detail how the
feature indicator 114 provides feature information for the selected features. In step S610, thefeature indicator 114 processes the confidence scores for each selected feature as desired. For example, thefeature indicator 114 thresholds, filters, combines, etc., the confidence score values as discussed above in preparation for providing feature information. - In step S620, the
feature indicator 114 generates information to display feature indications for each of the selected features based on the processed confidence scores. For example, thefeature indicator 114 generates signals to display feature indications in afeature indication window 6. - FIG. 18 is a flowchart describing in further detail how the
controller 111 controls media playback as desired. In step S710, a user optionally selects features using thekey playback buttons 9 on thegraphical user interface 10. For example, the user could select the “speaker MD” feature. In step S720, thecontroller 111 inputs a playback control signal received from a user and adjusts the media playback based on the features selected in step S710. For example, the user could select the play button, and themedia browser 100 adjusts the replay of the media file based on the “speaker MD” feature. As one example, themedia browser 100 could adjust the playback speed according to Equation 2: - R=C+M(1−C) (2)
- where R is the playback rate, C is the confidence score for the speaker MD feature, and M is a maximum playback rate, which is preferably set by a user. In this example, the confidence score C varies between 0 and 1. Therefore, as the confidence score increases (indicating that the speaker MD is speaking or is depicted in the media file), the playback rate approaches the value C. In contrast, as the confidence score decreases (indicating that the speaker MD is not featured in the media file), the playback rate approaches the maximum playback rate M, thereby speeding replay in sections where the speaker MD is not present in the media file.
- This is only one example of how the
media browser 100 can control the media playback controls based on a selected feature. Themedia browser 100 can also control the stop button to stop playback at a point in the media file where a feature is no longer present (e.g., a speaker leaves the room) or where a feature next occurs (e.g., stop at the next shot boundary). As discussed above, the index controls can be controlled to jump to a point in the media file where a feature is present in or absent from the media file. Further, thetimeline thumb 31 can be controlled based on a selected feature. For example, thethumb 31 could change its appearance, e.g., color, when thethumb 31 is moved over a media file portion where a selected feature is present, or thethumb 31 could be “indexed” so that a user can only move thethumb 31 within portions of the media file where a selected feature is present or absent. Preferably, control of the playback controls, including thethumb 31, is based on the confidence score value for a selected feature. Determination of whether a feature is present or not in a media file is preferably based on the confidence score value, e.g., if the confidence score exceeds/is below a desired threshold themedia browser 100 determines that the feature is present in/absent from the media file. - FIG. 19 is a flowchart describing in further detail another way how the
controller 111 optionally controls media playback. In step S730, a user selects a confidence score indication in afeature indication window 6. For example, a user can select (e.g., double click) anicon 8 displayed in afeature indication window 6. In step S740, thecontroller 111 accesses the media file and controls thedisplay 130 to begin playback at a point in the media file corresponding to the selected confidence score indication. Playback can be of selected portions of the media file of particular interest determined based on confidence score values. - While the invention has been described with reference to specific embodiments, the description of the specific embodiments is illustrative only and is not to be construed as limiting the scope of the invention. Various other modifications and changes may occur to those skilled in the art without departing from the spirit and scope of the invention as set forth herein.
Claims (3)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/934,486 US20020054083A1 (en) | 1998-09-11 | 2001-08-23 | Media browser using multimodal analysis |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/151,285 US6366296B1 (en) | 1998-09-11 | 1998-09-11 | Media browser using multimodal analysis |
US09/934,486 US20020054083A1 (en) | 1998-09-11 | 2001-08-23 | Media browser using multimodal analysis |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/151,285 Continuation US6366296B1 (en) | 1998-09-11 | 1998-09-11 | Media browser using multimodal analysis |
Publications (1)
Publication Number | Publication Date |
---|---|
US20020054083A1 true US20020054083A1 (en) | 2002-05-09 |
Family
ID=22538077
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/151,285 Expired - Lifetime US6366296B1 (en) | 1998-09-11 | 1998-09-11 | Media browser using multimodal analysis |
US09/934,486 Abandoned US20020054083A1 (en) | 1998-09-11 | 2001-08-23 | Media browser using multimodal analysis |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/151,285 Expired - Lifetime US6366296B1 (en) | 1998-09-11 | 1998-09-11 | Media browser using multimodal analysis |
Country Status (2)
Country | Link |
---|---|
US (2) | US6366296B1 (en) |
JP (1) | JP2000090121A (en) |
Cited By (36)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6711587B1 (en) | 2000-09-05 | 2004-03-23 | Hewlett-Packard Development Company, L.P. | Keyframe selection to represent a video |
US20040103372A1 (en) * | 1997-12-22 | 2004-05-27 | Ricoh Company, Ltd. | Multimedia visualization and integration environment |
US6782049B1 (en) * | 1999-01-29 | 2004-08-24 | Hewlett-Packard Development Company, L.P. | System for selecting a keyframe to represent a video |
USRE38609E1 (en) * | 2000-02-28 | 2004-10-05 | Webex Communications, Inc. | On-demand presentation graphical user interface |
EP1376584A3 (en) * | 2002-06-19 | 2004-12-22 | Microsoft Corporation | System and method for automatically generating video cliplets from digital video |
US20050151737A1 (en) * | 2004-01-14 | 2005-07-14 | Fuji Xerox Co., Ltd | Systems and methods for creating and updating an interactive 3D visualization of media indices |
US20050183041A1 (en) * | 2004-02-12 | 2005-08-18 | Fuji Xerox Co., Ltd. | Systems and methods for creating and interactive 3D visualization of indexed media |
US20050257151A1 (en) * | 2004-05-13 | 2005-11-17 | Peng Wu | Method and apparatus for identifying selected portions of a video stream |
US20060218192A1 (en) * | 2004-08-31 | 2006-09-28 | Gopalakrishnan Kumar C | Method and System for Providing Information Services Related to Multimodal Inputs |
US20070168413A1 (en) * | 2003-12-05 | 2007-07-19 | Sony Deutschland Gmbh | Visualization and control techniques for multimedia digital content |
US7296021B2 (en) | 2004-05-21 | 2007-11-13 | International Business Machines Corporation | Method, system, and article to specify compound query, displaying visual indication includes a series of graphical bars specify weight relevance, ordered segments of unique colors where each segment length indicative of the extent of match of each object with one of search parameters |
US20070286489A1 (en) * | 2005-07-01 | 2007-12-13 | International Business Machines Corporation | Methods and apparatus for filtering video packets for large-scale video stream monitoring |
US20080306921A1 (en) * | 2000-01-31 | 2008-12-11 | Kenneth Rothmuller | Digital Media Management Apparatus and Methods |
US20090157743A1 (en) * | 2007-12-12 | 2009-06-18 | Aten International Co., Ltd. | Media file generation method and media playback method |
US20090164904A1 (en) * | 2007-12-21 | 2009-06-25 | Yahoo! Inc. | Blog-Based Video Summarization |
US7672864B2 (en) | 2004-01-09 | 2010-03-02 | Ricoh Company Ltd. | Generating and displaying level-of-interest values |
US20100303365A1 (en) * | 2009-05-29 | 2010-12-02 | Min Zhang | Methods and apparatus to monitor a multimedia presentation including multiple content windows |
US20110035700A1 (en) * | 2009-08-05 | 2011-02-10 | Brian Meaney | Multi-Operation User Interface Tool |
US20110092251A1 (en) * | 2004-08-31 | 2011-04-21 | Gopalakrishnan Kumar C | Providing Search Results from Visual Imagery |
US20110093264A1 (en) * | 2004-08-31 | 2011-04-21 | Kumar Gopalakrishnan | Providing Information Services Related to Multimodal Inputs |
US20110093884A1 (en) * | 2000-01-14 | 2011-04-21 | Nds Limited. | Advertisements in an end-user controlled playback environment |
US20130166303A1 (en) * | 2009-11-13 | 2013-06-27 | Adobe Systems Incorporated | Accessing media data using metadata repository |
US20130290846A1 (en) * | 2012-04-27 | 2013-10-31 | Mobitv, Inc. | Search-based navigation of media content |
CN103399865A (en) * | 2013-07-05 | 2013-11-20 | 华为技术有限公司 | Method and device for multi-media file generation |
US20140032562A1 (en) * | 2012-07-26 | 2014-01-30 | Telefonaktiebolaget Lm Ericsson (Publ) | Apparatus and methods for user generated content indexing |
US20140324907A1 (en) * | 2005-11-29 | 2014-10-30 | Aol Inc. | Visually-Represented Results To Search Queries In Rich Media Content |
US20150082338A1 (en) * | 2000-03-28 | 2015-03-19 | Compass Innovations, LLC | Audiovisual Content Presentation Dependent On Metadata |
US20150095749A1 (en) * | 2013-09-27 | 2015-04-02 | Nokia Corporation | Visual Representation of a Character Identity and a Location Identity |
US9691068B1 (en) * | 2011-12-15 | 2017-06-27 | Amazon Technologies, Inc. | Public-domain analyzer |
US9846696B2 (en) | 2012-02-29 | 2017-12-19 | Telefonaktiebolaget Lm Ericsson (Publ) | Apparatus and methods for indexing multimedia content |
US10289810B2 (en) | 2013-08-29 | 2019-05-14 | Telefonaktiebolaget Lm Ericsson (Publ) | Method, content owner device, computer program, and computer program product for distributing content items to authorized users |
US10311038B2 (en) | 2013-08-29 | 2019-06-04 | Telefonaktiebolaget Lm Ericsson (Publ) | Methods, computer program, computer program product and indexing systems for indexing or updating index |
US10445367B2 (en) | 2013-05-14 | 2019-10-15 | Telefonaktiebolaget Lm Ericsson (Publ) | Search engine for textual content and non-textual content |
US11082722B2 (en) * | 2011-01-26 | 2021-08-03 | Afterlive.tv Inc. | Method and system for generating highlights from scored data streams |
US20230009515A1 (en) * | 2021-07-06 | 2023-01-12 | Rovi Guides, Inc. | Generating verified content profiles for user generated content |
US20230222164A1 (en) * | 2022-01-11 | 2023-07-13 | Kyndryl, Inc. | Autonomous webpage content summation |
Families Citing this family (209)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH11506575A (en) * | 1995-03-07 | 1999-06-08 | インターバル リサーチ コーポレイション | Information selective storage system and method |
US20030093790A1 (en) * | 2000-03-28 | 2003-05-15 | Logan James D. | Audio and video program recording, editing and playback systems using metadata |
US20060280437A1 (en) * | 1999-01-27 | 2006-12-14 | Gotuit Media Corp | Methods and apparatus for vending and delivering the content of disk recordings |
US6263507B1 (en) * | 1996-12-05 | 2001-07-17 | Interval Research Corporation | Browser for use in navigating a body of information, with particular application to browsing information represented by audiovisual data |
US5893062A (en) | 1996-12-05 | 1999-04-06 | Interval Research Corporation | Variable rate video playback with synchronized audio |
US7111009B1 (en) | 1997-03-14 | 2006-09-19 | Microsoft Corporation | Interactive playlist generation using annotations |
AU5926099A (en) | 1998-09-15 | 2000-04-03 | Microsoft Corporation | Annotation creation and notification via electronic mail |
US6956593B1 (en) | 1998-09-15 | 2005-10-18 | Microsoft Corporation | User interface for creating, viewing and temporally positioning annotations for media content |
JP2000207263A (en) * | 1998-11-12 | 2000-07-28 | Sony Corp | Processor and method for information processing and providing medium |
US7209942B1 (en) * | 1998-12-28 | 2007-04-24 | Kabushiki Kaisha Toshiba | Information providing method and apparatus, and information reception apparatus |
US6934461B1 (en) * | 1999-01-05 | 2005-08-23 | Interval Research Corporation | Low attention recording, with particular application to social recording |
JP2000253337A (en) * | 1999-02-24 | 2000-09-14 | Sony Corp | Method and device for controlling screen, method and device for reproducing video, method and device for recording video information, and computer readable recording medium |
US6262724B1 (en) * | 1999-04-15 | 2001-07-17 | Apple Computer, Inc. | User interface for presenting media information |
US7313808B1 (en) * | 1999-07-08 | 2007-12-25 | Microsoft Corporation | Browsing continuous multimedia content |
US7293280B1 (en) * | 1999-07-08 | 2007-11-06 | Microsoft Corporation | Skimming continuous multimedia content |
JP3424204B2 (en) * | 1999-07-23 | 2003-07-07 | インターナショナル・ビジネス・マシーンズ・コーポレーション | Method and apparatus for accessing video content using trajectory of object |
GB2354105A (en) * | 1999-09-08 | 2001-03-14 | Sony Uk Ltd | System and method for navigating source content |
US7155735B1 (en) | 1999-10-08 | 2006-12-26 | Vulcan Patents Llc | System and method for the broadcast dissemination of time-ordered data |
US6757682B1 (en) | 2000-01-28 | 2004-06-29 | Interval Research Corporation | Alerting users to items of current interest |
US6868440B1 (en) * | 2000-02-04 | 2005-03-15 | Microsoft Corporation | Multi-level skimming of multimedia content using playlists |
JP2001230994A (en) * | 2000-02-15 | 2001-08-24 | Fujitsu Ltd | Data processor |
US20060015925A1 (en) * | 2000-03-28 | 2006-01-19 | Gotuit Media Corp | Sales presentation video on demand system |
US7194186B1 (en) * | 2000-04-21 | 2007-03-20 | Vulcan Patents Llc | Flexible marking of recording data by a recording unit |
US6701014B1 (en) * | 2000-06-14 | 2004-03-02 | International Business Machines Corporation | Method and apparatus for matching slides in video |
US6882793B1 (en) | 2000-06-16 | 2005-04-19 | Yesvideo, Inc. | Video processing system |
US7490092B2 (en) | 2000-07-06 | 2009-02-10 | Streamsage, Inc. | Method and system for indexing and searching timed media information based upon relevance intervals |
EP1173013A3 (en) * | 2000-07-14 | 2004-04-21 | Sony Corporation | Remote control device for recording/reproducing apparatus and video signal recording/reproducing apparatus |
US8191103B2 (en) | 2000-08-30 | 2012-05-29 | Sony Corporation | Real-time bookmarking of streaming media assets |
US7254779B1 (en) * | 2000-09-06 | 2007-08-07 | Xanboo Inc. | Customizable presentation environment |
US6686838B1 (en) * | 2000-09-06 | 2004-02-03 | Xanboo Inc. | Systems and methods for the automatic registration of devices |
US6748375B1 (en) * | 2000-09-07 | 2004-06-08 | Microsoft Corporation | System and method for content retrieval |
CN100392749C (en) * | 2000-09-08 | 2008-06-04 | 皇家菲利浦电子有限公司 | Apparatus for reproducing information signal stored on storage medium |
US7890989B1 (en) | 2000-10-24 | 2011-02-15 | Sony Corporation | Automated context-sensitive updating on content in an audiovisual storage system |
JP2002204441A (en) * | 2000-10-26 | 2002-07-19 | Sony Corp | Information processing unit and information processing system |
US8078967B2 (en) | 2000-12-01 | 2011-12-13 | Jlb Ventures Llc | Method and apparatus for virtual editing of multimedia presentations |
US7177531B2 (en) * | 2000-12-05 | 2007-02-13 | Matsushita Electric Industrial Co., Ltd. | Record and playback apparatus and record medium |
US7401351B2 (en) * | 2000-12-14 | 2008-07-15 | Fuji Xerox Co., Ltd. | System and method for video navigation and client side indexing |
US7007243B2 (en) | 2000-12-20 | 2006-02-28 | Eastman Kodak Company | Method and apparatus for producing digital images with embedded image capture location icons |
US7020848B2 (en) * | 2000-12-20 | 2006-03-28 | Eastman Kodak Company | Comprehensive, multi-dimensional graphical user interface using picture metadata for navigating and retrieving pictures in a picture database |
US20020083473A1 (en) * | 2000-12-21 | 2002-06-27 | Philips Electronics North America Corporation | System and method for accessing a multimedia summary of a video program |
US20070300258A1 (en) * | 2001-01-29 | 2007-12-27 | O'connor Daniel | Methods and systems for providing media assets over a network |
JP4188568B2 (en) * | 2001-02-06 | 2008-11-26 | パイオニア株式会社 | Information reproducing apparatus and information editing method |
US20020120833A1 (en) * | 2001-02-15 | 2002-08-29 | Inventec Corporation | Method for automatic obtaining the ID number of items of a volume control program |
JP2002259720A (en) * | 2001-03-02 | 2002-09-13 | Internatl Business Mach Corp <Ibm> | Contents summarizing system, image summarizing system, user terminal unit, summary image producing method, summary image receiving method, and program |
EP1251515A1 (en) * | 2001-04-19 | 2002-10-23 | Koninklijke Philips Electronics N.V. | Method and system for selecting a position in an image sequence |
US20050005308A1 (en) * | 2002-01-29 | 2005-01-06 | Gotuit Video, Inc. | Methods and apparatus for recording and replaying sports broadcasts |
US20030009371A1 (en) * | 2001-07-03 | 2003-01-09 | Ravi Gauba | Interactive decision-making scenarios in an audio/video broadcast |
US20030028871A1 (en) * | 2001-07-20 | 2003-02-06 | Annie Wang | Behavior profile system and method |
US8515773B2 (en) | 2001-08-01 | 2013-08-20 | Sony Corporation | System and method for enabling distribution and brokering of content information |
US20030025720A1 (en) * | 2001-08-03 | 2003-02-06 | Clement Lau | System and method for common interest analysis among multiple users |
US7747943B2 (en) * | 2001-09-07 | 2010-06-29 | Microsoft Corporation | Robust anchoring of annotations to content |
US20030074447A1 (en) * | 2001-10-16 | 2003-04-17 | Rafey Richter A. | Intuitive mapping between explicit and implicit personalization |
CN1269105C (en) * | 2001-10-31 | 2006-08-09 | 皇家飞利浦电子股份有限公司 | Method of and system for transcribing dictations in text files and for revising the text |
US20030112276A1 (en) * | 2001-12-19 | 2003-06-19 | Clement Lau | User augmentation of content |
US7337455B2 (en) * | 2001-12-31 | 2008-02-26 | Koninklijke Philips Electronics N.V. | Method, apparatus, and program for evolving algorithms for detecting content in information streams |
JP3772117B2 (en) * | 2002-01-18 | 2006-05-10 | ソニー株式会社 | Information signal processing apparatus and information signal processing method |
US7739601B1 (en) * | 2002-01-23 | 2010-06-15 | Microsoft Corporation | Media authoring and presentation |
US20070113250A1 (en) * | 2002-01-29 | 2007-05-17 | Logan James D | On demand fantasy sports systems and methods |
US7243301B2 (en) | 2002-04-10 | 2007-07-10 | Microsoft Corporation | Common annotation framework |
US7568151B2 (en) * | 2002-06-27 | 2009-07-28 | Microsoft Corporation | Notification of activity around documents |
US7587318B2 (en) * | 2002-09-12 | 2009-09-08 | Broadcom Corporation | Correlating video images of lip movements with audio signals to improve speech recognition |
US20040059996A1 (en) * | 2002-09-24 | 2004-03-25 | Fasciano Peter J. | Exhibition of digital media assets from a digital media asset management system to facilitate creative story generation |
DE10297802B4 (en) * | 2002-09-30 | 2011-05-19 | Adobe Systems, Inc., San Jose | Method, storage medium and system for searching a collection of media objects |
EP1584048A1 (en) * | 2003-01-06 | 2005-10-12 | Koninklijke Philips Electronics N.V. | Method and apparatus for similar video content hopping |
EP1463258A1 (en) * | 2003-03-28 | 2004-09-29 | Mobile Integrated Solutions Limited | A system and method for transferring data over a wireless communications network |
US7233316B2 (en) | 2003-05-01 | 2007-06-19 | Thomson Licensing | Multimedia user interface |
JP2004364234A (en) * | 2003-05-15 | 2004-12-24 | Pioneer Electronic Corp | Broadcast program content menu creation apparatus and method |
US20040230910A1 (en) * | 2003-05-16 | 2004-11-18 | Brian Lin | Assistant editing display method for media clips |
JP4235490B2 (en) * | 2003-06-04 | 2009-03-11 | パイオニア株式会社 | Music program content menu creation apparatus and method |
US7512622B2 (en) * | 2003-06-11 | 2009-03-31 | Yahoo! Inc. | Method and apparatus for organizing and playing data |
US20070118812A1 (en) * | 2003-07-15 | 2007-05-24 | Kaleidescope, Inc. | Masking for presenting differing display formats for media streams |
US20050044561A1 (en) * | 2003-08-20 | 2005-02-24 | Gotuit Audio, Inc. | Methods and apparatus for identifying program segments by detecting duplicate signal patterns |
US20050071214A1 (en) * | 2003-09-25 | 2005-03-31 | Kover Arthur J. | Method and apparatus for obtaining web-based advertising research data |
US7596255B2 (en) * | 2003-11-26 | 2009-09-29 | Ge Medical Systems Information Technologies, Inc. | Image navigation system and method |
US20050154987A1 (en) * | 2004-01-14 | 2005-07-14 | Isao Otsuka | System and method for recording and reproducing multimedia |
JP4426853B2 (en) * | 2004-01-19 | 2010-03-03 | オリンパス株式会社 | Information display program, information display method, information display apparatus, and recording medium |
US7447331B2 (en) * | 2004-02-24 | 2008-11-04 | International Business Machines Corporation | System and method for generating a viewable video index for low bandwidth applications |
US20060031879A1 (en) * | 2004-04-30 | 2006-02-09 | Vulcan Inc. | Management and non-linear presentation of news-related broadcasted or streamed multimedia content |
US20060031916A1 (en) * | 2004-04-30 | 2006-02-09 | Vulcan Inc. | Management and non-linear presentation of broadcasted or streamed multimedia content |
US20060031885A1 (en) * | 2004-04-30 | 2006-02-09 | Vulcan Inc. | Management and non-linear presentation of music-related broadcasted or streamed multimedia content |
US7908339B2 (en) * | 2004-06-03 | 2011-03-15 | Maxsp Corporation | Transaction based virtual file system optimized for high-latency network connections |
US8812613B2 (en) * | 2004-06-03 | 2014-08-19 | Maxsp Corporation | Virtual application manager |
US9357031B2 (en) | 2004-06-03 | 2016-05-31 | Microsoft Technology Licensing, Llc | Applications as a service |
US7664834B2 (en) * | 2004-07-09 | 2010-02-16 | Maxsp Corporation | Distributed operating system management |
JP4123209B2 (en) * | 2004-09-07 | 2008-07-23 | ソニー株式会社 | Video material management apparatus and method, recording medium, and program |
US8768844B2 (en) | 2004-10-06 | 2014-07-01 | Sony Corporation | Method and system for content sharing and authentication between multiple devices |
JP4649944B2 (en) * | 2004-10-20 | 2011-03-16 | 富士ゼロックス株式会社 | Moving image processing apparatus, moving image processing method, and program |
US7783106B2 (en) * | 2004-11-12 | 2010-08-24 | Fuji Xerox Co., Ltd. | Video segmentation combining similarity analysis and classification |
US7624086B2 (en) * | 2005-03-04 | 2009-11-24 | Maxsp Corporation | Pre-install compliance system |
US7512584B2 (en) * | 2005-03-04 | 2009-03-31 | Maxsp Corporation | Computer hardware and software diagnostic and report system |
US8589323B2 (en) | 2005-03-04 | 2013-11-19 | Maxsp Corporation | Computer hardware and software diagnostic and report system incorporating an expert system and agents |
US8234238B2 (en) * | 2005-03-04 | 2012-07-31 | Maxsp Corporation | Computer hardware and software diagnostic and report system |
US7689631B2 (en) * | 2005-05-31 | 2010-03-30 | Sap, Ag | Method for utilizing audience-specific metadata |
US8089563B2 (en) * | 2005-06-17 | 2012-01-03 | Fuji Xerox Co., Ltd. | Method and system for analyzing fixed-camera video via the selection, visualization, and interaction with storyboard keyframes |
US7996771B2 (en) * | 2005-06-17 | 2011-08-09 | Fuji Xerox Co., Ltd. | Methods and interfaces for event timeline and logs of video streams |
US7623677B2 (en) * | 2005-06-17 | 2009-11-24 | Fuji Xerox Co., Ltd. | Methods and interfaces for visualizing activity across video frames in an action keyframe |
US20070016611A1 (en) * | 2005-07-13 | 2007-01-18 | Ulead Systems, Inc. | Preview method for seeking media content |
US7487206B2 (en) * | 2005-07-15 | 2009-02-03 | International Business Machines Corporation | Method for providing load diffusion in data stream correlations |
US20070027808A1 (en) * | 2005-07-29 | 2007-02-01 | Microsoft Corporation | Strategies for queuing events for subsequent processing |
GB2430101A (en) * | 2005-09-09 | 2007-03-14 | Mitsubishi Electric Inf Tech | Applying metadata for video navigation |
US20070185857A1 (en) * | 2006-01-23 | 2007-08-09 | International Business Machines Corporation | System and method for extracting salient keywords for videos |
US7720258B1 (en) | 2006-01-26 | 2010-05-18 | Adobe Systems Incorporated | Structured comparison of objects from similar images |
US7706577B1 (en) | 2006-01-26 | 2010-04-27 | Adobe Systems Incorporated | Exporting extracted faces |
US7716157B1 (en) | 2006-01-26 | 2010-05-11 | Adobe Systems Incorporated | Searching images with extracted objects |
US7813557B1 (en) | 2006-01-26 | 2010-10-12 | Adobe Systems Incorporated | Tagging detected objects |
US8259995B1 (en) | 2006-01-26 | 2012-09-04 | Adobe Systems Incorporated | Designating a tag icon |
US7978936B1 (en) * | 2006-01-26 | 2011-07-12 | Adobe Systems Incorporated | Indicating a correspondence between an image and an object |
US7813526B1 (en) | 2006-01-26 | 2010-10-12 | Adobe Systems Incorporated | Normalizing detected objects |
US7636450B1 (en) | 2006-01-26 | 2009-12-22 | Adobe Systems Incorporated | Displaying detected objects to indicate grouping |
US7694885B1 (en) | 2006-01-26 | 2010-04-13 | Adobe Systems Incorporated | Indicating a tag with visual data |
US20070204238A1 (en) * | 2006-02-27 | 2007-08-30 | Microsoft Corporation | Smart Video Presentation |
US7735101B2 (en) | 2006-03-28 | 2010-06-08 | Cisco Technology, Inc. | System allowing users to embed comments at specific points in time into media presentation |
US20080036917A1 (en) * | 2006-04-07 | 2008-02-14 | Mark Pascarella | Methods and systems for generating and delivering navigatable composite videos |
US8117246B2 (en) * | 2006-04-17 | 2012-02-14 | Microsoft Corporation | Registering, transfering, and acting on event metadata |
US20070256008A1 (en) * | 2006-04-26 | 2007-11-01 | Bedingfield James C Sr | Methods, systems, and computer program products for managing audio information |
US8701005B2 (en) * | 2006-04-26 | 2014-04-15 | At&T Intellectual Property I, Lp | Methods, systems, and computer program products for managing video information |
US20070256007A1 (en) * | 2006-04-26 | 2007-11-01 | Bedingfield James C Sr | Methods, systems, and computer program products for managing information by annotating a captured information object |
US7548937B2 (en) * | 2006-05-04 | 2009-06-16 | International Business Machines Corporation | System and method for scalable processing of multi-way data stream correlations |
US7954049B2 (en) | 2006-05-15 | 2011-05-31 | Microsoft Corporation | Annotating multimedia files along a timeline |
US8811396B2 (en) * | 2006-05-24 | 2014-08-19 | Maxsp Corporation | System for and method of securing a network utilizing credentials |
US8898319B2 (en) | 2006-05-24 | 2014-11-25 | Maxsp Corporation | Applications and services as a bundle |
US20070276852A1 (en) * | 2006-05-25 | 2007-11-29 | Microsoft Corporation | Downloading portions of media files |
US8140530B2 (en) * | 2006-08-03 | 2012-03-20 | Nec Corporation | Similarity calculation device and information search device |
US7956849B2 (en) | 2006-09-06 | 2011-06-07 | Apple Inc. | Video manager for portable multifunction device |
US8954851B2 (en) * | 2006-09-15 | 2015-02-10 | Microsoft Corporation | Adding video effects for video enabled applications |
US9317506B2 (en) * | 2006-09-22 | 2016-04-19 | Microsoft Technology Licensing, Llc | Accelerated data transfer using common prior data segments |
US20080077622A1 (en) * | 2006-09-22 | 2008-03-27 | Keith Robert O | Method of and apparatus for managing data utilizing configurable policies and schedules |
US7840514B2 (en) * | 2006-09-22 | 2010-11-23 | Maxsp Corporation | Secure virtual private network utilizing a diagnostics policy and diagnostics engine to establish a secure network connection |
US7792868B2 (en) * | 2006-11-10 | 2010-09-07 | Microsoft Corporation | Data object linking and browsing tool |
US7827490B2 (en) * | 2006-11-30 | 2010-11-02 | Microsoft Corporation | Media state user interface |
US8423821B1 (en) | 2006-12-21 | 2013-04-16 | Maxsp Corporation | Virtual recovery server |
US7844686B1 (en) | 2006-12-21 | 2010-11-30 | Maxsp Corporation | Warm standby appliance |
US7953640B2 (en) * | 2007-04-04 | 2011-05-31 | Disney Enterprises, Inc. | Method and system for determining a value of a portion of content |
US8091039B2 (en) * | 2007-04-13 | 2012-01-03 | Apple Inc. | Authoring interface which distributes composited elements about the display |
US8874468B2 (en) * | 2007-04-20 | 2014-10-28 | Google Inc. | Media advertising |
US9438860B2 (en) * | 2007-06-26 | 2016-09-06 | Verizon Patent And Licensing Inc. | Method and system for filtering advertisements in a media stream |
US8175418B1 (en) | 2007-10-26 | 2012-05-08 | Maxsp Corporation | Method of and system for enhanced data storage |
US8645515B2 (en) | 2007-10-26 | 2014-02-04 | Maxsp Corporation | Environment manager |
US8307239B1 (en) | 2007-10-26 | 2012-11-06 | Maxsp Corporation | Disaster recovery appliance |
US10216761B2 (en) * | 2008-03-04 | 2019-02-26 | Oath Inc. | Generating congruous metadata for multimedia |
US10552384B2 (en) | 2008-05-12 | 2020-02-04 | Blackberry Limited | Synchronizing media files available from multiple sources |
US8706690B2 (en) | 2008-05-12 | 2014-04-22 | Blackberry Limited | Systems and methods for space management in file systems |
US8140550B2 (en) * | 2008-08-20 | 2012-03-20 | Satyam Computer Services Limited Of Mayfair Centre | System and method for bounded analysis of multimedia using multiple correlations |
US9442933B2 (en) * | 2008-12-24 | 2016-09-13 | Comcast Interactive Media, Llc | Identification of segments within audio, video, and multimedia items |
US8713016B2 (en) | 2008-12-24 | 2014-04-29 | Comcast Interactive Media, Llc | Method and apparatus for organizing segments of media assets and determining relevance of segments to a query |
US11531668B2 (en) * | 2008-12-29 | 2022-12-20 | Comcast Interactive Media, Llc | Merging of multiple data sets |
US8176043B2 (en) | 2009-03-12 | 2012-05-08 | Comcast Interactive Media, Llc | Ranking search results |
US10705701B2 (en) | 2009-03-16 | 2020-07-07 | Apple Inc. | Device, method, and graphical user interface for moving a current position in content at a variable scrubbing rate |
US8533223B2 (en) * | 2009-05-12 | 2013-09-10 | Comcast Interactive Media, LLC. | Disambiguation and tagging of entities |
JP2010288015A (en) | 2009-06-10 | 2010-12-24 | Sony Corp | Information processing device, information processing method, and information processing program |
US9892730B2 (en) | 2009-07-01 | 2018-02-13 | Comcast Interactive Media, Llc | Generating topic-specific language models |
US9542489B2 (en) * | 2009-07-16 | 2017-01-10 | Bluefin Labs, Inc. | Estimating social interest in time-based media |
US9069884B2 (en) | 2009-09-08 | 2015-06-30 | International Business Machines Corporation | Processing special attributes within a file |
US8856636B1 (en) * | 2009-09-22 | 2014-10-07 | Adobe Systems Incorporated | Methods and systems for trimming video footage |
US8572488B2 (en) * | 2010-03-29 | 2013-10-29 | Avid Technology, Inc. | Spot dialog editor |
US8423555B2 (en) | 2010-07-09 | 2013-04-16 | Comcast Cable Communications, Llc | Automatic segmentation of video |
US10324605B2 (en) | 2011-02-16 | 2019-06-18 | Apple Inc. | Media-editing application with novel editing tools |
EP2659482B1 (en) * | 2010-12-30 | 2015-12-09 | Dolby Laboratories Licensing Corporation | Ranking representative segments in media data |
US8886015B2 (en) | 2011-01-28 | 2014-11-11 | Apple Inc. | Efficient media import |
US9026909B2 (en) | 2011-02-16 | 2015-05-05 | Apple Inc. | Keyword list view |
US9997196B2 (en) | 2011-02-16 | 2018-06-12 | Apple Inc. | Retiming media presentations |
US11747972B2 (en) | 2011-02-16 | 2023-09-05 | Apple Inc. | Media-editing application with novel editing tools |
US9288511B2 (en) | 2011-02-18 | 2016-03-15 | Futurewei Technologies, Inc. | Methods and apparatus for media navigation |
US10467289B2 (en) | 2011-08-02 | 2019-11-05 | Comcast Cable Communications, Llc | Segmentation of video according to narrative theme |
US9536564B2 (en) | 2011-09-20 | 2017-01-03 | Apple Inc. | Role-facilitated editing operations |
US20130073960A1 (en) | 2011-09-20 | 2013-03-21 | Aaron M. Eppolito | Audio meters and parameter controls |
US20130091421A1 (en) * | 2011-10-11 | 2013-04-11 | International Business Machines Corporation | Time relevance within a soft copy document or media object |
US9483110B2 (en) | 2011-11-07 | 2016-11-01 | International Business Machines Corporation | Adaptive media file rewind |
US9767194B2 (en) * | 2011-11-07 | 2017-09-19 | International Business Machines Corporation | Media file abbreviation retrieval |
JP5980604B2 (en) * | 2012-07-18 | 2016-08-31 | オリンパス株式会社 | Endoscope system |
US20140089467A1 (en) * | 2012-09-27 | 2014-03-27 | Andre Beck | Content stream delivery using pre-loaded segments |
JP2014126600A (en) * | 2012-12-25 | 2014-07-07 | Panasonic Corp | Voice recognition device, voice recognition method and television |
US8537983B1 (en) * | 2013-03-08 | 2013-09-17 | Noble Systems Corporation | Multi-component viewing tool for contact center agents |
US9495365B2 (en) * | 2013-03-15 | 2016-11-15 | International Business Machines Corporation | Identifying key differences between related content from different mediums |
US9804729B2 (en) | 2013-03-15 | 2017-10-31 | International Business Machines Corporation | Presenting key differences between related content from different mediums |
US9158435B2 (en) * | 2013-03-15 | 2015-10-13 | International Business Machines Corporation | Synchronizing progress between related content from different mediums |
JP6117621B2 (en) * | 2013-06-05 | 2017-04-19 | ソニー株式会社 | Information processing apparatus, information processing method, and information processing program |
TW201513095A (en) * | 2013-09-23 | 2015-04-01 | Hon Hai Prec Ind Co Ltd | Audio or video files processing system, device and method |
US9213903B1 (en) | 2014-07-07 | 2015-12-15 | Google Inc. | Method and system for cluster-based video monitoring and event categorization |
US9449229B1 (en) | 2014-07-07 | 2016-09-20 | Google Inc. | Systems and methods for categorizing motion event candidates |
US9501915B1 (en) | 2014-07-07 | 2016-11-22 | Google Inc. | Systems and methods for analyzing a video stream |
US10127783B2 (en) | 2014-07-07 | 2018-11-13 | Google Llc | Method and device for processing motion events |
US9082018B1 (en) | 2014-09-30 | 2015-07-14 | Google Inc. | Method and system for retroactively changing a display characteristic of event indicators on an event timeline |
US10140827B2 (en) | 2014-07-07 | 2018-11-27 | Google Llc | Method and system for processing motion event notifications |
CN110072131A (en) | 2014-09-02 | 2019-07-30 | 苹果公司 | Music user interface |
USD782495S1 (en) | 2014-10-07 | 2017-03-28 | Google Inc. | Display screen or portion thereof with graphical user interface |
US9361011B1 (en) | 2015-06-14 | 2016-06-07 | Google Inc. | Methods and systems for presenting multiple live video feeds in a user interface |
US11699266B2 (en) * | 2015-09-02 | 2023-07-11 | Interdigital Ce Patent Holdings, Sas | Method, apparatus and system for facilitating navigation in an extended scene |
US10506237B1 (en) | 2016-05-27 | 2019-12-10 | Google Llc | Methods and devices for dynamic adaptation of encoding bitrate for video streaming |
US10957171B2 (en) | 2016-07-11 | 2021-03-23 | Google Llc | Methods and systems for providing event alerts |
US10380429B2 (en) | 2016-07-11 | 2019-08-13 | Google Llc | Methods and systems for person detection in a video feed |
US10192415B2 (en) | 2016-07-11 | 2019-01-29 | Google Llc | Methods and systems for providing intelligent alerts for events |
US10992795B2 (en) | 2017-05-16 | 2021-04-27 | Apple Inc. | Methods and interfaces for home media control |
US11431836B2 (en) | 2017-05-02 | 2022-08-30 | Apple Inc. | Methods and interfaces for initiating media playback |
US10928980B2 (en) | 2017-05-12 | 2021-02-23 | Apple Inc. | User interfaces for playing and managing audio items |
US20220279063A1 (en) | 2017-05-16 | 2022-09-01 | Apple Inc. | Methods and interfaces for home media control |
CN111343060B (en) | 2017-05-16 | 2022-02-11 | 苹果公司 | Method and interface for home media control |
US10410086B2 (en) | 2017-05-30 | 2019-09-10 | Google Llc | Systems and methods of person recognition in video streams |
US11783010B2 (en) | 2017-05-30 | 2023-10-10 | Google Llc | Systems and methods of person recognition in video streams |
US10664688B2 (en) | 2017-09-20 | 2020-05-26 | Google Llc | Systems and methods of detecting and responding to a visitor to a smart home environment |
US11134227B2 (en) | 2017-09-20 | 2021-09-28 | Google Llc | Systems and methods of presenting appropriate actions for responding to a visitor to a smart home environment |
US11153619B2 (en) | 2018-07-02 | 2021-10-19 | International Business Machines Corporation | Cognitively derived multimedia streaming preferences |
KR20230039775A (en) | 2019-05-31 | 2023-03-21 | 애플 인크. | User interfaces for audio media control |
US11010121B2 (en) | 2019-05-31 | 2021-05-18 | Apple Inc. | User interfaces for audio media control |
US11113229B2 (en) * | 2019-06-03 | 2021-09-07 | International Business Machines Corporation | Providing a continuation point for a user to recommence consuming content |
US11893795B2 (en) | 2019-12-09 | 2024-02-06 | Google Llc | Interacting with visitors of a connected home environment |
US11442609B1 (en) * | 2020-04-08 | 2022-09-13 | Gopro, Inc. | Interface for setting speed and direction of video playback |
US11392291B2 (en) | 2020-09-25 | 2022-07-19 | Apple Inc. | Methods and interfaces for media control with dynamic feedback |
US11823713B1 (en) * | 2022-10-03 | 2023-11-21 | Bolt-On Ip Solutions, Llc | System and method for editing an audio stream |
Family Cites Families (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5136655A (en) | 1990-03-26 | 1992-08-04 | Hewlett-Pacard Company | Method and apparatus for indexing and retrieving audio-video data |
US5579471A (en) * | 1992-11-09 | 1996-11-26 | International Business Machines Corporation | Image query system and method |
US5655058A (en) | 1994-04-12 | 1997-08-05 | Xerox Corporation | Segmentation of audio data for indexing of conversational speech for real-time or postprocessing applications |
US5493677A (en) * | 1994-06-08 | 1996-02-20 | Systems Research & Applications Corporation | Generation, archiving, and retrieval of digital images with evoked suggestion-set captions and natural language interface |
US5664227A (en) | 1994-10-14 | 1997-09-02 | Carnegie Mellon University | System and method for skimming digital audio/video data |
US5708767A (en) | 1995-02-03 | 1998-01-13 | The Trustees Of Princeton University | Method and apparatus for video browsing based on content and structure |
US5729471A (en) * | 1995-03-31 | 1998-03-17 | The Regents Of The University Of California | Machine dynamic selection of one video camera/image of a scene from multiple video cameras/images of the scene in accordance with a particular perspective on the scene, an object in the scene, or an event in the scene |
US5987459A (en) * | 1996-03-15 | 1999-11-16 | Regents Of The University Of Minnesota | Image and document management system for content-based retrieval |
US5893110A (en) * | 1996-08-16 | 1999-04-06 | Silicon Graphics, Inc. | Browser driven user interface to a media asset database |
US6055543A (en) * | 1997-11-21 | 2000-04-25 | Verano | File wrapper containing cataloging information for content searching across multiple platforms |
-
1998
- 1998-09-11 US US09/151,285 patent/US6366296B1/en not_active Expired - Lifetime
-
1999
- 1999-08-24 JP JP23638199A patent/JP2000090121A/en active Pending
-
2001
- 2001-08-23 US US09/934,486 patent/US20020054083A1/en not_active Abandoned
Cited By (64)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8995767B2 (en) | 1997-12-22 | 2015-03-31 | Ricoh Company, Ltd. | Multimedia visualization and integration environment |
US20040175036A1 (en) * | 1997-12-22 | 2004-09-09 | Ricoh Company, Ltd. | Multimedia visualization and integration environment |
US20040103372A1 (en) * | 1997-12-22 | 2004-05-27 | Ricoh Company, Ltd. | Multimedia visualization and integration environment |
US8739040B2 (en) * | 1997-12-22 | 2014-05-27 | Ricoh Company, Ltd. | Multimedia visualization and integration environment |
US6782049B1 (en) * | 1999-01-29 | 2004-08-24 | Hewlett-Packard Development Company, L.P. | System for selecting a keyframe to represent a video |
US7643657B2 (en) | 1999-01-29 | 2010-01-05 | Hewlett-Packard Development Company, L.P. | System for selecting a keyframe to represent a video |
US20050002452A1 (en) * | 1999-01-29 | 2005-01-06 | Frederic Dufaux | System for selecting a keyframe to represent a video |
US20110093884A1 (en) * | 2000-01-14 | 2011-04-21 | Nds Limited. | Advertisements in an end-user controlled playback environment |
US8732750B2 (en) * | 2000-01-14 | 2014-05-20 | Cisco Technology, Inc. | Advertisements in an end-user controlled playback environment |
US20080306921A1 (en) * | 2000-01-31 | 2008-12-11 | Kenneth Rothmuller | Digital Media Management Apparatus and Methods |
US8229931B2 (en) * | 2000-01-31 | 2012-07-24 | Adobe Systems Incorporated | Digital media management apparatus and methods |
USRE38609E1 (en) * | 2000-02-28 | 2004-10-05 | Webex Communications, Inc. | On-demand presentation graphical user interface |
US20150082338A1 (en) * | 2000-03-28 | 2015-03-19 | Compass Innovations, LLC | Audiovisual Content Presentation Dependent On Metadata |
US10313714B2 (en) * | 2000-03-28 | 2019-06-04 | Tivo Solutions Inc. | Audiovisual content presentation dependent on metadata |
US6711587B1 (en) | 2000-09-05 | 2004-03-23 | Hewlett-Packard Development Company, L.P. | Keyframe selection to represent a video |
EP1376584A3 (en) * | 2002-06-19 | 2004-12-22 | Microsoft Corporation | System and method for automatically generating video cliplets from digital video |
US8238718B2 (en) | 2002-06-19 | 2012-08-07 | Microsoft Corporaton | System and method for automatically generating video cliplets from digital video |
US20070168413A1 (en) * | 2003-12-05 | 2007-07-19 | Sony Deutschland Gmbh | Visualization and control techniques for multimedia digital content |
US8209623B2 (en) | 2003-12-05 | 2012-06-26 | Sony Deutschland Gmbh | Visualization and control techniques for multimedia digital content |
US7672864B2 (en) | 2004-01-09 | 2010-03-02 | Ricoh Company Ltd. | Generating and displaying level-of-interest values |
US7079142B2 (en) | 2004-01-14 | 2006-07-18 | Fuji Xerox Co., Ltd. | Systems and methods for creating and updating an interactive 3D visualization of media indices |
US20050151737A1 (en) * | 2004-01-14 | 2005-07-14 | Fuji Xerox Co., Ltd | Systems and methods for creating and updating an interactive 3D visualization of media indices |
US7340678B2 (en) | 2004-02-12 | 2008-03-04 | Fuji Xerox Co., Ltd. | Systems and methods for creating an interactive 3D visualization of indexed media |
US20080104546A1 (en) * | 2004-02-12 | 2008-05-01 | Fuji Xerox Co., Ltd. | Systems and methods for creating an interactive 3d visualization of indexed media |
US20050183041A1 (en) * | 2004-02-12 | 2005-08-18 | Fuji Xerox Co., Ltd. | Systems and methods for creating and interactive 3D visualization of indexed media |
US7802188B2 (en) * | 2004-05-13 | 2010-09-21 | Hewlett-Packard Development Company, L.P. | Method and apparatus for identifying selected portions of a video stream |
US20050257151A1 (en) * | 2004-05-13 | 2005-11-17 | Peng Wu | Method and apparatus for identifying selected portions of a video stream |
US7296021B2 (en) | 2004-05-21 | 2007-11-13 | International Business Machines Corporation | Method, system, and article to specify compound query, displaying visual indication includes a series of graphical bars specify weight relevance, ordered segments of unique colors where each segment length indicative of the extent of match of each object with one of search parameters |
US7853582B2 (en) * | 2004-08-31 | 2010-12-14 | Gopalakrishnan Kumar C | Method and system for providing information services related to multimodal inputs |
US9639633B2 (en) | 2004-08-31 | 2017-05-02 | Intel Corporation | Providing information services related to multimodal inputs |
US20110092251A1 (en) * | 2004-08-31 | 2011-04-21 | Gopalakrishnan Kumar C | Providing Search Results from Visual Imagery |
US20110093264A1 (en) * | 2004-08-31 | 2011-04-21 | Kumar Gopalakrishnan | Providing Information Services Related to Multimodal Inputs |
US20060218192A1 (en) * | 2004-08-31 | 2006-09-28 | Gopalakrishnan Kumar C | Method and System for Providing Information Services Related to Multimodal Inputs |
US8370323B2 (en) | 2004-08-31 | 2013-02-05 | Intel Corporation | Providing information services related to multimodal inputs |
US20090080777A1 (en) * | 2005-07-01 | 2009-03-26 | International Business Machines Corporation | Methods and Apparatus for Filtering Video Packets for Large-Scale Video Stream Monitoring |
US7912284B2 (en) * | 2005-07-01 | 2011-03-22 | International Business Machines Corporation | Methods and apparatus for filtering video packets for large-scale video stream monitoring |
US20070286489A1 (en) * | 2005-07-01 | 2007-12-13 | International Business Machines Corporation | Methods and apparatus for filtering video packets for large-scale video stream monitoring |
US7545978B2 (en) * | 2005-07-01 | 2009-06-09 | International Business Machines Corporation | Methods and apparatus for filtering video packets for large-scale video stream monitoring |
US20140324907A1 (en) * | 2005-11-29 | 2014-10-30 | Aol Inc. | Visually-Represented Results To Search Queries In Rich Media Content |
US20090157743A1 (en) * | 2007-12-12 | 2009-06-18 | Aten International Co., Ltd. | Media file generation method and media playback method |
US20090164904A1 (en) * | 2007-12-21 | 2009-06-25 | Yahoo! Inc. | Blog-Based Video Summarization |
US9535988B2 (en) * | 2007-12-21 | 2017-01-03 | Yahoo! Inc. | Blog-based video summarization |
US20100303365A1 (en) * | 2009-05-29 | 2010-12-02 | Min Zhang | Methods and apparatus to monitor a multimedia presentation including multiple content windows |
US20110035700A1 (en) * | 2009-08-05 | 2011-02-10 | Brian Meaney | Multi-Operation User Interface Tool |
US20130166303A1 (en) * | 2009-11-13 | 2013-06-27 | Adobe Systems Incorporated | Accessing media data using metadata repository |
US11082722B2 (en) * | 2011-01-26 | 2021-08-03 | Afterlive.tv Inc. | Method and system for generating highlights from scored data streams |
US9691068B1 (en) * | 2011-12-15 | 2017-06-27 | Amazon Technologies, Inc. | Public-domain analyzer |
US9846696B2 (en) | 2012-02-29 | 2017-12-19 | Telefonaktiebolaget Lm Ericsson (Publ) | Apparatus and methods for indexing multimedia content |
US11789992B2 (en) * | 2012-04-27 | 2023-10-17 | Tivo Corporation | Search-based navigation of media content |
US20130290846A1 (en) * | 2012-04-27 | 2013-10-31 | Mobitv, Inc. | Search-based navigation of media content |
US9785639B2 (en) * | 2012-04-27 | 2017-10-10 | Mobitv, Inc. | Search-based navigation of media content |
US10628477B2 (en) | 2012-04-27 | 2020-04-21 | Mobitv, Inc. | Search-based navigation of media content |
US9633015B2 (en) * | 2012-07-26 | 2017-04-25 | Telefonaktiebolaget Lm Ericsson (Publ) | Apparatus and methods for user generated content indexing |
US20140032562A1 (en) * | 2012-07-26 | 2014-01-30 | Telefonaktiebolaget Lm Ericsson (Publ) | Apparatus and methods for user generated content indexing |
US10445367B2 (en) | 2013-05-14 | 2019-10-15 | Telefonaktiebolaget Lm Ericsson (Publ) | Search engine for textual content and non-textual content |
CN103399865A (en) * | 2013-07-05 | 2013-11-20 | 华为技术有限公司 | Method and device for multi-media file generation |
US10311038B2 (en) | 2013-08-29 | 2019-06-04 | Telefonaktiebolaget Lm Ericsson (Publ) | Methods, computer program, computer program product and indexing systems for indexing or updating index |
US10289810B2 (en) | 2013-08-29 | 2019-05-14 | Telefonaktiebolaget Lm Ericsson (Publ) | Method, content owner device, computer program, and computer program product for distributing content items to authorized users |
US9110869B2 (en) * | 2013-09-27 | 2015-08-18 | Nokia Technologies Oy | Visual representation of a character identity and a location identity |
US20150095749A1 (en) * | 2013-09-27 | 2015-04-02 | Nokia Corporation | Visual Representation of a Character Identity and a Location Identity |
US20230009515A1 (en) * | 2021-07-06 | 2023-01-12 | Rovi Guides, Inc. | Generating verified content profiles for user generated content |
US11615139B2 (en) * | 2021-07-06 | 2023-03-28 | Rovi Guides, Inc. | Generating verified content profiles for user generated content |
US20230222164A1 (en) * | 2022-01-11 | 2023-07-13 | Kyndryl, Inc. | Autonomous webpage content summation |
US11860946B2 (en) * | 2022-01-11 | 2024-01-02 | Kyndryl, Inc. | Autonomous webpage content summation |
Also Published As
Publication number | Publication date |
---|---|
US6366296B1 (en) | 2002-04-02 |
JP2000090121A (en) | 2000-03-31 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6366296B1 (en) | Media browser using multimodal analysis | |
US10031649B2 (en) | Automated content detection, analysis, visual synthesis and repurposing | |
US7046914B2 (en) | Automatic content analysis and representation of multimedia presentations | |
US7594177B2 (en) | System and method for video browsing using a cluster index | |
US5805733A (en) | Method and system for detecting scenes and summarizing video sequences | |
US7945142B2 (en) | Audio/visual editing tool | |
US6571054B1 (en) | Method for creating and utilizing electronic image book and recording medium having recorded therein a program for implementing the method | |
CN112740713B (en) | Method for providing key time in multimedia content and electronic device thereof | |
US5819103A (en) | Information recording/reproducing apparatus and method | |
US7702014B1 (en) | System and method for video production | |
US9100616B2 (en) | Relational display of images | |
US20050257151A1 (en) | Method and apparatus for identifying selected portions of a video stream | |
US20090158323A1 (en) | Method and apparatus for video navigation | |
US7099894B2 (en) | Multimedia information collection control apparatus and method | |
KR20140139859A (en) | Method and apparatus for user interface for multimedia content search | |
KR101440168B1 (en) | Method for creating a new summary of an audiovisual document that already includes a summary and reports and a receiver that can implement said method | |
JP3579111B2 (en) | Information processing equipment | |
US20040056881A1 (en) | Image retrieval system | |
Girgensohn et al. | Facilitating Video Access by Visualizing Automatic Analysis. | |
AU3724497A (en) | Digital video system having a data base of coded data for digital audio and ideo information | |
JPH0520367A (en) | Index generating system | |
JP2004234683A (en) | Information processor and information processing method | |
Budlong et al. | Multimedia Information Extraction | |
Smith et al. | Visualization Techniques |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: BANK ONE, NA, AS ADMINISTRATIVE AGENT, ILLINOIS Free format text: SECURITY AGREEMENT;ASSIGNOR:XEROX CORPORATION;REEL/FRAME:013111/0001 Effective date: 20020621 Owner name: BANK ONE, NA, AS ADMINISTRATIVE AGENT,ILLINOIS Free format text: SECURITY AGREEMENT;ASSIGNOR:XEROX CORPORATION;REEL/FRAME:013111/0001 Effective date: 20020621 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |
|
AS | Assignment |
Owner name: NATIONAL INSTITUTES OF HEALTH (NIH), U.S. DEPT. OF Free format text: CONFIRMATORY LICENSE;ASSIGNOR:SALK INSTITUTE FOR BIOLOGICAL STUDIES;REEL/FRAME:024793/0769 Effective date: 20010608 |
|
AS | Assignment |
Owner name: XEROX CORPORATION, CONNECTICUT Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:JPMORGAN CHASE BANK, N.A. AS SUCCESSOR-IN-INTEREST ADMINISTRATIVE AGENT AND COLLATERAL AGENT TO BANK ONE, N.A.;REEL/FRAME:061388/0388 Effective date: 20220822 |