Method and system for detecting scenes and summarizing video sequences

1. A method of summarizing a temporally ordered plurality of scenes in a video sequence including a plurality of frames, comprising the steps of:

detecting at least one scene in the plurality of frames, each scene including at least one related frame by:

determining for each frame at least one global measurement; and

determining a scene change between a pair of successive frames from a difference between the global measurements of each frame in the pair of frames, a Chi-squared value on a color histogram of each frame in the pair of frames, and a normalized motion compensated pixel difference between the frames;

determining at least one set of related scenes;

determining a representative frame for each set of related scenes; and

displaying each representative frame.

2. The method of claim 1, wherein the determination of the scene change is performed using a neural network receiving the difference between the global measurements, the Chi-squared value, and the normalized motion compensated pixel difference.

3. The method of claim 1, wherein the step of detecting at least one scene further comprises the steps of:

for each frame of the video sequence, beginning with a first frame, determining a motion compensated pixel luminance difference between a current frame and a next frame; and

responsive to the motion compensated pixel luminance difference being greater than a threshold value, associating the first frame with all frames up to and including the current frame to form a scene.

4. The method of claim 3, further comprising the step of:

equalizing a color histogram of each frame of the video sequence prior to determining the motion compensated pixel luminance difference.

5. The method of claim 1, wherein the step of detecting at least one scene further comprises the steps of:

for each frame of the video sequence, beginning with a first frame, determining a motion compensated edge map difference between a current frame and a next frame; and

responsive to the motion compensated edge map difference being greater than a threshold value, associating the first frame with all frames up to and including the current frame to form a scene.

6. The method of claim 1, wherein the step of detecting at least one scene further comprises the steps of:

for each frame of the video sequence, beginning with a first frame, determining a standard error of differences between a color histogram for a current frame and a color histogram of a next frame; and

responsive to the standard error being greater than a threshold value, associating the first frame with all frames up to and including the current frame to form a scene.

7. The method of claim 9, wherein the standard error is measured using a Chi-squared test.

8. The method of claim 1, wherein the step determining at least one set of related scenes further comprises the steps of:

determining for each scene a summary measure derived from all frames in the scene;

comparing the summary measure of each scene with the summary measures of selected other scenes using a standard error measure; and

identifying at least one set of scenes having approximately equal summary measures.

9. The method of claim 8 wherein the summary measure is an average color histogram for all frames in the scene.

10. The method of claim 8, further comprising the step of:

linking scenes in each group of scenes into a consolidated scene for display as a single continuous scene.

11. The method of claim 8, wherein the representative frame for each set of related scenes is a frame temporally medial to a first frame and a last frame in the set of related scenes.

12. The method of claim 8, wherein the representative frame for each set of related scenes is selected from a longest scene in the set of related scenes.

13. A method of removing redundant scenes in a plurality of frames, each scene including at least one frame, the method comprising the steps of:

detecting each scene in the plurality of frames, each scene including at least one related frame by:

determining for each frame at least one global measurement; and

determining a scene change between a pair of successive frames from a difference between the global measurements of each frame in the pair of frames, a Chi-squared value on a color histogram of each frame in the pair of frames, and a normalized motion compensated pixel difference between the frames;

determining for each scene, an average color histogram for all frames included the scene; and

comparing the average color histogram of each scene with the average color histogram of each of a selected number of subsequent scenes to identify scenes within the selected number of scenes having substantially similar average color histograms, each set of scenes having substantially similar average color histograms forming a set of related scenes.

14. The method of claim 13 wherein the step of comparing is performing by determining a Chi squared value for the average color histograms of each pair of scenes.

15. A method of removing redundant scenes in a plurality of frames, each scene including at least one frame, the method comprising the steps of:

detecting each scene in the plurality of frames, each scene including at least one related frame by:

determining for each frame at least one global measurement; and

determining a scene change between a pair of successive frames from a difference between the global measurements of each frame in the pair of frames, a Chi-squared value on a color histogram of each frame in the pair of frames, and a normalized motion compensated pixel difference between the frames;

determining for each scene a summary measure derived from all frames in the scene; and

comparing the summary measure of each scene with the summary measure of selected subsequent scenes using a standard error measure to identify at least one group of scenes having substantially similar summary measures.

16. The method of claim 15 wherein the summary measure is an average color histogram.

17. The method of claim 15, wherein the selected other scenes consists of a predetermined number of subsequent scenes.

18. The method of claim 15, further comprising the step of:

linking scenes in each group of scenes into a consolidated scene.

19. The method of claim 15, wherein the representative frame for each set of related scenes a frame temporally medial to a first frame and a last frame in the set of related scenes.

20. The method of claim 1, further comprising the steps of:

displaying a plurality of slices of frames as a rectangular prism; and

highlighting in the plurality of slices, the slices of frames either a first frame or a last frame of each scene.

21. The method of claim 20, further comprising the step of

retrieving in response to user selection of any frame, the scene including the selected frame.

22. The method of claim 20, further comprising the steps of:

determining at least one scene having a time duration in excess of a threshold;

displaying for the scene, and within the rectangular prism at the location of the scene, a compressed representation of all frames in the scene; and

responsive to a user selection of the compressed representation, displaying a plurality of slices for all frames in the scene.

23. A system for summarizing a temporally ordered plurality of scenes in a video sequence including a plurality of frames, comprising:

an input device for receiving an input video sequence;

a scene change detector coupled to the input device, the scene change detector detecting at least one scene in the plurality of frames, each scene including at least one related frame, the scene change detector detecting a scene change between a pair of successive frames from a difference between a global measurement of each frame in the pair of frames, a Chi-squared value on a color histogram of each frame in the pair of frames, and a normalized motion compensated pixel difference between the frames;

a related scene detector, coupled to the scene change detector to receive a set of scenes, and determining at least one set of related scenes, the related scene detector determining a representative frame for each set of related scenes;

a display for displaying the representative frame, and selected scenes; and

a user interface display driver coupled to the related scene detector to receive the representative frames, and displaying each representative frame on the display.

24. The system of claim 23, further comprising:

a movie bar generator coupled to receive the scenes from the scene change detector, and the related scenes from the related scene detector, the movie bar generator creating a slice of each frames in each scene, and displaying the slices as rectangular prism.