« PreviousContinue »
TIMELINE CORRELATION BETWEEN
MULTIPLE TIMELINE-ALTERED MEDIA
TECHNICAL FIELD 5
This invention relates to techniques of storing and rendering streaming multimedia content at different speeds, and for correlating timelines of media streams that have been timeline-altered by varying techniques and degrees.
BACKGROUND OF THE INVENTION 10
Multimedia streaming-the continuous delivery of synchronized media data like video, audio, text, and animationis a critical link in the digital multimedia revolution. Today, streamed media is primarily about video and audio, but a ^ richer, broader digital media era is emerging with a profound and growing impact on the Internet and digital broadcasting.
Synchronized media means multiple media objects that share a common timeline. Video and audio are examples of synchronized media—each is a separate data stream with its 20 own data structure, but the two data streams are played back in synchronization with each other. Virtually any media type can have a timeline. For example, an image object can change like an animated .gif file: text can change and move, and animation and digital effects happen over time. This 25 concept of synchronizing multiple media types is gaining greater meaning and currency with the emergence of more sophisticated media composition frameworks implied by MPEG-4, Dynamic HTML, and other media playback environments. The term "streaming" is used to indicate that the 30 data representing the various media types is provided over a network to a client computer on a realtime, as-needed basis, rather than being pre-delivered in its entirety before playback. Thus, the client computer renders streaming data as it is received from a network server, rather than waiting for an 35 entire "file" to be delivered.
The widespread availability of streaming multimedia enables a variety of informational content that was not previously available over the Internet or other computer networks. Live content is one significant example of such 40 content. Using streaming multimedia, audio, video, or audio/visual coverage of noteworthy events can be broadcast over the Internet as the events unfold. Similarly, television and radio stations can transmit their live content over the Internet. 45
A U.S. Patent Application entitled "Multimedia Timeline Modification in Networked Client/Server Systems," filed Sep. 15, 1998, Ser. No. 09/153,664, by inventors Anoop Gupta and Nosa D. Omoigui, describes a system that allows a user to vary the playback speed of streaming multimedia 50 content using time-scale modification technology. It was noted that both linear and non-linear timeline alteration techniques might be used.
When using linear techniques, time modification is applied consistently in times and across all individual 55 streams of a composite media stream. With non-linear techniques, on the other hand, some segments of an individual or composite stream might be more highly expanded or compressed (in time) than other segments. This presents problems in switching and synchronizing between different 60 versions of streams that have been non-linearly altered by different amounts or through the use of different non-linear techniques. These problems are addressed by the invention.
SUMMARY OF THE INVENTION 65
The invention utilizes time-scale modification so that a user can vary the speed of streaming content without
destroying its intelligibility. In accordance with the invention, a user selects multimedia content from a menu presented at a network client computer. In addition, the user selects a playback speed, indicating the speed at which the multimedia should be rendered relative to its default speed. In the case of non-linearly altered streams, the user selects from available levels of alteration, such as "normal," "pauses removed," and "summary" levels of non-linear time compression.
Multiple versions of the requested multimedia content are stored at a server. In response to the selected playback speed or selected level of non-linear compression, the appropriate version is streamed to the client computer.
In addition to storing multiple versions of the multimedia content, the server maintains data structures that indicate timeline correlations between the various stored versions. The client refers to these stored timeline correlations when changing between the versions in response to client requests, so that playback can be started anew with a new stream at the point in the content where the change was requested by the client.
In the embodiment described herein, one version of the multimedia stream is considered to be a primary or reference version. This is usually the version having an unaltered timeline, corresponding to a 1.0 speed factor. Associated with this primary media stream is a table or other data object that is indexed by the presentation times of the primary media stream. For each presentation time of the primary media stream, the table indicates the timeline-correlated presentation times of the other, timeline-altered, media streams.
In addition, further data objects or elements are provided in conjunction with the timeline-altered media streams. These data objects are indexed by the presentation times of the timeline-altered media stream, and map such presentation times to the time-correlated presentation times of the primary media stream. One way to index these data objects is to associate and send them with the actual data units of the time-altered media streams.
When switching playback from a first time-altered media stream to a second time-altered media stream, the client stops the first media stream and notes the presentation time (in the first media stream) at which playback was stopped. The client then refers to the table associated with the first media stream and from it determines the timeline-correlated presentation time in the primary media stream. With that information, the client refers to the table associated with the primary media stream, and determines the time-correlated presentation time in the second media stream. The client then initiates playback of the second media stream at or before this time-correlated presentation time.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block diagram of a networked client/server system in accordance with the invention.
FIG. 2 is a block diagram of a networked computer that can be used to implement either a server or a client in accordance with the invention.
FIG. 3 illustrates the flow of a composite media stream in accordance with one embodiment of the invention.
FIG. 4 illustrates timeline correlations between the timelines of the primary and timeline-altered media streams in accordance with one embodiment of the invention.
FIG. 5 illustrates the process used to find an appropriate presentation time in a second timeline-altered media stream,
when switching from a first timeline-altered media stream to the second timeline-altered media stream in accordance with one embodiment of the invention.
General Network Structure
FIG. 1 shows a client/server network system and environment in accordance with the invention. Generally, the system includes a network server computer 10 and a plurality of network client computers 11. The computers communicate with each other over a data communications network. The communications network in FIG. 1 comprises a public network 12 such as the Internet. The data communications network might also include local-area networks and private wide-area networks.
Server computer 10 has access to streaming media content in the form of different composite media streams. Some composite media streams might be stored as files in a database or other file storage system 13. Other composite media streams might be supplied to the server on a "live" basis from other data source components through dedicated communications channels or through the Internet itself.
In this discussion, the term "composite media stream" describes synchronized streaming data that represents a segment of multimedia content. The composite media stream has a timeline that establishes the speed at which the content is rendered. The composite media stream can be rendered to produce a plurality of different types of userperceivable media, including synchronized audio or sound, video graphics or motion pictures, animation, textual content, command script sequences, or other media types that convey time-varying information or content in a way that can be sensed and perceived by a human. A composite media stream comprises a plurality of individual media streams representing the multimedia content. Each of the individual media streams corresponds to and represents a different media type and each of the media streams can be rendered by a network client to produce a user-perceivable presentation using a particular presentation medium. The individual media streams have their own timelines, which are synchronized with each other so that the media streams can be rendered simultaneously for a coordinated multimedia presentation. The individual timelines define the timeline of the composite stream.
There are various standards for streaming media content and composite media streams. The "Advanced Streaming Format" (ASF) is an example of such a standard, including both accepted versions of the standard and proposed standards for future adoption. ASF specifies the way in which multimedia content is stored, streamed, and presented by the tools, servers, and clients of various multimedia vendors. ASF provides benefits such as local and network playback, extensible media types, component download, scalable media types, prioritization of streams, multiple language support, environment independence, rich inter-stream relationships, and expandability. Further details about ASF are available from Microsoft Corporation of Redmond, Washington.
Regardless of the streaming format used, an individual data stream contains a sequence of digital data sets or units that are rendered individually, in sequence, to produce an image, sound, or some other stimuli that is perceived by a human to be continuously varying. For example, an audio
data stream comprises a sequence of sample values that are converted to a pitch and volume to produce continuously varying sound. A video data stream comprises a sequence of digitally-specified graphics frames that are rendered in
5 sequence to produce a moving picture.
Typically, the individual data units of a composite media stream are interleaved in a single sequence of data packets. Various types of data compression might be used within a particular data format to reduce communications bandwidth
The sequential data units (such as audio sample values or video frames) are associated with both delivery times and presentation times, relative to an arbitrary start time. The delivery time of a data unit indicates when the data unit
15 should be delivered to a rendering client. The presentation time indicates when the value should be actually rendered. Normally, the delivery time of a data unit precedes its presentation time.
The presentation times determine the actual speed of
20 playback. For data streams representing actual events or performances, the presentation times correspond to the relative times at which the data samples were actually recorded. The presentation times of the various different individual data streams are consistent with each other so that the
streams remain coordinated and synchronized during playback.
Exemplary Computer Environment
3Q In the discussion below, the invention will be described in the general context of computer-executable instructions, such as program modules, being executed by one or more conventional personal computers. Generally, program modules include routines, programs, objects, components, data
35 structures, etc. that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or
40 programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. In a distributed computer environment, program modules may be located in both local and remote memory storage devices. FIG. 2 shows a general example of a computer 20 that can
45 be used as a network node or host in accordance with the invention. Computer 20 is shown as an example of a computer that can perform the functions of either server computer 10 or a client computer 11.
Computer 20 includes one or more processors or process
50 ing units 21, a system memory 22, and a system bus 23 that couples various system components including the system memory to processors 21.
The system bus 23 may be any of several types of bus structures including a memory bus or memory controller, a
55 peripheral bus, and a local bus using any of a variety of bus architectures. The system memory includes read only memory (ROM) 24 and random access memory (RAM) 25. A basic input/output system 26 (BIOS), containing the basic routines that help to transfer information between elements
60 within server computer 20, such as during start-up, is stored in ROM 24. Computer 20 further includes a hard disk drive 27 for reading from and writing to a hard disk, not shown, a magnetic disk drive 28 for reading from or writing to a removable magnetic disk 29, and an optical disk drive 30 for
65 reading from or writing to a removable optical disk 31 such as a CD ROM or other optical media. The hard disk drive 27, magnetic disk drive 28, and optical disk drive 30 are