US20110167174A1 - Method and System for In-Band Signaling of Multiple Media Streams - Google Patents

Method and System for In-Band Signaling of Multiple Media Streams Download PDF

Info

Publication number
US20110167174A1
US20110167174A1 US12/987,535 US98753511A US2011167174A1 US 20110167174 A1 US20110167174 A1 US 20110167174A1 US 98753511 A US98753511 A US 98753511A US 2011167174 A1 US2011167174 A1 US 2011167174A1
Authority
US
United States
Prior art keywords
source
streams
stream
destination
media
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US12/987,535
Other versions
US8208460B2 (en
Inventor
James Tighe
Frank Roland Zerangue, Jr.
Rowan McFarland
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Cisco Technology Inc
Original Assignee
Cisco Technology Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Cisco Technology Inc filed Critical Cisco Technology Inc
Priority to US12/987,535 priority Critical patent/US8208460B2/en
Publication of US20110167174A1 publication Critical patent/US20110167174A1/en
Application granted granted Critical
Publication of US8208460B2 publication Critical patent/US8208460B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/434Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
    • H04N21/4341Demultiplexing of audio and video streams
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L25/00Baseband systems
    • H04L25/02Details ; arrangements for supplying electrical power along data transmission lines
    • H04L25/14Channel dividing arrangements, i.e. in which a single bit stream is divided between several baseband channels and reassembled at the receiver
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/1066Session management
    • H04L65/1101Session protocols
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60Network streaming of media packets
    • H04L65/70Media network packetisation
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/80Responding to QoS
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H04N21/2368Multiplexing of audio and video streams
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/242Synchronization processes, e.g. processing of PCR [Program Clock References]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/4302Content synchronisation processes, e.g. decoder synchronisation
    • H04N21/4305Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/4302Content synchronisation processes, e.g. decoder synchronisation
    • H04N21/4307Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
    • H04N21/43072Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen of multiple content streams on the same device
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/631Multimode Transmission, e.g. transmitting basic layers and enhancement layers of the content over different transmission paths or transmitting with different error corrections, different keys or with different transmission protocols

Definitions

  • Embodiments of the invention relate in general to transmission of multiple media streams. More specifically, the embodiments of the invention relate to methods and systems enabling in-band signaling of multiple media streams in a network.
  • Transmission of two or more real-time media streams, for example, video and audio, in a network requires the aggregation of multiple digital voice channels, for example 56 Kbit, 64 Kbit digital voice channels. This aggregation is required to provide sufficient bandwidth—typically a minimum of 128 Kbit for video. Each channel may be routed between a source and a destination through a different network path, leading to varying channel latencies.
  • the multiple streams may also develop a varying relationship with each other, due to differences in the processing speed of network devices. Therefore, the multiple streams have to be synchronized at the time of playback.
  • IP Internet Protocol
  • the out-of-band protocol requires an additional signal channel alongside the data transmission channel, to carry signal messages for synchronization.
  • a scheme may be susceptible to problems arising due to the possible loss of signaling messages.
  • Another scheme uses the concept of ‘timestamps’.
  • This scheme requires either distributed ‘virtual wall clocks’ or forms of virtual ‘Round Trip Timestamps (RTTs)’ across an unknown and ever-changing network.
  • RTTs can drift, relative to each other, during a stream session, due to changing network dynamics, and hence may be inaccurate. If multiple related streams, such as audio, video, slides and mouse movements are used, then the complexity of using timestamps increases.
  • this scheme requires a connection to a dedicated timeserver, in order to maintain the virtual wall clocks.
  • FIG. 1 illustrates an environment for in-band signaling of multiple media streams wherein various embodiments of the invention can be practiced.
  • FIG. 2 illustrates a source, in accordance with an exemplary embodiment of the invention.
  • FIG. 3 illustrates a destination, in accordance with an exemplary embodiment of the invention.
  • FIG. 4 is a flowchart illustrating a method for in-band signaling of at least two simultaneous digital media streams in a network, in accordance with various embodiments of the invention.
  • FIG. 5 is a flowchart illustrating a method for synchronizing at least two simultaneous streams, in accordance with an exemplary embodiment of the invention.
  • FIG. 6 is a flowchart illustrating a method for re-synchronizing the synchronized streams, according to an exemplary embodiment of the invention.
  • FIG. 7 is a flowchart illustrating a method for re-synchronizing the synchronized streams, according to an exemplary embodiment of the invention.
  • Various embodiments of the invention provide methods, systems, and computer-readable media for in-band signaling of at least two simultaneous digital media streams in a network.
  • the at least two simultaneous digital streams may relate to a media session and the streams may be generated from multiple sources.
  • Various embodiments of the invention perform a temporal synchronization between ‘N’ simultaneous digital media streams relating to the media session.
  • the synchronization is performed by embedding unique marker packets within each of the N simultaneous digital media streams.
  • the unique marker packets are embedded simultaneously into each of the N simultaneous streams at regular intervals of time.
  • the time is determined by configuring a source Virtual Wall Clock (VWC), which may be configured by synchronizing separate source clocks within each of the multiple sources.
  • VWC Virtual Wall Clock
  • the synchronized streams are transmitted to multiple destinations; wherein, each of the multiple destinations corresponds to each of the multiple sources.
  • the previously synchronized streams are re-synchronized, based on the embedded unique marker packets, and a destination VWC.
  • the destination VWC is configured by synchronizing separate destination clocks on each of the corresponding destinations. After the re-synchronization, the streams are played back.
  • the N simultaneous digital media streams may be generated simultaneously from N number of related stream-generating devices.
  • the N related stream-generating devices may be N separate Data Signaling Processors (DSPs) in the same videophone.
  • DSPs Data Signaling Processors
  • the N related stream-generating devices may be in physically separate sources. For example, for a media session with N equal to two, the related stream-generating devices may be included in an Internet Protocol (IP) phone for audio generation and in a Personal Computer (PC) for video generation. Further, these devices may be connected, for example, through a direct wire connection (802.1).
  • IP Internet Protocol
  • PC Personal Computer
  • the related stream-generating devices have a ‘relationship’, and a ‘signaling path’ with each other.
  • the relationship is a temporal one, which is developed between the simultaneous streams generated by the related stream-generating devices by embedding unique marker packets within each of the simultaneous streams.
  • the unique marker packets are embedded within each of the related simultaneous streams, at the same instance.
  • the signaling path is the protocol capability between the related stream-generating devices, which generate the simultaneous streams. Examples of protocols include TCP.
  • the signaling path has deterministic latencies with small, if any, variance. Considering the above example, when the IP phone is activated, it starts communicating with the related video-generating device, which is the PC, to determine a communication delay between them. Thereafter, as the related audio and video streams are generated, the unique marker packets are inserted, at appropriate locations, into the RTP streams for synchronization of the streams.
  • FIG. 1 illustrates an environment for in-band signaling of multiple media streams wherein various embodiments of the invention can be practiced.
  • the environment includes sources 102 such as sources 102 a, 102 b and 102 c; a network device 104 ; destinations 106 , such as destinations 106 a, 106 b, and 106 c.
  • Sources 102 , network device 104 and destinations 106 are connected through a network.
  • the network may be, for example, a Wide Area Network (WAN) or a Local Area Network (LAN), a client-server network, a network with multitier architecture, or a peer-to-peer network.
  • Each of sources 102 comprises at least one of the related stream-generating devices, which generate digital media streams corresponding to a media session.
  • sources 102 generate and synchronize ‘N’ simultaneous media streams corresponding to the media session
  • destinations 106 receive the ‘N’ synchronized digital media streams.
  • the synchronized media streams may include streams generated from each of the following: a video, an audio, a slide, and a mouse movement.
  • destinations 106 a, 106 b, and 106 c may be separate DSPs in one media-rendering device.
  • destinations 106 a, 106 b, and 106 c may be separate DSPs in separate media-rendering devices.
  • there is the second signaling path between destinations 106 there is the second signaling path between destinations 106 .
  • sources 102 and destinations 106 can be a data-processing unit.
  • a data-processing unit include, but are not limited to, a mainframe computer, a supercomputer, an enterprise application server, a workstation, an embedded system, a videophone, an Internet Protocol (IP) phone, and a personal computer.
  • the personal computer can be, for example, a desktop computer, a laptop computer, a tablet computer, a Personal Digital Assistant (PDA), a Personal Video Recorder, and a wearable computer.
  • Network device 104 can be a network interface between sources 102 and destinations 106 .
  • the network interface between sources 102 and destinations 106 may be of different types.
  • Examples of network device 104 include a mixer, a bridge, a firewall, a gateway, a router, an ISDN terminal adapter, a hub, a switch, a multilayer switch, a line driver, a modem, a multiplexer, a network interface card, a protocol converter, a proxy, and so forth.
  • Network device 104 receives related streams and then outputs the related streams to destinations 106 at a fixed rate.
  • the fixed rate is equal to the rate of output of the related streams from sources 102 .
  • Network device 104 has an ‘input packet queue’ corresponding to the related streams.
  • the related streams line up in the input packet queue at an average output rate of network device 104 . If the queue becomes empty, network device 104 outputs synthesized data for the duration for which the queue is empty.
  • the synthesized data may include silence for an audio stream and a previous video frame for the video stream.
  • samples of related streams may be discarded to account for the synthesized data generated. This also holds true for lost packets of the related streams.
  • Sources 102 are connected to each other by means of a first signaling path.
  • destinations 106 are connected to each other by means of a second signaling path.
  • the first and second signaling paths include TCP connections.
  • the first and second signaling paths have deterministic latencies with small variances. Further, as described earlier, there exists a ‘relationship’ between sources 102 . Similarly, there exists a ‘relationship’ between destinations 106 .
  • FIG. 2 illustrates a source, in accordance with an exemplary embodiment of the invention.
  • Source 102 includes a stream-generating device 202 , encoders 204 , such as encoder 204 a, 204 b and 204 c, a synchronization module 206 , and a sender 208 .
  • Stream-generating device 202 generates media streams. Examples of stream-generating device 202 include. a digital video camera, a microphone, or a web camera.
  • Encoders 204 encode the streams generated by stream-generating device 202 for transmission. For example, encoder 204 a may encode an audio stream, encoder 204 b may encode a video stream, and encoder 204 c may encode a stream for ‘mouse-movement’.
  • the encoded streams may be compressed before transmission.
  • the audio media stream may be compressed by using the Global System for Mobile Communication (GSM)—13 kbps, G.729 (8 kbps) and G.723.3 (both 6.4 and 5.3 kbps), and a number of propriety media compression techniques
  • the video media stream may be compressed by using Moving Picture Experts Group (MPEG 1), MPEG 2, MPEG 4, H.261, and H.263.
  • MPEG 1 Moving Picture Experts Group
  • Encoders 204 encode the related streams and then send the encoded streams to synchronization module 206 .
  • Synchronization module 206 includes a source clock 210 and a marking module 212 .
  • Synchronization module 206 synchronizes the stream generated from stream-generating device 202 a, with simultaneous streams generated from sources 102 .
  • Each simultaneous stream corresponds to a media session and is generated at related stream-generating devices 202 of sources 102 .
  • synchronization module 206 of source 102 a can synchronize the media streams generated from the related stream-generating device 202 of source 102 a, with simultaneous streams generated from source 102 b.
  • Each source clock 210 within sources 102 maintains a local source time.
  • Each marking module 212 within sources 102 intermittently embeds unique marker packets into all the streams generated by stream-generating device 202 .
  • each source clock 210 may be synchronized to a configured source ‘Virtual Wall Clock’ (VWC).
  • VWC Virtual Wall Clock
  • the source VWC provides a first common clock time to sources 102 .
  • the source VWC time is updated at a pre-defined time interval by means of a ‘local time base synchronization technique’.
  • S1 source clock 210 of source 102 a
  • S2 source clock 210 of source 102 b
  • S1 sends a packet to S2 comprising TS10
  • S2 sends a packet to S1 comprising TS20
  • S1 calculates a ‘one-way trip delay’, hereinafter referred to as ‘dT’, in each direction as follows:
  • TS11 is an updated local time on S1 when dT is being calculated.
  • dTavg an average one-way trip delay
  • dT avg SUM( dT 1+ dT 2+ . . . + dTN )/ N (2)
  • TB1′ TB1 ⁇ ( TS 10 ⁇ TS 20 ⁇ dT avg) (3)
  • TB1 is an uncorrected local time base
  • TB1′ is a corrected local time base.
  • the corrected local time base TB1′ corresponds to the source VWC.
  • a destination VWC can be configured for the various destination clocks of destinations 106 .
  • the local time base synchronization technique takes into consideration a ‘Round Trip Delay’ (RTD) time and a ‘Clock Offset’ (CO) time for sending and receiving messages between source clock 210 of corresponding sources 102 , through the first signaling path.
  • RTD Red Trip Delay
  • CO Lock Offset
  • ‘t1’ is the local source time sampled by source clock 210 of source 102 a, and it is marked on an originally sampled 64-bit timestamp that is sent to source 102 b
  • ‘t2’ is the local source time sampled by source clock 210 of source 102 b, and it is marked on a timestamp sampled at source 102 b on receiving the 64-bit timestamp from source 102 a
  • ‘t3’ is the time on the 64-bit timestamp at source 102 b when it transmits the 64-bit timestamp to source 102 a, this takes into consideration the time delay due to internal processing of source 102 b
  • ‘t4’ is the time on a 64-bit timestamp at source 102 a when it receives the timestamp from source 102 b. This timestamp is used for updating the local time on source clock 210 of source 102 a.
  • the timestamps are normalized to the lowest precision value, based on the precision of source clock 210 of source 102 that has the lowest precision. For example, if source 102 a is an IP phone and source 102 b is a PC, then the timestamps would be normalized to the precision of source clock 210 of source 102 a, if the precision of the IP phone were lower than that of the PC.
  • the ‘Average Clock Offset’ (ACO) is determined.
  • the ACO is determined by averaging the COs over a series of samples. For example, the number of samples considered may be at least ten.
  • the source VWC for source 102 a and source 102 b is configured based on the ACO, as follows:
  • VWC t 3 ⁇ (Average Clock Offset) (3)
  • estimation error field may be a 32-bit fixed-point number indicating the estimated error of source clock 210 of source 102 a.
  • marking module 212 embeds a unique marker packet into the streams generated by related stream-generating device 202 .
  • the unique marker packet includes a 16-bit ‘Locally Unique Identifier’ (LUID), a 16-bit ‘previous RTP sequence number’, a 32-bit ‘marker ID’ value, a 16-bit ‘Source media type’ value, a 16-bit ‘Sub Source media ID’, and a 128-bit ‘unique identifier’.
  • LID Logical Unique Identifier
  • the 16-bit LUID is a unique identifier for each media session.
  • Each session that is generated by sources 102 has a new ID.
  • the session ID should be the same for all media streams of any one session on stream-generating devices 202 .
  • this session ID may be generated by source 102 a and communicated to other sources, such as source 102 b, and source 102 c.
  • the 16-bit previous RTP sequence number corresponds to a media stream packet that the unique marker packet follows.
  • the 32-bit marker ID value contains the sequence number of the unique marker packet. The sequence number may, for example, start from zero at the beginning of each media session and increase each time the unique marker packets are embedded into the related streams.
  • the 16-bit source media type value contains an identifier to identify the type of media streams.
  • the type of media streams include a video, audio, television, slides, music, and window. stream.
  • the 16-bit ‘Sub Source media ID’ contains a unique identifier for the media streams generated from source 102 a and the 128-bit unique identifier is a unique identifier for source 102 a.
  • FIG. 3 illustrates a destination, in accordance with an exemplary embodiment of the invention.
  • Each destination 106 includes a receiver 302 ; a re-synchronization module 304 ; decoders 306 , such as decoder 306 a, 306 b, 306 c; and a stream-rendering device 308 .
  • Receiver 302 receives a digital media stream along with the unique marker packet that is transmitted by sender 208 of a corresponding source from sources 102 .
  • receiver 302 of destination 106 a, receives the stream sent by sender 208 of source 102 a.
  • Re-synchronization module 304 re-synchronizes the streams received by receiver 302 , with related streams.
  • the re-synchronization is performed by relating like unique marker packets in each related stream.
  • the related streams are streams of the same media session, transmitted simultaneously by source 102 b, source 102 c, and so forth, and received at destinations such as 106 b, 106 c, and so forth.
  • receiver 302 After receiving the related streams, receiver 302 transmits the related streams to re-synchronization module 304 for resynchronization.
  • Re-synchronization module 304 includes a destination clock 310 , a synthesizing module 312 , and a buffer 314 .
  • the second common clock time of the destination VWC is updated at a pre-defined time interval by means of the ‘local time base synchronization technique’, as has been described earlier in the case of configuration of the source VWC.
  • synthesizing module 312 Based on the destination VWC and the embedded unique marker packet, synthesizing module 312 synthesizes the streams received by receiver 302 . For example, a synthesizing module 312 of destination 106 a synthesizes the streams received by a receiver 302 , of destination 106 a.
  • Buffer 314 stores the synthesized and re-synchronized streams for play back.
  • Stream-rendering device 308 accesses buffer 314 to play back the streams.
  • Decoders 306 decode the re-synchronized related streams before playback.
  • Stream-rendering device 308 plays back the decoded streams.
  • decoders 306 a, 306 b, and 306 c correspond to encoders 204 a, 204 b, and 204 c , respectively.
  • FIG. 4 is a flowchart illustrating a method for in-band signaling of at least two simultaneous digital media streams in a network, in accordance with various embodiments of the invention.
  • synchronization module 206 of source 102 a synchronizes a first stream with a destination stream.
  • the synchronized streams are transmitted to corresponding destinations 106 a, and 106 b by the corresponding sources 102 a, and 102 b, respectively.
  • the synchronized streams are transmitted by using the in-band signaling.
  • synchronization module 206 of sources 102 can synchronize a plurality of streams, which are generated from stream generating devices 202 of sources 102 .
  • source 102 b communicates to source 102 a the various additional delays, for example, for each supporting video codec.
  • Source 102 b communicates the delays by sending a ‘Video Frame Decode Delay Message’ to source 102 a.
  • the message includes the number of ‘Codec entry pairs’ that follow.
  • One Codec entry pair includes a ‘video codec type’ and a corresponding ‘video codec delay’.
  • the video codec delay is the time taken by source 102 b to decode a single frame for a previous video codec type of the second stream.
  • the video frame decode delay message is used by sources 102 in order to perform two functions: to share information about the codec supported by each of the sources 102 , and, to share information about the time required for decoding a single frame of the related streams.
  • each of the destinations 106 has the information about the type of media of the related streams and the longest delay time. Accordingly, the decoded media of the related streams is rendered in sync with reference to the actual decode time and play-out time.
  • source clock 210 of source 102 a is synchronized with source clock 210 of source 102 b to configure the source VWC. As described previously, this synchronization is performed on the basis of the local time base synchronization technique.
  • Source 102 a communicates to source 102 b the information about the start of media stream upon receipt or generation of an audio call.
  • Source 102 a communicates this information by sending a ‘Request Marker Start Message’ to source 102 b.
  • This message includes an ‘audio codec type’, and an ‘audio frame rate’.
  • the audio frame rate is the rate at which the audio frames of the media streams may be sent by source 102 a.
  • source 102 b sends a ‘Marker Start Notify Message’ to source 102 a.
  • This message provides information about the proper timing and location of where the unique marker packets have to be embedded in the media streams, based on the source VWC.
  • the marker start notify message includes, for example, the next value of the source VWC, count of the number of audio packets that are to be sent between each unique marker packet, an identification value for the next unique marker packet, and the Real Time Protocol (RTP) payload type.
  • RTP Real Time Protocol
  • sender 208 of source 102 a transmits a first stream to destination 106 a, which may be an IP phone, and sender 208 of source 102 b transmits a second stream to destination 106 b, which may be a PC.
  • Destination 106 a plays out the audio media stream after an appropriate delay period. It notifies destination 106 b as it. reaches each marker just before play out by using a ‘Play out Marker Globally Unique Identifier (GUID) Message’.
  • This message includes a 32-bit unique marker ID. For example, the time of play-out of first stream at stream-rendering device 308 of destination 106 a is performed, based on the destination VWC and the video frame decode delay message.
  • destination 106 a synchronizes with destination 106 b when destination 106 a receives the first unique marker packet from source 102 a.
  • Destinations 106 a and 106 b synchronize their respective local time base to the destination VWC.
  • destination 106 a communicates to destination 106 b the unique marker packet provided by source 102 a. If a time-base adjustment of destination 106 a is negative, with respect to the local time-base synchronization, synthesizing module 312 of destination 106 a will synthesize the first stream for a ‘first adjusted amount’ and then begin playback of the first stream. However, if the time-base adjustment is positive, destination 106 a will discard a ‘second adjusted amount’ from the first stream and then begin playback. This synchronizes the compensation of the first stream with the second stream.
  • synthesizing module 312 of destination 106 b begins synthesizing data and upon receiving the second stream, it discards the same by a ‘third adjusted amount’. This is done to compensate for the synthesized data and the difference of time between the receipt of the beginning unique marker of destination 106 a and the receipt of the beginning unique marker of destination 106 b. This can be described as follows:
  • TDIS 1 ( T SYN+( TS 1 B ⁇ TS 2 B )) (6)
  • TDIS1 is the third adjusted amount
  • TTYN is the amount of data that has already been synthesized by synthesizing module 312 of destination 106 b
  • TTIB is the time upon receipt of the beginning unique marker packet for the first stream
  • TS2B is the time upon receipt of the beginning unique marker packet for the second stream.
  • the fourth adjusted amount is calculated as follows:
  • TDIS 2 ( TS 1 B ⁇ TS 2 B ) (7)
  • TDIS2 is the fourth adjusted amount.
  • FIG. 5 is a flowchart illustrating a method for synchronizing at least two simultaneous streams, in accordance with an exemplary embodiment of the invention.
  • the source VWC is configured by using the local time base synchronization technique.
  • marking module 212 embeds a unique marker packet into all the streams generated by stream-generating device 202 at regular intervals based on the time provided by the source VWC. The unique marker packets help to synchronize the streams. These synchronized streams are then transmitted to corresponding destinations 106 .
  • FIG. 6 is a flowchart illustrating a method for re-synchronizing the synchronized streams, according to an exemplary embodiment of the invention.
  • a corresponding destination 106 receives one of the related streams from sender 208 from one of the sources 102 .
  • destination 106 a receives the first stream from source 102 a.
  • synthesizing module 312 of destination 106 a sends the ‘Play-out Marker GUID Message’ to the synthesizing module of destination 106 b.
  • a check is performed to verify if the play-out marker GUID message indicates that synthesis of the first stream needs to be performed before the play-out.
  • step 608 If the check performed at step 606 is true, then step 608 is performed; otherwise step 610 is performed.
  • synthesizing module 312 of destination 106 a synthesizes the first stream, based on the play-out marker GUID message. Synthesizing implies playing a synthesized data on the basis of the first adjusted amount.
  • step 610 a data from the first stream is discarded by the second adjusted amount.
  • the streams synthesized by synthesizing module 312 are stored in buffer 314 of destination 106 a.
  • decoders 306 of destination 106 a decode the streams stored in buffer 314 .
  • the decoded streams are played out by stream-rendering device 308 .
  • FIG. 7 is a flowchart illustrating a method for re-synchronizing the synchronized streams, according to an exemplary embodiment of the invention.
  • a check is performed by destination 106 a to ascertain whether receiver 302 in destination 106 b has received the second stream. If the check performed at step 702 is true, then step 704 is performed, otherwise, step 706 is performed.
  • synthesizing module 312 in receiver 302 b discards data on the second stream by the third adjusted amount.
  • synthesizing module 312 in destination 106 b discards the data on the second stream by the fourth adjusted amount.
  • the streams synthesized by synthesizing module 312 are stored in buffer 314 of destination 106 a.
  • decoders 306 of destination 106 a decode the streams stored in buffer 314 .
  • the decoded streams are played out by stream-rendering device 308 .
  • Various embodiments of the invention provide a method for synchronizing ‘N’ digital media streams, which relate to one media session.
  • the synchronization is performed by embedding unique marker packets within the simultaneous digital media streams of one media session.
  • Various embodiments of the invention use in-band signaling for communication between a source and a destination.
  • embodiments of the invention may be applied to various types of media that can be streamed. Further, embodiments of the invention are simple to implement and require less buffering capacity as no additional processing or encoding is required before the play back.
  • Embodiments of the invention do not require a connection with a dedicated timeserver in order to achieve synchronization of the simultaneous streams in a media session. Moreover, no additional processing or encoding is required to record and playback the streams in synchronization.
  • a ‘Method and System for In-band Signaling of at least of Multiple Media Streams’ can include any type of analysis, manual or automatic, to anticipate the needs for in-band signaling of multiple media streams.
  • peer can include any type of device, operation, or any other process.
  • the invention can operate between any two processes or entities including users, devices, functional systems, or combinations of hardware and software.
  • Peer-to-peer networks and any other networks or systems where the roles of client and server are switched, change dynamically, or are not even present, are within the scope of the invention.
  • routines of the present invention can be implemented using C, C++, Java, assembly language, etc.
  • Different programming techniques such as procedural or object oriented can be employed.
  • the routines can execute on a single processing device or multiple processors. Although the steps, operations, or computations may be presented in a specific order, this order may be changed in different embodiments. In some embodiments, multiple steps shown sequentially in this specification can be performed at the same time.
  • the sequence of operations described herein can be interrupted, suspended, or otherwise controlled by another process, such as an operating system, kernel, etc.
  • the routines can operate in an operating system environment or as stand-alone routines occupying all, or a substantial part, of the system processing.
  • a ‘computer’ for purposes of embodiments of the present invention may include any processor-containing device, such as a mainframe computer, personal computer, laptop, notebook, microcomputer, server, personal data manager or ‘PIM’ (also referred to as a personal information manager), smart cellular or other phone, so-called smart card, set-top box, or any of the like.
  • a ‘computer program’ may include any suitable locally or remotely executable program or sequence of coded instructions, which are to be inserted into a computer, well known to those skilled in the art. Stated more specifically, a computer program includes an organized list of instructions that, when executed, causes the computer to behave in a predetermined manner.
  • a computer program contains a list of ingredients (called variables) and a list of directions (called statements) that tell the computer what to do with the variables.
  • the variables may represent numeric data, text, audio or graphical images. If a computer is employed for presenting media via a suitable directly or indirectly coupled input/output (I/O) device, the computer would have suitable instructions for allowing a user to input or output (e.g., present) program code and/or data information respectively in accordance with the embodiments of the present invention.
  • I/O input/output
  • a ‘computer readable medium’ for purposes of embodiments of the present invention may be any medium that can contain, store, communicate, propagate, or transport the computer program for use by or in connection with the instruction execution system apparatus, system or device.
  • the computer readable medium can be, by way of example only but not by limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, system, device, propagation medium, or computer memory.
  • At least some of the components of an embodiment of the invention may be implemented by using a programmed general-purpose digital computer, by using application specific integrated circuits, programmable logic devices, or field programmable gate arrays, or by using a network of interconnected components and circuits. Connections may be wired, wireless, by modem, and the like.

Abstract

Methods and systems are provided for in-band signaling of at least two simultaneous digital media streams in a network, the two simultaneous streams being a part of a media session. Each of the at least two simultaneous streams is generated from a corresponding source. The generated simultaneous streams are synchronized by using a unique marker packet. Each synchronized stream is transmitted to a destination corresponding to each source.

Description

    BACKGROUND OF THE INVENTION
  • 1. Field of Invention
  • Embodiments of the invention relate in general to transmission of multiple media streams. More specifically, the embodiments of the invention relate to methods and systems enabling in-band signaling of multiple media streams in a network.
  • 2. Description of the Background Art
  • Transmission of two or more real-time media streams, for example, video and audio, in a network requires the aggregation of multiple digital voice channels, for example 56 Kbit, 64 Kbit digital voice channels. This aggregation is required to provide sufficient bandwidth—typically a minimum of 128 Kbit for video. Each channel may be routed between a source and a destination through a different network path, leading to varying channel latencies.
  • During the process of transmission, the multiple streams may also develop a varying relationship with each other, due to differences in the processing speed of network devices. Therefore, the multiple streams have to be synchronized at the time of playback.
  • There are ‘in-band’ and ‘out of band’ signaling protocols for the synchronization of multiple media streams. One such in-band signaling protocol inputs data packets into all the channels at the source. Thereafter, it analyzes the data at the destination to realign the channels. The variance in channel latencies is fixed once the connection is established. However, in the case of Internet Protocol (IP) streams, every packet for any given stream may be routed through completely different network paths or may incur variable delays from end to end.
  • The out-of-band protocol requires an additional signal channel alongside the data transmission channel, to carry signal messages for synchronization. However, such a scheme may be susceptible to problems arising due to the possible loss of signaling messages.
  • Another scheme uses the concept of ‘timestamps’. This scheme requires either distributed ‘virtual wall clocks’ or forms of virtual ‘Round Trip Timestamps (RTTs)’ across an unknown and ever-changing network. RTTs can drift, relative to each other, during a stream session, due to changing network dynamics, and hence may be inaccurate. If multiple related streams, such as audio, video, slides and mouse movements are used, then the complexity of using timestamps increases. Moreover, this scheme requires a connection to a dedicated timeserver, in order to maintain the virtual wall clocks.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 illustrates an environment for in-band signaling of multiple media streams wherein various embodiments of the invention can be practiced.
  • FIG. 2 illustrates a source, in accordance with an exemplary embodiment of the invention.
  • FIG. 3 illustrates a destination, in accordance with an exemplary embodiment of the invention.
  • FIG. 4 is a flowchart illustrating a method for in-band signaling of at least two simultaneous digital media streams in a network, in accordance with various embodiments of the invention.
  • FIG. 5 is a flowchart illustrating a method for synchronizing at least two simultaneous streams, in accordance with an exemplary embodiment of the invention.
  • FIG. 6 is a flowchart illustrating a method for re-synchronizing the synchronized streams, according to an exemplary embodiment of the invention.
  • FIG. 7 is a flowchart illustrating a method for re-synchronizing the synchronized streams, according to an exemplary embodiment of the invention.
  • DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION
  • Various embodiments of the invention provide methods, systems, and computer-readable media for in-band signaling of at least two simultaneous digital media streams in a network. The at least two simultaneous digital streams may relate to a media session and the streams may be generated from multiple sources. Various embodiments of the invention perform a temporal synchronization between ‘N’ simultaneous digital media streams relating to the media session. The synchronization is performed by embedding unique marker packets within each of the N simultaneous digital media streams. The unique marker packets are embedded simultaneously into each of the N simultaneous streams at regular intervals of time. The time is determined by configuring a source Virtual Wall Clock (VWC), which may be configured by synchronizing separate source clocks within each of the multiple sources. The synchronized streams are transmitted to multiple destinations; wherein, each of the multiple destinations corresponds to each of the multiple sources. At the corresponding destinations, the previously synchronized streams are re-synchronized, based on the embedded unique marker packets, and a destination VWC. The destination VWC is configured by synchronizing separate destination clocks on each of the corresponding destinations. After the re-synchronization, the streams are played back.
  • In accordance with various embodiments of the invention, the N simultaneous digital media streams may be generated simultaneously from N number of related stream-generating devices. In accordance with various embodiments of the invention, the N related stream-generating devices may be N separate Data Signaling Processors (DSPs) in the same videophone. In accordance with various embodiments, the N related stream-generating devices may be in physically separate sources. For example, for a media session with N equal to two, the related stream-generating devices may be included in an Internet Protocol (IP) phone for audio generation and in a Personal Computer (PC) for video generation. Further, these devices may be connected, for example, through a direct wire connection (802.1). The related stream-generating devices have a ‘relationship’, and a ‘signaling path’ with each other. The relationship is a temporal one, which is developed between the simultaneous streams generated by the related stream-generating devices by embedding unique marker packets within each of the simultaneous streams. The unique marker packets are embedded within each of the related simultaneous streams, at the same instance. The signaling path is the protocol capability between the related stream-generating devices, which generate the simultaneous streams. Examples of protocols include TCP. The signaling path has deterministic latencies with small, if any, variance. Considering the above example, when the IP phone is activated, it starts communicating with the related video-generating device, which is the PC, to determine a communication delay between them. Thereafter, as the related audio and video streams are generated, the unique marker packets are inserted, at appropriate locations, into the RTP streams for synchronization of the streams.
  • FIG. 1 illustrates an environment for in-band signaling of multiple media streams wherein various embodiments of the invention can be practiced. The environment includes sources 102 such as sources 102 a, 102 b and 102 c; a network device 104; destinations 106, such as destinations 106 a, 106 b, and 106 c. Sources 102, network device 104 and destinations 106 are connected through a network. The network may be, for example, a Wide Area Network (WAN) or a Local Area Network (LAN), a client-server network, a network with multitier architecture, or a peer-to-peer network. Each of sources 102 comprises at least one of the related stream-generating devices, which generate digital media streams corresponding to a media session. In accordance with various embodiments of the invention, sources 102 generate and synchronize ‘N’ simultaneous media streams corresponding to the media session, and destinations 106 receive the ‘N’ synchronized digital media streams. For example, for N equal to four, the synchronized media streams may include streams generated from each of the following: a video, an audio, a slide, and a mouse movement. According to an embodiment of the invention, destinations 106 a, 106 b, and 106 c may be separate DSPs in one media-rendering device. According to another embodiment of the invention, destinations 106 a, 106 b, and 106 c may be separate DSPs in separate media-rendering devices. However, there is the second signaling path between destinations 106.
  • In accordance with various embodiments of the invention, sources 102 and destinations 106 can be a data-processing unit. Examples of a data-processing unit include, but are not limited to, a mainframe computer, a supercomputer, an enterprise application server, a workstation, an embedded system, a videophone, an Internet Protocol (IP) phone, and a personal computer. The personal computer can be, for example, a desktop computer, a laptop computer, a tablet computer, a Personal Digital Assistant (PDA), a Personal Video Recorder, and a wearable computer.
  • Network device 104 can be a network interface between sources 102 and destinations 106. The network interface between sources 102 and destinations 106 may be of different types. Examples of network device 104 include a mixer, a bridge, a firewall, a gateway, a router, an ISDN terminal adapter, a hub, a switch, a multilayer switch, a line driver, a modem, a multiplexer, a network interface card, a protocol converter, a proxy, and so forth.
  • Network device 104 receives related streams and then outputs the related streams to destinations 106 at a fixed rate. In accordance with an embodiment, the fixed rate is equal to the rate of output of the related streams from sources 102. Network device 104 has an ‘input packet queue’ corresponding to the related streams. The related streams line up in the input packet queue at an average output rate of network device 104. If the queue becomes empty, network device 104 outputs synthesized data for the duration for which the queue is empty. In accordance with various embodiments of the invention, the synthesized data may include silence for an audio stream and a previous video frame for the video stream. When the related streams arrive at destinations 106, samples of related streams may be discarded to account for the synthesized data generated. This also holds true for lost packets of the related streams.
  • Sources 102 are connected to each other by means of a first signaling path. Similarly, destinations 106 are connected to each other by means of a second signaling path. Examples of the first and second signaling paths include TCP connections. In accordance with various embodiments of the invention, the first and second signaling paths have deterministic latencies with small variances. Further, as described earlier, there exists a ‘relationship’ between sources 102. Similarly, there exists a ‘relationship’ between destinations 106.
  • FIG. 2 illustrates a source, in accordance with an exemplary embodiment of the invention. Source 102 includes a stream-generating device 202, encoders 204, such as encoder 204 a, 204 b and 204 c, a synchronization module 206, and a sender 208. Stream-generating device 202 generates media streams. Examples of stream-generating device 202 include. a digital video camera, a microphone, or a web camera. Encoders 204 encode the streams generated by stream-generating device 202 for transmission. For example, encoder 204 a may encode an audio stream, encoder 204 b may encode a video stream, and encoder 204 c may encode a stream for ‘mouse-movement’. In accordance with various embodiments of the invention, the encoded streams may be compressed before transmission. For example, while the audio media stream may be compressed by using the Global System for Mobile Communication (GSM)—13 kbps, G.729 (8 kbps) and G.723.3 (both 6.4 and 5.3 kbps), and a number of propriety media compression techniques, the video media stream may be compressed by using Moving Picture Experts Group (MPEG 1), MPEG 2, MPEG 4, H.261, and H.263.
  • Encoders 204 encode the related streams and then send the encoded streams to synchronization module 206. Synchronization module 206 includes a source clock 210 and a marking module 212. Synchronization module 206 synchronizes the stream generated from stream-generating device 202 a, with simultaneous streams generated from sources 102. Each simultaneous stream corresponds to a media session and is generated at related stream-generating devices 202 of sources 102. For example, synchronization module 206 of source 102 a can synchronize the media streams generated from the related stream-generating device 202 of source 102 a, with simultaneous streams generated from source 102 b. Each source clock 210 within sources 102 maintains a local source time. Each marking module 212 within sources 102 intermittently embeds unique marker packets into all the streams generated by stream-generating device 202.
  • In accordance with various embodiments of the invention, each source clock 210 may be synchronized to a configured source ‘Virtual Wall Clock’ (VWC). The source VWC provides a first common clock time to sources 102. In accordance with an embodiment, the source VWC time is updated at a pre-defined time interval by means of a ‘local time base synchronization technique’. For example, the synchronization of source clock 210 of source 102 a, hereinafter referred to as ‘S1’, and source clock 210 of source 102 b, hereinafter referred to as ‘S2’, takes place as follows:
  • 1. S1 samples its local time base TS10
  • 2. S1 sends a packet to S2 comprising TS10
  • 3. S2 samples its local time base TS20
  • 4. S2 sends a packet to S1 comprising TS20
  • 5. S1 calculates a ‘one-way trip delay’, hereinafter referred to as ‘dT’, in each direction as follows:

  • dT=(TS11−TS10)/2  (1)
  • Where TS11 is an updated local time on S1 when dT is being calculated.
    According to various embodiments of the invention, an average one-way trip delay (dTavg) is calculated for ‘N’ number of trips, as follows:

  • dTavg=SUM(dT1+dT2+ . . . +dTN)/N  (2)
  • 6. S1 yields to S2 by correcting its local time base as follows:

  • TB1′=TB1−(TS10−TS20−dTavg)  (3)
  • Where TB1 is an uncorrected local time base, and TB1′ is a corrected local time base.
    In accordance with various embodiments of the invention, the corrected local time base TB1′ corresponds to the source VWC. Similarly, a destination VWC can be configured for the various destination clocks of destinations 106.
  • In accordance with various embodiments of the invention, the local time base synchronization technique takes into consideration a ‘Round Trip Delay’ (RTD) time and a ‘Clock Offset’ (CO) time for sending and receiving messages between source clock 210 of corresponding sources 102, through the first signaling path. The RTD and CO time can be calculated as follows:

  • RTD=(t4−t1)−(t3−t2)  (4)

  • CO=((t2−t1)+(t3−t4))/2  (5)
  • Where, for example,
    ‘t1’ is the local source time sampled by source clock 210 of source 102 a, and it is marked on an originally sampled 64-bit timestamp that is sent to source 102 b,
    ‘t2’ is the local source time sampled by source clock 210 of source 102 b, and it is marked on a timestamp sampled at source 102 b on receiving the 64-bit timestamp from source 102 a,
    ‘t3’ is the time on the 64-bit timestamp at source 102 b when it transmits the 64-bit timestamp to source 102 a, this takes into consideration the time delay due to internal processing of source 102 b, and
    ‘t4’ is the time on a 64-bit timestamp at source 102 a when it receives the timestamp from source 102 b. This timestamp is used for updating the local time on source clock 210 of source 102 a.
  • In accordance with various embodiments of the invention, the timestamps are normalized to the lowest precision value, based on the precision of source clock 210 of source 102 that has the lowest precision. For example, if source 102 a is an IP phone and source 102 b is a PC, then the timestamps would be normalized to the precision of source clock 210 of source 102 a, if the precision of the IP phone were lower than that of the PC.
  • In accordance with various embodiments of the invention, based on the timestamps and the CO, the ‘Average Clock Offset’ (ACO) is determined. The ACO is determined by averaging the COs over a series of samples. For example, the number of samples considered may be at least ten. Further, the source VWC for source 102 a and source 102 b is configured based on the ACO, as follows:

  • VWC=t3−(Average Clock Offset)  (3)
  • The difference between the time of VWC and the time of source clock 102 of source 102 b is transmitted to the source clock 210 of source 102 a in the ‘Estimated Error’ field of an NTP message for the synchronization of source clock 210 of source 102 a. The NTP messages are communicated between sources 102 in order to transmit time-information for the synchronization of each of source clock 210 of sources 102. In accordance with various embodiments of the invention, estimation error field may be a 32-bit fixed-point number indicating the estimated error of source clock 210 of source 102 a.
  • Based on the first common clock time provided by the configured source VWC, marking module 212 embeds a unique marker packet into the streams generated by related stream-generating device 202. In accordance with various embodiments of the invention, the unique marker packet includes a 16-bit ‘Locally Unique Identifier’ (LUID), a 16-bit ‘previous RTP sequence number’, a 32-bit ‘marker ID’ value, a 16-bit ‘Source media type’ value, a 16-bit ‘Sub Source media ID’, and a 128-bit ‘unique identifier’.
  • The 16-bit LUID is a unique identifier for each media session. Each session that is generated by sources 102 has a new ID. The session ID should be the same for all media streams of any one session on stream-generating devices 202. For example, this session ID may be generated by source 102 a and communicated to other sources, such as source 102 b, and source 102 c. The 16-bit previous RTP sequence number corresponds to a media stream packet that the unique marker packet follows. The 32-bit marker ID value contains the sequence number of the unique marker packet. The sequence number may, for example, start from zero at the beginning of each media session and increase each time the unique marker packets are embedded into the related streams. Therefore, the related unique marker packets are uniquely tied to the related stream-generating device and unique to each session. The 16-bit source media type value contains an identifier to identify the type of media streams. For example, the type of media streams include a video, audio, television, slides, music, and window. stream. The 16-bit ‘Sub Source media ID’ contains a unique identifier for the media streams generated from source 102 a and the 128-bit unique identifier is a unique identifier for source 102 a. After the unique marker packets are embedded, sender 208 of source 102, corresponding to destination 106; transmits the related streams. For example, sender 208 of source 102 a transmits streams to destination 106 a.
  • FIG. 3 illustrates a destination, in accordance with an exemplary embodiment of the invention. Each destination 106 includes a receiver 302; a re-synchronization module 304; decoders 306, such as decoder 306 a, 306 b, 306 c; and a stream-rendering device 308. Receiver 302 receives a digital media stream along with the unique marker packet that is transmitted by sender 208 of a corresponding source from sources 102. For example, receiver 302, of destination 106 a, receives the stream sent by sender 208 of source 102 a. Re-synchronization module 304 re-synchronizes the streams received by receiver 302, with related streams. The re-synchronization is performed by relating like unique marker packets in each related stream. The related streams are streams of the same media session, transmitted simultaneously by source 102 b, source 102 c, and so forth, and received at destinations such as 106 b, 106 c, and so forth. After receiving the related streams, receiver 302 transmits the related streams to re-synchronization module 304 for resynchronization.
  • Re-synchronization module 304 includes a destination clock 310, a synthesizing module 312, and a buffer 314. Destination-clocks 310 in destinations 106 a, 106 b, and 106 c, configure a destination VWC, in order to provide a second common clock time. In accordance with an embodiment of the invention, the second common clock time of the destination VWC is updated at a pre-defined time interval by means of the ‘local time base synchronization technique’, as has been described earlier in the case of configuration of the source VWC.
  • Based on the destination VWC and the embedded unique marker packet, synthesizing module 312 synthesizes the streams received by receiver 302. For example, a synthesizing module 312 of destination 106 a synthesizes the streams received by a receiver 302, of destination 106 a.
  • Buffer 314 stores the synthesized and re-synchronized streams for play back. Stream-rendering device 308 accesses buffer 314 to play back the streams.
  • Decoders 306 decode the re-synchronized related streams before playback. Stream-rendering device 308 plays back the decoded streams. In accordance with various embodiments of the invention, decoders 306 a, 306 b, and 306 c correspond to encoders 204 a, 204 b, and 204 c, respectively.
  • FIG. 4 is a flowchart illustrating a method for in-band signaling of at least two simultaneous digital media streams in a network, in accordance with various embodiments of the invention. At step 402, synchronization module 206 of source 102 a synchronizes a first stream with a destination stream. At step 404, the synchronized streams are transmitted to corresponding destinations 106 a, and 106 b by the corresponding sources 102 a, and 102 b, respectively. The synchronized streams are transmitted by using the in-band signaling. According to the various embodiments of the invention, synchronization module 206 of sources 102 can synchronize a plurality of streams, which are generated from stream generating devices 202 of sources 102.
  • In accordance with various embodiments of the invention, source 102 b communicates to source 102 a the various additional delays, for example, for each supporting video codec. Source 102 b communicates the delays by sending a ‘Video Frame Decode Delay Message’ to source 102 a. The message includes the number of ‘Codec entry pairs’ that follow. One Codec entry pair includes a ‘video codec type’ and a corresponding ‘video codec delay’. The video codec delay is the time taken by source 102 b to decode a single frame for a previous video codec type of the second stream. Therefore, the video frame decode delay message is used by sources 102 in order to perform two functions: to share information about the codec supported by each of the sources 102, and, to share information about the time required for decoding a single frame of the related streams. For example, during the play-out of the related streams at destinations 106, each of the destinations 106 has the information about the type of media of the related streams and the longest delay time. Accordingly, the decoded media of the related streams is rendered in sync with reference to the actual decode time and play-out time.
  • In accordance with an embodiment of the invention, source clock 210 of source 102 a is synchronized with source clock 210 of source 102 b to configure the source VWC. As described previously, this synchronization is performed on the basis of the local time base synchronization technique. Source 102 a communicates to source 102 b the information about the start of media stream upon receipt or generation of an audio call. Source 102 a communicates this information by sending a ‘Request Marker Start Message’ to source 102 b. This message includes an ‘audio codec type’, and an ‘audio frame rate’. The audio frame rate is the rate at which the audio frames of the media streams may be sent by source 102 a. For example, if two frames are sent after every 20 millisecond (ms), the audio frame rate is 20 ms. Thereafter, source 102 b sends a ‘Marker Start Notify Message’ to source 102 a. This message provides information about the proper timing and location of where the unique marker packets have to be embedded in the media streams, based on the source VWC. The marker start notify message includes, for example, the next value of the source VWC, count of the number of audio packets that are to be sent between each unique marker packet, an identification value for the next unique marker packet, and the Real Time Protocol (RTP) payload type.
  • According to various embodiments of the invention, sender 208 of source 102 a transmits a first stream to destination 106 a, which may be an IP phone, and sender 208 of source 102 b transmits a second stream to destination 106 b, which may be a PC. Destination 106 a plays out the audio media stream after an appropriate delay period. It notifies destination 106 b as it. reaches each marker just before play out by using a ‘Play out Marker Globally Unique Identifier (GUID) Message’. This message includes a 32-bit unique marker ID. For example, the time of play-out of first stream at stream-rendering device 308 of destination 106 a is performed, based on the destination VWC and the video frame decode delay message.
  • In accordance with various embodiments of the invention, destination 106 a synchronizes with destination 106 b when destination 106 a receives the first unique marker packet from source 102 a. Destinations 106 a and 106 b synchronize their respective local time base to the destination VWC. In addition, during the synchronization, destination 106 a communicates to destination 106 b the unique marker packet provided by source 102 a. If a time-base adjustment of destination 106 a is negative, with respect to the local time-base synchronization, synthesizing module 312 of destination 106 a will synthesize the first stream for a ‘first adjusted amount’ and then begin playback of the first stream. However, if the time-base adjustment is positive, destination 106 a will discard a ‘second adjusted amount’ from the first stream and then begin playback. This synchronizes the compensation of the first stream with the second stream.
  • If destination 106 b has not received the second stream from source 102 b, synthesizing module 312 of destination 106 b begins synthesizing data and upon receiving the second stream, it discards the same by a ‘third adjusted amount’. This is done to compensate for the synthesized data and the difference of time between the receipt of the beginning unique marker of destination 106 a and the receipt of the beginning unique marker of destination 106 b. This can be described as follows:

  • TDIS1=(TSYN+(TS1B−TS2B))  (6)
  • Where ‘TDIS1’ is the third adjusted amount,
    ‘TSYN’ is the amount of data that has already been synthesized by synthesizing module 312 of destination 106 b,
    ‘TSIB’ is the time upon receipt of the beginning unique marker packet for the first stream, and
    ‘TS2B’ is the time upon receipt of the beginning unique marker packet for the second stream.
  • If destination 106 b has already received the second stream it will discard a ‘fourth adjusted amount’ from the second stream and then begin playback. The fourth adjusted amount is calculated as follows:

  • TDIS2=(TS1B−TS2B)  (7)
  • Where ‘TDIS2’ is the fourth adjusted amount.
  • FIG. 5 is a flowchart illustrating a method for synchronizing at least two simultaneous streams, in accordance with an exemplary embodiment of the invention. At step 502, the source VWC is configured by using the local time base synchronization technique. At step 504, marking module 212 embeds a unique marker packet into all the streams generated by stream-generating device 202 at regular intervals based on the time provided by the source VWC. The unique marker packets help to synchronize the streams. These synchronized streams are then transmitted to corresponding destinations 106.
  • FIG. 6 is a flowchart illustrating a method for re-synchronizing the synchronized streams, according to an exemplary embodiment of the invention. At step 602, a corresponding destination 106 receives one of the related streams from sender 208 from one of the sources 102. For example, destination 106 a receives the first stream from source 102 a. At step 604, synthesizing module 312 of destination 106 a sends the ‘Play-out Marker GUID Message’ to the synthesizing module of destination 106 b. At step 606, a check is performed to verify if the play-out marker GUID message indicates that synthesis of the first stream needs to be performed before the play-out. If the check performed at step 606 is true, then step 608 is performed; otherwise step 610 is performed. At step 608, synthesizing module 312 of destination 106 a synthesizes the first stream, based on the play-out marker GUID message. Synthesizing implies playing a synthesized data on the basis of the first adjusted amount. At step 610, a data from the first stream is discarded by the second adjusted amount. At step 612, the streams synthesized by synthesizing module 312 are stored in buffer 314 of destination 106 a. At step 614, decoders 306 of destination 106 a decode the streams stored in buffer 314. At step 616, the decoded streams are played out by stream-rendering device 308.
  • FIG. 7 is a flowchart illustrating a method for re-synchronizing the synchronized streams, according to an exemplary embodiment of the invention. At step 702, a check is performed by destination 106 a to ascertain whether receiver 302 in destination 106 b has received the second stream. If the check performed at step 702 is true, then step 704 is performed, otherwise, step 706 is performed. At step 704, synthesizing module 312 in receiver 302 b discards data on the second stream by the third adjusted amount.
  • At step 706, synthesizing module 312 in destination 106 b discards the data on the second stream by the fourth adjusted amount. At step 708, the streams synthesized by synthesizing module 312 are stored in buffer 314 of destination 106 a. At step 710, decoders 306 of destination 106 a decode the streams stored in buffer 314. At step 712, the decoded streams are played out by stream-rendering device 308.
  • Various embodiments of the invention provide a method for synchronizing ‘N’ digital media streams, which relate to one media session. The synchronization is performed by embedding unique marker packets within the simultaneous digital media streams of one media session.
  • Various embodiments of the invention use in-band signaling for communication between a source and a destination.
  • Various embodiments of the invention may be applied to various types of media that can be streamed. Further, embodiments of the invention are simple to implement and require less buffering capacity as no additional processing or encoding is required before the play back.
  • Embodiments of the invention do not require a connection with a dedicated timeserver in order to achieve synchronization of the simultaneous streams in a media session. Moreover, no additional processing or encoding is required to record and playback the streams in synchronization.
  • Although the invention has been discussed with respect to specific embodiments thereof, these embodiments are merely illustrative; and not restrictive, of the invention. For example, a ‘Method and System for In-band Signaling of at least of Multiple Media Streams’ can include any type of analysis, manual or automatic, to anticipate the needs for in-band signaling of multiple media streams.
  • Although specific protocols have been used to describe the embodiments, other embodiments can use other transmission protocols or standards. Use of the terms ‘peer’, ‘client’, and ‘server’ can include any type of device, operation, or any other process. The invention can operate between any two processes or entities including users, devices, functional systems, or combinations of hardware and software. Peer-to-peer networks and any other networks or systems where the roles of client and server are switched, change dynamically, or are not even present, are within the scope of the invention.
  • Any suitable programming language can be used to implement the routines of the present invention including C, C++, Java, assembly language, etc. Different programming techniques such as procedural or object oriented can be employed. The routines can execute on a single processing device or multiple processors. Although the steps, operations, or computations may be presented in a specific order, this order may be changed in different embodiments. In some embodiments, multiple steps shown sequentially in this specification can be performed at the same time. The sequence of operations described herein can be interrupted, suspended, or otherwise controlled by another process, such as an operating system, kernel, etc. The routines can operate in an operating system environment or as stand-alone routines occupying all, or a substantial part, of the system processing.
  • In the description herein for embodiments of the present invention, numerous specific details are provided, such as examples of components and/or methods, to provide a thorough understanding of embodiments of the present invention. One skilled in the relevant art will recognize, however, that an embodiment of the invention can be practiced without one or more of the specific details, or with other apparatus, systems, assemblies, methods, components, materials, parts, and/or the like. In other instances, well-known structures, materials, or operations are not specifically shown or described in detail to avoid obscuring aspects of embodiments of the present invention.
  • Also in the description herein for embodiments of the present invention, a portion of the disclosure recited in the specification contains material, which is subject to copyright protection. Computer program source code, object code, instructions, text or other functional information that is executable by a machine may be included in an appendix, tables, figures or in other forms. The copyright owner has no objection to the facsimile reproduction of the specification as filed in the Patent and Trademark Office. Otherwise all copyright rights are reserved.
  • A ‘computer’ for purposes of embodiments of the present invention may include any processor-containing device, such as a mainframe computer, personal computer, laptop, notebook, microcomputer, server, personal data manager or ‘PIM’ (also referred to as a personal information manager), smart cellular or other phone, so-called smart card, set-top box, or any of the like. A ‘computer program’ may include any suitable locally or remotely executable program or sequence of coded instructions, which are to be inserted into a computer, well known to those skilled in the art. Stated more specifically, a computer program includes an organized list of instructions that, when executed, causes the computer to behave in a predetermined manner. A computer program contains a list of ingredients (called variables) and a list of directions (called statements) that tell the computer what to do with the variables. The variables may represent numeric data, text, audio or graphical images. If a computer is employed for presenting media via a suitable directly or indirectly coupled input/output (I/O) device, the computer would have suitable instructions for allowing a user to input or output (e.g., present) program code and/or data information respectively in accordance with the embodiments of the present invention.
  • A ‘computer readable medium’ for purposes of embodiments of the present invention may be any medium that can contain, store, communicate, propagate, or transport the computer program for use by or in connection with the instruction execution system apparatus, system or device. The computer readable medium can be, by way of example only but not by limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, system, device, propagation medium, or computer memory.
  • Reference throughout this specification to “one embodiment”, “an embodiment”, or “a specific embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention and not necessarily in all embodiments. Thus, respective appearances of the phrases “in one embodiment”, “in an embodiment”, or “in a specific embodiment” in various places throughout this specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, or characteristics of any specific embodiment of the present invention may be combined in any suitable manner with one or more other embodiments. It is to be understood that other variations and modifications of the embodiments of the present invention described and illustrated herein are possible in light of the teachings herein and are to be considered as part of the spirit and scope of the present invention.
  • Further, at least some of the components of an embodiment of the invention may be implemented by using a programmed general-purpose digital computer, by using application specific integrated circuits, programmable logic devices, or field programmable gate arrays, or by using a network of interconnected components and circuits. Connections may be wired, wireless, by modem, and the like.
  • It will also be appreciated that one or more of the elements depicted in the drawings/figures can also be implemented in a more separated or integrated manner, or even removed or rendered as inoperable in certain cases, as is useful in accordance with a particular application.
  • Additionally, any signal arrows in the drawings/Figures should be considered only as exemplary, and not limiting, unless otherwise specifically noted. Combinations of components or steps will also be considered as being noted, where terminology is foreseen as rendering the ability to separate or combine is unclear.
  • As used in the description herein and throughout the claims that follow, “a”, “an”, and “the” includes plural references unless the context clearly dictates otherwise. Also, as used in the description herein and throughout the claims that follow, the meaning of “in” includes “in” and “on” unless the context clearly dictates otherwise.
  • The foregoing description of illustrated embodiments of the present invention, including what is described in the abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed herein. While specific embodiments of, and examples for, the invention are described herein for illustrative purposes only, various equivalent modifications are possible within the spirit and scope of the present invention, as those skilled in the relevant art will recognize and appreciate. As indicated, these modifications may be made to the present invention in light of the foregoing description of illustrated embodiments of the present invention and are to be included within the spirit and scope of the present invention.
  • Thus, while the present invention has been described herein with reference to particular embodiments thereof, a latitude of modification, various changes and substitutions are intended in the foregoing disclosures, and it will be appreciated that in some instances some features of embodiments of the invention will be employed without a corresponding use of other features without departing from the scope and spirit of the invention as set forth. Therefore, many modifications may be made to adapt a particular situation or material to the essential scope and spirit of the present invention. It is intended that the invention not be limited to the particular terms used in following claims and/or to the particular embodiment disclosed as the best mode contemplated for carrying out this invention, but that the invention will include any and all embodiments and equivalents falling within the scope of the appended claims.

Claims (1)

1. A method of enabling a device to participate in an electronic conference, the method comprising:
enabling an electronic conferencing client to interface with a virtual wall clock that maintains synchronization with other electronic conferencing participants using a time stamp and an average clock offset between source clocks on the other electronic conferencing participants;
receiving a first stream from a first client and a second stream from a second client in the other electronic conferencing participants;
accessing a first marker from the first stream and a second marker from the second stream;
configuring, using the first marker and the second marker, the electronic conferencing client to update the virtual wall clock; and
transmitting, using the electronic conferencing client, a source stream with an updated marker that reflects an updated time for the virtual wall clock.
US12/987,535 2005-11-16 2011-01-10 Method and system for in-band signaling of multiple media streams Active US8208460B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US12/987,535 US8208460B2 (en) 2005-11-16 2011-01-10 Method and system for in-band signaling of multiple media streams

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US11/281,104 US7869420B2 (en) 2005-11-16 2005-11-16 Method and system for in-band signaling of multiple media streams
US12/987,535 US8208460B2 (en) 2005-11-16 2011-01-10 Method and system for in-band signaling of multiple media streams

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US11/281,104 Continuation US7869420B2 (en) 2005-11-16 2005-11-16 Method and system for in-band signaling of multiple media streams

Publications (2)

Publication Number Publication Date
US20110167174A1 true US20110167174A1 (en) 2011-07-07
US8208460B2 US8208460B2 (en) 2012-06-26

Family

ID=38040762

Family Applications (2)

Application Number Title Priority Date Filing Date
US11/281,104 Active 2029-04-26 US7869420B2 (en) 2005-11-16 2005-11-16 Method and system for in-band signaling of multiple media streams
US12/987,535 Active US8208460B2 (en) 2005-11-16 2011-01-10 Method and system for in-band signaling of multiple media streams

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US11/281,104 Active 2029-04-26 US7869420B2 (en) 2005-11-16 2005-11-16 Method and system for in-band signaling of multiple media streams

Country Status (1)

Country Link
US (2) US7869420B2 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9628866B2 (en) 2013-03-15 2017-04-18 Ricoh Company, Limited Distribution control system and distribution system
CN107943501A (en) * 2017-11-30 2018-04-20 深圳市东微智能科技股份有限公司 Embedded device upgrade method, device, computer equipment and storage medium
US10250665B2 (en) 2013-03-15 2019-04-02 Ricoh Company, Limited Distribution control system, distribution system, distribution control method, and computer-readable storage medium

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080175275A1 (en) * 2007-01-22 2008-07-24 Samsung Electronics Co., Ltd. Time synchronization method between nodes in network and apparatus for implementing the same
EP2073548A1 (en) * 2007-12-17 2009-06-24 Alcatel Lucent Method for synchronizing at least two streams
WO2009099366A1 (en) * 2008-02-05 2009-08-13 Telefonaktiebolaget L M Ericsson (Publ) A method of transmitting sychnronized speech and video
EP2124449A1 (en) * 2008-05-19 2009-11-25 THOMSON Licensing Device and method for synchronizing an interactive mark to streaming content
US10282373B2 (en) * 2009-04-17 2019-05-07 Excalibur Ip, Llc Subject-based vitality
US8140702B2 (en) * 2009-12-28 2012-03-20 General Instrument Corporation System and method of maximizing utility in media delivery network
KR101946861B1 (en) 2011-09-21 2019-02-13 삼성전자주식회사 Method and apparatus for synchronizing media data of multimedia broadcast service
ES2796873T3 (en) * 2013-09-20 2020-11-30 Koninklijke Kpn Nv Map timeline information between media streams
CN105723723B (en) * 2013-09-20 2019-01-08 皇家Kpn公司 Make timeline information interrelated between Media Stream
US9866883B2 (en) 2016-06-02 2018-01-09 Biamp Systems Corporation Dynamic delay equalization for media transport
US10887385B2 (en) 2017-09-20 2021-01-05 Akamai Technologies, Inc. Marker based reporting system for hybrid content delivery network and peer to peer network

Citations (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3735049A (en) * 1970-04-10 1973-05-22 Philips Corp Telecommunication system with time division multiplex
US5537409A (en) * 1993-07-16 1996-07-16 Pioneer Electronic Corporation Synchronizing system for time-divided video and audio signals
US5602992A (en) * 1993-11-29 1997-02-11 Intel Corporation System for synchronizing data stream transferred from server to client by initializing clock when first packet is received and comparing packet time information with clock
US5694397A (en) * 1994-09-29 1997-12-02 Gpt Limited Constant bit rate synchronization
US5844600A (en) * 1995-09-15 1998-12-01 General Datacomm, Inc. Methods, apparatus, and systems for transporting multimedia conference data streams through a transport network
US20020131398A1 (en) * 2001-03-13 2002-09-19 Fantasma Maintaining a global time reference among a group of networked devices
US6480902B1 (en) * 1999-05-25 2002-11-12 Institute For Information Industry Intermedia synchronization system for communicating multimedia data in a computer network
US20030072269A1 (en) * 2001-10-11 2003-04-17 Nippon Telegraph And Telephone Corporation Data transmission control method, program therefor and data transmission unit using the same
US20030110508A1 (en) * 2001-12-11 2003-06-12 Raj Bridgelall Dual transceiver module for use with imager and video cameras
US20030135635A1 (en) * 1996-03-08 2003-07-17 Levi Steven P. Active stream format for holding multiple media streams
US20040052275A1 (en) * 2002-09-13 2004-03-18 Tomokazu Murakami Recording apparatus, video camera and computer program
US20040090994A1 (en) * 2000-07-20 2004-05-13 Lockridge Terry Wayne Multi-media jitter removal in an asynchronous digital home network
US20040125787A1 (en) * 2002-12-31 2004-07-01 May Michael R. Method and apparatus for synchronized channel transmission
US20050074007A1 (en) * 2003-07-29 2005-04-07 Samuels Allen R. Transaction boundary detection for reduction in timeout penalties
US6898642B2 (en) * 2000-04-17 2005-05-24 International Business Machines Corporation Synchronous collaboration based on peer-to-peer communication
US20050169233A1 (en) * 2004-06-30 2005-08-04 Sharp Laboratories Of America, Inc. System clock synchronization in an ad hoc and infrastructure wireless networks
US20050175040A1 (en) * 2004-02-10 2005-08-11 Holborow Clive E. Method and system for multiplexing DOCSIS data into an MPEG transport stream
US20050190777A1 (en) * 2003-06-30 2005-09-01 Hess Kendal M. Simultaneous multiple channel receiver
US20050238057A1 (en) * 2002-06-26 2005-10-27 Tadamasa Toma Multiplexing device and demultiplexing device
US20060222126A1 (en) * 2005-03-31 2006-10-05 Stratus Technologies Bermuda Ltd. Systems and methods for maintaining synchronicity during signal transmission
US7188353B1 (en) * 1999-04-06 2007-03-06 Sharp Laboratories Of America, Inc. System for presenting synchronized HTML documents in digital television receivers
US20070071037A1 (en) * 2005-09-28 2007-03-29 Avaya Technology Corp. Synchronization watermarking in multimedia streams
US7433327B2 (en) * 2003-10-09 2008-10-07 Hewlett-Packard Development Company, L.P. Method and system for coordinating communication devices to create an enhanced representation of an ongoing event
US7558224B1 (en) * 2003-07-29 2009-07-07 Cisco Technology, Inc. Management of packet-based audio devices within acoustic spaces
US7792158B1 (en) * 2004-08-18 2010-09-07 Atheros Communications, Inc. Media streaming synchronization

Patent Citations (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3735049A (en) * 1970-04-10 1973-05-22 Philips Corp Telecommunication system with time division multiplex
US5537409A (en) * 1993-07-16 1996-07-16 Pioneer Electronic Corporation Synchronizing system for time-divided video and audio signals
US5602992A (en) * 1993-11-29 1997-02-11 Intel Corporation System for synchronizing data stream transferred from server to client by initializing clock when first packet is received and comparing packet time information with clock
US5694397A (en) * 1994-09-29 1997-12-02 Gpt Limited Constant bit rate synchronization
US5844600A (en) * 1995-09-15 1998-12-01 General Datacomm, Inc. Methods, apparatus, and systems for transporting multimedia conference data streams through a transport network
US20030135635A1 (en) * 1996-03-08 2003-07-17 Levi Steven P. Active stream format for holding multiple media streams
US7188353B1 (en) * 1999-04-06 2007-03-06 Sharp Laboratories Of America, Inc. System for presenting synchronized HTML documents in digital television receivers
US6480902B1 (en) * 1999-05-25 2002-11-12 Institute For Information Industry Intermedia synchronization system for communicating multimedia data in a computer network
US6898642B2 (en) * 2000-04-17 2005-05-24 International Business Machines Corporation Synchronous collaboration based on peer-to-peer communication
US20040090994A1 (en) * 2000-07-20 2004-05-13 Lockridge Terry Wayne Multi-media jitter removal in an asynchronous digital home network
US20020131398A1 (en) * 2001-03-13 2002-09-19 Fantasma Maintaining a global time reference among a group of networked devices
US20030072269A1 (en) * 2001-10-11 2003-04-17 Nippon Telegraph And Telephone Corporation Data transmission control method, program therefor and data transmission unit using the same
US20030110508A1 (en) * 2001-12-11 2003-06-12 Raj Bridgelall Dual transceiver module for use with imager and video cameras
US20050238057A1 (en) * 2002-06-26 2005-10-27 Tadamasa Toma Multiplexing device and demultiplexing device
US20040052275A1 (en) * 2002-09-13 2004-03-18 Tomokazu Murakami Recording apparatus, video camera and computer program
US20040125787A1 (en) * 2002-12-31 2004-07-01 May Michael R. Method and apparatus for synchronized channel transmission
US20050190777A1 (en) * 2003-06-30 2005-09-01 Hess Kendal M. Simultaneous multiple channel receiver
US20050074007A1 (en) * 2003-07-29 2005-04-07 Samuels Allen R. Transaction boundary detection for reduction in timeout penalties
US7558224B1 (en) * 2003-07-29 2009-07-07 Cisco Technology, Inc. Management of packet-based audio devices within acoustic spaces
US7433327B2 (en) * 2003-10-09 2008-10-07 Hewlett-Packard Development Company, L.P. Method and system for coordinating communication devices to create an enhanced representation of an ongoing event
US20050175040A1 (en) * 2004-02-10 2005-08-11 Holborow Clive E. Method and system for multiplexing DOCSIS data into an MPEG transport stream
US20050169233A1 (en) * 2004-06-30 2005-08-04 Sharp Laboratories Of America, Inc. System clock synchronization in an ad hoc and infrastructure wireless networks
US7792158B1 (en) * 2004-08-18 2010-09-07 Atheros Communications, Inc. Media streaming synchronization
US20060222126A1 (en) * 2005-03-31 2006-10-05 Stratus Technologies Bermuda Ltd. Systems and methods for maintaining synchronicity during signal transmission
US20070071037A1 (en) * 2005-09-28 2007-03-29 Avaya Technology Corp. Synchronization watermarking in multimedia streams

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9628866B2 (en) 2013-03-15 2017-04-18 Ricoh Company, Limited Distribution control system and distribution system
US10250665B2 (en) 2013-03-15 2019-04-02 Ricoh Company, Limited Distribution control system, distribution system, distribution control method, and computer-readable storage medium
CN107943501A (en) * 2017-11-30 2018-04-20 深圳市东微智能科技股份有限公司 Embedded device upgrade method, device, computer equipment and storage medium

Also Published As

Publication number Publication date
US20070110107A1 (en) 2007-05-17
US8208460B2 (en) 2012-06-26
US7869420B2 (en) 2011-01-11

Similar Documents

Publication Publication Date Title
US8208460B2 (en) Method and system for in-band signaling of multiple media streams
US7843974B2 (en) Audio and video synchronization
US7664057B1 (en) Audio-to-video synchronization system and method for packet-based network video conferencing
US7545794B2 (en) Timestamping network controller for streaming media applications
US9055332B2 (en) Lip synchronization in a video conference
EP1398931B1 (en) Synchronous play-out of media data packets
US7243150B2 (en) Reducing the access delay for transmitting processed data over transmission data
US7764713B2 (en) Synchronization watermarking in multimedia streams
KR100750669B1 (en) Method and device for multimedia streaming
CN101889422B (en) Method and system for synchronizing the output of terminals
EP2728830B1 (en) Method and system for synchronizing audio and video streams in media relay conferencing
US20080259966A1 (en) Synchronization of one or more source RTP streams at multiple receiver destinations
US20070047590A1 (en) Method for signaling a device to perform no synchronization or include a synchronization delay on multimedia stream
JP2006014299A (en) Synchronizing method of video/audio data of mobile communication terminal
KR20060114080A (en) System and method of providing multimedia streaming service
US10554704B2 (en) Low latency media mixing in packet networks
CN112584216A (en) Lip sound synchronization method and device
WO2011057588A1 (en) Streaming media server and method for synchronization in media stream switching
JP4042396B2 (en) Data communication system, data transmission apparatus, data reception apparatus and method, and computer program
JP3906678B2 (en) Data communication system, data transmission apparatus, data reception apparatus and method, and computer program
Jung et al. A client-driven media synchronization mechanism for RTP packet-based video streaming
Lauri et al. Synchronization of streamed audio between multiple playback devices over an unmanaged IP network
Malmgren et al. Synchronization of streamed audio between multiple playback devices over an unmanaged IP network
Ebnöther TV 2.0–Your Individual TV Experience!
Liang et al. Real-time communication over networks

Legal Events

Date Code Title Description
STCF Information on status: patent grant

Free format text: PATENTED CASE

FEPP Fee payment procedure

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

FPAY Fee payment

Year of fee payment: 4

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 8

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 12