US20040086268A1 - Decoder buffer for streaming video receiver and method of operation - Google Patents
Decoder buffer for streaming video receiver and method of operation Download PDFInfo
- Publication number
- US20040086268A1 US20040086268A1 US10/658,097 US65809703A US2004086268A1 US 20040086268 A1 US20040086268 A1 US 20040086268A1 US 65809703 A US65809703 A US 65809703A US 2004086268 A1 US2004086268 A1 US 2004086268A1
- Authority
- US
- United States
- Prior art keywords
- buffer
- decoder
- region
- data
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/24—Systems for the transmission of television signals using pulse code modulation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/637—Control signals issued by the client directed to the server or network components
- H04N21/6375—Control signals issued by the client directed to the server or network components for requesting retransmission, e.g. of data packets lost or corrupted during transmission from server
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/23406—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving management of server-side video buffer
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/235—Processing of additional data, e.g. scrambling of additional data or processing content descriptors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/435—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/44004—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving video buffer management, e.g. video decoder buffer or video display buffer
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/637—Control signals issued by the client directed to the server or network components
- H04N21/6377—Control signals issued by the client directed to the server or network components directed to server
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/65—Transmission of management data between client and server
- H04N21/658—Transmission by the client directed to the server
Definitions
- the present invention is directed, in general, to video processing systems and, more specifically, to a decoder buffer for use in a streaming video receiver.
- IP Internet protocol
- NACK negative automatic repeat request
- the present invention is embodied in an Integrated Transport Decoder (ITD) buffer model.
- ITD Integrated Transport Decoder
- One key advantage of the ITD model is that it eliminates the separation of a network-transport buffer, which is typically used for removing delay jitter and recovering lost data, from the video decoder buffer. This can significantly reduce the end-to-end delay, and optimize the usage of receiver resources (such as memory).
- Each of the access units is capable of holding at least one data packet associated with a selected frame in the streaming video.
- the decoder buffer comprises: 1) a first buffer region comprising at least one access unit capable of storing data packets that are less immediately needed by the video decoder; and 2) a re-transmission region comprising at least one access unit capable of storing data packets that are most immediately needed by the video decoder, wherein the decoder buffer, in response to a detection of a missing data packet in the re-transmission region requests that the streaming video transmitter retransmit the missing packet.
- At least one of the data packets are stored in the first buffer region for a period of time equal to a start-up delay time of the decoder buffer.
- the data packets are first stored in the first buffer region and are shifted into the re-transmission region.
- the first buffer region is separate from the re-transmission region.
- the first buffer region overlaps at least a portion of the re-transmission region.
- the first buffer region overlaps all of the re-transmission region.
- the first buffer region is separated from the re-transmission region by a second buffer region in which a late data packet is late with respect to an expected time of arrival of the late data packet, but is not sufficiently late to require a re-transmission of the late data packet.
- FIG. 1 illustrates an end-to-end transmission of streaming video from a streaming video transmitter through a data network to an exemplary streaming video receiver according to one embodiment of the present invention
- FIG. 2 illustrates an ideal encoder-decoder model of a video coding system
- FIG. 3 illustrates end-to-end transmission of streaming video from a compressed video source through a channel to an exemplary integrated transport decoder buffer and video decoder, without support for re-transmission, according to one embodiment of the present invention.
- FIG. 4 illustrates a sequence diagram showing the flow of data packets through different and distinct regions of exemplary ideal integrated transport decoder buffer.
- FIG. 5 illustrates a sequence diagram showing the flow of data packets through different over-lapping regions of exemplary integrated transport decoder buffer configured for the maximum outer boundary range.
- FIGS. 1 through 5 discussed below, and the various embodiments used to describe the principles of the present invention in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the invention. Those skilled in the art will understand that the principles of the present invention may be implemented in any suitably arranged streaming video receiver.
- FIG. 1 illustrates an end-to-end transmission of streaming video from streaming video transmitter 110 through data network 120 to streaming video receiver 130 , according to one embodiment of the present invention.
- streaming video transmitter 110 may be any one of a wide variety of sources of video frames, including a data network server, a television station, a cable network, a desktop personal computer (PC), or the like.
- Streaming video transmitter 110 comprises video frame source 112 , video encoder 114 and encoder buffer 116 .
- Video frame source 112 may be any device capable of generating a sequence of uncompressed video frames, including a television antenna and receiver unit, a video cassette player, a video camera, a disk storage device capable of storing a “raw” video clip, and the like.
- the uncompressed video frames enter video encoder 114 at a given picture rate (or “streaming rate”) and are compressed according to any known compression algorithm or device, such as an MPEG-4 encoder. Video encoder 114 then transmits the compressed video frames to encoder buffer 116 for buffering in preparation for transmission across data network 120 .
- Data network 120 may be any suitable IP network and may include portions of both public data networks, such as the Internet, and private data networks, such as an enterprise-owned local area network (LAN) or wide area network (WAN).
- Streaming video receiver 130 comprises decoder buffer 131 , video decoder 134 and video display 136 .
- Decoder buffer 131 receives and stores streaming compressed video frames from data network 120 . Decoder buffer 131 then transmits the compressed video frames to video decoder 134 as required.
- Video decoder 134 decompresses the video frames at the same rate (ideally) at which the video frames were compressed by video encoder 114 .
- Decoder buffer 131 further comprises integrated transport decoder (ITD) buffer 132 , ITD buffer monitor 138 and re-transmission controller 139 .
- ITD buffer 132 integrates both temporal and data-unit occupancy considerations in order to provide video decoder 134 with compressed video frames at a rate that is sufficient to avoid underflow conditions, during which video decoder 134 is starved for compressed video frames.
- ITD buffer 132 accomplishes this in cooperation with ITD buffer monitor 138 and re-transmission controller 139 .
- ITD buffer monitor 138 monitors the level of data-occupancy in ITD buffer 132 and detects missing data packets and potential underflow conditions.
- re-transmission controller 139 requests re-transmission of data missing from ITD buffer 132 in order to avoid underflow conditions.
- ITD buffer 132 , ITD buffer monitor 138 , and re-transmission controller 139 are implemented in a personal computer (PC) that receives streaming video and/or audio from, for example, the Internet over a high-speed data line.
- PC personal computer
- ITD buffer 132 may be implemented in main random access memory (RAM) of the PC or in RAM on a video card, and ITD buffer monitor 138 and re-transmission controller 139 may be implemented in the CPU of the PC.
- ITD buffer 132 may be embodied as computer executable instructions stored as a program on storage media 140 , such as a CD-ROM, computer diskette, or similar device, that may be loaded into removable disk port 141 in streaming video receiver 130 .
- Continuous decoding of compressed video frames is a key requirement of a real-time multimedia application, such as streaming video.
- a decoder-encoder buffer model is normally used to ensure that underflow and overflow events do not occur.
- These constraints limit the size (bit-wise) of video pictures that enter the encoder buffer.
- the constraints are usually expressed in terms of encoder-buffer bounds, which when adhered to by the encoder, guarantee continuous decoding and presentation of the compressed video stream at the receiver.
- FIG. 2 shows an ideal encoder-decoder model of a video coding system.
- uncompressed video frames 201 - 203 enter the compression engine of encoder 214 at a given picture-rate, X frames/second, as indicated by the Time(1) line.
- the compressed frames exit encoder 214 and enter encoder buffer 216 at the same X frames/second, as indicated by the Time(2) line.
- the compressed frames exit decoder buffer 216 and enter channel 220 at X frames/second.
- Channel 220 is a generic representation of any transmission medium, such as the Internet, that transfers compressed video frames from a transmitting source to a receiver.
- the delay of channel 220 ( ⁇ c ) is a constant value.
- the compressed frames exit channel 220 and enter decoder buffer 232 at the same X frames/second as at the input and the output of encoder 214 , as indicated by the Time(3) line.
- Decoder buffer 232 transmits the compressed frames to decoder 234 , which decompresses the frames and outputs decompressed frames 251 - 253 at the original X frames/second at which frames entered encoder 214 .
- the end-to-end buffering delay i.e., the total delay encountered in both encoder buffer 216 and decoder buffer 232
- the same piece of compressed video data e.g., a particular byte of the video stream
- encoding in encoder 214 and decoding in decoder 234 are instantaneous and require zero execution time and data packets are not lost.
- the encoder buffer bounds can be expressed using discrete-time summation.
- ⁇ is the end-to-end delay (i.e., including both encoder buffer 216 and decoder buffer 232 and channel delay ⁇ c ) in units of time.
- ⁇ is a constant number applicable to all frames entering the encoder-decoder buffer model.
- N the number of video frames
- the remainder of this disclosure will use time units specified in frame-duration intervals. For example, using the encoder time reference shown in FIG. 2, the n th frame enters encoder buffer 216 at time index “n”. The decoder time-reference of decoder buffer 232 is shifted by the channel delay ( ⁇ c ), with respect to encoder buffer 216 .
- the data rate (r) at the output of encoder (e) 214 during frame-interval “i” may be represented as r e (i).
- data rate is used generically. It could signify bit rate, byte rate, or even packet rate.
- the data rate at the input of decoder buffer 232 may be represented as r d (i).
- r e (iT) r d (iT+ ⁇ c ).
- r e (i) r d (i).
- B max d and B max e are the maximum decoder and encoder buffer sizes respectively.
- ITD buffer 132 minimizes underflow events by taking into consideration the above-described problems of the ideal buffer model and the ideal encoder-decoder buffer constraints. ITD buffer 132 is based on lost packet recovery using re-transmission.
- FIG. 3 is a simplified block diagram of exemplary end-to-end transmission of streaming video, without support for re-transmission.
- streaming video transmitter 110 has been replaced by compressed video source 305 and data network 120 has been replaced by channel 320 .
- Compressed video source 305 transmits data packets at rate r e (n) and channel 320 transmits data packets at rate r td (n). Since video re-transmission is not supported for this embodiment, ITD buffer monitor 138 and re-transmission controller 139 are omitted from the diagram.
- Streaming video receiver 130 has been simplified and is represented by ITD buffer 132 and video decoder 134 .
- ITD buffer 132 integrates temporal and data-unit occupancy models. ITD buffer 132 is divided into temporal segments of ‘T’ seconds each.
- the parameter T may be the frame period in a video sequence.
- the data packets (bits, bytes, or packets) associated with a given duration T are buffered in the corresponding temporal segment. All of the data packets associated with a temporal unit are referred to as an “access” unit.
- data packets 351 , 352 , and 353 comprise access unit A n+1
- data packet 354 comprises access unit A n+2
- data packets 355 and 356 comprise access unit A n+3 .
- the n th access unit, A n is being decoded by decoder 134 and access unit A n+1 is stored at the temporal segment nearest to the output of ITD buffer 132 .
- An access unit may be an audio frame, a video frame, or even a portion of a video frame, such as Group of Blocks (GOB). Therefore, the duration required to decode or display an access unit is the same as the duration of the temporal segment T.
- the rate at which data enters ITD buffer 132 is r td (n). The number of data packets in each access unit are not required to be the same. Compression algorithms used in video encoder 114 may compress the data packets in successive access units by different amounts, even though each access unit represents temporal units of the same duration.
- the three data packets 351 - 353 in access unit A n+1 may comprise a complete video frame, Frame 1 .
- the single data packet 354 in A n+2 may represent only those portions of Frame 2 that are different than Frame 1 . Nonetheless, data packet 354 is sufficient to create Frame 2 if the Frame 1 data is already known. Since Frame 1 and Frame 2 have the same duration, the temporal segment, T, is the same for A n+1 and A n+2 .
- Each temporal segment holds a maximum number of packets, K max , with each packet having a maximum size, b max , (in bits or bytes). Therefore, the maximum size of an access unit, S max , may be represented by S max ⁇ K max (b max ).
- Video encoder 114 is assumed to is begin each access-unit with a new packet that is present only in that access unit.
- the amount of data in ITD buffer 132 at time index n, B td (n), may be described by terms of B a (n) and B b (n).
- B a (n) represents the number of consecutive-and-complete access units in ITD buffer 132 at the beginning of interval n
- B b (n) represents the total consecutive amount of data in ITD buffer 132 at the end of interval n.
- T ⁇ B a (n) represents how much video in temporal units (e.g. seconds) that the ITD buffer 132 holds at time index n (without running into an underflow if no more data arrives).
- S j is the maximum size of the access unit for temporal segment j and U B a (n)+1 is the partial (incomplete) data of access unit A n+B a (n)+1 which is stored in temporal segment B a (n)+1 at the beginning of time index n.
- ITD buffer 132 When re-transmission is supported as an embodiment, ITD buffer 132 requires capability for a) outputting one temporal segment (T) worth of data at the beginning of every temporal time-interval n; b) detecting lost packet(s) and transmitting associated negative acknowledge (NACK) messages to the transmitter 110 or 305 ; c) continuously storing newly arrived primary (i.e., not re-transmitted) packets; and d) storing re-transmitted packets.
- the ideal ITD buffer 132 maintains the data rate of the video stream, without delays caused by re-transmission of any lost data.
- encoder buffer 116 may adjust its output data rate r e (n), with a corresponding adjustment by ITD buffer 132 .
- decoder buffer 131 adds buffering for the incoming video stream in order to compensate for the time required for detection and recovery of lost data and for the delay associated with a “real” world implementation. By delaying all incoming video streams by this compensation time, decoder buffer 131 outputs video stream data at a continuous rate as required for decoding.
- Re-transmission controller 139 and ITD buffer 132 incorporate processes for minimizing the time for detecting the absence of packets and transferring NACKs for re-transmission by streaming video transmitter 110 .
- the minimum duration of time needed for detecting a predetermined number of lost packets is represented by T L .
- T L is a function of the delay jitter caused by data arriving later than expected by ITD buffer 132 .
- Time T R includes the time required for streaming video receiver 130 to send a NACK to streaming video transmitter 110 and the time needed for the re-transmitted data to reach streaming video receiver 130 (assuming that the NACK and re-transmitted data are not lost).
- Exemplary decoder buffer 131 transfers a re-transmitted packet with a minimum delay (T L +T R ) for the lost packet interval. If the minimum delay experienced by any video data for an ideal decoder buffer 131 is represented by dd min , the amount of delay ⁇ R that may be added to the minimum ideal delay in order to account for the total delay for re-transmission is:
- Decoder buffer 131 adds delay ⁇ R buffering for all output data to video decoder 134 in order to provide time for decoding and transferring of the data, resulting in continuous video streams. Therefore, the total encoder buffer 116 to decoder buffer 132 output delay ( ⁇ TOT )may be represented by:
- ITD buffer 132 provides buffering (storage) for a minimum number of temporal segments (B min a ) as compensation for re-transmission time requirements and as prevention for an underflow event.
- the ITD buffer 132 sizing may be based, for example, on minimum and maximum boundaries for storing temporal segments. The process for determining these boundaries is described in the following paragraphs.
- the ITD buffer 132 In the absence of lost packets and delay jitter, at any time index n, the ITD buffer 132 provides the following occupancy capability:
- An ideal ITD buffer 132 has a maximum decoding delay (dd max ), where dd max ⁇ ideal . Consequently, in the absence of lost packets and delay jitter, ideal ITD buffer 132 satisfies the following requirement:
- the ideal ITD buffer 132 provides storage requirements for TB a (n) data, bounded as follows:
- ITD buffer 132 storage capability with consideration for delay jitter may be expressed as:
- T E is the delay jitter associated with packets arriving earlier than expected to ITD buffer 132 . Therefore, if B a max is the maximum number of temporal segments that ITD buffer 132 holds, then:
- ITD buffer 132 storage capability is based on the above equations, minimum ideal storage requirements, and delays associated with data transfers.
- ITD buffer 132 has a minimum size determined by ideal encoder buffer 116 which is represented by B max a .
- ITD buffer 132 provides added storage to adjust for delays introduced by ITD buffer 132 and for data arriving earlier than expected.
- ITD buffer 132 storage requirements (in temporal units) for accommodation of these exemplary delays is represented by T extra , as shown below.
- T extra u ( T L +T R ⁇ dd min )+ T E Equation 10
- ITD buffer 132 storage requirement for satisfying the B b max upper limit is shown by the following upper boundary relationship:
- An ideal ITD buffer 132 has a minimum decoding delay (dd min ) which is equal to zero and a maximum decoding delay (dd max ) which is equal to the ideal end-to-end buffering delay ( ⁇ ideal ).
- the ideal ITD buffer 132 is sized to provide extra minimum delay that is equal to T L +T R , where T L and T R are assumed to be integer-multiples of the duration T.
- ideal ITD buffer 132 is found to provide storage for the following number of temporal segments:
- N E [T E /T].
- ITD buffer 132 data enters from the right side of the diagram and exits to the video decoder 134 at the left side.
- the most recently received data is in a buffer area which is labeled “too-early for re-transmission request region” (too-early).
- ITD buffer 132 introduces buffer delays labeled N E , ⁇ N, or N L .
- ITD buffer 132 manages the ideal-buffer region as an ideal video buffer, i.e., data packets flow through this region and are only delayed by the inherent characteristics of the buffer element(s). Ideal ITD buffer 132 provides the remaining too-early buffer areas to compensate for delays associated with the transfer of video streams from streaming video transmitter 110 to decoder 131 (N E ), as well as delays caused by delayed or lost video packets (N L ).
- ITD buffer 132 provides delay N R in the re-transmission region in order to compensate for expected time requirements for the initiation and reception of re-transmission requests.
- Exemplary decoder buffer 131 initiates re-transmission requests during the time periods associated with the re-transmission region.
- N E represents the initial decoding delay (dd f ) which corresponds to the amount of delay encountered by the very first piece of data that enters the buffer prior to the decoding of the first picture (or access unit).
- This dd f is based on, among other things, the streaming video transmitter 110 and data network 120 data transmission rates during elapsed time dd f .
- ITD buffer 132 uses this same data rate for entering received data into its buffer (storage) regions.
- Ideal decoder buffer 131 recognizes the amount of data in its ITD buffer 132 regions just prior to the time that the first access unit is decoded as B 0 d data.
- ideal decoder buffer 131 re-transmission processing is comprised of the following procedures:
- the ideal-buffer region is filled until all data associated with the start-up delay are in the buffer. Since lost events may also occur during this time interval, these data may be treated in a special way, such as by using reliable transmission (e.g. using TCP) for them.
- B k is the amount of data stored in ideal ITD buffer 132 temporal segment k at any instant of time.
- ITD buffer 132 advances the content of all temporal storage segments by one segment toward the buffer output. Subsequently, ideal ITD buffer 132 repeats this process every T units of time. After N L +N R periods of T (i.e. after T L +T R ), decoder 134 starts decoding the first access unit. The time-period that starts when decoding of the first access unit begins is labeled T 1 . Hence, the beginning of any time period n (T n ) represents the time when access unit A n+k is moved to temporal segment k.
- Ideal ITD buffer 132 considers data missing in temporal segment N R of the re-transmission buffer region as lost. This condition occurs when:
- B N R (n) is the amount of data in temporal segment N R at time period n and S j is the size of access unit j.
- ideal ITD buffer 132 determines that data is missing, it sends a re-transmission request to streaming video transmitter 110 .
- Ideal ITD buffer 132 places arriving re-transmitted data into their corresponding temporal segments of the re-transmission region. Assuming the re-transmitted data are received, ideal ITD buffer 132 transfers the re-transmitted data to the video decoder 134 prior to the decoding times of their corresponding access units.
- FIG. 5 is a sequence diagram showing the flow of data packets through different regions of exemplary ITD buffer 132 with over-lap between the ideal buffer, N L , and re-transmission regions.
- ITD buffer 132 is configured for the maximum outer boundary where dd min ⁇ T L +T R , causing its ideal-buffer region to totally over-lap its re-transmission region.
- decoder buffer 131 transfers the received video stream to video decoder 134 after all of the data associated with the start-up delay arrives. Then, video decoder 134 decodes the first access unit without further delays. Decoder buffer 131 performs the re-transmission function as previously described.
- decoder buffer 131 provides data transfer between streaming video transmitter 110 and video decoder 134 for the general case when dd min has a value between the minimum and maximum boundary areas (i.e., when 0 ⁇ dd min ⁇ T L +T R ), with an additional delay of (T L +T R ⁇ dd min ).
Abstract
There is disclosed a decoder buffer capable of receiving streaming video data packets and storing the data packets in a plurality of access units. Each of the access units holds at least one data packet associated with a selected frame in the streaming video. The decoder buffer comprises: 1) a first buffer region comprising at least one access unit for storing data packets that are less immediately needed by the video decoder; and 2) a re-transmission region comprising at least one access unit for storing data packets that are most immediately needed by the video decoder. The decoder buffer, in response to a detection of a missing data packet in the re-transmission region, requests that the streaming video transmitter retransmit the missing packet.
Description
- The present invention is related to that disclosed in U.S. Provisional Patent Application No. 60/108,939, filed on Nov. 18, 1998, entitled “SCALABLE VIDEO STREAMING USING MPEG-4”, which is commonly assigned to the assignee of the present invention. The disclosure of this related provisional patent application is incorporated herein by reference for all purposes as if fully set forth herein.
- The present invention is directed, in general, to video processing systems and, more specifically, to a decoder buffer for use in a streaming video receiver.
- Real-time streaming of multimedia content over Internet protocol (IP) networks has become an increasingly common application in recent years. A wide range of interactive and non-interactive multimedia Internet applications, such as news con-demand, live TV viewing, video conferencing, and many others rely on end-to-end streaming solutions. Unlike a “downloaded” video file, which may be retrieved first in “non-real” time and viewed or played back later, streaming video applications require a video source to encode and to transmit a video signal over a network to a video receiver, which must decode and display the video signal in real time. The receiver relies on a decoder buffer to receive encoded video data packets from the network and to transfer the packets to a video decoder.
- Two problems arise when a streaming video signal is transmitted across a non-guaranteed Quality-of-Service (QoS) network, such as the Internet. First, end-to-end variations in the network (e.g., delay jitter) between the streaming video transmitter and the streaming video receiver mean that the end-to-end delay is not constant. Second, there is usually a significant packet loss rate across non-QoS networks, often requiring re-transmission. The lost data packet must be recovered prior to the time the corresponding frame must be decoded. If not, an underflow event occurs. Furthermore, if prediction-based compression is used, an underflow due to lost data packets may not only impact the current frame being processed, but may affect many subsequent frames.
- It is well-known that re-transmission of lost packets is a viable means of recovery for continuous media communication over packet networks. Many applications use a negative automatic repeat request (NACK) in conjunction with re-transmission of the lost packet. These approaches take into consideration both the round-trip delay and the delay jitter between the sender and the receiver(s).
- For example, an end-to-end model with re-transmission for packet voice transmission has been developed. This model takes advantage of the fact that voice data consists of periods of silence separated by brief talk-spurt segments. The model also assumes that each talk-spurt consists of a fixed number of fixed-size packets. However, this model is not general enough to capture the characteristics of compressed video (which can have variable number of bytes or packets per video frame).
- There is therefore a need in the art for improved streaming video receivers that compensate for variations inherent in a non-QoS network. In particular, there is a need for an improved receiver decoder buffer that takes into consideration both transport delay parameters (e.g., end-to-end delay and delay jitter) and video encoder buffer constraints. More particularly, there is a need for an improved decoder buffer that eliminates the separation between the network transport buffer, which is typically used to remove delay jitter and to recover lost data, and the video decoder buffer.
- The present invention is embodied in an Integrated Transport Decoder (ITD) buffer model. One key advantage of the ITD model is that it eliminates the separation of a network-transport buffer, which is typically used for removing delay jitter and recovering lost data, from the video decoder buffer. This can significantly reduce the end-to-end delay, and optimize the usage of receiver resources (such as memory).
- It is a primary object of the present invention to provide, for use with a video decoder capable of decoding streaming video, a decoder buffer capable of receiving from a streaming video transmitter data packets comprising the streaming video and storing the data packets in a plurality of access units. Each of the access units is capable of holding at least one data packet associated with a selected frame in the streaming video. The decoder buffer comprises: 1) a first buffer region comprising at least one access unit capable of storing data packets that are less immediately needed by the video decoder; and 2) a re-transmission region comprising at least one access unit capable of storing data packets that are most immediately needed by the video decoder, wherein the decoder buffer, in response to a detection of a missing data packet in the re-transmission region requests that the streaming video transmitter retransmit the missing packet.
- In one embodiment of the present invention, at least one of the data packets are stored in the first buffer region for a period of time equal to a start-up delay time of the decoder buffer.
- In another embodiment of the present invention, the data packets are first stored in the first buffer region and are shifted into the re-transmission region.
- In still another embodiment of the present invention, the first buffer region is separate from the re-transmission region.
- In yet another embodiment of the present invention, the first buffer region overlaps at least a portion of the re-transmission region.
- In a further embodiment of the present invention, the first buffer region overlaps all of the re-transmission region.
- In a further embodiment of the present invention, the first buffer region is separated from the re-transmission region by a second buffer region in which a late data packet is late with respect to an expected time of arrival of the late data packet, but is not sufficiently late to require a re-transmission of the late data packet.
- The foregoing has outlined rather broadly the features and technical advantages of the present invention so that those skilled in the art may better understand the detailed description of the invention that follows. Additional features and advantages of the invention will be described hereinafter that form the subject of the claims of the invention. Those skilled in the art should appreciate that they may readily use the conception and the specific embodiment disclosed as a basis for modifying or designing other structures for carrying out the same purposes of the present invention. Those skilled in the art should also realize that such equivalent constructions do not depart from the spirit and scope of the invention in its broadest form.
- Before undertaking the DETAILED DESCRIPTION, it may be advantageous to set forth definitions of certain words and phrases used throughout this patent document: the terms “include” and “comprise,” as well as derivatives thereof, mean inclusion without limitation; the term “or,” is inclusive, meaning and/or; the phrases “associated with” and “associated therewith,” as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, or the like; and the term “controller” means any device, system or part thereof that controls at least one operation, such a device may be implemented in hardware, firmware or software, or some combination of at least two of the same. It should be noted that the functionality associated with any particular controller may be centralized or distributed, whether locally or remotely. Definitions for certain words and phrases are provided throughout this patent document, those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.
- For a more complete understanding of the present invention, and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, wherein like numbers designate like objects, and in which:
- FIG. 1 illustrates an end-to-end transmission of streaming video from a streaming video transmitter through a data network to an exemplary streaming video receiver according to one embodiment of the present invention;
- FIG. 2 illustrates an ideal encoder-decoder model of a video coding system;
- FIG. 3 illustrates end-to-end transmission of streaming video from a compressed video source through a channel to an exemplary integrated transport decoder buffer and video decoder, without support for re-transmission, according to one embodiment of the present invention.
- FIG. 4 illustrates a sequence diagram showing the flow of data packets through different and distinct regions of exemplary ideal integrated transport decoder buffer.
- FIG. 5 illustrates a sequence diagram showing the flow of data packets through different over-lapping regions of exemplary integrated transport decoder buffer configured for the maximum outer boundary range.
- FIGS. 1 through 5, discussed below, and the various embodiments used to describe the principles of the present invention in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the invention. Those skilled in the art will understand that the principles of the present invention may be implemented in any suitably arranged streaming video receiver.
- Additionally, those skilled in the art will readily understand that while the embodiment of the present invention described below is principally oriented towards streaming video, this is by way of illustration only. In fact, the improved integrated transport decoder buffer described below may be readily adapted for use in connection with streaming audio data or other streaming data that must be supplied to a decoder at a required rate.
- FIG. 1 illustrates an end-to-end transmission of streaming video from
streaming video transmitter 110 throughdata network 120 to streamingvideo receiver 130, according to one embodiment of the present invention. Depending on the application,streaming video transmitter 110 may be any one of a wide variety of sources of video frames, including a data network server, a television station, a cable network, a desktop personal computer (PC), or the like.Streaming video transmitter 110 comprisesvideo frame source 112,video encoder 114 andencoder buffer 116.Video frame source 112 may be any device capable of generating a sequence of uncompressed video frames, including a television antenna and receiver unit, a video cassette player, a video camera, a disk storage device capable of storing a “raw” video clip, and the like. - The uncompressed video frames enter
video encoder 114 at a given picture rate (or “streaming rate”) and are compressed according to any known compression algorithm or device, such as an MPEG-4 encoder.Video encoder 114 then transmits the compressed video frames toencoder buffer 116 for buffering in preparation for transmission acrossdata network 120.Data network 120 may be any suitable IP network and may include portions of both public data networks, such as the Internet, and private data networks, such as an enterprise-owned local area network (LAN) or wide area network (WAN). - Streaming
video receiver 130 comprisesdecoder buffer 131,video decoder 134 andvideo display 136.Decoder buffer 131 receives and stores streaming compressed video frames fromdata network 120.Decoder buffer 131 then transmits the compressed video frames tovideo decoder 134 as required.Video decoder 134 decompresses the video frames at the same rate (ideally) at which the video frames were compressed byvideo encoder 114. -
Decoder buffer 131 further comprises integrated transport decoder (ITD)buffer 132,ITD buffer monitor 138 andre-transmission controller 139. In accordance with the principles of the present invention,ITD buffer 132 integrates both temporal and data-unit occupancy considerations in order to providevideo decoder 134 with compressed video frames at a rate that is sufficient to avoid underflow conditions, during whichvideo decoder 134 is starved for compressed video frames. -
ITD buffer 132 accomplishes this in cooperation withITD buffer monitor 138 andre-transmission controller 139. ITD buffer monitor 138 monitors the level of data-occupancy inITD buffer 132 and detects missing data packets and potential underflow conditions. In response to notification fromITD buffer monitor 138,re-transmission controller 139 requests re-transmission of data missing fromITD buffer 132 in order to avoid underflow conditions. In an advantageous embodiment of the present invention,ITD buffer 132,ITD buffer monitor 138, andre-transmission controller 139 are implemented in a personal computer (PC) that receives streaming video and/or audio from, for example, the Internet over a high-speed data line. In such an embodiment,ITD buffer 132 may be implemented in main random access memory (RAM) of the PC or in RAM on a video card, andITD buffer monitor 138 andre-transmission controller 139 may be implemented in the CPU of the PC. To implementITD buffer 132 in a PC environment,ITD buffer 132 may be embodied as computer executable instructions stored as a program onstorage media 140, such as a CD-ROM, computer diskette, or similar device, that may be loaded intoremovable disk port 141 in streamingvideo receiver 130. - Continuous decoding of compressed video frames is a key requirement of a real-time multimedia application, such as streaming video. To meet this requirement, a decoder-encoder buffer model is normally used to ensure that underflow and overflow events do not occur. These constraints limit the size (bit-wise) of video pictures that enter the encoder buffer. The constraints are usually expressed in terms of encoder-buffer bounds, which when adhered to by the encoder, guarantee continuous decoding and presentation of the compressed video stream at the receiver.
- FIG. 2 shows an ideal encoder-decoder model of a video coding system. Under this ideal model, uncompressed video frames201-203 enter the compression engine of
encoder 214 at a given picture-rate, X frames/second, as indicated by the Time(1) line. The compressed frames exitencoder 214 and enterencoder buffer 216 at the same X frames/second, as indicated by the Time(2) line. Similarly, the compressed frames exitdecoder buffer 216 and enterchannel 220 at X frames/second.Channel 220 is a generic representation of any transmission medium, such as the Internet, that transfers compressed video frames from a transmitting source to a receiver. In the ideal case, the delay of channel 220 (δc) is a constant value. - Next, the compressed frames exit
channel 220 and enterdecoder buffer 232 at the same X frames/second as at the input and the output ofencoder 214, as indicated by the Time(3) line.Decoder buffer 232 transmits the compressed frames todecoder 234, which decompresses the frames and outputs decompressed frames 251-253 at the original X frames/second at which frames enteredencoder 214. - Ideally, the end-to-end buffering delay (i.e., the total delay encountered in both
encoder buffer 216 and decoder buffer 232) is constant. However, the same piece of compressed video data (e.g., a particular byte of the video stream) encounters different delays inencoder buffer 216 anddecoder buffer 232. In the ideal model, encoding inencoder 214 and decoding indecoder 234 are instantaneous and require zero execution time and data packets are not lost. - The encoder buffer bounds can be expressed using discrete-time summation. In discrete-time domain analysis, Δ is the end-to-end delay (i.e., including both
encoder buffer 216 anddecoder buffer 232 and channel delay δc) in units of time. For a given video coding system, Δ is a constant number applicable to all frames entering the encoder-decoder buffer model. - To simplify the discrete-time analysis, it is assumed that the end-to-end buffering delay (ΔT=Δ−δc) is an integer-multiple of the frame duration (T). Therefore, NΔ=N(Δ−δc)/T represents the delay of the encoder and decoder buffers in terms of the number of video frames (N). For the purposes of clarity and brevity in describing the principles of the present invention, the remainder of this disclosure will use time units specified in frame-duration intervals. For example, using the encoder time reference shown in FIG. 2, the nth frame enters
encoder buffer 216 at time index “n”. The decoder time-reference ofdecoder buffer 232 is shifted by the channel delay (δc), with respect toencoder buffer 216. -
- where Bmax d and Bmax e are the maximum decoder and encoder buffer sizes respectively.
- In the ideal case, it is also assumed that
encoder 214 starts transmitting data immediately after the first frame entersencoder 214. Therefore, the start-up delay ddf (i.e., the delay time the first piece of data from the first picture spends indecoder buffer 232 prior to decoding) equals the end-to-end, encoder-decoder buffer delay: ddf=ΔT=T·ΔN. - In one embodiment of the present invention,
ITD buffer 132 minimizes underflow events by taking into consideration the above-described problems of the ideal buffer model and the ideal encoder-decoder buffer constraints.ITD buffer 132 is based on lost packet recovery using re-transmission. - FIG. 3 is a simplified block diagram of exemplary end-to-end transmission of streaming video, without support for re-transmission. For the purposes of simplicity and clarity, streaming
video transmitter 110 has been replaced by compressedvideo source 305 anddata network 120 has been replaced bychannel 320.Compressed video source 305 transmits data packets at rate re(n) andchannel 320 transmits data packets at rate rtd(n). Since video re-transmission is not supported for this embodiment,ITD buffer monitor 138 andre-transmission controller 139 are omitted from the diagram. Streamingvideo receiver 130 has been simplified and is represented byITD buffer 132 andvideo decoder 134. - As noted above,
ITD buffer 132 integrates temporal and data-unit occupancy models.ITD buffer 132 is divided into temporal segments of ‘T’ seconds each. By way of example, the parameter T may be the frame period in a video sequence. The data packets (bits, bytes, or packets) associated with a given duration T are buffered in the corresponding temporal segment. All of the data packets associated with a temporal unit are referred to as an “access” unit. By way of example,data packets data packet 354 comprises access unit An+2, anddata packets - During time interval n, the nth access unit, An, is being decoded by
decoder 134 and access unit An+1 is stored at the temporal segment nearest to the output ofITD buffer 132. An access unit may be an audio frame, a video frame, or even a portion of a video frame, such as Group of Blocks (GOB). Therefore, the duration required to decode or display an access unit is the same as the duration of the temporal segment T. During the time-interval n, the rate at which data entersITD buffer 132 is rtd(n). The number of data packets in each access unit are not required to be the same. Compression algorithms used invideo encoder 114 may compress the data packets in successive access units by different amounts, even though each access unit represents temporal units of the same duration. - For example, the three data packets351-353 in access unit An+1 may comprise a complete video frame,
Frame 1. Thesingle data packet 354 in An+2 may represent only those portions ofFrame 2 that are different thanFrame 1. Nonetheless,data packet 354 is sufficient to createFrame 2 if theFrame 1 data is already known. SinceFrame 1 andFrame 2 have the same duration, the temporal segment, T, is the same for An+1 and An+2. - Each temporal segment holds a maximum number of packets, Kmax, with each packet having a maximum size, bmax, (in bits or bytes). Therefore, the maximum size of an access unit, Smax, may be represented by Smax≦Kmax(bmax).
Video encoder 114 is assumed to is begin each access-unit with a new packet that is present only in that access unit. - The amount of data in
ITD buffer 132 at time index n, Btd(n), may be described by terms of Ba(n) and Bb(n). Ba(n) represents the number of consecutive-and-complete access units inITD buffer 132 at the beginning of interval n, and Bb(n) represents the total consecutive amount of data inITD buffer 132 at the end of interval n. For Ba(n), temporal segments containing partial data are not counted, and all segments following a partial segment are also not counted even if they contain a complete, access-unit worth of data. Hence, T·Ba(n) represents how much video in temporal units (e.g. seconds) that theITD buffer 132 holds at time index n (without running into an underflow if no more data arrives). -
- where Sj is the maximum size of the access unit for temporal segment j and UB a (n)+1 is the partial (incomplete) data of access unit An+B a (n)+1 which is stored in temporal segment Ba(n)+1 at the beginning of time index n.
- When re-transmission is supported as an embodiment,
ITD buffer 132 requires capability for a) outputting one temporal segment (T) worth of data at the beginning of every temporal time-interval n; b) detecting lost packet(s) and transmitting associated negative acknowledge (NACK) messages to thetransmitter ideal ITD buffer 132 maintains the data rate of the video stream, without delays caused by re-transmission of any lost data. In other words, if re(n) is the transmission data rate used by anidealized video encoder 114 under lossless circumstances,ideal ITD buffer 132 will maintain this date rate without degradation caused by the re-transmission process. Depending upon the number of retransmission requests,encoder buffer 116 may adjust its output data rate re(n), with a corresponding adjustment byITD buffer 132. - In one embodiment,
decoder buffer 131 adds buffering for the incoming video stream in order to compensate for the time required for detection and recovery of lost data and for the delay associated with a “real” world implementation. By delaying all incoming video streams by this compensation time,decoder buffer 131 outputs video stream data at a continuous rate as required for decoding.Re-transmission controller 139 andITD buffer 132 incorporate processes for minimizing the time for detecting the absence of packets and transferring NACKs for re-transmission by streamingvideo transmitter 110. The minimum duration of time needed for detecting a predetermined number of lost packets is represented by TL. In general, TL is a function of the delay jitter caused by data arriving later than expected byITD buffer 132. - The minimum amount of time needed for streaming
video receiver 130 to recover a packet after being declared lost is represented by TR. Time TR includes the time required for streamingvideo receiver 130 to send a NACK to streamingvideo transmitter 110 and the time needed for the re-transmitted data to reach streaming video receiver 130 (assuming that the NACK and re-transmitted data are not lost). -
Exemplary decoder buffer 131 transfers a re-transmitted packet with a minimum delay (TL+TR) for the lost packet interval. If the minimum delay experienced by any video data for anideal decoder buffer 131 is represented by ddmin, the amount of delay ΔR that may be added to the minimum ideal delay in order to account for the total delay for re-transmission is: - ΔR ≧u(T L +T R −dd min)
Equation 3 - where u(x)=x for x>0, and u(x)=0 for x≦0.
-
Decoder buffer 131 adds delay ΔR buffering for all output data tovideo decoder 134 in order to provide time for decoding and transferring of the data, resulting in continuous video streams. Therefore, thetotal encoder buffer 116 todecoder buffer 132 output delay (ΔTOT)may be represented by: - ΔTOT=Δideal+ΔR≧Δideal +u(T L +T R −dd min)
Equation 4 -
ITD buffer 132 provides buffering (storage) for a minimum number of temporal segments (Bmin a) as compensation for re-transmission time requirements and as prevention for an underflow event. TheITD buffer 132 sizing may be based, for example, on minimum and maximum boundaries for storing temporal segments. The process for determining these boundaries is described in the following paragraphs. - In the absence of lost packets and delay jitter, at any time index n, the
ITD buffer 132 provides the following occupancy capability: - TB a(n)≧TB min a =T L +T R Equation 5
- An
ideal ITD buffer 132 has a maximum decoding delay (ddmax), where ddmax≦Δideal. Consequently, in the absence of lost packets and delay jitter,ideal ITD buffer 132 satisfies the following requirement: - T·B a(n)≦dd max +u(T L +T R −dd min)≦Δideal +u(T L +T R −dd min) Equation 6
- Further, in the absence of lost data and delay jitter, the
ideal ITD buffer 132 provides storage requirements for TBa(n) data, bounded as follows: - T L +T R ≦T·B a(n)≦dd max +u(T L +T R −dd min) Equation 7
-
ITD buffer 132 storage capability with consideration for delay jitter may be expressed as: - T R ≦T·B a(n)≦dd max +u(T L +T R −dd min)+T E Equation 8
- where TE is the delay jitter associated with packets arriving earlier than expected to ITD
buffer 132. Therefore, if Ba max is the maximum number of temporal segments thatITD buffer 132 holds, then: - T·B a max ≦dd max +u(T L +T R dd min)+T E Equation 9
-
-
ITD buffer 132 storage capability is based on the above equations, minimum ideal storage requirements, and delays associated with data transfers.ITD buffer 132 has a minimum size determined byideal encoder buffer 116 which is represented by Bmax a.ITD buffer 132 provides added storage to adjust for delays introduced byITD buffer 132 and for data arriving earlier than expected.ITD buffer 132 storage requirements (in temporal units) for accommodation of these exemplary delays is represented by Textra, as shown below. - T extra =u(T L +T R −dd min)+T E Equation 10
- Using this relationship,
ITD buffer 132 storage requirement for satisfying the Bb max upper limit (in temporal units), is shown by the following upper boundary relationship: - B max b ≧B max d +R max ·T extra =B max d +R max [u(T L +T R −dd min)+T E] Equation 11
- An
ideal ITD buffer 132 has a minimum decoding delay (ddmin) which is equal to zero and a maximum decoding delay (ddmax) which is equal to the ideal end-to-end buffering delay (Δideal). Theideal ITD buffer 132 is sized to provide extra minimum delay that is equal to TL+TR, where TL and TR are assumed to be integer-multiples of the duration T. The minimum time delay requirement is found by substituting the ideal buffer region ddmin=0 and dmax=Δideal into previously described equation for ΔTOT. This extra buffer requirement stores NL+NR temporal segments, where NR=TR/T and N=TL/T. Thus,ideal ITD buffer 132 is found to provide storage for the following number of temporal segments: - B max a ≧N L +N R+┌(T E +dd max)/T┐ Equation 12
- Since the maximum decoding delay, ddmax=Δideal=ΔT, corresponds to ΔN temporal segments, Bb max is further described as follows:
- B a max ≧N R +N L +ΔN=N E Equation 13
- where NE=[TE/T].
- FIG. 4 is a sequence diagram showing the flow of data packets through different regions of
exemplary ITD buffer 132 under the assumption that ddmin=0 (the lower boundary level) and dmax=Δideal.ITD buffer 132 data enters from the right side of the diagram and exits to thevideo decoder 134 at the left side. The most recently received data is in a buffer area which is labeled “too-early for re-transmission request region” (too-early). Depending on the location in the too-early region of the buffer,ITD buffer 132 introduces buffer delays labeled NE, ΔN, or NL. The area of this too-early buffer region which comprises the ideal delay ΔN, is labeled as the ideal-buffer region.ITD buffer 132 manages the ideal-buffer region as an ideal video buffer, i.e., data packets flow through this region and are only delayed by the inherent characteristics of the buffer element(s).Ideal ITD buffer 132 provides the remaining too-early buffer areas to compensate for delays associated with the transfer of video streams from streamingvideo transmitter 110 to decoder 131 (NE), as well as delays caused by delayed or lost video packets (NL). -
ITD buffer 132 provides delay NR in the re-transmission region in order to compensate for expected time requirements for the initiation and reception of re-transmission requests.Exemplary decoder buffer 131 initiates re-transmission requests during the time periods associated with the re-transmission region. - It is important to note that the ideal-buffer and re-transmission regions may overlap, depending on the values of the different delay parameters (ddmin, TR, TL) However, for the exemplary
ideal ITD buffer 132 with ddmin=0, the re-transmission and ideal-buffer regions do not overlap. - For
ITD buffer 132, NE represents the initial decoding delay (ddf) which corresponds to the amount of delay encountered by the very first piece of data that enters the buffer prior to the decoding of the first picture (or access unit). This ddf is based on, among other things, thestreaming video transmitter 110 anddata network 120 data transmission rates during elapsed time ddf. In the ideal case,ITD buffer 132 uses this same data rate for entering received data into its buffer (storage) regions.Ideal decoder buffer 131 recognizes the amount of data in itsITD buffer 132 regions just prior to the time that the first access unit is decoded as B0 d data. This B0 d data, also referred to as “start-up-delay” data, is determined from the following relationship: - When ddmin=0,
ideal decoder buffer 131 re-transmission processing is comprised of the following procedures: - 1. The ideal-buffer region is filled until all data associated with the start-up delay are in the buffer. Since lost events may also occur during this time interval, these data may be treated in a special way, such as by using reliable transmission (e.g. using TCP) for them. The ideal condition for lossless data is satisfied when:
- where Bk is the amount of data stored in
ideal ITD buffer 132 temporal segment k at any instant of time. - 2. After Equation 15 is satisfied,
ITD buffer 132 advances the content of all temporal storage segments by one segment toward the buffer output. Subsequently,ideal ITD buffer 132 repeats this process every T units of time. After NL+NR periods of T (i.e. after TL+TR),decoder 134 starts decoding the first access unit. The time-period that starts when decoding of the first access unit begins is labeled T1. Hence, the beginning of any time period n (Tn) represents the time when access unit An+k is moved to temporal segment k. - 3.
Ideal ITD buffer 132 considers data missing in temporal segment NR of the re-transmission buffer region as lost. This condition occurs when: - B N
R (n)<S n+NR Equation 16 - where BN
R (n) is the amount of data in temporal segment NR at time period n and Sj is the size of access unit j. Whenideal ITD buffer 132 determines that data is missing, it sends a re-transmission request to streamingvideo transmitter 110. - 4.
Ideal ITD buffer 132 places arriving re-transmitted data into their corresponding temporal segments of the re-transmission region. Assuming the re-transmitted data are received,ideal ITD buffer 132 transfers the re-transmitted data to thevideo decoder 134 prior to the decoding times of their corresponding access units. - FIG. 5 is a sequence diagram showing the flow of data packets through different regions of
exemplary ITD buffer 132 with over-lap between the ideal buffer, NL, and re-transmission regions. For this case,ITD buffer 132 is configured for the maximum outer boundary where ddmin≧TL+TR, causing its ideal-buffer region to totally over-lap its re-transmission region. Thus,decoder buffer 131 transfers the received video stream tovideo decoder 134 after all of the data associated with the start-up delay arrives. Then,video decoder 134 decodes the first access unit without further delays.Decoder buffer 131 performs the re-transmission function as previously described. - In a similar manner,
decoder buffer 131, provides data transfer betweenstreaming video transmitter 110 andvideo decoder 134 for the general case when ddmin has a value between the minimum and maximum boundary areas (i.e., when 0<ddmin<TL+TR), with an additional delay of (TL+TR−ddmin). - Although the present invention has been described in detail, those skilled in the art should understand that they can make various changes, substitutions and alterations herein without departing from the spirit and scope of the invention in its broadest form.
Claims (21)
1. For use with a video decoder capable of decoding streaming video, a decoder buffer capable of receiving from a streaming video transmitter data packets comprising said streaming video and storing said data packets in a plurality of access units, each of said access units capable of holding at least one data packet associated with a selected frame in said streaming video, wherein said decoder buffer comprises:
a first buffer region comprising at least one access unit capable of storing data packets that are less immediately needed by said video decoder; and
a re-transmission region comprising at least one access unit capable of storing data packets that are most immediately needed by said video decoder, wherein said decoder buffer, in response to a detection of a missing data packet in said re-transmission region requests that said streaming video transmitter retransmit said missing packet.
2. The decoder buffer set forth in claim 1 wherein at least one of said data packets are stored in said first buffer region for a period of time equal to a start-up delay time of said decoder buffer.
3. The decoder buffer set forth in claim 1 wherein said data packets are first stored in said first buffer region and are shifted into said re-transmission region.
4. The decoder buffer set forth in claim 1 wherein said first buffer region is separate from said re-transmission region.
5. The decoder buffer set forth in claim 1 wherein said first buffer region overlaps at least a portion of said re-transmission region.
6. The decoder buffer set forth in claim 5 wherein said first buffer region overlaps all of said re-transmission region.
7. The decoder buffer set forth in claim 1 wherein said first buffer region is separated from said re-transmission region by a second buffer region in which a late data packet is late with respect to an expected time of arrival of said late data packet, but is not sufficiently late to require a re-transmission of said late data packet.
8. A receiver capable of receiving encoded streaming data comprising:
a device capable of at least one of: 1) displaying streaming video data associated with said encoded streaming data and 2) audibly playing streaming audio data associated with said encoded streaming data;
a decoder capable of decoding said encoded streaming data; and
a decoder buffer capable of receiving from a streaming data transmitter data packets comprising said encoded streaming data and storing said data packets in a plurality of access units, each of said access units capable of holding at least one data packet associated with a selected portion of said encoded streaming data, wherein said decoder buffer comprises:
a first buffer region comprising at least one access unit capable of storing data packets that are less immediately needed by said decoder; and
a re-transmission region comprising at least one access unit capable of storing data packets that are most immediately needed by said decoder, wherein said decoder buffer, in response to a detection of a missing data packet in said re-transmission region requests that said streaming video transmitter retransmit said missing packet.
9. The receiver set forth in claim 8 wherein at least one of said data packets are stored in said first buffer region for a period of time equal to a start-up delay time of said decoder buffer.
10. The receiver set forth in claim 8 wherein said data packets are first stored in said first buffer region and are shifted into said re-transmission region.
11. The receiver set forth in claim 8 wherein said first buffer region is separate from said re-transmission region.
12. The receiver set forth in claim 8 wherein said first buffer region overlaps at least a portion of said re-transmission region.
13. The receiver set forth in claim 12 wherein said first buffer region overlaps all of said re-transmission region.
14. The receiver set forth in claim 8 wherein said first buffer region is separated from said re-transmission region by a second buffer region in which a late data packet is late with respect to an expected time of arrival of said late data packet, but is not sufficiently late to require a re-transmission of said late data packet.
15. For use with a video decoder capable of decoding streaming video, a method of buffering the streaming video comprising the steps of:
receiving from a streaming video transmitter data packets comprising the streaming video and storing the data packets in a plurality of access units in a decoder buffer, each of the access units capable of holding at least one data packet associated with a selected frame in the streaming video;
storing data packets that are less immediately needed by the video decoder in a first buffer region of the decoder buffer comprising at least one access unit capable of storing data packets; and
storing data packets that are most immediately needed by the video decoder in a re-transmission region of the decoder buffer comprising at least one access unit, wherein the decoder buffer, in response to a detection of a missing data packet in the re-transmission region, requests that the streaming video transmitter retransmit the missing packet.
16. The decoder buffer set forth in claim 15 wherein at least one of the data packets are stored in the first buffer region for a period of time equal to a start-up delay time of the decoder buffer.
17. The decoder buffer set forth in claim 15 wherein the data packets are first stored in the first buffer region and are shifted into the re-transmission region.
18. The decoder buffer set forth in claim 15 wherein the first buffer region is separate from the re-transmission region.
19. The decoder buffer set forth in claim 15 wherein the first buffer region overlaps at least a portion of the re-transmission region.
20. The decoder buffer set forth in claim 19 wherein the first buffer region overlaps all of the re-transmission region.
21. The decoder buffer set forth in claim 15 wherein the first buffer region is separated from the re-transmission region by a second buffer region in which a late data packet is late with respect to an expected time of arrival of the late data packet, but is not sufficiently late to require a re-transmission of the late data packet.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/658,097 US20040086268A1 (en) | 1998-11-18 | 2003-09-09 | Decoder buffer for streaming video receiver and method of operation |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10893998P | 1998-11-18 | 1998-11-18 | |
US09/365,463 US6629318B1 (en) | 1998-11-18 | 1999-08-02 | Decoder buffer for streaming video receiver and method of operation |
US10/658,097 US20040086268A1 (en) | 1998-11-18 | 2003-09-09 | Decoder buffer for streaming video receiver and method of operation |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/365,463 Division US6629318B1 (en) | 1998-11-18 | 1999-08-02 | Decoder buffer for streaming video receiver and method of operation |
Publications (1)
Publication Number | Publication Date |
---|---|
US20040086268A1 true US20040086268A1 (en) | 2004-05-06 |
Family
ID=26806445
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/365,463 Expired - Lifetime US6629318B1 (en) | 1998-11-18 | 1999-08-02 | Decoder buffer for streaming video receiver and method of operation |
US10/658,097 Abandoned US20040086268A1 (en) | 1998-11-18 | 2003-09-09 | Decoder buffer for streaming video receiver and method of operation |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/365,463 Expired - Lifetime US6629318B1 (en) | 1998-11-18 | 1999-08-02 | Decoder buffer for streaming video receiver and method of operation |
Country Status (8)
Country | Link |
---|---|
US (2) | US6629318B1 (en) |
EP (1) | EP1050166B1 (en) |
JP (1) | JP4524042B2 (en) |
KR (1) | KR100704134B1 (en) |
CN (1) | CN1171458C (en) |
DE (1) | DE69934092T2 (en) |
ES (1) | ES2277464T3 (en) |
WO (1) | WO2000030356A1 (en) |
Cited By (36)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20040019658A1 (en) * | 2001-03-26 | 2004-01-29 | Microsoft Corporation | Metadata retrieval protocols and namespace identifiers |
US20050010955A1 (en) * | 2003-05-15 | 2005-01-13 | Elia Eric J. | Method and system for playing video |
US20050008074A1 (en) * | 2003-06-25 | 2005-01-13 | Van Beek Petrus J.L. | Wireless video transmission system |
US20050071876A1 (en) * | 2003-09-30 | 2005-03-31 | Van Beek Petrus J. L. | Wireless video transmission system |
US20060095944A1 (en) * | 2004-10-30 | 2006-05-04 | Demircin Mehmet U | Sender-side bandwidth estimation for video transmission with receiver packet buffer |
US20060095943A1 (en) * | 2004-10-30 | 2006-05-04 | Demircin Mehmet U | Packet scheduling for video transmission with sender queue control |
US20060095942A1 (en) * | 2004-10-30 | 2006-05-04 | Van Beek Petrus J | Wireless video transmission system |
US20060212478A1 (en) * | 2005-03-21 | 2006-09-21 | Microsoft Corporation | Methods and systems for generating a subgroup of one or more media items from a library of media items |
US20060218187A1 (en) * | 2005-03-25 | 2006-09-28 | Microsoft Corporation | Methods, systems, and computer-readable media for generating an ordered list of one or more media items |
US20060230065A1 (en) * | 2005-04-06 | 2006-10-12 | Microsoft Corporation | Methods, systems, and computer-readable media for generating a suggested list of media items based upon a seed |
US20060288041A1 (en) * | 2005-06-20 | 2006-12-21 | Microsoft Corporation | Providing community-based media item ratings to users |
US20070016599A1 (en) * | 2005-07-15 | 2007-01-18 | Microsoft Corporation | User interface for establishing a filtering engine |
US20070038672A1 (en) * | 2005-08-11 | 2007-02-15 | Microsoft Corporation | Single action media playlist generation |
US20070039055A1 (en) * | 2005-08-11 | 2007-02-15 | Microsoft Corporation | Remotely accessing protected files via streaming |
US20070067480A1 (en) * | 2005-09-19 | 2007-03-22 | Sharp Laboratories Of America, Inc. | Adaptive media playout by server media processing for robust streaming |
US20070097802A1 (en) * | 2005-10-27 | 2007-05-03 | Microsoft Corporation | Enhanced table of contents (TOC) identifiers |
US20070147371A1 (en) * | 2005-09-26 | 2007-06-28 | The Board Of Trustees Of Michigan State University | Multicast packet video system and hardware |
US20070153916A1 (en) * | 2005-12-30 | 2007-07-05 | Sharp Laboratories Of America, Inc. | Wireless video transmission system |
GB2434512A (en) * | 2006-01-20 | 2007-07-25 | Motorola Inc | Missed content item distribution |
US20070234170A1 (en) * | 2006-03-29 | 2007-10-04 | Samsung Electronics Co., Ltd. | Method and system for communication of video information over wireless channels |
US20070240191A1 (en) * | 2006-03-24 | 2007-10-11 | Samsung Electronics Co., Ltd. | Method and system for transmission of uncompressed video over wireless communication channels |
US20070236599A1 (en) * | 2006-03-31 | 2007-10-11 | Sharp Laboratories Of America, Inc. | Accelerated media coding for robust low-delay video streaming over time-varying and bandwidth limited channels |
US20070244856A1 (en) * | 2006-04-14 | 2007-10-18 | Microsoft Corporation | Media Search Scope Expansion |
US20070245387A1 (en) * | 2006-03-29 | 2007-10-18 | Samsung Electronics Co., Ltd. | Method and system for video stream transmission over wireless channels |
US20080069201A1 (en) * | 2006-09-18 | 2008-03-20 | Sharp Laboratories Of America, Inc. | Distributed channel time allocation for video streaming over wireless networks |
US20090138774A1 (en) * | 2007-11-27 | 2009-05-28 | Samsung Electronics Co., Ltd. | System and method for wireless communication of uncompressed video using selective retransmission |
US20090201380A1 (en) * | 2008-02-12 | 2009-08-13 | Decisive Analytics Corporation | Method and apparatus for streamlined wireless data transfer |
US20090259460A1 (en) * | 2008-04-10 | 2009-10-15 | City University Of Hong Kong | Silence-based adaptive real-time voice and video transmission methods and system |
US7652993B2 (en) | 2006-11-03 | 2010-01-26 | Sharp Laboratories Of America, Inc. | Multi-stream pro-active rate adaptation for robust video transmission |
US20100189182A1 (en) * | 2009-01-28 | 2010-07-29 | Nokia Corporation | Method and apparatus for video coding and decoding |
US20100205128A1 (en) * | 2009-02-12 | 2010-08-12 | Decisive Analytics Corporation | Method and apparatus for analyzing and interrelating data |
US20100235314A1 (en) * | 2009-02-12 | 2010-09-16 | Decisive Analytics Corporation | Method and apparatus for analyzing and interrelating video data |
US8018850B2 (en) | 2004-02-23 | 2011-09-13 | Sharp Laboratories Of America, Inc. | Wireless video transmission system |
US20130064306A1 (en) * | 2011-05-16 | 2013-03-14 | Broadcom Corporation | Variable Link Rate Streaming For Audio And Video Content From Home Media Server |
US20130170561A1 (en) * | 2011-07-05 | 2013-07-04 | Nokia Corporation | Method and apparatus for video coding and decoding |
US10652612B2 (en) | 2014-11-04 | 2020-05-12 | Gt Systems Pty Ltd. | Media distribution and management system and apparatus |
Families Citing this family (82)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
ATE188793T1 (en) | 1994-10-12 | 2000-01-15 | Touchtunes Music Corp | INTELLIGENT SYSTEM FOR NUMERICAL AUDIOVISUAL REPRODUCTION |
US7188352B2 (en) | 1995-07-11 | 2007-03-06 | Touchtunes Music Corporation | Intelligent digital audiovisual playback system |
US7424731B1 (en) | 1994-10-12 | 2008-09-09 | Touchtunes Music Corporation | Home digital audiovisual information recording and playback system |
US8661477B2 (en) | 1994-10-12 | 2014-02-25 | Touchtunes Music Corporation | System for distributing and selecting audio and video information and method implemented by said system |
FR2753868A1 (en) | 1996-09-25 | 1998-03-27 | Technical Maintenance Corp | METHOD FOR SELECTING A RECORDING ON AN AUDIOVISUAL DIGITAL REPRODUCTION SYSTEM AND SYSTEM FOR IMPLEMENTING THE METHOD |
FR2769165B1 (en) | 1997-09-26 | 2002-11-29 | Technical Maintenance Corp | WIRELESS SYSTEM WITH DIGITAL TRANSMISSION FOR SPEAKERS |
FR2781582B1 (en) | 1998-07-21 | 2001-01-12 | Technical Maintenance Corp | SYSTEM FOR DOWNLOADING OBJECTS OR FILES FOR SOFTWARE UPDATE |
FR2781580B1 (en) | 1998-07-22 | 2000-09-22 | Technical Maintenance Corp | SOUND CONTROL CIRCUIT FOR INTELLIGENT DIGITAL AUDIOVISUAL REPRODUCTION SYSTEM |
FR2781591B1 (en) | 1998-07-22 | 2000-09-22 | Technical Maintenance Corp | AUDIOVISUAL REPRODUCTION SYSTEM |
US8028318B2 (en) | 1999-07-21 | 2011-09-27 | Touchtunes Music Corporation | Remote control unit for activating and deactivating means for payment and for displaying payment status |
US6629318B1 (en) * | 1998-11-18 | 2003-09-30 | Koninklijke Philips Electronics N.V. | Decoder buffer for streaming video receiver and method of operation |
US8726330B2 (en) | 1999-02-22 | 2014-05-13 | Touchtunes Music Corporation | Intelligent digital audiovisual playback system |
US6499060B1 (en) * | 1999-03-12 | 2002-12-24 | Microsoft Corporation | Media coding for loss recovery with remotely predicted data units |
FR2796482B1 (en) | 1999-07-16 | 2002-09-06 | Touchtunes Music Corp | REMOTE MANAGEMENT SYSTEM FOR AT LEAST ONE AUDIOVISUAL INFORMATION REPRODUCING DEVICE |
US7177278B2 (en) | 1999-12-09 | 2007-02-13 | Broadcom Corporation | Late frame recovery method |
FR2805377B1 (en) | 2000-02-23 | 2003-09-12 | Touchtunes Music Corp | EARLY ORDERING PROCESS FOR A SELECTION, DIGITAL SYSTEM AND JUKE-BOX FOR IMPLEMENTING THE METHOD |
FR2805072B1 (en) | 2000-02-16 | 2002-04-05 | Touchtunes Music Corp | METHOD FOR ADJUSTING THE SOUND VOLUME OF A DIGITAL SOUND RECORDING |
FR2805060B1 (en) | 2000-02-16 | 2005-04-08 | Touchtunes Music Corp | METHOD FOR RECEIVING FILES DURING DOWNLOAD |
FR2808906B1 (en) | 2000-05-10 | 2005-02-11 | Touchtunes Music Corp | DEVICE AND METHOD FOR REMOTELY MANAGING A NETWORK OF AUDIOVISUAL INFORMATION REPRODUCTION SYSTEMS |
FR2811175B1 (en) | 2000-06-29 | 2002-12-27 | Touchtunes Music Corp | AUDIOVISUAL INFORMATION DISTRIBUTION METHOD AND AUDIOVISUAL INFORMATION DISTRIBUTION SYSTEM |
FR2811114B1 (en) | 2000-06-29 | 2002-12-27 | Touchtunes Music Corp | DEVICE AND METHOD FOR COMMUNICATION BETWEEN A SYSTEM FOR REPRODUCING AUDIOVISUAL INFORMATION AND AN ELECTRONIC ENTERTAINMENT MACHINE |
FR2814085B1 (en) | 2000-09-15 | 2005-02-11 | Touchtunes Music Corp | ENTERTAINMENT METHOD BASED ON MULTIPLE CHOICE COMPETITION GAMES |
US7444593B1 (en) * | 2000-10-04 | 2008-10-28 | Apple Inc. | Disk space management and clip remainder during edit operations |
FR2818768B1 (en) * | 2000-12-26 | 2003-03-21 | Thomson Multimedia Sa | TEMPORAL STAMPING DEVICE AND METHOD, RECEIVER, DELAYED BROADCAST ASSEMBLY AND PACKAGE SUITE THEREOF |
US7103669B2 (en) * | 2001-02-16 | 2006-09-05 | Hewlett-Packard Development Company, L.P. | Video communication method and system employing multiple state encoding and path diversity |
US7164680B2 (en) * | 2001-06-04 | 2007-01-16 | Koninklijke Philips Electronics N.V. | Scheme for supporting real-time packetization and retransmission in rate-based streaming applications |
WO2003009577A2 (en) * | 2001-07-18 | 2003-01-30 | Polycom Israel Ltd. | A system and method for improving the quality of video communication over a packet-based network |
US7117521B2 (en) * | 2001-08-31 | 2006-10-03 | Intel Corporation | Method to measure the perceived quality of streaming media |
US7646816B2 (en) * | 2001-09-19 | 2010-01-12 | Microsoft Corporation | Generalized reference decoder for image or video processing |
US9646339B2 (en) | 2002-09-16 | 2017-05-09 | Touchtunes Music Corporation | Digital downloading jukebox system with central and local music servers |
US8151304B2 (en) | 2002-09-16 | 2012-04-03 | Touchtunes Music Corporation | Digital downloading jukebox system with user-tailored music management, communications, and other tools |
US11029823B2 (en) | 2002-09-16 | 2021-06-08 | Touchtunes Music Corporation | Jukebox with customizable avatar |
US7822687B2 (en) | 2002-09-16 | 2010-10-26 | Francois Brillon | Jukebox with customizable avatar |
US8332895B2 (en) | 2002-09-16 | 2012-12-11 | Touchtunes Music Corporation | Digital downloading jukebox system with user-tailored music management, communications, and other tools |
US8103589B2 (en) | 2002-09-16 | 2012-01-24 | Touchtunes Music Corporation | Digital downloading jukebox system with central and local music servers |
US8584175B2 (en) | 2002-09-16 | 2013-11-12 | Touchtunes Music Corporation | Digital downloading jukebox system with user-tailored music management, communications, and other tools |
US10373420B2 (en) | 2002-09-16 | 2019-08-06 | Touchtunes Music Corporation | Digital downloading jukebox with enhanced communication features |
US6865627B2 (en) * | 2002-12-27 | 2005-03-08 | Microsoft Corp | Regulating real-time data capture rates to match processor-bound data consumption rates |
US8345754B2 (en) | 2003-09-07 | 2013-01-01 | Microsoft Corporation | Signaling buffer fullness |
US8582659B2 (en) | 2003-09-07 | 2013-11-12 | Microsoft Corporation | Determining a decoding time stamp from buffer fullness |
US7724827B2 (en) | 2003-09-07 | 2010-05-25 | Microsoft Corporation | Multi-layer run level encoding and decoding |
KR100678891B1 (en) * | 2004-07-30 | 2007-02-05 | 삼성전자주식회사 | Method and apparatus for contents' attribute adaptive buffer control in audio-video data receiving |
US8634413B2 (en) | 2004-12-30 | 2014-01-21 | Microsoft Corporation | Use of frame caching to improve packet loss recovery |
TWI289797B (en) * | 2005-02-04 | 2007-11-11 | Via Tech Inc | External digital communication routing module |
US8102878B2 (en) * | 2005-09-29 | 2012-01-24 | Qualcomm Incorporated | Video packet shaping for video telephony |
US8514711B2 (en) * | 2005-10-21 | 2013-08-20 | Qualcomm Incorporated | Reverse link lower layer assisted video error control |
US8406309B2 (en) * | 2005-10-21 | 2013-03-26 | Qualcomm Incorporated | Video rate adaptation to reverse link conditions |
US8842555B2 (en) * | 2005-10-21 | 2014-09-23 | Qualcomm Incorporated | Methods and systems for adaptive encoding of real-time information in packet-switched wireless communication systems |
US8548048B2 (en) * | 2005-10-27 | 2013-10-01 | Qualcomm Incorporated | Video source rate control for video telephony |
US7609709B2 (en) * | 2006-01-23 | 2009-10-27 | Alcatel-Lucent Usa Inc. | Video aware traffic management |
US8451850B2 (en) * | 2006-02-10 | 2013-05-28 | Scott W. Lewis | Method and system for distribution of media including a gigablock |
US8566894B2 (en) * | 2006-02-10 | 2013-10-22 | Scott W. Lewis | Method and system for distribution of media |
US20070192819A1 (en) * | 2006-02-10 | 2007-08-16 | Lewis Scott W | System for distribution of media utilized with a receiver/set top box |
US9740552B2 (en) * | 2006-02-10 | 2017-08-22 | Percept Technologies Inc. | Method and system for error correction utilized with a system for distribution of media |
US8245264B2 (en) * | 2006-05-26 | 2012-08-14 | John Toebes | Methods and systems to reduce channel selection transition delay in a digital network |
US7471218B2 (en) | 2006-09-18 | 2008-12-30 | National Semiconductor Corporation | Methods and systems for efficiently storing and retrieving streaming data |
US8537197B2 (en) * | 2007-01-10 | 2013-09-17 | Qualcomm Incorporated | Content- and link-dependent coding adaptation for multimedia telephony |
US9330529B2 (en) | 2007-01-17 | 2016-05-03 | Touchtunes Music Corporation | Game terminal configured for interaction with jukebox device systems including same, and/or associated methods |
US9171419B2 (en) | 2007-01-17 | 2015-10-27 | Touchtunes Music Corporation | Coin operated entertainment system |
US8767839B2 (en) * | 2007-01-22 | 2014-07-01 | Qualcomm Incorporated | Error filter to differentiate between reverse link and forward link video data errors |
US7693070B2 (en) * | 2007-03-15 | 2010-04-06 | International Business Machines Corporation | Congestion reducing reliable transport packet retry engine |
US9953481B2 (en) | 2007-03-26 | 2018-04-24 | Touchtunes Music Corporation | Jukebox with associated video server |
JP2009027720A (en) | 2007-07-23 | 2009-02-05 | Polycom Inc | System and method executing lost packet recovery with congestion avoidance |
US10290006B2 (en) | 2008-08-15 | 2019-05-14 | Touchtunes Music Corporation | Digital signage and gaming services to comply with federal and state alcohol and beverage laws and regulations |
US8332887B2 (en) | 2008-01-10 | 2012-12-11 | Touchtunes Music Corporation | System and/or methods for distributing advertisements from a central advertisement network to a peripheral device via a local advertisement server |
US8797850B2 (en) * | 2008-01-10 | 2014-08-05 | Qualcomm Incorporated | System and method to adapt to network congestion |
WO2010005569A1 (en) | 2008-07-09 | 2010-01-14 | Touchtunes Music Corporation | Digital downloading jukebox with revenue-enhancing features |
US8108538B2 (en) * | 2008-08-21 | 2012-01-31 | Voltaire Ltd. | Device, system, and method of distributing messages |
KR101748448B1 (en) | 2009-03-18 | 2017-06-16 | 터치튠즈 뮤직 코포레이션 | Entertainment server and associated social networking services |
US9292166B2 (en) | 2009-03-18 | 2016-03-22 | Touchtunes Music Corporation | Digital jukebox device with improved karaoke-related user interfaces, and associated methods |
US10564804B2 (en) | 2009-03-18 | 2020-02-18 | Touchtunes Music Corporation | Digital jukebox device with improved user interfaces, and associated methods |
US10719149B2 (en) | 2009-03-18 | 2020-07-21 | Touchtunes Music Corporation | Digital jukebox device with improved user interfaces, and associated methods |
US9030987B2 (en) | 2009-06-30 | 2015-05-12 | Fosco Bianchetti | Systems and methods for transmission of uninterrupted radio, television programs and additional data services through wireless networks |
CA2881456A1 (en) | 2010-01-26 | 2011-08-04 | Touchtunes Music Corporation | Digital jukebox device with improved user interfaces, and associated methods |
FR2959636B1 (en) * | 2010-04-28 | 2012-07-13 | Canon Kk | METHOD FOR ACCESSING A SPATIO-TEMPORAL PART OF A VIDEO SEQUENCE OF IMAGES |
KR101672253B1 (en) * | 2010-12-14 | 2016-11-03 | 삼성전자주식회사 | Apparatus and method for providing streaming service in portable terminal |
GB2522772B (en) | 2011-09-18 | 2016-01-13 | Touchtunes Music Corp | Digital jukebox device with karaoke and/or photo booth features, and associated methods |
US11151224B2 (en) | 2012-01-09 | 2021-10-19 | Touchtunes Music Corporation | Systems and/or methods for monitoring audio inputs to jukebox devices |
CN103327315B (en) * | 2012-03-22 | 2016-12-21 | 上海算芯微电子有限公司 | Dynamically can rollback bit stream buffer modular system and method |
WO2015070070A1 (en) | 2013-11-07 | 2015-05-14 | Touchtunes Music Corporation | Techniques for generating electronic menu graphical user interface layouts for use in connection with electronic devices |
EP3123293A4 (en) | 2014-03-25 | 2017-09-27 | Touchtunes Music Corporation | Digital jukebox device with improved user interfaces, and associated methods |
US11252344B2 (en) | 2017-12-27 | 2022-02-15 | Adasky, Ltd. | Method and system for generating multiple synchronized thermal video streams for automotive safety and driving systems |
Citations (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5109384A (en) * | 1988-11-02 | 1992-04-28 | Tseung Lawrence C N | Guaranteed reliable broadcast network |
US5550847A (en) * | 1994-10-11 | 1996-08-27 | Motorola, Inc. | Device and method of signal loss recovery for realtime and/or interactive communications |
US5652627A (en) * | 1994-09-27 | 1997-07-29 | Lucent Technologies Inc. | System and method for reducing jitter in a packet-based transmission network |
US5666484A (en) * | 1988-09-14 | 1997-09-09 | Hitachi, Ltd. | Control method for distributed processing system |
US5680322A (en) * | 1994-05-30 | 1997-10-21 | Matsushita Electric Industrial Co., Ltd. | Method and apparatus for dynamic image data transmission |
US5751883A (en) * | 1995-06-07 | 1998-05-12 | International Business Machines Corporation | Multimedia direct access storage device and formatting method |
US5768527A (en) * | 1996-04-23 | 1998-06-16 | Motorola, Inc. | Device, system and method of real-time multimedia streaming |
US5768533A (en) * | 1995-09-01 | 1998-06-16 | National Semiconductor Corporation | Video coding using segmented frames and retransmission to overcome channel errors |
US5918002A (en) * | 1997-03-14 | 1999-06-29 | Microsoft Corporation | Selective retransmission for efficient and reliable streaming of multimedia packets in a computer network |
US5930480A (en) * | 1996-10-10 | 1999-07-27 | Apple Computer, Inc. | Software architecture for controlling data streams based on linked command blocks |
US6026097A (en) * | 1997-03-13 | 2000-02-15 | 8 X 8, Inc. | Data processor having controlled scalable input data source and method thereof |
US6031818A (en) * | 1997-03-19 | 2000-02-29 | Lucent Technologies Inc. | Error correction system for packet switching networks |
US6032180A (en) * | 1996-09-26 | 2000-02-29 | Fujitsu Limited | Image data transmission system, video server unit, and client unit for displaying image data |
US6104757A (en) * | 1998-05-15 | 2000-08-15 | North Carolina State University | System and method of error control for interactive low-bit rate video transmission |
US6141385A (en) * | 1996-03-28 | 2000-10-31 | Nec Corporation | MPEG coded picture decoding apparatus |
US6172983B1 (en) * | 1997-03-13 | 2001-01-09 | Siemens Information And Communication Networks, Inc. | Hub dominated method and system for managing network collisions |
US6275471B1 (en) * | 1998-05-12 | 2001-08-14 | Panasonic Technologies, Inc. | Method for reliable real-time multimedia streaming |
US6310897B1 (en) * | 1996-09-02 | 2001-10-30 | Kabushiki Kaisha Toshiba | Information transmitting method, encoder/decoder of information transmitting system using the method, and encoding multiplexer/decoding inverse multiplexer |
US6629318B1 (en) * | 1998-11-18 | 2003-09-30 | Koninklijke Philips Electronics N.V. | Decoder buffer for streaming video receiver and method of operation |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5481543A (en) * | 1993-03-16 | 1996-01-02 | Sony Corporation | Rational input buffer arrangements for auxiliary information in video and audio signal processing systems |
EP0735776B1 (en) * | 1995-03-29 | 2004-01-28 | Hitachi, Ltd. | Decoder for compressed and multiplexed video and audio data |
US5623483A (en) * | 1995-05-11 | 1997-04-22 | Lucent Technologies Inc. | Synchronization system for networked multimedia streams |
US5822524A (en) * | 1995-07-21 | 1998-10-13 | Infovalue Computing, Inc. | System for just-in-time retrieval of multimedia files over computer networks by transmitting data packets at transmission rate determined by frame size |
-
1999
- 1999-08-02 US US09/365,463 patent/US6629318B1/en not_active Expired - Lifetime
- 1999-11-18 KR KR1020007007867A patent/KR100704134B1/en not_active IP Right Cessation
- 1999-11-18 DE DE69934092T patent/DE69934092T2/en not_active Expired - Lifetime
- 1999-11-18 CN CNB998039969A patent/CN1171458C/en not_active Expired - Fee Related
- 1999-11-18 JP JP2000583253A patent/JP4524042B2/en not_active Expired - Fee Related
- 1999-11-18 ES ES99972409T patent/ES2277464T3/en not_active Expired - Lifetime
- 1999-11-18 WO PCT/EP1999/008927 patent/WO2000030356A1/en active IP Right Grant
- 1999-11-18 EP EP99972409A patent/EP1050166B1/en not_active Expired - Lifetime
-
2003
- 2003-09-09 US US10/658,097 patent/US20040086268A1/en not_active Abandoned
Patent Citations (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5666484A (en) * | 1988-09-14 | 1997-09-09 | Hitachi, Ltd. | Control method for distributed processing system |
US5109384A (en) * | 1988-11-02 | 1992-04-28 | Tseung Lawrence C N | Guaranteed reliable broadcast network |
US5680322A (en) * | 1994-05-30 | 1997-10-21 | Matsushita Electric Industrial Co., Ltd. | Method and apparatus for dynamic image data transmission |
US5652627A (en) * | 1994-09-27 | 1997-07-29 | Lucent Technologies Inc. | System and method for reducing jitter in a packet-based transmission network |
US5550847A (en) * | 1994-10-11 | 1996-08-27 | Motorola, Inc. | Device and method of signal loss recovery for realtime and/or interactive communications |
US5751883A (en) * | 1995-06-07 | 1998-05-12 | International Business Machines Corporation | Multimedia direct access storage device and formatting method |
US5768533A (en) * | 1995-09-01 | 1998-06-16 | National Semiconductor Corporation | Video coding using segmented frames and retransmission to overcome channel errors |
US6141385A (en) * | 1996-03-28 | 2000-10-31 | Nec Corporation | MPEG coded picture decoding apparatus |
US6085252A (en) * | 1996-04-23 | 2000-07-04 | Motorola Inc. | Device, system and method for real-time multimedia streaming |
US5768527A (en) * | 1996-04-23 | 1998-06-16 | Motorola, Inc. | Device, system and method of real-time multimedia streaming |
US6310897B1 (en) * | 1996-09-02 | 2001-10-30 | Kabushiki Kaisha Toshiba | Information transmitting method, encoder/decoder of information transmitting system using the method, and encoding multiplexer/decoding inverse multiplexer |
US6032180A (en) * | 1996-09-26 | 2000-02-29 | Fujitsu Limited | Image data transmission system, video server unit, and client unit for displaying image data |
US5930480A (en) * | 1996-10-10 | 1999-07-27 | Apple Computer, Inc. | Software architecture for controlling data streams based on linked command blocks |
US6026097A (en) * | 1997-03-13 | 2000-02-15 | 8 X 8, Inc. | Data processor having controlled scalable input data source and method thereof |
US6172983B1 (en) * | 1997-03-13 | 2001-01-09 | Siemens Information And Communication Networks, Inc. | Hub dominated method and system for managing network collisions |
US5918002A (en) * | 1997-03-14 | 1999-06-29 | Microsoft Corporation | Selective retransmission for efficient and reliable streaming of multimedia packets in a computer network |
US6031818A (en) * | 1997-03-19 | 2000-02-29 | Lucent Technologies Inc. | Error correction system for packet switching networks |
US6275471B1 (en) * | 1998-05-12 | 2001-08-14 | Panasonic Technologies, Inc. | Method for reliable real-time multimedia streaming |
US6104757A (en) * | 1998-05-15 | 2000-08-15 | North Carolina State University | System and method of error control for interactive low-bit rate video transmission |
US6629318B1 (en) * | 1998-11-18 | 2003-09-30 | Koninklijke Philips Electronics N.V. | Decoder buffer for streaming video receiver and method of operation |
Cited By (61)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20040019658A1 (en) * | 2001-03-26 | 2004-01-29 | Microsoft Corporation | Metadata retrieval protocols and namespace identifiers |
US20050010955A1 (en) * | 2003-05-15 | 2005-01-13 | Elia Eric J. | Method and system for playing video |
US9176643B2 (en) * | 2003-05-15 | 2015-11-03 | Comcast Cable Holdings, Llc | Method and system for playing video |
US7274740B2 (en) | 2003-06-25 | 2007-09-25 | Sharp Laboratories Of America, Inc. | Wireless video transmission system |
US20050008074A1 (en) * | 2003-06-25 | 2005-01-13 | Van Beek Petrus J.L. | Wireless video transmission system |
US20050071876A1 (en) * | 2003-09-30 | 2005-03-31 | Van Beek Petrus J. L. | Wireless video transmission system |
US9325998B2 (en) | 2003-09-30 | 2016-04-26 | Sharp Laboratories Of America, Inc. | Wireless video transmission system |
US8018850B2 (en) | 2004-02-23 | 2011-09-13 | Sharp Laboratories Of America, Inc. | Wireless video transmission system |
US7784076B2 (en) | 2004-10-30 | 2010-08-24 | Sharp Laboratories Of America, Inc. | Sender-side bandwidth estimation for video transmission with receiver packet buffer |
US7797723B2 (en) | 2004-10-30 | 2010-09-14 | Sharp Laboratories Of America, Inc. | Packet scheduling for video transmission with sender queue control |
US20060095942A1 (en) * | 2004-10-30 | 2006-05-04 | Van Beek Petrus J | Wireless video transmission system |
US20060095943A1 (en) * | 2004-10-30 | 2006-05-04 | Demircin Mehmet U | Packet scheduling for video transmission with sender queue control |
US20060095944A1 (en) * | 2004-10-30 | 2006-05-04 | Demircin Mehmet U | Sender-side bandwidth estimation for video transmission with receiver packet buffer |
US8356327B2 (en) | 2004-10-30 | 2013-01-15 | Sharp Laboratories Of America, Inc. | Wireless video transmission system |
US7756388B2 (en) | 2005-03-21 | 2010-07-13 | Microsoft Corporation | Media item subgroup generation from a library |
US20060212478A1 (en) * | 2005-03-21 | 2006-09-21 | Microsoft Corporation | Methods and systems for generating a subgroup of one or more media items from a library of media items |
US20060218187A1 (en) * | 2005-03-25 | 2006-09-28 | Microsoft Corporation | Methods, systems, and computer-readable media for generating an ordered list of one or more media items |
US20060230065A1 (en) * | 2005-04-06 | 2006-10-12 | Microsoft Corporation | Methods, systems, and computer-readable media for generating a suggested list of media items based upon a seed |
US7533091B2 (en) | 2005-04-06 | 2009-05-12 | Microsoft Corporation | Methods, systems, and computer-readable media for generating a suggested list of media items based upon a seed |
US7890513B2 (en) | 2005-06-20 | 2011-02-15 | Microsoft Corporation | Providing community-based media item ratings to users |
US20060288041A1 (en) * | 2005-06-20 | 2006-12-21 | Microsoft Corporation | Providing community-based media item ratings to users |
US7580932B2 (en) | 2005-07-15 | 2009-08-25 | Microsoft Corporation | User interface for establishing a filtering engine |
US20070016599A1 (en) * | 2005-07-15 | 2007-01-18 | Microsoft Corporation | User interface for establishing a filtering engine |
US20070039055A1 (en) * | 2005-08-11 | 2007-02-15 | Microsoft Corporation | Remotely accessing protected files via streaming |
US20070038672A1 (en) * | 2005-08-11 | 2007-02-15 | Microsoft Corporation | Single action media playlist generation |
US7680824B2 (en) | 2005-08-11 | 2010-03-16 | Microsoft Corporation | Single action media playlist generation |
US7681238B2 (en) | 2005-08-11 | 2010-03-16 | Microsoft Corporation | Remotely accessing protected files via streaming |
US20070067480A1 (en) * | 2005-09-19 | 2007-03-22 | Sharp Laboratories Of America, Inc. | Adaptive media playout by server media processing for robust streaming |
US20070147371A1 (en) * | 2005-09-26 | 2007-06-28 | The Board Of Trustees Of Michigan State University | Multicast packet video system and hardware |
US20070097802A1 (en) * | 2005-10-27 | 2007-05-03 | Microsoft Corporation | Enhanced table of contents (TOC) identifiers |
US7688686B2 (en) | 2005-10-27 | 2010-03-30 | Microsoft Corporation | Enhanced table of contents (TOC) identifiers |
US9544602B2 (en) | 2005-12-30 | 2017-01-10 | Sharp Laboratories Of America, Inc. | Wireless video transmission system |
US20070153916A1 (en) * | 2005-12-30 | 2007-07-05 | Sharp Laboratories Of America, Inc. | Wireless video transmission system |
GB2434512A (en) * | 2006-01-20 | 2007-07-25 | Motorola Inc | Missed content item distribution |
GB2434512B (en) * | 2006-01-20 | 2008-03-12 | Motorola Inc | Content Item Distribution |
US20070240191A1 (en) * | 2006-03-24 | 2007-10-11 | Samsung Electronics Co., Ltd. | Method and system for transmission of uncompressed video over wireless communication channels |
US8363675B2 (en) | 2006-03-24 | 2013-01-29 | Samsung Electronics Co., Ltd. | Method and system for transmission of uncompressed video over wireless communication channels |
US20070234170A1 (en) * | 2006-03-29 | 2007-10-04 | Samsung Electronics Co., Ltd. | Method and system for communication of video information over wireless channels |
US8432938B2 (en) | 2006-03-29 | 2013-04-30 | Samsung Electronics Co., Ltd. | Method and system for video stream transmission over wireless channels |
US20070245387A1 (en) * | 2006-03-29 | 2007-10-18 | Samsung Electronics Co., Ltd. | Method and system for video stream transmission over wireless channels |
US20070236599A1 (en) * | 2006-03-31 | 2007-10-11 | Sharp Laboratories Of America, Inc. | Accelerated media coding for robust low-delay video streaming over time-varying and bandwidth limited channels |
US7652994B2 (en) | 2006-03-31 | 2010-01-26 | Sharp Laboratories Of America, Inc. | Accelerated media coding for robust low-delay video streaming over time-varying and bandwidth limited channels |
US20070244856A1 (en) * | 2006-04-14 | 2007-10-18 | Microsoft Corporation | Media Search Scope Expansion |
US8861597B2 (en) | 2006-09-18 | 2014-10-14 | Sharp Laboratories Of America, Inc. | Distributed channel time allocation for video streaming over wireless networks |
US20080069201A1 (en) * | 2006-09-18 | 2008-03-20 | Sharp Laboratories Of America, Inc. | Distributed channel time allocation for video streaming over wireless networks |
US7652993B2 (en) | 2006-11-03 | 2010-01-26 | Sharp Laboratories Of America, Inc. | Multi-stream pro-active rate adaptation for robust video transmission |
US8205126B2 (en) | 2007-11-27 | 2012-06-19 | Samsung Electronics Co., Ltd. | System and method for wireless communication of uncompressed video using selective retransmission |
US20090138774A1 (en) * | 2007-11-27 | 2009-05-28 | Samsung Electronics Co., Ltd. | System and method for wireless communication of uncompressed video using selective retransmission |
US20090201380A1 (en) * | 2008-02-12 | 2009-08-13 | Decisive Analytics Corporation | Method and apparatus for streamlined wireless data transfer |
US20090259460A1 (en) * | 2008-04-10 | 2009-10-15 | City University Of Hong Kong | Silence-based adaptive real-time voice and video transmission methods and system |
US8438016B2 (en) * | 2008-04-10 | 2013-05-07 | City University Of Hong Kong | Silence-based adaptive real-time voice and video transmission methods and system |
US20100189182A1 (en) * | 2009-01-28 | 2010-07-29 | Nokia Corporation | Method and apparatus for video coding and decoding |
WO2010086501A1 (en) * | 2009-01-28 | 2010-08-05 | Nokia Corporation | Method and apparatus for video coding and decoding |
US20100235314A1 (en) * | 2009-02-12 | 2010-09-16 | Decisive Analytics Corporation | Method and apparatus for analyzing and interrelating video data |
US8458105B2 (en) | 2009-02-12 | 2013-06-04 | Decisive Analytics Corporation | Method and apparatus for analyzing and interrelating data |
US20100205128A1 (en) * | 2009-02-12 | 2010-08-12 | Decisive Analytics Corporation | Method and apparatus for analyzing and interrelating data |
US20130064306A1 (en) * | 2011-05-16 | 2013-03-14 | Broadcom Corporation | Variable Link Rate Streaming For Audio And Video Content From Home Media Server |
US20130170561A1 (en) * | 2011-07-05 | 2013-07-04 | Nokia Corporation | Method and apparatus for video coding and decoding |
US10652612B2 (en) | 2014-11-04 | 2020-05-12 | Gt Systems Pty Ltd. | Media distribution and management system and apparatus |
AU2015342719B2 (en) * | 2014-11-04 | 2020-08-20 | Gt Systems Pty Ltd | Media distribution and management system and apparatus |
AU2020213277B2 (en) * | 2014-11-04 | 2022-07-07 | Gt Systems Pty Ltd | Media distribution and management system and apparatus |
Also Published As
Publication number | Publication date |
---|---|
JP2002530948A (en) | 2002-09-17 |
JP4524042B2 (en) | 2010-08-11 |
US6629318B1 (en) | 2003-09-30 |
EP1050166B1 (en) | 2006-11-22 |
EP1050166A1 (en) | 2000-11-08 |
ES2277464T3 (en) | 2007-07-01 |
DE69934092D1 (en) | 2007-01-04 |
CN1171458C (en) | 2004-10-13 |
KR20010034213A (en) | 2001-04-25 |
DE69934092T2 (en) | 2007-06-21 |
CN1293871A (en) | 2001-05-02 |
WO2000030356A1 (en) | 2000-05-25 |
KR100704134B1 (en) | 2007-04-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6629318B1 (en) | Decoder buffer for streaming video receiver and method of operation | |
US6700893B1 (en) | System and method for controlling the delay budget of a decoder buffer in a streaming data receiver | |
US5918002A (en) | Selective retransmission for efficient and reliable streaming of multimedia packets in a computer network | |
US7707303B2 (en) | Method and devices for controlling retransmissions in data streaming | |
US8516346B2 (en) | Packet transmission apparatus, communication system and program | |
EP1361690B1 (en) | Method and apparatus for retransmitting data packets based on channel conditions | |
EP1397899B1 (en) | Real-time packetization and retransmission in streaming applications | |
US8306058B2 (en) | Communication system and techniques for transmission from source to destination | |
US20050152397A1 (en) | Communication system and techniques for transmission from source to destination | |
JPH09191314A (en) | Continuous data transmission method and its transmitter | |
EP1301041A1 (en) | Video data transmission method and apparatus | |
US7530089B1 (en) | System and method for improving video quality using a constant bit rate data stream | |
EP1947859A1 (en) | Video transmission method and system | |
Huszák et al. | TFRC-Based Selective Retransmission for Multimedia Applications. | |
KR100701032B1 (en) | Video data transmission control system for network and method therefore | |
JP2001148717A (en) | Data server device | |
Riley et al. | Low latency video communications over high bandwidth-delay networks using FEC | |
KR20080027622A (en) | Apparatus and method for video on demand service of duplex communication television |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO PAY ISSUE FEE |