METHOD AND APPARATUS FOR DECODING PACKETIZED DATA
A method for decoding a packetized video signal including at least one encoded frame. In one case, the method includes receiving at least one FEC packet at a receiving station. The receiving station uses embedded data associated with the FEC packet to obtain more accurate knowledge of the packet loss state of the media packets. This improved knowledge can allow the receiver to make better use of packet retransmission requests. The embedded data associated with the FEC packet can include in some cases a base sequence number and a packet mask.
Latest Google Patents:
- Thermal Mitigation for An Electronic Speaker Device and Associated Apparatuses and Methods
- NETWORK ADDRESS TRANSLATION FOR VIRTUAL MACHINES
- LINK MARGIN IMPROVEMENTS USING A VARIABLE PHYSICAL LAYER SYMBOL RATE
- Multi-Output Decoders for Multi-Task Learning of ASR and Auxiliary Tasks
- BROWSING HIERARCHICAL DATASETS
This application is related to co-pending application Ser. No. ______ (Attorney Docket No. GOGL-266-A) filed concurrently herewith and entitled “METHOD AND APPARATUS FOR REQUESTING RETRANSMISSION OF MISSING PACKETS IN DATA COMMUNICATIONS” which is hereby incorporated by reference in its entirety.
TECHNICAL FIELDThe present invention relates to the field of data communications including techniques for error correction.
BACKGROUNDMany kinds of data are transmitted over the Internet and other networks, including video and audio data. Data can for example be transmitted from one computer or other transmitting station to another remote computer or other receiving station. Data transmission over networks such as the Internet is frequently accomplished by packetizing the message to be transmitted—that is, by dividing the message into packets which are reassembled at the receiving end to reconstruct the original message.
Packets may be lost or delayed during transmission, resulting in corruption of the message. This can be especially problematic when it occurs during real time transmission of data (such as during voice over IP (VOIP) session or video conferencing). A video frame with lost or missing packets is referred to as an incomplete or partial frame.
Methods have been proposed to address the problem of packet loss. These methods include forward error correction (hereinafter “FEC” coding) and negative acknowledgment requests (hereinafter, “NACK” requests). FEC coding involves the transmitting station generating redundant FEC packets. These redundant FEC packets are transmitted along with the underlying source (i.e., video or other media) packets. The FEC packets can be used by a receiving station to reconstruct packets that are lost during transmission. Generally speaking, NACK requests are sent by the receiving station to request retransmission of lost packets that the receiving station cannot reconstruct using FEC packets. The receiving station can detect lost packets by identifying gaps in the packet sequence numbers of the packets that are received. The packet sequence numbers can be included in a packet header, such as an RTP header.
SUMMARYEmbodiments are disclosed for decoding correcting errors in data communications. One aspect of the disclosed embodiments is a method for decoding a packetized video signal including at least one encoded frame. The method includes receiving the packetized video signal over a network. The packetized video signal has at least one received packet associated with the encoded frame and having embedded data. The method also includes identifying at least one lost packet that is missing from the packetized video signal as received over the network. At least one packet loss state value based at least in part upon at least a portion of the embedded data.
In accordance with another aspect of the disclosed embodiments, an apparatus is provided for decoding a packetized signal that has at least one encoded frame including a first packet with packet header information. The apparatus includes a memory and a processor configured to execute instructions stored in the memory. As explained in the detailed description, memory and processor, although referred to in the singular, can each include one or more devices (including devices of different type) such as parallel processors or multicore processors. The processor is programmed to identify, by means of a gap in the sequence of packet sequence numbers in the packetized signal, a lost packet that is missing from the packetized signal. The processor determines whether the lost packet is likely to be a source packet or an error correction packet, based on the packet header information of the first packet.
In accordance with another aspect of the disclosed embodiments, a computer program product is provided that includes computer readable medium having instructions encoded thereon. As explained in the detailed description, the computer readable media can include random access memory, a hard drive, removable media or any other memory device. The instructions cause the processor to receive the packetized video signal including at least a first packet associated with the at least one encoded frame and having a packet header information. The instructions also cause the processor to identify a second packet that is missing from the packetized video signal as received; and determine whether the second packet is likely to be a source packet or an error correction packet, based on the packet header information of the first packet. The packet header information is at least one of a packet sequence number, a base sequence number or a packet mask.
The description herein makes reference to the accompanying drawings wherein like reference numerals refer to like parts throughout the several views, and wherein:
Disclosed are methods and apparatuses for selectively retransmitting lost packets. One drawback to retransmission of video data is that the request for the retransmission of packets can result in a delay as the receiver waits for lost packets to complete an incomplete or partial frame. Another drawback of retransmission is that network congestion is sometimes increased if too many lost packets are retransmitted. It is the case, however, that in a video transmission, some packets are of a nature that their loss does not degrade video quality to an unacceptable level. For example, a partial frame might be capable of decoding with an acceptable quality level even though one of the packets containing part of the frame has been lost. Even if there is some degradation in quality from decoding a partial frame, end users may rather endure that degradation than experience a delay in rendering the frame (which delay might be occasioned by waiting for retransmission of the missing packets), particularly in the case of real time video communications.
Thus, in some of the embodiments described below, a determination is made by a receiving station, a transmitting station, or both to selectively request or re-transmit lost packets based on an assessment that considers the cost of the retransmission and the quality of decoding partial frames. In some embodiments, the assessment can be made by using a function of the distortion occasioned by decoding a frame having a missing packet and the delay of re-transmitting the missing packet.
In making a determination as to whether a missing packet should be re-transmitted, it is useful to have information about the missing packet. For example, when both NACK and FEC are used in a video transmission, it is sometimes the case that FEC packets are lost in transmission. In some cases, the receiving station not need to request retransmission of the lost FEC packet, such as for example when the underlying source packets have been successfully transmitted. If the missing packet were known to be an FEC packet, then a determination could be made to forego retransmission. Also, it is useful to know which frame or partition the missing packets are from to determine whether the partial frame can be decoded with an acceptable degradation in quality, such as during bursty packet loss or when the first or last packet of a frame is lost. The term “packet loss state” is used herein to refer to information about packet loss, such as the number of packets lost and/or the frames (or, in some cases, partitions within frames) from which packets are lost, whether lost packets are source or error correcting packets. In some of the embodiments described below, the packet loss state can be determined at least in part using information embedded in the received FEC packets. This information can be used, along with other inputs, to selectively determine whether a missing packet or packets should be re-transmitted.
Memories 20 and 24 are random access memory (RAM) although any other suitable type of storage device can be used. Generally, processors 18, 22 can receive program instructions and data from memory 20, 24, which can be used by the processor for performing the methods described below.
Although
In this example, transmitting station 12 and receiving station 14 are used to communicate in real time to permit parties at each station to engage in a videoconference over the Internet. The designation of a station 12 or 14 as “receiving” or “transmitting” is arbitrary and for purposes of illustration. With two-way real time communication, one station 12, 14 will be the transmitting station with respect to a particular message, and the other station 12, 14 will be the receiving station; but these roles can be reversed depending on which station is transmitting a particular message.
Transmitting station 12 and receiving station 14 can be implemented in a wide variety of configurations, including for example on servers in a video conference system. Alternatively, transmitting station 12 can be implemented on a server and receiving station 14 can be implemented on a mobile device such as a mobile telephone or other hand-held communications or computing device. Alternatively, both transmitting station 12 and receiving station 14 can be hand-held devices. Alternatively, receiving station 14 can be a stationary personal computer rather than a mobile device.
FEC packets 40 and 42 are generated by transmitting station 12 using a packet mask.
For example, referring to
Referring still to
Referring still to
Recovered media packets 81 which are generated as output by FEC decoder stage 80 are accepted as input by de-packetization stage 82 and by NACK list generator 88. NACK list generator 88 in turn generates as output a NACK list 85 based on FEC header data 83 received as input from FEC decoder stage 80, and the packet header data of the recovered packets 81. NACK list 85 is received as input by retransmission selector stage 86. The functions of NACK list generator stage 88 and retransmission selector stage 86 are described below.
De-packetization stage 82 re-assembles recovered media packets 81 to generate as output an encoded stream 90 corresponding to encoded stream 74 generated by encoder in transmitting station. Note that encoded stream 90 may not include all of the data in encoded stream 74 as some packets of packetized output signal 78 have been lost and remain unrecovered. Encoded stream 90 is accepted as input to video decoder stage 84. Video decoder stage 84 decodes encoded stream 90 in accordance with the video encoding method used by transmitting station to generate an output video that is suitable for rendition on a display (not shown), which may be a peripheral to receiving station 14.
Video decoder stage 84 also generates as output a video property signal 92 which is accepted as input by retransmission selector stage 86. Video property 92 includes decode state information as well as video properties, which can include any property of the rendered video such as for example play-back rate. Retransmission selector stage 86 also accepts as input recovered media packets 81 (or in some cases just the packet header data of recovered media packets 81) from FEC decoder 80 and NACK list 85 from NACK list generator stage 88. As explained below in more detail, retransmission selector stage 86 determines which, if any packets originally included in packetized output signal 78 have been lost and are missing from media packets 81 and determines whether to request retransmission of those lost packets. If it is determined if to request retransmission of a lost packets then retransmission selector stage 86 generates as output a NACK or other retransmission request 94. Retransmission request 94 is transmitted via network 16 to transmitting station 12.
Back at transmitting station 12, retransmission request 94 is received as input to retransmission logic stage 70. As mentioned above, retransmission logic stage 70 also receives as input the packetized output signal 78 generated by FEC encoder stage 68. Other inputs can be accepted by retransmission logic stage 70 including inputs pertaining to the state of network 16. As described below, retransmission logic stage 70 determines whether to accept retransmission request 94. If retransmission logic stage 70 determines to accept retransmission request 94, then it generates as output retransmitted packets 96 which are retransmitted to receiving station 14 via network 16. Retransmitted packets 96 can optionally be protected with FEC coding.
The operation of the NACK list generator 88 is explained in reference to exemplary transmissions of packetized video data as shown in
Referring to
Referring to
Referring to
The base sequence number field (
In some exemplary embodiments, there are two implementations of base sequence numbers, namely equal protection (illustrated in
Because all of FEC packets 116 illustrated in
Referring to
FEC packets 124 exemplify unequal protection because the source packets “00” through “04” are protected by differing numbers of FEC packets 124. In particular, two of FEC packets 124 (containing the packet sequence numbers “05” and “06”) each contain a base sequence number of “00” and a five-bit mask which includes that FEC packet “05” protects source packets “00” and “01” and FEC packet “06” protects only source packet “01”. The remaining one of FEC packets 124 (packet “07”) has a base sequence number of “02” and a five bit mask which indicates that FEC packet “07” protects “02”, “03” and “04”. Summarizing, source packet “01” is protected by one FEC packet (“05”), source packet “02” is protected by all three FEC packets, source packet “03” is protected by two FEC packets (“06” and “07”) and source packets “03” and “04” are protected by one FEC packet (“07”). This type of base sequence number setting can be used to let receiving station 14 determine, in some cases, the number of packets of the first partition. For example, if FEC packet 07 is received, and at least one of FEC packet 05 or 06 (or the receiver knows by some other means that the first packet sequence number is 00), then the receiver can deduce that there are 2 first partition packets (base sequence number 02−base sequence number 00).
Operation of the NACK list generator stage 88 (
For example, NACK list generator stage 88 searches portion 110′ for packets, 128 that were received in portion 110′ and that have packet sequence numbers that are higher than the packet sequence numbers of the missing packets “07” and “08.” NACK list generator stage 88 scans packets 128 for base sequence numbers that fall within the range of the packet sequence numbers of the missing packets (i.e., “07” and “08”). In this example, NACK list generator stage 88 determines that one FEC packet 128 (specifically, packet “11”) has a base sequence number equal to “08”. It can thus be determined that missing packet “08” is a source packet, and it is the first packet of a frame (because it corresponds to the base sequence number). The NACK list generator 88 also determines that FEC packet “11” belongs to Frame 01 (this information is also in the FEC packet's header via the frame time-stamp data). Because FEC packet “11” belongs to Frame 01, the NACK list generator stage 88 can also determine that missing packet “08” belongs to Frame 01. With the state of the missing packet “08” resolved, the NACK list generator 88 can determine that the preceding missing packet (i.e., the missing packet with packet sequence number “07”) belongs to the preceding frame, Frame 00, because missing packet “07” precedes the first packet (“08”) in Frame 01. Because FEC packets in this case are grouped together at the end of each frame, it can be further determined that missing packet “07” is also an FEC packet because the packet (“06”) that immediate proceeds packet “07” is itself an FEC packet.
The operation of NACK list generator stage 88 is further explained by way a second example in reference to
For example, NACK list generator stage 88 searches portion 110″ for packets 130 that were received in portion 110″ and that have packet sequence numbers that are higher than the packet sequence numbers of the missing packets (i.e., “00” and “01”). NACK list generator stage 88 scans packets 130 for base sequence numbers that fall within the range of the packet sequence number of the missing packets “00” and “01”. NACK list generator stage 88 determines that packet “05” has a base sequence number of “00”. It can thus be determined that missing packet “00” is a source packet (because it corresponds to the base sequence number) and also the first packet of frame 00, since the NACK list generator stage 88 also determines that FEC packet “05” belongs to Frame 00 (this information is also in the FEC packet's header via the frame time-stamp). By further inspecting the mask of FEC packet “05,” NACK list generator stage 88 can determine whether the next missing packet (“01”) is also a source frame packet. For example, in this case the mask value of FEC packet is “1 1 1 1 1” (as shown in
As exemplified above, a packet mask can be used to determine a lower bound on the number of packets in the frame. The lower bound of the frame packet number (n) is determined as the nth most significant bit of the FEC packet. From all the FEC packets received for the same frame (i.e., all FEC packets with same time-stamp), receiving station 14 will take the largest n value as the lower bound on frame packet number. This quantity is very useful for NACK list generation, and for estimating the distortion of the incomplete frame.
Thus, by inspection of the header information of those FEC packets that are successfully transmitted, NACK list generator stage 88 can deduce with some degree of confidence (although, in some cases, not absolute certainty) whether missing packets are FEC or source packets to which frame or the missing packets belong, whether the missing packet is the first packet of the frame, and lower bound of the number of packets in the frame. Furthermore in the case of unequal protection, the number of packets for the first partition may also be deduced (e.g.,
Referring to
Generally speaking, retransmission selector stage 86 is designed to request retransmission of packets by balancing the cost of a delay of the retransmission against the benefit of reduced distortion. The delay-distortion cost function can be applied to the whole frame to do selective retransmission at the frame level, or may be applied at the packet level for selective retransmission of a subset of missing packets. Retransmission logic of a different form has been employed in some conventional video systems. Such systems accept as inputs only the network state parameters such as round-trip time (“RTT”) and play-back time (“PB”) of the video renderer. For example, if RTT is low and PB is large, then retransmission can be requested (since ample time is available for the play-back). However, in some applications such as real-time video conferencing, play-back time is usually small. Conventional retransmission approaches not necessarily use selective re-transmission and do not always yield good results.
In one exemplary embodiment, retransmission selector stage 86 determines whether to request transmitting station 12 to re-transmit lost packets based on an assessment that considers the cost of the retransmission delay and the quality of decoding partial frames. Still referring to
From these inputs, retransmission selector stage 86 can derive one or more different types of information that it uses to determine whether to request retransmission of missing packets. For example, retransmission selector stage 86 can incorporate in its decision-making logic video content state information derived from the content of the video signal itself, including video resolution, an estimate of the encoding rate (from the average size of decoded frames, content classes) or frame type (e.g., key or delta frame).
Other types of information which can be derived from inputs recovered media packets 81, FEC header data 85 and video properties 92 by retransmission selector 86 include packet loss state. Packet loss state information can include one or more parameters such as:
-
- N: estimate of the total number of source packets in the current frame
- M: estimate of the number of missing/lost packets
- P: relative importance score for each missing packet
The relative importance score (P) for each missing packet can be based on the type of data contained in the packet. For example, a packet containing data from the first partition (which may include motion vectors and other relatively more important data) can be assigned a higher importance score. The relative importance score of the packet can also be a function of the type of frame, with reference frames affording their packets a higher importance score. The specific values used in the importance score will depend on particular implementations. In one implementation, the relative importance score P may be computed as a distortion value (as in Equation 2) computed on a per-packet basis. The importance score may be computed based on offline-computed empirical data.
Error concealment (EC) knowledge is another type of information that can be used by retransmission selector 86. Both sending station 12 and receiving station 14 are aware of the error concealment (EC) algorithm used by the decoder. This awareness may be used in estimating the quality degradation to the decoded frame which would result from the packet loss state. The packet loss cost function may be conditioned, for example, on whether the EC is effective at recovering motion vectors (first partition data), or residual data (second or higher partition data). This effectiveness may in turn be a function of the number or percentage of missing packets in the frame, the frame's resolution and the frame type (e.g., key or delta frame). An additional variant can be the state of the decoder (e.g., whether or not the previous frame was decoded successfully, as this would impact the quality of the recovered frame), which is related to error propagation discussed below.
Error propagation (EP) potential is another exemplary type of information that retransmission selector 86 can use to decide whether to request retransmission of a missing packet. Error propagation potential refers to the potential of a missing packet to affect the propagation of errors in later transmitted frames. For example, packets of reference frames may be considered to have higher error propagation potential than non-referenced frames.
Model DescriptionIn some cases, information used by retransmission selector 86 is acquired through inspection of at least one of the three inputs depicted in
Cost function J can be applied on missing packets across a frame to determine whether to request missing packets. If the number of missing packets is sufficiently large so as to raise network congestion then the cost junction J can be applied again on a packet-by-packet basis to select a subset of the missing packets for re-transmission (e.g., to request selective re-transmission of packets at a frame or packet level).
In one exemplary embodiment, the cost function (J) employed by retransmission selector stage 86 is determined as follows:
J=D+λQ (Equation 1)
where D is defined as the estimated distortion due to the missing packets, Q is defined as the acceptable delay (that is, delay as a function of RTT and PB, i.e., Q=F(RTT,PB)), and λ is defined as a weight that controls the trade-off between the distortion and delay. Thus, the retransmission of packets is requested if J exceeds a threshold. Note that the calculation of J, D and Q as described herein are illustrative examples only. In one embodiment, Q can be represented by describe values as follows:
An example for the determination of the estimated distortion D is now provided. In one application D is a function of two distortion components:
D=Dc+Ds (Equation 2)
where Dc is the estimated distortion of the current frame (error recovery), and Ds is the estimated impact on distortion on subsequent frames (error propagation).
In some cases, Dc can be based on at least one or more of the types of input, such as the type of error concealment algorithm used decoder, video properties (e.g., content, resolution, bit rate) and packet loss state information derived from recovered media packets 81 and embedded data in the FEC header data 83. In addition, Dc may be further estimated using knowledge regarding the quality of previous frames. For example, the computation of Dc can be based on whether the previous frame was complete (because in such cases, the EC algorithm may perform better).
In another alternative embodiment, Dc can be determined using state variables in accordance with the following:
Dc=DcF(P,bpp,Dc{t-1}) (Equation 3)
Where Dc is defined as a function F of P, the packet loss state, bpp which represents bits per pixel and Dc{t-1}) which is the decoder distortion state of the previous frame. The bits per pixel is based on an estimated of the encoding rate and the resolution of the video, as determined from the video properties 92.
The value of D can be output using a predetermined error distortion classification scheme such as described in Table 4 below:
In some case, Ds reflects the error propagation potential of a given packet. In a predicted-encoding scheme such as H.264 or VP8, the decoding of a given frame can affect the quality of subsequently decoded frames. The Ds score in some case captures the probability that a packet will be referenced. This probability can be estimated by first determining whether the current frame to which the packet belongs is a reference frame. The information as to whether the current frame is a reference frame is included in the video properties input accepted by retransmission section stage from video decoder stage 84. This frame type information can also be included in the codec header contained in some packets. In one implementation, Ds will be set to 0 (Null) if the frame is a non-reference frame, 1 (Partial) if it is a delta frame and 2 (Full) if it is a key frame.
Thus, with packet loss state information available (and having knowledge of the error concealment algorithm's capabilities), retransmission selector stage 86 can, for a particular packet or group of missing packets determines whether or not to request retransmission by assessing how well video decoder stage 84 will be able to minimize distortion if the packet or group of packets is missing.
In one embodiment, a determination is made as to whether the missing packets fall into partitions and if so how many missing packets fall into the first partition and how many fall into the second or higher partitions. If the number of packets missing from the first partition is greater than zero, then a determination of the Dc score is made for those packets based on how effective the decoder's error concealment algorithm is at recovering from first partition errors given the number of number of missing packets in the first petition. The determination can be based on empirical testing and can vary from implementation to implementation. If the number of packets missing from the second and higher partition is greater than zero, then a determination of Dc is made for those packets based on how effective the decoder's error concealment algorithm is at recovering from two or more partition errors given the number of missing packets in the first petition. The determination can be based on empirical testing and can vary from implementation to implementation.
The quality of the decoded frame given the packet loss state can depend on the video content state. Video decoder stage 84 has access to this basic information, in some cases without incurring significant additional processing. For example, from past coded frames, decoder can extract the following: video resolution, an estimate of the encoding rate (from average size of decoded frames), content classes and other video properties which are output from decoder to retransmission selection stage.
In one exemplary embodiment, a look up table is used to determine whether to request retransmission based on given values of Dc and Ds and the acceptable delay. One such table is illustrated as Table 5 below. In Table 5, the inputs are the values of Dc and Ds as indicated in the rows of the table and the acceptable delay as indicated by the three columns under the heading “acceptable delay.” The output is indicted by the legend “Y” for request retransmission and “N” for do not request retransmission. For example, referring to the first row of Table 5, if the value of Dc is 0, and the value of Ds is 0, then the output for any level of acceptable delay (i.e., high or medium) is “N”.
In an alternative embodiment, retransmission logic is also included in retransmission logic stage 70 of transmitting station 14. Retransmission logic stage 70 accepts as input a request from retransmission selector stage 86. Additional decision processing can take place at retransmission logic stage 70 before fulfilling the request. For example, retransmission logic stage 70 can determine the congestion of network 16. If congestion is determined to be high, then retransmission logic stage 70 can provide only a subset of the packets that are included in the retransmission request 94. Retransmission logic stage 70 can also determine if any requested packets are non-source packets and then refrain from retransmitting those packets.
In another alternative embodiment, the selective retransmission logic employed by retransmission selector stage 86 can be implemented in two stages. A first stage assesses cost function J for all packets at a frame level. The second stage then assesses the cost function J for individual packets within the frame. This two-stage approach is useful in when a frame is missing several packets and it is desired to reduce the number retransmitted packets to reduce network congestion.
In yet another embodiment, the selective retransmission based on the distortion-delay cost function is supplemented with a waiting time parameter. Once the retransmission selector 86 makes a decision on which packets in the NACK list to be sent to the sender for retransmission, the receiver may decide to wait only up to the time and then decode the (possibly incomplete) frames, regardless of whether all the missing packets have been received. The parameter may be fixed or determined dynamically based on the various factors, such as the packet inter-arrival jitter, round-trip time, and frame rate of incoming video stream for example. The relevance of this waiting time parameter may be for cases where the re-transmitted packets are further delayed due to sudden increases in network congestion or cross-traffic.
Referring to
At block 146, retransmission selector 86 determines a value of acceptable delay (such as for example Q) in rendering the frame. This can be accomplished using the values of PB and RTT, for example, as described above.
At block 148, retransmission selector 86 determines a distortion value (such as for example D) of the frame as rendered without the at least one missing packet. In one embodiment, the process of determining a distortion value begins by extracting a packet loss state, such as the total number of packets in the frame and the number of missing packets and their affiliation (such as partition information). Next, the value of Dc is determined as described above. In some cases, the value of Dc is determined based on the packet loss state and the decoding state of previous frames. Dc can in some cases be a value as set forth above in Table 4.
Processing at block 148 can also include determination of a Ds value, which can be performed as described above. The value of Ds can in some cases be a value as set forth above in Table 4. In one implementation, the Ds value can be based on the packet loss state. In alternative embodiments, Ds can be determined based on the current frame type in accordance for example with Table 6 below:
At a block 150, retransmission selector 86 determines a cost value (such as for example J) as function of the value of acceptable delay Q and the distortion value D.
At decision block 152, if the cost value J exceeds a threshold, then at a block 154 receiving station 14 initiates a request over the network for retransmission of the at least one missing packet; processing then terminates at a block 156. If at decision block 152, the cost value does not exceed the threshold, then processing terminates at block 156 without requesting retransmission.
Alternatively, the decision to retransmit can be based on the results of a pre-calculated look-up table (such as Table 5 above). Such a table can incorporate user-preferences to determine the optimal trade-off between delay and quality. The use of a table effectively performs the steps of determining a cost value and determining whether that cost value exceeds a threshold.
Processing at blocks 148 and 150 can be performed selectively on a per-frame or per-packet basis. For example, given the values of Dc, Ds and Q, retransmission selector 86 can determine whether all of the missing source packets in the frame should be retransmitted. Alternatively, the decision to retransmit packets can be made on a packet-by-packet basis depending on factors such as network state. For example, if the network is congested, requests for retransmission can be limited to the most important packets in the NACK list 85, such as for example those packets most important for adequate decoding based on per-packet Dc and Ds models. Alternatively, importance can be determined by use of a lookup table such as for example Table 7 below. The importance, as shown in Table 7, may be based on the partition ID of the packet (2 columns in the table) and the frame type (3 rows in the table).
In an alternative embodiment, off-line study can be conducted on packet losses and the impact of those losses on decoded frame quality. The off-line study can include for example the following steps. A simulation is provided of a frame that is encoded, packetized and sent to a decoder with transmission over the network. For purposes of the simulation, prior transmitted frames can be correctly decoded. The simulation is conducted over a range of parameters, including for example:
Frame resolution such as dimension and frame rate;
Frame type (such as delta or key);
Target bit rate;
Packet loss such as percentage of total loss, relative percentage per partition; or
Variety of test sequences.
Packet loss can then be simulated and decoding can take place over an error-prone frame. The error-prone frame is created to monitor the number, location and/or affiliation of the dropped packets. The following values are monitored to the decoded error-prone frame:
Decode error such as whether the frame was decoded or dropped; and
PSNR/SSIM and other objective metrics to quantify the quality of the frame
The results of the monitoring are then clustered into classes, such as four classes. Exemplary classes include Bad (decoder was unable to decode the frame) and {Poor, Acceptable and Good}. Threshold values for use in the cost function J are then computed using k-means clustering on the objective metrics. The threshold values can be further computed given frame dimension and target bit rate. The analysis can also be conducted on a per-frame and per-packet basis.
All or a portion of embodiments of the present invention can take the form of a computer program product accessible from, for example, a computer-usable or computer-readable medium such as memories 20, 24. A computer-usable or computer-readable medium can be any device that can, for example tangibly contain, store, communicate, and/or transport the program for use by or in connection with any processor. The medium can be, for example, an electronic, magnetic, optical, electromagnetic, or a semiconductor device. Other suitable mediums are also available. For example, computer instructions for implementing the techniques described above can be stored on the computer-readable medium (such as memories 20, 24). These instructions, when executed by a computer, cause the computer to form the above-described techniques.
References herein to “determining” a value include both determining the actual value or an estimate of the actual value, whether by means of calculations, accepting user input for the value, retrieving the value from memory of any kind, receiving the value from an external resources such as a server, selecting or identifying the value from a set of values, or otherwise creating or ascertaining the value (or an estimate of the value) by any means whatsoever using a processor, such as a computer or any other device of any kind capable of processing information.
The above-described embodiments have been described in order to allow easy understanding of the present invention and do not limit the present invention. On the contrary, the invention is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structure as is permitted under the law.
Claims
1-18. (canceled)
19. A apparatus for processing a packetized signal including at least one encoded frame having at least one received packet having a packet header information, comprising:
- memory; and
- a processor configured to execute instructions stored in the memory to: identify, by means of a gap in the sequence of packet sequence numbers in the packetized signal, a lost packet that is missing from the packetized signal; and determine whether the lost packet is likely to be a source packet or an error correction packet, based on the packet header information of the at least one received packet.
20. The apparatus of claim 19, wherein the processor is further configured to execute instructions stored in the memory to:
- request retransmission of the lost packet based on the determination of whether the lost packet is likely to be a source packet or an error correction packet.
21. The apparatus of claim 19, wherein the packet header information is at least one of a packet sequence number, a base sequence number or a packet mask.
22. The apparatus of claim 19, wherein the processor is further configured to execute instructions stored in the memory to:
- determine whether the lost packet contains information associated with the at least one encoded frame based at least in part on the packet header information.
23. The apparatus of claim 19, wherein the at least one encoded frame includes at least a first partition, and wherein the processor is further configured to execute instructions stored in the memory to:
- determine whether the at least one lost packet contains information associated with the first partition based at least in part on the packet header information.
24. The apparatus of claim 19, wherein the processor is further configured to execute instructions stored in the memory to:
- determine the total number of packets in the at least one encoded frame, including the lost packet, based at least in part on the packet header information.
25. The apparatus of claim 19, wherein packetized signal is a packetized video signal.
26-30. (canceled)
31. A method for processing a packetized signal including at least one encoded frame having at least one received packet having a packet header information, the method comprising:
- identifying, by means of a gap in the sequence of packet sequence numbers in the packetized signal, a lost packet that is missing from the packetized signal; and
- determining whether the lost packet is likely to be a source packet or an error correction packet, based on the packet header information of the at least one received packet.
32. The method of claim 31, further comprising:
- requesting retransmission of the lost packet based on the determination of whether the lost packet is likely to be a source packet or an error correction packet.
33. The method of claim 31, wherein the packet header information is at least one of a packet sequence number, a base sequence number or a packet mask.
34. The method of claim 31, further comprising:
- determining whether the lost packet contains information associated with the at least one encoded frame based at least in part on the packet header information.
35. The method of claim 31, wherein the at least one encoded frame includes at least a first partition, the method further comprising:
- determining whether the at least one lost packet contains information associated with the first partition based at least in part on the packet header information.
36. The method of claim 31, further comprising:
- determining the total number of packets in the at least one encoded frame, including the lost packet, based at least in part on the packet header information.
37. The method of claim 31, wherein packetized signal is a packetized video signal.
38. A computer program product comprising a non-transitory computer readable medium having instructions encoded thereon for processing a packetized signal including at least one encoded frame having at least one received packet having a packet header information, wherein the instructions, when executed by a processor, cause the processor to:
- identify, by means of a gap in the sequence of packet sequence numbers in a packetized signal, a lost packet that is missing from the packetized signal; and
- determine whether the lost packet is likely to be a source packet or an error correction packet, based on the packet header information of the at least one received packet.
39. The computer program product of claim 38, wherein the instructions further cause the processor to:
- request retransmission of the lost packet based on the determination of whether the lost packet is likely to be a source packet or an error correction packet.
40. The computer program product of claim 38, wherein the packet header information is at least one of a packet sequence number, a base sequence number or a packet mask.
41. The computer program product of claim 38, wherein the instructions further cause the processor to:
- determine whether the lost packet contains information associated with at least one encoded frame based at least in part on the packet header information.
42. The computer program product of claim 38, wherein at least one encoded frame includes at least a first partition, and wherein the instructions further cause the processor to:
- determine whether the at least one lost packet contains information associated with the first partition based at least in part on the packet header information.
43. The computer program product of claim 38, wherein the instructions further cause the processor to:
- determine the total number of packets in the at least one encoded frame, including the lost packet, based at least in part on the packet header information.
44. The computer program product of claim 38, wherein packetized signal is a packetized video signal.
Type: Application
Filed: Nov 7, 2016
Publication Date: Mar 2, 2017
Applicant: GOOGLE INC. (Mountain View, CA)
Inventors: Marco Paniconi (Campbell, CA), Mikhal Shemer (Berkeley, CA)
Application Number: 15/344,629