Abstract: Data packets from different layers of a layered data stream are correlated in time. The layered data stream has a base layer and one or more enhancement layers. The method builds a table of entries, with each entry including one or more received data packets. Each data packet has a corresponding time stamp indicating when that data packet can be sent to a decoder for processing. The entries in the table are indexed according to the time stamps of the received data packets, where each entry is associated with a temporal window and data packets that have time stamps within a given temporal window are included within the entry associated with the given temporal window.