Method and apparatus for controlling transmission flow using explicit rate control and queuing without data rate supervision

Abstract

Packet flow rate control techniques are enhanced by the interactive and early invocation of packet queuing to control short flows of packets and to eliminate undershoot and overshoot of a targeted flow rate. Packet queuing involves the scheduled release of packets in accordance with flow policies (priorities) to achieve a pre-selected outgoing target flow rate. The combination of controlled packet queuing and packet flow rate control with appropriate mechanisms for favoring one over the other improves the efficiency of data transmission.

Description

CROSS REFERENCE TO RELATED APPLICATIONS

This continuation application claims the benefit of U.S. patent application Ser. No. 10/140,351 filed on May 6, 2002 now U.S. Pat. No. 7,236,459, which is incorporated herein by reference.

STATEMENT AS TO RIGHTS TO INVENTIONS MADE UNDER FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

NOT APPLICABLE

REFERENCE TO A “SEQUENCE LISTING,” A TABLE, OR A COMPUTER PROGRAM LISTING APPENDIX SUBMITTED ON A COMPACT DISK

NOT APPLICABLE

BACKGROUND OF THE INVENTION

This invention relates to digital packet telecommunications, and particularly to management of flow of data, that is, the volume of data per unit of time across heterogeneous network boundaries. It is particularly useful in a digitally-switched packet telecommunications environment normally not subject to data flow rate control. The present invention is intended to work in an environment having a metered-release of acknowledgements and a window control mechanism.

This invention represents an augmentation of the capabilities disclosed in the work of Robert Packer, as for example described in U.S. Pat. Nos. 6,018,516; 5,802,106; 6,038,216; 6,046,980; 6,205,120; 6,285,658; 6,298,041 and 6,115,357. The Packer packet flow rate control mechanisms taught therein controlled size of the sliding window, amount of acknowledged data and timing of acknowledgement delivery.

The ubiquitous TCP/IP protocol suite intentionally omits explicit supervision of the rate of data transport over the various media which comprise a network. While there are certain perceived advantages, this characteristic of TCP/IP has the consequence of juxtaposing very high-speed packet flows and very low-speed packet flows in potential conflict for network resources, which results in inefficiencies. Certain pathological loading conditions can result in instability, overload and data transfer stoppage. Therefore, it is desirable to provide some mechanism to optimize efficiency of data transfer while minimizing the risk of data loss. Data flow rate capacity information is a key factor for use in resource allocation decisions.

The technology of interest is based largely on the TCP/IP protocol suite, where IP, or Internet Protocol, is the network layer protocol and TCP, or Transmission Control Protocol, is the transport layer protocol. At the network level, IP provides a “datagram” delivery service. By contrast, TCP builds a transport level service over the datagram service to provide guaranteed, sequential delivery of a byte stream between two IP hosts.

Conventional TCP flow control mechanisms operate exclusively at the end stations to limit the rate at which TCP endpoints emit data. However, TCP lacks explicit data rate control. In fact, until the work of Packer, there was no concept of coordination of data rates among multiple flows.

The basic TCP flow control mechanism is a sliding window superimposed on a range of bytes beyond the last explicitly-acknowledged byte. Its sliding operation limits the amount of unacknowledged transmissible data that a TCP endpoint can emit.

The sliding window flow control mechanism works in conjunction with the Retransmit Timeout Mechanism (RTO), which is a timeout to prompt a retransmission of unacknowledged data. The timeout length is based on a running average of the Round Trip Time (RTT) for acknowledgment receipt, i.e., if an acknowledgment is not received within (typically) the smoothed RTT+4*mean deviation, then packet loss is inferred and the data pending acknowledgment is retransmitted.

Data rate flow control mechanisms which are operative end-to-end without explicit data rate control draw a strong inference of congestion from packet loss (inferred, typically, by RTO). TCP end systems, for example, will ‘back-off’, i.e., inhibit transmission in increasing multiples of the base RTT average as a reaction to consecutive packet loss.

While TCP rate control has significant advantages, there are certain conditions where the response time needed to adjust rate control mechanisms is less than can be provided by Packer packet flow rate control techniques.

SUMMARY OF THE INVENTION

According to the invention, in a packet-based communication system where acknowledgment packets are employed in the control of the flow rate of packets, packet flow rate control techniques are enhanced by the interactive and early invocation of packet queuing to control short flows of packets and to eliminate overshoot of a targeted flow rate. Packet queuing according to the invention may involve the scheduled release of packets in accordance with flow policies (priorities) to achieve a preselected outgoing target flow rate. In a specific embodiment of the invention, packets that arrive from a data source at the beginning of a flow before rate control is effective, the packets are forwarded by metering them out at the allocated bandwidth below the bandwidth capacity of the channel based on an expected capacity of the channel. The queuing of packets terminates and the queue is emptied as the feedback-based rate control mechanism using acknowledgments begins to moderate the rate of packet release from the data source. Packet rate control uses window size (TCP window size), acknowledgment rate, and number of bytes acknowledged. The combination of controlled packet queuing and network flow rate control with appropriate mechanisms for favoring one over the other improves the efficiency of data transmission.

The invention will be better understood by reference to the following detailed description in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is first flow chart of operation of an integrated packet rate control mechanism according to the invention.

FIG. 2 is a second flow chart of operation of an augmented packet rate control mechanism in FIG. 1 according to the invention.

FIG. 3 is a third flow chart of operation of a normal rate control and sizing mechanism in FIG. 1 according to the invention.

DETAILED DESCRIPTION OF THE INVENTION

• • Referring to FIG. 1, a flow chart is shown of overall or integrated system operation of a controller at a portal in the flow path between an arbitrary data source and an arbitrary data destination according to the invention. The invention is explained in the context of Transmission Control Protocol (TCP), which is a well-known transmission protocol used on the Internet. However, in general any network protocol or network application which employs a feedback mechanism to moderate bandwidth allocated to a source can be rate controlled through an implementation of this mechanism. Network transport protocols that fall into this category include AppleTalk ADSP, AppleTalk ATP, some sub-protocols of SNA, SNA/IP, and RTP/IP. Some applications with explicit timing feedback, such as Real Audio over UDP, can also be controlled with a specific implementation of the described mechanism.
  • In the protocol, a network packet arrives at the controller (Step A) and is immediately tested to determine if it is a new flow (Step B). If not, the packet is passed through, where it is subsequently check to determine if this flow is controlled or not (Step C). (A control block for this flow may be retrieved at this point to facilitate control.) If not, the packet is transmitted on in the network (Step D). The controller is thus transparent to the packet.

If the flow (Step B) is a new flow, then the controller sets up a local control block with flow information for this new flow, such as source and destination IP addresses and TCP ports of the flow, as well as the state of the flow (e.g., time of receipt of packet, time of last packet received, ACK number, Sequence number, last ACKs, and window size) (Step E), and then the controller checks to see if flow is to be controlled (Step C).

If flow is to be controlled, the controller determines whether it is a TCP flow (Step F). If not, then the flow control is passed on to other control mechanisms appropriate to the flow type (Step G), and which are not a part of this specific embodiment.

If it is a TCP flow, then the controller checks to determine if a TCP SYN (synchronization) flag or RST (reset) flag set (Step H), in which case, the packet is passed to the network. Otherwise, the packet is tested to determine whether it contains data with a new ACK number (an ACK which has not been received before) (Step I). If not, it tested for data (Step J) and if not, it is passed on for normal TCP ACK processing (Step K, FIG. 3).

If the packet contains both data and new ACK information, then the data and the ACK are separated (Step L). this is done as follows. A new packet is created with no data, and the new ACK information is copied into the new packet. This new packet is forwarded for normal TCP ACK processing (Step K). The data packet is stripped of the new ACK information (the acknowledgment number in the TCP is set back to the last acknowledgment number that was forwarded in this direction), and the packet, which now contains only data, is forwarded to TCP data processing (Step M, FIG. 2). By separating the data from the ACK, the data can be forwarded as soon as possible without regard to the timing of the forwarded ACK packets that are metering the sender's data transmission rate.

Referring to FIG. 2, with the environment set up to take advantage of the invention, the local control block for the flow is augmented and updated (Step N) by recording the highest sequence number (SEQ) received, checking for missing data according to missing sequence numbers, and recording the time of arrival of the packet at the controller. Then in Step O, a new estimate of the flow's data rate is made. This is done by updating an Exponentially Weighted Moving Average (EWMA) with the latest flow rate measurement. The latest flow rate measurement is the number of bytes in the packet received divided by the interval since the receipt of the last received packet. This value is the measured flow rate. Thereafter the controller determines the target data rate (Step P) by making a new estimate of the flow's potential rate (whether to increase or decrease), requesting bandwidth from the distribution mechanism (not shown), and receiving from the bandwidth distribution mechanism an assignment of bandwidth or target rate in bytes per second. The target rate is acquired on every incoming packet, since the target rate information is used in subsequent control steps.

The state of the initial packet in a flow is State 0 (zero) of possible states 0, 1 and 2. The states indicate whether a queue is on buildup, draindown or whether the queuing mechanism is finished. This queuing process is typically not used after an initial period related to the beginning of a new flow or a restart of a flow after a pause.

If the tested packet has achieved a state equal to 2 (Step Q), then there will be no queuing, and the packet is transmitted without undue delay (Step R). If the state is zero or 1, then the controller tests to see if there is already a packet queued for this flow (Step S). If not, then the controller tests to determined whether the packet's inbound arrival rate exceeds the assigned target rate (Step T) (as determined by Step P). If not, the packet is transmitted (Step R). If its rate exceeds the target rate, then it is scheduled for delayed release (Step U), and it is released as scheduled.

If there is a data packet already queued for flow (Step S), then the state flag is again tested for State 0 or 1, to determine the state of the queue (Step V). If the state is zero (queue buildup), then the controller tests to see if the number of packets in the queue is greater than a trigger level, as selected by the operator (Step W). (In TCP where the target rate is not more than an order of magnitude different than the incoming rate, a queue of 2-4 is expected to be sufficient). If the queue is not “full,” the packet is added to the queue (Step X), and the packet is dealt with as part of the scheduled release of the queue (the next transmission of a packet in the previous scheduling performed in of Step U). However, if the packet count exceeds the trigger, the state is set to State 1 (Step Y) before the packet is added to the queue (Step X).

If the state is State 1 (Not State=0, Step V), the queue is in draindown state, whether or not it is being emptied. The controller checks to see if the number of packets in the queue has fallen below a draindown trigger level (Step Z). The draindown trigger level may be different than the buildup trigger level. If not, then the packet is added to the queue (Step X). Otherwise, the state is set to 2 (Step AA) before adding the packet to the queue. State 2 indicates that the queuing is done. Packets are released from the queue in due course through the scheduling of transmission.

Referring now to FIG. 3, the mechanism for rate control with Acknowledgment is illustrated. This process takes over from the queuing mechanism, which is the initial rate control mechanism, after the decisions of the controller outlined in FIG. 1 are completed. The ACK packet (Step K) prompts the updating of the ACK information of the flow's control block (Step AB) and tests to determine whether there is an ACK which has been queued (Step AC). If there is already an ACK for this flow that has been scheduled for future release, then the current ACK is simply deleted (Step AD), and the controllers wait for the next timed ACK transmission or release. If there is not an ACK in the system, this ACK packet is used to determine data rates and ACK rates (Step AE). This is done by dividing the number of bytes acknowledged between the time of receipt of the current ACK and time of receipt of a previous ACK. This rate may be averaged over several ACK times using the EWMA technique mentioned previously.

The ACK rate is then tested (Step AF). If the ACK rate does not exceed the assigned ACK rate as specified by the bandwidth manager (Step T, which is based on data flow rate), the window size is modified to be consistent with prior window sizes (Step AG) and the ACK is transmitted to the data source (Step AH). If the ACK rate does exceed the assigned ACK rate, then, since there is no scheduled ACK, the controller modifies the ACK to be consistent with the data rate assigned to the flow. the ACK may be modified by changing the number of bytes acknowledged by the packet (never sending an ACK that is less than previously sent ACKs), or it may reduce the advertised window size (which can be done by holding one edge of the window constant, since reducing the window from previous packets without advancing the acknowledgment number is a violation of the TCP protocol specification) (Step AI). The ACK may be and often is delayed in time. This time delay is useful in conjunction with ACK modifications to induce the data sender to send further data packets at the assigned data rate. Then the controller modifies the ACK packet according to the prior calculations (Step AJ), and schedules the ACK packet for later transmission (Step AK). After the scheduled delay, the ACK packet is transmitted (Step AL) and the controller checks to see if all bytes which have been acknowledged by the receiver have been forwarded to the sender (Step AM). If so, the process is done. Otherwise, the ACK is recycled (Step AN) and new determinations are made of delay, byte count and window size (Step AI) in according with the acknowledgment-based rate control mechanism.

The combination of initial queuing of packets and Acknowledgment-based rate control provides an effective mechanism for introduction of new flows in a bandwidth limited packet transmission environment, where speeds need to be controlled. It is most useful in an environment where fast and slow rates must be merged, and it inhibits undesired effects manifest in traffic speed oscillation.

The invention has been explained with reference to specific embodiments. Other embodiments will be evident to those of ordinary skill in the art. It is therefore not intended that the invention be limited, except as indicated by the appended claims.

Claims

1. In a digital data communication network employing packets for conveying information, a method for use at a controller between a data source and a data destination in a flow path for controlling flow of data, the method comprising:

receiving, at the controller, a data packet of a Transport Control Protocol (TCP) data flow from a data source;

queuing, the data packet of the TCP data flow on a queue and subsequently transmitting the data packet at a scheduled delay to achieve a target rate for the TCP data flow, if a measured rate of the TCP data flow exceeds the target rate or a prior packet of the data flow is stored on the queue;

entering a queue drain-down phase when a first threshold number of packets of the TCP data flow are stored on the queue, wherein the queue drain-down phase comprises: queuing the data packet of the TCP data flow on a queue and subsequently transmitting the data packet at a scheduled delay to achieve a target rate for the TCP data flow, if a prior packet of the data flow is stored on the queue; bypassing, after entering the queue drain-down phase, the queuing step of the drain-down phase for subsequent packets of the TCP data flow when the number of packets of the TCP data flow stored on the queue falls below a second threshold number of packets; and

controlling window size, amount of acknowledged data and timing of response of acknowledgments of data in the flow path between the data source and the data destination.

2. The method according to claim 1 wherein said queuing step is initiated immediately at the start of a TCP data flow.

3. The method according to claim 1, wherein the data packet comprises data, and wherein the method further comprises determining whether the data packet includes a new acknowledgment of data; and, if so, creating a first new packet including the data of the received packet and a second new packet including the new acknowledgment of data; and applying separate processing to the first and second new packets.

4. The method according to claim 1 wherein the first and second threshold are the same.

5. The method according to claim 3 further comprising modifying one or more attributes of the acknowledgment of data corresponding to the second new packet.

6. The method according to claim 5 wherein the modifying step comprises one or more of: modifying an advertised window size, modifying an amount of data acknowledged, and queuing the second new packet for a scheduled delay.

7. The method of claim 1 wherein synchronization or reset packets are scheduled for immediate transmission.