NESTED WEIGHTED ROUND ROBIN QUEUING

Abstract

In a Mobile Ad Hoc Network (MANET), nested weighted round robin queues are employed to selectively provide channel access for traffic according to a priority or Quality of Service (QoS) for data. By nesting queues within other queues, and applying a weighted round robin technique to serve each queue, relatively arbitrary service metrics may be achieved including nodal QoS for class-based traffic, avoidance of queue starvation, and so forth. Prioritized queues may also be provided for preemptive delivery of high priority traffic.

Description

RELATED APPLICATIONS

This application claims the benefit of the following U.S. Provisional Patent Applications, each of which is incorporated by reference herein in its entirety:

U.S. App. No. 60/976,730 filed on Oct. 1, 2007;

U.S. App. No. 60/976,735 filed on Oct. 1, 2007;

U.S. App. No. 60/976,740 filed on Oct. 1, 2007;

U.S. App. No. 60/976,744 filed on Oct. 1, 2007;

U.S. App. No. 60/976,747 filed on Oct. 1, 2007; and

U.S. App. No. 60/976,748 filed on Oct. 1, 2007.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

This invention was made with support of the United States Government under Contract MDA972-01-9-0022. The United States Government may have certain rights in the invention.

BACKGROUND

This application relates to queuing data for transmission in a Mobile Ad Hoc Network (MANET), and more particularly to queuing prioritized data according to a weighted nested round robin queue. There remains a need for improved handling of multiple traffic types in a wireless ad hoc network.

SUMMARY

In a Mobile Ad Hoc Network (MANET), nested weighted round robin queues are employed to selectively provide channel access for traffic according to a priority or Quality of Service (QoS) for data. By nesting queues within other queues, and applying a weighted round robin technique to serve each queue, relatively arbitrary service metrics may be achieved including nodal QoS for class-based traffic, avoidance of queue starvation, and so forth. Prioritized queues may also be provided for preemptive delivery of high priority traffic.

In one aspect, there is disclosed herein a method that includes storing a plurality of data packets in a plurality of queues for transmission in a number of time slots from a node of a mobile ad hoc network, each one of the plurality of queues having a weight; selecting a first data packet from the plurality of data packets for transmission in one of the number of time slots according to a first weighted round robin schedule that is weighted to serve a first group of the plurality of queues according to their respective weights; and selecting a second data packet from the plurality of data packets according to a second weighted round robin schedule that is weighted to serve a second group of the plurality of queues according to their respective weights, wherein the first weighted round robin schedule includes a weight for the second round robin schedule and periodically serves the second weighted round robin schedule according to the weight, thereby selecting the second data packet in the first weighted round robin schedule for transmission in one of the number of time slots. The method may include preemptively selecting data packets from a prioritized queue until the prioritized queue is empty.

In another aspect, a computer program product disclosed herein include computer executable code that, when executing on one or more computing devices, performs the steps of: storing a plurality of data packets in a plurality of queues for transmission in a number of time slots from a node of a mobile ad hoc network, each one of the plurality of queues having a weight; selecting a first data packet from the plurality of data packets for transmission in one of the number of time slots according to a first weighted round robin schedule that is weighted to serve a first group of the plurality of queues according to their respective weights; and selecting a second data packet from the plurality of data packets according to a second weighted round robin schedule that is weighted to serve a second group of the plurality of queues according to their respective weights, wherein the first weighted round robin schedule includes a weight for the second round robin schedule and periodically serves the second weighted round robin schedule according to the weight, thereby selecting the second data packet in the first weighted round robin schedule for transmission in one of the number of time slots.

In another aspect, a device disclosed herein includes a data source that provides a plurality of data packets; a queue that schedules the plurality of data packets for transmission according to a weighted round robin, the weighted round robin including at least one weight for a nested weighted round robin queue, the nested weighted round robin queue served according to its weight in the weighted round robin, thereby providing scheduled packets; a radio that provides an air interface to a mobile ad hoc network including links to a plurality of neighboring nodes; and a signal processor that prepares the scheduled packets for transmission over the air interface.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention and the following detailed description of certain embodiments thereof may be understood by reference to the following figures wherein:

FIG. 1 is a block diagram of a Mobile Ad Hoc Network (MANET).

FIG. 2 is a block diagram of a MANET having multiple backhaul access points.

FIG. 3 is a block diagram of a node in a MANET.

FIG. 4 shows a queue architecture 400 that may be used with a nested weighted round robin queuing system.

FIG. 5 shows a scheduling algorithm for use with the queue structure of FIG. 4.

FIG. 6 shows a queue structure containing packets in various queues.

FIG. 7 shows a scheduling sequence for the queues of FIG. 6.

DETAILED DESCRIPTION

The following description details certain embodiments of a prioritized nested weighted round robin queuing mechanism used to meter data output from a node of a Mobile Ad Hoc Network (MANET). Use of a prioritized queue helps to meet strict Quality of Service requirements, while nested weighted round robin queues provide multilayer metered access capable of skewing user data to meet various user and system requirements, such as traffic differentiated according to service types as described in the Internet Engineering Task Force (IETF) Request For Comment (RFC) 2475, incorporated herein by reference in its entirety. While the invention is described below in relation to MANETs, it will be understood that the principles of the invention may be suitably applied in any environment where multiple traffic types are managed within a network.

So-called “infrastructure” networks employ base stations at fixed locations to form a substantially fixed network infrastructure. The base stations may enable communication among the wireless devices of the network, between a wireless device and another device on another network, and so on. This general approach is employed, for example, in 802.11 or WiFi networks, as well as in cellular telephony networks. By contrast, ad hoc wireless communications networks are formed in an ad hoc manner among any number of participating nodes that may periodically join, leave, or move within the ad hoc network. Although such networks do not belong to any fixed network infrastructure, they may support conventional network communications such as point-to-point or broadcast communications, and may be adapted for use with Internet Protocol or similar, well-established networking protocols.

In general, a Mobile Ad Hoc Network (MANET) is an ad hoc wireless network in which some (or all) of the participating devices—also referred to herein as “nodes”—are mobile. Thus the topography of a MANET may change not only as nodes enter and leave the network, but as nodes move relative to one another within the network. As the network topology changes, communications routes through the network may also vary in terms of availability and in terms of quality. While the invention(s) disclosed herein have broad applicability, they may be particularly useful in a MANET environment where the context of continuously changing node-to-node links poses challenges to, and opportunities for, maintaining traffic flow.

FIG. 1 shows a Mobile Ad Hoc Network (MANET) that may be used with the systems and methods described herein. In general, a MANET 100 may include subscriber devices 102, access points 104, and backhaul access points 108 (for coupling to a core network 110 such as the Internet), and subscriber devices 110, all generally interconnected as shown in FIG. 1. Without limiting the generality of the foregoing, one or more of the subscriber devices 102 may be a stationary device 112 that does not move within the MANET 100. It will be understood that the device-to-device links illustrated in FIG. 1 are for purposes of illustration only, and in no way are intended to limit the nature or number of links between devices in the MANET 100, which may be created, removed, and/or modified over time according to any corresponding protocols followed by the devices within the MANET 100. In general, the links among devices within the MANET 100 are wireless links, although wired links may optionally be employed in various locations such as between the backhaul access point 108 and the core networks 110. In order to maintain the MANET 100, typically one or more protocols are shared among the participating devices to control creation, removal, and modification of individual data links between devices, and to route traffic and control information among the devices. The term protocol as used herein generally refers to any and all such rules, procedures, and/or algorithms used in maintaining the MANET 100, unless a specific protocol is explicitly stated or otherwise clear from the context.

Subscriber devices 102 may include any general purpose nodes participating in the MANET 100 according to suitable protocols. It will be understood that while subscriber devices 102 may include terminal nodes that send or receive data, in a MANET 100 as described herein subscriber devices 102 may also suitably be employed as intermediate nodes to route traffic to and from other subscriber devices 102. Thus an ad hoc network as described herein is generally extensible, and as new subscriber devices 102 appear within the MANET 100, they may form a part of the MANET 100 fabric that routes traffic among other nodes. In general, subscriber devices 102 may include any network or computing devices that include a wireless interface, network protocol stack(s), and the like adapted to participate in the MANET 100. The Internet Protocol may usefully be employed in subscriber devices 102 within the MANET 100 in order to use well-established addressing schemes and the like. A subscriber device 102 may include without limitation a cellular phone, personal digital assistant, wireless electronic mail client, laptop computer, palmtop computer, desktop computer, video device, digital camera, electrical instrument, sensor, detector, display, media player, navigation device, smart phone, a wireless networking card, or any other device that might usefully participate in a network. In some embodiments subscriber devices may include a GPS receiver providing a position and timing reference. In embodiments, each subscriber device 102 may be authenticated and/or authorized before being granted access to the MANET 100.

Access points 104 may be provided to establish a permanent or otherwise generally stable infrastructure to the MANET 100. In one embodiment, the access points 104 may employ identical network functionality and protocol stacks as subscriber devices 102. However, an access point 104 may have a number of differences related to their dedicated function within the MANET 100. In one aspect, the access points 104 may have no associated computing device that originates or consumes network traffic. That is, the access points 104 may simply form a fixed mesh of participants in the MANET 100 and relay traffic among other network participants. An access point 104 may also include a physical connection to a power infrastructure so that it may be physically installed at a location and operate autonomously without requiring regular maintenance for battery changes and the like. In another aspect, access points 104 may include some minimal supplemental circuitry related to, e.g., status and diagnostics, or for receiving software updates and the like. This may improve continuity of coverage across a physical region where subscriber devices 102 may or may not be present with any regularity, and may ensure that wireless network resources are available in a desired area. In embodiments the access point 104 may be of a size and weight making it suitable for mounting and/or concealment in a variety of locations including indoor and outdoor locations, and including mounting on walls, floors, ground, ceilings, roofs, utility poles, and so forth.

Each access point 104 may include or utilize a timing reference such as any of the Network Timing Protocols described in RFC 778, RFC 891, RFC 956, RFC 958, RFC 1305, RFC 1361, RFC 1769, RFC 2030, and RFC 4330, all published by The Internet Engineering Task Force. Each access point may also, or instead, include a GPS receiver providing a position and timing reference. In embodiments the wireless access points 104 may have a greater transmit power and/or a greater antenna gain than mobile subscriber devices 102, thus providing greater physical coverage than some other devices within the MANET 100.

The MANET 100 may include one or more backhaul access points 108 that generally operate to connect nodes within the MANET 100 to a core network 110 such as the Internet. On one interface, a backhaul access point 108 may have a wireless radio interface, protocol stack(s) and other components of other nodes within the MANET 100. On another interface, the backhaul access point 108 may provide any suitable interface to the core network 110. The backhaul access point 108 may, for example, be deployed at a fiber access point or the like that provides high-speed data capacity Internet traffic. For example and without limitation, the fiber access point may include a Gig-E router site or an OC-3/12 add-drop multiplexer site. In an embodiment the backhaul access point 108 may include two Gig-E interfaces for backhaul connections. It will be understood that any number of a variety of suitable interfaces for backhaul connections may be usefully employed with a backhaul access point 108 as described herein.

A backhaul access point 108 may serve multiple access points 104 within the MANET 100, and may distribute network load across those access points 104. Alternatively, a single backhaul access point 108 may serve a single access point 104. In some embodiments, the number of access points 104 served by a backhaul access point 108 may relate to the amount of intra-MANET traffic and extra-MANET traffic, the nature and direction of multicast versus unicast data, and so forth. This association between backhaul access points 108 and access points 104 may change from time to time depending on the presence of other subscriber devices 102 within the area, network conditions, and so forth. In some cases an access point 104 may for a time be associated with more than one backhaul access point.

The core networks 110 may provide access to network resources outside the MANET 100. The core networks 114 may connect disparate, geographically remote and/or local instances of the MANET 100 to form a single network. The core networks 110 may include any and all forms of IP networks, including LANs, MANs, WANs, and so on. The core networks 110 may also or instead include the public Internet. In other embodiments the core networks 110 may consist exclusively of a single zone of administrative control, or a number of zones of administrative control, or some combination of an administrative zone and any of the foregoing.

The stationary device 112 may include any subscriber device 102 that, for whatever reason, does not physically move within the MANET 100. In general, such fixed physical points within the MANET 100 may provide useful routing alternatives for traffic that can be exploited for load balancing, redundancy, and so forth. This may include, for example, a fixed desktop computer within the MANET 100.

Details of various MANET 100 protocols—referred to collectively herein as the MANET Wireless Protocol (MWP)—are provided below. In general, any of the nodes above that participate in the MANET 100 according to the MWP may include a hardware platform enabling radio software and firmware upgrades, which may include for example a dedicated or general purpose computing device, memory, digital signal processors, radio-frequency components, an antenna, and any other suitable hardware and/or software suitable for implementing the MWP in participating nodes.

In embodiments, any of the foregoing devices, such as one of the access points 104, may also include an adapter for other networks such as an Ethernet network adapter or equivalent IP network adapter, router, and the like, so that non-MANET 100 equipment can participate in the MANET 100 through the device. It will also be appreciated that, while a connection to other core networks 110 is shown, this connection is optional. A MANET 100 (with or without fixed access points 104) may be maintained independently without connections to any other networks, and may be usefully employed for the sole purpose of trafficking data among subscriber devices 102.

FIG. 2 is a block diagram of a MANET having multiple backhaul access points. In general, the MANET 100 may include subscriber devices 102 (not shown), access points 104, and backhaul access points 108 for connecting to core networks 110, and an edge router 202 that facilitates routing between the MANET 100 and the core networks 110.

The edge router 202 may include any devices or systems for maintaining connectivity between the MANET 100 and the core networks 110, and may further support or enhance network activity within the MANET 100. For example, the edge router 202 may include an industry standard and/or proprietary Address Resolution Protocol server, an application server, a Virtual Private Network server, a Network Address Translation server, a firewall, a Domain Name System server, a Dynamic Host Configuration Protocol server, and/or an Operations, Administration, Maintenance and Provisioning server, as well as any combination of the foregoing. These various components may be integrated into the edge router 202, or may be provided as separate (physical and/or logical) systems that support operation of the edge router 202. These supporting systems may in general support operations such as broadband Internet connectivity within the MANET 100 and the like, broadcast communications crossing between the MANET 100 and the core networks 110, and so forth, as well as the use of multiple backhaul access points 108 to efficiently route inter-MANET traffic among subscriber devices 102.

FIG. 3 is a block diagram of a node in a MANET. The node may be any of the devices described above, such as a subscriber device 102, access point 104, or backhaul access point. In general the node 300 may include data sources 302, a data link 304, a signal processor 306, a radio 308, data queues 310, routing information 312, and neighborhood information 314. It will be understood that the following description is general in nature, and that numerous arrangements of processing, storage, and radio frequency hardware may be suitably employed to similar affect. This description is intended to outline certain operations of a MANET node relevant to the systems and methods described herein, and in no way limits the invention to the specific architecture shown in FIG. 3.

The data sources 302 may include any applications or other hardware and/or software associated with the node 300. This may include, for example, programs running on a laptop or other portable computing device, a web server or client, a multimedia input and/or output sources such as a digital camera or video, and so forth. More generally any device, sensor, detector, or the like that might send or receive data may operate as a data source 302 in the node 300. It will be further understood that some nodes such as access points 104 may not have independent data sources 302, and may function exclusively as MANET 100 network elements that relay data among other nodes and/or provide network stability as generally described above.

The data link 304 may include hardware and/or software implementing data link layer functionality such as neighbor management, segmentation and reassembly of data packets, Quality of Service (QoS) management, data queue servicing, channel access, adaptive data rates, and any other suitable data link functions. In general, the data link 304 controls participation of the data sources 302, and more generally the node 300, in a MANET. It will be understood that the data link 304 in FIG. 3 may implement any number of lower layer (e.g., physical layer) or higher layer (e.g., routing, transport, session, presentation, application) protocols from a conventional Open Systems Interconnection (OSI) Model, or any such protocols and related functions may be implemented elsewhere within the node 300, such as in an IP stack executing on the data source 302, or in firmware within the signal processor 306 or radio 308, or in additional functional blocks not depicted in FIG. 3. For example, routing protocols may be implemented within hardware/software of the data link 304 in order to ensure that nodes in the MANET 100 share appropriate routing functions. Thus it will be appreciated that while the certain elements discussed herein might suitably be placed within the data link layer of a formal protocol stack, the systems and methods of this disclosure might also or instead be implemented with variations to a conventional protocol stack, or without any formal protocol stack whatsoever.

The data link 304 may include a link manager that collects neighbor information from the data link layer, and may form and maintains the neighborhood information 314 for the node 300. This table may be used to establish routes to neighbors, and may be updated periodically with information from one and two hop neighbors as described further below. The link manager may monitor statistics on all active links for a node on a link-by-link basis in order to support link quality calculations and other functions described herein.

The signal processor 306 may include waveform processing and timing functions associated with transceiving data at the node 300. This may include, for example, network timing, time-slot and/or frame-based waveform configuration, maintenance of one or more families of Orthogonal Frequency Division Multiplexing waveform modes (or other transmit mode waveforms), receiver detection of waveform modes, error correction coding, and so forth. In general, the signal processor 306 may be implemented in any suitable combination of digital signal processors, field programmable gate arrays, application-specific integrated circuits, microprocessors, or other general or special-purpose computing devices.

In one embodiment, a family of Orthogonal Frequency Division Multiplexing (OFDM) waveforms may be employed for adaptive data rate communications. The modes of the OFDM waveforms may, for example, include 7.2 MHz Quadrature Phase-Shift Keying (QPSK), 4.8 MHz QPSK, 2.4 MHz QPSK, 1.2 MHz QPSK, 1.2 MHz Binary Phase-Shift Keying (BPSK), or the like. The effective data rate for transmit waveforms may be affected by other parameters such as error correction. In order to facilitate implementation of an adaptive rate system, the transmit modes may be organized into an ordered list of monotonically increasing data rates matched to correspondingly decreasing signal robustness, thus permitting unique mapping of link quality to transmit mode. In one aspect, the actual waveform mode selected to transmit data on a link may be adaptively selected according to any suitable evaluation of link quality for links to neighboring nodes.

The radio 308 in general operates to transmit data from the data queue(s) 310, as organized and encoded by the data link 304 and the signal processor 306 (along with any control information, packet header information, and so forth), over a wireless air interface to other nodes in a MANET, and to perform complementary data reception. The radio 308 may include any radio frequency analog circuitry and the like, and may be coupled to the signal processor 306 which converts data and control information between a digital representation used within the node 300, and an analog representation used in radio frequency communications with other nodes. In embodiments, a low power radio 308 may be employed, such as where the node 300 is a battery-powered mobile device. In other embodiments, a high-power radio 308 may be employed, such as where the node 300 is an access point or backhaul access point connected to a fixed power infrastructure. In an embodiment, the radio 308 and signal processor 306 provide adaptive data rate coding capable of changing transmit modes, error correction, and the like according to measured link quality.

The data queue(s) 310 may include any data for transmission from the node 300. This may include, for example, data from the data sources 302, data that is relayed by the node 300 from other nodes in the MANET, and/or control information scheduled for transmission within data packets from the node 300. The data queue(s) 310 may be organized in any suitable fashion, and may include a single first-in-first-out queue, multiple queues, prioritized queues, and the like. In one embodiment, the node 300 may include multiple prioritized queues to assist in providing various service levels, such as for QoS traffic. In general, data in the data queue(s) 310 is delivered according to any suitable queuing mechanism to the data link 304, signal processor 306, and radio 308 for transmission within the MANET.

Routing information 312 such as a routing or forwarding table may be provided to support routing functions by the node 300. In general, this may include, for example, a destination address or identifier, a cost of a path to the destination (using any suitably cost calculation), and a next hop on that path. Other information such as quality of service and other metrics for various routes and links may also be provided for more refined routing decisions.

Neighborhood information 314 may be maintained in a database, flat file, routing table, or other suitably organized volatile or non-volatile storage within the node 300. The neighborhood information 314 generally supports the creation and maintenance of the MANET as well as routing functions of each MANET node. Within the MANET, each node may interact with other nodes to autonomously identify and maintain local network connections, shift capacity, dynamically form routes throughout the network, and so on. The routing functions of the node (as supported by the neighbourhood information 314) may accommodate delay-sensitive (e.g. voice) traffic, delay-tolerant traffic with quality of service (QoS) prioritization, and so on.

The neighborhood information 314 may include an identification of neighboring nodes along with information relating to those nodes. This may include one-hop neighbors (i.e., neighboring nodes in direct wireless communication with the node 300), two-hop neighbors (i.e., neighboring nodes that communicate with the node 300 through only one other node), or any other nodes or participants within the MANET. In one aspect, neighborhood information 314 includes link quality information for the radio 308, which may be obtained from any combination of physical layer and data link data, and may be employed to adapt the data rate of communications according to currently present channel conditions. The neighborhood information may also include QoS data used to select next hops for QoS data. Other useful information may include bandwidth utilization, node weights, node position (either logical or physical), and queue latency for each QoS type and/or other priority type.

In one aspect, the neighborhood information 314 may be gathered during periodic exchanges (such as during control transmissions) with neighboring nodes, which may occur under control of the link manager of the data link 304. For example, the node 300 may determine output bandwidth (i.e., data transmit requirements) for each link that the node 300 has with a neighbor, and may transmit this to one-hop neighbors. Similarly, the node 300 may receive output bandwidth from each one-hop neighbor. Using this data, each node 300 may further calculate its own input bandwidth (i.e., data receive requirements) from each link to a neighboring node, and this information may in turn be exchanged with one hop neighbors. Following a system-wide exchange with one-hop neighbors, the node 300 (and every other node in the MANET) may calculate a node weight that represents relative output requirements for the node 300. For example, the node weight, W, may be calculated as:

$\begin{array}{cc}W=\frac{{\mathrm{BW}}_{\mathrm{out}}}{{\mathrm{BW}}_{\mathrm{out}}+{\mathrm{BW}}_{\mathrm{in}}}& \left[\mathrm{Eq}.1\right]\end{array}$

where BW_out is the total output or transmit requirements for each link of the node 300, and BW_in is the total input or receive requirements for each link of the node 300. Finally, the node 300 may transmit the node weight to each neighboring node, and may in turn receive a node weight from each neighboring node. It will be appreciated that the node weight, W, may be further processed for use with other neighborhood information 314, such as by limiting the value according to the number of bits used for control information, or by providing a supplemental adjustment to the node weight to further refine control of routing or other MANET functions. Sharing of information for maintenance of the neighborhood information 314 may be controlled, for example, by the data link 304, which may apply any suitable technique to determine when to share information with one hop neighbors. In one aspect, the data link 304 may transmit data whenever a change is detected in the MANET such as an addition or deletion of a node.

In another aspect, for a MANET that has location-aware nodes 300 (e.g., using Global Positioning System (GPS) data, signal strength data, and so forth), the neighborhood information 314 may include position data in order to support location-based routing and the like.

Having described in general terms a MANET that can implement the prioritized nested weighted round robin queuing as described herein, the description now turns to a more detailed treatment of embodiments of a queuing mechanism.

FIG. 4 shows a queue architecture 400 that may be used with a prioritized nested weighted round robin queuing system. The queue architecture 400 may be deployed, for example, in the data queue(s) 310 of FIG. 3. The queue architecture 400 may include a packet in-flow counter 402, a packet overflow counter 404, a queue depth meter 406, priority queues and weighted round robin queues 410. The queue structure 400 may be stored in a volatile or non-volatile memory using any suitable list, ordered list, buffer, index, or other memory structure suitable for storing queues of data, along with code or other mechanisms for adding and removing data from queues 412, counting packet flow in and out, measuring queue depth, and so forth. In general, each queue 412 operates as a first-in-first-out buffer for packets that are to be transmitted from the node described above.

The packet in-flow counter 402, packet overflow counter 404, and queue depth meter 406 may be used to monitor performance of the queues 408, 410, and where appropriate, may provide feedback to adjust weights of the WRR queues 410 in order to adjust scheduling of packets according to traffic conditions. For example, the packet in-flow counter 402 may count packets as they arrive for queuing, and may provide aggregate counts and/or queue-specific counts. The packet overflow counter 404 may count packets that were dropped by the queues due to aging, buffer overflow, or the like. The queue depth meter 406 may provide a depth for each queue, and may be updated continuously as packets are added to, delivered from, or overflowed out of the queues.

Packet sources 414 may include any sources of data within the node, such as application software executing on the node or data received at the node for relay to another node in the MANET. In general, packet sources 414 may feed the queues using explicit or implicit prioritization data including without limitation traffic types or other prioritization, transmit mode, or link quality data tagged to data from the packet sources 414, or associated with links of the air interface that will be used to transmit the data. In one embodiment, the packets sources 414 may include traffic with QoS levels, Voice-Over-IP traffic, streaming video, and the like. Traffic may be identified using IETF RFC 2475 DiffServ Code Points (DSCPs) such as Expedited Forwarding, Assured Delivery and Best Efforts. Each class of traffic may be further divided into sub-types for prioritization within the class. More generally, any explicit or implicit prioritization scheme may be employed, and however such traffic is categorized, a suitable delivery mechanism may be deployed using the systems and methods described herein.

A queue server 414 may operate to serve the queues 412 by selecting data from the queues 412 according to a scheduling algorithm, and scheduling the data for transmission over an air interface of the node, such as in a time slot of a Time Division Multiple Access system. In general, the priority queues 408, if any, will receive immediate and complete service from the queue server 414, so that any data placed in these queues 412 will be immediately scheduled for transmission. The priority queues 408 may be prioritized to provide multiple priority levels to this preemptive service. Thus a priority 1 or “High” priority queue will always be immediately serviced. A priority 2 or “Medium” priority queue will always be immediately serviced unless there is priority 1 data. A priority 3 or “Low” priority queue will always be immediately serviced unless there is priority 1 and/or priority 2 data. In embodiments, there may be no prioritized queues, and the nested weighted round robin queuing may be used exclusively to schedule data. The weighted round robin (WRR) queues 410 may include any number of data queues. As depicted, the WRR queues 410 include three queues (Q1-Q3), one of which is a nested WRR queue including three additional queues (Q4-Q6). Each WRR queue 410 has a weight associated with it, as indicated in parenthesis below the queue label (e.g., Q1 has a weight of 6, Q2 has a weight of 3, etc.). The queue server 416 controls the manner in which data is removed from these various queues and scheduled for delivery, as described in more detail below.

FIG. 5 shows a scheduling algorithm for use with the queue structure of FIG. 4. Although not explicitly depicted in FIG. 5, it will be understood that each time a packet is “selected”, the packet may be placed into a time slot or otherwise scheduled for transmission in order of selection. It will further be understood that while the following description refers to packets, the process 500 described herein may be employed more generally with data of varying lengths and types without departing from the scope of this disclosure. The process 500 may begin 502 by determining whether there is priority data in a preemptive priority queue as shown in step 504.

As shown in step 506, if there is data in one of the priority queues, then a packet may be selected from the priority queues and the process may return to step 504 where the priority queues are again checked for data. Steps 504 and 506 may be repeated until the priority queues are empty. In one embodiment, a single preemptive priority queue may be employed. In another embodiment, a number of preemptive queues may be employed, which may be relatively prioritized so that one of the preemptive queues having a highest priority is emptied first, followed in order by any preemptive queues of decreasing priority.

As shown in step 508, if there is no priority data, the process 500 may continue to determine if it is time to serve a nested WRR queue. A weighted round robin schedule generally serves a WRR queue according to queue weight. However, in a nested WRR, one of the queues refers to a nested WRR queue which has its own queue schedule, but is accessed only periodically when the non-nested WRR queue reaches the nested queue in its own schedule. Thus if it is not time to serve the nested queue, the process 500 may proceed to step 510 where a packet is selected from the WRR queue according to a WRR schedule, and then return to step 504 where priority data (if any) is once again given preemptive attention. If it is time to serve a nested WRR—i.e., the nested queues are the current queue in the non-nested WRR schedule—then the process 500 may proceed to step 512 where a packet is selected from the nested WRR queue according to the nested WRR schedule. The process 500 may then return to step 504 where priority data (if any) is given preemptive attention.

It will be understood that the above illustration is provided by way of illustration and not limitation, and that numerous additions, deletions, or modifications to the steps above may be made without departing from the generality of this disclosure. For example, in one embodiment the nested WRR queue schedule may restart at its beginning each time it is served by the non-nested WRR. In another embodiment, the sequence of the nested WRR queue may be preserved between requests, so that each time the non-nested WRR queue returns to the nested WRR queue, the nested WRR queue may pick up where it left of in its sequence. Similarly, the non-nested WRR queue may either reset or continue each time it is pre-empted by priority data. As another example, the prioritized queues may be implemented asynchronously and separately from the WRR queues. In such embodiments, a preemptive queue may operate continuously, and may pause and pre-empt the WRR queue(s) whenever priority data is present.

A detailed example is now provided of dequeuing data according to a prioritized nested weighted round robin scheduling mechanism.

FIG. 6 shows a queue structure 600 containing packets in various queues. In particular, the queue structure contains the following packets:

10 packets in a high priority queue, PR1,

7 packets in a low priority queue, PR3,

9 packets in a first WRR queue, Q1,

7 packets in a second WRR queue, Q2,

7 packets in a nested WRR queue, Q4,

2 packets in a nested WRR queue, Q5, and

5 packets in a nested WRR queue, Q6.

FIG. 7 illustrates a scheduling sequence for the queues of FIG. 6. The sequence 700 is presented as a first timeline 702, a second timeline 704, and a third timeline 706, which collectively illustrate a sequence of packets served from the queue structure of FIG. 6. In a first timeline 702 representing eighteen serially scheduled packets, it can be seen that initially all ten packets in PR1 are pre-emptively scheduled. Second, all seven packets in the lower priority queue, PR2 are pre-emptively scheduled. After the preemptive data has been scheduled, the remaining queues may be served in weighted round robin fashion. This begins with a packet from Q1 as shown in the first timeline 602.

As shown in the second timeline 704, scheduling may proceed to serve queues Q1, Q2, and Q3 (nest) in round robin fashion according to queue weights. With weights of 6, 3, and 1 respectively, a weighted round robin will serve Q1 six times, Q2 three times, and Q3 once over every ten packet selections. In a weighted round robin schedule, the order may vary in any fashion provided the result is a correspondingly weighted service of the queues. As illustrated, six consecutive packets are selected from Q1, beginning with the last packet in the first timeline 702, and concluding with the first five packets in the second timeline 704. In weighted round robin fashion, three packets may then be selected from Q2. At this point, the nested WRR queues are served in proportion to their weight in the (non-nested) WRR queues. That is, with a weight of 1, the nested WRR queues are served once for each cycle of the WRR queues. As illustrated, this results in a selection of one packet from one of the nested WRR queues (Q4 in this example), followed by a return to the non-nested WRR queuing. The nesting is denoted in the timelines 702, 704, 706 by use of parenthesis in the WRR queue that signify the nesting point, with the actual selected packet shown in the corresponding time slot for the nested WRR queues.

At this point, there are no remaining priority packets, and the following packets in the WRR queues:

3 packets in a first WRR queue, Q1,

4 packets in a second WRR queue, Q2,

6 packets in a nested WRR queue, Q4,

2 packets in a nested WRR queue, Q5, and

5 packets in a nested WRR queue, Q6.

Returning to the non-nested WRR queues, Q1, Q2, and Q3 are once again served according to weight. Thus the three remaining packets in Q1 are scheduled, followed by three of the Q2 packets, followed by one reference to the nested WRR queues. Returning again to the top of the non-nested WRR schedule, only one packet remains in the non-nested queues, which packet is scheduled immediately for delivery. In the balance of the second timeline 704, all of the remaining packets for delivery are in the nested WRR queue, which may then proceed to serve packets from Q4, Q5, and Q6 in weighted fashion. As shown in FIG. 6, these weights are 5, 3, and 1 respectively. Thus, having already provided two packets from Q4, three additional packets may then be served from this queue as shown in the last time slot of the second timeline 704 and the first two time slots of the third timeline 706. Next, three packets may be served from Q5. However, only two packets remain, so these are served in sequence to empty queue Q5. Finally, one packet from Q6 may be served. At this point, the following packets remain for delivery from the queue structure:

2 packets in a nested WRR queue, Q4, and

4 packets in a nested WRR queue, Q6.

At this point, the nested WRR continues to serve packets from the queue structure according to the respective weights of Q4 (5) and Q6 (1). Accordingly, the remaining 2 packets are delivered from Q4, followed by the remaining 4 packets from Q6. At this point, all of the queues are empty, and no further scheduling will occur until further data is provided to the queue structure.

It will appreciated that the general notion of nesting round robin queues may be readily extended to accommodate multiple layers of nesting such as a top level WRR queue that contains a first nested WRR queue, with at least one additional nested WRR queue that is nested within the first nested WRR queue. The structure may also be extended by providing multiple groups of WRR queues at each level of nesting. Thus for example, a WRR queue may include a first queue having a weight of 7, a second queue having a weight of 3, a first nested WRR queue having a weight of 4, and a second nested WRR queue having a weight of 2. In this embodiment the first nested WRR queue will be accessed twice as often as the second nested WRR queue, and packets in the nested WRR queues will collectively receive (2+4=) 6 out of every (7+3+4+2=) 16 time slots, while scheduling within each nested queue may be arbitrarily established by weighting the queues therein. Thus as a general matter, relatively arbitrary, multilayer scheduling may be provided in order to achieve various service levels or other routing and network objectives.

Other variations and enhancements to the foregoing may also be provided. For example, the general approach described above may be adapted for use with directional antennas by using destination-based queuing instead of, or in addition to, traffic type(s). In other embodiments, the queues may be explicitly tied to certain traffic types, and weights may be periodically adjusted for these traffic types according to queue depth. In other embodiments, weights may be periodically adjusted according to node weights (described above) to improve the chances of meeting various service level commitments for traffic. Also as noted above, transitions between prioritized queues, WRR queues, and nested WRR queues may be managed in a number of fashions including restarting each group of queues each time it is accessed, or returning to a point in the schedule for that group of queues where a last access or service was made.

A wide range of software and hardware platforms may be used to deploy the systems and methods described herein. Generally, the system components may be realized in hardware, software, or some combination of these. The components may be realized in one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable devices, along with internal and/or external memory such as read-only memory, programmable read-only memory, electronically erasable programmable read-only memory, random access memory, dynamic random access memory, double data rate random access memory, Rambus direct random access memory, flash memory, or any other volatile or non-volatile memory for storing program instructions, program data, and program output or other intermediate or final results. The components may also, or instead, include one or more application specific integrated circuits (ASICs), dedicated semiconductor devices, programmable gate arrays, programmable array logic devices, or any other device that may be configured to process electronic signals.

Any combination of the above circuits and components, whether packaged discretely, as a chip, as a chip set, or as a die, may be suitably adapted to use with the systems described herein. It will further be appreciated that the above components may be realized as computer executable code created using a structured programming language such as C, an object oriented programming language such as C++, or any other high-level or low-level programming language that may be compiled or interpreted to run on one of the above devices, as well as heterogeneous combinations of processors, processor architectures, or combinations of different hardware and software. Any such combination of hardware and software suitable for use in an ad hoc network as described herein may be employed without departing from the scope of this disclosure.

Those skilled in the art will recognize, or will be able to ascertain using no more than routine experimentation, numerous equivalents to the systems and methods described herein. Such equivalents are considered to fall within the scope of the present invention. Moreover, the embodiments described herein are intended to exemplify the invention and not to limit it. While the invention is described above in connection with certain preferred embodiments, other embodiments may be understood by those of ordinary skill in the art. All such variations, modifications, extensions, additions, omissions, and the like as would be apparent to one of ordinary skill in the art are intended to fall within the scope of this disclosure, which is to be interpreted in the broadest sense allowable by law.

Claims

1. A method comprising:

storing a plurality of data packets in a plurality of queues for transmission in a number of time slots from a node of a mobile ad hoc network, each one of the plurality of queues having a weight;

selecting a first data packet from the plurality of data packets for transmission in one of the number of time slots according to a first weighted round robin schedule that is weighted to serve a first group of the plurality of queues according to their respective weights; and

selecting a second data packet from the plurality of data packets according to a second weighted round robin schedule that is weighted to serve a second group of the plurality of queues according to their respective weights, wherein the first weighted round robin schedule includes a weight for the second round robin schedule and periodically serves the second weighted round robin schedule according to the weight, thereby selecting the second data packet in the first weighted round robin schedule for transmission in one of the number of time slots.

2. The method of claim 1 further comprising preemptively selecting data packets from a prioritized queue until the prioritized queue is empty.

3. The method of claim 2 further comprising:

providing a plurality of prioritized queues, each one of the prioritized queues having a priority; and

preemptively selecting data packets from the plurality of prioritized queues according to the priority until each one of the prioritized queues is empty.

4. The method of claim 2 further comprising assigning a Quality of Service level to the prioritized queue, whereby data having the corresponding Quality of Service is placed into the prioritized queue and preemptively scheduled for transmission.

5. The method of claim 1 further comprising assigning a weight to at least one of the plurality of queues according to a Quality of Service level for that queue.

6. The method of claim 1 wherein at least one of the second group of the plurality of queues has a lowest priority of the plurality of queues.

7. The method of claim 1 wherein the second group of the plurality of queues includes at least one best-efforts queue for which delivery is not assured.

8. The method of claim 1 wherein the weight used by the first weighted round robin schedule to serve the second weighted round robin schedule is a lowest one of the weights used by the first weighted round robin schedule.

9. The method of claim 1 wherein the first weighted round robin schedule serves a plurality of additional weighted round robin schedules.

10. The method of claim 1 wherein the second weighted round robin schedule serves a third weighted round robin schedule.

11. The method of claim 1 further comprising adjusting one or more weights for the first weighted round robin schedule according to a queue depth for one or more of the first group of the plurality of queues.

12. The method of claim 1 further comprising adjusting one or more weights for the second weighted round robin schedule according to a queue depth for one or more weights of the second group of the plurality of queues.

13. The method of claim 1 further comprising filling at least one of the number of time slots with data from a current one of the queues before moving to a next queue in the weighted round robin schedule.

14. A computer program product comprising computer executable code that, when executing on one or more computing devices, performs the steps of:

storing a plurality of data packets in a plurality of queues for transmission in a number of time slots from a node of a mobile ad hoc network, each one of the plurality of queues having a weight;

selecting a first data packet from the plurality of data packets for transmission in one of the number of time slots according to a first weighted round robin schedule that is weighted to serve a first group of the plurality of queues according to their respective weights; and

selecting a second data packet from the plurality of data packets according to a second weighted round robin schedule that is weighted to serve a second group of the plurality of queues according to their respective weights, wherein the first weighted round robin schedule includes a weight for the second round robin schedule and periodically serves the second weighted round robin schedule according to the weight, thereby selecting the second data packet in the first weighted round robin schedule for transmission in one of the number of time slots.

15. The computer program product of claim 14 further comprising computer executable code that performs the step of providing a prioritized queue and preemptively selecting data packets from the prioritized queue until the prioritized queue is empty.

16. The computer program product of claim 15 further comprising computer executable code that performs the steps of:

providing a plurality of prioritized queues, each one of the prioritized queues having a priority; and

preemptively selecting data packets from the plurality of prioritized queues according to the priority until each one of the prioritized queues is empty.

17. The computer program product of claim 15 further comprising computer executable code that performs the step of assigning a Quality of Service level to the prioritized queue, whereby data having the corresponding Quality of Service is placed into the prioritized queue and preemptively scheduled for transmission.

18. The computer program product of claim 14 further comprising computer executable code that performs the step of assigning a weight to at least one of the plurality of queues according to a Quality of Service level for that queue.

19. The computer program product of claim 14 wherein at least one of the second group of the plurality of queues has a lowest priority of the plurality of queues.

20. A device comprising:

a data source that provides a plurality of data packets;

a queue that schedules the plurality of data packets for transmission according to a weighted round robin, the weighted round robin including at least one weight for a nested weighted round robin queue, the nested weighted round robin queue served according to its weight in the weighted round robin, thereby providing scheduled packets;

a radio that provides an air interface to a mobile ad hoc network including links to a plurality of neighboring nodes; and

a signal processor that prepares the scheduled packets for transmission over the air interface.