SYSTEM FOR DELAY OPTIMIZATION FOR SCHEDULING IN BUFFERLESS CROSSBAR SWITCHES

Abstract

A system for delay optimization scheduling in bufferless crossbar switches includes a plurality of line cards, each line card having an ingress half, an egress half, and a partial scheduler, wherein each line card is configured to transmit scheduling information to a horizontal control broadcast network and a vertical control broadcast network; a plurality of couplers connected by control links to the line cards in a two-dimensional grid, organizing the line cards into rows and columns; a bufferless crossbar switch connected by data path links to each line card to the ingress half and the egress half; wherein control information is distributed in a first stage and a second stage of broadcasts.

Description

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. Ser. No. 11/047,510, filed Jan. 31, 2005, the contents of which are incorporated by reference herein in their entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to network computing, data transmission, Gigabit Ethernet, high speed data transmission, high speed networking, data coding and encoding, data assembly and formatting, cluster computers, parallel processing computers, InfiniBand networks, high performance computing, supercomputing and, in particular, to delay optimization, electrical and crossbar switches, and scheduling.

2. Description of Related Art

FIG. 1 shows a prior art data crossbar switch scheduling structure. The scheduling structure includes a control broadcast network 100 and partial schedulers 102 in each line card that separately control the operation of each line card on the basis of global control information. There are a number of pairs of ingress line cards 104 and egress line cards 106 equal to the number of ports on the data crossbar switch 108. The control broadcast network 100 broadcasts requests from the ingress half 104 of each line card to the partial schedulers 102 of all the other line cards.

In operation of the scheduling structure of FIG. 1, each line card can calculate, based on received requests, which line card its ingress half should send to and which line card its egress half should receive from, at any given time.

There is a need to improve upon the prior art data scheduling structure in several ways. First, in the prior art scheduling structure, each partial scheduler must have a control input port for receiving requests (control information) from every other line card on the switch. Therefore, the total number of control input ports for the whole switching fabric scales as n². For low numbers of line cards (e.g., 8 or less), the number of control input ports is manageable (e.g., 64 or less). However, for switching fabrics with high port counts (e.g., 32 or 64), the number of control input ports scales dramatically (e.g., to 1,024 or 4,096), drastically increasing the cost of the overall switching fabric. Second, in the prior art scheduling structure, the control broadcast network must replicate the control information from each ingress line card half to every other line card half. This replication operation also results in O(N²) scaling of the number of outputs of the control broadcast network and O(N) scaling of the replication factor for each input port's information, which becomes extremely difficult to implement as the number of switch ports is increased. Third, in the prior art scheduling structure, each partial scheduler must receive a separate control request from each ingress line card half. Each request is packed and transmitted as a packet generated by the ingress card half and received by each partial scheduler. Because each packet has some per-packet overhead, for packet framing and error correction and because the ratio of this overhead to useful request payload information decreases with larger requests, the overhead could be reduced by aggregating requests together, such that each partial scheduler only needs to receive fewer individual packets containing request information.

There is a need for scalability to large systems, such as 32 or 64 line cards that require 1,024 or 4,096 control ports in conventional switches. There is a need for there is a need for scalability of broadcast network replication operations and network outputs and there is a need to reduce per-packet overhead.

BRIEF SUMMARY OF THE INVENTION

The present invention is directed to a system for delay optimization scheduling in bufferless crossbar switches, including a plurality of line cards, each line card having an ingress half, an egress half, and a partial scheduler, wherein each line card is configured to transmit scheduling information to a horizontal control broadcast network and a vertical control broadcast network; a plurality of couplers connected by control links to the line cards in a two-dimensional grid, organizing the line cards into rows and columns; a bufferless crossbar switch connected by data path links to each line card to the ingress half and the egress half; wherein control information is distributed in a first stage and a second stage of broadcasts.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the present invention will become better understood with regard to the following description, appended claims, and accompanying drawings, where:

FIG. 1 is a block diagram showing a prior art data crossbar switch scheduling structure;

FIG. 2 is a block diagram showing an exemplary embodiment of a switch fabric;

FIG. 3 is a block diagram showing an exemplary embodiment of a logically partitioned switch fabric; and

FIG. 4 is a block diagram showing another exemplary embodiment of a switch fabric.

DETAILED DESCRIPTION OF THE INVENTION

Exemplary methods, systems, and storage mediums are described for delay optimization scheduling in bufferless crossbar switches, such as replicated distributed responseless crossbar switches.

FIG. 2 shows an exemplary embodiment of a switch fabric. The n-way control broadcast network 100 of FIG. 1 is replaced by multiple smaller broadcast networks in the exemplary embodiment of FIG. 2. The switch fabric improves scalability at manageable costs by arranging the control broadcast network as multiple intersecting and redundant two-stage control multicast networks, with distinct but redundant aggregation in each stage.

In this exemplary embodiment, a crossbar data switch 200 is in the center of a number of line cards 202 (or adapters). The line cards 202 are logically arranged in a two-dimensional grid with both horizontal and vertical directions. This horizontal and vertical organization describes the connectivity between the line cards on control links (dashed lines) and may or may not have any relationship to the physical positioning of the line cards 202 relative to each other.

The exemplary embodiment of FIG. 2 replaces the control broadcast network 100 of FIG. 1 with a number of smaller control broadcast networks. Each smaller control broadcast network operates in either the horizontal or vertical direction to link all the line cards 202 in a row or in a column. In this exemplary embodiment, control information is distributed in two stages of broadcasts, across rows and, then, across columns.

This exemplary embodiment includes a two-dimensional fanout broadcast network, which may be implemented with either electrical or optical fanout technology. Logically, the switch 200 is in the center and the line cards 202 are in a two-dimensional grid topology. There are multiple broadcast networks with one along each row and column. Each line card 202 broadcasts to all the cards in its row and column. Each line card 202 aggregates requests in its row to all cards in its column and aggregates requests in its column to all cards in its row.

There are many advantages to this exemplary embodiment. Aggregation of messages across a row and column reduces the per-packet overhead to a single message per row or column. The number of control ports on all the line cards is 2*N*sqrt(N), not N² as in the prior art. For N=64 or 256, this is a factor of 4 or 8 improvement. Each horizontal/vertical broadcast is sqrt(N), rather than N as in the prior art. This saves connections, which is especially important for all-optical architectures, considering the expense of optical components. Another advantage is the simplicity of assuring that each line card 202 may receive two copies of all scheduling data (one distributed first by row and, then, by column and one distributed first by column and, then, by row). This duplicated distribution of data assures that errors are better detected than in the prior art.

In some embodiments, a minimum delay topology is determined for the switch control network based on various switch parameters. Some examples of switch parameters include the number of nodes, the delay overhead for initiating a message, the delay per node hop delay, and others. Cost optimization of the control network may be based on the cost of the transmitter, the cost of the receiver, the cost of the broadcast, the cost of aggregation logic, and the like.

One exemplary embodiment operates as follows. Each line card 202 transmits information approximately simultaneously to the horizontal and vertical control broadcast networks to which the line card 202 is attached. The information transmitted is the information necessary for determining appropriate schedules, such as the status of ingress queues, ingress traffic prioritization, egress buffer availability on the egress portions of the line cards, and other information. Each line card 202 receives a number of requests from the control broadcast networks to which the line card 202 is attached. The requests are from the other line cards 202 in its row and the other line cards in its column. Each line card 202 aggregates the requests from its row in a single message. This message is forwarded to all the other line cards 202 in its column. Each line card 202 aggregates the requests from its column in a separate single message. This message is forwarded to all the other line cards 202 in its row. Of course, other embodiments may do columns first and then rows or any other order of steps.

Once each line card 202 has received the aggregated, forwarded requests from its row and column, it has received requests from all the other line cards 202 on the switch fabric. In fact, each line card 202 received two redundant copies of each request, which serve as a basis for detecting errors in transmission or errors in operation.

The partial scheduler 102 (FIG. 1) in each line card 202 determines the portion of the overall schedule (i.e., a partial schedule) that applies to it based on the aggregated and redundant requests from all the line cards 202.

Once the partial schedules are determined for each line card 202, the line cards 202 send data through the data crossbar switch 200 to/from their ingress sections to their scheduled output ports.

FIG. 3 shows an exemplary embodiment of a logically partitioned switch fabric. The logical grid of FIG. 2 may be divided into smaller square or rectangular grids in various ways in different exemplary embodiments. In FIG. 3, there are four partitions. With partitioning, the switch has some of the advantages of a local area network (LAN) switch without the associated overhead. When this is applied to server architectures that also permit logical partitioning of processor and storage resources, the switches may be partitioned in a complimentary manner. Partitioning is usually accomplished by using a mask in the control plane to limit the range of ports than can address each other. By partitioning (or zoning) the switch, it is possible to provide an effect equivalent to several smaller switches, which is convenient for some types of parallel processing applications. This exemplary embodiment enables new switch architectures with logical partitioning, zoning, and load balancing by masking the control plane.

In some embodiments, different quality of service levels are provided across different switch zones. In some embodiments, traffic is segregated for various purposes. For example, inter-switch traffic and switch-to-node traffic may be partitioned into different zones. In some embodiments, the logical partitions are re-configured in response to changing data traffic conditions. This provides the possibility of load balancing on the switch as well as optimizing utilization of the attached server and/or storage resources. In some embodiments, one or more partitions are configured as backups in case of a failure affecting another partition to increase the reliability of the switch.

For technologies where the per-packet overhead is large enough, it may take less time for a line card 2020 to receive 2*sqrt(N) messages, even if half of those messages contain aggregated requests across a whole node or column. In this case, the latency of scheduling may be reduced. The latency of scheduling is the time from the initial transmission of requests until the data can be transmitted through the crossbar.

In one embodiment, requests are originally transmitted both horizontally and vertically and then aggregated and transmitted both vertically and horizontally. At the end of distribution, each line card 202 receives requests from each requesting line card 202 through two different paths with two different aggregations. This allows error checking and filtering out of corrupted messages, before the partial schedulers 102 perform scheduling work.

There are many ways to implement the control broadcast in each of the two dimensions, horizontal and vertical. The architecture of the exemplary embodiment shown in FIG. 2 is well suited to optical cross connect switches. FIG. 2 shows a row or column control broadcast network implemented using optical n×n couplers 204 (with n=R or n=C, where typically R*C=N) in each of the rows and columns. The data paths to and from all nodes are interconnected using a crossbar switch 200 using any of several available electronic or optical technologies to construct the data crossbar.

Suppose an exemplary embodiment has 32 ports with 4 rows and 8 columns, (i.e., R=4 and C=8). In this example, 4×4 couplers and 8×8 couplers are used. Each line card 202 sends requests through a 44×44 coupler to the line cards in each row of its column. Then, the 8×8 couplers in each row are used to fan out requests to the other line cards 202 in each column of each row. This distribution across all rows and all columns replicates data from all the line cards to all the other line cards.

Another exemplary embodiment implements row and column broadcasts with electronic switching application-specific integrated circuits (ASICs). In some embodiments, separate electronic switching ASICs are implemented for the control broadcast networks 100 separately for each row and each column. These embodiments would have electronic switching ASICs in place of the couplers 204 shown in FIG. 2.

Another exemplary embodiment has a single switch for each row and column pair, in order to halve the number of electronic switching ASICs, while doubling the number of ports per ASIC. With this topology, each of the electronic switching ASICs only performs control broadcasting within a row or column separately and the line cards 202 perform aggregation of row and column requests, before broadcasting them to across their columns and rows.

FIG. 4 shows this exemplary embodiment of a switch fabric. Because FIG. 4 shows a 64-port switch fabric with 8 columns and 8 rows, 16-port electronic packet switched ASICs 404 are used to distribute and replicate control information across the rows and columns. FIG. 4 shows eight separate 16-port switches 404 (one switch per row and column, for redundancy and fast connectivity) instead of 16 separate 8-port switches (per row and per column). The packet switches may be InfiniBand switches, or switches of another architecture that is capable of transporting packets carrying control traffic.

The exemplary embodiments of the present invention have many advantages. Because the number of rows and columns are each proportional to sqrt(N), the number of control input ports in the line cards 202 for a switching fabric is 2*N*sqrt(N), rather than N². For N=64, for example, this is reduction of a factor of 4 in overall input ports, possibly significantly decreasing the cost of the control broadcast network 100. Because control broadcast occurs separately in horizontal and vertical directions, each broadcast is only a sqrt(N)-way broadcast, rather than an N-way broadcast. For certain types of broadcast technologies, such as optical broadcast, this may be a significant difference. Each line card receives and manages 2*sqrt(N) different control request messages. Although some of the messages contain aggregated and forwarded requests across a row or column, the per-message overhead may be low so that the bandwidth utilization for each line card 202 is reduced. The grid structure allows the switch to be logically partitioned into any desired configuration of subsystems.

As described above, the exemplary embodiments of the invention may be embodied in software, hardware, firmware or any combination thereof. Embodiments of the invention may also be embodied in the form of computer program code containing instructions embodied in tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other computer-readable storage medium, wherein, when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for practicing the invention. The present invention can also be embodied in the form of computer program code. For example, whether stored in a storage medium, loaded into and/or executed by a computer, or transmitted over some transmission medium, such as over electrical wiring or cabling, through fiber optics, or via electromagnetic radiation, when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for practicing the invention. When implemented on a general-purpose microprocessor, the computer program code segments configure the microprocessor to create specific logic circuits.

While the invention has been described with reference to exemplary embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the scope of the invention. Furthermore, various components may be implemented in hardware, software, or firmware or any combination thereof. Finally, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from the essential scope thereof. Therefore, it is intended that the invention is not to be limited to the particular embodiment disclosed as the best or only mode contemplated for carrying out this invention, but that the invention will include all embodiments falling within the scope of the appended claims. Moreover, the use of the terms first, second, etc. do not denote any order or importance, but rather the terms first, second, etc. are used to distinguish one element from another. Furthermore, the use of the terms a, an, etc. do not denote a limitation of quantity, but rather denote the presence of at least one of the referenced item.

Claims

1. A system for delay optimization scheduling in bufferless crossbar switches, comprising:

a plurality of line cards, each line card having an ingress half, an egress half, and a partial scheduler, wherein each line card is configured to transmit scheduling information to a horizontal control broadcast network and a vertical control broadcast network;

a plurality of couplers connected by control links to the line cards in a two-dimensional grid, organizing the line cards into rows and columns;

a bufferless crossbar switch connected by data path links to each line card to the ingress half and the egress half;

wherein control information is distributed in a first stage and a second stage of broadcasts.

2. The system of claim 1, wherein the bufferless crossbar switch is a replicated distributed responseless crossbar switch.

3. The system of claim 1, wherein each coupler operates as a control broadcast network in either a horizontal or a vertical direction.

4. The system of claim 1, wherein control information is distributed across the rows in the first stage and across the columns in the second stage, so that each line card receives redundant control information from all the other line cards.

5. The system of claim 1, wherein a minimum delay topology is determined based on one or more of: a number of nodes, a delay overhead for initiating a message, and a delay per node hop delay.

6. The system of claim 1, wherein the two-dimensional grid is logically partitioned into a plurality of partitions by using a mask in a control plane to limit a range of addressable ports.

7. The system of claim 6, wherein different levels of quality of service are provided in different partitions.

8. The system of claim 6, wherein inter-switch traffic and switch-to-node traffic are partitioned into different partitions.

9. The system of claim 6, wherein the partitions are re-configurable in response to changing traffic conditions.

10. The system of claim 6, wherein at least one partition is configured as a backup.

11. The system of claim 1, wherein each line card transmits the scheduling information approximately simultaneously to the horizontal and the vertical control broadcast networks connected to that line card.