SYSTEM OF IMPLEMENTING SWITCH DEVICES IN A SERVER SYSTEM
A system of implementing devices in a system. At least some of the illustrative embodiments are systems comprising a plurality of devices of a first type coupled to a coupling medium, and a first and second slots (where each slot enables coupling of a device of a second type to the coupling medium). The first and second slots accept second type devices one each in each slot, and the first and second slots accept a single device spanning both slots.
This application is a continuation of U.S. patent application Ser. No. 11/554,294, filed Oct. 30, 2006, titled “A System of Implementing Switch Devices in a Server System,” which claims the benefit of U.S. provisional patent application Ser. No. 60/813,752, filed Jun. 14, 2006, titled “Methods to Minimize Backplane and Enclosure Complexity for Modular Servers,” which are hereby incorporated herein by reference in their entirety for all purposes.
BACKGROUNDA major component of electronic information access and retrieval is network accessible computer systems known as servers. High density servers (also known as blade servers) are implemented in “enclosures” mounted within racks. The servers electrically couple to a first side of a midplane board within the enclosure, and switch devices couple to a second side of the midplane board, thus forming complete server systems.
By way of the midplane board, the servers may couple to multiple types of switch devices. One type of switch device is a network switch device, which acts to perform message routing for a communication network. The servers communicate with the network switch devices across the midplane board using communication-based protocols, such as Ethernet.
Other switch devices with which the servers may communicate are storage switch devices. The servers themselves may have no, or relatively little, onboard hard drive storage space. The hard drive storage space is made accessible to the servers through the storage switch devices coupled to the midplane board. The servers communicate with storage switch devices across the midplane board using storage-based protocols, such as Fibre Channel®, Serial Attached SCSI (SAS), and may also communicate with storage devices using other non-storage-based protocols, such as Peripheral Components Interconnect express (PCIe).
Yet still other switch devices with which the servers may communicate are parallel computing switch devices, which switch devices network the servers together for purposes of parallel computing. The servers communicate with parallel computing switch devices across the midplane board using cluster-networking protocols, such as InfiniBand®.
In order to support the possible switch devices to which the servers may need to communicate, the midplane board implements dedicated slots and signal paths for each type communication protocol. For example, a slot (and its respective connector on the midplane board) may be dedicated only to InfiniBand communication, and no other switch device will be operable in the slot. Creating dedicated slots and signal paths limits flexibility of a single enclosure, and may require manufactures to create multiple midplane boards and enclosures to support the differing needs of consumers.
For a detailed description of exemplary embodiments, reference will now be made to the accompanying drawings in which:
Certain terms are used throughout the following description and claims to refer to particular system components. As one skilled in the art will appreciate, computer companies may refer to a component by different names. This document does not intend to distinguish between components that differ in name but not function.
In the following discussion and in the claims, the terms “including” and “comprising” are used in an open-ended fashion, and thus should be interpreted to mean “including, but not limited to . . . .” Also, the term “couple” or “couples” is intended to mean either: an indirect, direct, optical or wireless electrical connection; or an indirect or direct mechanical connection. Thus, if a first device couples to a second device, that connection may be through a direct connection, or through an indirect connection via other devices and connections.
DETAILED DESCRIPTIONThe following discussion is directed to various embodiments of the invention. Although one or more of these embodiments may be preferred, the embodiments disclosed should not be interpreted, or otherwise used, as limiting the scope of the disclosure, including the claims. In addition, one skilled in the art will understand that the following description has broad application, and the discussion of any embodiment is meant only to be exemplary of that embodiment, and not intended to intimate that the scope of the disclosure, including the claims, is limited to that embodiment.
Regardless of the precise function of the servers 10, the servers 10 may communicate with other devices to implement the desired functionality. In order to communicate with other computer systems connected to the Internet 12, the servers 10 may couple to a network communications switch device 14. Although only one network communications switch device 14 is shown, any number of network communication switch devices 14 may be implemented (e.g., a number such that each server 10 couples to two different network communication switch devices for purposes of fault tolerance). The network communications switch device may perform message packet routing between the Internet 12 and the servers 10. The servers 10 communicate with the network communications switch device 14 by way of a network communication protocol (e.g., Ethernet protocol).
In some embodiments the servers 10 implement little or no on-board long term non-volatile storage (e.g., hard drive space), and thus in order to have access to long term non-volatile storage the servers 10 may couple to a storage switch device 16. Although only one storage switch device 16 is shown, any number of storage switch devices 16 may be implemented. The storage switch device 16 in turn couples to long term non-volatile storage, such as an internal or external hard drive 18. In some embodiments, the storage switch device 16 couples to a plurality of hard drives and implements a redundant array of independent (or inexpensive) disks (RAID) for purposes of fault tolerance with respect to drive failure. The servers 10 communicate with the storage switch device 16 using a storage communication-based protocol (e.g., Fibre Channel®, Serial Attached SCSI (SAS) or Peripheral Components Interconnect express (PCIe)).
In yet still other embodiments, the servers 10 participate in a parallel computing architecture, and thus may need to couple to each other and other computer systems in a specialized manner to implement the parallel computing. In accordance with these embodiments, the servers 10 couple to a parallel computing switch device 20, which performs message-passing packet switching for parallel computing operations. The servers 10 communicate with the message-passing packet switching device 20 using a parallel computing communication-based protocol (e.g., InfiniBand®, RDMA-over-Ethernet).
Still referring to
In accordance with at least some embodiments, the enclosure 30 has eight collimated slots, thus accepting sixteen half-height servers, eight full-height servers, or combinations of full- and half-height servers. However, enclosure 30 may accommodate any number of servers, and the servers need not necessarily be in vertically oriented columns.
The physical layer connections used by each communication protocol may differ not only as between the different protocols, but also within a protocol. Considering first the network communication protocols, some Ethernet communications (e.g., 1000-Base-KX and 10G-Base-KR) utilize four electrical traces at the physical layer for communication (the four traces being two differential communication pairs comprising one differential receive pair and one differential transmit pair). However, 10G-Base-KX4 Ethernet uses sixteen electrical traces at the physical layer. As for storage network communication, Fibre Channel uses four electrical traces, Serial Attached SCSI (SAS) uses four traces for 1x SAS and eight trances for 2x SAS, PCIe uses from four traces (for 1x PCIe) to sixteen traces (for 4x PCIe). As for communications for parallel computing, InfiniBand 4x uses sixteen traces at the physical layer. In the related art, traces on the midplane boards are dedicated to particular protocols (e.g., traces designed for InfiniBand would not be used for Ethernet, and vice versa). Dedicating midplane board traces to particular protocols limits the implementation flexibility of each midplane board (and thus each enclosure within which a midplane board is installed). For example, a slot design for use with a parallel computing switch device communicating using the InfiniBand protocol could not be used for a network communications switch device communicating using Ethernet in the related art.
In accordance with the various embodiments, traces on the midplane board 22 between the switch device connectors 42 and the server connectors 38 are not dedicated to any particular communications protocol, and indeed may carry signals from differing protocols depending on the type of switch device coupled to the midplane board 22. It follows that the switch device slots (e.g., 44 and 46) are not limited to any particular type of switch device. Moreover, a single-wide slot (which itself may accept switch devices of multiple types) may be considered with a contiguous single-wide slot to form a double-wide slot to accept switch devices having a double-wide form factor, and using more traces for communication. Attention now turns to the midplane board 22 layout to implement the various embodiments.
In accordance with the various embodiments, each switch device connector 42 couples to each server connector 38 by way of electrical traces on the midplane board 22. More particularly, each switch device connector 42 couples to each server connector 38 by way of at least eight electrical traces of the midplane board 22 in a star topology (additional traces, e.g., clock signal traces, may also be used). For an illustrative system having eight switch device connectors 42, sixteen server connectors 38 and eight traces for each server to switch device connection, each switch device connector 42 thus implements 128 connection points or pins (eight pins per connection times 16 server connectors) dedicated to communication between servers and switch devices. Additional pins may be present, for example, to supply power to the switch devices and to allow communication between the switch devices and the enclosure manager 40.
Considering illustrative single-wide slot 44 and its respective connector 42A, a single-wide switch device installed in the single-wide slot 44 thus has the capability of communicating to each and every server in the system over at least eight electrical traces (if the server are half-height servers), and the switch device can communicate to a full-height server over 16 electrical traces (because the full height server couples to two server connectors 38). As for the single-wide switch device and the half-height server, the switch device can communicate using any protocol requiring eight traces or less (e.g., 1000-Base-KX Ethernet (four traces), 10G-Base-KR Ethernet (four traces), Fibre Channel (four traces), 1x Serial Attached SCSI (four traces), 2x Serial Attached SCSI (eight traces), 1x PCIe (four traces) or 2x PCIe (eight traces)). Thus, a single-wide switch device (coupling to one switch device connector 42) could be any now existing or after developed switch device needing eight or fewer traces for communication (e.g., network communications switch device or storage switch device).
As for a single-wide switch device and the full-height server (connecting to two server connectors 38), the switch device can communicate using any communication protocol requiring eight or less traces (e.g., all of those discussed for the single-wide switch device and half-height server, along with 2x PCIe (eight traces) and 2x InfiniBand (eight traces)). A full-height server double the number of connections to the switch devices, compared to a half-height server. Doubling the number of connections doubles the number of ports. Thus, in the case of the single-wide switch device and the full-height server, the switch device may be any of the switch devices discussed with respect to
Now consider a double-wide switch device that occupies two single-wide slots (e.g., 44 and 46) and thus couples to two switch device connectors 42. In this illustrative situation, the double-wide switch device has sixteen available traces to each half-height server (coupled to a single server connector 38). In the illustrative case of occupying slots 44 and 46, eight traces of the sixteen are available through switch device connector 42A, and eight traces through switch device connector 42B. A double-wide switch device can communicate using any communication protocol having sixteen traces (e.g., 10G-Base-KX4, 4x PCIe and 4x InfiniBand). As for using a double-wide switch device and a full-height server (coupled to two server connectors 38), the double-wide switch device has 32 available traces for communication, sixteen through each switch device connector 42.
Now consider a switch device that not only couples to two switch device connectors in the same horizontal plane (e.g., 42A and 42B, and thus a double-wide), but also connects to two switch device connectors in contiguous slots (e.g., connectors 42C and 42D, and thus a double-wide, double-high switch device). In these illustrative situations, the switch device has 32 available traces to each half-height server (eight traces each through connectors 42A, 42B, 42C and 42D). As for using a double-wide, double-high switch device and a full-height server (coupled to two server connectors 38), the double-wide, double-high switch device has 64 traces for communication, sixteen traces through each switch device connector 42.
As discussed above, each switch device connector 42 couples to each and every server connector 38. If follows that each switch device coupled within the enclosure 30 likewise has the ability to communicate with each server over at least eight electrical traces, and more traces if either the switch device couples to multiple switch device connectors 42 or the server couples to multiple server connectors 38. However, while each server and switch device may have the ability to communicate, each server may not need to communicate with each switch device. In order to selectively couple the servers to the switch devices, computing systems 1000 in accordance with at least some embodiments use electrical boards (termed mezzanine boards) within each server which allow selectively coupling of the server to switch devices.
Inside the illustrative server 50 reside mezzanine connectors 60A and 60B, one mezzanine connector 60 for each server connector 52. In accordance with embodiments of the invention, coupling of the traces between the server connectors 52 and the corresponding mezzanine connectors 60 vary as between server connector and mezzanine connector pairs. In this way mezzanine cards 62A and 62B, having the same configuration, may couple to different switch devices 56 dependent upon which mezzanine connector 60 is used by the mezzanine card 62. In the illustrative situation of
Still referring to
Attention now turns to assignment of traces to either transmit or receive for particular implemented protocols. In situations where single-wide switch devices are used, the transmit and receive pairs by definition reside within the same switch device connector 42. However, in situations where a double-wide switch devices, or double-wide, double-high switch devices are used, the electrical traces on the midplane board 22 between the switch device and each server span multiple switch device connectors 42. In these situations, and in accordance with at least some embodiments, traces assigned to transmitting from the switch device to the server may be within different connectors than traces assigned to receiving from servers by the switch devices.
Returning now to
The above discussion is meant to be illustrative of the principles and various embodiments of the present invention. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.
Claims
1. A system, comprising:
- a plurality of first connectors on a coupling medium to couple to a first type of device;
- first and second slots enabling coupling of a second type of device to the coupling medium;
- a second connector associated with the first slot, wherein the second connector communicatively couples to each of the first connectors; and
- a third connector associated with the second slot, wherein the third connector is communicatively coupled to each of the first connectors;
- wherein, when each of the first and second slots accepts a separate second type of device, the second type of device in the first slot communicates with at least one of the first type of devices using a first bandwidth communication protocol; and
- wherein, when the first and second slots accept a single second type of device spanning both slots, the single second type of device communicates with at least one of the first type of devices using a second bandwidth communication protocol, the second bandwidth being greater than the first bandwidth.
2. The system as defined in claim 1 wherein the second type of devices are selected from the group consisting of: a device that couples at least one server to a communications network; a device that couples at least one server to a storage device; and a device that couples to at least one server for purposes of parallel computing.
3. The system as defined in claim 1 further comprising:
- third and fourth slots, where each of the third and fourth slot enables coupling of a device of the second type to the coupling medium;
- wherein the third and fourth switch slots is to accept a single second type device spanning both the third and fourth slots, and wherein the first, second, third and fourth slots accept a single second device spanning the first, second, third and fourth slots.
4. A system, comprising:
- a plurality of first connectors on a coupling medium to couple to a first type of device; and
- first, second, third, and fourth slots to enable coupling of a second type of device through the coupling medium to each of the first connectors, said second type of device including at least one of a single-wide device, a double-wide device and a double-wide, double-high device;
- wherein, when any of the first, second, third and fourth slots accepts a single-wide device, such single-wide device is to communicate with at least one of the first type of devices;
- wherein, when a pair of the first, second, third, and fourth slots accepts a double-wide device spanning the pair of slots, such double-wide device is to communicate with at least one of the first type of devices; and
- wherein, when the first, second, third, and fourth slots accept a double-wide, double-high device spanning all four switch slots, the double-wide, double-high device communicates with at least one of the first type of devices.
5. The system of claim 4 wherein communication bandwidth is different among the single-wide device, double-wide device, and double-wide, double-high device.
6. The system of claim 5 wherein the communication bandwidth of the double-wide, double-high device is greater than that of the double-wide device, and the communication bandwidth of the double-wide device is greater than that of the single-wide device.
Type: Application
Filed: Oct 31, 2014
Publication Date: Feb 26, 2015
Inventor: Kevin B. LEIGH (Houston, TX)
Application Number: 14/530,175
International Classification: G06F 13/40 (20060101); G06F 13/42 (20060101);