ADVANCED FLUID CONNECTION DESIGN FOR MULTIPLE PHASE SYSTEM
A cooling arrangement for an electronic rack of a data center having a rack manifold with liquid supply line to receive first cooling liquid from a cooling liquid source and liquid return line to return first warmer liquid back to the cooling liquid source. A plurality of server chassis are arranged in a stack, each server chassis including one or more fluid cooling devices associated with one or more information technology (IT) components. A plurality of connector modules are attached to the servers, each providing fluid connections between the fluid cooling devices and the cooling liquid source on the racks; wherein each of the connector module comprises a frame having mounts configured for attaching the frame onto the server a plurality of channels formed in the frame to accept the fluid connectors; and, a plurality of fluid connectors slidably mounted in the channels.
Embodiments of the present invention relate generally to data center and electronics cooling. More particularly, embodiments of the invention relate to modular fluid connection and hardware design particularly beneficial for multiple phase systems.
BACKGROUNDCooling is a prominent factor in a computer system and data center design. The number of high performance electronics components such as high performance processors packaged inside servers has steadily increased, thereby increasing the amount of heat generated and dissipated during the ordinary operations of the servers. The reliability of servers used within a data center decreases if the environment in which they operate is permitted to increase in temperature over time. Maintaining a proper thermal environment is critical for normal operations of these servers in data centers, as well as the server performance and lifetime. It requires more effective and efficient cooling solutions especially in the cases of cooling these high performance servers.
The servers and different types of IT equipment, which can be understood as different compute, storage and other functions based systems include multiple connectors are provided in the rear to deliver electrical communication, power, cooling fluids, etc. Such connectors may be blind mate connectors, wherein the male-female connection is done blindly by self-alignment of the mating connectors, such that a user need not reach the back of the unit to connect or disconnect the various connectors. To enable such blind connections, the connectors must be located at a fixed orientation to accept the mating connection. However, with the incorporation of advanced multi-phase cooling systems, different systems may require different arrangement for the fluid connectors, thus exacerbating the design of fluid cooling connections.
The interoperability of different cooling elements is important to accommodate different rack configurations, fluid systems and rack level fluid port availabilities in actual uses.
Embodiments of the invention are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements.
Various embodiments and aspects of the inventions will be described with reference to details discussed below, and the accompanying drawings will illustrate the various embodiments. The following description and drawings are illustrative of the invention and are not to be construed as limiting the invention. Numerous specific details are described to provide a thorough understanding of various embodiments of the present invention. However, in certain instances, well-known or conventional details are not described in order to provide a concise discussion of embodiments of the present inventions.
Reference in the specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in conjunction with the embodiment can be included in at least one embodiment of the invention. The appearances of the phrase “in one embodiment” in various places in the specification do not necessarily all refer to the same embodiment.
Disclosed embodiments provide a flexible interface for connectors in a computing system, especially suitable for blind mate fluid connectors in systems that may use multiple phase fluid cooling. In such systems fluid flow may rely on gravity and/or fluid pressure, such that the locations of the connectors may differ for each supply and return lines of each FRU within the enclosure. Disclosed embodiments introduce an interface that enables to easily rearrange the location of the blind mate connectors for the fluids.
In disclosed embodiments the interface comprises a frame mountable onto the computing enclosure and having a plurality of channels or tracks. Each channel may slidably accept one or more connectors, such that the connector may slide within the channel to different positions within the channel. Preset positions may be configured for each channel, such that each connector may be slide to one of the preset position. Each pair of connectors may be interconnected such that they move in unison, so as to maintain a fixed spatial orientation there-between. Thus, moving one connector would force the paired connector to move as well to maintain the spatial orientation.
In one aspect, a connector module for electronic rack includes a frame having mounts configured for attaching the frame onto the electronic rack, a plurality of channels formed in the frame to accept fluid connectors, and a plurality of fluid connectors slidably mounted in the channels. The connector module further includes an interconnector attached to at least two fluid connectors thereby maintaining fixed spatial orientation between the two fluid connectors. Each of the fluid connector further comprises a locking mechanism affixing the fluid connector in a fixed location within the channel. The locking mechanism comprises tight fit of the fluid connector within the channel imparting motion resistance of the fluid connector within the channel preventing free slide. The locking mechanism includes arms extending from the fluid connector and engaging teeth within the channel. The frame further includes a plurality of locators positioned at preset locations and configured for locking the fluid connectors at the preset locations. Each of the fluid connector includes an interconnect kit configured for interconnecting each of the fluid connector to another fluid connector to maintain fixed spatial orientation.
According to another aspect, an electronic rack includes a rack manifold having a rack liquid supply line to receive first cooling liquid from a cooling liquid source and a rack liquid return line to return first warmer liquid back to the cooling liquid source; a plurality of server chassis arranged in a stack, each server chassis including one or more fluid cooling devices associated with one or more information technology (IT) components; and a plurality of connector modules attached to the electronic rack, each providing fluid connections between the fluid cooling devices and the cooling liquid source. Each connector module includes components as described above.
According to a further aspect, a method for providing cooling fluid to an electronic rack of a plurality of server chassis includes mounting a plurality of connector modules to the electronic rack, each connector module provided with at least two channels; attaching a plurality of fluid connectors to the connector modules by attaching at least one fluid connector in each of the channels; and for each of the server chassis, verifying spatial orientation of inlet connector and outlet connector and sliding two of the plurality of connectors to match the spatial orientation of the inlet connector and outlet connector, and securing the two connectors in the spatial orientation, and then mounting the server chassis in the electronic rack by engaging the two connectors with the inlet connector and outlet connector.
The securing the two connectors comprises positioning the two connectors in selected predefined dedicated locations identified by pre-arranged locators applied to the channels. The method further includes mounting a multi-device chassis having multiple cooling devices by engaging the multi-device chassis with multiple pairs of fluid connectors. The method further includes, for each pair of fluid connectors, affixing a riser connector at higher elevation than a return connector.
The following disclosure starts by providing background information regarding the application of the disclosed embodiments, and then proceeds to disclose specific embodiments. In this context, the embodiments may refer simply to connectors, and such reference is intended to include blind mate fluid connectors, including such connectors that may be engaged and disengaged without fluid leaks. Such connectors may include a valve that closes when a mating connector is disconnected, so as to prevent fluid from leaking.
In one embodiment, each of the electronic racks (e.g., electronic racks 110A-110N) includes a housing to house a number of IT components arranged in a stack operating therein. The electronic racks can include a cooling liquid manifold, a number of server slots (e.g., standard shelves or chassis configured with an identical or similar form factor), and a number of server chassis (also referred to as server blades or server shelves) capable of being inserted into and removed from the server slots. Each server chassis represents a computing node having one or more processors, a memory, and/or a persistent storage device (e.g., hard disk), where a computing node may include one or more servers operating therein. At least one of the processors is attached to a liquid cold plate (also referred to as a cold plate assembly) to receive cooling liquid. In addition, one or more optional cooling fans are associated with the server chassis to provide air cooling to the computing nodes contained therein. Note that the cooling system 120 may be coupled to multiple data center systems such as data center system 100.
In one embodiment, cooling system 120 includes an external liquid loop connected to a cooling tower or a dry cooler external to the building/housing container. The cooling system 120 can include, but is not limited to evaporative cooling, free air, rejection to large thermal mass, and waste heat recovery designs. Cooling system 120 may include or be coupled to a cooling liquid source that provide cooling liquid.
In one embodiment, each server chassis is coupled to the cooling liquid manifold modularly, such that a server chassis can be removed from the electronic rack without affecting the operations of remaining server chassis in the electronic rack and the cooling liquid manifold. In another embodiment, each server chassis is coupled to the cooling liquid manifold through a quick-release coupling assembly having a server liquid intake connector and a server liquid outlet connector coupled to a flexible hose to distribute the cooling liquid to the processors. The server liquid intake connector is to receive cooling liquid via a rack liquid intake connector from a cooling liquid manifold mounted on a backend of the electronic rack. The server liquid outlet connector is to emit warmer or hotter liquid carrying the heat exchanged from the processors to the cooling liquid manifold via a rack liquid outlet connector and then back to a coolant distribution unit (CDU) within the electronic rack.
In one embodiment, the cooling liquid manifold disposed on the backend of each electronic rack is coupled to liquid supply line 132 (also referred to as a room supply manifold) to receive cooling liquid from cooling system 120. The cooling liquid is distributed through a liquid distribution loop attached to a cold plate assembly on which a processor is mounted to remove heat from the processors. A cold plate is configured similar to a heat sink with a liquid distribution tube attached or embedded therein. The resulting warmer or hotter liquid carrying the heat exchanged from the processors is transmitted via liquid return line 131 (also referred to as a room return manifold) back to cooling system 120.
Liquid supply/return lines 131-132 are referred to as data center or room liquid supply/return lines (e.g., global liquid supply/return lines), which supply cooling liquid to all of the electronic racks of rows 101-102. The liquid supply line 132 and liquid return line 131 are coupled to a heat exchanger of a CDU located within each of the electronic racks, forming a primary loop. The secondary loop of the heat exchanger is coupled to each of the server chassis in the electronic rack to deliver the cooling liquid to the cold plates of the processors.
In one embodiment, data center system 100 further includes an optional airflow delivery system 135 to generate an airflow to cause the airflow to travel through the air space of the server chassis of the electronic racks to exchange heat generated by the computing nodes due to operations of the computing nodes (e.g., servers) and to exhaust the airflow exchanged heat to an external environment or a cooling system (e.g., air-to-liquid heat exchanger) to reduce the temperature of the airflow. For example, air supply system 135 generates an airflow of cool/cold air to circulate from aisle 103 through electronic racks 110A-110N to carry away exchanged heat.
The cool airflows enter the electronic racks through their frontends and the warm/hot airflows exit the electronic racks from their backends. The warm/hot air with exchanged heat is exhausted from room/building or cooled using a separate cooling system such as an air-to-liquid heat exchanger. Thus, the cooling system is a hybrid liquid-air cooling system, where a portion of the heat generated by a processor is removed by cooling liquid via the corresponding cold plate, while the remaining portion of the heat generated by the processor (or other electronics or processing devices) is removed by airflow cooling. Moreover, the liquid cooling may be multi-phase system wherein fluid flows in liquid or gaseous phase.
In addition, for at least some of the server chassis 203, an optional fan module (not shown) is associated with the server chassis. Each of the fan modules includes one or more cooling fans. The fan modules may be mounted on the backends of server chassis 203 or on the electronic rack to generate airflows flowing from frontend 204, traveling through the air space of the sever chassis 203, and existing at backend 205 of electronic rack 200.
In one embodiment, CDU 201 mainly includes heat exchanger 211, liquid pump 212, and a pump controller (not shown), and some other components such as a liquid reservoir, a power supply, monitoring sensors and so on. Heat exchanger 211 may be a liquid-to-liquid or multi-phase heat exchanger. Heat exchanger 211 includes a first loop with inlet and outlet ports having a first pair of liquid connectors coupled to external liquid supply/return lines 131-132 to form a primary loop. The connectors coupled to the external liquid supply/return lines 131-132 may be disposed or mounted on backend 205 of electronic rack 200. The liquid supply/return lines 131-132, also referred to as room liquid supply/return lines, may be coupled to cooling system 120 as described above.
In addition, heat exchanger 211 further includes a second loop with two ports having a second pair of liquid connectors coupled to liquid manifold 225 (also referred to as a rack manifold) to form a secondary loop, which may include a supply manifold (also referred to as a rack liquid supply line or rack supply manifold) to supply cooling liquid to server chassis 203 and a return manifold (also referred to as a rack liquid return line or rack return manifold) to return warmer liquid back to CDU 201. Note that CDUs 201 can be any kind of CDUs commercially available or customized ones. Thus, the details of CDUs 201 will not be described herein.
Each of server chassis 203 may include one or more IT components (e.g., central processing units or CPUs, general/graphic processing units (GPUs), memory, and/or storage devices). Each IT component may perform data processing tasks, where the IT component may include software installed in a storage device, loaded into the memory, and executed by one or more processors to perform the data processing tasks. Server chassis 203 may include a host server (referred to as a host node) coupled to one or more compute servers (also referred to as computing nodes, such as CPU server and GPU server). The host server (having one or more CPUs) typically interfaces with clients over a network (e.g., Internet) to receive a request for a particular service such as storage services (e.g., cloud-based storage services such as backup and/or restoration), executing an application to perform certain operations (e.g., image processing, deep data learning algorithms or modeling, etc., as a part of a software-as-a-service or SaaS platform). In response to the request, the host server distributes the tasks to one or more of the computing nodes or compute servers (having one or more GPUs) managed by the host server. The compute servers perform the actual tasks, which may generate heat during the operations.
Electronic rack 200 further includes optional RMU 202 configured to provide and manage power supplied to servers 203, and CDU 201. RMU 202 may be coupled to a power supply unit (not shown) to manage the power consumption of the power supply unit. The power supply unit may include the necessary circuitry (e.g., an alternating current (AC) to direct current (DC) or DC to DC power converter, battery, transformer, or regulator, etc.,) to provide power to the rest of the components of electronic rack 200.
In one embodiment, RMU 202 includes optimization module 221 and rack management controller (RMC) 222. RMC 222 may include a monitor to monitor operating status of various components within electronic rack 200, such as, for example, computing nodes 203, CDU 201, and the fan modules. Specifically, the monitor receives operating data from various sensors representing the operating environments of electronic rack 200. For example, the monitor may receive operating data representing temperatures of the processors, cooling liquid, and airflows, which may be captured and collected via various temperature sensors. The monitor may also receive data representing the fan power and pump power generated by the fan modules and liquid pump 212, which may be proportional to their respective speeds. These operating data are referred to as real-time operating data. Note that the monitor may be implemented as a separate module within RMU 202.
Based on the operating data, optimization module 221 performs an optimization using a predetermined optimization function or optimization model to derive a set of optimal fan speeds for the fan modules and an optimal pump speed for liquid pump 212, such that the total power consumption of liquid pump 212 and the fan modules reaches minimum, while the operating data associated with liquid pump 212 and cooling fans of the fan modules are within their respective designed specifications. Once the optimal pump speed and optimal fan speeds have been determined, RMC 222 configures liquid pump 212 and cooling fans of the fan modules based on the optimal pump speeds and fan speeds.
As an example, based on the optimal pump speed, RMC 222 communicates with a pump controller of CDU 201 to control the speed of liquid pump 212, which in turn controls a liquid flow rate of cooling liquid supplied to the liquid manifold 225 to be distributed to at least some of server chassis 203. Similarly, based on the optimal fan speeds, RMC 222 communicates with each of the fan modules to control the speed of each cooling fan of the fan modules, which in turn control the airflow rates of the fan modules. Note that each of fan modules may be individually controlled with its specific optimal fan speed, and different fan modules and/or different cooling fans within the same fan module may have different optimal fan speeds.
Note that the rack configuration as shown in
The rack fluid distribution system may be equipped with different types of connectors, positioned at different locations and/or at different spacing and orientations, including different integration locations such as parallel or staggered. This may increase the difficulty for implementing liquid cooled server on the rack, especially when accounting to multi-phase cooling arrangements. Therefore, disclosed embodiments introduce an adapter design, which may be used for adapting different server systems on different rack fluid distribution hardware designs. The various embodiments enable each configuration of blind mate connectors for different connection requirements of the computing systems.
A beneficial feature of disclosed embodiments is that since the connectors are movable to different locations, the rack side may not need to integrate with many connectors. Since the fluid connectors on the server side can be moved to provide different fluid connection design, a smaller number of connectors are required, thereby providing for cost reduction.
In addition, the different types of servers may require different fluid systems for thermal management. Disclosed embodiments of the connector adapter helps satisfy these different requirements. Similarly, the rack configurations may be different for different IT equipment and systems. The rack level distribution may be designed in different manners, and/or the rack level fluid ports may be in different availabilities, especially when servers are already populated in the rack thus occupying some ports. The individual servers may have different cooling modules designed and implemented for them. Therefore, the fluid flowing directions may be different and the difference in the fluid flowing direction, e.g., using thermosiphon, may generate different performances. Disclosed embodiments enable to easily change the locations of fluid inlet and outlet based on different cooling module designs and the rack liquid systems.
In the following, embodiments for fluid connection module, especially beneficial for multi-phase cooling systems or heterogeneous racks, are disclosed. The connection module may function as an interface for multiple blind mate fluid connectors, wherein internal channels or tracks enable adjusting the position of the connectors. The entire unit can be an integrated unit on a server chassis or a separate unit easily installed or disassembled from the server chassis. The connectors are interconnected in an embodiment for blind mating applications and the internal channels are also being slidable for adjusting the locations of the connectors. The flexible design ensures matching with different rack level design specifications. In an embodiment, the connection module may have two types of channels, one for the connectors and the other one for the internal connection (intra-connection) of the connectors. The variation in locations of the channels as well as the connectors can be used for assembling different cooling fluid operating requirements, either single phase or two phase fluid.
The ability to slide the fluid connectors within the channels enable further advantages in various use cases. For example, the rack ports may be occupied by other systems on an existing rack, so the solution enables more possibilities to utilize different rack ports which are available as well as adjusting other systems connections to enable new system installation. Also, the variation in the internal channel allows for integrating multiple sets of connectors and sub-loops from a server chassis.
Connectors 422 and 424 are slidably mounted onto the housing 405. In this embodiment, connectors 422 and 424 are paired, in that one serves as intake and one as return for one liquid cooling element. Each of connectors 422 and 424 may slide in channel or track 430 so as to change its position in the housing 405. In one example, the channel 430 is formed to have tight fit to the connectors, such that the connectors may slide by application of force, but once placed in one position they remain in that position. In this sense, tight fit means that the connectors do not slide by force of gravity, but requires an additional application of force to change elevation. In such embodiment, no mechanical locking mechanism is needed to secure the connectors in a given position, albeit a locking mechanism may be used in other embodiments if desired.
Also, in order to easily blind mate to corresponding connectors, e.g., of a serve chassis, the relative orientation of the paired connectors 422 and 424 is maintained by use of an interconnection 440. The interconnection 440 ensures that the paired connectors 422 and 424 are maintained in relative orientation that conforms to standard rack configuration. This feature ensures that blind mating can be easily accomplished.
In disclosed embodiments the interconnection of the paired connectors also assists the locking of the two connectors, preventing any unintentional motion, e.g., from gravity. In one embodiment, during the blind mating procedure, different directions of forces may be loaded on the connectors 424 and 422 when mating with the corresponding connectors on the rack level fluid distribution. The interconnection also functions as additional protection.
In one embodiment, the two channels maybe merged to one, while two dedicated channels may provide additional structure enhancement.
According to an optional feature, toothed tracks 626 are provided within the space 631, so that when the connector is rotated to an engaging position, illustrated by connector 622, the engagement arms 628 engage the toothed track and affix the connector in its position. Conversely, when the connector is rotated as illustrated by connector 624, the connector is free to slide within channel 630 to be positioned anywhere along the channel 630. As illustrated in
In the example of
The embodiment illustrated in
it should be appreciated that the embodiments illustrated in
Incidentally, the callout in
A heterogeneous rack would have equipment mounted therein having different U sizes. The embodiment of
In
In the foregoing specification, embodiments of the invention have been described with reference to specific exemplary embodiments thereof. It will be evident that various modifications may be made thereto without departing from the broader spirit and scope of the invention as set forth in the following claims. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.
Claims
1. A connector module for an electronic rack, comprising:
- a frame having mounts configured for attaching the frame onto the electronic rack;
- a plurality of channels formed in the frame to accept fluid connectors; and
- a plurality of fluid connectors slidably mounted in the channels; and,
- wherein each of the fluid connectors further comprises a locking mechanism affixing the fluid connector in a fixed location within the channel.
2. The module of claim 1, further comprising an interconnector attached to at least two fluid connectors thereby maintaining fixed spatial orientation between the two fluid connectors.
3. (canceled)
4. The module of claim 1, wherein the locking mechanism comprises tight fit of the fluid connector within a channel imparting motion resistance of the fluid connector within a channel preventing free slide.
5. The module of claim 1, wherein the locking mechanism comprises arms extending from the fluid connector and engaging teeth within the channel.
6. The module of claim 1, wherein the frame further comprises a plurality of locators positioned at preset locations and configured for locking the fluid connectors at the preset locations.
7. The module of claim 1 wherein each of the fluid connectors comprises an interconnect kit configured for interconnecting each of the fluid connector to another fluid connector to maintain fixed spatial orientation.
8. An electronic rack of a data center, comprising:
- a rack manifold having a rack liquid supply line to receive first cooling liquid from a cooling liquid source and a rack liquid return line to return first warmer liquid back to the cooling liquid source;
- a plurality of server chassis arranged in a stack, each server chassis including one or more fluid cooling devices associated with one or more information technology (IT) components; and
- a plurality of connector modules attached to the electronic rack, each providing fluid connections between the fluid cooling devices and the cooling liquid source; wherein each of the connector module comprises: a frame having mounts configured for attaching the frame onto the electronic rack, a plurality of channels formed in the frame to accept the fluid connectors, and a plurality of fluid connectors slidably mounted in the channels; and, a plurality of interconnectors, each attached to at least two fluid connectors thereby maintaining fixed spatial orientation between the two fluid connectors.
9. The electronic rack of claim 8, wherein the fluid cooling devices are selected from cooling plates and condensers.
10. The electronic rack of claim 8, wherein at least one of the fluid cooling device comprises a riser gas line and a liquid return line, and wherein the riser gas line is connected to a connector mounted on a first connector module and the liquid return line is connected to a connector mounted on a second connector module, wherein the first connector module is mounted vertically higher than the second connector module.
11. The electronic rack of claim 8, wherein at least one of the fluid cooling devices comprises a hot liquid line and a cold liquid line, and wherein the hot liquid line is connected to a first connector mounted on a first channel and the cold liquid line is connected to a second connector mounted on a second channel, wherein the first connector is mounted at same vertical height as the second connector.
12. The electronic rack of claim 8, wherein the plurality of fluid connectors comprises a plurality of paired inlet and return connectors, wherein a subset of the paired inlet and return connectors are mounted at same vertical height and remainder of the paired inlet connectors are mounted with the inlet at a higher vertical height than the return connector.
13. (canceled)
14. The electronic rack of claim 8, wherein each of the fluid connectors further comprises a locking mechanism affixing the fluid connector in a fixed location within the channel.
15. The electronic rack of claim 14, wherein the locking mechanism comprises one of: a tight fit of the fluid connector within a channel imparting motion resistance of the fluid connector within a channel preventing free slide; or arms extending from the fluid connector and engaging teeth within the channel.
16. The electronic rack of claim 8, wherein the frame further comprises a plurality of locators positioned at preset locations and configured for locking the fluid connectors at the preset locations.
17. A method for providing cooling fluid to an electronic rack of a plurality of server chassis, comprising:
- mounting a frame with a plurality of connector modules to the electronic rack using mounts, each connector module provided with at least two channels;
- attaching a plurality of fluid connectors to the connector modules by attaching at least one fluid connector in each of the channels; and
- for each of the server chassis, verifying spatial orientation of inlet connector and outlet connector and sliding two of the plurality of connectors to match the spatial orientation of the inlet connector and outlet connector, and securing the two connectors in the spatial orientation using a locking mechanism, and then mounting the server chassis in the electronic rack by engaging the two connectors with the inlet connector and outlet connector.
18. The method of claim 17, wherein securing the two connectors comprises positioning the two connectors in selected predefined dedicated locations identified by pre-arranged locators applied to the channels.
19. The method of claim 17, further comprising mounting a multi-device chassis having multiple cooling devices by engaging the multi-device chassis with multiple pairs of fluid connectors.
20. The method of claim 19, wherein for each pair of fluid connectors, affixing a riser connector at higher elevation than a return connector.
Type: Application
Filed: Feb 23, 2021
Publication Date: Aug 25, 2022
Inventor: Tianyi GAO (San Jose, CA)
Application Number: 17/182,564