Data mover selection system

- Dell Products L.P.

A data mover selection system includes a fabric manager system coupled to computing devices that are coupled to a memory system. The fabric manager system receives respective local data mover selection information from the computing devices that identifies data mover device(s) accessible to those computing device, and generates global data mover selection information that includes each data mover device accessible to the computing devices. When the fabric manager system receives a first data transfer request to transfer data between first and second memory locations in the memory system, it uses the global data mover selection information to identify a data mover device having the highest priority for performing data transfers between the first and second memory locations in the memory system, and transmits a first data transfer instruction to that data mover device to cause that data mover device to perform the first data transfer operation.

Skip to: Description  ·  Claims  ·  References Cited  · Patent History  ·  Patent History
Description
BACKGROUND

The present disclosure relates generally to information handling systems, and more particularly to selecting a data mover device for performing data transfers between memory locations accessible to an information handling system.

As the value and use of information continues to increase, individuals and businesses seek additional ways to process and store information. One option available to users is information handling systems. An information handling system generally processes, compiles, stores, and/or communicates information or data for business, personal, or other purposes thereby allowing users to take advantage of the value of the information. Because technology and information handling needs and requirements vary between different users or applications, information handling systems may also vary regarding what information is handled, how the information is handled, how much information is processed, stored, or communicated, and how quickly and efficiently the information may be processed, stored, or communicated. The variations in information handling systems allow for information handling systems to be general or configured for a specific user or specific use such as financial transaction processing, airline reservations, enterprise data storage, or global communications. In addition, information handling systems may include a variety of hardware and software components that may be configured to process, store, and communicate information and may include one or more computer systems, data storage systems, and networking systems.

Information handling systems such as, for example, server computing devices and/or other computing devices known in the art, sometimes utilize data mover devices in order to perform data transfers between memory locations that are accessible to the computing device(s). As will be appreciated by one of skill in the art in possession of the present disclosure, data movers may be made accessible to a computing device to offload data transfer operations from the processing system in the computing device, with current data mover devices implemented as part of a processor package in the processing system, a built-in controller in the computing device, an add-in card in the computing device, and/or in a variety of other manners known in the art. However, the use of data mover devices in some computing device configurations can raise some issues.

For example, the processing system in some server computing devices may include multiple processor subsystems (e.g., multiple Central Processing Units (CPUs)) coupled together by processor interconnect(s) (e.g., Ultra Path Interconnect(s) (UPI(s)) provided in processing systems available from INTEL® Corporation of Santa Clara, Calif., United States), the memory system may provide a respective memory subsystem for each processor subsystem (with each respective memory subsystem often called the “local memory” for its associated processor subsystem), and the processing system/memory system may be configured in a Non-Uniform Memory Access (NUMA) design in which each processing subsystem/memory subsystem combination provides a respective “NUMA node”, with memory access times for processing subsystems depending on the memory subsystem location relative to the processor subsystem performing the memory access, and processor subsystems capable of accessing their local memory subsystem faster than non-local memory subsystems (i.e., the memory subsystem that is local to the other processor subsystem(s)). Furthermore, server computing devices may be connected together via a network and may operate to make their memory system available by other server computing device, while dedicated shared memory resources coupled to the network may be assigned to (and detached from) server computing devices over time, and the data mover devices accessible to those server computing devices operating to perform data transfers between any accessible memory resources.

In such configurations, a data mover device may be shared by one or more of the NUMA nodes and/or by different server computing devices, a respective data mover device may be provided with each NUMA node and/or each server computing device, and/or multiple data mover devices may be provided with one or more NUMA nodes and/or one or more computing devices. In multi-data-mover-device systems, the selection of a data mover device to perform any particular data transfer is conventionally performed via “round-robin” techniques that attempt to distribute data transfer operations evenly across the available data mover devices that are available to the server computing device requesting the data transfer. However, similarly to the processing subsystems discussed above, memory access times for some data mover devices will depend on the memory subsystem location relative to the data mover device performing the data transfer, and thus the round-robin data mover device selection techniques discussed above can result in inefficient data transfers between memory locations accessible to the server computing device (e.g., a data mover device selected to perform the data transfer may provide for slower data transfers relative to at least one of the other data mover devices accessible to the server computing device).

Accordingly, it would be desirable to provide data mover selection system that addresses the issues discussed above.

SUMMARY

According to one embodiment, an Information Handling System (IHS) a processing system; and a memory system that is coupled to the processing system and that includes instructions that, when executed by the processing system, cause the processing system to provide an fabric manager engine that is configured to: receive, from each of a plurality of computing devices, respective local data mover selection information that identifies at least one data mover device accessible to that computing device; generate, using the respective local data mover selection information received from each of the plurality of computing devices, global data mover selection information that includes each data mover device accessible to the plurality of computing devices; receive, from a first computing device included in the plurality of computing devices, a first data transfer request that provides for the transfer of data from a first memory location in a memory system to a second memory location in the memory system; identify, using the global data mover selection information, a first data mover device for performing the first data transfer operation based on the first data mover device having a higher priority relative to other data mover devices included in the global data mover selection information for performing data transfers from the first memory location in the memory system to the second memory location in the memory system; and transmit, in response to identifying the first data mover device for performing the first data transfer operation, a first data transfer instruction to the first data mover device that is configured to cause the first data mover device to perform the first data transfer operation to transfer data from the first memory location in the memory system to the second memory location in the memory system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view illustrating an embodiment of an Information Handling System (IHS).

FIG. 2 is a schematic view illustrating an embodiment of a computing device that may utilize the data mover selection system of the present disclosure.

FIG. 3 is a flow chart illustrating an embodiment of a method for selecting a data mover device.

FIG. 4A is a schematic view illustrating an embodiment of the computing device of FIG. 2 operating during the method of FIG. 3.

FIG. 4B is a schematic view illustrating an embodiment of the computing device of FIG. 2 operating during the method of FIG. 3.

FIG. 5 is a schematic view illustrating an embodiment of a BIOS database that may be provided in the computing device of FIG. 2 operating during the method of FIG. 3.

FIG. 6 is a schematic view illustrating an embodiment of the computing device of FIG. 2 operating during the method of FIG. 3.

FIG. 7 is a schematic view illustrating an embodiment of the computing device of FIG. 2 operating during the method of FIG. 3.

FIG. 8 is a schematic view illustrating an embodiment of the computing device of FIG. 2 operating during the method of FIG. 3.

FIG. 9 is a schematic view illustrating an embodiment of the computing device of FIG. 2 operating during the method of FIG. 3.

FIG. 10 is a schematic view illustrating an embodiment of the computing device of FIG. 2 operating during the method of FIG. 3.

FIG. 11 is a schematic view illustrating an embodiment of the computing device of FIG. 2 operating during the method of FIG. 3.

FIG. 12 is a schematic view illustrating an embodiment of the computing device of FIG. 2 operating during the method of FIG. 3.

FIG. 13 is a schematic view illustrating an embodiment of a networked system that includes a plurality of the computing devices of FIG. 2 coupled to a fabric manager system, and that may provide the data mover selection system of the present disclosure.

FIG. 14 is a schematic view illustrating an embodiment of a fabric manager system that may be provided with the networked system of FIG. 13.

FIG. 15 is a flow chart illustrating an embodiment of a method for selecting a data mover device.

FIG. 16 is a schematic view illustrating an embodiment of the networked system of FIG. 13 operating during the method of FIG. 15.

FIG. 17 is a schematic view illustrating an embodiment of a fabric manager database that may be provided in the fabric manager system of FIG. 14 during the operation of the networked system of FIG. 13 during the method of FIG. 15.

FIG. 18 is a schematic view illustrating an embodiment of the networked system of FIG. 13 operating during the method of FIG. 15.

FIG. 19A is a schematic view illustrating an embodiment of the networked system of FIG. 13 operating during the method of FIG. 15.

FIG. 19B is a schematic view illustrating an embodiment of the networked system of FIG. 13 operating during the method of FIG. 15.

 DETAILED DESCRIPTION

For purposes of this disclosure, an information handling system may include any instrumentality or aggregate of instrumentalities operable to compute, calculate, determine, classify, process, transmit, receive, retrieve, originate, switch, store, display, communicate, manifest, detect, record, reproduce, handle, or utilize any form of information, intelligence, or data for business, scientific, control, or other purposes. For example, an information handling system may be a personal computer (e.g., desktop or laptop), tablet computer, mobile device (e.g., personal digital assistant (PDA) or smart phone), server (e.g., blade server or rack server), a network storage device, or any other suitable device and may vary in size, shape, performance, functionality, and price. The information handling system may include random access memory (RAM), one or more processing resources such as a central processing unit (CPU) or hardware or software control logic, ROM, and/or other types of nonvolatile memory. Additional components of the information handling system may include one or more disk drives, one or more network ports for communicating with external devices as well as various input and output (I/O) devices, such as a keyboard, a mouse, touchscreen and/or a video display. The information handling system may also include one or more buses operable to transmit communications between the various hardware components.

In one embodiment, IHS 100, FIG. 1, includes a processor 102, which is connected to a bus 104. Bus 104 serves as a connection between processor 102 and other components of IHS 100. An input device 106 is coupled to processor 102 to provide input to processor 102. Examples of input devices may include keyboards, touchscreens, pointing devices such as mouses, trackballs, and trackpads, and/or a variety of other input devices known in the art. Programs and data are stored on a mass storage device 108, which is coupled to processor 102. Examples of mass storage devices may include hard discs, optical disks, magneto-optical discs, solid-state storage devices, and/or a variety of other mass storage devices known in the art. IHS 100 further includes a display 110, which is coupled to processor 102 by a video controller 112. A system memory 114 is coupled to processor 102 to provide the processor with fast storage to facilitate execution of computer programs by processor 102. Examples of system memory may include random access memory (RAM) devices such as dynamic RAM (DRAM), synchronous DRAM (SDRAM), solid state memory devices, and/or a variety of other memory devices known in the art. In an embodiment, a chassis 116 houses some or all of the components of IHS 100. It should be understood that other buses and intermediate circuits can be deployed between the components described above and processor 102 to facilitate interconnection between the components and the processor 102.

Referring now to FIG. 2, an embodiment of a computing device 200 is illustrated that may utilize the data mover selection system of the present disclosure. In an embodiment, the computing device 200 may be provided by the IHS 100 discussed above with reference to FIG. 1 and/or may include some or all of the components of the IHS 100, and in specific examples may be provided by a server computing device. However, while illustrated and discussed as being provided by a server computing device, one of skill in the art in possession of the present disclosure will recognize that the functionality of the computing device 200 discussed below may be provided by other devices that are configured to operate similarly as the computing device 200 discussed below. In the illustrated embodiment, the computing device 200 includes a chassis 302 that houses the components of the computing device 200, only some of which are illustrated below. For example, as discussed below, the chassis 302 may house a processing system (e.g., which may include the processor 102 discussed above with reference to FIG. 1) and a memory system (e.g., which may include the memory 114 discussed above with reference to FIG. 1) that is coupled to the processing system and that includes instructions that, when executed by the processing system, cause the processing system to provide an operating system engine and/or application engine(s) that are configured to perform the functionality of the operating systems, applications, and/or computing devices discussed below.

In the example illustrated in FIG. 2, the processing system and memory system housed in the chassis 202 are provided in a Non-Uniform Memory Access (NUMA) configuration including a pair of nodes 204 and 206 (e.g., “NUMA nodes”). However, while only two nodes 204 and 206 are illustrated and described in the examples below, one of skill in the art in possession of the present disclosure will recognize that NUMA configurations may include additional nodes that are similar to the nodes 204 and 206 discussed herein. In the illustrated embodiment, the node 204 includes a processing subsystem 204a that is part of the processing system provided in the chassis 202 and that may be provided by a Central Processing Unit (CPU) or other processing subsystems known in the art. The node 204 also includes a memory subsystem 204b that is part of the memory system provided in the chassis 202, that is coupled to the processing subsystem 204a, and that may be provided by Dual Inline Memory Modules (DIMMs), memory controllers, and/or other memory components known in the art. The node 204 also includes a data mover device 204c that is coupled to the processing subsystem 204a and the memory subsystem 204b, and that is illustrated as being included as part of a processing subsystem package (e.g., a CPU package that provides the processing subsystem 204a/CPU) while being a separate component from the processor core(s) (i.e., in order to allow the data mover device 204c to offload data transfer operations from those processor core(s)).

Similarly, the node 206 includes a processing subsystem 206a that is part of the processing system provided in the chassis 202 and that may be provided by a Central Processing Unit (CPU) or other processing subsystems known in the art. As illustrated, the processing subsystem 204a in the node 204 and the processing subsystem 206a in the node 206 may be coupled together by a processing subsystem interconnect 207 (e.g., the UPI discussed above). The node 206 also includes a memory subsystem 206b that is part of the memory system provided in the chassis 202, that is coupled to the processing subsystem 206a, and that may be provided by Dual Inline Memory Modules (DIMMs) and/or other memory devices known in the art. The node 206 also includes a data mover device 206c that is coupled to the processing subsystem 206a and the memory subsystem 206b, and that is illustrated as being included as part of a processing subsystem package (e.g., a CPU package that provides the processing subsystem 204a/CPU) while being a separate component from the processor core(s) (i.e., in order to allow the data mover device 206c to offload data transfer operations from those processor core(s)).

However, while respective data mover devices 204c and 206c are illustrated and described below as being provided with each node, one of skill in the art in possession of the present disclosure will recognize that other data mover device configurations will fall within the scope of the present disclosure as well. For example, either of the nodes 204 and 206 may include multiple data mover devices, or may not include a data mover devices. In specific examples, the data mover devices of the present disclosure may be provided by a Pass-Through Direct Memory Access (PTDMA) engine provided by ADVANCED MICRO DEVICES® of Santa Clara, Calif., United States; a Data Streaming Accelerator (DSA) or Crystal Beach Direct Memory Access (CBDMA) engine available from INTEL® Corporation of Santa Clara, Calif., United States; and/or any other data mover device that one of skill in the art in possession of the present disclosure would recognize that enabling the direct memory-to-memory data transfers discussed herein. Furthermore, while illustrated as being provided as part of a processing subsystem package in the node, one of skill in the art in possession of the present disclosure will recognize that data mover devices may be provided as part of a built-in controller, as part of an add-in card that is connected to a motherboard in the computing device that is also coupled to the nodes 204 and 206, and/or in a variety of other data mover device configurations that will fall within the scope of the present disclosure as well.

In specific examples, the data mover devices of the present disclosure may be integrated into a Central Processing Unit (CPU) System on a Chip (SoC) such as with the AMD® PTDMA engine or INTEL® CBDMA engine discussed above, implemented as discrete Peripheral Component Interconnect express (PCIe) add-in cards that are localized to specific CPUs, and/or in any other manner that would be apparent to one of skill in the art in possession of the present disclosure. As will be appreciated by one of skill in the art in possession of the present disclosure, CPU SoC systems may provide many physical functions, with each associated with a different “distance” to memory channels that provide access to a memory subsystem. For example, the AMD® PTDMA engine discussed above provides each PTDMA engine in the same quadrant an equal distance to two available memory channels in that quadrant, but a longer distance to six available memory channels in the other quadrants. Furthermore, one of skill in the art in possession of the present disclosure will recognize that for memory-to-memory data transfers via a data mover device, memory read operations are associated with longer latencies than memory write operations, thus providing relatively lower latencies when reading from local memory subsystems and writing to remote memory subsystems

As such, the processor subsystem 204a/memory subsystem 204b may provide a first NUMA node (e.g., “NUMA node 0”) that includes the data mover device 204c, and the processor subsystem 206a/memory subsystem 206b may provide a second NUMA node (e.g., “NUMA node 1”) that includes the data mover device 206c and that is coupled to the first NUMA node via the processing subsystem interconnect/UPI 207. However, while particular processing subsystem/memory subsystem nodes are described in a two-processing subsystem/memory subsystem node configuration, one of skill in the art in possession of the present disclosure will recognize that other processing subsystem/memory subsystem node systems will fall within the scope of the present disclosure as well. Furthermore, one of skill in the art in possession of the present disclosure will recognize that the nodes 204 and 206 illustrated in FIG. 2 provide an example of a NUMA configuration in which local memory subsystems are provided for each processing subsystem in a multi-processor system, and memory subsystem access times depend on the relative location of the memory subsystem and the processing subsystem performing the memory access operations, with processing subsystems able to access their local memory subsystems faster than memory subsystems that are not local (i.e., memory subsystems that are local to another processing subsystem.) However, while a NUMA memory design is illustrated and discussed below, other processing system/memory system configurations may benefit from the teachings of the present disclosure and thus are envisioned as falling within its scope as well.

The chassis 202 also houses a Basic Input/Output System (BIOS) 208 that one of skill in the art in possession of the present disclosure will recognize may be provided by firmware, and used to perform hardware initialization during booting operations (e.g., Power-On StartUp (POST)) for the computing device 200, as well as provide runtime services for an operating systems and/or other applications/programs provided by the computing device 200. As such, the BIOS 210 may be provided by a BIOS processing system (not illustrated, but which may include the processor 102 discussed above with reference to FIG. 1) and a BIOS memory system (not illustrated, but which may be provided by the memory 114 discussed above with reference to FIG. 1) that includes instruction that, when executed by the BIOS processing system, cause the BIOS processing system to provide a BIOS engine that is configured to performs the operations of the BIOS 210 discussed below. Furthermore, while discussed as a BIOS, one of skill in the art in possession of the present disclosure will recognize that the BIOS 210 may be provided according to the Unified Extensible Firmware Interface (UEFI) specification, which defines a software interface between operating systems and platform firmware and which was provided to replace legacy BIOS firmware, while remaining within the scope of the present disclosure as well.

The chassis 202 may also house a storage system (not illustrated, but which may include the storage 108 discussed above with reference to FIG. 1) that is coupled to the BIOS 208 (e.g., via a coupling between the storage system and the BIOS processing system) and that includes a BIOS database 210 that is configured to store any of the information utilized by the BIOS 208 discussed below. However, while a specific computing device 200 has been illustrated, one of skill in the art in possession of the present disclosure will recognize that computing devices (or other devices operating according to the teachings of the present disclosure in a manner similar to that described below for the computing device 200) may include a variety of components and/or component configurations for providing conventional computing device functionality, as well as the functionality discussed below, while remaining within the scope of the present disclosure as well.

Referring now to FIG. 3, an embodiment of a method 300 for selecting data mover devices is illustrated. As discussed below, the systems and methods of the present disclosure provide for the selection of one of a plurality of data mover devices for which to perform data transfer operations between memory locations based on an “affinity” of that data mover device to at least one of those memory locations that provides that data mover device a higher priority for performing the data transfer that the other data mover devices. For example, the data mover selection system of the present disclosure may include a first data mover device and a second data mover device that are both coupled to a memory system, and an operating system that is coupled to the first data mover device and the second data mover device. The operating system determines that a first data transfer operation provides for the transfer of data from a first memory location in the memory system to a second memory location in the memory system, identifies the first data mover device for performing the first data transfer operation based on the first data mover device having a higher priority relative to the second data mover device for performing data transfers from the first memory location in the memory system to the second memory location in the memory system and, in response, transmits a first data transfer instruction to the first data mover device that is configured to cause the first data mover device to perform the first data transfer operation to transfer data from the first memory location in the memory system to the second memory location in the memory system. As such, data transfers may be performed by data mover devices that provide more efficient data transfers (relative to conventional “round robin” data mover device selections) based on their affinity to one or more of the memory locations involved in those data transfers.

The method 300 begins at block 302 where a BIOS generates a data mover selection table during initialization operations. In an embodiment, at block 302, a BIOS engine in the BIOS 208 may operate to generate a data mover selection table during, for example, initialization operations for the computing device 200. For example, with reference to FIG. 4A, the BIOS 208 may operate during boot operations for the computing device 200 to perform discovery operations 400 that provide for the discovery or other identification of the nodes 204 and 206, the processing subsystems 204a and 206a, the memory subsystems 204b and 206b, and/or the data mover devices 204c and 206c. In an embodiment, following the discovery operations, the BIOS engine in the BIOS 208 may operate to utilize the discovered information to generate a data mover selection table that may be provided by, for example, an Advanced Configuration and Power Interface (ACPI) construct. As illustrated in FIG. 4B, the BIOS 208 may then perform data mover selection table storage operations 402 to store the data mover selection table in the BIOS database 210.

With reference to FIG. 5, an embodiment of a data mover selection table 500 that may be generated and stored in BIOS database 210 at block 302. As will be appreciated by one of skill in the art in possession of the present disclosure, the BIOS engine in the BIOS 208 may generate the data mover selection table 500 that maps memory boundaries in the memory subsystems 204b and 206b to the data mover devices 204c and 206c based on an “affinity” or “distance” of the data mover devices to those memory boundaries, with data mover devices having higher “affinities”/smaller “distances” to particular memory subsystems prioritized for performing data transfers associated with those memory subsystems. As such, the specific example of the data mover selection table 500 corresponding to the computing device 200 illustrated in FIG. 2 maps memory locations to data mover devices by providing “source” memory location/address ranges in a first column in the data mover selection table 500, providing “destination” memory location/address ranges in a first row in the data mover selection table 500, and identifying the priority/priorities of the data mover devices 204c and/or 206c for data transfers between any particular combination of a source memory location/address and a destination memory location/address.

As such, with reference to the specific example provided in FIG. 5, for data transfers between a source memory location/address and a destination memory location/address that are both included in the memory location/address range of “0-0xFFF” that is provided by the memory subsystem 204b in this example, the data mover device 204c is identified as having priority for performing those data transfers due to the data mover device 204c being located in the node 204 with the memory subsystem 204b and, thus, having a higher “affinity”/smaller “distance” relative to that memory subsystem 204b that provides for more efficient data transfers (e.g., relative to data transfers performed by the data mover device 206c that is located in the node 206). Similarly, for data transfers between a source memory location/address and a destination memory location/address that are both included in the memory location/address range of “0x1000-0x1FFF” that is provided by the memory subsystem 204b in this example, the data mover device 204c is identified as having priority for performing those data transfers due to the data mover device 204c being located in the node 204 with the memory subsystem 204b and, thus, having a higher “affinity”/smaller “distance” relative to that memory subsystem 204b that provides for more efficient data transfers (e.g., relative to data transfers performed by the data mover device 206c that is located in the node 206).

Similarly, for data transfers between a source memory location/address that is included in the memory location/address range of “0-0xFFF” that is provided by the memory subsystem 204b in this example and a destination memory location/address that is included in the memory location/address range of “0x1000-0x1FFF” that is provided by the memory subsystem 204b in this example, the data mover device 204c is identified as having priority for performing those data transfers due to the data mover device 204c being located in the node 204 with the memory subsystem 204b and, thus, having a higher “affinity”/smaller “distance” relative to that memory subsystem 204b that provides for more efficient data transfers (e.g., relative to data transfers performed by the data mover device 206c that is located in the node 206). Similarly, for data transfers between a source memory location/address that is included in the memory location/address range of “0x1000-0x1FFF” that is provided by the memory subsystem 204b in this example and a destination memory location/address that is included in the memory location/address range of “0-0xFFF” that is provided by the memory subsystem 204b in this example, the data mover device 204c is identified as having priority for performing those data transfers due to the data mover device 204c being located in the node 204 with the memory subsystem 204b and, thus, having a higher “affinity”/smaller “distance” relative to that memory subsystem 204b that provides for more efficient data transfers (e.g., relative to data transfers performed by the data mover device 206c that is located in the node 206).

Similarly, for data transfers between a source memory location/address and a destination memory location/address that are both included in the memory location/address range of “0x2000-0x2FFF” that is provided by the memory subsystem 204b in this example, the data mover device 206c is identified as having priority for performing those data transfers due to the data mover device 206c being located in the node 206 with the memory subsystem 206b and, thus, having a higher “affinity”/smaller “distance” relative to that memory subsystem 206b that provides for more efficient data transfers (e.g., relative to data transfers performed by the data mover device 204c that is located in the node 204). Similarly, for data transfers between a source memory location/address and a destination memory location/address that are both included in the memory location/address range of “0x3000-0x3FFF” that is provided by the memory subsystem 206b in this example, the data mover device 206c is identified as having priority for performing those data transfers due to the data mover device 206c being located in the node 206 with the memory subsystem 206b and, thus, having a higher “affinity”/smaller “distance” relative to that memory subsystem 206b that provides for more efficient data transfers (e.g., relative to data transfers performed by the data mover device 204c that is located in the node 204).

Similarly, for data transfers between a source memory location/address that is included in the memory location/address range of “0x2000-0x2FFF” that is provided by the memory subsystem 206b in this example and a destination memory location/address that is included in the memory location/address range of “0x3000-0x3FFF” that is provided by the memory subsystem 206b in this example, the data mover device 206c is identified as having priority for performing those data transfers due to the data mover device 206c being located in the node 206 with the memory subsystem 206b and, thus, having a higher “affinity”/smaller “distance” relative to that memory subsystem 206b that provides for more efficient data transfers (e.g., relative to data transfers performed by the data mover device 204c that is located in the node 204). Similarly, for data transfers between a source memory location/address that is included in the memory location/address range of “0x3000-0x3FFF” that is provided by the memory subsystem 206b in this example and a destination memory location/address that is included in the memory location/address range of “0x2000-0x2FFF” that is provided by the memory subsystem 206b in this example, the data mover device 206c is identified as having priority for performing those data transfers due to the data mover device 206c being located in the node 206 with the memory subsystem 206b and, thus, having a higher “affinity”/smaller “distance” relative to that memory subsystem 206b that provides for more efficient data transfers (e.g., relative to data transfers performed by the data mover device 204c that is located in the node 204).

Similarly, for data transfers between a source memory location/address that is included in the memory location/address range of “0-0xFFF” that is provided by the memory subsystem 204b in this example and a destination memory location/address that is included in the memory location/address range of “0x2000-0x2FFF” that is provided by the memory subsystem 206b in this example, the data mover device 204c is identified as having first priority for performing those data transfers due to the data mover device 204c being located in the node 204 with the memory subsystem 204b that provides the source of the data for the data transfer and, thus, having a higher “affinity”/smaller “distance” relative to that memory subsystem 204b that provides for more efficient source data transfers (e.g., relative to source data transfers performed by the data mover device 206c that is located in the node 206). Furthermore, the data mover device 206c is identified as having second priority for performing those data transfers due to the data mover device 206c being located in the node 206 with the memory subsystem 206b that provides the destination of the data for the data transfer. As such, this embodiment of the present disclosure prioritizes data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the source of the data for the data transfer over data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the destination of the data for the data transfer.

Similarly, for data transfers between a source memory location/address that is included in the memory location/address range of “0-0xFFF” that is provided by the memory subsystem 204b in this example and a destination memory location/address that is included in the memory location/address range of “0x3000-0x3FFF” that is provided by the memory subsystem 206b in this example, the data mover device 204c is identified as having first priority for performing those data transfers due to the data mover device 204c being located in the node 204 with the memory subsystem 204b that provides the source of the data for the data transfer and, thus, having a higher “affinity”/smaller “distance” relative to that memory subsystem 204b that provides for more efficient source data transfers (e.g., relative to source data transfers performed by the data mover device 206c that is located in the node 206). Furthermore, the data mover device 206c is identified as having second priority for performing those data transfers due to the data mover device 206c being located in the node 206 with the memory subsystem 206b that provides the destination of the data for the data transfer. As such, this embodiment of the present disclosure prioritizes data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the source of the data for the data transfer over data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the destination of the data for the data transfer.

Similarly, for data transfers between a source memory location/address that is included in the memory location/address range of “0x1000-0x1FFF” that is provided by the memory subsystem 204b in this example and a destination memory location/address that is included in the memory location/address range of “0x2000-0x2FFF” that is provided by the memory subsystem 206b in this example, the data mover device 204c is identified as having first priority for performing those data transfers due to the data mover device 204c being located in the node 204 with the memory subsystem 204b that provides the source of the data for the data transfer and, thus, having a higher “affinity”/smaller “distance” relative to that memory subsystem 204b that provides for more efficient source data transfers (e.g., relative to source data transfers performed by the data mover device 206c that is located in the node 206). Furthermore, the data mover device 206c is identified as having second priority for performing those data transfers due to the data mover device 206c being located in the node 206 with the memory subsystem 206b that provides the destination of the data for the data transfer. As such, this embodiment of the present disclosure prioritizes data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the source of the data for the data transfer over data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the destination of the data for the data transfer.

Similarly, for data transfers between a source memory location/address that is included in the memory location/address range of “0x1000-0xF1FF” that is provided by the memory subsystem 204b in this example and a destination memory location/address that is included in the memory location/address range of “0x3000-0x3FFF” that is provided by the memory subsystem 206b in this example, the data mover device 204c is identified as having first priority for performing those data transfers due to the data mover device 204c being located in the node 204 with the memory subsystem 204b that provides the source of the data for the data transfer and, thus, having a higher “affinity”/smaller “distance” relative to that memory subsystem 204b that provides for more efficient source data transfers (e.g., relative to source data transfers performed by the data mover device 206c that is located in the node 206). Furthermore, the data mover device 206c is identified as having second priority for performing those data transfers due to the data mover device 206c being located in the node 206 with the memory subsystem 206b that provides the destination of the data for the data transfer. As such, this embodiment of the present disclosure prioritizes data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the source of the data for the data transfer over data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the destination of the data for the data transfer.

Similarly, for data transfers between a source memory location/address that is included in the memory location/address range of “0x2000-0x2FFF” that is provided by the memory subsystem 206b in this example and a destination memory location/address that is included in the memory location/address range of “0-0xFFF” that is provided by the memory subsystem 204b in this example, the data mover device 206c is identified as having first priority for performing those data transfers due to the data mover device 206c being located in the node 206 with the memory subsystem 206b that provides the source of the data for the data transfer and, thus, having a higher “affinity”/smaller “distance” relative to that memory subsystem 206b that provides for more efficient source data transfers (e.g., relative to source data transfers performed by the data mover device 204c that is located in the node 204). Furthermore, the data mover device 204c is identified as having second priority for performing those data transfers due to the data mover device 204c being located in the node 204 with the memory subsystem 204b that provides the destination of the data for the data transfer. As such, this embodiment of the present disclosure prioritizes data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the source of the data for the data transfer over data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the destination of the data for the data transfer.

Similarly, for data transfers between a source memory location/address that is included in the memory location/address range of “0x2000-0x2FFF” that is provided by the memory subsystem 206b in this example and a destination memory location/address that is included in the memory location/address range of “0x1000-0x1FFF” that is provided by the memory subsystem 204b in this example, the data mover device 206c is identified as having first priority for performing those data transfers due to the data mover device 206c being located in the node 206 with the memory subsystem 206b that provides the source of the data for the data transfer and, thus, having a higher “affinity”/smaller “distance” relative to that memory subsystem 206b that provides for more efficient source data transfers (e.g., relative to source data transfers performed by the data mover device 204c that is located in the node 204). Furthermore, the data mover device 204c is identified as having second priority for performing those data transfers due to the data mover device 204c being located in the node 204 with the memory subsystem 204b that provides the destination of the data for the data transfer. As such, this embodiment of the present disclosure prioritizes data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the source of the data for the data transfer over data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the destination of the data for the data transfer.

Similarly, for data transfers between a source memory location/address that is included in the memory location/address range of “0x3000-0x3FFF” that is provided by the memory subsystem 206b in this example and a destination memory location/address that is included in the memory location/address range of “0-0xFFF” that is provided by the memory subsystem 204b in this example, the data mover device 206c is identified as having first priority for performing those data transfers due to the data mover device 206c being located in the node 206 with the memory subsystem 206b that provides the source of the data for the data transfer and, thus, having a higher “affinity”/smaller “distance” relative to that memory subsystem 206b that provides for more efficient source data transfers (e.g., relative to source data transfers performed by the data mover device 204c that is located in the node 204). Furthermore, the data mover device 204c is identified as having second priority for performing those data transfers due to the data mover device 204c being located in the node 204 with the memory subsystem 204b that provides the destination of the data for the data transfer. As such, this embodiment of the present disclosure prioritizes data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the source of the data for the data transfer over data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the destination of the data for the data transfer.

Similarly, for data transfers between a source memory location/address that is included in the memory location/address range of “0x3000-0x3FFF” that is provided by the memory subsystem 206b in this example and a destination memory location/address that is included in the memory location/address range of “0x1000-0x1FFF” that is provided by the memory subsystem 204b in this example, the data mover device 206c is identified as having first priority for performing those data transfers due to the data mover device 206c being located in the node 206 with the memory subsystem 206b that provides the source of the data for the data transfer and, thus, having a higher “affinity”/smaller “distance” relative to that memory subsystem 206b that provides for more efficient source data transfers (e.g., relative to source data transfers performed by the data mover device 204c that is located in the node 204). Furthermore, the data mover device 204c is identified as having second priority for performing those data transfers due to the data mover device 204c being located in the node 204 with the memory subsystem 204b that provides the destination of the data for the data transfer. As such, this embodiment of the present disclosure prioritizes data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the source of the data for the data transfer over data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the destination of the data for the data transfer.

However, while a specific example, of a data mover selection table 500 has been described for the specific configuration of the computing device 200 illustrated in FIG. 2, one of skill in the art in possession of the present disclosure will recognize that data mover selection tables may differ based on the configuration of the computing device for which they are generated (e.g., the number of nodes in the computing device, the location of the data mover devices, the memory subsystem and/or memory location/address ranges associated with the data mover devices, etc.), as well as based on a variety of other system features that will fall within the scope of the present disclosure as well. For example, while the specific data mover selection table 500 discussed above prioritizes data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the source of the data for the data transfer over data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the destination of the data for the data transfer, the prioritization of data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the destination of the data for the data transfer over data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem that provides the source of the data for the data transfer will fall within the scope of the present disclosure as well. Thus, data mover selection tables (and/or other techniques for providing for the selection of data mover devices according to the teachings of the present disclosure) may vary from the specific examples described herein while remaining within the scope of the present disclosure as well.

Furthermore, in some embodiments, the information in the data mover selection table 500 may be relatively static during runtime operations for the computing device 200 and following the initialization operations for the computing device 200. However, one of skill in the art in possession of the present disclosure will recognize how the data mover selection table 500 may be generated each time the computing device is initialized in order to, for example, allow for the movement of data mover devices (e.g., provided on a PCIe add-in card) to be reflected in the data mover selection table 500. As such, dynamic modification to the data mover selection table 500 across system boots (or during runtime in some examples) is envisioned as falling within the scope of the present disclosure.

The method 300 then proceeds to block 304 where an operating system determines that a data transfer operation provides for the transfer of data between memory locations. As illustrated in FIG. 6, in an embodiment of block 304, an operating system 600 and an application 602 may be provided by, for example, one or more of the processing subsystems 204a and 206a executing instructions stored on one or more of the memory subsystems 204b and 206b. As also illustrated in FIG. 6, at block 304, the application 602 may perform data transfer request operations 604 that may include, for example, a data transfer request that requests the performance of a data transfer between memory locations/addresses provided by the memory subsystem 204b and/or 206b. In a specific example, the operating system 600 may include an operating system driver that provides user-level abstraction for the querying of an operating system kernel in the operating system for data mover resources for data transfers and, thus, at block 304 the operating system driver may receive the data transfer request from the application 602.

Furthermore, one of skill in the art in possession of the present disclosure will recognize how the application 602 may query the operating system driver provided by the node to which the processing subsystems core(s)/thread(s) executing the application 602 are affinitized to. In other words, based on information received from the application 602, the operating system driver for the operating system 600 may identify the source address, destination address, and the size of memory block of data that needs to be moved in a memory transaction, and may present that information to the operating system kernel (i.e., the operating system driver may query the operating system kernel for which data mover to be use in the memory transaction, and the operating system kernel may then access the information provided in the data mover lookup table and return the data mover that the operating system driver should use for that memory transaction). However, while a specific configuration for providing data transfer operations to the operating system 600 has been described, one of skill in the art in possession of the present disclosure will appreciate that data transfer requests that request data transfers between memory locations may be provided to an operating system by a variety of components and in a variety of manners that will fall within the scope of the present disclosure as well.

The method 300 then proceeds to block 306 where the operating system identifies a data mover device in the data mover selection table with a highest priority for transferring data between the memory locations. With reference to FIG. 7, in an embodiment of block 306, the operating system 600 may operate to perform data mover selection operations 700 that include the accessing of the data mover selection table 500 stored in the BIOS database 210 and the selection of a data mover device for performing the data transfer operations determined at block 304. Continuing with the specific example discussed above, upon receiving the data transfer request from the application 602, the operating system driver in the operating system 600 may operate to send a data mover device selection request the operating system kernel in the operating system 600 to select a data mover device for performing the data transfer operation associated with the data transfer request, and the operating system kernel in the operating system 600 will operate to access the data mover selection table 500 in the BIOS database 210 in order to select a data mover device for performing the data transfer operations.

With reference to the data mover selection table 500 discussed above with reference to FIG. 5, at block 306 the operating system 600 may use the memory locations/addresses identified in the data transfer request to identify a data mover device for performing the data transfer operations. For example, if the source memory location/address falls in the range of “0-0xFFF” and the destination memory location/address falls in the range of “0x1000-0x1FFF”, the operating system 600 may identify the data mover device 204c for performing the data transfer operations (i.e., because the data mover device 204c is the only data mover device identified/prioritized for performing data transfers between those memory locations/addresses). In another example, if the source memory location/address falls in the range of “0x1000-0x1FFF” and the destination memory location/address falls in the range of “0x2000-0x2FFF”, the operating system 600 may identify the data mover device 204c for performing the data transfer operations (i.e., because the data mover device 204c is prioritized over the data mover device 206c for performing data transfers between those memory locations/addresses). In another example, if the source memory location/address falls in the range of “0x3000-0x3FFF” and the destination memory location/address falls in the range of “0-0xFFF”, the operating system 600 may identify the data mover device 206c for performing the data transfer operations (i.e., because the data mover device 206c is prioritized over the data mover device 204c for performing data transfers between those memory locations/addresses). Similarly, if the source memory location/address falls in the range of “0x2000-0x2FFF” and the destination memory location/address falls in the range of “0x3000-0x3FFF”, the operating system 600 may identify the data mover device 206c for performing the data transfer operations (i.e., because the data mover device 206c is the only data mover device identified/prioritized for performing data transfers between those memory locations/addresses).

As such, one of skill in the art in possession of the present disclosure will appreciate how the data mover selection table 500 allows the operating system 600 to select, for any data transfer request that provides for the transfer of data between memory locations, a data mover device that is configured to perform the most efficient data transfer between those memory locations (e.g., based on that data mover device having the highest “affinity”/smallest “distance” relative to one or more of those memory locations, and/or on other factors that would be apparent to one of skill in the art in possession of the present disclosure.) However, while a specific data mover selection table has been described as being utilized to select a data mover device for a data transfer operation based on particular data transfer efficiency characteristics, one of skill in the art in possession of the present disclosure will recognize that the selection of a data mover device for performing a data transfer operation in other manners and/or based on other data mover device selection characteristics will fall within the scope of the present disclosure as well.

The method 300 then proceeds to decision block 308 where it is determined whether the identified data mover device exceeds a data transfer operation threshold. In an embodiment, at decision block 308, the operating system 600 may operate to determine whether the data mover device selected at block 306 is currently operating such that it exceeds a data transfer operation threshold. As will be appreciated by one of skill in the art in possession of the present disclosure, any data mover device selected at block 306 may already be performing one or more data transfer operations, and the data mover selection system of the present disclosure may define a data transfer operation threshold above which a data mover device should not be utilized to perform a requested data transfer operation (i.e., despite its selection/ident cation at block 306). As such, for any data mover device selection/identification at block 306, the operating system 600 may perform a check to determine the operating level of that data mover device in order to ensure that data mover device will not be overloaded if it performs the data transfer operations determined at block 304.

If, at decision block 308, it is determined that the identified data mover device exceeds the data transfer operation threshold, the method 300 proceeds to block 310 where the identified data mover device is ignored. In an embodiment, at block 310 and in response to determining that the identified data mover device exceeds the data transfer operation threshold, the operating system 600 may operate to ignore that data mover device and the method 300 will return to block 306. As such, in the event a data mover device is selected at block 306 and determined to exceed the data transfer operation threshold at block 308 of a first iteration of the method 300, that data mover device will be ignored at block 310, and a different data mover device will be selected at block 306 of second iteration of the method 300. Thus, one of skill in the art in possession of the present disclosure will recognize how the method 300 may loop through blocks 306, 308, and 310 until a data mover device is selected/identified that does not exceed the data transfer operation threshold. As such, following any iteration of the method 300 in which a data mover device is identified that exceeds the data transfer threshold, the next “best” data mover device may be identified until a data mover device is identified that does not exceed the data transfer threshold. One of skill in the art in possession of the present disclosure will appreciate that, in some embodiments, changing data transfer operations by a data mover device may result in the same data mover device that was identified in a previous iteration of the method 300 being identified in a subsequent iteration of the method 300. Furthermore, rather than perform the iterative process discussed above, in the event a data mover is identified that exceeds the data transfer threshold, the method 300 may simply operate to identify the next “best” data mover device and proceed to block 312, discussed in further detail below.

In a specific example, at block 306 on a first iteration of the method 300, the data mover device 206c may have been identified by the operating system 600 as having first priority for performing data transfers between a source memory location/address that is included in the memory location/address range of “0x3000-0x3FFF” that is provided by the memory subsystem 206b in this example and a destination memory location/address that is included in the memory location/address range of “0x1000-0x1FFF” that is provided by the memory subsystem 204b in this example (i.e., due to the data mover device 206c being located in the node 206 with the memory subsystem 206b that provides the source of the data for the data transfer and, thus, having a higher “affinity”/smaller “distance” relative to that memory subsystem 206b that provides for more efficient source data transfers relative to source data transfers performed by the data mover device 204c that is located in the node 204). At decision block 308, the operating system 600 may determine that the data mover device 206c exceeds the data transfer operation threshold and, in response, the operating system 600 will operate to ignore the data mover device 206c at block 310. Subsequently, at block 306 on a second iteration of the method 300, the data mover device 204c will be identified by the operating system 600 as having second (and now highest) priority for performing those data transfers (i.e., due to the data mover device 204c being located in the node 204 with the memory subsystem 204b that provides the destination of the data for the data transfer.)

As such, the prioritization of the data mover devices in the data mover selection table 500 allows lower priority data mover devices to be selected over higher priority data mover devices in the event the higher priority data mover devices exceed the data transfer operation threshold. As will be appreciated by one of skill in the art in possession of the present disclosure, in some embodiments and in the event only a single data mover device is identified for performing data transfers between different memory location/address ranges (e.g., the data mover device 204c identified for performing data transfers between the source memory range “0-0xFFF” and the destination memory range “0x1000-0x1FFF” in the data mover selection table 500), that data mover device may be selected/identified for performing the data transfer operations despite the fact that it exceeds the data transfer operation threshold. However, in other embodiments and in the event only a single data mover device is identified for performing data transfers between different memory location/address ranges (e.g., the data mover device 206c identified for performing data transfers between the source memory range “0x3000-0x3FFF” and the destination memory range “0x3000-0x3FFF” in the data mover selection table 500), the operating system 600 may select and/or identify a different data mover device for performing the data transfer operations in the event the data mover device identified in the data mover selection table 500 exceeds the data transfer operation threshold. As such, one of skill in the art in possession of the present disclosure will recognize that the data transfer operation threshold may be used to prevent the overloading of data mover devices in a variety of manners that will fall within the scope of the present disclosure as well.

If at decision block 308, it is determined that the identified data mover device does not exceed the data transfer operation threshold, the method 300 proceeds to block 312 where the operating system transmits a data transfer instruction to the identified data mover device. With reference to FIG. 8, in an embodiment of block 312 and in response to the selection/identification of the data mover device 204c at block 306, the operating system 600 may perform data transfer instruction operations 800 to transfer a data transfer instruction to the data mover device 204c. With reference to FIG. 11, in an embodiment of block 312 and in response to the selection/identification of the data mover device 206c at block 306, the operating system 600 may perform data transfer instruction operations 1100 to transfer a data transfer instruction to the data mover device 206c. However, while specific examples are provided, as discussed below data mover devices may be provided in different configurations and/or locations within the computing device 200, and thus the transmission of data transfer instructions to any of those data mover devices will fall within the scope of the present disclosure as well.

The method 300 then proceeds to block 314 where the identified data mover device transfers data between the memory locations. With reference to FIG. 9, in an embodiment of block 314, the data mover device 204c may receive the data transfer instructions as part of the data transfer instruction operations 800 from the operating system 600 and, in response, perform data transfer operations 900 included in those data transfer instructions. In this example, the data transfer instructions instruct the transfer of data between a source memory location/address included in the memory location/address range “0-0xFFF” provided by the memory subsystem 204b, and a destination memory location/address included in the memory location/address range “0x1000-0x1FFF” provided by the memory subsystem 204b, and the data transfer operations 900 provide for the transfer of data from a data location 902a in the memory subsystem 204b (included in the memory location/address range “0-0x1 FFF”) to a data location 902b in the memory subsystem 204b (included in the memory location/address range “0x1000-0x1FFF”). As will be appreciated by one of skill in the art in possession of the present disclosure, following the data transfer operations 900, the data mover device 204c may provide a data transfer confirmation to the operating system 600, and the operating system 600 may provide a data transfer confirmation to the application 602.

With reference to FIG. 10, in an embodiment of block 314, the data mover device 204c may receive the data transfer instructions as part of the data transfer instruction operations 800 from the operating system 600 and, in response, perform data transfer operations 1000 included in those data transfer instructions. In this example, the data transfer instructions instruct the transfer of data between a source memory location/address included in the memory location/address range “0x1000-0x1FFF” provided by the memory subsystem 204b, and a destination memory location/address included in the memory location/address range “0x2000-0x2FFF” provided by the memory subsystem 206b, and the data transfer operations 1000 provide for the transfer of data from a data location 1002a in the memory subsystem 204b (included in the memory location/address range “0x1000-0x1FFF”) to a data location 1002b in the memory subsystem 206b (included in the memory location/address range “0x2000-0x2FFF”). As will be appreciated by one of skill in the art in possession of the present disclosure, following the data transfer operations 1000, the data mover device 204c may provide a data transfer confirmation to the operating system 600, and the operating system 600 may provide a data transfer confirmation to the application 602.

With reference to FIG. 12, in an embodiment of block 314, the data mover device 206c may receive the data transfer instructions as part of the data transfer instruction operations 1100 from the operating system 600 and, in response, perform data transfer operations 1200 included in those data transfer instructions. In this example, the data transfer instructions instruct the transfer of data between a source memory location/address included in the memory location/address range “0-0xFFF” provided by the memory subsystem 204b, and a destination memory location/address included in the memory location/address range “0x3000-0x3FFF” provided by the memory subsystem 206b, and the data transfer operations 1200 provide for the transfer of data from a data location 1202a in the memory subsystem 204b (included in the memory location/address range “0-0xFFF”) to a data location 1202b in the memory subsystem 206b (included in the memory location/address range “0x3000-0x3FFF”). As will be appreciated by one of skill in the art in possession of the present disclosure, following the data transfer operations 1200, the data mover device 206c may provide a data transfer confirmation to the operating system 600, and the operating system 600 may provide a data transfer confirmation to the application 602.

Thus, systems and methods have been described that provide for the selection of one of a plurality of data mover devices for which to perform data transfer operations between memory locations provided in one or more NUMA nodes based on an “affinity” of that data mover device to at least one of those memory locations that provides that data mover device a higher priority for performing the data transfer that the other data mover devices. For example, the data mover selection system of the present disclosure may include a first data mover device and a second data mover device that are both coupled to a memory system provided by a plurality of NUMA nodes, and an operating system that is coupled to the first data mover device and the second data mover device. The operating system determines that a first data transfer operation provides for the transfer of data from a first memory location in the memory system provided by the plurality of NUMA nodes to a second memory location in the memory system provided by the plurality of NUMA nodes, identifies the first data mover device for performing the first data transfer operation based on the first data mover device having a higher priority relative to the second data mover device for performing data transfers from the first memory location in the memory system provided by the plurality of NUMA nodes to the second memory location in the memory system provided by the plurality of NUMA nodes and, in response, transmits a first data transfer instruction to the first data mover device that is configured to cause the first data mover device to perform the first data transfer operation to transfer data from the first memory location in the memory system provided by the plurality of NUMA nodes to the second memory location in the memory system provided by the plurality of NUMA nodes. As such, more efficient data transfers may be performed by data mover devices (relative to conventional “round robin” data mover device selections) based on their affinity to one or more of the memory locations involved in those data transfers.

Referring now to FIG. 13, an embodiment of a networked system 1300 is illustrated. In the illustrated embodiment, the networked system 1300 includes a plurality of computing devices 1302a, 1302b, and up to 1302c, any or all of which may be provided by the computing device 200 discussed above with reference to FIG. 2. As such, any or all of the computing devices 1302a-1302c may be provided by the IHS 100 discussed above with reference to FIG. 1, and/or may include some or all of the components of the IHS 100, and in specific examples, may be provided by server computing devices and/or other computing devices known in the art. However, while illustrated and discussed as being provided by server computing devices, one of skill in the art in possession of the present disclosure will recognize that computing devices provided in the networked system 1300 may include any devices that may be configured to operate similarly as the computing devices 1302a-1302c discussed below. In the illustrated embodiment, each of the computing devices 1302a-1302c may be coupled to a network 1304 that may be provided by a Local Area Network (LAN), the Internet, combinations thereof, and/or any other network that would be apparent to one of skill in the art in possession of the present disclosure. As will be appreciated by one of skill in the art in possession of the present disclosure, the network 1304 may be utilized to provide a memory fabric/memory links such as those available via protocols promulgated by the Gen-Z consortium, the Compute Express Link (CXL) standard, and/or other network connectors memory systems known in the art. As such, the network 1304 may provide an extension of memory links (e.g., Double Data Rate (DDR) memory links, DDR-T memory links, etc.) via the Gen-Z protocol, CXL standard, and or other techniques that enables memory semantics for moving data with a data mover device.

In the illustrated embodiment, a fabric manager system 1306 is coupled to the computing devices 1302a-1302c via the network 1304. In an embodiment, the fabric manager system 1306 may be provided by the IHS 100 discussed above with reference to FIG. 1, and/or may include some or all of the components of the IHS 100, and in specific examples, may be provided by server computing devices and/or other computing devices known in the art. However, while illustrated and discussed as being provided by server computing devices, one of skill in the art in possession of the present disclosure will recognize that fabric manager systems provided in the networked system 1300 may include any devices that may be configured to operate similarly as the fabric manager system 1306 discussed below. However, while a specific networked system 1300 has been illustrated and described, one of skill in the art in possession of the present disclosure will recognize that networked systems utilizing the data mover selection system of the present disclosure may include a variety of components and component configurations while remaining within the scope of the present disclosure as well.

Referring now to FIG. 14, an embodiment of a fabric manager system 1400 is illustrated that may provide the fabric manager system 1306 discussed above with reference to FIG. 13. As such, the fabric manager system 1400 may be provided by the IHS 100 discussed above with reference to FIG. 1 and/or may include some or all of the components of the IHS 100, and in specific examples may be provided by a server computing device. Furthermore, while illustrated and discussed as being provided by a server computing device, one of skill in the art in possession of the present disclosure will recognize that the functionality of the fabric manager system 1400 discussed below may be provided by other devices that are configured to operate similarly as the fabric manager system 1400 discussed below. In the illustrated embodiment, the fabric manager system 1400 includes a chassis 1402 that houses the components of the fabric manager system 1400, only some of which are illustrated below. For example, the chassis 1402 may house a processing system (not illustrated, but which may include the processor 102 discussed above with reference to FIG. 1) and a memory system (not illustrated, but which may include the memory 114 discussed above with reference to FIG. 1) that is coupled to the processing system and that includes instructions that, when executed by the processing system, cause the processing system to provide a fabric manager engine 1404 that is configured to perform the functionality of the fabric manager engines and/or fabric manager systems discussed below, as well as functionality including the composing and mapping of a data mover device to a specific computing device, the remapping of a data mover device to a different computing device, the composing of a data mover device to two or more computing devices, and/or other functionality that would be apparent to one of skill in the art in possession of the present disclosure.

The chassis 1402 may also house a storage system (not illustrated, but which may include the storage 108 discussed above with reference to FIG. 1) that is coupled to the fabric manager engine 1404 (e.g., via a coupling between the storage system and the processing system) and that includes a fabric manager database 1406 that is configured to store any of the information utilized by the fabric manager engine 1404 discussed below. The chassis 1402 may also house a communication system 1408 that is coupled to the fabric manager engine 1404 (e.g., via a coupling between the communication system 1408 and the processing system) and that may be provided by a Network Interface Controller (NIC), wireless communication systems (e.g., BLUETOOTH®, Near Field Communication (NFC) components, WiFi components, etc.), and/or any other communication components that would be apparent to one of skill in the art in possession of the present disclosure. While a specific fabric manager system 1400 has been illustrated, one of skill in the art in possession of the present disclosure will recognize that fabric manager systems (or other devices operating according to the teachings of the present disclosure in a manner similar to that described below for the fabric manager system 1400) may include a variety of components and/or component configurations for providing conventional fabric management functionality (e.g., composing network-accessible memory subsystems for computing devices in the networked subsystem 1300), as well as the functionality discussed below, while remaining within the scope of the present disclosure as well.

Referring now to FIG. 15, an embodiment of a method 1500 for selecting a data mover device is illustrated. As discussed below, the systems and methods of the present disclosure provide for the selection of a data mover device for use in performing a data transfer in a server computing device cluster and/or memory fabric environment. For example, the data mover selection system of the present disclosure includes a fabric manager system coupled to a plurality of computing devices that are coupled to a memory system, with the fabric manager system operating to receive respective local data mover selection information from each of the plurality of computing devices that identifies at least one data mover device accessible to that computing device, and using the respective local data mover selection information to generate global data mover selection information that includes each data mover device accessible to the plurality of computing devices. Subsequently, when the fabric manager system receives a first data transfer request from a first computing device that provides for the transfer of data from a first memory location in the memory system to a second memory location in the memory system, it uses the global data mover selection information to identify a first data mover device for performing the first data transfer operation based on the first data mover device having a higher priority relative to other data mover devices included in the global data mover selection information for performing data transfers from the first memory location in the memory system to the second memory location in the memory system. The fabric manager system may then transmit a first data transfer instruction to the first data mover device that is configured to cause the first data mover device to perform the first data transfer operation to transfer data from the first memory location in the memory system to the second memory location in the memory system. As such, data transfers may be performed by data mover devices that provide more efficient data transfers (relative to conventional “round robin” data mover device selections) based on their affinity to one or more of the memory locations involved in those data transfers.

The method 1500 begins at block 1502 where a fabric manager system generates a global data mover selection table. In an embodiment, during or prior to the method 1500, each of the computing devices 1302a-1302c may operate according to the method 300 in order to generate local data mover selection information provided by the data mover selection table as discussed above with reference to block 302. As such, each of the computing devices 1302a-1302c may reset, reboot, and/or otherwise initialize and, in response, a BIOS engine in the BIOS of those computing devices 1302a-1302c may operate to generate a local data mover selection table that is similar to the data mover selection table 500 discussed above with reference to FIG. 5, and that one of skill in the art in possession of the present disclosure will recognize is “local” and/or otherwise specific to that computing device based on the memory subsystem and data mover devices accessible to that computing device. However, in some embodiment, the local data mover selection information/tables for any of the computing devices 1302a-1302c may be generated by the fabric manager system 1306 while remaining within the scope of the present disclosure as well.

With reference to FIG. 16, in an embodiment of block 1502, each of the computing devices 1302a-1302c may operate to perform local data mover selection information transmission operations 1600 in order to transmit their local data mover selection information (e.g., their local data mover selection tables) via the network 1304 to the fabric manager system 1306. For example, the BIOS engine that provides to the BIOS in each computing device 1302a-1302c, or the operating system 600 in each computing device 1302a-1302c, may operate at block 1502 to perform the local data mover selection information transmission operations 1600 at block 1502. However, while specific techniques for publishing/announcing local data mover selection information to the fabric manager system 1306 are discussed above, one of skill in the art in possession of the present disclosure will appreciate that a variety of techniques that provide for the publishing or other announcement of the local data mover selection information for each computing device 1302a-1302c to the fabric manager system 1306 will fall within the scope of the present disclosure as well. As such, at block 1502, the fabric manager engine 1404 in the fabric manager system 1306/1400 may receive the local data mover selection information via its communication system 1408 from each of the computing devices 1302a-1302c.

In an embodiment, at block 1502 and in response to receiving the local data mover selection information from each of the computing devices 1302a-1302c, the fabric manager engine 1404 in the fabric manager system 1306/1400 may operate to generate global data mover selection information that is provided in the examples discussed below in a global data mover selection table. For example, in response to receiving the local data mover selection information from each of the computing devices 1302a-1302c, the fabric manager engine 1404 in the fabric manager system 1306/1400 may map each data mover device in the networked system 1300 to the computing devices 1302a-1302c that are configured to access those data mover devices (as identified in the local data mover selection information/tables). In a specific example, the following data-mover-device-to-computing-device mapping may be generated at block 1502:

DATA MOVER DEVICE COMPUTING DEVICE IDENTIFIER IDENTIFIER DATA MOVER DEVICE A COMPUTING DEVICE 1302a DATA MOVER DEVICE B COMPUTING DEVICE 1302a DATA MOVER DEVICE C COMPUTING DEVICE 1302b DATA MOVER DEVICE D COMPUTING DEVICE 1302b DATA MOVER DEVICE E COMPUTING DEVICE 1302a COMPUTING DEVICE 1302b DATA MOVER DEVICE F COMPUTING DEVICE 1302a COMPUTING DEVICE 1302b

As will be appreciated by one of skill in the art in possession of the present disclosure, the example of the data-mover-device-to-computing-device mapping above is a simplified example having a pair of computing devices 1302a and 1302b, with the computing device 1302a reporting in its local data mover selection information/table that it is configured to access data mover devices A, B, E, and F, and the computing device 1302b reporting in its local data mover selection information/table that it is configured to access data mover devices C, D, E, and F. However, one of skill in the art in possession of the present disclosure will appreciate that data-mover-device-to-computing-device mappings may map any number of data mover devices to any number of computing devices while remaining within the scope of the present disclosure as well.

Furthermore, in the example below, the fabric manager engine 1404 in the fabric manager system 1306/1400 may identify a “free” memory pool (e.g., memory subsystem(s) in the computing device 1302c that are shared with the computing devices 1302a and/or 1302b, other memory subsystems in the network system 1300 that may be configured for use by the computing devices 1302a and/or 1302b, etc.) that is accessible by the computing devices 1302a and/or 1302b and one or more data mover devices (e.g., the data mover device G discussed below). For example, the networked system 1300 may include a Gen-Z memory fabric that provides network-connected memory subsystems that are accessible by one or more of the computing devices 1302a-1302c, and one of skill in the art in possession of the present disclosure will appreciate that the fabric manager system 1306 may be configured to provision such network-connected memory subsystems, and data mover devices that are capable of transferring data to/from those memory subsystems, to any of the computing devices 1302a-1302c.

Thus, in an embodiment of block 1502, the fabric manager engine 1404 in the fabric manager system 1306/1400 may utilize the data-mover-device-to-computing-device mapping to generate global data mover selection information. In the examples provided below, the global data mover selection information is provided in a global data mover selection table that describes the data mover device(s) capable of accessing each memory location/address range provided by a host and/or a network connected memory subsystem, and assigns priorities to each data mover device capable of accessing a particular memory location/address range when more than one data mover device is capable of accessing that memory location/address range. However, one of skill in the art in possession of the present disclosure will recognize that the global data mover selection information may be provided in a variety of manners that will fall within the scope of the present disclosure as well.

With reference to FIG. 17, an embodiment of a global data mover selection table 1700 that may be generated and stored in fabric manager database 1406 at block 1502. As will be appreciated by one of skill in the art in possession of the present disclosure, the fabric manager engine 1404 in the fabric manager system 1306/1400 may generate the global data mover selection table 1700 that maps memory boundaries in the memory subsystems available in the network system 1300 to the data mover devices based on an “affinity” or “distance” of the data mover devices to those memory boundaries, with data mover devices having higher “affinities”/smaller “distances” to particular memory subsystems prioritized for performing data transfers associated with those memory subsystems. As such, the specific example of the global data mover selection table 1700 corresponding to the computing devices 1302a and 1302b illustrated in FIG. 1300 maps memory locations to data mover devices by providing “source” memory location/address ranges in particular computing devices or a free memory pool in a first column in the global data mover selection table 1700, providing “destination” memory location/address ranges in particular computing devices or a free memory pool in a first row in the global data mover selection table 1700, and identifying the priority/priorities of the data mover devices for data transfers between any particular combination of a source memory location/address and a destination memory location/address.

As such, with reference to the specific example provided in FIG. 17, for data transfers between a source memory location/address and a destination memory location/address that are both included in a memory location/address range in the computing device 1302a in this example, the data mover device A is identified as having a first priority for performing those data transfers due to the data mover device A having a highest “affinity”/smaller “distance” relative to that memory location/address range that provides for more efficient data transfers (e.g., relative to data transfers performed by other data mover devices in the networked system 1300), and the data mover device B is identified as having second priority for performing those data transfers due to the data mover device B having a second highest “affinity”/smaller “distance” relative to that memory location/address range that provides for more efficient data transfers (e.g., relative to data transfers performed by other data mover devices in the networked system 1300 except the data mover device A).

Similarly, for data transfers between a source memory location/address that is included in a memory location/address range in the computing device 1302a and a destination memory location/address that is included in a memory location/address range in the computing device 1302b in this example, the data mover device E is identified as having a first priority for performing those data transfers due to the data mover device E having a highest “affinity”/smaller “distance” relative to that memory location/address range that provides for more efficient data transfers (e.g., relative to data transfers performed by other data mover devices in the networked system 1300), and the data mover device F is identified as having second priority for performing those data transfers due to the data mover device F having a second highest “affinity”/smaller “distance” relative to that memory location/address range that provides for more efficient data transfers (e.g., relative to data transfers performed by other data mover devices in the networked system 1300 except the data mover device D).

Similarly, for data transfers between a source memory location/address that is included in a memory location/address range in the computing device 1302a and a destination memory location/address that is included in a memory location/address range in the free memory pool in this example, the data mover device E is identified as having priority for performing those data transfers due to the data mover device E having a highest “affinity”/smaller “distance” relative to that memory location/address range that provides for more efficient data transfers (e.g., relative to data transfers performed by other data mover devices in the networked system 1300).

Similarly, for data transfers between a source memory location/address that is included in a memory location/address range in the computing device 1302b and a destination memory location/address that is included in a memory location/address range in the computing device 1302a in this example, the data mover device E is identified as having a first priority for performing those data transfers due to the data mover device E having a highest “affinity”/smaller “distance” relative to that memory location/address range that provides for more efficient data transfers (e.g., relative to data transfers performed by other data mover devices in the networked system 1300), and the data mover device F is identified as having second priority for performing those data transfers due to the data mover device F having a second highest “affinity”/smaller “distance” relative to that memory location/address range that provides for more efficient data transfers (e.g., relative to data transfers performed by other data mover devices in the networked system 1300 except the data mover device E).

Similarly, for data transfers between a source memory location/address and a destination memory location/address that are both included in a memory location/address range in the computing device 1302b in this example, the data mover device C is identified as having a first priority for performing those data transfers due to the data mover device C having a highest “affinity”/smaller “distance” relative to that memory location/address range that provides for more efficient data transfers (e.g., relative to data transfers performed by other data mover devices in the networked system 1300), and the data mover device D is identified as having second priority for performing those data transfers due to the data mover device D having a second highest “affinity”/smaller “distance” relative to that memory location/address range that provides for more efficient data transfers (e.g., relative to data transfers performed by other data mover devices in the networked system 1300 except the data mover device C).

Similarly, for data transfers between a source memory location/address that is included in a memory location/address range in the computing device 1302b and a destination memory location/address that is included in a memory location/address range in the free memory pool in this example, no data mover device is identified as having priority for performing those data transfers due to none of the data mover devices in the networked system 1300 having a highest “affinity”/smaller “distance” relative to that memory location/address range that provides for more efficient data transfers (e.g., relative to data transfers performed by other data mover devices in the networked system 1300).

Similarly, for data transfers between a source memory location/address that is included in a memory location/address range in the free memory pool and a destination memory location/address that is included in a memory location/address range in the computing device 1302a in this example, the data mover device G is identified as having priority for performing those data transfers due to the data mover device G having a highest “affinity”/smaller “distance” relative to that memory location/address range that provides for more efficient data transfers (e.g., relative to data transfers performed by other data mover devices in the networked system 1300).

Similarly, for data transfers between a source memory location/address that is included in a memory location/address range in the free memory pool and a destination memory location/address that is included in a memory location/address range in the computing device 1302b in this example, the data mover device G is identified as having priority for performing those data transfers due to the data mover device G having a highest “affinity”/smaller “distance” relative to that memory location/address range that provides for more efficient data transfers (e.g., relative to data transfers performed by other data mover devices in the networked system 1300), and the data mover device D is identified as having second priority for performing those data transfers due to the data mover device D having a second highest “affinity”/smaller “distance” relative to that memory location/address range that provides for more efficient data transfers (e.g., relative to data transfers performed by other data mover devices in the networked system 1300 except the data mover device G).

Similarly, for data transfers between a source memory location/address and a destination memory location/address that are both included in a memory location/address range in the free memory pool in this example, the data mover device G is identified as having priority for performing those data transfers due to the data mover device G having a highest “affinity”/smaller “distance” relative to that memory location/address range that provides for more efficient data transfers (e.g., relative to data transfers performed by other data mover devices in the networked system 1300).

However, while a specific example of a global data mover selection table 1700 has been described for the specific configuration of the networked system 1300 illustrated in FIG. 13, one of skill in the art in possession of the present disclosure will recognize that global data mover selection tables may differ based on the configuration of the networked system for which they are generated (e.g., the number of computing devices in the networked system, the location of the data mover devices in the networked system, the memory subsystem and/or memory location/address ranges associated with the data mover devices, etc.), as well as based on a variety of other system features that will fall within the scope of the present disclosure as well. For example, while the specific global data mover selection table 1700 discussed above prioritizes data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem in the computing device that provides the source of the data for the data transfer over data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem in the computing device that provides the destination of the data for the data transfer, the prioritization of data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem in the computing device that provides the destination of the data for the data transfer over data mover devices with a higher “affinity”/smaller “distance” relative to the memory subsystem in the computing device that provides the source of the data for the data transfer will fall within the scope of the present disclosure as well. Thus, global data mover selection tables (and/or other techniques for providing for the selection of data mover devices according to the teachings of the present disclosure) may vary from the specific examples described herein while remaining within the scope of the present disclosure as well.

Furthermore, in some embodiments, the information in the global data mover selection table 1700 may be relatively static during runtime operations for the computing devices 1302a-1302b and following the initialization operations for the computing devices 1302a-1302c However, one of skill in the art in possession of the present disclosure will recognize how the global data mover selection table 1700 may be modified during runtime operations for the computing devices 1302a-1302c in order to, for example, allow for the movement of data mover devices (e.g., provided on a PCIe add-in card) to be reflected in the global data mover selection table 1700, the addition or removal of computing devices to/from the networked system 1300, the addition or removal of memory subsystems to/from the networked system 1300, the addition or removal of data mover devices to/from the networked system 1300, etc. As discussed above, the fabric manager system 1306 may be configured to remap a data mover device to a different computing device, which one of skill in the art in possession of the present disclosure would recognize would result in corresponding changes to the global data mover selection table 1700. As such, dynamic modification to the global data mover selection table 1700 across system boots (and during runtime in some examples) is envisioned as falling within the scope of the present disclosure.

As discussed below, in some embodiments of the method 1500, the fabric manager system 1306 may operate to manage data mover device selection using the global data mover selection information/table for the computing devices 1302a-1302c. However, in other embodiments of the method 1500 discussed below, the computing devices 1302a-1302c may manage data mover device selection via their local data mover selection information/tables that have been supplemented or otherwise updated by the fabric manager system 1306 using information from (or similar to that discussed above in) the global data mover selection information/table. For example, similar to the transmission of the local data mover selection information by each of the computing devices 1302a-1302c discussed above, each computing device 1302a-1302c in the networked system 1300 may submit networked system resources (e.g., fabric resources such as memory subsystems and data mover devices) that it wishes to utilize to the fabric manager system 1306, and the fabric manager engine 1404 in the fabric manager system 1306/1400 will operate to configure those networked system resources for use by those computing devices 1302a-1302c. As such, the fabric manager engine 1404 in the fabric manager system 1306/1400 may orchestrate data mover device and memory subsystem assignments to the computing devices 1302a-1302c based on networked system resource requests received from those computing devices 1302a-1302c.

Similarly as discussed above, based on the networked system resource requests received from the computing devices 1302a-1302c, the fabric manager engine 1404 in the fabric manager system 1306/1400 may create the data-mover-device-to-computing-device mapping, and may use that data-mover-device-to-computing-device mapping to instruct each computing device 1302a-1302c to update its local data mover selection information/table to identify the networked system resources that have been configured for its use. For example, FIG. 18 illustrates the fabric manager engine 1404 in the fabric manager system 1306/1400 performing data mover selection information updating operations 1800 to instruct updates to the local data mover selection information on each of the computing devices 1302a-1302c to identify the networked system resources that have been configured for use by those computing devices 1302a-1302c. Thus, in such embodiments, the local data mover selection information/tables in each computing device 1302a-1302c may be updated (e.g., from the form illustrated and discussed above with reference to FIG. 5) to identify prioritized data mover device(s) for use in performing data transfers between any network-accessible memory subsystems (i.e., memory locations/address ranges) in the networked subsystem 1300.

As discussed in further detail below, the updating of the local data mover selection information/table in each computing device 1302a-1302c with a portion of the global data mover selection information/table pertinent to that computing device allows each computing device to select a data mover device for performing any data transfer between memory locations/address ranges in memory subsystems accessible to that computing device. As will be appreciated by one of skill in the art in possession of the present disclosure, such techniques may be utilized to reduce congestion in the fabric manager system 1306 as, once the local data mover selection information/tables have been updated in each of the computing devices 1302a-1302c according to those techniques, the fabric manager system 1306 is not needed to select data mover devices, providing for faster data mover device selection and eliminating the fabric manager system as a point of failure. However, one of skill in the art in possession of the present disclosure will also appreciate that such techniques may complicate the identification of overloaded data mover devices across the networked system 1300 by the fabric manager system 1306, as well as runtime updates to the data mover selection information on the computing devices 1302a-1302c during their runtime operations.

The method 1500 then proceeds to block 1504 where a data transfer request to transfer data between memory locations is received. As illustrated in FIG. 19A, in an embodiment of block 1504 in which the fabric manager system 1306 orchestrates data mover device selection, the computing device 1302a may perform data transfer request operations 1900 that may include, for example, a data transfer request that requests the performance of a data transfer between memory locations/addresses provided by memory subsystems in the networked system 1300. In some examples, the computing devices 1302a-1302c may be configured to select data mover devices themselves when that data mover device is to be used in performing a data transfer between memory locations local to that computing device, while being configured to transmit the data transfer request to have the fabric manager system 1306 select a data mover device when that data mover device is to be used in performing a data transfer with at least one memory locations that is connected to that computing device via the fabric (e.g., an external/non-local memory subsystem, a network connected memory subsystem, etc.).

As such, at block 1504, the fabric manager engine 1404 in the fabric manager system 1306/1400 may receive the data transfer request provided via the data transfer request operations 1900 via its communication system 1308. In a specific example, the fabric manager system 1306 may assign a unique identifier to each computing device 1302a-1302c that allow the fabric manager system 1306 to identify data transfer requests from those computing devices 1302a-1302c. Furthermore, any data transfer request provided by a computing device 1302a-1302c may include a source memory location identifier, a source memory range, a destination memory location identifier, and a destination memory range, which one of skill in the art in possession of the present disclosure will recognize will allow the fabric manager system 1306 to perform the data mover device selection operations discussed below. However, in another embodiment of block 1504 in which the computing devices themselves orchestrate data mover device selection, the data transfer request at block 1504 may be received by an operating system in the computing device 1302a from an application in the computing device 1302a in substantially the same manner as described above with reference to FIG. 6.

The method 1500 then proceeds to block 1506 where a data mover device is identified in a data mover selection table with a highest priority for transferring data between the memory locations. In an embodiment of block 306 in which the fabric manager system 1306 orchestrates data mover device selection, the fabric manager engine 1404 in the fabric manager system 1306/1400 may operate to perform data mover selection operations that include the accessing of the global data mover selection table 1700 stored in the fabric manager database 1306 and the selection of a data mover device for performing the data transfer operations determined at block 1504.

With reference to the data mover selection table 1700 discussed above with reference to FIG. 17, at block 1506 the fabric manager engine 1404 in the fabric manager system 1306/1400 may use the memory locations/addresses identified in the data transfer request to identify a data mover device for performing the data transfer operations. For example, if the source memory location/address falls in the range included in the computing device 1302a and the destination memory location/address falls in the range included in the computing device 1302a, the fabric manager engine 1404 in the fabric manager system 1306/1400 may identify the data mover device A for performing the data transfer operations (i.e., because the data mover device A is identified/prioritized over the data mover device B for performing data transfers between those memory locations/addresses). In another example, if the source memory location/address falls in the range included in the computing device 1302b and the destination memory location/address falls in the range included in the computing device 1302a, the fabric manager engine 1404 in the fabric manager system 1306/1400 may identify the data mover device E for performing the data transfer operations (i.e., because the data mover device E is prioritized over the data mover device F for performing data transfers between those memory locations/addresses).

In another example, if the source memory location/address falls in the range included in the free memory pool and the destination memory location/address falls in the range included in the computing device 1302a, the fabric manager engine 1404 in the fabric manager system 1306/1400 may identify the data mover device G for performing the data transfer operations (i.e., because the data mover device 206c is the only data mover device identified/prioritized for performing data transfers between those memory locations/addresses). Similarly, if the source memory location/address falls in the range included in the free memory pool and the destination memory location/address falls in the range included in the computing device 1302b, the fabric manager engine 1404 in the fabric manager system 1306/1400 may identify the data mover device G for performing the data transfer operations (i.e., because the data mover device 206c is prioritized over the data mover device D for performing data transfers between those memory locations/addresses).

As such, one of skill in the art in possession of the present disclosure will appreciate how the global data mover selection table 1700 allows the fabric manager system 1306 to select, for any data transfer request that provides for the transfer of data between memory locations in the networked system 1300, a data mover device that is configured to perform the most efficient data transfer between those memory locations (e.g., based on that data mover device having the highest “affinity”/smallest “distance” relative to one or more of those memory locations, and/or on other factors that would be apparent to one of skill in the art in possession of the present disclosure.) However, while a specific global data mover selection table has been described as being utilized to select a data mover device for a data transfer operation based on particular data transfer efficiency characteristics, one of skill in the art in possession of the present disclosure will recognize that the selection of a data mover device for performing a data transfer operation in other manners and/or based on other data mover device selection characteristics will fall within the scope of the present disclosure as well.

In an embodiment of block 1506 in which the computing device 1302a orchestrates data mover device selection, the computing device 1302a may operate to perform data mover selection operations that include the accessing of its local data mover selection table that was updated as discussed above by the fabric manager system 1300, and the selection of a data mover device for performing the data transfer operations determined at block 1504 in substantially the same manner as discussed above with reference to block 306 of the method 300 (but with the local data mover selection table that has been updated with network-connected memory subsystems and data mover devices accessible to the computing device 1302a).

The method 1500 then proceeds to decision block 1508 where it is determined whether the identified data mover device exceeds a data transfer operation threshold. In embodiments in which the fabric manager system 1306 orchestrates data mover device selection, at decision block 1508 the fabric manager engine 1404 in the fabric manager system 1306/1400 may operate to determine whether the data mover device selected at block 1506 is currently operating such that it exceeds a data transfer operation threshold. In embodiments in which the computing device 1302a orchestrates data mover device selection, at decision block 1508 the computing device 1302a may operate to determine whether the data mover device selected at block 1506 is currently operating such that it exceeds a data transfer operation threshold. As will be appreciated by one of skill in the art in possession of the present disclosure, any data mover device selected at block 1506 may already be performing one or more data transfer operations, and the data mover selection system of the present disclosure may define a data transfer operation threshold above which a data mover device should not be utilized to perform a requested data transfer operation (i.e., despite its selection/identification at block 1506). As such, for any data mover device selection/identification at block 1506, a check may be performed to determine the operating level of that data mover device in order to ensure that data mover device will not be overloaded if it performs the data transfer operations determined at block 1504.

If, at decision block 1508, it is determined that the identified data mover device exceeds the data transfer operation threshold, the method 1500 proceeds to block 1510 where the identified data mover device is ignored. In embodiments in which the fabric manager system 1306 orchestrates data mover device selection, at block 1510 and in response to determining that the identified data mover device exceeds the data transfer operation threshold, the fabric manager engine 1404 in the fabric manager system 1306/1400 may operate to ignore that data mover device and the method 1500 will return to block 1506. In embodiments in which the computing device 1302a orchestrates data mover device selection, at block 1510 and in response to determining that the identified data mover device exceeds the data transfer operation threshold, the computing device 1302a may operate to ignore that data mover device and the method 1500 will return to block 1506. As such, in the event a data mover device is selected at block 1506 and determined to exceed the data transfer operation threshold at block 1508 of a first iteration of the method 1500, that data mover device will be ignored at block 1510, and a different data mover device will be selected at block 1506 of second iteration of the method 1500. Thus, one of skill in the art in possession of the present disclosure will recognize how the method 1500 may loop through blocks 1506, 1508, and 1510 until a data mover device is selected/identified that does not exceed the data transfer operation threshold.

In a specific example, at block 1506 on a first iteration of the method 1500, the data mover device A may have been identified by the fabric manager engine 1404 in the fabric manager system 1306/1400 as having first priority for performing data transfers between a source memory location/address that is included in the memory location/address range in the computing device 1302a in this example and a destination memory location/address that is included in the memory location/address range in the computing device 1302a (i.e., due to the data mover device A being located in the computing device 1302a that provides the source of the data for the data transfer and, thus, having a higher “affinity”/smaller “distance” relative to source data transfers performed by other data mover devices in the networked system 1300). At decision block 1508, the fabric manager engine 1404 in the fabric manager system 1306/1400 may determine that the data mover device A exceeds the data transfer operation threshold and, in response, the fabric manager engine 1404 in the fabric manager system 1306/1400 will operate to ignore the data mover device A at block 1510. Subsequently, at block 1506 on a second iteration of the method 1500, the data mover device B will be identified by the fabric manager engine 1404 in the fabric manager system 1306/1400 as having second (and now highest) priority for performing those data transfers.

As such, the prioritization of the data mover devices in the global data mover selection table 1700 allows lower priority data mover devices to be selected over higher priority data mover devices in the event the higher priority data mover devices exceed the data transfer operation threshold. As will be appreciated by one of skill in the art in possession of the present disclosure, in some embodiments and in the event only a single data mover device is identified for performing data transfers between different memory location/address ranges (e.g., the data mover device G identified for performing data transfers between the source memory range and the destination memory range in the free memory pool in the global data mover selection table 1700), that data mover device may be selected/identified for performing the data transfer operations despite the fact that it exceeds the data transfer operation threshold. However, in other embodiments and in the event only a single data mover device is identified for performing data transfers between different memory location/address ranges, the fabric manager engine 1404 in the fabric manager system 1306/1400 may select and/or identify a different data mover device for performing the data transfer operations in the event the data mover device identified in the global data mover selection table 1700 exceeds the data transfer operation threshold.

In embodiments in which the computing device 1302a orchestrates data mover device selection, at block 1510 and in response to determining that the identified data mover device exceeds the data transfer operation threshold, the computing device 1302a may operate to ignore the selected data mover device in substantially the same manner as described above with reference to block 310 in the method 300. As such, one of skill in the art in possession of the present disclosure will recognize that the data transfer operation threshold may be used to prevent the overloading of data mover devices in a variety of manners that will fall within the scope of the present disclosure as well.

If at decision block 1508, it is determined that the identified data mover device does not exceed the data transfer operation threshold, the method 1500 proceeds to block 1512 where a data transfer instruction is transmitted to the identified data mover device. With reference to FIG. 19B, in an embodiment of block 1512 and in response to the selection/identification of the data mover device by the fabric manager system 1306 at block 1506, the fabric manager engine 1404 in the fabric manager system 1306/1400 may perform data transfer instruction operations 1902 to transfer a data transfer instruction to the data mover device identified at block 1506 (which is located in the computing device 1302a in this example.) Furthermore, in embodiments in which the computing device 1302a orchestrates data mover selection, at block 1512 and in response to the selection/identification of the data mover device at block 1506, the computing device 1302a may perform data transfer instruction operations 1902 to transfer a data transfer instruction to the data mover device identified at block 1506.

The method 1500 then proceeds to block 1514 where the identified data mover device transfers data between the memory locations. In an embodiment of block 1514, the data mover device may receive the data transfer instructions as part of the data transfer instruction operations discussed above (e.g., from the fabric manager system 1306 or the computing devices 1302a) and, in response, perform data transfer operations included in those data transfer instructions. For example, the data transfer instructions may instruct the data mover E to transfer data between a source memory location/address included in the memory location/address range provided by the computing device 1302a, and a destination memory location/address included in the memory location/address range provided by the free memory pool, and the data transfer operations provide for the transfer of data by the data mover E from a first data location in a memory subsystem (included in the memory location/address range provided by the computing device 1302a) to a second data location in a memory subsystem (included in the memory location/address range provided by the free memory pool). As will be appreciated by one of skill in the art in possession of the present disclosure, following the data transfer operations and in embodiments in which the fabric manager system 1306 orchestrates data mover selection, the data mover device E may provide a data transfer confirmation to the fabric manager system 1306, and the fabric manager system 1306 may provide a data transfer confirmation to the computing device 1302a. Furthermore, following the data transfer operations and in embodiments in which the computing device 1302a orchestrates data mover selection, the data mover device E may provide a data transfer confirmation to the computing device 1302a, and the computing device 1302a may provide a data transfer confirmation to the application that requested the data transfer.

Thus, systems and methods have been described that provide for the selection of a data mover device for use in performing a data transfer in a server computing device cluster and/or memory fabric environment. For example, the data mover selection system of the present disclosure includes a fabric manager system coupled to a plurality of computing devices that are coupled to a memory system, with the fabric manager system operating to receive respective local data mover selection information from each of the plurality of computing devices that identifies at least one data mover device accessible to that computing device, and using the respective local data mover selection information to generate global data mover selection information that includes each data mover device accessible to the plurality of computing devices. Subsequently, when the fabric manager system receives a first data transfer request from a first computing device that provides for the transfer of data from a first memory location in the memory system to a second memory location in the memory system, it uses the global data mover selection information to identify a first data mover device for performing the first data transfer operation based on the first data mover device having a higher priority relative to other data mover devices included in the global data mover selection information for performing data transfers from the first memory location in the memory system to the second memory location in the memory system. The fabric manager system may then transmit a first data transfer instruction to the first data mover device that is configured to cause the first data mover device to perform the first data transfer operation to transfer data from the first memory location in the memory system to the second memory location in the memory system. As such, data transfers may be performed by data mover devices that provide more efficient data transfers (relative to conventional “round robin” data mover device selections) based on their affinity to one or more of the memory locations involved in those data transfers.

Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. Accordingly, it is appropriate that the appended claims be construed broadly and in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A data mover selection system, comprising:

at least one network-connected memory subsystem that is included in a memory system;
a plurality of computing devices that each include a respective computing device memory subsystem that is included in the memory system; and
a fabric manager system that is coupled to the plurality of computing devices and the at least one network-connected memory subsystem, wherein the fabric manager system is configured to: receive, from each of the plurality of computing devices in response to initialization of that computing device, respective local data mover selection information that identifies at least one data mover device accessible to that computing device; generate, using the respective local data mover selection information received from each of the plurality of computing devices in response to initialization of those computing devices, initialization global data mover selection information that prioritizes data mover devices accessible to each of the plurality of computing devices such that data mover devices that provide relatively lower data transfer distances between any pair of memory subsystems included in the memory system are prioritized over data mover devices that provide relatively higher data transfer distances between that pair of memory subsystems included in the memory system; identify, during runtime of the plurality of computing devices following the initialization of the plurality of computing devices, a data mover device availability modification to the data mover devices that were identified as accessible to the plurality of computing devices in the respective local data mover selection information; generate, in response to identifying the data mover device availability modification, updated global data mover selection information that prioritizes data mover devices accessible to each of the plurality of computing devices following the data mover device availability modification such that data mover devices that provide relatively lower data transfer distances between any pair of memory subsystems included in the memory system following the data mover device availability modification are prioritized over data mover devices that provide relatively higher data transfer distances between that pair of memory subsystems included in the memory system following the data mover device availability modification; receive, from a first computing device included in the plurality of computing devices, a first data transfer request that provides for the transfer of data between a first pair of memory subsystems in the memory system; identify, using the updated global data mover selection information, a first data mover device for performing the first data transfer operation based on the first data mover device providing a first lowest data transfer distance between the first pair of memory subsystems in the memory system the relative to other data mover devices included in the updated global data mover selection information; and transmit, in response to identifying the first data mover device for performing the first data transfer operation, a first data transfer instruction to the first data mover device that is configured to cause the first data mover device to perform the first data transfer operation to transfer data between the first pair of memory subsystems in the memory system.

2. The system of claim 1, wherein the fabric manager system is configured to:

receive, from the first computing device, a second data transfer operation that provides for the transfer of data between a second pair of memory subsystems in the memory system;
identify, using the updated global data mover selection information, the first data mover device for performing the second data transfer operation based on the first data mover device providing a second lowest data transfer distance between the second pair of memory subsystems in the memory system relative to other data mover devices included in the updated global data mover selection information;
determine that the first data mover device is exceeding a data transfer operation threshold; and
transmit, in response to determining that the first data mover device is exceeding the data transfer operation threshold, a second data transfer instruction to a second data mover device that is configured to cause the second data mover device to perform the second data transfer operation to transfer data between the second pair of memory subsystems in the memory system.

3. The system of claim 1, wherein the data mover device availability modification includes a movement of a first data mover device that was identified as accessible to a first computing device in first local data mover selection information to a second computing device.

4. The system of claim 1, wherein the data mover device availability modification includes an addition of a first data mover device that was not identified as accessible to any of the plurality of computing devices in the respective local data mover selection information to a first computing device.

5. The system of claim 1, wherein the data mover device availability modification includes a removal of a first data mover device that was identified as accessible to a first computing devices in a first local data mover selection information from the first computing device.

6. The system of claim 1, wherein the first pair of memory subsystems in the memory system include a first network-connected memory subsystem included in the at least one network-connected memory subsystem, and wherein the fabric manager system is configured to:

configure the first network-connected memory subsystem such that the first network-connected memory subsystem is accessible to the first computing device; and
configure the first data mover device such that the first data mover device is accessible to the first computing device.

7. An Information Handling System (IHS), comprising:

a processing system that is configured to execute instructions to provide a fabric manager engine that is configured to: receive, from each of a plurality of computing devices that each include a respective computing device memory subsystem that is include in a memory subsystem with at least one network-connected memory subsystem and in response to the initialization of that computing device, respective local data mover selection information that identifies at least one data mover device accessible to that computing device; generate, using the respective local data mover selection information received from each of the plurality of computing devices in response to the initialization of those computing devices, initialization global data mover selection information that prioritizes data mover devices accessible to the plurality of computing devices such that data mover devices that provide relatively lower data transfer distances between any pair of memory subsystems included in the memory system are prioritized over data mover devices that provide relatively higher data transfer distances between that pair of memory subsystems included in the memory system; identify, during runtime of the plurality of computing devices following the initialization of the plurality of computing devices, a data mover device availability modification to the data mover devices that were identified as accessible to the plurality of computing devices in the respective local data mover selection information; generate, in response to identifying the data mover device availability modification, updated global data mover selection information that prioritizes data mover devices accessible to each of the plurality of computing devices following the data mover device availability modification such that data mover devices that provide relatively lower data transfer distances between any pair of memory subsystems included in the memory system following the data mover device availability modification are prioritized over data mover devices that provide relatively higher data transfer distances between that pair of memory subsystems included in the memory system following the data mover device availability modification; receive, from a first computing device included in the plurality of computing devices, a first data transfer request that provides for the transfer of data between a first pair of memory subsystems in the memory system; identify, using the updated global data mover selection information, a first data mover device for performing the first data transfer operation based on the first data mover device providing a first lowest data transfer distance between the first pair of memory subsystems in the memory system relative to other data mover devices included in the updated global data mover selection information; and transmit, in response to identifying the first data mover device for performing the first data transfer operation, a first data transfer instruction to the first data mover device that is configured to cause the first data mover device to perform the first data transfer operation to transfer data between the first pair of memory location subsystems in the memory system.

8. The IHS of claim 7, wherein the fabric manager engine is configured to:

receive, from the first computing device, a second data transfer operation that provides for the transfer of data between a second pair of memory subsystems in the memory system;
identify, using the updated global data mover selection information, the first data mover device for performing the second data transfer operation based on the first data mover device providing a second lowest data transfer distance between the second pair of memory location subsystems in the memory system relative to other data mover devices included in the updated global data mover selection information;
determine that the first data mover device is exceeding a data transfer operation threshold; and
transmit, in response to determining that the first data mover device is exceeding the data transfer operation threshold, a second data transfer instruction to a second data mover device that is configured to cause the second data mover device to perform the second data transfer operation to transfer data between the second pair of memory subsystems in the memory system.

9. The IHS of claim 7, wherein the data mover device availability modification includes a movement of a first data mover device that was identified as accessible to a first computing device in first local data mover selection information to a second computing device.

10. The IHS of claim 7, wherein the data mover device availability modification includes an addition of a first data mover device that was not identified as accessible to any of the plurality of computing devices in the respective local data mover selection information to a first computing device.

11. The IHS of claim 7, wherein the data mover device availability modification includes a removal of a first data mover device that was identified as accessible to a first computing devices in a first local data mover selection information from the first computing device.

12. The IHS of claim 7, wherein the first pair of memory subsystems in the memory system include a first network-connected memory subsystem included in the at least one network-connected memory subsystem, and wherein the fabric manager system is configured to:

configure the first network-connected memory subsystem such that the first network-connected memory subsystem is accessible to the first computing device.

13. The IHS of claim 7, wherein the fabric manager system is configured to:

configure the first data mover device such that the first data mover device is accessible to the first computing device.

14. A method for selecting a data mover device comprising:

receiving, by a fabric manager system from each of a plurality of computing devices that each include a respective computing device memory subsystem that is include in a memory subsystem with at least one network-connected memory subsystem and in response to the initialization of that computing device, respective local data mover selection information that identifies at least one data mover device accessible to that computing device;
generating, by the fabric manager system using the respective local data mover selection information received from each of the plurality of computing devices in response to the initialization of those computing devices, initialization global data mover selection information that prioritizes data mover devices accessible to the plurality of computing devices such that data mover devices that provide relatively lower data transfer distances between any pair of memory subsystems included in the memory system are prioritized over data mover devices that provide relatively higher data transfer distances between that pair of memory subsystems included in the memory system;
identify, by the fabric manager system during runtime of the plurality of computing devices following the initialization of the plurality of computing devices, a data mover device availability modification to the data mover devices that were identified as accessible to the plurality of computing devices in the respective local data mover selection information;
generate, by the fabric manager system in response to identifying the data mover device availability modification, updated global data mover selection information that prioritizes data mover devices accessible to each of the plurality of computing devices following the data mover device availability modification such that data mover devices that provide relatively lower data transfer distances between any pair of memory subsystems included in the memory system following the data mover device availability modification are prioritized over data mover devices that provide relatively higher data transfer distances between that pair of memory subsystems included in the memory system following the data mover device availability modification;
receiving, by the fabric manager system from a first computing device included in the plurality of computing devices, a first data transfer request that provides for the transfer of data from between a first pair of memory subsystems in the memory system;
identifying, by the fabric manager system using the updated global data mover selection information, a first data mover device for performing the first data transfer operation based on the first data mover device providing a first lowest data transfer distance between the first pair of memory subsystems in the memory system relative to other data mover devices included in the updated global data mover selection information; and
transmitting, by the fabric manager system in response to identifying the first data mover device for performing the first data transfer operation, a first data transfer instruction to the first data mover device that is configured to cause the first data mover device to perform the first data transfer operation to transfer data between the first pair of memory subsystems in the memory system.

15. The method of claim 14, further comprising:

receiving, by the fabric manager system from the first computing device, a second data transfer operation that provides for the transfer of data between a second pair of memory subsystems in the memory system;
identifying, by the fabric manager system using the updated global data mover selection information, the first data mover device for performing the second data transfer operation based on the first data mover device providing a second lowest data transfer distance between the second pair of memory subsystems in the memory system relative to other data mover devices included in the updated global data mover selection information;
determining, by the fabric manager system, that the first data mover device is exceeding a data transfer operation threshold; and
transmitting, by the fabric manager system in response to determining that the first data mover device is exceeding the data transfer operation threshold, a second data transfer instruction to a second data mover device that is configured to cause the second data mover device to perform the second data transfer operation to transfer data between the second pair of memory subsystems in the memory system.

16. The method of claim 14, wherein the data mover device availability modification includes a movement of a first data mover device that was identified as accessible to a first computing device in first local data mover selection information to a second computing device.

17. The method of claim 14, wherein the data mover device availability modification includes an addition of a first data mover device that was not identified as accessible to any of the plurality of computing devices in the respective local data mover selection information to a first computing device.

18. The method of claim 14, wherein the data mover device availability modification includes a removal of a first data mover device that was identified as accessible to a first computing devices in a first local data mover selection information from the first computing device.

19. The method of claim 14, the first pair of memory subsystems in the memory system include a first network-connected memory subsystem included in the at least one network-connected memory subsystem, and wherein the method further comprises:

configuring, by the fabric manager system, the first network-connected memory subsystem such that the first network-connected memory subsystem is accessible to the first computing device.

20. The method of claim 14, further comprising:

configuring, by the fabric manager system, the first data mover device such that the first data mover device is accessible to the first computing device.
Referenced Cited
U.S. Patent Documents
6208622 March 27, 2001 Makrucki
7440842 October 21, 2008 Vorona
7769722 August 3, 2010 Bergant
7844756 November 30, 2010 Cannon
8706833 April 22, 2014 Bergant
9147024 September 29, 2015 Kathail
10061759 August 28, 2018 Chang
10265022 April 23, 2019 Rauhala
20050193245 September 1, 2005 Hayden
20070055702 March 8, 2007 Fridella
20070110046 May 17, 2007 Farrell
20070233953 October 4, 2007 Shen
20080075010 March 27, 2008 Song
20080104325 May 1, 2008 Narad
20080177958 July 24, 2008 Cannon
20080192724 August 14, 2008 Kondo
20090003499 January 1, 2009 Chiu
20090022061 January 22, 2009 Walley
20090058635 March 5, 2009 LaLonde
20100094532 April 15, 2010 Vorona
20110246997 October 6, 2011 Sitaraman
20110295804 December 1, 2011 Erofeev
20120083995 April 5, 2012 Vorona
20130073670 March 21, 2013 Das
20140222873 August 7, 2014 Nakadai
20140378160 December 25, 2014 Song
20150087333 March 26, 2015 Song
20160236869 August 18, 2016 Kimura
20170161508 June 8, 2017 Yoshida
20170374111 December 28, 2017 Hinohara
20180295070 October 11, 2018 Tonouchi
20200293039 September 17, 2020 Kim
20200293193 September 17, 2020 Littlefield
20200348874 November 5, 2020 Iyer
Patent History
Patent number: 11429302
Type: Grant
Filed: Jul 29, 2020
Date of Patent: Aug 30, 2022
Patent Publication Number: 20220035551
Assignee: Dell Products L.P. (Round Rock, TX)
Inventors: Shyam Iyer (Cedar Park, TX), Srinivas Giri Raju Gowda (Santa Clara, CA), Anh Dinh Luong (Austin, TX)
Primary Examiner: Sheng Jen Tsai
Application Number: 16/941,716
Classifications
Current U.S. Class: Determination Of Communication Parameters (370/252)
International Classification: G06F 12/00 (20060101); G06F 13/00 (20060101); G06F 3/06 (20060101);