Abstract: Apparatuses, systems, methods, and techniques to perform graphics processing unit (GPU) initiated networking (GIN). In at least one embodiment, generation of a work queue element (WQE) is offloaded from the GPU to separate hardware (e.g., an accelerator or processor in a network interface controller). In at least one embodiment, the GPU generates a GPU-friendly descriptor (GFD), instead of a WQE, that contains information to create a WQE and transfers the GFD to the separate hardware, which performs the creation of the WQE, processes a completion queue entry (CQE), and provides a notification to the GPU that processing the WQE has been completed.

Abstract: An apparatus and method for reducing the memory bandwidth of executing machine learning models. A computing system includes two or more processing nodes, each including at least one or more processors and a corresponding local memory. Switch circuitry communicates with at least the local memories and a system memory of the computing system. The switch includes multiple direct memory access (DMA) interfaces. Each of one or more processing nodes stores multiple embedding rows of embedding tables. A processor of the processing node identifies two or more embedding rows as source operands of a reduction operation. The switch executes memory access requests to retrieve data of the two or more embedding rows from the corresponding local memory, and generates a result by performing the reduction operation. The switch sends the result to the local memory.

Abstract: Systems and methods for atomic memory operations in a remote direct memory access network are disclosed. A system includes a network interface card (NIC) comprising a first memory and one or more processors coupled to the first memory. The one or more processors are to receive an atomic memory operation (AMO) remote procedure call (RPC) comprising a memory address and an AMO type. The one or more processors are further to retrieve a value corresponding to the memory address of the AMO RPC from a second memory. The one or more processors are further to perform an AMO corresponding to the AMO type on the value from the second memory to obtain a modified value. The one or more processors are further to store the modified value in the first memory.

Abstract: An apparatus and method for efficiently performing remote memory access requests among multiple processing nodes. In various implementations, a computing system has a first node and a second node. Each of these nodes has a corresponding virtual address space in the computing system, and each node assigns subdivisions of the virtual address spaces to multiple clients of the node. The nodes assign a subset of a first virtual address space of a first client in the first node to remote data stored in a second virtual address space of a second client in the second node. Remote presence check (RPC) circuits of the nodes assign the second virtual address space to a subset of a network physical address (NPA) space. The RPC circuits of the nodes use assignments to the NPA space to verify that address mappings in TLBs are still available prior to routing memory access requests.

Abstract: Embodiments herein describe generating packets that combine an atomic operation (e.g., an atomic fetch) with a data operation (e.g., a put). Previous remote atomics first transmit a packet to a remote node that provides a lock for the data. If the lock is granted, the node transmits another packet containing a data operation which can read or write data. However, the embodiments herein can use a relax range-based atomics where the packet uses a range to reserve space in a dataset (e.g., a buffer) at the destination node for the data operation.

Abstract: Non-blocking processing system are described. In accordance with the described techniques, a pending range store receives, at a start of a bulk memory operation, a pending memory range of the bulk memory operation. A logic unit includes at least one of check conflict logic or check address logic. The logic unit detects a conflicting memory access based on a target address of the pending memory range conflicting with a memory access request separate from the bulk memory operation. The logic unit performs at least a portion of the bulk memory operation associated with the target address before the memory access request is allowed to proceed.

Abstract: A device includes a plurality of processing elements (PEs). A symmetric memory is allocated in each of the plurality of PEs. The device includes a switch connected to the plurality of PEs. The switch is to: receive, from a first processing element (PE) of the plurality of PEs, a message that includes a buffer offset, compute, based on the buffer offset, a first memory address of a first buffer in a first symmetric memory of the first PE and a second memory address of a second buffer in a second symmetric memory of a second PE of the plurality of PEs, and initiate, based on the first memory address and the second memory address, a direct memory access operation to access the first buffer and the second buffer.

Abstract: Techniques are described for efficiently performing integrated matrix multiplication operations to compute an output matrix based on two input matrices. Data corresponding to a first input tile of a first input matrix is retrieved and stored in a local buffer. For each output tile of the output matrix, a non-blocking remote call is initiated to retrieve data corresponding to a next tile of the first input matrix into the local buffer and, concurrently with the processing of this remote call, iteratively computes the output tile using the data of the first input matrix stored in the local buffer and a unidimensional sequence of input tiles from the second input matrix. The output matrix is generated based on the iteratively computed output tiles.

Abstract: An apparatus and method for reducing the memory bandwidth of executing machine learning models. A computing system includes two or more processing nodes, each including at least one or more processors and a corresponding local memory. Switch circuitry communicates with at least the local memories and a system memory of the computing system. The switch includes multiple direct memory access (DMA) interfaces. Each of one or more processing nodes stores multiple embedding rows of embedding tables. A processor of the processing node identifies two or more embedding rows as source operands of a reduction operation. The switch executes memory access requests to retrieve data of the two or more embedding rows from the corresponding local memory, and generates a result by performing the reduction operation. The switch sends the result to the local memory.

Abstract: Non-blocking processing system are described. In accordance with the described techniques, a pending range store receives, at a start of a bulk memory operation, a pending memory range of the bulk memory operation. A logic unit includes at least one of check conflict logic or check address logic. The logic unit detects a conflicting memory access based on a target address of the pending memory range conflicting with a memory access request separate from the bulk memory operation. The logic unit performs at least a portion of the bulk memory operation associated with the target address before the memory access request is allowed to proceed.

Abstract: A device includes a plurality of processing elements (PEs). A symmetric memory is allocated in each of the plurality of PEs. The device includes a switch connected to the plurality of PEs. The switch is to: receive, from a first processing element (PE) of the plurality of PEs, a message that includes a buffer offset, compute, based on the buffer offset, a first memory address of a first buffer in a first symmetric memory of the first PE and a second memory address of a second buffer in a second symmetric memory of a second PE of the plurality of PEs, and initiate, based on the first memory address and the second memory address, a direct memory access operation to access the first buffer and the second buffer.

Abstract: A device includes a communication scheduler to generate schedule trees for scheduling data communication among a plurality of nodes configured to perform a collective operation using data contributed from the plurality of nodes. The device includes data reduction logic to: identify one or more skewed nodes among the plurality of nodes, perform, according to a first set of schedule trees, a first operation to generate partial results based on data contributed from non-skewed nodes, and perform, according to a second set of schedule trees, a second operation to generate final results based on the partial results and data contributed from the one or more skewed nodes.

Abstract: A framework disclosed herein extends a relaxed, scoped memory model to a system that includes nodes across a commodity network and maintains coherency across the system. A new scope, cluster scope, is defined, that allows for memory accesses at scopes less than cluster scope to operate on locally cached versions of remote data from across the commodity network without having to issue expensive network operations. Cluster scope operations generate network commands that are used to synchronize memory across the commodity network.

Abstract: A processing system includes a first processor couplable to a first memory and a second memory. In response to a page migration trigger for a page in the first memory, the first processor is configured to, responsive to the page being a read-only page storing code for execution, initiate migration of the page to a code cache portion of a second memory associated with a second processor and shared by multiple processes executing at the second processor, and to configure each process of a set of processes executing at the second processor to access and execute the code from the code cache portion.

Abstract: In accordance with described techniques for PE-centric all-to-all communication, a distributed computing system includes processing elements, such as graphics processing units, distributed in clusters. An all-to-all communication procedure is performed by the processing elements that are each configured to generate data packets in parallel for all-to-all data communication between the clusters. The all-to-all communication procedure includes a first stage of intra-cluster parallel data communication between respective processing elements of each of the clusters; a second stage of inter-cluster data exchange for all-to-all data communication between the clusters; and a third stage of intra-cluster data distribution to the respective processing elements of each of the clusters.

Abstract: A distributed cache network used for machine learning is provided which comprises a network fabric having file systems which store data and a plurality of processing devices, each comprising cache memory and a processor configured to execute a training of a machine learning model and selectively cache portions of the data based on a frequency with which the data is accessed by the processor. Each processing device stores metadata identifying portions of data which are cached in the cache memory and other portions of the data which are cached in other processing devices of the network. When requested data is not cached in another processing device, the portion of requested data is accessed from a network file system via a client to server channel and is accessed from another processing device via a client to client channel when the requested data is cached in the other processing device.

Abstract: An approach is provided for coalescing network commands in a GPU that implements a SIMT architecture. Compatible next network operations from different threads are coalesced into a single network command packet. This reduces the number of network command packets generated and issued by threads, thereby increasing efficiency, and improving throughput. The approach is applicable to any number of threads and any thread organization methodology, such as wavefronts, warps, etc.

Abstract: An electronic device includes one or more data producing nodes and a data consuming node. Each data producing node separately generates two or more portions of a respective block of data. Upon completing generating each portion of the two or more portions of the respective block of data, each data producing node communicates that portion of the respective block of data to the data consuming node. Upon receiving corresponding portions of the respective blocks of data from each of the one or more data producing nodes, the data consuming node performs operations for a model using the corresponding portions of the respective blocks of data.

Abstract: A framework disclosed herein extends a relaxed, scoped memory model to a system that includes nodes across a commodity network and maintains coherency across the system. A new scope, cluster scope, is defined, that allows for memory accesses at scopes less than cluster scope to operate on locally cached versions of remote data from across the commodity network without having to issue expensive network operations. Cluster scope operations generate network commands that are used to synchronize memory across the commodity network.

Abstract: A framework disclosed herein extends a relaxed, scoped memory model to a system that includes nodes across a commodity network and maintains coherency across the system. A new scope, cluster scope, is defined, that allows for memory accesses at scopes less than cluster scope to operate on locally cached versions of remote data from across the commodity network without having to issue expensive network operations. Cluster scope operations generate network commands that are used to synchronize memory across the commodity network.