Abstract: Systems, apparatuses and methods may provide for technology that identifies an embedding table associated with a neural network. The neural network is associated with a plurality of compute nodes. The technology further identifies a number of entries of the embedding table, and determines whether to process gradients associated with the embedding table as dense gradients or sparse gradients based on the number of entries.

Abstract: A method of performing ring allreduce operations is disclosed. The method includes sending a chunk of a message in a receive buffer at a current index of a send buffer to a next node in a virtual ring of nodes, receiving a chunk of the message from a previous node in the virtual ring of nodes and store the chunk at the current index of the receive buffer, and reducing a chunk in a send buffer at a previous index of the receive buffer and a chunk in the receive buffer at a previous index of the receive buffer and storing a result at the previous index of the receive buffer. The method includes repeating the sending, receiving and storing, and reducing and storing steps until all chunks of the message are reduced, and sending reduced chunks to the next node and receive reduced chunks from the previous node.

Abstract: A method and system to support scheduling of memory store instructions across atomic regions in binary translation in a processing unit or processor. In one embodiment of the invention, the processing unit has a store buffer that allows store instructions to be issued in different order than the source binary program order but still retire in source binary program order. This facilitates a small atomic region that maps to each iteration of a source binary code and these atomic regions are joined together into a pipelined region. In one embodiment of the invention, the processing unit executes commit instruction(s) once every loop iteration instead of executing the commit instruction(s) once after the loop exit.

Abstract: A method and system to support scheduling of memory store instructions across atomic regions in binary translation in a processing unit or processor. In one embodiment of the invention, the processing unit has a store buffer that allows store instructions to be issued in different order than the source binary program order but still retire in source binary program order. This facilitates a small atomic region that maps to each iteration of a source binary code and these atomic regions are joined together into a pipelined region. In one embodiment of the invention, the processing unit executes commit instruction(s) once every loop iteration instead of executing the commit instruction(s) once after the loop exit.

Abstract: A technique includes performing multiple aligned accesses to a memory to retrieve data of a string misaligned with respect to boundaries of the memory by an offset. Based on the offset, a subset of the data is selected, and the subset is stored in a register.