Abstract: The technology disclosed herein pertains to a system and method for scaling storage using peer-to-peer NVMe communication, the method including selecting one of a plurality of NVMe devices as principal device to control communication with a host via a PCI switch, designating remainder of the plurality of NVMe devices as subordinate devices, and controlling the communication between the host and the subordinate devices using a PCI P2P DMA between the principal device and the subordinate devices.

Abstract: Systems and methods for moving data between a storage device and a processing element are disclosed. The storage device may include: a non-volatile storage medium; a first interface for communicating with a computing device; and a second interface for communicating with a processing element over a data communications network. The processing element may include a processing circuit that is configured to: receive a first request from the computing device via the first interface, where the first request identifies a first memory address associated with the first memory; retrieve first data from the non-volatile storage medium based on the first request; and transmit the first data to the processing element via the second interface for storing the first data in the first memory based on the first memory address.

Abstract: One or more embodiments of the present disclosure are directed to systems and methods for host and device cooperation. In some embodiments, a storage device comprises a first memory for storing data; a second memory configured with a cache coherent protocol; and a processing circuit coupled to the first memory and the second memory. The processing circuit is configured to: store, in the second memory, information for managing the storage device, wherein the information is shared with a computing device via the cache coherent protocol; receive a first value from the computing device, the first value associated with a first block of the first memory; and take an action with respect to the first block based on the first value, wherein the action is for managing the first memory.

Abstract: A distributed data storage system can be configured with a host connected to a device and a distribution module. The distribution module identifies a manufacturing origin of the device and diverts a system operation from an upstream component connected to the distribution module to the device in response to the identified manufacturing origin of the device. The manufacturer installed operating parameters of the device are then used to complete the system operation.

Abstract: A distributed data storage system may connect a host to a device and a dedication module. The dedication module can identify a manufacturing origin of the device before dedicating a type of host-generated data access request to the device. The type of host-generated data access request is chosen by the dedication module in response to an operating capability correlated to the device by the dedication module based on the identified manufacturing origin. At least one host-generated request of the dedicated type is then completed with the device.

Abstract: Apparatus and method for executing hidden computational functions in a distributed data processing environment. In some embodiments, a trust boundary includes a target device such as a storage device, and a source device such as a client device in a computer network. A storage device processor executes a hidden command function (HCF) routine to accumulate HCF output data in a local cache responsive to an HCF command received from the client device over a data interface. The processor further establishes a smaller retention boundary within the trust boundary that includes the storage device and excludes the client device. The HCF output data are stored locally in a non-volatile memory (NVM) of the storage device while not transferring any portion of the HCF output data outside the retention boundary, including to the client device. The HCF routine can update a block-chain ledger or take some other form to provide data security.

Abstract: A data storage system employing distributed memories can have at least one host connected to a plurality of data storage devices via a network controller. One or more performance bottlenecks through the network controller may be identified with a performance module. A peer group consisting of at least two of the plurality of data storage devices is created with the performance module in response to the identified performance bottleneck so that a task can be assigned by the performance module to the peer group. The task may be chosen to mitigate the performance bottleneck by avoiding involvement of the network controller in the task.

Abstract: A distributed data storage system can be configured with a host connected to a first device and an initialization module that performs a default initialization procedure on the first device in response to detecting the first device has an unknown manufacturing origin. Conducting the default initialization procedure may allow the first device to service data access requests from the host. In response to the connection of a second device to the host and initialization module, a manufacturing origin of the second device is identified with the initialization module before the initialization module customizes the default initialization procedure to a custom procedure that is executed to allow the second device to satisfy a background operation prescribed by the initialization module.

Abstract: A data storage system may connect a host to a device and a hardware module that are utilized to satisfy at least one host-generated data access request to the device. A portion of the device is set as dormant by the hardware module prior to identifying an origin of the device with the hardware module during the satisfaction of the at least one host-generated data access request. In response to the identified origin, the previously dormant portion of the device is activated and subsequently utilized to execute a task assigned by the hardware module.

Abstract: A data storage system may connect a host to a device and a hardware module that are utilized to satisfy at least one host-generated data access request to the device. A portion of the device is set as dormant by the hardware module prior to identifying an origin of the device with the hardware module during the satisfaction of the at least one host-generated data access request. In response to the identified origin, the previously dormant portion of the device is activated and subsequently utilized to execute a task assigned by the hardware module.

Abstract: Apparatus and method for executing hidden computational functions in a distributed data processing environment. In some embodiments, a trust boundary includes a target device such as a storage device, and a source device such as a client device in a computer network. A storage device processor executes a hidden command function (HCF) routine to accumulate HCF output data in a local cache responsive to an HCF command received from the client device over a data interface. The processor further establishes a smaller retention boundary within the trust boundary that includes the storage device and excludes the client device. The HCF output data are stored locally in a non-volatile memory (NVM) of the storage device while not transferring any portion of the HCF output data outside the retention boundary, including to the client device. The HCF routine can update a block-chain ledger or take some other form to provide data security.

Abstract: Method and apparatus for offloading upstream processing tasks to peer groups of downstream data storage devices. A peer control circuit forms a peer group of storage devices in response to a detected processing bottleneck associated with a network controller. One of the storage devices in the peer group is designated as a primary device, and is responsible for interface communications, for subdividing the processing task for execution by secondary devices in the peer group, and coordinating overall execution. The peer group and the processing task are selected to avoid or minimize the processing bottleneck at the network controller level while maintaining ongoing data transfer performance at the storage device level. A list of available device resources and capabilities may be maintained by the peer control circuit. Offloaded tasks can include data rebuilds, cryptographic functions, new device authentication operations, and the like. Multiple overlapping peer groups can be formed as needed.

Abstract: A distributed data storage system can be configured with a host connected to a first device and an initialization module that performs a default initialization procedure on the first device in response to detecting the first device has an unknown manufacturing origin. Conducting the default initialization procedure may allow the first device to service data access requests from the host. In response to the connection of a second device to the host and initialization module, a manufacturing origin of the second device is identified with the initialization module before the initialization module customizes the default initialization procedure to a custom procedure that is executed to allow the second device to satisfy a background operation prescribed by the initialization module.

Abstract: A distributed data storage system may connect a host to a device and a dedication module. The dedication module can identify a manufacturing origin of the device before dedicating a type of host-generated data access request to the device. The type of host-generated data access request is chosen by the dedication module in response to an operating capability correlated to the device by the dedication module based on the identified manufacturing origin. At least one host-generated request of the dedicated type is then completed with the device.

Abstract: A distributed data storage system can be configured with a host connected to a device and a distribution module. The distribution module identifies a manufacturing origin of the device and diverts a system operation from an upstream component connected to the distribution module to the device in response to the identified manufacturing origin of the device. The manufacturer installed operating parameters of the device are then used to complete the system operation.

Abstract: Apparatus and method for maintaining real-time coherency between a local cache of a target device and a client cache of a source device during execution of a distributed computational function. In some embodiments, a source device, such as a host computer, is coupled via a network interface to a target device, such as a data storage device. A storage compute function (SCF) command is transferred from the source device to the target device. A local cache of the target device accumulates output data during the execution of an associated SCF over an execution time interval. Real-time coherency is maintained between the contents of the local cache and a client cache of the source device, so that the client cache retains continuously updated copies of the contents of the local cache during execution of the SCF. The coherency can be carried out on a time-based granularity or an operational granularity.

Abstract: Methods, apparatuses, and computer-readable media for streaming arbitrarily large amounts of data through computational storage programs of a computational storage device. A computational storage device comprises a storage media, a computational storage processor, and a controller. A firmware of the controller comprises a plurality of streaming drivers, each associated with a data source or data destination of the storage device. The firmware further comprises a buffer abstraction layer operable to read data from a data source through an associated ingress streaming driver of the plurality of streaming drivers to provide a source data stream for a computational storage program executing on the computational storage processor. The buffer abstraction layer is further operable to receive a destination data stream from the computational storage program and write data to a data destination through an associated egress streaming driver of the plurality of streaming drivers.

Abstract: A system includes a storage device and a computational storage processor. The storage device includes media. The computational storage processor is configured to, after issuance of a single command from a host device, receive data corresponding to the command, process the data as the data is received using a filter program and provide results data from the processed data.

Abstract: A system includes a storage device and a computational storage processor. The storage device includes media. The computational storage processor is configured to, after issuance of a single command from a host device, receive data corresponding to the command, process the data as the data is received using a filter program and provide results data from the processed data.

Abstract: A multi-function NVMe subsystem includes a plurality of primary controllers, and a plurality of queue resources. The multi-function NVMe subsystem also includes a plurality of policies with each different policy of the plurality of policies differently dictating how the plurality of queue resources is divided amongst different primary controllers of the plurality of primary controllers.