Abstract: Provided herein are systems, apparatuses and methods (i.e., utilities) that allow for increasing the bandwidth of a processing complex of a storage controller. The utilities utilize a symmetrical approach where PCIe switches overcome limitations of prior art processor complexes. The symmetrical approach provided by the disclosed utilities as incorporated into a storage controller provides equal access from any host path/channel to any drive path/channel (i.e., storage element). More specifically, a first or a first set of PCIe switches connect front-end PCIe host bus adaptors, which are connectable to host systems, to front-end data paths of a plurality of PCIe memory controllers. A second or second set of PCIe switches connect backend host bus adapters, which are connectable to storage elements, to back-end data paths of the plurality of PCIe memory controllers. The symmetrical architecture provides at least twice the bandwidth of prior art architectures.

Abstract: Distributed Compute Engine (DCE) memory controller in a data storage environment contains embedded logic and arithmetic functionality for Boolean logical and arithmetic operations. “Write” or “Read” requests which are received from data generating entities, contain a Physical Address field identifying an address of a data block to be retrieved from the external memory, and a Control bits field identifying a type of computational operation to be performed. The DCE memory controller decodes the request, and applies the desired compute operation autonomically to the contents of an external memory and/or the incoming data without burdening the CPU with the computational activity.

Abstract: Asynchronous namespace maintenance in a distributed replicated data storage system is disclosed. An access device/program serving as a front end to the distributed replicated data storage system updates a batch of updated meta data about stored data items when data items are stored in the distributed replicated data storage system. When the elapsed time since the last batch of data item meta data was stored exceeds a first threshold value or the current batch size exceeds a second threshold value, the access device/program stores the current batch of updated meta data as an object in the distributed replicated data storage system, receiving a batch object identifier for the stored batch of updated meta data, and distributes the batch object identifier to other access devices and/or access programs which retrieve the batch of updated meta data and update their namespaces.

Abstract: System and methods for managing I/O write requests of host systems to physical storage. A storage subsystem includes a plurality of storage devices where each storage device is configured to provide data storage. A controller is connected to the plurality of storage devices for executing the I/O write requests from the host systems and is further connected to a plurality of solid state drives in a parity RAID configuration. Non-valid portions of cache lines stored in stripes of the parity RAID configured second level cache are filled with known default values. Upon receiving an I/O write request to write new data to a non-valid portion of a cache line existing in the second level cache, the new data is spliced into the cache line and a new parity value is calculated without reading the known default values saving a read and an XOR operation.

Abstract: A data storage system having mutable objects incorporating time is described herein. According to the systems and methods described herein, a data item may be partitioned into parts (data objects) and stored as an index object. As the object storage system provides immutable objects, when a new version of a data item needs to be stored, only those parts (data objects) of the data item that changed need be saved rather than the entire data item. The systems and methods described herein allow for efficient storage, access and manipulation of mutable data items using an underlying immutable object system.

Abstract: A searchable data storage system is described herein. The storage system includes zones that are independent, and autonomous from each other. The zones include nodes that are independent and autonomous. The nodes include storage devices. When a data item is stored, a local database is updated with information about the newly stored data item. When a search for a data item meeting certain metadata criteria is received, multiple concurrent searches are conducted across all storage devices in all nodes in all zones of the storage system. The configuration of the data storage system allows a parallel concurrent search at constituent storage devices to be performed quickly.

Abstract: The present data storage system employs a memory controller with embedded logic to selectively XOR incoming data with data written in the memory to generate XOR parity data. The memory controller automatically performs XOR operations on incoming data based upon the address range associated with the memory “write” request. The system provides data migration and parity generation in a simple and effective manner and attains reduction in cost and power consumption. The memory controller may be built on the basis of FPGAs, thus providing an economical and miniature system.

Abstract: Data storage systems and methods for storing data in computing nodes of a super computer or compute cluster are described herein. The super computer storage may be coupled with a primary storage system. In addition to a CPU and memory, non-volatile memory is included with the computing nodes as local storage. The super computer includes a plurality of computing groups, each including a plurality of computing nodes. There is one burst buffer fabric per group and one input/output node per group. When data bursts occur, data may be stored by a first computing node on the local storage of a second computing node in the computing group through the burst buffer fabric without interrupting the CPU in the second computing node. Further, the local storage of other computing nodes may be used to store redundant copies of data from a first computing node to make the super computer data resilient.

Abstract: Data storage systems and methods for storing data in computing nodes of a super computer or compute cluster are described herein. The super computer storage may be integrated with or coupled with a primary storage system. In addition to a CPU and memory, non-volatile memory is included with the computing nodes as local storage. A high speed interconnect remote direct memory access (HRI) unit is also included with each computing node. When data bursts occur, data may be stored by a first computing node on the local storage of a second computing node through the HRI units of the computing nodes, bypassing the CPU of the second computing node. Further, the local storage of other computing nodes may be used to store redundant copies of data from a first computing node to make the super computer data resilient while not interfering with the CPU of the other computing nodes.

Abstract: Data re-protection in a distributed replicated data storage system is disclosed. The method may be implemented on a server or controller. A method includes storing first data in a first zone and storing a replica of the first data in a second zone. The zones are at different, separate locations. When an actual or impending failure with the first data in the first zone is detected, the system automatically initiates transitioning to a copy of impacted data at the first zone obtained from the second zone. The transitioning includes creating a remote copy of the impacted data at the second zone within a local area network before transferring the copy to the first zone over a wide area network. The methods allow the system to return to a fully protected state faster than if the impacted data was transferred from the second zone to the first zone without making a copy at the second zone.

Abstract: An object based file system for storing and accessing objects is disclosed. The file system may be implemented as a method in hardware, firmware, software, or a combination thereof. The method may include receiving from an application program an object write request. A selected storage node on which to store the object may be selected, including identifying a least busy storage node and/or a least full storage node. The object and the object write request may be sent to the selected storage node. A write success message may be received from the selected storage node. The successful writing of the object may be reported to the application program. An object identifier and object data may be stored in a database.

Abstract: A resilient distributed replicated data storage system is described herein. The storage system includes zones that are independent, and autonomous from each other. The zones include nodes that are independent and autonomous. The nodes include storage devices. When a data item is stored, it is partitioned into a plurality of data objects and a plurality of parity objects are calculated. Reassembly instructions are created for the data item. The data objects, parity objects and reassembly instructions are spread across nodes and zones in the storage system according to a policy for the data item. When a zone is inaccessible, a virtual zone is created and used until the intended zone is available. When a read request is received, the data item is prepared from the lowest latency nodes according to the reassembly instructions, and a virtual zone is accessed in place of a real zone when the real zone is inaccessible.

Abstract: A data storage system having mutable objects incorporating time is described herein. According to the systems and methods described herein, a data item may be partitioned into parts (data objects) and stored as an index object. As the object storage system provides immutable objects, when a new version of a data item needs to be stored, only those parts (data objects) of the data item that changed need be saved rather than the entire data item. The systems and methods described herein allow for efficient storage, access and manipulation of mutable data items using an underlying immutable object system.

Abstract: System and methods for managing I/O write requests of host systems to physical storage. A storage subsystem includes a plurality of storage devices where each storage device is configured to provide data storage. At least a pair of redundant controllers is connected to the plurality of storage devices for executing the I/O write requests from the host systems. A received I/O write request is initially saved in a controller memory of one of the controllers and mirrored in controller memory of the other controller. In one embodiment, the I/O write request is transferred to a flash memory device for subsequent transfer to the storage devices. Once transferred to the flash memory device, the I/O write request may be flushed from the controller memories. The I/O write request may then be transferred to the storage devices from the flash memory device as a background operation.

Abstract: Asynchronous namespace maintenance in a distributed replicated data storage system is disclosed. An access device/program serving as a front end to the distributed replicated data storage system updates a batch of updated meta data about stored data items when data items are stored in the distributed replicated data storage system. When the elapsed time since the last batch of data item meta data was stored exceeds a first threshold value or the current batch size exceeds a second threshold value, the access device/program stores the current batch of updated meta data as an object in the distributed replicated data storage system, receiving a batch object identifier for the stored batch of updated meta data, and distributes the batch object identifier to other access devices and/or access programs which retrieve the batch of updated meta data and update their namespaces.

Abstract: Asymmetric distributed replicated data storage systems and methods are described herein. The storage system includes zones that are independent, and autonomous. The zones include nodes that are independent and autonomous. The nodes include storage devices. When a data item is stored, it is partitioned into a plurality of data objects and a plurality of parity objects using erasure coding. The data objects and parity objects are spread across all nodes and zones in the storage system asymmetrically such that a first zone includes all of the data objects and no parity objects while the remaining zones include subsets of the data objects and all of the parity objects. The systems and methods provide for data resiliency while keeping the amount of storage space required relatively low.

Abstract: A resilient distributed replicated data storage system is described herein. The storage system includes zones that are independent, and autonomous from each other. The zones include nodes that are independent and autonomous. The nodes include storage devices. When a data item is stored, it is partitioned into a plurality of data objects and a plurality of parity objects are calculated. Reassembly instructions are created for the data item. The data objects, parity objects and reassembly instructions are spread across nodes and zones in the storage system according to a policy for the data item. When a zone is inaccessible, a virtual zone is created and used until the intended zone is available. When a read request is received, the data item is prepared from the lowest latency nodes according to the reassembly instructions, and a virtual zone is accessed in place of a real zone when the real zone is inaccessible.

Abstract: A control processor manages the power budget in a drive enclosure and is within the drive enclosure which monitors in real time a redundantly configured power supply unit, drives, interposers, and temperature sensors, and determines the power settings for each drive to avoid overload and overheating in the system. The control processor dynamically adjusts the mode of operation as needed during operation through the SAS interposer. A localized monitoring and control mechanism eliminates the need for extraneous coordination of information across various entities that access the storage. Data tunneling takes place directly between the compute nodes and target drives through the SAS expander and interposer, and does not need buffering the pending IO requests in the DRAM.

Abstract: A system for data migration between high performance computing architectures and data storage disks includes an intermediate data migration handling system which has an intermediate data storage module coupled to the computer architecture to store data received, and a data controller module which includes data management software supporting the data transfer activity between the intermediate data storage module and the disk drives in an orderly manner independent of the random I/O activity of the computer architecture. RAID calculations are performed on the data prior to storage in the intermediate storage module, as well as when reading data from it for assuring data integrity, and carrying out reconstruction of corrupted data. The data transfer to the disk drives is actuated in sequence determined by the data management software based on minimization of seeking time, tier usage, predetermined time since the previous I/O cycle, or fullness of the intermediate data storage module.

Abstract: Data storage systems and methods for storing data are described herein. The storage system includes a first storage node is configured to issue a first delivery request to a first set of other storage nodes in the storage system, the first delivery request including a first at least one data operation for each of the first set of other storage nodes and issuing at least one other delivery request, while the first delivery request remains outstanding, the at least one other delivery request including a first commit request for each of the first set of other storage nodes. The first node causes the first at least one data operation to be made active within the storage system in response to receipt of a commit indicator along with a delivery acknowledgement regarding one of the at least one other delivery request.