Abstract: A system and method for efficiently storing and accessing large volumes of metadata persistent on Non-Volatile Memory (NVM) storage systems is provided. The system applies log-structured, Copy-on-Write (CoW) B+ tree methods, and supports a core-affine data and resource partitioning approaches on the system's architecture and platform with a high-degree of parallelism within the CPU, NVMe storage, and networking devices. The subject system and method efficiently indexes both in-core (DRAM resident) and out-of-core (NVM resident) metadata, supports a variety of data access patterns, supports CoW features and provides verifiable data safety and integrity capabilities. The present system minimizes latencies over all aspects of the metadata management and access path by leveraging core-affine resource partitioning with runtime environment providing lightweight user-level threads with low-latency context switching that execute within the exclusive context of a dedicated CPU core, and partitioned resources.

Abstract: A system and method for efficiently storing and accessing large volumes of metadata persistent on Non-Volatile Memory (NVM) storage systems is provided. The system applies log-structured, Copy-on-Write (CoW) B+ tree methods, and supports a core-affine data and resource partitioning approaches on the system's architecture and platform with a high-degree of parallelism within the CPU, NVMe storage, and networking devices. The subject system and method efficiently indexes both in-core (DRAM resident) and out-of-core (NVM resident) metadata, supports a variety of data access patterns, supports CoW features and provides verifiable data safety and integrity capabilities. The present system minimizes latencies over all aspects of the metadata management and access path by leveraging core-affine resource partitioning with runtime environment providing lightweight user-level threads with low-latency context switching that execute within the exclusive context of a dedicated CPU core, and partitioned resources.

Abstract: Systems and methods for dynamic context based data protection and distribution are described herein. In the method data replication in a multi-zone storage system automatically changes based on historical storage trends and/or external information obtained from third party information sources, such as feeds, websites, news alerts, emergency alerts, weather alerts and databases. The methods include monitoring third party information sources and evaluating a correlation between event data from the third party information sources and key words that are user provided and/or system derived. A correlation between the event data and data storage history is evaluated. A storage policy is derived based on the results of the correlation evaluation(s). The derived storage policy and corresponding key words are stored in a database. Data is stored in storage arrays according to storage policies corresponding to the key words identified in the data.

Abstract: A data storage system allowing for ingest of data when certain storage is unavailable is described herein. The storage system includes zones that are independent and autonomous from each other. The zones include nodes that are independent and autonomous. The nodes include storage devices. When data is to be stored in the data storage system according to a specified storage policy and the specified storage policy cannot be achieved, the data is stored according to a fallback storage policy. This allows a client to be able to continue executing without having to wait for a storage anomaly to be corrected or pass. After the data is stored according to a fallback storage policy, the data is at a later time stored according to the specified storage policy.

Abstract: The present invention is directed to a distributed data migration and storage system and method configured for fully parallel writing and retrieval of data objects to (and from) the Web Object Scaler (WOS) object storage system with highly efficient accessibility and using a simplified set of filing commands. The data objects migrate between data generating entities and the WOS object storage through an intermediate Burst Buffer tier (supported by the Infinite Memory Engine) without the utilization of a serial process substantially simultaneously through parallel channels. The metadata for each object is captured with appropriate pointers for efficient finding of the object in the WOS storage system. The WOS nodes return the stored objects' ID (OIDs) corresponding to the objects storage address which are used for the object access.

Abstract: A data storage system having mutable objects incorporating time is described herein. According to the systems and methods described herein, a data item may be partitioned into parts (data objects) and stored as an index object. As the object storage system provides immutable objects, when a new version of a data item needs to be stored, only those parts (data objects) of the data item that changed need be saved rather than the entire data item. The systems and methods described herein allow for efficient storage, access and manipulation of mutable data items using an underlying immutable object system.

Abstract: Data storage systems and methods for storing data are described herein. An intermediate storage system is coupled with a High Performance Computing (HPC) system. A plurality of burst engines of the intermediate storage system are directly connected to compute nodes of the HPC. The burst engines are operative to receive file level data from the compute nodes at data transfer rates that at least match to the burst mode output rate of the compute node. The burst engines store the data to high bandwidth storage devices of the intermediate storage system at a data rate that at least matches the burst output rate of the compute node. During a compute phase of the compute node, data is replayed from the intermediate storage devices to a primary storage system at a lower data transfer rate.

Abstract: A data migrating system and method are provided in which a Burst Buffer Network Aggregator (BBNA) process is configured either on the File Servers or on the File System's dedicated I/O nodes to coalesce data fragments stored in participating Burst Buffer nodes under the direction of a primary BB node appointed by a data generating entity prior to transfer of the full data stripe into the File System. The “write” request in the form of a full data stripe is distributed into a plurality of data fragments among participating BB nodes along with corresponding metadata.

Abstract: A data migration system supports a low-latency and reduced overhead data storage protocol for data storage sharing in a non-collision fashion which does not require inter-communication and permanent arbitration between data storage controllers to decide on the data placement/routing. The multiple data fragments of data sets are prevented from routing to the same storage devices by a multi-step selection protocol which selects (in a first phase of the selection routine) a healthy highest ranked drive enclosure, and further selects (in a second phase of the selection routine) a healthy highest-ranked data storage controller residing in the selected drive enclosure, for routing data fragments to different storage pools assigned to the selected data storage devices for exclusive “writing” and data modification. The selection protocol also contemplates various failure scenarios in a data placement collision free manner.

Abstract: Systems and methods for automated firmware update with rollback are described herein. The systems include a plurality of storage zones, each storage zone including a plurality of storage nodes, each storage node including a plurality of storage media. The method includes monitoring storage system activity and parameters and maintaining a data storage system usage and parameter database containing system activity information. When a firmware update is available, data storage system activity is evaluated. Storage nodes needing the firmware update are identified. The firmware update is run on available storage nodes identified as needing the firmware update. The impact of the firmware update is evaluated and a rollback of the firmware update is initiated on all firmware updated storage nodes when parameter variations are significant and/or result in degraded performance.

Abstract: Systems and methods that combine a silicon storage volume with a hard disk drive (HDD) volume in a storage system that uses in band hinting to select the volume for storing actual data and meta data based on the demands of high performance computing are described. A storage system with an application processor, a storage processor, a silicon storage volume including a plurality of SSDs and an HDD with a much larger number of HDDs efficiently handles write requests from a high performance computer. A high performance computing device prepares internal meta data and actual data write requests by specifying in the internal write requests whether the data is actual data or meta data using an existing field. The storage processor receives the internal write requests and manages storage of the meta data in the silicon storage volume and actual data in the HDD volume.

Abstract: Systems and methods for storing data in a data storage system that allows dynamic context-based data protection and distribution are disclosed. The method includes receiving a storage request and evaluating whether a storage policy is specified in the storage request. When the storage request specifies a storage policy, the data is stored according to the specified policy. When the storage request does not specify a storage policy, meta data and/or the data item itself may be evaluated to identify a storage policy. When a storage policy has been identified, store the data is stored according to the identified policy. When a storage policy has not been specified and cannot be identified, an error is returned.

Abstract: The system and routine for data caching leverages the properties of Network-Attached Non-Volatile Memories (NANVMs) to provide virtualized secure node-local storage services to the network users with reduced data movement across the NANVMs. The caching routine reserves storage resources (storage partitions) on NANVM devices, migrates data required for the target application execution to the allocated storage partitions, and directs the network clients to dynamically “mount” to the storage partitions based on application data requirements. Only those clients and applications that present valid credentials and satisfactory computing capabilities can access the data in the specific storage partitions. Several clients can have an access to the same storage partitions without duplication or replicating the data. A Global Data Indexing sub-system supports the efficient operation of the subject system.

Abstract: A system and method for data migration between data generating entities and de-clustered RAID storage employs a data re-distribution approach with the purpose of eliminating drives thrashing and attain an even I/O activity distribution for Physical Disk Extents (PDEs) residing on Physical Disks. The system monitors the I/O activity map, identifies competing over-used PDEs residing on the same physical disk drive, and if the drive thrashing is detected, re-directs “write” or “read” requests from the over-used PDEs to a different physical disk or to underused PDEs on the same physical disk.

Abstract: Variable Redundancy Distributed (VRD) RAID controller in a data storage environment contains embedded RAID logic permitting to choose and compute a desired redundancy coding scheme from a plurality thereof pre-programmed and embedded in a Compute Engine in the VRD RAID controller. “Write” or “Read” requests which are received from data generating entities, contain information identifying a type of the redundancy coding scheme of interest. The controller decodes the request, and automatically applies the desired computation to the incoming data without burdening the CPU with the computational activity. The variable redundancy computational ability of the subject systems provides an extremely versatile and flexible tool for RAID operations.

Abstract: Systems and methods are provided for expanding the available memory of a storage controller. The systems and methods utilize a PCIe memory controller connected to the backend interface of the storage controller. Memory of the PCIe memory controller is memory mapped to controller memory of the storage controller. The PCIe connection allows the storage controller to access the memory of the PCIe memory controller with latencies similar to that of the controller memory.

Abstract: Data storage systems and methods for storing data in computing nodes of a super computer or compute cluster are described herein. The super computer storage may be coupled with a primary storage system. In addition to a CPU and memory, non-volatile memory is included with the computing nodes as local storage. The super computer includes a plurality of computing groups, each including a plurality of computing nodes. There is one burst buffer fabric per group and one input/output node per group. When data bursts occur, data may be stored by a first computing node on the local storage of a second computing node in the computing group through the burst buffer fabric without interrupting the CPU in the second computing node. Further, the local storage of other computing nodes may be used to store redundant copies of data from a first computing node to make the super computer data resilient.

Abstract: Asynchronous namespace maintenance in a distributed replicated data storage system is disclosed. An access device/program serving as a front end to the distributed replicated data storage system updates a batch of updated meta data about stored data items when data items are stored in the distributed replicated data storage system. When the elapsed time since the last batch of data item meta data was stored exceeds a first threshold value or the current batch size exceeds a second threshold value, the access device/program stores the current batch of updated meta data as an object in the distributed replicated data storage system, receiving a batch object identifier for the stored batch of updated meta data, and distributes the batch object identifier to other access devices and/or access programs which retrieve the batch of updated meta data and update their namespaces.

Abstract: A failure resilient distributed replicated data storage system is described herein. The storage system includes zones that are independent, and autonomous from each other. The zones include nodes that are independent and autonomous. The nodes include storage devices. When a data item is stored, it is partitioned into a plurality of data objects and a plurality of parity objects calculated. Reassembly instructions are created for the data item. The data objects and parity objects are spread across all nodes and zones in the storage system. Reassembly instructions are also spread across the zones. When a read request is received, the data item is prepared from the lowest latency nodes according to the reassembly instructions. This provides for data resiliency while keeping the amount of storage space required relatively low.

Abstract: Method and system for data migration between data generating entities and data storage devices protected by de-clustered RAID algorithm are enhanced by dynamically controlling the I/O activity towards the data storage devices (NVM devices) based on their remaining lifespan (health) with the goal to prevent multiple devices selected for writing a parity stripe information from simultaneous failures. This feature is rendered by polling the remaining health of NVM devices in the RAID pool, computing a weighted lifespan for each NVM device, comparing the latter to an average of all NVM devices in the pool, and adjusting the I/O activity towards the NVM device of interest accordingly. If the weighted lifespan exceeds the average lifespan in the pool, the allowed I/O activity is increased, and if the weighted lifespan is below the average for the pool, then the device in question is sent less “writes”.