Abstract: Embodiments of a distributed virtual array data storage system and method are disclosed. Storage nodes made up of relatively unsophisticated disks with associated processors are scalable to store very large amounts of data. The storage nodes communicate with servers directly over a network through, for example, an Ethernet connection. Control of the storage nodes and access to the storage nodes is handled entirely on the server side of the system by distributed virtual array (DVA) software running on the server side and employing a particular protocol over the standard network connection. In an embodiment, server-side virtual machine (VM) hosts host application VMs that are associated with vDisks. The DVA software distributes data for the vDisk over the storage nodes. In the case of failure of one or more of the storage nodes, the DVA software reconstructs the data, for example by reading redundant data from surviving storage nodes.

Abstract: A system for storing files comprises a processor and a memory. The processor is configured to break a file into one or more segments; store the one or more segments in a first storage unit; and add metadata to the first storage unit so that the file can be accessed independent of a second storage unit, wherein a single namespace enables access for files stored in the first storage unit and the second storage unit. The memory is coupled to the processor and configured to provide the processor with instructions.

Abstract: A data storage system has a plurality of hosts that request writes of data to a relatively high-performance storage device (RHPSD) such as NVRAM, which may be one of a plurality of RHPSDs in a plurality of storage nodes. A storage management system receives the write requests and writes received data to the RHPSD. According to an allocation policy, the storage management system indicates to at least one of the hosts that it should limit its use of RHPSD and it then discards data upon indication from that host to do so. Before being discarded from the RHPSD, the data may be written to a persistent storage device.

Abstract: A system for storing data includes a performance storage unit, a performance storage transfer manager, a segment storage system, and a performance segment storage unit. The performance storage unit is for storing a data stream or a data block in. The data stream or the data block includes one or more data items. The performance storage transfer manager manages a transfer of the one or more data items to be automatically stored in the segment storage system over a network. The segment storage system is for storing a stored data item of the one or more data items as a set of segments. The performance segment storage unit is for storing the set of segments in the event that the stored data item has been stored using the segment storage system.

Abstract: A system for processing data includes a data storage device and a processor. The data storage device stores a set of data. The processor is configured to divide the set of data in the data storage system into a set of segments; compute a set of fingerprints, wherein the set of fingerprints comprises a fingerprint for each segment of the set of segments; store the set of fingerprints in a new snapshot; identify a second set of fingerprints in the new snapshot that are not already in a fingerprint index; cause a second set of segments associated with the second set of fingerprints to be stored in a backup data storage system; and cause the second set of fingerprints to be added to the fingerprint index.

Abstract: An operating state of each of a plurality of storage units of a storage system is periodically monitored, including a storage capacity, a throughput, and overlap of clients associated with the storage units. In response to a request to redistribute data from a first of the storage units to another storage unit, a cost factor for each of remaining storage units to relocate the data of the first storage unit to each of the remaining storage units is determined. A cost factor of each of the remaining storage units is determined based on at least one of the storage capacity, the throughput, or the overlap of clients of the storage unit. A second of the storage units having a lowest cost factor amongst the remaining storage units is selected. At least a portion of the data of the first storage unit is migrated to the second storage unit.

Abstract: A system for storing data comprises a performance storage unit and a performance segment storage unit. The system further comprises a determiner. The determiner determines whether a requested data is stored in the performance storage unit. The determiner determines whether the requested data is stored in the performance segment storage unit in the event that the requested data is not stored in the performance storage unit.

Abstract: Transmitting or storing subsegments is disclosed. A data stream or a data block is received and broken into a plurality of segments. For at least one segment, the segment is broken into a plurality of subsegments. A previously stored or transmitted segment similar to the at least one segment is identified. A fingerprint is computed for at least one subsegment. And, using the fingerprint for the at least one subsegment, determining whether the at least one subsegment is identical to a subsegment of the previously stored or transmitted segment without directly comparing the content of the at least one subsegment with the content of the subsegment of the previously stored or transmitted segment.

Abstract: A distributed storage system has at least one host and at least one storage node. Hosts write data to and read data from storage devices within the storage nodes. A host upgrade manager (HUM) is installed in at least one of the hosts; a secondary upgrade manager (SUM) is installed in at least one of the nodes; and a primary upgrade manager (PUM) is installed within the distributed storage system itself, for example, in one of the nodes or hosts. Host and node storage system software is upgraded by transferring, via the respective HUM and SUM, upgraded versions of the software from the PUM to at least one host or node detected to be running an outdated version. The host and node storage system software components comprise different, cooperating portions of overall storage management software.

Abstract: Techniques for performing data encryption on data to be stored within a storage system are provided. A client application executing on a host machine may generate a data storage write request to write data to a storage system. A host-side module, executing on the host machine receives the write request. The host-side module is configured to generate one or more fingerprints for the data corresponding to the write request, where the one or more fingerprints are unique identifiers used to identify data blocks that make up the data. The host-side module generates encrypted data by encrypting the data blocks using an encryption technique. The encrypted data is then sent to a storage node within the storage system. Deduplication may be performed on the encrypted data using the one or more generated fingerprints.

Abstract: In a Distributed Virtual Array data storage system, a storage pool receives, from at least one writing entity within a host, a request to write a container of data that is identified by a unique key that is independent of physical storage identifiers and that comprises a set of data chunks. The key indicates a plurality of storage devices in which to store the data chunks, which are accordingly stored in those storage devices. Different combinations, each comprising a layout map, of storage devices are represented in a layout table. Information in the container key is used to select which layout map to use for the chunks of each container. An error-correction chunk is preferably also stored along with the chunks of each container so as to enable data container reconstruction in case of storage device failure.

Abstract: A distributed storage system has at least one host and at least one storage node. Hosts write data to and read data from storage devices within the storage nodes. A host upgrade manager (HUM) is installed in at least one of the hosts; a secondary upgrade manager (SUM) is installed in at least one of the nodes; and a primary upgrade manager (PUM) is installed within the distributed storage system itself, for example, in one of the nodes or hosts. Host and node storage system software is upgraded by transferring, via the respective HUM and SUM, upgraded versions of the software from the PUM to at least one host or node detected to be running an outdated version. The host and node storage system software components comprise different, cooperating portions of overall storage management software.

Abstract: Data in a storage system is deduplicated after receiving from at least one writing entity requests for a plurality of write operations for a corresponding plurality of data blocks in a storage object. The received blocks are buffered and sorted in order and a sequence of clumps is created from the buffered blocks, where each clump comprises a grouping of at least one of the sorted, buffered blocks. A boundary is determined between at least one pair of clumps based at least in part on the content of at least one of the buffered blocks, and it is then determined whether at least one of the clumps is a duplicate of a previously stored clump.

Abstract: A method of managing duplicate segments from a segmented file storage system is disclosed. The method comprises indexing a segment according to a key for the segments wherein the index includes an identification of a first data location where the segment is stored and identifying a duplicate segment having the same key that is stored in a second location. The method further comprises determining that the duplicate segment is an undesired duplicate segment and eliminating the undesired duplicate segment.

Abstract: A data storage system has a plurality of hosts that request writes of data to a relatively high-performance storage device (RHPSD) such as NVRAM, which may be one of a plurality of RHPSDs in a plurality of storage nodes. A storage management system receives the write requests and writes received data to the RHPSD. According to an allocation policy, the storage management system indicates to at least one of the hosts that it should limit its use of RHPSD and it then discards data upon indication from that host to do so. Before being discarded from the RHPSD, the data may be written to a persistent storage device.

Abstract: A system for storing data comprises a performance storage system for storing one or more data items. A data item of the one or more data items comprises a data file or a data block. The system further comprises a segment storage system for storing a snapshot of a stored data item of the one or more data items in the performance storage system. The taking of the snapshot of the stored data item enables recall of the stored data item as stored at a time of the snapshot. At least one newly stored segment is stored as a reference to a previously stored segment.

Abstract: A system for processing data includes a data storage device and a processor. The data storage device stores a set of data. The processor is configured to divide the set of data in the data storage system into a set of segments; compute a set of fingerprints, wherein the set of fingerprints comprises a fingerprint for each segment of the set of segments; store the set of fingerprints in a new snapshot; identify a second set of fingerprints in the new snapshot that are not already in a fingerprint index; cause a second set of segments associated with the second set of fingerprints to be stored in a backup data storage system; and cause the second set of fingerprints to be added to the fingerprint index.

Abstract: A system and method are disclosed for providing efficient data storage. A plurality of data segments is received in a data stream. The system determines whether a data segment has been stored previously in a low latency memory. In the event that the data segment is determined to have been stored previously, an identifier for the previously stored data segment is returned.

Abstract: A cost function is determined for assigning first deduplicating storage units of a first storage system for replication onto second deduplicating storage units of a second storage system. One or more of the first storage units in the first storage system are assigned to one or more of the second storage units in the second storage system based on a minimized cost resulting from the cost function.

Abstract: In a Distributed Virtual Array data storage system, data chunks making up data containers are stored in a key order in storage devices according to layouts specified in a layout data structure. When any of the storage devices becomes inaccessible, the storage devices iteratively return, in storage order, their stored chunks. Chunks belonging to the same container are identified and, if that container had a chunk in the inaccessible storage device, are used to reconstruct the inaccessible chunk. The chunks of the container are then stored according to an updated layout. The keys are independent of physical storage locations.