Abstract: Mechanisms are provided for data transfer reduction in scale out architectures. When a compute node receives a write input/output (I/O) request for a data stream, the compute node separates the data stream into chunks and generates fingerprints for the individual chunks. Fingerprints are then sent to a scale out node and compared to fingerprints of chunks already maintained at the scale out node. Write data transfers are only made for chunks not already maintained at the scale out node. For a read I/O request for a data stream, fingerprints for chunks of the data stream are requested by the compute node from a scale out node. Fingerprints received are compared to fingerprints of chunks already maintained at the compute node and read data transfers are only made for chunks not already maintained at the compute node.

Abstract: In accordance with embodiments of the present disclosure, a storage system may include a storage cluster comprising a plurality of network attached storage nodes, one or more backup devices communicatively coupled to the storage cluster, and a cluster-wide data server executing on the plurality of network attached storage nodes and configured to manage communication of backup data between the plurality of network attached storage nodes and the one or more backup devices.

Abstract: Techniques and mechanisms are provided to allow for selective optimization, including deduplication and/or compression, of portions of files and data blocks. Data access is monitored to generate a heat index for identifying sections of files and volumes that are frequently and infrequently accessed. These frequently used portions may be left non-optimized to reduce or eliminate optimization I/O overhead. Infrequently accessed portions can be more aggressively optimized.

Abstract: Techniques and mechanisms are provided to allow for selective optimization, including deduplication and/or compression, of portions of files and data blocks. Data access is monitored to generate a heat index for identifying sections of files and volumes that are frequently and infrequently accessed. These frequently used portions may be left non-optimized to reduce or eliminate optimization I/O overhead. Infrequently accessed portions can be more aggressively optimized.

Abstract: Mechanisms are provided for data transfer reduction in scale out architectures. When a compute node receives a write input/output (I/O) request for a data stream, the compute node separates the data stream into chunks and generates fingerprints for the individual chunks. Fingerprints are then sent to a scale out node and compared to fingerprints of chunks already maintained at the scale out node. Write data transfers are only made for chunks not already maintained at the scale out node. For a read I/O request for a data stream, fingerprints for chunks of the data stream are requested by the compute node from a scale out node. Fingerprints received are compared to fingerprints of chunks already maintained at the compute node and read data transfers are only made for chunks not already maintained at the compute node.

Abstract: Techniques and mechanisms are provided for migrating data blocks around a cluster during node addition and node deletion. Migration requires no downtime, as a newly added node is immediately operational while the data blocks are being moved. Blockmap files and deduplication dictionaries need not be updated.

Abstract: Techniques and mechanisms provide a storage optimization manager. Data may be optimized and maintained on various nodes in a cluster. Particular nodes may be overburdened while other nodes remain relatively unused. Techniques are provided to efficiently optimize data onto nodes to enhance operational efficiency. Data access requests for optimized data are monitored and managed to allow for intelligent maintenance of optimized data.

Abstract: Techniques and mechanisms are provided to support live file optimization. Active I/O access to an optimization target is monitored during optimization. Active files need not be taken offline or made unavailable to an application during optimization and retain the ability to support file operations such as read, write, unlink, and truncate while an optimization engine performs deduplication and/or compression on active file ranges.

Abstract: Techniques and mechanisms are provided to allow for selective optimization, including deduplication and/or compression, of portions of files and data blocks. Data access is monitored to generate a heat index for identifying sections of files and volumes that are frequently and infrequently accessed. These frequently used portions may be left non-optimized to reduce or eliminate optimization I/O overhead. Infrequently accessed portions can be more aggressively optimized.