Abstract: A classifier component receives an instruction to determine whether any data fields in a datastore are sensitive data fields in which sensitive data is stored. The classifier component analyzes the set of data fields and determines that a data field is a sensitive data field. The classifier component causes information that classifies the data field as a sensitive data field to a data catalog without sending content of any data field in the set of data fields to the data catalog. A data security component subsequently accesses a query made to the datastore, the query including a data field name that identifies the data field. The data security component determines, based on the data catalog, that the query requested content from a sensitive data field, and stores, by the data security component, information that the query requested the content from the sensitive data field.

Abstract: Techniques and mechanisms described herein provide for global deduplication in a cloud-based storage system. According to various embodiments, a global segment reference map can be created for data segments when a data segment has not been previously added to the global segment reference map. For each data segment not added to the global segment reference map, those data segments can be deleted from a cloud storage location.

Abstract: Techniques and mechanisms described herein provide for verification of data across cloud-based and on-premises data storage systems. According to various embodiments, a backup client implemented on a first compute node can store a data file in a backup data repository. A data guardianship can store first data file state information describing the data file in a key-value store accessible via the internet. A data verification instance can analyze the backup data repository to verify that the data file is stored intact in the backup data repository.

Abstract: Techniques and mechanisms are provided for migrating data blocks around a cluster during node addition and node deletion. Migration requires no downtime, as a newly added node is immediately operational while the data blocks are being moved. Blockmap files and deduplication dictionaries need not be updated.

Abstract: Techniques and mechanisms described herein provide for facilitating communications between one or more client machines and one or more cloud storage providers. According to various embodiments, a virtual machine may communicate with one or more client machines via a standard data storage protocol. The virtual machine may also communicate with one or more cloud storage providers via the internet. The virtual machine may then perform operations such as receiving data from a client machine and storing it to the cloud, retrieving data from the cloud and providing it to the client machine, querying data stored in the cloud, reporting on and verifying data stored in the cloud, and transferring and/or transforming data stored in the cloud.

Abstract: Techniques and mechanisms described herein provide for verifying data across cloud-based and on-premises data storage systems. According to various embodiments, data can be received by a gateway from a client machine and stored in a file data repository accessible via the internet. The stored data can have a common master namespace. Indications of updates to the master namespace can be received. Updated namespace data and metadata can be transmitted to the gateway.

Abstract: In general, certain embodiments of the present disclosure provide techniques or mechanisms for creating efficient and consistent replication of file overwrites. According to various embodiments, a file map with a sequence file number is created for a data file on a source. The file map includes one or more nodes that each include a sequence node number. When an entry in the file map is changed, the sequence file number is incremented, and the corresponding node is updated by, among other things, setting the sequence node number to equal the current sequence file number. All other nodes in the file map that point to the updated node are similarly updated. When replication occurs, only the updated nodes in the file map are replicated and any unreplicated non-updated nodes are maintained.

Abstract: Techniques and mechanisms described herein facilitate the strengthening of a data fingerprint. According to various embodiments, a data stream may be parsed into a plurality of data chunks. A hash function may be applied to each of the data chunks to determine a respective data chunk hash value. A respective data chunk fingerprint for each of the plurality of data chunks may be transmitted to a remote server. Each data chunk fingerprint may include the respective data chunk hash value and one or more respective intermediate hash values.

Abstract: Techniques and mechanisms described herein facilitate the replication of data between storage nodes. According to various embodiments, a request to provide a data chunk to a target storage node may be received at a source data storage node. A reference data chunk may be identified based on fingerprint information associated with the requested data chunk. The reference data chunk may be stored on the target storage node. The reference data chunk and the requested data chunk may each include a first data portion. Data chunk reconstruction information may be transmitted from the source data storage node to the target data storage node. The data chunk reconstruction information may identify the reference data chunk. The data chunk reconstruction information may include data difference information for constructing the requested data chunk at the target data storage node based on the reference data chunk.

Abstract: Techniques and mechanisms described herein facilitate the encryption of content using content-based encryption keys. According to various embodiments, data stream may include one or more data chunks. A client machine may apply a hash function to a data chunk to determine a fingerprint value. A cryptographic protocol shared with a remote server may be applied to the fingerprint value to determine a data chunk encryption key. The data chunk encryption key may be used to encrypt the data chunk, and the encrypted data chunk may be sent to the remote server for storage.

Abstract: Systems, methods, and devices are disclosed herein for data deduplication and storage. Methods include identifying first criteria associated with a data block included in a plurality of data blocks, the data block being identified based on a blockmap and an index, the blockmap characterizing a file stored in a deduplication repository, the index identifying data store identifiers, reference count data, and second criteria associated with the plurality of data blocks. The second criteria include the first criteria, and the first criteria characterize input/output (I/O) activity associated with the data block. The methods may also include identifying data stores based on the first criteria, the data stores being included in data storage tiers. The data stores are identified based on the I/O activity associated with the data block. The methods further include updating the first criteria stored in the index and associated with the data block.

Abstract: Techniques and mechanisms are provided for migrating data blocks around a cluster during node addition and node deletion. Migration requires no downtime, as a newly added node is immediately operational while the data blocks are being moved. Blockmap files and deduplication dictionaries need not be updated.

Abstract: Techniques and mechanisms described herein facilitate the replication of data between storage nodes. According to various embodiments, a request to provide a data chunk to a target storage node may be received at a source data storage node. A reference data chunk may be identified based on fingerprint information associated with the requested data chunk. The reference data chunk may be stored on the target storage node. The reference data chunk and the requested data chunk may each include a first data portion. Data chunk reconstruction information may be transmitted from the source data storage node to the target data storage node. The data chunk reconstruction information may identify the reference data chunk. The data chunk reconstruction information may include data difference information for constructing the requested data chunk at the target data storage node based on the reference data chunk.

Abstract: Techniques and mechanisms described herein facilitate the encryption of content using content-based encryption keys. According to various embodiments, data stream may include one or more data chunks. A client machine may apply a hash function to a data chunk to determine a fingerprint value. A cryptographic protocol shared with a remote server may be applied to the fingerprint value to determine a data chunk encryption key. The data chunk encryption key may be used to encrypt the data chunk, and the encrypted data chunk may be sent to the remote server for storage.

Abstract: Techniques and mechanisms are provided for migrating data blocks around a cluster during node addition and node deletion. Migration requires no downtime, as a newly added node is immediately operational while the data blocks are being moved. Blockmap files and deduplication dictionaries need not be updated.

Abstract: Systems, methods, and devices are disclosed herein for data deduplication and storage. Methods include identifying first criteria associated with a data block included in a plurality of data blocks, the data block being identified based on a blockmap and an index, the blockmap characterizing a file stored in a deduplication repository, the index identifying data store identifiers, reference count data, and second criteria associated with the plurality of data blocks. The second criteria include the first criteria, and the first criteria characterize input/output (I/O) activity associated with the data block. The methods may also include identifying data stores based on the first criteria, the data stores being included in data storage tiers. The data stores are identified based on the I/O activity associated with the data block. The methods further include updating the first criteria stored in the index and associated with the data block.

Abstract: Techniques and mechanisms described herein facilitate the encryption of content using content-based encryption keys. According to various embodiments, data stream may include one or more data chunks. A client machine may apply a hash function to a data chunk to determine a fingerprint value. A cryptographic protocol shared with a remote server may be applied to the fingerprint value to determine a data chunk encryption key. The data chunk encryption key may be used to encrypt the data chunk, and the encrypted data chunk may be sent to the remote server for storage.

Abstract: A system provides file aware block level deduplication in a system having multiple clients connected to a storage subsystem over a network such as an Internet Protocol (IP) network. The system includes client components and storage subsystem components. Client components include a walker that traverses the namespace looking for files that meet the criteria for optimization, a file system daemon that rehydrates the files, and a filter driver that watches all operations going to the file system. Storage subsystem components include an optimizer resident on the nodes of the storage subsystem. The optimizer can use idle processor cycles to perform optimization. Sub-file compression can be performed at the storage subsystem.

Abstract: Techniques and mechanisms are provided to allow for selective optimization, including deduplication and/or compression, of portions of files and data blocks. Data access is monitored to generate a heat index for identifying sections of files and volumes that are frequently and infrequently accessed. These frequently used portions may be left non-optimized to reduce or eliminate optimization I/O overhead. Infrequently accessed portions can be more aggressively optimized.

Abstract: In general, certain embodiments of the present disclosure provide techniques or mechanisms for creating efficient and consistent replication of file overwrites. According to various embodiments, a file map with a sequence file number is created for a data file on a source. The file map includes one or more nodes that each include a sequence node number. When an entry in the file map is changed, the sequence file number is incremented, and the corresponding node is updated by, among other things, setting the sequence node number to equal the current sequence file number. All other nodes in the file map that point to the updated node are similarly updated. When replication occurs, only the updated nodes in the file map are replicated and any unreplicated non-updated nodes are maintained.