Abstract: A system hosts a file system representing a replica of file systems hosted on a plurality of nodes, each node storing data on a first storage medium. The system stores data on a second storage medium that is slower than the first storage medium. The system stores on the second storage medium a file received from one of the nodes when the file remains un-accessed by the nodes for a predetermined period of time. The first storage medium on the plurality of nodes stores a pointer to the file and not the data content of the file stored on the second storage medium. The system provides the file from the second storage medium to the first storage medium on one of the nodes while providing the plurality of nodes read and write access to the file stored on the second storage medium using the pointer on the first storage medium.

Abstract: A system hosts a file system representing a replica of file systems hosted on a plurality of nodes, each node storing data on a first storage medium. The system stores data on a second storage medium that is slower than the first storage medium. The system stores on the second storage medium a file received from one of the nodes when the file remains un-accessed by the nodes for a predetermined period of time. The first storage medium on the plurality of nodes stores a pointer to the file and not the data content of the file stored on the second storage medium. The system provides the file from the second storage medium to the first storage medium on one of the nodes while providing the plurality of nodes read and write access to the file stored on the second storage medium using the pointer on the first storage medium.

Abstract: A system hosts a file system representing a replica of file systems hosted on a plurality of nodes, each node storing data on a first storage medium. The system stores data on a second storage medium that is slower than the first storage medium. The system stores on the second storage medium a file received from one of the nodes when the file remains un-accessed by the nodes for a predetermined period of time. The first storage medium on the plurality of nodes stores a pointer to the file and not the data content of the file stored on the second storage medium. The system provides the file from the second storage medium to the first storage medium on one of the nodes while providing the plurality of nodes read and write access to the file stored on the second storage medium using the pointer on the first storage medium.

Abstract: A system hosts a file system representing a replica of file systems hosted on a plurality of nodes, each node storing data on a first storage medium. The system stores data on a second storage medium that is slower than the first storage medium. The system stores on the second storage medium a file received from one of the nodes when the file remains un-accessed by the nodes for a predetermined period of time. The first storage medium on the plurality of nodes stores a pointer to the file and not the data content of the file stored on the second storage medium. The system provides the file from the second storage medium to the first storage medium on one of the nodes while providing the plurality of nodes read and write access to the file stored on the second storage medium using the pointer on the first storage medium.

Abstract: A system includes determination of a resource stored in a cloud storage service, identification of a first cloud service associated with the resource, determination of a first pre-backup callback and a first post-backup callback associated with the first cloud service, transmission of the first pre-backup callback to the first cloud service, reception of a return of the first pre-backup callback from the first cloud service, transmission, in response to receipt of the return, of an instruction to the cloud storage service to backup the resource, and transmission of the first post-backup callback to the first cloud service.

Abstract: A system includes determination of a resource stored in a cloud storage service, identification of a first cloud service associated with the resource, determination of a first pre-backup callback and a first post-backup callback associated with the first cloud service, transmission of the first pre-backup callback to the first cloud service, reception of a return of the first pre-backup callback from the first cloud service, transmission, in response to receipt of the return, of an instruction to the cloud storage service to backup the resource, and transmission of the first post-backup callback to the first cloud service.

Abstract: Data streams may be stored in a chunk store in the form of stream maps and data chunks. Data chunks corresponding to a data stream may be stored in a chunk container, and a stream map corresponding to the data stream may point to the data chunks in the chunk container. Multiple stream maps may be stored in a stream container, and may point to the data chunks in the chunk container in a manner that duplicate data chunks are not present. Techniques are provided herein for localizing the storage of related data chunks in such chunk containers, for locating data chunks stored in chunk containers, for storing data streams in chunk stores in localized manners that enhance locality and decrease defragmentation, and for reorganizing stored data streams in chunks stores.

Abstract: Techniques are described herein that are capable of heterogeneously optimizing a file. Heterogeneous optimization involves optimizing regions of a file non-uniformly. For example, the regions of the file may be optimized to different extents. In accordance with this example, a different optimization technique may be used to optimize each region or subset of the regions. In one aspect, optimization designations are assigned to respective regions of a file based on access patterns that are associated with the respective regions. The file may be a database file, a virtualized storage file, or other suitable type of file. Each optimization designation indicates an extent to which the respective region is to be optimized. Each region may be optimized to the extent that is indicated by the respective optimization designation that is assigned to that region.

Abstract: Techniques are described herein that are capable of heterogeneously optimizing a file. Heterogeneous optimization involves optimizing regions of a file non-uniformly. For example, the regions of the file may be optimized to different extents. In accordance with this example, a different optimization technique may be used to optimize each region or subset of the regions. In one aspect, optimization designations are assigned to respective regions of a file based on access patterns that are associated with the respective regions. The file may be a database file, a virtualized storage file, or other suitable type of file. Each optimization designation indicates an extent to which the respective region is to be optimized. Each region may be optimized to the extent that is indicated by the respective optimization designation that is assigned to that region.

Abstract: The subject disclosure is directed towards partially recalling file ranges of deduplicated files based on tracking dirty (write modified) ranges (user writes) in a way that eliminates or minimizes reading and writing already-optimized adjacent data. The granularity of the ranges does not depend on any file-system granularity for tracking ranges. In one aspect, lazy flushing of tracking data that preserves data-integrity and crash-consistency is provided. In one aspect, also described is supporting granular partial recall on an open file while a data deduplication system is optimizing that file.

Abstract: A system hosts a file system representing a replica of file systems hosted on a plurality of nodes, each node storing data on a first storage medium. The system stores data on a second storage medium that is slower than the first storage medium. The system stores on the second storage medium a file received from one of the nodes when the file remains un-accessed by the nodes for a predetermined period of time. The first storage medium on the plurality of nodes stores a pointer to the file and not the data content of the file stored on the second storage medium. The system provides the file from the second storage medium to the first storage medium on one of the nodes while providing the plurality of nodes read and write access to the file stored on the second storage medium using the pointer on the first storage medium.

Abstract: Techniques are described herein that are capable of optimizing (i.e., deduplicating) data in a virtualization environment. For example, optimization designations (a.k.a. deduplication designations) may be assigned to respective regions of a virtualized storage file. A virtualized storage file is a file that is configured to be mounted as a disk or a volume to provide a file system interface for accessing hosted files. In accordance with this example, each optimization designation indicates an extent to which the respective region is to be optimized (i.e., deduplicated). In another example, a virtualized storage file is mounted to provide a virtual disk that includes hosted files. In accordance with this example, optimization designations are assigned to the respective hosted files. In further accordance with this example, each optimization designation indicates an extent to which the respective hosted file is to be optimized.

Abstract: Disclosed herein is a system and method that can be used to synchronize files from a hosted file system where the client endpoint devices have direct access to the files stored on a hosted file store/system. When the client synchronizes with the files stored on the hosted file store the synchronization process identifies the files that need to be synchronized with the client and prepares batches of files to be synced. During the process of synchronization the files are not locked from other user's being able to use the files. If the file is modified by another endpoint during the file synchronization process the file will not be synchronized with the client device during the particular synchronization process. In this way synchronization becomes possible with the hosted service without impacting the performance of other devices that use and access the data in the hosted file store.

Abstract: Techniques for backup and restore of optimized data streams are described. A chunk store includes each optimized data stream as a plurality of chunks including at least one data chunk and corresponding optimized stream metadata. The chunk store includes data chunks in a deduplicated manner. Optimized data streams stored in the chunk store are identified for backup. At least a portion of the chunk store is stored in backup storage according to an optimized backup technique, an un-optimized backup technique, an item level backup technique, or a data chunk identifier backup technique. Optimized data streams stored in the backup storage may be restored. A file reconstructor includes a callback module that generates calls to a restore application to request optimized stream metadata and any referenced data chunks from the backup storage. The file reconstructor reconstructs the data streams from the referenced data chunks.

Abstract: The subject disclosure is directed towards a data deduplication technology in which a hash index service's index is partitioned into subspace indexes, with less than the entire hash index service's index cached to save memory. The subspace index is accessed to determine whether a data chunk already exists or needs to be indexed and stored. The index may be divided into subspaces based on criteria associated with the data to index, such as file type, data type, time of last usage, and so on. Also described is subspace reconciliation, in which duplicate entries in subspaces are detected so as to remove entries and chunks from the deduplication system. Subspace reconciliation may be performed at off-peak time, when more system resources are available, and may be interrupted if resources are needed. Subspaces to reconcile may be based on similarity, including via similarity of signatures that each compactly represents the subspace's hashes.

Abstract: File system-awareness of ghosting performed by one or more tiering engines allows a file system to receive and store metadata indicating an identifier of the tiering engine ghosting the file extents and a storage location of the ghosted file extents for later use by the tiering engine. The file system is able to receive and process requests to read and write to a file having ghosted extents.

Abstract: Described is caching classification-related metadata for a file in an alternate data stream of that file. When a file is classified (e.g., for data management), the classification properties are cached in association with the file, along with classification-related metadata that indicates the state of the file at the time of caching. The classification-related metadata in the alternate data stream is then useable in determining whether the classification properties are valid and up-to-date when next accessed, or whether the file needs to be reclassified. If the properties are valid and up-to-date, they may be used without requiring the computationally costly steps of reclassification. Also described is using more than one alternate data stream for the cache, and extending the classification-related metadata through a defined extension mechanism.

Abstract: Disclosed herein is a system and method that can be used to synchronize files from a hosted file system where the client endpoint devices have direct access to the files stored on a hosted file store/system. When the client synchronizes with the files stored on the hosted file store the synchronization process identifies the files that need to be synchronized with the client and prepares batches of files to be synced. During the process of synchronization the files are not locked from other user's being able to use the files. If the file is modified by another endpoint during the file synchronization process the file will not be synchronized with the client device during the particular synchronization process. In this way synchronization becomes possible with the hosted service without impacting the performance of other devices that use and access the data in the hosted file store.

Abstract: A host server hosting one or more virtual machines can back up host volumes and the one or more virtual machines installed thereon in an application-consistent manner. In one implementation, a host-level requestor instructs a host-level writer to identify which virtual machines qualify for application-consistent backups. The host-level requestor then instructs the host-level writer to initiate virtual machine backups through guest-level requesters in each appropriately-configured virtual machine, wherein the virtual machines create application-consistent backups within the virtual machine volumes. The host-level requester then initiates snapshots of the server volumes on the host-level. The virtual machine-level snapshots can thus be retrieved from within the host-level snapshots of the server volumes.

Abstract: A host server hosting one or more virtual machines can back up host volumes and the one or more virtual machines installed thereon in an application-consistent manner. In one implementation, a host-level requestor instructs a host-level writer to identify which virtual machines qualify for application-consistent backups. The host-level requestor then instructs the host-level writer to initiate virtual machine backups through guest-level requesters in each appropriately-configured virtual machine, wherein the virtual machines create application-consistent backups within the virtual machine volumes. The host-level requester then initiates snapshots of the server volumes on the host-level. The virtual machine-level snapshots can thus be retrieved from within the host-level snapshots of the server volumes.