Abstract: The subject disclosure is directed towards data deduplication (optimization) performed by phases/modules of a modular data deduplication pipeline. At each phase, the pipeline allows modules to be replaced, selected or extended, e.g., different algorithms can be used for chunking or compression based upon the type of data being processed. The pipeline facilitates secure data processing, batch processing, and parallel processing. The pipeline is tunable based upon feedback, e.g., by selecting modules to increase deduplication quality, performance and/or throughput. Also described is selecting, filtering, ranking, sorting and/or grouping the files to deduplicate, e.g., based upon properties and/or statistical properties of the files and/or a file dataset and/or internal or external feedback.

Abstract: A host server hosting one or more virtual machines can back up host volumes and the one or more virtual machines installed thereon in an application-consistent manner. In one implementation, a host-level requestor instructs a host-level writer to identify which virtual machines qualify for application-consistent backups. The host-level requestor then instructs the host-level writer to initiate virtual machine backups through guest-level requesters in each appropriately-configured virtual machine, wherein the virtual machines create application-consistent backups within the virtual machine volumes. The host-level requester then initiates snapshots of the server volumes on the host-level. The virtual machine-level snapshots can thus be retrieved from within the host-level snapshots of the server volumes.

Abstract: Techniques for backup and restore of optimized data streams are described. A chunk store includes each optimized data stream as a plurality of chunks including at least one data chunk and corresponding optimized stream metadata. The chunk store includes data chunks in a deduplicated manner. Optimized data streams stored in the chunk store are identified for backup. At least a portion of the chunk store is stored in backup storage according to an optimized backup technique, an un-optimized backup technique, an item level backup technique, or a data chunk identifier backup technique. Optimized data streams stored in the backup storage may be restored. A file reconstructor includes a callback module that generates calls to a restore application to request optimized stream metadata and any referenced data chunks from the backup storage. The file reconstructor reconstructs the data streams from the referenced data chunks.

Abstract: The subject disclosure is directed towards a data deduplication technology in which a hash index service's index is partitioned into subspace indexes, with less than the entire hash index service's index cached to save memory. The subspace index is accessed to determine whether a data chunk already exists or needs to be indexed and stored. The index may be divided into subspaces based on criteria associated with the data to index, such as file type, data type, time of last usage, and so on. Also described is subspace reconciliation, in which duplicate entries in subspaces are detected so as to remove entries and chunks from the deduplication system. Subspace reconciliation may be performed at off-peak time, when more system resources are available, and may be interrupted if resources are needed. Subspaces to reconcile may be based on similarity, including via similarity of signatures that each compactly represents the subspace's hashes.

Abstract: Described is the backup and/or restore of virtual disks In general, metadata is backed up for restoring a virtual disk. To restore the disk, a physical disk is created, with the virtual disk the created on a partition of the physical disk. Backup and restore is described for nested virtual disks, including for block level restore. Further described is backing up of critical virtual disks and their containers, and virtual disk backup with respect to basic disks and dynamic volumes.

Abstract: The subject disclosure is directed towards changing a file from a fully deduplicated state to a partially deduplicated state in which some of the file data is deduplicated in a chunk store, and some is recalled into the file, that is, in the file's storage volume. A partial recall mechanism such as in a file system filter tracks (e.g., via a bitmap in a file reparse point) whether file data is maintained in the chunk store or has been recalled to the file. Data is recalled from the chunk store as needed, and committed (e.g., flushed) to the file. Also described is efficiently returning the file to a fully deduplicated state by using the tracking information to determine which parts of the file are already deduplicated into the chunk store so as to avoid their further deduplication processing.

Abstract: The subject disclosure is directed towards data deduplication (optimization) performed by phases/modules of a modular data deduplication pipeline. At each phase, the pipeline allows modules to be replaced, selected or extended, e.g., different algorithms can be used for chunking or compression based upon the type of data being processed. The pipeline facilitates secure data processing, batch processing, and parallel processing. The pipeline is tunable based upon feedback, e.g., by selecting modules to increase deduplication quality, performance and/or throughput. Also described is selecting, filtering, ranking, sorting and/or grouping the files to deduplicate, e.g., based upon properties and/or statistical properties of the files and/or a file dataset and/or internal or external feedback.

Abstract: Techniques are described herein that are capable of optimizing (i.e., deduplicating) data in a virtualization environment. For example, optimization designations (a.k.a. deduplication designations) may be assigned to respective regions of a virtualized storage file. A virtualized storage file is a file that is configured to be mounted as a disk or a volume to provide a file system interface for accessing hosted files. In accordance with this example, each optimization designation indicates an extent to which the respective region is to be optimized (i.e., deduplicated). In another example, a virtualized storage file is mounted to provide a virtual disk that includes hosted files. In accordance with this example, optimization designations are assigned to the respective hosted files. In further accordance with this example, each optimization designation indicates an extent to which the respective hosted file is to be optimized.

Abstract: Techniques are described herein that are capable of heterogeneously optimizing a file. Heterogeneous optimization involves optimizing regions of a file non-uniformly. For example, the regions of the file may be optimized to different extents. In accordance with this example, a different optimization technique may be used to optimize each region or subset of the regions. In one aspect, optimization designations are assigned to respective regions of a file based on access patterns that are associated with the respective regions. The file may be a database file, a virtualized storage file, or other suitable type of file. Each optimization designation indicates an extent to which the respective region is to be optimized. Each region may be optimized to the extent that is indicated by the respective optimization designation that is assigned to that region.

Abstract: Data streams may be stored in a chunk store in the form of stream maps and data chunks. Data chunks corresponding to a data stream may be stored in a chunk container, and a stream map corresponding to the data stream may point to the data chunks in the chunk container. Multiple stream maps may be stored in a stream container, and may point to the data chunks in the chunk container in a manner that duplicate data chunks are not present. Techniques are provided herein for localizing the storage of related data chunks in such chunk containers, for locating data chunks stored in chunk containers, for storing data streams in chunk stores in localized manners that enhance locality and decrease defragmentation, and for reorganizing stored data streams in chunks stores.

Abstract: Described is the backup and/or restore of virtual disks In general, metadata is backed up for restoring a virtual disk. To restore the disk, a physical disk is created, with the virtual disk the created on a partition of the physical disk. Backup and restore is described for nested virtual disks, including for block level restore. Further described is backing up of critical virtual disks and their containers, and virtual disk backup with respect to basic disks and dynamic volumes.

Abstract: A method for concurrent data migration includes classifying files to be migrated into plural jobs, selecting media to which to migrate each job, and using plural drives concurrently to write the jobs to the media. The selection of a medium is performed in a way that prevents the number of writeable media from exceeding the number of available drives, unless no allocated medium has sufficient space to store any files in a migration job. A medium is preferentially selected that has already been allocated for writing, has space to store at least one file in the job, is not in use for another job, and can be robotically mounted on a drive. If such a medium does not exist, then the set of available media is canvassed to locate an alternative medium.

Abstract: Described is a technology, such as implemented in an operating system security system, by which a resource's metadata (e.g., including data properties) is evaluated against an audit rule or audit rules associated with that resource (e.g., object). The audit rule may be associated with all such resources corresponding to a resource manager, and/or by a resource-specific audit rule. When a resource is accessed, each audit rule is processed against the metadata to determine whether to generate an audit event for that rule. The audit rule may be in the form of one or more conditional expressions. Audit events may be maintained and queried to obtain audit information for various usage scenarios.

Abstract: Aspects of the subject matter described herein relate to transactions. In aspects, a consistency requester such as a shadow copy service requests to have a consistent view of a set of resources. In response, transactions that are in a problematic state are allowed to exit the problematic state while new and existing transactions that are not in the problematic state are allowed to perform any work except work that takes them into the problematic state. After no transactions are in the problematic state, a consistent view of the set of resources is available. This view may be used by the consistency requester as desired. Afterwards, the consistency requester may indicate that a consistent view is no longer needed. Transactions are then allowed to enter the problematic state. Transactions may also be allowed to enter the problematic state if a timeout elapses.

Abstract: Described is a technology by which access to a resource is determined by evaluating a resource label of the resource against a user claim of an access request, according to policy decoupled from the resource. The resource may be a file, and the resource label may be obtained by classifying the file into classification properties, such that a change to the file may change its resource label, thereby changing which users have access to the file. The resource label-based access evaluation may be logically combined with a conventional ACL-based access evaluation to determine whether to grant or deny access to the resource.

Abstract: Described is caching classification-related metadata for a file in an alternate data stream of that file. When a file is classified (e.g., for data management), the classification properties are cached in association with the file, along with classification-related metadata that indicates the state of the file at the time of caching. The classification-related metadata in the alternate data stream is then useable in determining whether the classification properties are valid and up-to-date when next accessed, or whether the file needs to be reclassified. If the properties are valid and up-to-date, they may be used without requiring the computationally costly steps of reclassification. Also described is using more than one alternate data stream for the cache, and extending the classification-related metadata through a defined extension mechanism.

Abstract: Described is a technology in which data items (e.g., files) are processed through an extensible data processing pipeline, including a classification pipeline, to facilitate management of the data items based upon their classifications. A discovery module locates data items to process. An independent classification pipeline obtains metadata (properties) associated with each discovered data item, and one or more classifiers classify the data item based on the metadata. An independent policy module applies policy to each data item based upon its classification. Multiple classifiers may be invoked, based upon various criteria. Predefined ordering of the classifiers, authoritative classifiers and/or an aggregation mechanism handle any classification conflicts. Different types of classifiers may be provided, and each classifier may correspond to automatic classification rules; the classifier may directly change a property, (e.g.

Abstract: Described is the backup and/or restore of virtual disks In general, metadata is backed up for restoring a virtual disk. To restore the disk, a physical disk is created, with the virtual disk the created on a partition of the physical disk. Backup and restore is described for nested virtual disks, including for block level restore. Further described is backing up of critical virtual disks and their containers, and virtual disk backup with respect to basic disks and dynamic volumes.

Abstract: Method and system for establishing and maintaining quotas. An auto quota is defined and applied to a directory. Input and output is monitored to detect a successful operation that involves a subdirectory of the directory. A determination is made as to whether to apply a quota associated with the auto quota to the subdirectory. If the determination is that the quota is to be applied to the subdirectory, it is automatically applied.

Abstract: Method and system for maintaining namespace consistency between selected objects maintained by a file system and a filter associated therewith. Metadata regarding selected objects of a file system is maintained by a filter while the filter is attached to the file system and persisted in non-volatile storage. The namespace of the file system may be changed while the filter is unattached from the file system. Afterwards, when the filter is attached to the file system, the namespace of the filter is synchronized with the namespace of the file system for the selected objects.