Abstract: The disclosed embodiments disclose techniques for maintaining global name consistency for a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. During operation, a cloud controller receives a client request to perform a namespace operation upon a filesystem object. The cloud controller contacts the cloud controller that manages (“owns”) the portion of the global namespace for the distributed filesystem that includes the filesystem object. This second cloud controller ensures the consistency of the filesystem object across the distributed filesystem during the namespace operation.

Abstract: The disclosed embodiments disclose techniques for sending interim notifications to clients of a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. During operation, a cloud controller receives a client request to perform a namespace operation upon a filesystem object. The cloud controller determines that it will need to contact at least one of another peer cloud controller or a cloud storage system to service the request, and sends an interim notification to the client to notify the client that the request is pending.

Abstract: A garbage collector scans containers of a storage system, each container containing a plurality of segments referenced by files of the storage system. For each of the containers being scanned constructing a working live segment record (LSR) file in memory, including generating a container LSR for each of the containers being scanned in a sequential order according to container identifiers of the containers, generating segment LSRs associated with the container LSR, each of the segment LSRs corresponding to one of the segments contained in the container, and for each of the segments contained in the container, indicating in a corresponding segment LSR whether the segment is a live segment. The segment LSRs of each container LSR of the working LSR file are translated from the memory into a persistent LSR for each of the containers in a persistent LSR file stored a persistent storage.

Abstract: Containers of a storage system are scanned, each container containing segments of files, where each file is represented by a file tree having segments in a hierarchical structure. The container live segment records (LSRs) corresponding to one of the containers are created, each of the container LSRs including segment LSRs corresponding to segments contained therein. After the segment LSRs of the container LSRs have been created for all segments of the containers, the segment LSRs of the container LSRs are sequentially traversed based on levels of segments specified in the corresponding segment LSRs to determine and indicate in the corresponding segment LSRs whether the segments are live segments. After all of the segment LSRs of the container LSRs have been traversed, a garbage collection operation is performed to reclaim storage space of segments that are not live segments indicated in the segment LSRs of the container LSRs.

Abstract: Techniques for using a bloom filter in deduplication are described herein. A change log comprising a plurality of data blocks may be received. Values associated with the data blocks may be hashed and compared with a bloom filter. The comparison with the bloom filter identifies data blocks from the change log as unique data blocks or potential duplicate data blocks. A bit by bit comparison of the potential duplicate data blocks and previous data blocks may be performed to determine if any of the potential duplicate data blocks are identical to any of previous data blocks. Such data blocks of the change log that are identified as being identical may be deduplicated.

Abstract: The disclosed embodiments disclose techniques for sending interim notifications to clients of a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. During operation, a cloud controller receives a client request to perform a namespace operation upon a filesystem object. The cloud controller determines that it will need to contact at least one of another peer cloud controller or a cloud storage system to service the request, and sends an interim notification to the client to notify the client that the request is pending.

Abstract: The disclosed embodiments disclose techniques for maintaining global name consistency for a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. During operation, a cloud controller receives a client request to perform a namespace operation upon a filesystem object. The cloud controller contacts the cloud controller that manages (“owns”) the portion of the global namespace for the distributed filesystem that includes the filesystem object. This second cloud controller ensures the consistency of the filesystem object across the distributed filesystem during the namespace operation.

Abstract: In a method and apparatus for scalable deduplication, a data set is partitioned into multiple logical partitions, where each partition can be deduplicated independently. Each data block of the data set is assigned to exactly one partition, so that any two or more data blocks that are duplicates of each are always be assigned to the same logical partition. A hash algorithm generates a fingerprint of each data block in the volume, and the fingerprints are subsequently used to detect possible duplicate data blocks as part of deduplication. In addition, the fingerprints are used to ensure that duplicate data blocks are sent to the same logical partition, prior to deduplication. A portion of the fingerprint of each data block is used as a partition identifier to determine the partition to which the data block should be assigned. Once blocks are assigned to partitions, deduplication can be done on partitions independently.

Abstract: In a method and apparatus for scalable deduplication, a data set is partitioned into multiple logical partitions, where each partition can be deduplicated independently. Each data block of the data set is assigned to exactly one partition, so that any two or more data blocks that are duplicates of each are always be assigned to the same logical partition. A hash algorithm generates a fingerprint of each data block in the volume, and the fingerprints are subsequently used to detect possible duplicate data blocks as part of deduplication. In addition, the fingerprints are used to ensure that duplicate data blocks are sent to the same logical partition, prior to deduplication. A portion of the fingerprint of each data block is used as a partition identifier to determine the partition to which the data block should be assigned. Once blocks are assigned to partitions, deduplication can be done on partitions independently.