Abstract: Disclosed is an improved approach to implement I/O and storage device management in a virtualization environment. According to some approaches, a Service VM is employed to control and manage any type of storage device, including directly attached storage in addition to networked and cloud storage. The Service VM implements the Storage Controller logic in the user space, and can be migrated as needed from one node to another. IP-based requests are used to send I/O request to the Service VMs. The Service VM can directly implement storage and I/O optimizations within the direct data access path, without the need for add-on products.

Abstract: Techniques for cloning, writing to, and reading from file system metadata. Cloning involves identifying a first set of pointers included h a first root node in a file system metadata tree structure that stores file system metadata n leaf nodes of the tree structure, creating a first copy of the first root node that includes the first set of pointers, creating a second copy of the first root node that includes the first set of pointers, associating the first copy with a first view, and associating the second copy with a second view. Reading generally involves traversing the tree structure towards a target leaf node that contains data to be read. Writing generally involves traversing the tree structure in the same manner, but also creating copies of any nodes to be modified if those nodes are deemed to have a different treeID than a particular root node.

Abstract: Disclosed is an improved approach to implement I/O and storage device management in a virtualization environment. According to some approaches, a Service VM is employed to control and manage any type of storage device, including directly attached storage in addition to networked and cloud storage. The Service VM implements the Storage Controller logic in the user space, and can be migrated as needed from one node to another. IP-based requests are used to send I/O request to the Service VMs. The Service VM can directly implement storage and I/O optimizations within the direct data access path, without the need for add-on products.

Abstract: Data associated with a source system is ingested. After the data is ingested, a post-processing metadata conversion process is performed including by selecting an entry of a chunk metadata data structure and determining that a data chunk associated with the selected entry is not referenced by at least a threshold number of objects. In response to determining that the data chunk associated with the selected entry is not referenced by at least the threshold number of objects, metadata of a tree data structure node corresponding to a chunk identifier associated with the data chunk is updated to store a reference to a chunk file storing the data chunk and the selected entry is removed from the chunk metadata data structure.

Abstract: Backup data associated with a primary storage system is received. At least a view of the received backup data associated with the primary storage system is generated. The view of the of the received backup data associated with the primary storage system includes a tree data structure comprising a root node, a first plurality of child nodes, and a first plurality of leaf nodes. At least some of the first plurality of leaf nodes include corresponding pointers to a corresponding binary large object. A read request for data exposed by the view of the received backup data associated with the primary storage system is received from an external system. In response to receiving the read request, the view of the received backup data associated with the primary storage system is exposed to a requesting system utilizing a protocol associated with the external system.

Abstract: Data is ingested from a source system including by storing a plurality of data chunks in one or more chunk files and storing corresponding chunk identifiers associated with the plurality of data chunks in a first data structure. After data ingestion is complete, one or more duplicate data chunks that were stored during the data ingestion are determined and a second data structure is updated to include one or more entries corresponding to one or more determined duplicate data chunks.

Abstract: According to one embodiment of the present disclosure, a first set of file system objects included in performing a requested file system operation is identified in response to a request to perform a file system operation. An update intent corresponding to the requested file system operation is inserted into a data structure associated with each identified file system object. Each file system object corresponding to the corresponding data structure is modified as specified by the update intent in that data structure. After modifying the file system object corresponding to the corresponding data structure, the update intent is removed from that data structure.

Abstract: A method and apparatus for maintaining file system consistency in a distributed system using version vectors is presented. A method generally includes comparing incarnation and transaction identifiers of a current version vector associated with a file with incarnation and transaction identifiers of a last completed version vector associated with the file. Upon determining that a current version vector reflects operations on the file that are either earlier than or the same as the identifiers in the last completed version vector, the node performing one or more file system operations on the file.

Abstract: An inode is associated with an incomplete file system operation to a file system object is determined based on an update intent stored in the inode. It is determined that a set of inodes is associated with the incomplete file system operation based on the update intent stored in the inode. The update intent indicates the set of inodes is associated with the incomplete file system operation. It is determined to complete the incomplete file system operation based on evaluating the set of inodes that is associated with the incomplete file system operation. Evaluating the set of inodes that is associated with the incomplete file system operation includes determining whether the set of inodes that is associated with the incomplete file system operation stores a corresponding update intent.

Abstract: A plurality of portions of a content file are stored. It is determined that the content file has a size that is greater than a threshold size. In response to determining that the content file has the size that is greater than the threshold size, a plurality of component file metadata structures are generated for each of the plurality of portions of the content file. A component file metadata structure of the plurality of component file metadata structures corresponds to one of the portions of the content file. Each of the plurality of component file metadata structures includes corresponding metadata that enables data chunks associated with a corresponding portion of the content file to be located.

Abstract: Techniques are disclosed which allow a secondary storage system to provide data to non-production workloads in conjunction with performing data backup and protection tasks. As disclosed, a secondary storage system exposes backup data stored by the secondary storage system to other workloads, such as test and development applications, data analytics, etc. These non-production workloads can run at the same time the secondary storage system provides backup services to a primary storage system. This consolidation eliminates the need for an enterprise to deploy separate storage clusters for analytics, test and development applications, etc. and eliminates unnecessary copies of data.

Abstract: A request to write data corresponding to at least a first portion of a file is received. It is determined whether to perform the request either as an in-place write or as an out-of-place write. Performing the in-place write comprises performing a write to a low latency storage device, and performing the out-of-place write comprises performing a write to a higher latency storage device. The request is performed as either the in-place write or the out-of-place write based on the determination. Performing the request as the in-place write includes writing the data to a first location on a storage tier storing the first portion of the file, and performing the request as the out-of-place write includes writing the data to a second location on one of a plurality of storage tiers of a computing node, other than the first location.

Abstract: A size associated with a content file is determined to be greater than a threshold size. Contents of the content file split across a plurality of component files are stored. Metadata, for the content file, is updated to reference a plurality of component file metadata structures for the component files. A node of the metadata is configured to track different sizes of portions of the content file stored in different component files of the plurality of component files. File metadata of the content file is split across the plurality of component file metadata structures and each component file metadata structure of the plurality of component file metadata structures specifies a corresponding structure organizing data components for a corresponding portion of the content file.

Abstract: Embodiments described herein provide techniques for maintaining consistency in a distributed system (e.g., a distributed secondary storage system). According to one embodiment of the present disclosure, a first set of file system objects included in performing the requested file system operation is identified in response to a request to perform a file system operation. An update intent corresponding to the requested file system operation is inserted into an inode associated with each identified file system object. Each file system object corresponding to the inode is modified as specified by the update intent in that inode. After modifying the file system object corresponding to the inode, the update intent is removed from that inode.

Abstract: A method and system for performing incremental backup of a network attached storage (NAS) device are described. A storage capture instance associated with a first time instance is received from a network attached storage device. At least a portion of metadata of tracked network packets associated with the network attached storage device is also received. At least one changed content item of the network attached storage device that has changed since the first time instance is identified by analyzing the at least the portion of the metadata of the tracked network packets received. An incremental backup of the network attached storage device is performed at a second time instance based at least in part on the at least one changed content item identified.

Abstract: Data is ingested from a source system including by storing a plurality of data chunks in one or more chunk files and storing corresponding chunk identifiers associated with the plurality of data chunks in a first data structure. After data ingestion is complete, one or more duplicate data chunks that were stored during the data ingestion are determined and a second data structure is updated to include one or more entries corresponding to one or more determined duplicate data chunks.

Abstract: Disclosed is an improved approach to implement I/O and storage device management in a virtualization environment. According to some approaches, a Service VM is employed to control and manage any type of storage device, including directly attached storage in addition to networked and cloud storage. The Service VM implements the Storage Controller logic in the user space, and can be migrated as needed from one node to another. IP-based requests are used to send I/O request to the Service VMs. The Service VM can directly implement storage and I/O optimizations within the direct data access path, without the need for add-on products.

Abstract: Disclosed is an improved approach for implementing and maintaining writable snapshots. An efficient approach is provided for implementing snapshots that can be used to immediately create snapshots without incurring any detectable delays in providing access to the new snapshots. Also described are improved metadata structures that can be used to implement and maintain the writable snapshots.

Abstract: An improved architecture is provided which enables significant convergence of the components of a system to implement virtualization. The infrastructure is VM-aware, and permits scaled out converged storage provisioning to allow storage on a per-VM basis, while identifying I/O coming from each VM. The current approach can scale out from a few nodes to a large number of nodes. In addition, the inventive approach has ground-up integration with all types of storage, including solid-state drives. The architecture of the invention provides high availability against any type of failure, including disk or node failures. In addition, the invention provides high performance by making I/O access local, leveraging solid-state drives and employing a series of patent-pending performance optimizations.

Abstract: Techniques are disclosed for managing a high performance, fault-tolerant, strongly consistent, distributed key-value store system. The key-value store may store information, such as metadata for a distributed file system. Fault-tolerance means that the distributed key-value store continues to provide access to values in the key-value store in spite of a certain number of node failures. To provide this capability, the key-value store may store copies of (key, value) pair on N+1 nodes in order to provide fault tolerance for the failure of up to N nodes. In addition, metadata describing which nodes store a given value is stored on 2N+1 nodes and the distributed key-value store is sized such that there are 3N+1 nodes in a cluster. Doing so allows the key, value store to tolerate a failure of N nodes, while still maintaining a consistent and available key-value store.