Abstract: In general, techniques for efficient data retrieval from a backup system are described. An example computing system includes one or more storage devices and processing circuitry having access to the one or more storage devices and configured to: process an input to generate a filter, wherein the input indicates a context for one or more queries; apply the filter to backup data to obtain filtered data from the backup data; generate an index of embeddings from the filtered data; process, based on the index of embeddings, a query to generate a response for the query; and output the response.

Abstract: A computing system comprising a storage device storing instructions, and processing circuitry that accesses the one or more storage devices and configured with the instructions to implement a change events stream. The processing circuitry may expose, via a single application programming interface executed by a data management platform, a unified difference data access layer that provides an abstraction layer by which to obtain difference data between two or more events, and interface, via the single application programming interface, with the unified difference data access layer to obtain the difference data. The processing circuitry may also publish the difference data to a change event stream, receive, from an application, a request to access at least a portion of the difference data published to the change event stream, and output, responsive to the request and to the application, at least the portion of the difference data published to the change event stream.

Abstract: Disclosed is an improved approach for implementing and maintaining writable snapshots. An efficient approach is provided for implementing snapshots that can be used to immediately create snapshots without incurring any detectable delays in providing access to the new snapshots. Also described are improved metadata structures that can be used to implement and maintain the writable snapshots.

Abstract: An inode is associated with an incomplete file system operation to a file system object is determined based on an update intent stored in the inode. It is determined that a set of inodes is associated with the incomplete file system operation based on the update intent stored in the inode. The update intent indicates the set of inodes is associated with the incomplete file system operation. It is determined to complete the incomplete file system operation based on evaluating the set of inodes that is associated with the incomplete file system operation. Evaluating the set of inodes that is associated with the incomplete file system operation includes determining whether the set of inodes that is associated with the incomplete file system operation stores a corresponding update intent. The incomplete file system operation is completed based on the evaluation of the set of inodes that is associated with the incomplete file system operation.

Abstract: An improved architecture is provided which enables significant convergence of the components of a system to implement virtualization. The infrastructure is VM-aware, and permits scaled out converged storage provisioning to allow storage on a per-VM basis, while identifying I/O coming from each VM. The current approach can scale out from a few nodes to a large number of nodes. In addition, the inventive approach has ground-up integration with all types of storage, including solid-state drives. The architecture of the invention provides high availability against any type of failure, including disk or node failures. In addition, the invention provides high performance by making I/O access local, leveraging solid-state drives and employing a series of patent-pending performance optimizations.

Abstract: Disclosed is an improved approach to implement I/O and storage device management in a virtualization environment. According to some approaches, a Service VM is employed to control and manage any type of storage device, including directly attached storage in addition to networked and cloud storage. The Service VM implements the Storage Controller logic in the user space, and can be migrated as needed from one node to another. IP-based requests are used to send I/O request to the Service VMs. The Service VM can directly implement storage and I/O optimizations within the direct data access path, without the need for add-on products.

Abstract: Techniques are disclosed for allocating shared resources to nodes in a distributed computing network system. Nodes request a lock for each instance of a computing resource (e.g., a virtual IP address associated with a service provided by the distributed computing network system) from a distributed lock manager. The distributed lock manager maintains a queue of requests for each instance of the shared resource. Upon receiving a lock from the distributed lock manager, the receiving node performs a fairness allocation protocol to determine whether to accept the lock. If so determined, the shared computing resources associated with the lock is configured.

Abstract: Disclosed is an improved approach to implement I/O and storage device management in a virtualization environment. According to some approaches, a Service VM is employed to control and manage any type of storage device, including directly attached storage in addition to networked and cloud storage. The Service VM implements the Storage Controller logic in the user space, and can be migrated as needed from one node to another. IP-based requests are used to send I/O request to the Service VMs. The Service VM can directly implement storage and I/O optimizations within the direct data access path, without the need for add-on products.

Abstract: A plurality of portions of a content file are stored. It is determined that the content file has a size that is greater than a threshold size. In response to determining that the content file has the size that is greater than the threshold size, a plurality of component file metadata structures are generated for each of the plurality of portions of the content file. A component file metadata structure of the plurality of component file metadata structures corresponds to one of the portions of the content file. Each of the plurality of component file metadata structures includes corresponding metadata that enables data chunks associated with a corresponding portion of the content file to be located.

Abstract: Data associated with a source system is ingested. After the data is ingested, a post-processing metadata conversion process is performed including by selecting an entry of a chunk metadata data structure and determining that a data chunk associated with the selected entry is not referenced by at least a threshold number of objects. In response to determining that the data chunk associated with the selected entry is not referenced by at least the threshold number of objects, metadata of a tree data structure node corresponding to a chunk identifier associated with the data chunk is updated to store a reference to a chunk file storing the data chunk and the selected entry is removed from the chunk metadata data structure.

Abstract: Backup data associated with a primary storage system is received. At least a view of the received backup data associated with the primary storage system is generated. The view of the of the received backup data associated with the primary storage system includes a tree data structure comprising a root node, a first plurality of child nodes, and a first plurality of leaf nodes. At least some of the first plurality of leaf nodes include corresponding pointers to a corresponding binary large object. A read request for data exposed by the view of the received backup data associated with the primary storage system is received from an external system. In response to receiving the read request, the view of the received backup data associated with the primary storage system is exposed to a requesting system utilizing a protocol associated with the external system.

Abstract: In general, techniques for efficient data retrieval from a backup system are described. An example computing system includes one or more storage devices and processing circuitry having access to the one or more storage devices and configured to: process an input to generate a filter, wherein the input indicates a context for one or more queries; apply the filter to backup data to obtain filtered data from the backup data; generate an index of embeddings from the filtered data; process, based on the index of embeddings, a query to generate a response for the query; and output the response.

Abstract: Disclosed is an improved approach to implement I/O and storage device management in a virtualization environment. According to some approaches, a Service VM is employed to control and manage any type of storage device, including directly attached storage in addition to networked and cloud storage. The Service VM implements the Storage Controller logic in the user space, and can be migrated as needed from one node to another. IP-based requests are used to send I/O request to the Service VMs. The Service VM can directly implement storage and I/O optimizations within the direct data access path, without the need for add-on products.

Abstract: Techniques for cloning, writing to, and reading from file system metadata. Cloning involves identifying a first set of pointers included h a first root node in a file system metadata tree structure that stores file system metadata n leaf nodes of the tree structure, creating a first copy of the first root node that includes the first set of pointers, creating a second copy of the first root node that includes the first set of pointers, associating the first copy with a first view, and associating the second copy with a second view. Reading generally involves traversing the tree structure towards a target leaf node that contains data to be read. Writing generally involves traversing the tree structure in the same manner, but also creating copies of any nodes to be modified if those nodes are deemed to have a different treeID than a particular root node.

Abstract: Disclosed is an improved approach to implement I/O and storage device management in a virtualization environment. According to some approaches, a Service VM is employed to control and manage any type of storage device, including directly attached storage in addition to networked and cloud storage. The Service VM implements the Storage Controller logic in the user space, and can be migrated as needed from one node to another. IP-based requests are used to send I/O request to the Service VMs. The Service VM can directly implement storage and I/O optimizations within the direct data access path, without the need for add-on products.

Abstract: Data associated with a source system is ingested. After the data is ingested, a post-processing metadata conversion process is performed including by selecting an entry of a chunk metadata data structure and determining that a data chunk associated with the selected entry is not referenced by at least a threshold number of objects. In response to determining that the data chunk associated with the selected entry is not referenced by at least the threshold number of objects, metadata of a tree data structure node corresponding to a chunk identifier associated with the data chunk is updated to store a reference to a chunk file storing the data chunk and the selected entry is removed from the chunk metadata data structure.

Abstract: Backup data associated with a primary storage system is received. At least a view of the received backup data associated with the primary storage system is generated. The view of the of the received backup data associated with the primary storage system includes a tree data structure comprising a root node, a first plurality of child nodes, and a first plurality of leaf nodes. At least some of the first plurality of leaf nodes include corresponding pointers to a corresponding binary large object. A read request for data exposed by the view of the received backup data associated with the primary storage system is received from an external system. In response to receiving the read request, the view of the received backup data associated with the primary storage system is exposed to a requesting system utilizing a protocol associated with the external system.

Abstract: Data is ingested from a source system including by storing a plurality of data chunks in one or more chunk files and storing corresponding chunk identifiers associated with the plurality of data chunks in a first data structure. After data ingestion is complete, one or more duplicate data chunks that were stored during the data ingestion are determined and a second data structure is updated to include one or more entries corresponding to one or more determined duplicate data chunks.

Abstract: A method and apparatus for maintaining file system consistency in a distributed system using version vectors is presented. A method generally includes comparing incarnation and transaction identifiers of a current version vector associated with a file with incarnation and transaction identifiers of a last completed version vector associated with the file. Upon determining that a current version vector reflects operations on the file that are either earlier than or the same as the identifiers in the last completed version vector, the node performing one or more file system operations on the file.

Abstract: According to one embodiment of the present disclosure, a first set of file system objects included in performing a requested file system operation is identified in response to a request to perform a file system operation. An update intent corresponding to the requested file system operation is inserted into a data structure associated with each identified file system object. Each file system object corresponding to the corresponding data structure is modified as specified by the update intent in that data structure. After modifying the file system object corresponding to the corresponding data structure, the update intent is removed from that data structure.