Abstract: A data management system manages concurrent readers and writers for large file scans. The data management system may read a plurality of data chunks of the file starting from different offsets and generate a bounded number of read requests, which causes a data chuck identifiable by a data offset to be loaded into a data buffer. The system may queue the loaded data chunks for generating write requests to release the loaded data chunks. One or more write requests are generated responsive to one or more data chunks being associated with a consecutive order of data offsets being successfully loaded to data buffers. The system may write data chunks released from the buffer-rounded reading stage to the data storage in a checkpointed writing stage. The checkpointed writing stage creates a checkpoint based on the data offset of the data chunks that have been completely transferred to the data storage.

Abstract: A data management system manages concurrent readers and writers for large file scans. The data management system may read a plurality of data chucks of the file starting from different offsets and generate a bounded number of read requests, which causes a data chuck identifiable by a data offset to be loaded into a data buffer. The system may queue the loaded data chucks for generating write requests to release the loaded data chunks. One or more write requests are generated responsive to one or more data chunks being associated with a consecutive order of data offsets being successfully loaded to data buffers. The system may write data chucks released from the buffer-rounded reading stage to the data storage in a checkpointed writing stage. The checkpointed writing stage creates a checkpoint based on the data offset of the data chucks that have been completely transferred to the data storage.

Abstract: A data management system manages concurrent readers and writers for large file scans. The data management system may read a plurality of data chucks of the file starting from different offsets and generate a bounded number of read requests, which causes a data chuck identifiable by a data offset to be loaded into a data buffer. The system may queue the loaded data chucks for generating write requests to release the loaded data chunks. One or more write requests are generated responsive to one or more data chunks being associated with a consecutive order of data offsets being successfully loaded to data buffers. The system may write data chucks released from the buffer-rounded reading stage to the data storage in a checkpointed writing stage. The checkpointed writing stage creates a checkpoint based on the data offset of the data chucks that have been completely transferred to the data storage.

Abstract: A system includes a data store and a data management server. The data store stores a plurality of backup snapshots that capture states of a device at different times. The data management server receives a request to restore the device that is potentially malware affected. The data management server retrieves a first backup snapshot of the device. The data management server determines that a first file stored in the first backup snapshot is malware affected. The data management server checks one or more corresponding versions of the first file captured in one or more previous backup snapshots to identify a clean version of the first file. The data management server determines that a second file stored in the first backup snapshot is clean. The data management server restores data in the device.

Abstract: A backup management system may include a data management server, a warm-tier data store, and a cold-tier data store. Snapshots may be captured from various client devices. A data block stored in the warm-tier data store may be referenced by multiple backup snapshots and/or referenced by one or more users. When a data block's total reference count is equal to the cold reference count or equal to or less than a threshold total reference count, the data management server may determine that the data block is ready to be migrated to the cold-tier data store. The data management server may send the data block into a candidate queue. In the queue, data blocks with similar retention periods or similar expected restoration may be grouped as a unit. The unit may be transmitted to the cold-tier data store in a single write request.

Abstract: A file system may include an object storage, a merged index, and a distributed database. When a file is stored in the file system, the file may be converted to an object and be stored in the object storage. The deduplication index of the file may be stored in the distributed database. The namespace metadata of the file may be stored in the merged index. The merged index generates namespace entries of the file when the file is created, deleted, and/or modified. A namespace entry may be associated with a specific file and may include a creation version and a deletion version. When a file is deleted or modified, instead of modifying the existing namespace entries, new entries associated with different versions and including different creation or deletion versions are created. The status of a file may be monitored by one or more entries associated with a file.

Abstract: Disclosed embodiments include a method (system and non-transitory computer-readable medium) for backing up updated portions of a plurality files having hierarchical relationships through object storage. In one or more embodiments, a file is segregated into chunks, and objects corresponding to the chunks are generated for storage at an object storage. For a chunk, an object for storing the chunk and additional objects for storing mapping information are generated. The mapping information may include path information identifying a path of the file in a hierarchical structure, a file version list identifying a version of the file, a chunk list describing an association between the file and the chunks, a chunk version list identifying a version of the chunk, etc. When a portion of the file is updated, objects corresponding to the updated portion of the file can be generated, and stored at the object storage.

Abstract: A system includes a data store and a data management server. The data store stores a plurality of backup snapshots that capture states of a device at different times. The data management server receives a request to restore the device that is potentially malware affected. The data management server retrieves a first backup snapshot of the device. The data management server determines that a first file stored in the first backup snapshot is malware affected. The data management server checks one or more corresponding versions of the first file captured in one or more previous backup snapshots to identify a clean version of the first file. The data management server determines that a second file stored in the first backup snapshot is clean. The data management server restores data in the device.

Abstract: A system, method, and non-transitory computer-readable medium provide backup and archive services for computing devices. Typically, multiple backup snapshots are archived together in each archive cycle. A catalogue for the current archive is efficiently created by starting with a copy of the previous archive catalogue and updating it based on metadata associated with the backup snapshots.

Abstract: A file system may include an object storage, a merged index, and a distributed database. When a file is stored in the file system, the file may be converted to an object and be stored in the object storage. The deduplication index of the file may be stored in the distributed database. The namespace metadata of the file may be stored in the merged index. The merged index generates namespace entries of the file when the file is created, deleted, and/or modified. A namespace entry may be associated with a specific file and may include a creation version and a deletion version. When a file is deleted or modified, instead of modifying the existing namespace entries, new entries associated with different versions and including different creation or deletion versions are created. The status of a file may be monitored by one or more entries associated with a file.

Abstract: A data storage system allows data to be encrypted and de-duplicated at the same system. By way of example, a server of the data storage system may request a client device which intends to upload a data block to transmit a first fingerprint of the data block to the server. The first fingerprint may be derived from the plaintext of the data block. The server may apply a one-way function to the first fingerprint to generate an encryption key and transmit the encryption key to the client device. The client device uses the encryption key to encrypt the data block and generates a second fingerprint which is derived from the ciphertext of the data block. The server uses both the first fingerprint and the second fingerprint to verify the data block and the legitimacy of the client attempting to upload the data block.

Abstract: Aspects of the current patent document include systems and methods to perform search in an index system. In one embodiment, an index system may be implemented in an object storage. A distributed database index is used in conjunction with the object storage. In some cases, data stored in the distributed database may be encrypted and moved to object storage. The object storage stores a plurality of blocks containing words. Each block can contain a large number of words, such as one million words.

Abstract: A backup management system may include a data management server, a warm-tier data store, and a cold-tier data store. Snapshots may be captured from various client devices. A data block stored in the warm-tier data store may be referenced by multiple backup snapshots and/or referenced by one or more users. When a data block's total reference count is equal to the cold reference count or equal to or less than a threshold total reference count, the data management server may determine that the data block is ready to be migrated to the cold-tier data store. The data management server may send the data block into a candidate queue. In the queue, data blocks with similar retention periods or similar expected restoration may be grouped as a unit. The unit may be transmitted to the cold-tier data store in a single write request.

Abstract: Disclosed embodiments include a method (system and non-transitory computer-readable medium) for backing up updated portions of a plurality files having hierarchical relationships through object storage. In one or more embodiments, a file is segregated into chunks, and objects corresponding to the chunks are generated for storage at an object storage. For a chunk, an object for storing the chunk and additional objects for storing mapping information are generated. The mapping information may include path information identifying a path of the file in a hierarchical structure, a file version list identifying a version of the file, a chunk list describing an association between the file and the chunks, a chunk version list identifying a version of the chunk, etc. When a portion of the file is updated, objects corresponding to the updated portion of the file can be generated, and stored at the object storage.

Abstract: Disclosed embodiments include a method (system and non-transitory computer-readable medium) for backing up updated portions of a plurality files having hierarchical relationships through object storage. In one or more embodiments, a file is segregated into chunks, and objects corresponding to the chunks are generated for storage at an object storage. For a chunk, an object for storing the chunk and additional objects for storing mapping information are generated. The mapping information may include path information identifying a path of the file in a hierarchical structure, a file version list identifying a version of the file, a chunk list describing an association between the file and the chunks, a chunk version list identifying a version of the chunk, etc. When a portion of the file is updated, objects corresponding to the updated portion of the file can be generated, and stored at the object storage.

Abstract: When a backup client sends a request to back up a file to a backup server, the file and an index (e.g., checksum, hash, encryption, etc.) of the file are stored on the backup server in an efficient deduplication storage. If a backup client sends a request to back up a modified version of a file already stored on a backup server, the modified portion of the file is stored. In addition, an index of the modified portion is generated and stored along with the modified portions on the backup server. The indices can be used to reconstruct the file or modified version of the file when retrieved. The efficient deduplication storage method ensures that multiple copies of files or portions of files do not exist on the servers.

Abstract: A file system may include an object storage, a merged index, and a distributed database. When a file is stored in the file system, the file may be converted to an object and be stored in the object storage. The deduplication index of the file may be stored in the distributed database. The namespace metadata of the file may be stored in the merged index. The merged index generates namespace entries of the file when the file is created, deleted, and/or modified. A namespace entry may be associated with a specific file and may include a creation version and a deletion version. When a file is deleted or modified, instead of modifying the existing namespace entries, new entries associated with different versions and including different creation or deletion versions are created. The status of a file may be monitored by one or more entries associated with a file.

Abstract: Data from computing devices is backed-up regularly, storing a snapshot of the data and corresponding metadata in a data store. The backup data are stored for a relatively short period of time before being archived to long-term storage. For snapshots with files with identical data that are not archived together, archive storage space and computing resources may be conserved by not storing duplicates of the data. When the data is added to the archive storage, the archive storage location is added to backup reference entries for other files with identical data. When all files referencing an archive storage location are expired from the backup data store, an archive retention period is initiated, and an entry is added to a time-based archive expiration database indicating the storage location and an expiration time for the archived data. At the expiration time, the archived data is designated for deletion from the archive.

Abstract: Disclosed embodiments include a method (system and non-transitory computer storage readable medium) for load-balancing a distributed database. The distributed database includes one or more storage machines configured to store a plurality of partitions, where each partition includes key-value pairs. In one embodiment, the distributed database prepares for load-balancing by determining a partition to redistribute (or repartition) and generating smaller partitions of the determined partition. In one aspect, each of the smaller partitions is smaller than the determined partition. The redistribution of the partition can occur, when an amount of requests to access one or more key-value pairs stored in the database increases beyond a predetermined request level or when the size of a partition exceeds a predetermined size. Key-value pairs of the determined partition can be split into different sets of key-value pairs, and each set of key-value pair is copied to a corresponding smaller partition.

Abstract: Disclosed embodiments include a method (system and non-transitory computer-readable medium) for backing up updated portions of a plurality files having hierarchical relationships through object storage. In one or more embodiments, a file is segregated into chunks, and objects corresponding to the chunks are generated for storage at an object storage. For a chunk, an object for storing the chunk and additional objects for storing mapping information are generated. The mapping information may include path information identifying a path of the file in a hierarchical structure, a file version list identifying a version of the file, a chunk list describing an association between the file and the chunks, a chunk version list identifying a version of the chunk, etc. When a portion of the file is updated, objects corresponding to the updated portion of the file can be generated, and stored at the object storage.