Abstract: In some examples, a data management system processes snapshots of a distributed file system, the distributed file system having files, each file comprising multiple data chunks. The data management system performs operations including storing file-to-chunk mapping in file system metadata; creating, for each chunk, a chunk generation ID by associating each chunk with a file system generation ID; in a next-generation snapshot of the distributed file system, incrementing, for all chunks in the next-generation snapshot, the respective chunk generation IDs; and taking a snapshot of the file system metadata and storing an updated file-to-chunk mapping associated with the next-generation snapshot.

Abstract: Techniques for data management are described. A monotonically increasing time source that indicates an elapsed time since a reference event may be activated. Multiple snapshots of a computing object may be generated in accordance with a schedule for backing up the computing object, where the schedule may include a retention duration for retaining snapshots. Based on generating the snapshots, timestamps for the snapshots may be stored, where the timestamps may indicate respective values of the monotonically increasing time source. As part of an expiration job, a reference value of the monotonically increasing time source may be identified based on the retention duration and a current value indicated by the monotonically increasing time source. Also, a snapshot of the snapshots may be expired based on a timestamp of the snapshot corresponding to a value of the monotonically increasing time source that is less than the reference value.

Abstract: In some examples, a data management and storage (DMS) platform, comprises peer DMS nodes in a node cluster, a distributed data store comprising local and cloud storage, and at least one processor configured to perform operations in a method of creating a local consolidated patch file from a patch file chain stored in the cloud storage. The operations include, in a first dry-run phase, creating a logical patch file image of data blocks in one or more cloud patch files stored in the cloud storage; in a second data-transfer phase, downloading at least some of the data blocks from the cloud patch files identified by the logical patch file image, the second data-transfer phase comprising a coalescing operation to construct a set of coalesced reads of the data blocks; and creating and storing, in the local storage, the local consolidated patch file using the downloaded data blocks.

Abstract: In some examples, a data management system generates snapshots in a distributed file system based on a protocol or a user triggered event, The data management system identifies a snappable file in a distributed file system and a first data block in the snappable file, the first data block including data and attribute data. The system scans an index file to access the attribute data of the first data block and initiates construction of a patch file based on the accessed attribute data. The system repeats the scanning of the index file to access attribute data of at least a further second data block, the second data block including data and attribute data, and completes construction of the patch file based on the accessed attribute data of the first and second data blocks. The system generates conversion simulation information by collecting attribute data for all the data blocks of the constructed patch file, and writes the simulation information to a patch file image.

Abstract: Techniques for data management are described. A monotonically increasing time source that indicates an elapsed time since a reference event may be activated. Multiple snapshots of a computing object may be generated in accordance with a schedule for backing up the computing object, where the schedule may include a retention duration for retaining snapshots. Based on generating the snapshots, timestamps for the snapshots may be stored, where the timestamps may indicate respective values of the monotonically increasing time source. As part of an expiration job, a reference value of the monotonically increasing time source may be identified based on the retention duration and a current value indicated by the monotonically increasing time source. Also, a snapshot of the snapshots may be expired based on a timestamp of the snapshot corresponding to a value of the monotonically increasing time source that is less than the reference value.

Abstract: A method for recovering files from a filesystem stored across sparse files in a cloud environment is described. According to the method, a data management system may receive a request to read the files. The data management system may identify one or more target address ranges corresponding to the files indicated via the request. The data management system may read index information for the sparse files in the cloud environment. The index information may indicate respective address ranges for data blocks within the sparse files. The data management system may identify one or more data blocks within one or more sparse files as corresponding to address ranges that overlap with the one or more target address ranges based on the index information. The data management system may transmit, to the cloud environment, one or more read requests for the identified one or more data blocks.

Abstract: A method of consolidating snapshots includes receiving a request to consolidate a first snapshot with a second snapshot into a third snapshot, the first and second snapshots stored in separate backup files, each backup file organized as a directory where data parts of the first and second snapshots can be hard linked to locations outside of the backup file, comparing the data parts of the first and second snapshots to determine if any second snapshot data part fully overlaps with any first snapshot data part, responsive to determining that a second snapshot data part fully overlaps with a first snapshot data part, hard linking the determined second snapshot data part into the third snapshot, and storing the third snapshot in the backup file.

Abstract: In some examples, a data management and storage (DMS) platform, comprises peer DMS nodes in a node cluster, a distributed data store comprising local and cloud storage, and at least one processor configured to perform operations in a method of creating a local consolidated patch file from a patch file chain stored in the cloud storage. The operations include, in a first dry-run phase, creating a logical patch file image of data blocks in one or more cloud patch files stored in the cloud storage; in a second data-transfer phase, downloading at least some of the data blocks from the cloud patch files identified by the logical patch file image, the second data-transfer phase comprising a coalescing operation to construct a set of coalesced reads of the data blocks; and creating and storing, in the local storage, the local consolidated patch file using the downloaded data blocks.

Abstract: A method of consolidating snapshots includes receiving a request to consolidate a first snapshot with a second snapshot into a third snapshot, the first and second snapshots stored in separate backup files, each backup file organized as a directory where data parts of the first and second snapshots can be hard linked to locations outside of the backup file, comparing the data parts of the first and second snapshots to determine if any second snapshot data part fully overlaps with any first snapshot data part, responsive to determining that a second snapshot data part fully overlaps with a first snapshot data part, hard linking the determined second snapshot data part into the third snapshot, and storing the third snapshot in the backup file.

Abstract: A computer-implemented method at a data management system comprises: generating, with one or more processors, a containerized runtime in a memory in communication with the one or more processors; instantiating, with the one or more processors, an app in the runtime; receiving, with the one or more processors, a request from the app for data; retrieving, with the one or more processors, a copy of the requested data from a data source; and transmitting, with the one or more processors, the data to the containerized runtime for the app to operate on.

Abstract: In some examples, a data management system generates snapshots in a distributed file system based on a protocol or a user triggered event, The data management system identifies a snappable file in a distributed file system and a first data block in the snappable file, the first data block including data and attribute data. The system scans an index file to access the attribute data of the first data block and initiates construction of a patch file based on the accessed attribute data. The system repeats the scanning of the index file to access attribute data of at least a further second data block, the second data block including data and attribute data, and completes construction of the patch file based on the accessed attribute data of the first and second data blocks. The system generates conversion simulation information by collecting attribute data for all the data blocks of the constructed patch file, and writes the simulation information to a patch file image.

Abstract: Systems and methods herein describe a fraud detection system. The fraud detection system receives a transaction request comprising a set of transaction data, accesses a set of historical transaction data from one or more historical data sources, generates a weight score for each data source of the one or more historical data sources, generates a fraud score for the set of transaction data, the fraud score generated using a machine-learning model trained to analyze the historical transaction data and the generated weight scores for the one or more historical data sources, determines that the fraud score surpasses a threshold score, and in response to determining that the fraud score surpasses the threshold score, voids the transaction request.

Abstract: Techniques for data management are described. A monotonically increasing time source that indicates an elapsed time since a reference event may be activated. Multiple snapshots of a computing object may be generated in accordance with a schedule for backing up the computing object, where the schedule may include a retention duration for retaining snapshots. Based on generating the snapshots, timestamps for the snapshots may be stored, where the timestamps may indicate respective values of the monotonically increasing time source. As part of an expiration job, a reference value of the monotonically increasing time source may be identified based on the retention duration and a current value indicated by the monotonically increasing time source. Also, a snapshot of the snapshots may be expired based on a timestamp of the snapshot corresponding to a value of the monotonically increasing time source that is less than the reference value.

Abstract: A method for recovering files from a filesystem stored across sparse files in a cloud environment is described. According to the method, a data management system may receive a request to read the files. The data management system may identify one or more target address ranges corresponding to the files indicated via the request. The data management system may read index information for the sparse files in the cloud environment. The index information may indicate respective address ranges for data blocks within the sparse files. The data management system may identify one or more data blocks within one or more sparse files as corresponding to address ranges that overlap with the one or more target address ranges based on the index information. The data management system may transmit, to the cloud environment, one or more read requests for the identified one or more data blocks.

Abstract: In some examples, a data management and storage (DMS) platform comprises peer DMS nodes in a node cluster, a distributed data store comprising local and cloud storage, and at least one processor configured to perform operations in a method of creating a local consolidated patch file from a patch file chain stored in the cloud storage. Example operations comprise, in a first dry-run phase, creating a patch file image of data blocks in one or more cloud patch files stored in the cloud storage; in a second data-transfer phase, downloading at least some of the data blocks from the cloud patch files identified by the patch file image; and creating and storing, in the local storage, the local consolidated patch file using the downloaded data blocks.

Abstract: A storage system may store one or more snapshots of a computing system to support backup and restoration of data stored at the computing system. The storage system may identify an expiration of a first snapshot indicating a first set of physical storage locations to which first data of the computing system was stored as part of a first backup procedure. The storage system may identify a first subset physical storage locations of the first set as storing a first portion of the first data that is superseded by second data associated with a second snapshot. Based on identifying the first subset, the storage system may delete the first portion of the first data from the first subset of physical storage locations and retain a second portion of the first data at a second subset of physical storage locations.

Abstract: In some examples, a data aggregator and anonymizer is provided for selective encryption of test data.

Abstract: In some examples, a data management and storage (DMS) platform comprises peer DMS nodes in a node cluster, a distributed data store comprising local and cloud storage, and at least one processor configured to perform operations in a method of creating a local consolidated patch file from a patch file chain stored in the cloud storage. Example operations comprise, in a first dry-run phase, creating a patch file image of data blocks in one or more cloud patch files stored in the cloud storage; in a second data-transfer phase, downloading at least some of the data blocks from the cloud patch files identified by the patch file image; and creating and storing, in the local storage, the local consolidated patch file using the downloaded data blocks.