Abstract: Techniques unmasking ransomware attacks are disclosed. In some embodiments, a computer system performs operations comprising: generating a first prediction that a file system comprising a plurality of files has been attacked by ransomware based on snapshot metadata of the file system using a snapshot-level machine learning prediction model, the snapshot metadata comprising a plurality of file change data indicating a plurality of file change events that have been performed on the file system; in response to the first prediction, generating a classification for each one of the files based on the file change data using a file-level machine learning prediction model, the classification indicating whether the files have been targeted by the ransomware for encryption; determining that one or more files have been targeted by the ransomware based on the classification; and displaying the classification for the one or more files on a computing device of a user.

Abstract: Techniques for implementing a scalable automated training framework for anomaly and ransomware detection are disclosed. In some embodiments, a computer system performs operations comprising: instantiating a plurality of virtual machines, each one of the virtual machines being loaded with a corresponding file system; simulating user actions and ransomware on the virtual machines, the simulating of user actions and ransomware on the virtual machines causing changes to the corresponding file systems of the virtual machines; for each one of the plurality of virtual machines, generating a corresponding metadata file based on one or more corresponding snapshots of the virtual machine, the one or more corresponding snapshots indicating the changes to the corresponding file system of the virtual machine; and training a ransomware detection model using a machine learning algorithm and training data, the training data being based on the corresponding metadata files of the virtual machines.

Abstract: In one approach, filesets to be backed up are divided into partitions and snapshots are pulled for each partition. In one architecture, a data management and storage (DMS) cluster includes a plurality of peer DMS nodes and a distributed data store implemented across the peer DMS nodes. One of the peer DMS nodes receives fileset metadata for the fileset and defines a plurality of partitions for the fileset based on the fileset metadata. The peer DMS nodes operate autonomously to execute jobs to pull snapshots for each of the partitions and to store the snapshots of the partitions in the distributed data store.

Abstract: Subject matter related to data management is disclosed. Snapshots of a target object in a computing system may be stored, where the snapshot may correspond to multiple versions of the target object. An indication that information in one or more of the snapshots may be stored. A request to restore a snapshot of the target object may be received. In response to the request, a determination of whether the snapshot of the target object is prohibited from being restored may be made based on the indication that information in one or more of the snapshots is flagged for quarantine. A determination of whether to initiate a restoration operation in response to the request to restore the snapshot may also be made based on whether the snapshot is prohibited from being restored.

Abstract: Methods and systems for automatically upgrading or synchronizing a remote data management agent running on a remote host machine (e.g., a hardware server) to a particular version that is in-sync with a corresponding version used by a cluster of data storage nodes controlling the remote data management agent are described. The remote agent may be initially installed on the remote host and subsequent updates to the remote agent may be performed using the remote agent itself without requiring intervention by the remote host. The remote agent may comprise a backup agent and a bootstrap agent that are each exposed in different network ports or associated with different port numbers or networking addresses. The backup agent may perform data backup related tasks for backing up files stored on the remote host and the bootstrap agent may perform upgrade related tasks for upgrading the backup agent.

Abstract: In some examples, a cluster protection system comprises at least one processor and a memory storing instructions which, when executed by the at least one processor, cause the system to perform operations comprising identifying a target cluster or an object in a container management framework, identifying application data and metadata associated with the target cluster or the object, generating a first snapshot of the target cluster or the object, the first snapshot including at least the metadata, storing the first snapshot in offsite cloud storage, generating a second snapshot of the target cluster, the second snapshot including at least the application data, and storing the second snapshot in a persistent volume in onsite storage.

Abstract: In one approach, filesets to be backed up are divided into partitions and snapshots are pulled for each partition. In one architecture, a data management and storage (DMS) cluster includes a plurality of peer DMS nodes and a distributed data store implemented across the peer DMS nodes. One of the peer DMS nodes receives fileset metadata for the fileset and defines a plurality of partitions for the fileset based on the fileset metadata. The peer DMS nodes operate autonomously to execute jobs to pull snapshots for each of the partitions and to store the snapshots of the partitions in the distributed data store.

Abstract: Techniques for implementing a scalable automated training framework for anomaly and ransomware detection are disclosed. In some embodiments, a computer system performs operations comprising: instantiating a plurality of virtual machines, each one of the virtual machines being loaded with a corresponding file system; simulating user actions and ransomware on the virtual machines, the simulating of user actions and ransomware on the virtual machines causing changes to the corresponding file systems of the virtual machines; for each one of the plurality of virtual machines, generating a corresponding metadata file based on one or more corresponding snapshots of the virtual machine, the one or more corresponding snapshots indicating the changes to the corresponding file system of the virtual machine; and training a ransomware detection model using a machine learning algorithm and training data, the training data being based on the corresponding metadata files of the virtual machines.

Abstract: Techniques unmasking ransomware attacks are disclosed. In some embodiments, a computer system performs operations comprising: generating a first prediction that a file system comprising a plurality of files has been attacked by ransomware based on snapshot metadata of the file system using a snapshot-level machine learning prediction model, the snapshot metadata comprising a plurality of file change data indicating a plurality of file change events that have been performed on the file system; in response to the first prediction, generating a classification for each one of the files based on the file change data using a file-level machine learning prediction model, the classification indicating whether the files have been targeted by the ransomware for encryption; determining that one or more files have been targeted by the ransomware based on the classification; and displaying the classification for the one or more files on a computing device of a user.

Abstract: A lightweight deduplication system can perform resource efficient data deduplication using an extent index and a content index. The extent index can store full fingerprints of data segments to be deduplicated and the content index can store shortened versions of the full fingerprints. The system can alternate between the extent and content indexes, and cache portions of the indices to perform lightweight data deduplication. Further, the system can be configured with an efficient heuristic approach for selecting content index data lookups for chains of volumes for deduplication, such as a long chain of snapshots.

Abstract: A lightweight deduplication system can perform resource efficient data deduplication using an extent index and a content index. The extent index can store full fingerprints of data segments to be deduplicated and the content index can store shortened versions of the full fingerprints. The system can alternate between the extent and content indexes, and cache portions of the indices to perform lightweight data deduplication. Further, the system can be configured with an efficient heuristic approach for selecting content index data lookups for chains of volumes for deduplication, such as a long chain of snapshots.

Abstract: Methods and systems for efficiently capturing snapshots of a computing application or environment over time and transferring the snapshots to an integrated data management and storage system are described. A snapshot agent may detect that one or more electronic files associated with the computing application or environment are greater than a threshold file size and in response perform an incremental backup optimization in which the snapshot agent may identify files that have been touched since a previous snapshot by accessing file system metadata (e.g., last modified timestamps) or utilizing a tracking agent to detect potential file changes that have occurred since the previous snapshot was captured. The snapshot agent may then generate fingerprints for data blocks of the touched files, which may reduce the total number of fingerprints needing to be generated to identify the changed data blocks corresponding with a current snapshot to be transferred.

Abstract: In one approach, filesets to be backed up are divided into partitions and snapshots are pulled for each partition. In one architecture, a data management and storage (DMS) cluster includes a plurality of peer DMS nodes and a distributed data store implemented across the peer DMS nodes. One of the peer DMS nodes receives fileset metadata for the fileset and defines a plurality of partitions for the fileset based on the fileset metadata. The peer DMS nodes operate autonomously to execute jobs to pull snapshots for each of the partitions and to store the snapshots of the partitions in the distributed data store.

Abstract: Methods and systems for efficiently capturing snapshots of a computing application or environment over time and transferring the snapshots to an integrated data management and storage system are described. A snapshot agent may detect that one or more electronic files associated with the computing application or environment are greater than a threshold file size and in response perform an incremental backup optimization in which the snapshot agent may identify files that have been touched since a previous snapshot by accessing file system metadata (e.g., last modified timestamps) or utilizing a tracking agent to detect potential file changes that have occurred since the previous snapshot was captured. The snapshot agent may then generate fingerprints for data blocks of the touched files, which may reduce the total number of fingerprints needing to be generated to identify the changed data blocks corresponding with a current snapshot to be transferred.

Abstract: Methods and systems for automatically upgrading or synchronizing a remote data management agent running on a remote host machine (e.g., a hardware server) to a particular version that is in-sync with a corresponding version used by a cluster of data storage nodes controlling the remote data management agent are described. The remote agent may be initially installed on the remote host and subsequent updates to the remote agent may be performed using the remote agent itself without requiring intervention by the remote host. The remote agent may comprise a backup agent and a bootstrap agent that are each exposed in different network ports or associated with different port numbers or networking addresses. The backup agent may perform data backup related tasks for backing up files stored on the remote host and the bootstrap agent may perform upgrade related tasks for upgrading the backup agent.

Abstract: A garbage collector of a storage system traverses a namespace of a file system of the storage system to identify segments that are alive in a breadth-first manner. The namespace includes information identifying files that are represented by segments arranged in a plurality of levels in a hierarchy, where an upper level segment includes one or more references to one or more lower level segments, and at least one segment is referenced by multiple files. All live segments of an upper level are identified before any of live segments of a lower level are identified. Upon all live segments of all levels have been identified, the live segments are copied from their original storage locations to a new storage location, and a storage space associated with the original storage locations is reclaimed.

Abstract: A garbage collector of a storage system traverses a namespace of a file system of the storage system to verify data integrity of segments. The namespace identifies files that are represented by segments arranged in multiple levels in a hierarchy, where an upper level segment includes one or more references to one or more lower level segments, and at least one segment is referenced by multiple files. Traversing the namespace includes computing and verifying checksums all segments in a level-by-level manner, where checksums of an upper level are verified before any of checksums of a lower level are verified. Upon all checksums of all levels have been verified, a garbage collection process is performed on the segments stored in the storage system.