Abstract: A system can, based on a retrieval-augmented generation (RAG) process ingesting data, query a search system to identify at least one first portion of the data that has at least one third generation identifier that is greater than second generation identifiers in a checkpoint; create second chunks from the at least one first portion of the data; in response to second chunks being duplicates of first chunks, based on hash values, store first identifiers of the second chunks; remove the stored chunk identifiers from second identifiers of chunks that correspond to chunks that existed in a RAG system prior to the ingesting, to produce stale chunk identifiers; remove, from the RAG system, third chunks that are identified by the stale chunk identifiers, and, in response to the second chunks being determined to be unique relative to the first chunks, store the second chunks in the RAG system.

Abstract: A system can store a group of checkpoints that comprises respective pairs comprising respective identifications of at least some respective data stored in a storage system and respective second generation identifiers that correspond to the at least some respective data. The system can, based on ingesting data from the storage system and to send the data to be ingested by a first retrieval-augmented generation system, identify a first checkpoint among the group of checkpoints, query a search system to identify a first portion of the data having respective third generation identifiers that are greater than the respective second generation identifiers in the first checkpoint, wherein the search system stores respective metadata of the at least some respective data from the storage system, and ingest the first portion of the data into the first retrieval-augmented generation system.

Abstract: A system can execute a search system that stores respective metadata of respective data from the storage system, wherein the respective metadata comprises respective first generation identifiers that indicate respective updates to the respective data. The system can maintain a checkpoint that comprises pairs, respective pairs of the pairs comprising identifications of at least some of the respective data stored in the storage system and respective second generation identifiers that correspond to the respective data. The system can, as part of an iteration of ingesting data from the storage system, query the search system to identify a first portion of the data having respective third generation identifiers that are greater than the respective second generation identifiers in the checkpoint, and ingest the first portion of the data into a retrieval-augmented generation system while refraining from ingesting a second portion of the data.

Abstract: A system can, based on a retrieval-augmented generation (RAG) process ingesting data, query a search system to identify at least one first portion of the data that has at least one third generation identifier that is greater than second generation identifiers in a checkpoint; create second chunks from the at least one first portion of the data; in response to second chunks being duplicates of first chunks, based on hash values, store first identifiers of the second chunks; remove the stored chunk identifiers from second identifiers of chunks that correspond to chunks that existed in a RAG system prior to the ingesting, to produce stale chunk identifiers; remove, from the RAG system, third chunks that are identified by the stale chunk identifiers, and, in response to the second chunks being determined to be unique relative to the first chunks, store the second chunks in the RAG system.

Abstract: A system can store a group of checkpoints that comprises respective pairs comprising respective identifications of at least some respective data stored in a storage system and respective second generation identifiers that correspond to the at least some respective data. The system can, based on ingesting data from the storage system and to send the data to be ingested by a first retrieval-augmented generation system, identify a first checkpoint among the group of checkpoints, query a search system to identify a first portion of the data having respective third generation identifiers that are greater than the respective second generation identifiers in the first checkpoint, wherein the search system stores respective metadata of the at least some respective data from the storage system, and ingest the first portion of the data into the first retrieval-augmented generation system.

Abstract: A system can store a checkpoint that comprises pairs and first hash values that correspond to the at least some respective first data, wherein respective pairs of the pairs comprise respective identifications of at least some respective first data stored in a storage system and respective second generation identifiers that correspond to the respective data. The system can query a search system to identify at least one first portion of the data that has at least one respective third generation identifier that is greater than the respective second generation identifiers, determine respective second hash values for respective second data of the at least one first portion of the data, identify at least one second portion of the data for which the respective second hash values do not match any of the first hash values, and ingest the at least one second portion of the data into the retrieval-augmented generation system.

Abstract: One example method includes receiving from a node, in an HSAN that includes multiple nodes, an ADD_DATA request to add an entry to a distributed ledger of the HSAN, the request comprising a user ID that identifies the node, a hash of a data segment, and a storage location of the data segment at the node, performing a challenge-and-response process with the node to verify that the node has a copy of the data that was the subject of the entry, making a determination that a replication factor X has not been met, and adding the entry to the distributed ledger upon successful conclusion of the challenge-and-response process.

Abstract: One example method includes receiving, from an entity, a proposed entry for a ledger, where the ledger is shared and accessible by multiple users and includes a whitelist and a blacklist, determining, or assigning, a credibility score and rate limiter value for the entity, comparing the credibility score and rate limiter value with respective credibility score and rate limiter value thresholds, determining that the credibility score and rate limiter value meet or exceed the respective credibility score and rate limiter value thresholds, and submitting the proposed entry to the ledger.

Abstract: One example method includes receiving from a node, in an HSAN that includes multiple nodes, an ADD_DATA request to add an entry to a distributed ledger of the HSAN, the request comprising a user ID that identifies the node, a hash of a data segment, and a storage location of the data segment at the node, performing a challenge-and-response process with the node to verify that the node has a copy of the data that was the subject of the entry, making a determination that a replication factor X has not been met, and adding the entry to the distributed ledger upon successful conclusion of the challenge-and-response process.

Abstract: One example method includes receiving from a node, in an HSAN that includes multiple nodes, an ADD_DATA request to add an entry to a distributed ledger of the HSAN, the request comprising a user ID that identifies the node, a hash of a data segment, and a storage location of the data segment at the node, performing a challenge-and-response process with the node to verify that the node has a copy of the data that was the subject of the entry, making a determination that a replication factor X has not been met, and adding the entry to the distributed ledger upon successful conclusion of the challenge-and-response process.

Abstract: One example method includes receiving a group of files, two or more of the files being of different respective file types, creating a backup saveset that includes the group of files, classifying each of the files in the backup saveset based in part on the respective file types of the files, assigning a respective storage media type to each of the classified files in the backup saveset, and transmitting the backup saveset to a storage site.

Abstract: One example method includes receiving an input that indicates selection of a cloud storage provider, receiving one or more product selection inputs, each of the product selection inputs indicating selection of a respective data protection product, receiving an input indicating an instance size, assembling the inputs together to define a data protection configuration, and automatically generating a script which, when executed by one or more hardware processors, deploys the data protection configuration in a cloud storage environment of the selected cloud storage provider.

Abstract: One example method includes receiving an input that indicates selection of a cloud storage provider, receiving one or more product selection inputs, each of the product selection inputs indicating selection of a respective data protection product, receiving an input indicating an instance size, assembling the inputs together to define a data protection configuration, and automatically generating a script which, when executed by one or more hardware processors, deploys the data protection configuration in a cloud storage environment of the selected cloud storage provider.

Abstract: One example method includes receiving an input that indicates selection of a cloud storage provider, receiving one or more product selection inputs, each of the product selection inputs indicating selection of a respective data protection product, receiving an input indicating an instance size, assembling the inputs together to define a data protection configuration, and automatically generating a script which, when executed by one or more hardware processors, deploys the data protection configuration in a cloud storage environment of the selected cloud storage provider.

Abstract: One example method includes receiving an input that indicates selection of a cloud storage provider, receiving one or more product selection inputs, each of the product selection inputs indicating selection of a respective data protection product, receiving an input indicating an instance size, assembling the inputs together to define a data protection configuration, and automatically generating a script which, when executed by one or more hardware processors, deploys the data protection configuration in a cloud storage environment of the selected cloud storage provider.

Abstract: One example method includes receiving an input that indicates selection of a cloud storage provider, receiving one or more product selection inputs, each of the product selection inputs indicating selection of a respective data protection product, receiving an input indicating an instance size, assembling the inputs together to define a data protection configuration, and automatically generating a script which, when executed by one or more hardware processors, deploys the data protection configuration in a cloud storage environment of the selected cloud storage provider.

Abstract: One example method includes receiving from a node, in an HSAN that includes multiple nodes, an ADD_DATA request to add an entry to a distributed ledger of the HSAN, the request comprising a user ID that identifies the node, a hash of a data segment, and a storage location of the data segment at the node, performing a challenge-and-response process with the node to verify that the node has a copy of the data that was the subject of the entry, making a determination that a replication factor X has not been met, and adding the entry to the distributed ledger upon successful conclusion of the challenge-and-response process.

Abstract: One example method includes receiving, from an entity, a proposed entry for a ledger, where the ledger is shared and accessible by multiple users and includes a whitelist and a blacklist, determining, or assigning, a credibility score and rate limiter value for the entity, comparing the credibility score and rate limiter value with respective credibility score and rate limiter value thresholds, determining that the credibility score and rate limiter value meet or exceed the respective credibility score and rate limiter value thresholds, and submitting the proposed entry to the ledger.

Abstract: One example method includes receiving a group of files, two or more of the files being of different respective file types, creating a backup saveset that includes the group of files, classifying each of the files in the backup saveset based in part on the respective file types of the files, assigning a respective storage media type to each of the classified files in the backup saveset, and transmitting the backup saveset to a storage site.

Abstract: One example method includes receiving an input that indicates selection of a cloud storage provider, receiving one or more product selection inputs, each of the product selection inputs indicating selection of a respective data protection product, receiving an input indicating an instance size, assembling the inputs together to define a data protection configuration, and automatically generating a script which, when executed by one or more hardware processors, deploys the data protection configuration in a cloud storage environment of the selected cloud storage provider.