Abstract: Providing for high availability in a data analytics pipeline without replicas, including: creating a data analytics pipeline, wherein each component of the data analytics pipeline is deployed within a container; creating a failover container; detecting that a component within the data analytics pipeline has failed; and responsive to detecting that the component within the data analytics pipeline has failed, deploying the component within the data analytics pipeline that has failed in the failover container.

Abstract: Ensuring resiliency to storage device failures in a storage system, including: determining a number of storage device failures within a particular write group that are to be tolerated by the storage system; for a plurality of datasets stored within the storage system, writing each dataset to at least a predetermined number of storage devices within the particular write group, wherein the predetermined number of storage devices is greater than the number of storage device failures within the particular write group that are to be tolerated by the storage system; and responsive to recovering from a system interruption: determining a number of readable storage devices that contain a copy of the dataset; and if the number of readable storage devices that contain a copy of the dataset is not greater than the number of failures that are to be tolerated, writing the dataset to one or more additional storage devices.

Abstract: An implementation of the disclosure provides a system comprising a storage array comprising a storage controller coupled to the storage array. The storage controller comprising a processing device to remap a plurality of deduplication references in a deduplication map to point to an earlier occurrence of duplicate data of a data block for the deduplication map. The processing device further to update an entry of the deduplication map associated with the plurality of deduplication references with a record indicating that the entry is no longer referenced and trim the entry from the deduplication map that is associated with the record.

Abstract: A method begins with a processing module selecting one of a plurality of dispersed storage (DS) processing modules for facilitating access to a dispersed storage network (DSN) memory. The method continues with the processing module sending a DSN memory access request to the one of the plurality of DS processing modules. The method continues with the processing module selecting another one of the plurality of DS processing modules when no response is received within a given time frame or when the response to the access request does not include an access indication. The method continues with the processing module sending the DSN memory access request to the another one of the plurality of DS processing modules.

Abstract: A method of failure detection in a storage system is performed by the storage system. The method includes detecting a failure in a nonvolatile random access memory device that is in or coupled to a storage device having storage memory. The storage system has multiple NVRAM devices and multiple storage devices that have storage memory. The method includes taking a portion or all of the NVRAM device offline. Taking a portion or all of the NVRAM device offline is responsive to detecting the failure. Taking a portion or all of the NVRAM device off-line is while keeping online the storage memory of the storage device, sufficient ones of the NVRAM devices, and sufficient ones of the storage devices to provide reliable access to data and metadata in the storage system.

Abstract: A method begins with a processing module transmitting list slice requests to distributed storage network (DSN) storage units regarding storage errors associated with a data segment. The method continues with the processing module receiving list slice response messages and depending on the list slice response messages, determining whether a first threshold priority or a second threshold priority number of error-free EDSs associated with the first data segment has been stored. The method proceeds with the processing module, depending on the number of error-free EDSs associated with the first data segment that have been stored, issuing read slice requests and rebuilding one or more EDSs associated with the data segment.

Abstract: A method includes obtaining, by a computing device of a storage network, provenance information for data associated with a set of storage units of the storage network, where the data is error encoded into a set of encoded data slices, in accordance with error encoding parameters, for storage in the set of storage units. The method further includes determining, by the computing device, probable error locations associated with the set of storage units based on the provenance information. The method further includes scanning, by the computing device, the probable error locations to determine whether an error exists for the set of encoded data slices.

Abstract: Methods and apparatus for determining if a value is stored in a hash table is provided. The method includes breaking the value into address bits, prefix bits, and signature bits. The method includes determining a container in a compressed index at an address specified by the address bits, the container comprised of a prefix table with bits set corresponding to the prefix bits determined by aggregate values associated with the container and a signature table containing the signature bits determined by the aggregate values associated with the container. The method includes determining a result based on a function of the prefix and signature tables and the determined prefix and signature bits.

Abstract: Data associated with a write request is stored at a storage device of multiple solid-state storage devices. A determination as to whether the data stored at the storage device is readable is made by determining whether a number of subsequent programming operations have been performed since the data was stored at the storage device. A notification that the stored data is readable from the storage device is generated upon determining that the data is readable.

Abstract: A method for execution by a dispersed storage and task (DST) processing unit includes obtaining audit records for an audit object and determining when the audit object is complete. When the audit object is complete, aggregating the audit records of the audit object within the audit object by generating the audit object to include the audit records; generating identifier (ID) information and generating integrity information. Fields of the audit object are populated with the audit records, the ID information, and the integrity information and a name of the audit object is determined for storage of the audit object and the name of the audit object in a dispersed storage network (DSN).

Abstract: Mediating between storage systems synchronously replicating a dataset, including: requesting, by a first storage system in response to detecting a triggering event, a lock for a shared resource from a mediation service; requesting, by a second storage system in response to detecting the triggering event, the lock for the shared resource from the mediation service; and responsive to acquiring the lock from the mediation service, the first storage system, instead of the second storage system, processing data storage requests directed to the dataset that is synchronously replicated across the first storage system and the second storage system.

Abstract: A storage system is provided. The storage system includes a plurality of storage nodes, each of the plurality of storage nodes having a plurality of storage units with storage memory. The system includes a first network coupling the plurality of storage nodes and a second network coupled to at least a subset of the plurality of storage units of each of the plurality of storage nodes such that one of the plurality of storage units of a first one of the plurality of storage nodes can initiate or relay a command to one of the plurality of storage units of a second one of the plurality of storage nodes via the second network without the command passing through the first network.

Abstract: A method for dynamically configuring a storage system to facilitate independent scaling of resources is provided. The method includes detecting a change to a topology of the storage system consisting of different sets of blades configured within one of a plurality of chassis and reconfiguring the storage system to change an allocation of resources to one or more authorities responsive to detecting the change to the topology of the storage system.

Abstract: A method for execution by a processing module of a distributed storage includes transmitting a request to retrieve a set of encoded data slices (EDSs) to a plurality of storage nodes followed by receiving a threshold number of EDSs from one or more of the plurality of storage nodes, and decoding the EDSs to produce a transposed encrypted data segment. The method continues with the processing module partitioning the encrypted data segment into an encoded encryption key and encrypted data, performing a hash function on the encrypted data to produce a digest resultant and combining the digest resultant with the encoded encryption key to generate combined key data. The method then continues with decoding the combined key data to recover an encryption key and decrypting the encrypted data using the encryption key to recover a data segment.

Abstract: A method for use by a source storage unit of a dispersed storage network includes identifying a slice name corresponding to a slice to migrate from a source storage unit to a destination storage unit; sending the slice to migrate to the destination storage unit; generating a slice verification request and sending the slice verification request to the destination storage unit; receiving an integrity value from the destination storage unit; and determining when the integrity value compares favorably to the slice verification request. When the source storage unit determines that the integrity value compares favorably to the slice verification request, a slice name assignment associated with the slice name is updated and the slice is deleted from the source storage unit.

Abstract: A method for distribution of directories in a storage system is provided. The method includes distributing information, regarding location in the storage system of a plurality of files in a directory, to a plurality of owners in a plurality of storage nodes of the storage system, wherein ownership of differing subsets of the plurality of files of the directory is distributed among differing owners in differing storage nodes. The method includes receiving a request for location information in the storage system of a file of the plurality of files in the directory and determining, based on a file name of the file and an identity of the directory, which of the plurality of owners has ownership of the file and the location information for the file.

Abstract: A method for a transactional commit in a storage unit is provided. The method includes receiving a logical record from a storage node into a transaction engine of a storage unit of the storage node and writing the logical record into a data structure of the transaction engine. The method includes writing, to a command queue of the transaction engine, an indication to perform an atomic update using the logical record and transferring each portion of the logical record from the data structure of the transaction engine to non-persistent memory of the storage unit as a committed transaction. A storage unit for a storage system is also provided.

Abstract: A method includes receiving a set of difference lists from a set of storage units of the DSN, where the set of storage units store a plurality of sets of encoded data slices, wherein a first difference list identifies first encoded data slices that have first indicators that are different than corresponding first indicators of the first encoded data slices included in a shared common list. The method continues by determining, for a set of encoded data slices of the plurality of sets of encoded data slices, whether a storage inconsistency exists based on one or more indicators associated with the encoded data slice included in the set of difference lists. When the storage inconsistency exists, the method continues by flagging for rebuilding encoded data slices of the set of encoded data slices associated with the storage inconsistency.

Abstract: A storage network operates by: generating metadata for a data object; first disperse storage error encoding the metadata to produce a set of metadata slices, wherein the first disperse storage error encoding utilizes first dispersal parameters, the first dispersal parameters including a first decode threshold of 1; generating sets of first data slices via a second disperse storage error encoding of data segments associated with the data object, wherein the second disperse storage error encoding utilizes second dispersal parameters, the second dispersal parameters different from the first dispersal parameters and the second dispersal parameters including a second decode threshold greater than 1; producing an additional data segment associated with the data object wherein the additional data segment is different from the data segments and the metadata; and third disperse storage error encoding the additional data segment to produce a set of second data slices, wherein the third disperse storage error encoding u

Abstract: Methods and systems for use in a dispersed storage network to prioritize data rebuilding operations. In various examples, a device receives data loss information from a set of storage units. Based on the data loss information, data slice errors are detected regarding data stored by the storage units, and corresponding rebuild requests are issued to the storage units. The device also determines a rebuild rate based on a rate of rebuilding associated with the rebuild requests. The device further receives storage error information regarding errors associated with storage requests to the storage units and, based on the data loss information and storage error information, determines a data loss rate. The rebuild rate and the data loss rate are provided to the storage units for use in prioritizing the rebuild requests, such that when the rebuild rate compares unfavorably to the data loss rate, rebuild requests are prioritized over storage requests.