Abstract: Providing for high availability in a data analytics pipeline without replicas, including: creating a data analytics pipeline, wherein each component of the data analytics pipeline is deployed within a container; creating a failover container; detecting that a component within the data analytics pipeline has failed; and responsive to detecting that the component within the data analytics pipeline has failed, deploying the component within the data analytics pipeline that has failed in the failover container.

Abstract: A method for execution by a processing system in dispersed storage and task network (DSTN) that includes a processor, includes: identifying a slice name of a slice in error of a set of slices stored in a set of dispersed storage (DS) units; identifying a number of slice errors of the set of slices; generating a queue entry that includes the slice name of the slice in error, a rebuilding task indicator, an identity of the set of slices, and the number of slice errors; identifying a rebuilding queue based on the number of slice errors, wherein the rebuilding queue is associated with one of: the set of DS units or another set of DS units; and facilitating storing the queue entry in the identified rebuilding queue.

Abstract: Systems and method for determining a storage location according to legal requirements. According to one example a dispersed storage (DS) processing unit receives data to store in a dispersed storage network (DSN) and assesses attributes of data to produce a determination of whether the data falls into one or more defined legal categories. The DS processing unit also determines one or more attributes of a storage location in the DSN. The DS processing unit then determines whether or not to select the storage location based on the determination and based on one or more attributes of the storage location. If storing data of the identified category would be illegal the DS processing unit evaluates the another storage location, and so on. If no available storage location is found the DS processing unit returns an error indicating that no legal storage location is found.

Abstract: A method begins by one or more processing modules of one or more computing devices of a dispersed storage network (DSN) receiving a delete data request for a data object of a set of data objects, where each data object of the set of data objects has substantially identical data to other data objects of the set of data objects and where only one plurality of sets of encoded data slices corresponding to one data object of the set of data objects is stored in memory of the DSN. The method continues with the one or more processing modules determining whether the data object is associated with a data ID and when the data object is associated with a data ID, identifying linking information associated with the data object. Linking information is updated to indicate that the set of data objects includes one fewer copy of the data object and then it is determined whether the Data ID is associated with a link-object and when the Data ID is associated with a link-object the link-object is deleted from the DSN memory.

Abstract: A system and method for efficiently distributing data among multiple storage devices. A data storage array receives read and write requests from multiple client computers. The data storage array includes multiple storage devices, each with multiple allocation units (AUs). A storage controller within the data storage array determines a RAID layout for use in storing data. In response to determining a failure of a first AU, the storage controller begins reconstructing in a second AU the data stored in the first AU. For read and write requests targeting data in the first AU, the request is serviced by the first AU responsive to determining no error occurs when accessing the first AU.

Abstract: A system and method for utilizing unmapped and unknown states in a storage system. When a first portion of a first medium is determined to be unreachable from any other mediums, the first portion of the first medium may be put into an unmapped state, and its data may be discarded and the corresponding storage locations may be freed. During replication of the first medium to a replica storage array, the state of the first portion of the first medium may be translated from the unmapped state into an unknown state on the replica storage array. If another storage array has the data of the first portion of the first medium, this data may be used to overwrite the first portion of the first medium on the replica storage array, converting the first portion of the first medium from the unknown state into the mapped state.

Abstract: A method for managing processing power in a storage system is provided. The method includes providing a plurality of blades, each of a first subset having a storage node and storage memory, and each of a second, differing subset having a compute-only node. The method includes distributing authorities across the plurality of blades, to a plurality of nodes including at least one compute-only node, wherein each authority has ownership of a range of user data.

Abstract: A method for page writes for triple or higher level cell flash memory is provided. The method includes receiving data in a storage system, from a client that is agnostic of page write requirements for triple or higher level cell flash memory, wherein the page write requirements specify an amount of data and a sequence of writing data for a set of pages to assure read data coherency for the set of pages. The method includes accumulating the received data, in random-access memory (RAM) in the storage system to satisfy the page write requirements for the triple or higher level cell flash memory in the storage system. The method includes writing at least a portion of the accumulated data in accordance with the page write requirements, from the RAM to the triple level cell, or the higher level cell, flash memory in the storage system as an atomic write.

Abstract: Providing authorization and authentication in a cloud for a user of a storage array includes: receiving, by a storage array access module from a client-side array services module, a token representing authentication of user credentials and authorized access privileges defining one or more storage array services accessible by the user, where the token is generated by a cloud-based security module upon authentication of the user credentials and identification of authorized access privileges for the user; receiving, by the storage array access module from the user, a user access request to one or more storage array services; and determining, by the storage array access module, whether to grant the user access request in dependence upon the authorized access privileges represented by the token.

Abstract: A method for execution by a dispersed storage and task (DST) execution unit includes identifying a plurality of pending operations. A resource availability level to support execution of at least one of the plurality of pending operations is determined, and a required resource level to execute the at least one of the plurality of pending operations is determined. A balance factor between at least two types of the plurality of pending operations is determined based on the resource availability level and the required resource level. Determination of required timing of the execution of the at least one of the plurality of pending operations is coordinated with at least one other DST execution unit. An operation execution schedule is updated based on the required resource levels, the resource availability level, the balance factor, and the required timing of the execution.

Abstract: A method for storing data in a storage system having solid-state memory is provided. The method includes determining a first portion of the solid-state memory having a lower read latency than a second portion of the solid-state memory, relative to each other or to a threshold. The method includes writing data bits of erasure coded data to the first portion of the solid-state memory, and writing one or more parity bits of the erasure coded data to the second portions of the solid-state memory. A storage system is also provided.

Abstract: A method of encryption management with host-side data reduction includes identifying data to be written to a storage array and compressing the data to generate compressed data. The method further includes encrypting the compressed data to generate an encrypted data packet. The method further includes adding, by a processing device of a host, a padding bit pattern to the encrypted data packet to generate a data block for storage. The method further includes sending the data block to the storage array.

Abstract: A computing device includes an interface configured to interface and communicate with a dispersed storage network (DSN), a memory that stores operational instructions, and a processing module operably coupled to the interface and memory such that the processing module, when operable within the computing device based on the operational instructions, is configured to perform various operations. For example, the computing device generates a prioritized request that includes at least one of a task for execution or a priority level based on information stored within a storage unit (SU) of a plurality of storage units (SUs) implemented within the DSN. Note that the information corresponds to a data object that is related to a set of encoded data slices (EDSs) that are distributedly stored within the DSN. The computing device then transmits the prioritized request to the SU and receives, from the SU, a response to the prioritized request.

Abstract: Methods for use in a storage unit of a dispersed storage network (DSN) to securely store cryptographic key information. In various examples, the storage unit receives a slice access request relating to a key slice generated by performing a dispersed storage error encoding function on an encryption key. When the slice access request includes a request to store the key slice, the storage unit encrypts the key slice using a local key and stores the encrypted key slice (e.g., in a key region of a storage vault). When the slice access request includes a request to recover a key slice stored in the storage unit, the encrypted key slice is recovered from memory and decrypted using the local key to produce a decrypted key slice for provision to the requesting entity. For rebuilding operations, the storage unit may instead return a zero information gain (ZIG) representation of the key slice.

Abstract: Ensuring resiliency to storage device failures in a storage system, including: determining a number of storage device failures within a particular write group that are to be tolerated by the storage system; for a plurality of datasets stored within the storage system, writing each dataset to at least a predetermined number of storage devices within the particular write group, wherein the predetermined number of storage devices is greater than the number of storage device failures within the particular write group that are to be tolerated by the storage system; and responsive to recovering from a system interruption: determining a number of readable storage devices that contain a copy of the dataset; and if the number of readable storage devices that contain a copy of the dataset is not greater than the number of failures that are to be tolerated, writing the dataset to one or more additional storage devices.

Abstract: A method of applying an address space to data storage in a non-volatile solid-state storage is provided. The method includes receiving a plurality of portions of user data for storage in the non-volatile solid-state storage and assigning to each successive one of the plurality of portions of user data one of a plurality of sequential, nonrepeating addresses of an address space. The address range of the address space exceeds a maximum number of addresses expected to be applied during a lifespan of the non-volatile solid-state storage. The method includes writing each of the plurality of portions of user data to the non-volatile solid-state storage such that each of the plurality of portions of user data is identified and locatable for reading via the one of the plurality of sequential, nonrepeating addresses of the address space.

Abstract: A system and method for offset protection data in a RAID array. A computer system comprises client computers and data storage arrays coupled to one another via a network. A data storage array utilizes solid-state drives and Flash memory cells for data storage. A storage controller within a data storage array is configured to store user data in a first page of a first storage device of the plurality of storage devices; generate intra-device protection data corresponding to the user data, and store the intra-device protection data at a first offset within the first page. The controller is further configured to generate inter-device protection data corresponding to the first page, and store the inter-device protection data at a second offset within a second page in a second storage device of the plurality of storage devices, wherein the first offset is different from the second offset.

Abstract: Systems, methods, and computer readable storage mediums for discovering volumes which are good candidates for space reclamation. A storage subsystem identifies the file system storage capacity for a given volume from the file system metadata of the given volume. Then, the storage subsystem compares the file system capacity of the given volume to the allocated capacity on the storage subsystem. If the allocated capacity is greater than the file system capacity by a given threshold, the storage subsystem marks the given volume as a candidate for space reclamation and generates an alert to the user to reclaim the space of the given volume.

Abstract: A storage cluster includes a plurality of storage nodes. Each of the plurality of storage nodes includes nonvolatile solid-state memory and each of the plurality of storage nodes is configured to cooperate with others of the plurality of storage nodes having differing storage capacities in applying erasure coding. The plurality of storage nodes are configured to distribute the user data and metadata throughout the plurality of storage nodes.

Abstract: A system and method for offset protection data in a RAID array. A computer system comprises client computers and data storage arrays coupled to one another via a network. A data storage array utilizes solid-state drives and Flash memory cells for data storage. A storage controller within a data storage array is configured to store user data in a first page of a first storage device of the plurality of storage devices; generate intra-device protection data corresponding to the user data, and store the intra-device protection data at a first offset within the first page. The controller is further configured to generate inter-device protection data corresponding to the first page, and store the inter-device protection data at a second offset within a second page in a second storage device of the plurality of storage devices, wherein the first offset is different from the second offset.