Abstract: A data management services architecture includes architectural components that run in both a storage and compute domains. The architectural components redirect storage requests from the storage domain to the compute domain, manage resources allocated from the compute domain, ensure compliance with a policy that governs resource consumption, deploy program code for data management services, dispatch service requests to deployed services, and monitor deployed services. The architectural components also include a service map to locate program code for data management services, and service instance information for monitoring deployed services and dispatching requests to deployed services. Since deployed services can be stateless or stateful, the services architecture also includes state data for the stateful services, with supporting resources that can expand or contract based on policy and/or service demand. The architectural components also include containers for the deployed services.

Abstract: A data management services architecture includes architectural components that run in both a storage and compute domains. The architectural components redirect storage requests from the storage domain to the compute domain, manage resources allocated from the compute domain, ensure compliance with a policy that governs resource consumption, deploy program code for data management services, dispatch service requests to deployed services, and monitor deployed services. The architectural components also include a service map to locate program code for data management services, and service instance information for monitoring deployed services and dispatching requests to deployed services. Since deployed services can be stateless or stateful, the services architecture also includes state data for the stateful services, with supporting resources that can expand or contract based on policy and/or service demand. The architectural components also include containers for the deployed services.

Abstract: A data management services architecture includes architectural components that run in both a storage and compute domains. The architectural components redirect storage requests from the storage domain to the compute domain, manage resources allocated from the compute domain, ensure compliance with a policy that governs resource consumption, deploy program code for data management services, dispatch service requests to deployed services, and monitor deployed services. The architectural components also include a service map to locate program code for data management services, and service instance information for monitoring deployed services and dispatching requests to deployed services. Since deployed services can be stateless or stateful, the services architecture also includes state data for the stateful services, with supporting resources that can expand or contract based on policy and/or service demand. The architectural components also include containers for the deployed services.

Abstract: A data management services architecture includes architectural components that run in both a storage and compute domains. The architectural components redirect storage requests from the storage domain to the compute domain, manage resources allocated from the compute domain, ensure compliance with a policy that governs resource consumption, deploy program code for data management services, dispatch service requests to deployed services, and monitor deployed services. The architectural components also include a service map to locate program code for data management services, and service instance information for monitoring deployed services and dispatching requests to deployed services. Since deployed services can be stateless or stateful, the services architecture also includes state data for the stateful services, with supporting resources that can expand or contract based on policy and/or service demand. The architectural components also include containers for the deployed services.

Abstract: A data management services architecture includes architectural components that run in both a storage and compute domains. The architectural components redirect storage requests from the storage domain to the compute domain, manage resources allocated from the compute domain, ensure compliance with a policy that governs resource consumption, deploy program code for data management services, dispatch service requests to deployed services, and monitor deployed services. The architectural components also include a service map to locate program code for data management services, and service instance information for monitoring deployed services and dispatching requests to deployed services. Since deployed services can be stateless or stateful, the services architecture also includes state data for the stateful services, with supporting resources that can expand or contract based on policy and/or service demand. The architectural components also include containers for the deployed services.

Abstract: Methods and systems for a networked computing system are provided. One method includes detecting that a processor executable, policy decision point (PDP) has not responded to a request for accessing data associated with a storage system; predicting a response to the request using a machine-learned, request-response association maintained by a processor executable training device; and presenting the predicted response to a processor executable, policy enforcement point (PEP) for granting access to the data and denying access to the data, based on the predicted response.

Abstract: A rebuild node of a storage system can assess risk of the storage system not being able to provide a data object. The rebuild node(s) uses information about data object fragments to determine health of a data object, which relates to the risk assessment. The rebuild node obtains object fragment information from nodes throughout the storage system. With the object fragment information, the rebuild node(s) can assess object risk based, at least in part, on the object fragments indicated as existing by the nodes. To assess object risk, the rebuild node(s) treats absent object fragments (i.e., those for which an indication was not received) as lost. When too many object fragments are lost, an object cannot be rebuilt. The erasure coding technique dictates the threshold number of fragments for rebuilding an object. The risk assessment per object influences rebuild of the objects.

Abstract: Methods and systems for a networked computing system are provided. One method includes detecting that a processor executable, policy decision point (PDP) has not responded to a request for accessing data associated with a storage system; predicting a response to the request using a machine-learned, request-response association maintained by a processor executable training device; and presenting the predicted response to a processor executable, policy enforcement point (PEP) for granting access to the data and denying access to the data, based on the predicted response.

Abstract: A method includes receiving a write request to write a current data block to a Shingled Magnetic Recording (SMR) storage device. In response, the current data block is written to a current physical block in an open zone of the SMR storage device. A corresponding copy of the current data block is written to a nonvolatile memory. A determination is made of whether a wandering write error occurred during the writing of the data to the open zone. In response to the wandering write error occurring, for each of the number of written physical blocks in the open zone that have the corresponding copy in the nonvolatile memory, the data in the physical block is validated using the corresponding copy. In response to validation, the data in the corresponding copy is written as corrected data for the physical block to a new zone in the SMR storage device.

Abstract: A storage system writes an object across zones of a set of zones (“zone set”). Each zone of a zone set is contributed from an independently accessible storage medium. To create a zone set, the storage system arbitrarily selects disks to contribute a zone for membership in the zone set. This results in a fairly even distribution of zone sets throughout the storage system, which increases fault tolerance of the storage system. Although disk selection for zone set membership is arbitrary, the arbitrary selection can be from a pool of disks that satisfy one or more criteria (e.g., health or activity based criteria). In addition, weights can be assigned to disks to influence the arbitrary selection. Although manipulating the arbitrary selection with weights or by reducing the pool of disks reduces the arbitrariness, this evenly distributes zone sets while accounting for client demand and/or disk health.

Abstract: A data management services architecture includes architectural components that run in both a storage and compute domains. The architectural components redirect storage requests from the storage domain to the compute domain, manage resources allocated from the compute domain, ensure compliance with a policy that governs resource consumption, deploy program code for data management services, dispatch service requests to deployed services, and monitor deployed services. The architectural components also include a service map to locate program code for data management services, and service instance information for monitoring deployed services and dispatching requests to deployed services. Since deployed services can be stateless or stateful, the services architecture also includes state data for the stateful services, with supporting resources that can expand or contract based on policy and/or service demand. The architectural components also include containers for the deployed services.

Abstract: A data management services architecture includes architectural components that run in both a storage and compute domains. The architectural components redirect storage requests from the storage domain to the compute domain, manage resources allocated from the compute domain, ensure compliance with a policy that governs resource consumption, deploy program code for data management services, dispatch service requests to deployed services, and monitor deployed services. The architectural components also include a service map to locate program code for data management services, and service instance information for monitoring deployed services and dispatching requests to deployed services. Since deployed services can be stateless or stateful, the services architecture also includes state data for the stateful services, with supporting resources that can expand or contract based on policy and/or service demand. The architectural components also include containers for the deployed services.

Abstract: An archival cloud storage service can be created with cost efficient components for large scale data storage and can efficiently use these components. A frontend of the cloud storage service presents an asynchronous storage interface to consuming devices of the cloud storage service. Providing an asynchronous storage service interface avoids at least some of the state data overhead that accompanies a time constrained interface (e.g., a request-response based interface with timeouts in seconds). Backend nodes of the cloud storage service periodically query the frontend servers to select requests that the backend nodes can fulfill. Each backend node selects requests based on backend characteristics information, likely dynamic characteristics, of the backend node. Thus, the storage system underlying the cloud storage service can be considered a self-organizing storage system.

Abstract: Failed capacity of a distributed storage system is determined. The distributed storage system includes a plurality of storage nodes, wherein the plurality of storage nodes include at least one storage device to store data objects, wherein the data objects have been divided into constituent fragments in the distributed storage system. Protection capacity of the distributed storage system is determined. Protection capacity includes the data fragments generated to allow the data objects to be rebuilt in response to at least a part of the data objects being either lost or corrupted. A probability is determined that the failed capacity overlaps with the used capacity of the distributed storage system prior to a next periodically scheduled maintenance of the distributed storage system. In response to the probability exceeding a risk threshold, a next maintenance of the distributed storage system is scheduled that comprises reducing the failed capacity.

Abstract: A data storage system uses the free space that is not yet filled with data after the deployment of the data store. The free space is used to store additional ‘opportunistic’ protection information for stored data, possibly above and beyond the specified protection level. As the system fills up, the additional protection information is deleted to make room for more data and specified protection information.

Abstract: Systems and techniques for managing data storage are disclosed. In some aspects, a front-end node responds to a request to write an object by dividing the object into multiple source data segments. The front-end node generates redundancy data for the multiple source data segments using a rateless erasure encoding. The front-end node associates a respective subset of the redundancy data with each of the multiple source data segments, wherein each subset of redundancy data and associated source data segment form an encoded segment. The rateless erasure encoding further includes defining multiple segment-level fragments within each of the encoded segments. The front-end node transmits each of the encoded segments to a selected one of multiple storage nodes, wherein each of the selected storage nodes are selected based on a determined storage layout of the encoded segments across the multiple storage nodes.

Abstract: An archival cloud storage service can be created with cost efficient components for large scale data storage and can efficiently use these components. A frontend of the cloud storage service presents an asynchronous storage interface to consuming devices of the cloud storage service. Providing an asynchronous storage service interface avoids at least some of the state data overhead that accompanies a time constrained interface (e.g., a request-response based interface with timeouts in seconds). Backend nodes of the cloud storage service periodically query the frontend servers to select requests that the backend nodes can fulfill. Each backend node selects requests based on backend characteristics information, likely dynamic characteristics, of the backend node. Thus, the storage system underlying the cloud storage service can be considered a self-organizing storage system.

Abstract: Technology is disclosed for deferring storage operations (e.g., writes or reads) during hostile events. When a data storage device experiences a hostile event, e.g., a vibration, shock, etc. contact by a head of the data storage device with a disk surface can cause errors or indeed damage. The technology can cause a data storage device to suspend storage operations until the hostile event is no longer detected.

Abstract: Failed capacity of a distributed storage system is determined. The distributed storage system includes a plurality of storage nodes, wherein the plurality of storage nodes include at least one storage device to store data objects, wherein the data objects have been divided into constituent fragments in the distributed storage system. Protection capacity of the distributed storage system is determined. Protection capacity includes the data fragments generated to allow the data objects to be rebuilt in response to at least a part of the data objects being either lost or corrupted. A probability is determined that the failed capacity overlaps with the used capacity of the distributed storage system prior to a next periodically scheduled maintenance of the distributed storage system. In response to the probability exceeding a risk threshold, a next maintenance of the distributed storage system is scheduled that comprises reducing the failed capacity.

Abstract: A data storage system uses the free space that is not yet filled with data after the deployment of the data store. The free space is used to store additional ‘opportunistic’ protection information for stored data, possibly above and beyond the specified protection level. As the system fills up, the additional protection information is deleted to make room for more data and specified protection information.