Abstract: Embodiments of the present systems and methods may provide the capability ensure that data is persisted and accessed correctly without depending on eventually consistent list operations on the object store. For example, in an embodiment, a computer-implemented method for data distribution may comprise attempting to persist a plurality of data parts from a plurality of processing tasks, generating a manifest including information indicating those attempts to persist data parts that have succeeded, and persisting the manifest with the data parts that have been successfully persisted.

Abstract: A system and method for recovering a database and restoring an index following a failure of the database is disclosed. The method receives a change to a record in the database. The change is stored in a persistent data store, the persistent data store is divided into a plurality of segments. The volatile index is updated in volatile memory with a pointer to the record in the persistent data store. A shadow index is generated in the persistent data store, where the shadow index is a persistent copy of the volatile index and is not updated at the same time as the volatile index. The shadow thread is executed on the plurality of records where the shadow thread scans each record in the persistent storage device to populate and update the shadow index, wherein the shadow thread operates as a background operation on the persistent data store.

Abstract: Machines, systems and methods for enhancing data recovery in a data storage system, the method comprising determining whether one or more data storage mediums in a data storage system are unavailable; determining data that are at a risk of loss, due to said one or more data storage mediums being unavailable; from among the data that is determined to be at the risk of loss, identifying data that is highly vulnerable to loss; and creating one or more temporary replicas of the data that is highly vulnerable to loss.

Abstract: Machines, systems and methods for recovering data objects in a distributed data storage system, the method comprising storing one or more replicas of a first data object on one or more clusters in one or more data centers connected over a data communications network; recording health information about said one or more replicas, wherein the health information comprises data about availability of a replica to participate in a restoration process; calculating a query-priority for the first data object; querying, based on the calculated query-priority, the health information for the one or more replicas to determine which of the one or more replicas is available for restoration of the object data; calculating a restoration-priority for the first data object based on the health information for the one or more replicas; and restoring the first data object from the one or more of the available replicas, based on the calculated restoration-priority.

Abstract: Machines, systems and methods for recovering data objects in a distributed data storage system, the method comprising storing one or more replicas of a first data object on one or more clusters in one or more data centers connected over a data communications network; recording health information about said one or more replicas, wherein the health information comprises data about availability of a replica to participate in a restoration process; calculating a query-priority for the first data object; querying, based on the calculated query-priority, the health information for the one or more replicas to determine which of the one or more replicas is available for restoration of the object data; calculating a restoration-priority for the first data object based on the health information for the one or more replicas; and restoring the first data object from the one or more of the available replicas, based on the calculated restoration-priority.

Abstract: Embodiments of the present systems and methods may provide the capability ensure that data is persisted and accessed correctly without depending on eventually consistent list operations on the object store. For example, in an embodiment, a computer-implemented method for data distribution may comprise attempting to persist a plurality of data parts from a plurality of processing tasks, generating a manifest including information indicating those attempts to persist data parts that have succeeded, and persisting the manifest with the data parts that have been successfully persisted.

Abstract: Machines, systems and methods for recovering data objects in a distributed data storage system, the method comprising storing one or more replicas of a first data object on one or more clusters in one or more data centers connected over a data communications network; recording health information about said one or more replicas, wherein the health information comprises data about availability of a replica to participate in a restoration process; calculating a query-priority for the first data object; querying, based on the calculated query-priority, the health information for the one or more replicas to determine which of the one or more replicas is available for restoration of the object data; calculating a restoration-priority for the first data object based on the health information for the one or more replicas; and restoring the first data object from the one or more of the available replicas, based on the calculated restoration-priority.

Abstract: Machines, systems and methods for recovering data objects in a distributed data storage system, the method comprising storing one or more replicas of a first data object on one or more clusters in one or more data centers connected over a data communications network; recording health information about said one or more replicas, wherein the health information comprises data about availability of a replica to participate in a restoration process; calculating a query-priority for the first data object; querying, based on the calculated query-priority, the health information for the one or more replicas to determine which of the one or more replicas is available for restoration of the object data; calculating a restoration-priority for the first data object based on the health information for the one or more replicas; and restoring the first data object from the one or more of the available replicas, based on the calculated restoration-priority.

Abstract: A method for storage systems improvement includes collecting information that indicates one or more failure correlations for disks in a storage system. The disks are then separated into a plurality of virtual failure domains based on the indicated one or more failure correlations. The method then determines that all data objects of a set of redundant data objects are included in a first virtual failure domain. Responsive to determining that all data objects of the set of redundant data objects are included in the first virtual failure domain, the method then migrates at least one data object of the set of redundant data objects from a first disk in the first virtual failure domain to a second disk in a second virtual failure domain.

Abstract: Machines, systems and methods for enhancing data recovery in a data storage system, the method comprising determining whether one or more data storage mediums in a data storage system are unavailable; determining data that are at a risk of loss, due to said one or more data storage mediums being unavailable; from among the data that is determined to be at the risk of loss, identifying data that is highly vulnerable to loss; and creating one or more temporary replicas of the data that is highly vulnerable to loss.

Abstract: Machines, systems and methods for enhancing data recovery in a data storage system, the method comprising determining whether one or more data storage mediums in a data storage system are unavailable; determining data that are at a risk of loss, due to said one or more data storage mediums being unavailable; from among the data that is determined to be at the risk of loss, identifying data that is highly vulnerable to loss; and creating one or more temporary replicas of the data that is highly vulnerable to loss.

Abstract: Aspects of the present invention include a method, system and computer program product for performing data deduplication for eventually consistent distributed data storage (DDS) system. The method includes receiving data content from one or more clients by a DDS system, wherein the one or more clients do not coordinate transmitting of the data content. The method also includes calculating a hash for the data content by the distributed data storage system, writing the data content to an object used for data deduplication, wherein a name of the object is based on the hash and determining whether the data content is present in the distributed data storage system based on the name of an object previously stored on the DDS system. The method further includes keeping track of a number of references to the data content and delaying deletion of the data content for a predetermined period of time.

Abstract: Embodiments include evaluating durability and availability of a distributed storage system. Aspects include receiving a configuration of the distributed storage system, identifying a failure model for each component of the distributed storage system. Aspects also include generating a series of failure events for each component of the distributed storage system based on the failure model and calculating a recovery time for each failed component based on a network recovery bandwidth, a disk recovery bandwidth, a total capacity of simultaneous failed storage devices and a resiliency scheme used by the in the distributed storage system. Aspects further include collecting data regarding the series of failures and the recovery times, calculating an observed distribution of component failures from the collected data and calculating the availability and durability of the distributed storage system based on the observed distribution of component failures and using probabilistic durability and availability models.

Abstract: A method for storage systems improvement includes collecting information that indicates one or more failure correlations for disks in a storage system. The disks are then separated into a plurality of virtual failure domains based on the indicated one or more failure correlations. The method then determines that all data objects of a set of redundant data objects are included in a first virtual failure domain. Responsive to determining that all data objects of the set of redundant data objects are included in the first virtual failure domain, the method then migrates at least one data object of the set of redundant data objects from a first disk in the first virtual failure domain to a second disk in a second virtual failure domain.

Abstract: Embodiments include evaluating durability and availability of a distributed storage system. Aspects include receiving a configuration of the distributed storage system, identifying a failure model for each component of the distributed storage system. Aspects also include generating a series of failure events for each component of the distributed storage system based on the failure model and calculating a recovery time for each failed component based on a network recovery bandwidth, a disk recovery bandwidth, a total capacity of simultaneous failed storage devices and a resiliency scheme used by the in the distributed storage system. Aspects further include collecting data regarding the series of failures and the recovery times, calculating an observed distribution of component failures from the collected data and calculating the availability and durability of the distributed storage system based on the observed distribution of component failures and using probabilistic durability and availability models.

Abstract: Machines, systems and methods for handling a client request in a hierarchical multi-tenant data storage system, the method comprising processing a request in subtasks, wherein a subtask is executed with a minimal set of privileges associated with a specific subtenant; extracting a claimed n-level hierarchy of a tenant and sub-tenant identities from the request; extracting authentication signatures or credentials that correspond to a level in the hierarchy; for a first level in the hierarchy, sending the request to a dedicated subtenant authenticator with privilege to validate credentials for a subtenant at the first level; and receiving a confirmation from the dedicated subtenant authenticator, whether the request is authentic.

Abstract: Methods, storage systems and computer program products implement embodiments of the present invention that include defining, for an entity, a policy access control list including one or more access rules, each of the access rules including one or more user conditions and one or more entity conditions. Upon receiving a request from a user to access a given entity, one or more user attributes associated with the user and one or more entity attributes associated with the given entity are identified. For each of the access rules, the one or more user conditions are applied to the one or more user attributes, the one or more entity conditions are applied to the one or more entity attributes. Access to the given content entity is granted to the user upon determining that a minimum threshold of the one or more user conditions and the one or more entity conditions are met.

Abstract: Machines, systems and methods for optimizing data replication in a distributed storage network, the method comprising determining a need to create a replica for a data item in a remote failure zone in a data storage network; creating a temporary replica of the data item in a local failure zone defined in the data storage network, in response to determining that it is beneficial to create the temporary replica in the local failure zone based on a cost versus reliability improvement analysis; attempting to create the replica in the remote failure zone; and removing the temporary replica from the local failure zone, in response to successfully creating the replica in the remote failure zone.

Abstract: Machines, systems and methods for handling a client request in a hierarchical multi-tenant data storage system, the method comprising processing a request in subtasks, wherein a subtask is executed with a minimal set of privileges associated with a specific subtenant; extracting a claimed n-level hierarchy of a tenant and sub-tenant identities from the request; extracting authentication signatures or credentials that correspond to a level in the hierarchy; for a first level in the hierarchy, sending the request to a dedicated subtenant authenticator with privilege to validate credentials for a subtenant at the first level; and receiving a confirmation from the dedicated subtenant authenticator, whether the request is authentic.

Abstract: Methods, storage systems and computer program products implement embodiments of the present invention that include defining, for an entity, a policy access control list including one or more access rules, each of the access rules including one or more user conditions and one or more entity conditions. Upon receiving a request from a user to access a given entity, one or more user attributes associated with the user and one or more entity attributes associated with the given entity are identified. For each of the access rules, the one or more user conditions are applied to the one or more user attributes, the one or more entity conditions are applied to the one or more entity attributes. Access to the given content entity is granted to the user upon determining that a minimum threshold of the one or more user conditions and the one or more entity conditions are met.