Abstract: Embodiments of the present systems and methods may provide the capability ensure that data is persisted and accessed correctly without depending on eventually consistent list operations on the object store. For example, in an embodiment, a computer-implemented method for data distribution may comprise attempting to persist a plurality of data parts from a plurality of processing tasks, generating a manifest including information indicating those attempts to persist data parts that have succeeded, and persisting the manifest with the data parts that have been successfully persisted.

Abstract: Embodiments of the present systems and methods may provide techniques to provide host side encryption while maintaining compression and deduplication benefits and providing communication between the host and the storage system that does not leak information about the data compressibility/deduplication properties. For example, in an embodiment, a method may comprise compressing, at a computer system, an original sector of data, generating a new sector of data including a first part including metadata and padding data, and a second part including the original sector of data that has been compressed and encrypted using a data encryption key (DEK), encrypting, at the computer system, the new sector of data using a data reduction key (DRK), and transmitting, at the computer system, the encrypted new sector of data to a storage system.

Abstract: An example operation may include one or more of dividing a data object into a plurality of parts, hashing the data object to generate a hashed data object and hashing the plurality of parts to generate a plurality of hashed parts, generating a plurality of slices where each slice includes the hashed data object and a different respective hashed part, and distributing the plurality of slices across a plurality of blockchain peers, respectively.

Abstract: Embodiments of the present systems and methods may provide techniques to provide simple and accurate estimate of memory requirements for application invocation in a serverless environment. For example, a method may comprise selecting sample invocations of functions as a service from a larger plurality of invocations, submitting for execution the plurality of sample invocations and, for each sample invocation, submitting a specification of a memory size to be used for execution of each sample invocation, determining, whether the specification of the memory size to be used for execution of each sample invocation results in unsuccessful execution of at least some of the sample invocations due to insufficient memory and, if so, adjusting the specification of the memory size for at least some of the sample invocations, and submitting for execution at least those invocations in the larger plurality of invocations that were not included in the plurality of sample invocations.

Abstract: An example operation may include one or more of dividing a data object into a plurality of parts, hashing the data object to generate a hashed data object and hashing the plurality of parts to generate a plurality of hashed parts, generating a plurality of slices where each slice includes the hashed data object and a different respective hashed part, and distributing the plurality of slices across a plurality of blockchain peers, respectively.

Abstract: Embodiments of the present systems and methods may provide techniques to provide simple and accurate estimate of memory requirements for application invocation in a serverless environment. For example, a method may comprise selecting sample invocations of functions as a service from a larger plurality of invocations, submitting for execution the plurality of sample invocations and, for each sample invocation, submitting a specification of a memory size to be used for execution of each sample invocation, determining, whether the specification of the memory size to be used for execution of each sample invocation results in unsuccessful execution of at least some of the sample invocations due to insufficient memory and, if so, adjusting the specification of the memory size for at least some of the sample invocations, and submitting for execution at least those invocations in the larger plurality of invocations that were not included in the plurality of sample invocations.

Abstract: In some examples, a system for executing instructions can include a processor to detect data to be transmitted to a storage device in response to a write operation. The processor can also determine that the data comprises a compressible characteristic that enables compression of the data to a size below a threshold value. Additionally, the processor can generate a modified data block by encrypting the compressed data, and adding a padding to the compressed and encrypted data. Furthermore, the processor can transmit the modified data block to the storage device.

Abstract: Embodiments of the present systems and methods may provide techniques to provide host side encryption while maintaining compression and deduplication benefits and providing communication between the host and the storage system that does not leak information about the data compressibility/deduplication properties. For example, in an embodiment, a method may comprise compressing, at a computer system, an original sector of data, generating a new sector of data including a first part including metadata and padding data, and a second part including the original sector of data that has been compressed and encrypted using a data encryption key (DEK), encrypting, at the computer system, the new sector of data using a data reduction key (DRK), and transmitting, at the computer system, the encrypted new sector of data to a storage system.

Abstract: In some examples, a system for executing instructions can include a processor to detect data to be transmitted to a storage device in response to a write operation. The processor can also determine that the data comprises a compressible characteristic that enables compression of the data to a size below a threshold value. Additionally, the processor can generate a modified data block by encrypting the compressed data, and adding a padding to the compressed and encrypted data. Furthermore, the processor can transmit the modified data block to the storage device.

Abstract: A system and method for recovering a database and restoring an index following a failure of the database is disclosed. The method receives a change to a record in the database. The change is stored in a persistent data store, the persistent data store is divided into a plurality of segments. The volatile index is updated in volatile memory with a pointer to the record in the persistent data store. A shadow index is generated in the persistent data store, where the shadow index is a persistent copy of the volatile index and is not updated at the same time as the volatile index. The shadow thread is executed on the plurality of records where the shadow thread scans each record in the persistent storage device to populate and update the shadow index, wherein the shadow thread operates as a background operation on the persistent data store.

Abstract: An object-based data storage system includes a memory and a processor for executing machine executable instructions configured for implementing logical containers for data objects each having a global identifier. The containers are configured for storing metadata including a first parameterization value descriptive of a number of storage nodes and a second parameterization value descriptive of a classification of the data objects. The machine executable instructions are further configured for implementing a first object storage ring for addressing storage locations across the multiple storage nodes using a surjective function.

Abstract: Embodiments of the present systems and methods may provide the capability ensure that data is persisted and accessed correctly without depending on eventually consistent list operations on the object store. For example, in an embodiment, a computer-implemented method for data distribution may comprise attempting to persist a plurality of data parts from a plurality of processing tasks, generating a manifest including information indicating those attempts to persist data parts that have succeeded, and persisting the manifest with the data parts that have been successfully persisted.

Abstract: An object-based data storage system includes a memory and a processor for executing machine executable instructions configured for implementing logical containers for data objects each having a global identifier. The containers are configured for storing metadata including a first parameterization value descriptive of a number of storage nodes and a second parameterization value descriptive of a classification of the data objects. The machine executable instructions are further configured for implementing a first object storage ring for addressing storage locations across the multiple storage nodes using a surjective function.

Abstract: A computing facility, including a storage management system belonging to a first trust zone having a first privilege level, a metadata management system belonging to a second trust zone having a second privilege level higher than the first privilege level, and a security management system belonging to a third trust zone having a third privilege level higher than or equal to the second privilege level. The storage management system is and configured to store multiple content entities, and the metadata management system is configured to manage, for each of the multiple content entities, metadata including a respective content encryption key and a respective retention time, each of the content entities being encrypted by its respective content encryption key. The security management system is configured to manage a master encryption key used to create the respective content encryption keys, and to confirm expiration of the respective retention times.

Abstract: Methods, storage systems and computer program products implement embodiments of the present invention that include defining, for an entity, a policy access control list including one or more access rules, each of the access rules including one or more user conditions and one or more entity conditions. Upon receiving a request from a user to access a given entity, one or more user attributes associated with the user and one or more entity attributes associated with the given entity are identified. For each of the access rules, the one or more user conditions are applied to the one or more user attributes, the one or more entity conditions are applied to the one or more entity attributes. Access to the given content entity is granted to the user upon determining that a minimum threshold of the one or more user conditions and the one or more entity conditions are met.

Abstract: Methods, computing systems and computer program products implement embodiments of the present invention that include detecting multiple sets of storage objects stored in a data facility including multiple server racks, each of the server racks including a plurality of server computers, each of the storage objects in each set being stored in a separate one of the server racks and including one or more data objects and one or more protection objects. A specified number of the storage objects are identified in a given server rack, each of the identified storage objects being stored in a separate one of the server computers, and one or more server computers in the given server rack not storing any of the identified storage objects are identified. Finally, in the identified one or more server computers, an additional protection object is created and managed for the identified storage objects.

Abstract: Methods, computing systems and computer program products implement embodiments of the present invention that include detecting multiple sets of storage objects stored in a data facility including multiple server racks, each of the server racks including a plurality of server computers, each of the storage objects in each set being stored in a separate one of the server racks and including one or more data objects and one or more protection objects. A specified number of the storage objects are identified in a given server rack, each of the identified storage objects being stored in a separate one of the server computers, and one or more server computers in the given server rack not storing any of the identified storage objects are identified. Finally, in the identified one or more server computers, an additional protection object is created and managed for the identified storage objects.

Abstract: Methods, storage systems and computer program products implement embodiments of the present invention that include defining, for an entity, a policy access control list including one or more access rules, each of the access rules including one or more user conditions and one or more entity conditions. Upon receiving a request from a user to access a given entity, one or more user attributes associated with the user and one or more entity attributes associated with the given entity are identified. For each of the access rules, the one or more user conditions are applied to the one or more user attributes, the one or more entity conditions are applied to the one or more entity attributes. Access to the given content entity is granted to the user upon determining that a minimum threshold of the one or more user conditions and the one or more entity conditions are met.

Abstract: Methods, storage systems and computer program products implement embodiments of the present invention that include defining, for an entity, a policy access control list including one or more access rules, each of the access rules including one or more user conditions and one or more entity conditions. Upon receiving a request from a user to access a given entity, one or more user attributes associated with the user and one or more entity attributes associated with the given entity are identified. For each of the access rules, the one or more user conditions are applied to the one or more user attributes, the one or more entity conditions are applied to the one or more entity attributes. Access to the given content entity is granted to the user upon determining that a minimum threshold of the one or more user conditions and the one or more entity conditions are met.

Abstract: Methods, computing systems and computer program products implement embodiments of the present invention that include detecting multiple sets of storage objects stored in a data facility including multiple server racks, each of the server racks including a plurality of server computers, each of the storage objects in each set being stored in a separate one of the server racks and including one or more data objects and one or more protection objects. A specified number of the storage objects are identified in a given server rack, each of the identified storage objects being stored in a separate one of the server computers, and one or more server computers in the given server rack not storing any of the identified storage objects are identified. Finally, in the identified one or more server computers, an additional protection object is created and managed for the identified storage objects.