Abstract: A system, method and apparatus for storing metadata in a metadata store in a robust and efficient manner including receiving a request from a client to perform a data transaction, updating a key-value pair in a metadata store based on the request, entering the data transaction in a transaction log, updating a read cache with the key-value pair, and replicating the last transaction log entry in at least one other storage node in the metadata store.

Abstract: A system, method and apparatus for storing metadata in a metadata store in a robust and efficient manner including receiving a request from a client to perform a data transaction, updating a key-value pair in a metadata store based on the request, entering the data transaction in a transaction log, replicating the last transaction log entry in at least one other storage node in the metadata store and, in response to a threshold event, migrating a first portion of the transaction log from the a data storage medium to a second data storage medium of each of the plurality of data storage nodes.

Abstract: The present disclosure relates to systems and methods for dynamic sharding of a database comprising data identifiable by keys comprised within a global ordered range. When handling a request for data of at least one key: providing the request to a predetermined shard store; the predetermined shard store verifying, by means of its local subrange collection, whether the at least one key is present in a local subrange of a shard stored on the predetermined shard store; and when at least one key is not present in a local subrange of a shard stored on the predetermined shard store, returning a message comprising the local subrange collection of the predetermined shard store.

Abstract: A data object can be encoded into a plurality of encoded data fragments and stored on backend storage elements in a distributed encoded storage system. The identifiers and metadata corresponding to each encoded fragment of the data object can be stored in a single metadata unit, which is stored on the backend as encoded fragments. The identifiers of the metadata fragments can be associated with the data object and stored on a low latency frontend storage device. Thus, the amount of metadata per data object stored on expensive low latency frontend storage is reduced to the fragment identifiers. The fragment identifiers can be quickly retrieved, and used to retrieve the identifiers and metadata corresponding to the encoded data fragments from the backend, for retrieval of the data object itself.

Abstract: A data object can be encoded into multiple encoded data fragments and stored on the backend of a distributed encoded storage system. The identifiers and metadata corresponding to each fragment of the data object can be stored in a first section of a metadata unit, and the initial part of the data object in a second section. The metadata unit is encoded into multiple metadata fragments, which are stored on the backend. The identifiers of the metadata fragments can be associated with the data object and stored on a fast frontend storage device. In response to a request to access the data object, the identifiers are used to retrieve the metadata fragments from the backend, and decode the metadata unit. The initial part of the data object is retrieved from the metadata unit and transmitted to the requesting client application to begin processing the data object.

Abstract: A data object can be encoded into a plurality of encoded data fragments and stored on backend storage elements in a distributed encoded storage system. The identifiers and metadata corresponding to each encoded fragment of the data object can be stored in a single metadata unit, which is stored on the backend as encoded fragments. The identifiers of the metadata fragments can be associated with the data object and stored on a low latency frontend storage device. Thus, the amount of metadata per data object stored on expensive low latency frontend storage is reduced to the fragment identifiers. The fragment identifiers can be quickly retrieved, and used to retrieve the identifiers and metadata corresponding to the encoded data fragments from the backend, for retrieval of the data object itself.

Abstract: A data object can be encoded into multiple encoded data fragments and stored on the backend of a distributed encoded storage system. The identifiers and metadata corresponding to each fragment of the data object can be stored in a first section of a metadata unit, and the initial part of the data object in a second section. The metadata unit is encoded into multiple metadata fragments, which are stored on the backend. The identifiers of the metadata fragments can be associated with the data object and stored on a fast frontend storage device. In response to a request to access the data object, the identifiers are used to retrieve the metadata fragments from the backend, and decode the metadata unit. The initial part of the data object is retrieved from the metadata unit and transmitted to the requesting client application to begin processing the data object.

Abstract: The present disclosure relates to a system and methods of implementing an integer-value database using a single I/O operation. In particular, the present disclosure relates to methods of writing and reading information to a database using key/value pairs, including receiving, at a database management system, a value to be written to a database, the database including a plurality of segments stored on a storage medium, and assigning, by the database management system, an assigned key to the value based on keys previously used in the database. The method may further include storing, by the database management system, the assigned key and the value to a segment at a virtual end of the database, wherein the segment is identified in a mapping index by an offset and the mapping index identifies a first key in the segment, and returning the assigned key.

Abstract: Techniques for distributing data in a distributed data storage system using a hierarchy rule that is generated based on a spreading policy and a set of tolerable failures specified by a user in absence of system deployment information are disclosed. The system includes a controller node which receives a request including a spreading policy and a protection level for spreading a first data object. The controller node determines a hierarchy rule corresponding to the spreading policy based on the protection level. The controller node distributes the first data object in the system using the hierarchy rule and the spreading policy. The controller node receives a reconfiguration of system deployment. The controller node distributes a second data object in the system based on providing protection of the protection level to the second data object without affecting protection of the same protection level applied to the first data object.

Abstract: A computer-implemented method for redistributing data between memory clusters in a key value store. The data is redistributed according to a load balancing algorithm. Memory clusters are sorted into lists according to the number of intervals each cluster contains. Migration jobs are created by the load balancing algorithm to push data fragments from a larger cluster to a smaller cluster. Data fragments, or intervals, are selected for migration according to corresponding data fragments on a target cluster or the number of occupying data fragments on a target cluster. The redistribution of data helps avoid fragmentation of key ranges of intervals, decrease the overall number of non-adjacent key ranges, and to conform to the routing table requirements.

Abstract: A system and method for seamlessly and automatically handling outdated resources in a cloud and/or performing a multiple-version upgrade within the cloud with little visibility of interim operations to end users are presented. Mechanisms ensure that outdated resources do not communicate with the rest of the cloud (for example, via distributed data models and Application Programming Interfaces (APIs) before they are upgraded to a target software level. A multiple-version upgrade is implemented in the cloud by a single upgrade operation, by which internal paths for upgrading through various interim versions of software are concealed from end users of the cloud.

Abstract: A system, method and apparatus for storing metadata in a metadata store in a robust and efficient manner including receiving a request from a client to perform a data transaction, updating a key-value pair in a metadata store based on the request, entering the data transaction in a transaction log, updating a read cache with the key-value pair, and replicating the last transaction log entry in at least one other storage node in the metadata store.

Abstract: A system, method and apparatus for storing metadata in a metadata store in a robust and efficient manner including receiving a request from a client to perform a data transaction, updating a key-value pair in a metadata store based on the request, entering the data transaction in a transaction log, replicating the last transaction log entry in at least one other storage node in the metadata store and, in response to a threshold event, migrating a first portion of the transaction log from the a data storage medium to a second data storage medium of each of the plurality of data storage nodes.

Abstract: A set of encoded data fragments is grouped into a container object in sequential order. Each encoded data fragment is a specific fragment size, and the container object is a specific container object size. The sequential order of the set of encoded data fragments can be tracked in a log in memory, such that the location of any one of the data fragments in the container object can be determined. The container object can be stored directly on a specific backend storage element, without using a file system. A corresponding container object identifier identifies the physical storage location of the container object on the backend storage element. The container object identifier is tracked in the log in memory, such that the physical location on the backend storage element of any specific one of the set of encoded data fragments in the container object can be determined.

Abstract: Techniques for distributing data in a distributed data storage system using a hierarchy rule that is generated based on a spreading policy and a set of tolerable failures specified by a user in absence of system deployment information are disclosed. The system includes a controller node which receives a request including a spreading policy and a protection level for spreading a first data object. The controller node determines a hierarchy rule corresponding to the spreading policy based on the protection level. The controller node distributes the first data object in the system using the hierarchy rule and the spreading policy. The controller node receives a reconfiguration of system deployment. The controller node distributes a second data object in the system based on providing protection of the protection level to the second data object without affecting protection of the same protection level applied to the first data object.

Abstract: A computer-implemented method for redistributing data between memory clusters in a key value store. The data is redistributed according to a load balancing algorithm. Memory clusters are sorted into lists according to the number of intervals each cluster contains. Migration jobs are created by the load balancing algorithm to push data fragments from a larger cluster to a smaller cluster. Data fragments, or intervals, are selected for migration according to corresponding data fragments on a target cluster or the number of occupying data fragments on a target cluster. The redistribution of data helps avoid fragmentation of key ranges of intervals, decrease the overall number of non-adjacent key ranges, and to conform to the routing table requirements.

Abstract: Techniques for distributing data in a distributed data storage system using a hierarchy rule that is generated based on a spreading policy and a set of tolerable failures specified by a user in absence of system deployment information are disclosed. The system includes a controller node which receives a request including a spreading policy and a protection level for spreading a first data object. The controller node determines a hierarchy rule corresponding to the spreading policy based on the protection level. The controller node distributes the first data object in the system using the hierarchy rule and the spreading policy. The controller node receives a reconfiguration of system deployment. The controller node distributes a second data object in the system based on providing protection of the protection level to the second data object without affecting protection of the same protection level applied to the first data object.

Abstract: The present disclosure relates to systems and methods for dynamic sharding of a database comprising data identifiable by keys comprised within a global ordered range. When handling a request for data of at least one key: providing the request to a predetermined shard store; the predetermined shard store verifying, by means of its local subrange collection, whether the at least one key is present in a local subrange of a shard stored on the predetermined shard store; and when at least one key is not present in a local subrange of a shard stored on the predetermined shard store, returning a message comprising the local subrange collection of the predetermined shard store.

Abstract: Techniques for distributing data in a distributed data storage system using a hierarchy rule that is generated based on a spreading policy and a set of tolerable failures specified by a user in absence of system deployment information are disclosed. The system includes a controller node which receives a request including a spreading policy and a protection level for spreading a first data object. The controller node determines a hierarchy rule corresponding to the spreading policy based on the protection level. The controller node distributes the first data object in the system using the hierarchy rule and the spreading policy. The controller node receives a reconfiguration of system deployment. The controller node distributes a second data object in the system based on providing protection of the protection level to the second data object without affecting protection of the same protection level applied to the first data object.

Abstract: A system and method for seamlessly and automatically handling outdated resources in a cloud and/or performing a multiple-version upgrade within the cloud with little visibility of interim operations to end users are presented. Mechanisms ensure that outdated resources do not communicate with the rest of the cloud (for example, via distributed data models and Application Programming Interfaces (APIs) before they are upgraded to a target software level. A multiple-version upgrade is implemented in the cloud by a single upgrade operation, by which internal paths for upgrading through various interim versions of software are concealed from end users of the cloud.