Abstract: A snapshot of data from a table associated with a particular user may be generated. Tree data structures that are distributed across multiple computer systems may be accessed. Each of the tree structures may include data associated with one or more users. At least one tree data structure of the tree data structures that includes data associated with the particular user of the one or more users may be identified. The at least one tree data structure may then be filtered. Filtering may comprise identifying only data that is associated with the particular user. A snapshot of the data associated with the particular user may be generated. Generating the snapshot of the data associated with the particular user comprises generating a data structure that is configured to map to each data page of the at least one tree data structure that includes data associated with the particular user.

Abstract: A system receives a request to store data at a first layer of servers in a cluster, configured to authenticate and authorize the request. The system compresses the data upon authenticating and authorizing the request and encrypts the compressed data at the first layer of servers when encryption is enabled. The system sends the request and the encrypted data to a second layer of servers in the cluster, configured to store data structures used to manage data storage in a third layer of servers in the cluster, and to distribute the request and the encrypted data to the third layer of servers using the data structures. The system stores the encrypted data in the third layer of servers. Encrypting the data at the first layer of servers reduces latency associated with transferring the data between the first, second, and third layers of servers.

Abstract: One example provides a method of migrating a data partition from a first storage cluster to a second storage cluster, the method including determining that the data partition meets a migration criteria for migrating from the first storage cluster to the second storage cluster, on the first storage cluster, preparing partition metadata to be transferred, the partition metadata describing one or more streams within the data partition and one or more extents within each stream, transferring the partition metadata from the first storage cluster to the second storage cluster, directing new transactions associated with the data partition to the second storage cluster, including while the one or more extents reside at the first storage cluster, on the first storage cluster, changing an access attribute of the one or more extents within the data partition to read-only, and on the second storage cluster, performing new ingress for the data partition.

Abstract: Described herein is a system method for point in time blob restore. In response to receipt of a request to restore a particular blob range to a particular point in time, read and/or write operations on the particular blob range requested to be restored are blocked. Blob(s) in the particular blob range which have changed since the particular point in time are determined based, at least in part, upon information stored in a change log and/or a deleted blob store. A determination is made as to whether the requested restore operation can be performed, for example, based upon a restoration policy. When it is determined that the requested restore operation can be performed, blob(s) determined to have changed since the particular point in time based, at least in part, upon information stored in the change log and/or the deleted blob store are restored.

Abstract: In various embodiments, methods and systems for implementing hash-based partitioning in distributed computing systems are provided. At a high level, a distributed computing system having an underlying range-based partitioning architecture for storage may be configured as a hash-based partitioning system, for example, a hybrid range-hash table storage. An operations engine of the hash-based partitioning system receives a tenant request to provision input/output operations per second (IOPS). The tenant request comprises a requested number of IOPS. Based on the tenant request, a provisioning operation to provision IOPS in a hybrid range-hash table storage with hash-based partitioning is determined. The provisioning operation is selected from one of the following: a table creation provisioning operation, an IOPS increase provisioning operation, and an IOPS decrease provisioning operation. The selected provisioning operation is executed for a corresponding table.

Abstract: A cloud storage system includes a processor and a non-transitory computer-readable medium to store blob table management instructions for execution by the processor. The blob table management instructions are configured to manage a plurality of storage requests for a blob stored in a storage stamp as snapshots in a blob table and selectively create a user snapshot of at least one of the snapshots in the blob table. When automatic snapshots are enabled, the blob table management instructions are configured to receive a first request to overwrite the blob. If the first request does not further specify a key of the one of the snapshots in the blob table, the blob table management instructions are configured to add a new snapshot to the blob table and maintain storage of a prior snapshot of the blob for a maximum period.

Abstract: Described herein is a system and method for a user accessible log of object storage change-events system. A change is detected within a cloud storage based on an unstructured storage object stored in the cloud storage. Contextual attributes of the change are identified from an updated state of the log file that stores information about the unstructured storage object. An output is provided based, at least in part, upon the detected change and at least some of the contextual attributes, wherein the output is an immutable, read-only log of change events.

Abstract: Methods, systems, and computer programs are presented for providing file system functions on a cloud storage system based on blob storage. A cloud storage system comprises storage clusters, and each storage cluster comprises a plurality of storage nodes (each including a partition layer that stores data in blobs organized within containers, and a stream layer for streaming blob data), a hierarchical namespace service for implementing file system functions on the data stored in the blobs and, a front end layer for processing user requests to access and manage data stored on the blobs. The file system functions include flat namespace functions and hierarchical namespace functions. Further, the front end layer accesses the partition layer to access the blobs based on addresses of blobs and accesses the hierarchical namespace service to identify the blobs.

Abstract: Examples are disclosed that relate to querying data in a distributed storage system. One example provides, at a front-end node, a method comprising receiving a query regarding data stored in an object, obtaining information identifying data nodes at which blocks of the object are stored, sending, to a first data node, a query regarding data in a first block, and receiving results for the query regarding the data in the first block comprising an incomplete record. The method further comprises sending, to a second data node, a query regarding data in a second block comprising the incomplete record, the second block being stored on a different physical device than the first block, and receiving results for the query regarding the data in the second block. The method further comprises outputting the results for the queries regarding the data in the first block and the data in the second block.

Abstract: Methods, systems, and programs provide for executing distributed transactions in a cloud storage system with a hierarchical namespace. One method includes receiving a request with operations to be executed atomically. Further, nodes are identified for executing the operations, each node having a respective clock and having at least part of a transactions table for controlling updates to entities. Each clock is one of a loosely-synchronized, a strictly-synchronized clock, a logical, or a physical clock. Additionally, the nodes process the operations, which includes setting a commit timestamp (CS) to a value of the clock in the node if the node is a first node in the processing. One node coordinates the transactions, and may be one of the nodes executing transactions. If the clock in the node is less than a current value of the CS, the node waits for the clock to reach the current value of the CS and the CS is updated.

Abstract: A distributed storage system includes multiple partitions. A block received for storage is striped across the partitions if it meets predetermined criteria. Multiple tables are maintained for indexing the blocks and the subblocks of blocks that are partitioned. The tables for the subblocks and the tables for the corresponding blocks of the subblocks are updated to include metadata for the subblocks for retrieving the subblocks in response to a request for a corresponding stored block.

Abstract: A service enables a command that refers to a file system object using a hierarchical namespace identifier to be executed against the file system object in a flat namespace. The service selectively distributes the command to one of a plurality of name resolution nodes based on a directory name included in the hierarchical namespace identifier. The identified node resolves the directory name to a flat namespace identifier that is used to execute the command against the flat namespace. After communicating with at least one storage node to resolve a directory name, each name resolution node stores a mapping of the directory name to the corresponding flat namespace identifier in a cache, so that subsequent resolutions of that directory name may be performed more efficiently. Cache entries may be invalidated when an operation occurs that impacts the relevant mapping and/or based on system considerations such as cache expiry.

Abstract: Techniques for implementing zone redundant computing services in distributed computing systems are described herein. In one embodiment, a computing device is configured to query a database in a first availability zone for a list of computing services corresponding to a user account and provided by servers in a second availability zone different than the first availability zone. The computing device can then selects one or more computing services provided by the servers in the second availability zone from the queried list of computing services and transmit an instruction to the servers in the second availability zone for performing the requested data operation.

Abstract: Methods, systems, and apparatuses are provided for a storage system that implements a hierarchical namespace service. A storage system includes a plurality of physical nodes and a plurality of sets of virtual nodes. Each set of virtual nodes is managed by a corresponding physical node. Each virtual node is configured to manage a respective set of directory blocks. Each directory block is a respective partition of a storage namespace and is managed by a corresponding single virtual node. Each virtual node maintains a directory block map. The directory block map maps file system object names in a hierarchical namespace to entity block identifiers in the flat namespace for entity blocks (files and folders) stored in directories corresponding to the managed set of directory blocks. Load balancing may be performed by moving virtual nodes between physical nodes, and by splitting directory blocks.

Abstract: A system receives data of one or more types from one or more sources having distinct identities for storing the data in a distributed storage system. The system stores metadata associated with storing the data in data structures in the distributed storage system. The system selects a portion of the data and a portion of the metadata associated with storing the data in the distributed storage system. The system compresses the selected portions of the data and the metadata, encrypts the compressed data and metadata using a single key or using a separate key for each data type, and stores the encrypted compressed data and metadata in the distributed storage system. The system also encrypts unencrypted metadata and associated data during defragmentation procedure.

Abstract: Live migration of partitions can be achieved using server-to-server communications to coordinate and control partition migration and rebuild a persistent state of the partition on a destination server while the partition is served on the source server. The live migration approach breaks the heavy dependency on the partition master by eliminating most of the server-to-master communications, delegating coordination and control of the migration to the source and destination servers. After the migration is initiated by the partition master, the source server sends in-memory cache entries indicating the persistent state of the partition to the destination server. Once transferred, the two servers can cooperate to complete the handover, which loads the partition on the destination server using the cache entries received from the source server. Once the source server offloads the partition, it can redirect traffic for the partition to the destination server by triggering an update to the partition map.

Abstract: A service enables a command that refers to a file system object using a hierarchical namespace identifier to be executed against the file system object in a flat namespace. The service selectively distributes the command to one of a plurality of name resolution nodes based on a directory name included in the hierarchical namespace identifier. The identified node resolves the directory name to a flat namespace identifier that is used to execute the command against the flat namespace. After communicating with at least one storage node to resolve a directory name, each name resolution node stores a mapping of the directory name to the corresponding flat namespace identifier in a cache, so that subsequent resolutions of that directory name may be performed more efficiently. Cache entries may be invalidated when an operation occurs that impacts the relevant mapping and/or based on system considerations such as cache expiry.

Abstract: Techniques for managing resources among clusters of computing devices in a computing system are described herein. In one embodiment, a method includes receiving, via a computer network, a resource reassignment message indicating that a server is reassigned from a first cluster to a second cluster and in response to the received resource reassignment message, establishing communications with the server reassigned from the first cluster to the second cluster via the computer network. The method further includes subsequent to establishing communications with the server via the computer network, assigning a compute load to the server reassigned from the first cluster to the second cluster without physically relocating the server from the first cluster to the second cluster.

Abstract: A system stores data in data units in a cluster in a cloud computing system, the data stored in the data units being encrypted or unencrypted depending on whether encryption is enabled or disabled when storing data in the data units. The system identifies one or more data units to defragment and defragments the identified data units by writing the data from the identified data units to one or more new data units and by releasing the identified data units for storing new data. The system encrypts unencrypted data from the identified data units when writing the data from the identified data units to the one or more new data units.

Abstract: Prefetching metadata stream sub-portions and/or log stream sub-portions to recreate partition states in a distributed computing system. A system monitors performance metrics associated with a first node running an instance of a partition and, based thereon, cause a second node to duplicate a desired operational state of the partition. The desired operational state may result from transactions being performed on the partition after a previous operational state. Re-creating the desired operational state may include causing multiple prefetch buffers to simultaneously load sub-portions of the transactions and then commencing a replay operation after an initial subset of the transactions is loaded but prior to the entirety of the transactions being fully loaded. The system may preemptively analyze individual prefetch buffers to identify partial buffers that have not fully loaded. Then, the system may cause other prefetch buffers to assist the partial buffers in fully loading a corresponding subset of transactions.