Abstract: A method includes receiving data objects, determining a predicted lifespan of each data object, and instantiating multiple shard files. Each shard file has an associated predicted lifespan range. The method also includes writing each data object into a corresponding shard file having the associated predicted lifespan range that includes the predicted lifespan of the respective data object and storing the shard files in a distributed system. The method also includes determining whether any stored shard files satisfy a compaction criteria based on a number of deleted data objects in each corresponding stored shard file. For each stored shard file satisfying the compaction criteria, the method also includes compacting the stored shard file by rewriting the remaining data objects of the stored shard file into a new shard file.

Abstract: A system and method for storing data in a distributed network having a plurality of datacenters distributed over a plurality of geographic regions. The method may involve receiving data, including metadata, uploaded to a first datacenter of the distributed network, receiving access information about previous data that was previously stored in the plurality of datacenters of the distributed network, predicting one or more of the plurality of geographic regions from which the uploaded data will be accessed based on the metadata and the access information, and instructing the uploaded data to be transferred from the first datacenter to one or more second datacenters located at each of the one or more predicted geographic regions.

Abstract: A method includes receiving data objects, determining a predicted lifespan of each data object, and instantiating multiple shard files. Each shard file has an associated predicted lifespan range. The method also includes writing each data object into a corresponding shard file having the associated predicted lifespan range that includes the predicted lifespan of the respective data object and storing the shard files in a distributed system. The method also includes determining whether any stored shard files satisfy a compaction criteria based on a number of deleted data objects in each corresponding stored shard file. For each stored shard file satisfying the compaction criteria, the method also includes compacting the stored shard file by rewriting the remaining data objects of the stored shard file into a new shard file.

Abstract: A method includes receiving data objects, determining a predicted lifespan of each data object, and instantiating multiple shard files. Each shard file has an associated predicted lifespan range. The method also includes writing each data object into a corresponding shard file having the associated predicted lifespan range that includes the predicted lifespan of the respective data object and storing the shard files in a distributed system. The method also includes determining whether any stored shard files satisfy a compaction criteria based on a number of deleted data objects in each corresponding stored shard file. For each stored shard file satisfying the compaction criteria, the method also includes compacting the stored shard file by rewriting the remaining data objects of the stored shard file into a new shard file.

Abstract: A system and method for storing data in a distributed network having a plurality of datacenters distributed over a plurality of geographic regions. The method may involve receiving data, including metadata, uploaded to a first datacenter of the distributed network, receiving access information about previous data that was previously stored in the plurality of datacenters of the distributed network, predicting one or more of the plurality of geographic regions from which the uploaded data will be accessed based on the metadata and the access information, and instructing the uploaded data to be transferred from the first datacenter to one or more second datacenters located at each of the one or more predicted geographic regions.

Abstract: A method includes identifying a first table including data. The first table has associated metadata, an associated replication state, an associated replication log file including replication logs logging mutations of the first table, and an associated replication configuration file including a first association that associates the first table with a replication family. The method includes inserting a second association in the replication configuration file that associates a second table having a non-loadable state with the replication family. The association of the second table with the replication family causes persistence of any replication logs in the replication log file that correspond to any mutations of the first table during the existence of the second table. The method further includes generating a third table from the first table, the metadata associated with the first table, and the associated replication state of the first table.

Abstract: A method includes identifying a first table including data. The first table has associated metadata, an associated replication state, an associated replication log file including replication logs logging mutations of the first table, and an associated replication configuration file including a first association that associates the first table with a replication family. The method includes inserting a second association in the replication configuration file that associates a second table having a non-loadable state with the replication family. The association of the second table with the replication family causes persistence of any replication logs in the replication log file that correspond to any mutations of the first table during the existence of the second table. The method further includes generating a third table from the first table, the metadata associated with the first table, and the associated replication state of the first table.

Abstract: Techniques for synchronization between data structures for original locks and mirror lock data structures are disclosed herein. The mirror lock data structures are being maintained during various scenarios including volume move and aggregate relocation, in order to preserve the non-disruptive persistent operation on storage initiated by clients. According to one embodiment, a storage node determines a plurality of data container locks to be synchronized to a partner node of the storage node and transfers metadata that indicates states of variables that represent the plurality of data container locks to the partner node in a batch. When a client initiates a data access operation that causes an attempt to modify a data container lock of the plurality of data container locks, the storage node sends a retry code to a client that prompts the client to retry the data access operation after a predetermined time period.

Abstract: A method includes receiving data objects, determining a predicted lifespan of each data object, and instantiating multiple shard files. Each shard file has an associated predicted lifespan range. The method also includes writing each data object into a corresponding shard file having the associated predicted lifespan range that includes the predicted lifespan of the respective data object and storing the shard files in a distributed system. The method also includes determining whether any stored shard files satisfy a compaction criteria based on a number of deleted data objects in each corresponding stored shard file. For each stored shard file satisfying the compaction criteria, the method also includes compacting the stored shard file by rewriting the remaining data objects of the stored shard file into a new shard file.

Abstract: A method includes identifying a first table including data. The first table has associated metadata, an associated replication state, an associated replication log file including replication logs logging mutations of the first table, and an associated replication configuration file including a first association that associates the first table with a replication family. The method includes inserting a second association in the replication configuration file that associates a second table having a non-loadable state with the replication family. The association of the second table with the replication family causes persistence of any replication logs in the replication log file that correspond to any mutations of the first table during the existence of the second table. The method further includes generating a third table from the first table, the metadata associated with the first table, and the associated replication state of the first table.

Abstract: Techniques for synchronization between data structures for original locks and mirror lock data structures are disclosed herein. The mirror lock data structures are being maintained during various scenarios including volume move and aggregate relocation, in order to preserve the non-disruptive persistent operation on storage initiated by clients. According to one embodiment, a storage node determines a plurality of data container locks to be synchronized to a partner node of the storage node and transfers metadata that indicates states of variables that represent the plurality of data container locks to the partner node in a batch. When a client initiates a data access operation that causes an attempt to modify a data container lock of the plurality of data container locks, the storage node sends a retry code to a client that prompts the client to retry the data access operation after a predetermined time period.

Abstract: Techniques for synchronization between data structures for original locks and mirror lock data structures are disclosed herein. The mirror lock data structures are being maintained during various scenarios including volume move and aggregate relocation, in order to preserve the non-disruptive persistent operation on storage initiated by clients. According to one embodiment, a storage node determines a plurality of data container locks to be synchronized to a partner node of the storage node and transfers metadata that indicates states of variables that represent the plurality of data container locks to the partner node in a batch. When a client initiates a data access operation that causes an attempt to modify a data container lock of the plurality of data container locks, the storage node sends a retry code to a client that prompts the client to retry the data access operation after a predetermined time period.