Abstract: Methods for modeling performance of tiered storage of a data processing service given a decrease in the storage capacity of a warm storage tier of the tiered storage are disclosed. Metadata of the warm storage tier is used to track hits due to incoming queries on data blocks that are stored in the warm storage tier. The metadata prioritizes data block identifiers that correspond to the data blocks stored in the warm storage tier by frequency of hits due to the incoming queries, or various other prioritization schemes. One or more partitions of the metadata may be set that correspond to respective downsized storage capacity scenarios of the warm storage tier. When an incoming query targets a data block within a given partition of the metadata, a hit counter is incremented to track the hit rate that would be made on the downsized warm storage tier corresponding to that partition.

Abstract: Methods for modeling performance of tiered storage of a data processing service given a decrease in the storage capacity of a warm storage tier of the tiered storage are disclosed. Metadata of the warm storage tier is used to track hits due to incoming queries on data blocks that are stored in the warm storage tier. The metadata prioritizes data block identifiers that correspond to the data blocks stored in the warm storage tier by frequency of hits due to the incoming queries, or various other prioritization schemes. One or more partitions of the metadata may be set that correspond to respective downsized storage capacity scenarios of the warm storage tier. When an incoming query targets a data block within a given partition of the metadata, a hit counter is incremented to track the hit rate that would be made on the downsized warm storage tier corresponding to that partition.

Abstract: A data processing service generates a data classifier tree for managing data files of a data table. The data classifier tree may be configured as a KD-classifier tree and includes a plurality of nodes and edges. A node of the data classifier tree may represent a splitting condition with respect to key-values for a respective key. A node of the data classifier tree may be associated with one or more data files assigned to the node. The data files assigned to the node each include a subset of records having key-values that satisfy the conditions represented by the node and parent nodes of the node. The data processing service may efficiently cluster the data in the data table while reducing the number of data files that are rewritten when data is modified or added to the data table.

Abstract: Disclosed is a configuration for managing the organization of data tables in cloud-based storage. The configuration receives metrics for data processing operations on the data table. Metrics include at least one of a size of the data table, a size of each file in the data table, and metadata describing the data table. The configuration automatically executes a cost-benefit analysis based on the one or more metrics for each candidate maintenance operation in a plurality of candidate maintenance operations. The configuration automatically selects a maintenance operation from the candidate maintenance operations to automate based on the cost-benefit analysis of the one or more candidate maintenance operations. The selected maintenance operation is automated and scheduled on the data table.

Abstract: Methods for improving performance of tiered storage of a data processing service by detecting and reducing thrashing of data blocks between warm and cold storage tiers are disclosed. In order to understand the frequency of hits by incoming queries to data blocks that are not currently stored in the warm storage tier, the elapsed time between query hits to the respective data blocks may be tracked using timers. Times below a given amount of time may be used to indicate thrashing. For example, recently evicted data blocks that are subsequently hit by a query within a short amount of time since eviction may indicate thrashing. In scenarios in which thrashing may be occurring, a threshold corresponding to the number of times a given data block in the cold storage tier receives a query hit before being added to the warm storage tier may be turned on.

Abstract: Disclosed is a configuration for managing the organization of data tables in cloud-based storage. The configuration receives metrics for data processing operations on the data table. Metrics include at least one of a size of the data table, a size of each file in the data table, and metadata describing the data table. The configuration automatically executes a cost-benefit analysis based on the one or more metrics for each candidate maintenance operation in a plurality of candidate maintenance operations. The configuration automatically selects a maintenance operation from the candidate maintenance operations to automate based on the cost-benefit analysis of the one or more candidate maintenance operations. The selected maintenance operation is automated and scheduled on the data table.

Abstract: A data processing service generates a data classifier tree for managing data files of a data table. The data classifier tree may be configured as a KD-classifier tree and includes a plurality of nodes and edges. A node of the data classifier tree may represent a splitting condition with respect to key-values for a respective key. A node of the data classifier tree may be associated with one or more data files assigned to the node. The data files assigned to the node each include a subset of records having key-values that satisfy the conditions represented by the node and parent nodes of the node. The data processing service may efficiently cluster the data in the data table while reducing the number of data files that are rewritten when data is modified or added to the data table.

Abstract: Disclosed is a configuration for managing the organization of data tables in cloud-based storage. The configuration receives metrics for data processing operations on the data table. Metrics include at least one of a size of the data table, a size of each file in the data table, and metadata describing the data table. The configuration automatically executes a cost-benefit analysis based on the one or more metrics for each candidate maintenance operation in a plurality of candidate maintenance operations. The configuration automatically selects a maintenance operation from the candidate maintenance operations to automate based on the cost-benefit analysis of the one or more candidate maintenance operations. The selected maintenance operation is automated and scheduled on the data table.

Abstract: A protocol for implementing ACID transactions that provides snapshot isolation in a distributed setting that does not require synchronized clocks is described. The protocol ensures at commit time that transactions touching common objects do not commit out of order. The protocol can be used in the context of a distributed data lake built on an object store in which clients can transactionally add or remove objects from logical tables.

Abstract: Methods for modeling performance of tiered storage of a data processing service given an increase in the storage capacity of a warm storage tier of the tiered storage are disclosed. Buffers in the warm storage tier are used to store data block identifiers corresponding to a set of data blocks that would be stored in the warm storage tier given the increase in storage capacity in addition to those already stored in the warm storage tier. When an incoming query targets a data block that has a corresponding data block identifier in one of the buffers, a hit counter is incremented in order to track the hit rate that would be made on the up-sized warm storage tier. In response to adding the data block targeted by the query to the warm storage tier, one or more evictions from the warm storage tier may additionally be triggered.

Abstract: Techniques for using tree data structures to maintain a transactionally consistent set with support for time-travel queries are described. When a transaction commits, a new version of the tree data structure is created using a copy-on-write based method such that the tree shares internal nodes with previous trees to save space. This approach may be used in the implementation of a transactional data catalog in which the files that make up a table are stored in a transactional set.

Abstract: Techniques for implementing systems using transactional version sets are described. Transactional version sets or t-sets include a collection of elements, each having a collection of metadata. A t-set is transactional in that a sequence of updates to one or more t-sets are made within an atomic transaction. A t-set is versioned since each committed transaction that updates it produces a new timestamped version that can be accessed via time-travel queries.

Abstract: Techniques for resizing a distributed database are described. A request to resize a distributed database is received. The distributed database stores data organized into one or more rows of one or more tables. Each node of the first plurality of nodes is assigned a portion of the data. A portion of the data assigned to a first node in the first plurality of nodes is selected to be assigned to a second node in a second plurality of nodes. The number of nodes in the first and second plurality of nodes is different, and the first and second plurality of nodes include at least one common node. Metadata of the selected portion of the data is transferred from the first node to the second node. The metadata that includes a location of the selected portion of the data within a provider network.

Abstract: Data in a storage system is deduplicated after receiving from at least one writing entity requests for a plurality of write operations for a corresponding plurality of data blocks in a storage object. The received blocks are buffered and sorted in order and a sequence of clumps is created from the buffered blocks, where each clump comprises a grouping of at least one of the sorted, buffered blocks. A boundary is determined between at least one pair of clumps based at least in part on the content of at least one of the buffered blocks, and it is then determined whether at least one of the clumps is a duplicate of a previously stored clump.

Abstract: Data in a storage system is deduplicated after receiving from at least one writing entity requests for a plurality of write operations for a corresponding plurality of data blocks in a storage object. The received blocks are buffered and sorted in order and a sequence of clumps is created from the buffered blocks, where each clump comprises a grouping of at least one of the sorted, buffered blocks. A boundary is determined between at least one pair of clumps based at least in part on the content of at least one of the buffered blocks, and it is then determined whether at least one of the clumps is a duplicate of a previously stored clump.

Abstract: Data in a storage system is deduplicated after receiving from at least one writing entity requests for a plurality of write operations for a corresponding plurality of data blocks in a storage object. The received blocks are buffered and sorted in order and a sequence of clumps is created from the buffered blocks, where each clump comprises a grouping of at least one of the sorted, buffered blocks. A boundary is determined between at least one pair of clumps based at least in part on the content of at least one of the buffered blocks, and it is then determined whether at least one of the clumps is a duplicate of a previously stored clump.

Abstract: A machine-implemented method includes automatically determining that a host device is restarting from a disruptive stoppage of operations and that in-process write transactions by the host device to respective pages of non-volatile storage may have been interrupted. The method includes, in response to the determination, automatically scanning the non-volatile storage for all metadata-containing storage pages with respective identifications S(i) and having corresponding metadata relating each respective storage page S(i) to a corresponding data page P(j) and a corresponding version number V(k). The method includes automatically identifying scanned storage pages S(i) that have for their corresponding data page P(j) a most recent version number HV(k) and, in some cases, a secondmost recent version number.

Abstract: A cyclic commit protocol is used to store relationships between transactions and is used by the technology to determine whether a transaction is committed or not. The protocol allows creation of a cycle of transactions which can be used to recover the state of a storage device after a host failure by identifying the last committed version of intention records as committed or uncommitted based on the data stored in the physical pages.

Abstract: Data in a storage system is deduplicated after receiving from at least one writing entity requests for a plurality of write operations for a corresponding plurality of data blocks in a storage object. The received blocks are buffered and sorted in order and a sequence of clumps is created from the buffered blocks, where each clump comprises a grouping of at least one of the sorted, buffered blocks. A boundary is determined between at least one pair of clumps based at least in part on the content of at least one of the buffered blocks, and it is then determined whether at least one of the clumps is a duplicate of a previously stored clump.

Abstract: A cyclic commit protocol is used to store relationships between transactions and is used by the technology to determine whether a transaction is committed or not. The protocol allows creation of a cycle of transactions which can be used to recover the state of a storage device after a host failure by identifying the last committed version of intention records as committed or uncommitted based on the data stored in the physical pages.