Abstract: A distributed system is adapted to manage the performance of distributed processes. In one aspect, multiple stripes associated with a data item are stored in a distributed storage. The stored stripes may include one or more stripes of redundancy information for the data item. The stored stripes may include one or more copies of stripes for the data item. A distributed process including at least one task is performed. During performance of the distributed process, a determination is made as to whether to perform an accelerated data retrieval operation. Responsive to a determination to perform an accelerated data retrieval operation, at more than the minimal number stripes information required to reconstruct the data item is requested from the distributed storage. After a sufficient subset of stripes associated with the data item is received, the data item is reconstructed using the subset.

Abstract: A distributed system is adapted to manage the performance of distributed processes. In one aspect, multiple stripes associated with a data item are stored in a distributed storage. The stored stripes include one or more stripes of redundancy information for the data item. A distributed process including at least one task is performed. During performance of the distributed process, a determination is made as to whether to perform an accelerated data retrieval operation. Responsive to a determination to perform an accelerated data retrieval operation, at least one of the one or more stripes of redundancy information for the data item is requested from the distributed storage. Other stripes associated with the data item may also be requested from the distributed storage. After a sufficient subset of stripes associated with the data item is received, the data item is reconstructed using the subset.

Abstract: Accesses to a number of data blocks stored in a distributed storage are observed. Following observation of the accesses, the stored data blocks are redistributed. In one aspect, redistribution of the data blocks includes determining the access patterns for one or more of the data blocks based on the observed accesses, and determining the storage sizes for the one or more data blocks. Thereafter, based on the determined access patterns and determined storage sizes, the one or more data blocks are sorted. Subsequently, the one or more data blocks are redistributed or rebalanced across a number of storage devices of the distributed storage based on the sorting. In one aspect, the one or more data blocks are redistributed according to either a uniform distribution scheme or a proportional distribution scheme.

Abstract: Embodiments are directed to generating and managing an advertising campaign based on a third-party sales listing. In particular, an identifier associated with a sales listing posted over a third-party service from a user of the third-party service is received. Sales listing data for the sales listing is retrieved from the third-party service using the identifier associated with the sales listing. Thereafter, an advertising campaign is automatically generated based at least in part on the retrieved sales listing data. In one aspect, generation of the advertising campaign includes generating an advertising creative using the retrieved sales listing data. Subsequently, bidding on an advertising opportunity in an auction is performed according to the generated advertising campaign. Responsive to winning the auction, an advertisement based on the advertising creative is provided to a content provider associated with the advertising opportunity.

Abstract: A distributed storage system maintains multiple logically independent file systems. Each file system includes a data set stored by a storage device of the distributed storage system. During operation, access pattern levels for the multiple logically independent file systems are determined. Thereafter, the data sets included in the multiple logically independent file systems are redistributed across multiple storage devices of the distributed storage. Redistribution of a particular data set is based at least in part on the particular file system including the particular data set and on the determined access pattern levels for the multiple logically independent file systems. In addition, each disk of a plurality of disks in the distributed storage includes a physically separated partition dedicated to storing the data of the file system that is most frequently accessed. The distribution of data is based at least in part on the presence of the physically separated partition.

Abstract: A column set server adapts to data use patterns by data consumers by modifying how a table produced by a data producer is partitioned into separate column sets to reduce the waste incurred in accessing the data by multiple consumers of the data. For example, the column set server adjusts a column set distribution when a new consumer process is added, when one is retired, or when relative data set size ratios change.

Abstract: A distributed computing system executes a MapReduce job on streamed data that includes an arbitrary amount of imbalance with respect to the frequency distribution of the data keys in the dataset. A map task module maps the dataset to a coarse partitioning, and generates a list of the top K keys with the highest frequency among the dataset. A sort task module employs a plurality of sorters to read the coarse partitioning and sort the data into buckets by data key. The values for the top K most frequent keys are separated into single-key buckets. The other less frequently occurring keys are assigned to buckets that each have multiple keys assigned to it. Then, more than one worker is assigned to each single-key bucket. The output of the multiple workers assigned to each respective single-key bucket is stitched together.

Abstract: A distributed system is adapted to manage the performance of distributed processes. In one aspect, multiple stripes associated with a data item are stored in a distributed storage. The stored stripes include one or more stripes of redundancy information for the data item. A distributed process including at least one task is performed. During performance of the distributed process, a determination is made as to whether to perform an accelerated data retrieval operation. Responsive to a determination to perform an accelerated data retrieval operation, at least one of the one or more stripes of redundancy information for the data item is requested from the distributed storage. Other stripes associated with the data item may also be requested from the distributed storage. After a sufficient subset of stripes associated with the data item is received, the data item is reconstructed using the subset.

Abstract: A distributed system is adapted to manage the performance of distributed processes. In one aspect, multiple stripes associated with a data item are stored in a distributed storage. The stored stripes include one or more stripes of redundancy information for the data item. A distributed process including at least one task is performed. During performance of the distributed process, a determination is made as to whether to perform an accelerated data retrieval operation. Responsive to a determination to perform an accelerated data retrieval operation, at least one of the one or more stripes of redundancy information for the data item is requested from the distributed storage. Other stripes associated with the data item may also be requested from the distributed storage. After a sufficient subset of stripes associated with the data item is received, the data item is reconstructed using the subset.

Abstract: A distributed system is adapted to manage the performance of distributed processes. In one aspect, multiple stripes associated with a data item are stored in a distributed storage. The stored stripes include one or more stripes of redundancy information for the data item. A distributed process including at least one task is performed. During performance of the distributed process, a determination is made as to whether to perform an accelerated data retrieval operation. Responsive to a determination to perform an accelerated data retrieval operation, at least one of the one or more stripes of redundancy information for the data item is requested from the distributed storage. Other stripes associated with the data item may also be requested from the distributed storage. After a sufficient subset of stripes associated with the data item is received, the data item is reconstructed using the subset.

Abstract: Accesses to a number of data blocks stored in a distributed storage are observed. Following observation of the accesses, the stored data blocks are redistributed. In one aspect, redistribution of the data blocks includes determining the access patterns for one or more of the data blocks based on the observed accesses, and determining the storage sizes for the one or more data blocks. Thereafter, based on the determined access patterns and determined storage sizes, the one or more data blocks are sorted. Subsequently, the one or more data blocks are redistributed or rebalanced across a number of storage devices of the distributed storage based on the sorting. In one aspect, the one or more data blocks are redistributed according to either a uniform distribution scheme or a proportional distribution scheme.

Abstract: A column set server adapts to data use patterns by data consumers by modifying how a table produced by a data producer is partitioned into separate column sets to reduce the waste incurred in accessing the data by multiple consumers of the data. For example, the column set server adjusts a column set distribution when a new consumer process is added, when one is retired, or when relative data set size ratios change.

Abstract: Embodiments are directed to generating and managing an advertising campaign based on a third-party listing. In particular, an identifier associated with a listing posted over a third-party service from a user of the third-party service is received. Listing data for the listing is retrieved from the third-party service using the identifier associated with the listing. Thereafter, an advertising campaign is automatically generated based at least in part on the retrieved listing data. In one aspect, generation of the advertising campaign includes generating an advertising creative using the retrieved listing data. Subsequently, an advertising opportunity is selected according to the generated advertising campaign. Following the selection, an advertisement based on the advertising creative is provided to a content provider associated with the advertising opportunity.

Abstract: A distributed storage system maintains multiple logically independent file systems. Each file system includes a data set stored by a distributed storage of the distributed storage system. During operation, access pattern levels for the multiple logically independent file systems are determined. Thereafter, the data sets included in the multiple logically independent file systems are redistributed across multiple storage devices of the distributed storage. In one aspect, redistribution of a particular data set is based at least in part on the particular file system including the particular data set and on the determined access pattern levels for the multiple logically independent file systems. In one implementation, redistribution is performed according to a uniform redistribution scheme. In another implementation, redistribution is performed according to a proportional distribution scheme.

Abstract: Accesses to a number of data blocks stored in a distributed storage are observed. Following observation of the accesses, the stored data blocks are redistributed. In one aspect, redistribution of the data blocks includes determining the access patterns for one or more of the data blocks based on the observed accesses, and determining the storage sizes for the one or more data blocks. Thereafter, based on the determined access patterns and determined storage sizes, the one or more data blocks are sorted. Subsequently, the one or more data blocks are redistributed or rebalanced across a number of storage devices of the distributed storage based on the sorting. In one aspect, the one or more data blocks are redistributed according to either a uniform distribution scheme or a proportional distribution scheme.

Abstract: A distributed storage system maintains multiple logically independent file systems. Each file system includes a data set stored by a storage device of the distributed storage system. During operation, access pattern levels for the multiple logically independent file systems are determined. Thereafter, the data sets included in the multiple logically independent file systems are redistributed across multiple storage devices of the distributed storage. Redistribution of a particular data set is based at least in part on the particular file system including the particular data set and on the determined access pattern levels for the multiple logically independent file systems. In addition, each disk of a plurality of disks in the distributed storage includes a physically separated partition dedicated to storing the data of the file system that is most frequently accessed. The distribution of data is based at least in part on the presence of the physically separated partition.

Abstract: A distributed system is adapted to manage the performance of distributed processes. In one aspect, multiple stripes associated with a data item are stored in a distributed storage. The stored stripes include one or more stripes of redundancy information for the data item. A distributed process including at least one task is performed. During performance of the distributed process, a determination is made as to whether to perform an accelerated data retrieval operation. Responsive to a determination to perform an accelerated data retrieval operation, at least one of the one or more stripes of redundancy information for the data item is requested from the distributed storage. Other stripes associated with the data item may also be requested from the distributed storage. After a sufficient subset of stripes associated with the data item is received, the data item is reconstructed using the subset.

Abstract: A distributed computing system executes a MapReduce job on streamed data that includes an arbitrary amount of imbalance with respect to the frequency distribution of the data keys in the dataset. A map task module maps the dataset to a coarse partitioning, and generates a list of the top K keys with the highest frequency among the dataset. A sort task module employs a plurality of sorters to read the coarse partitioning and sort the data into buckets by data key. The values for the top K most frequent keys are separated into single-key buckets. The other less frequently occurring keys are assigned to buckets that each have multiple keys assigned to it. Then, more than one worker is assigned to each single-key bucket. The output of the multiple workers assigned to each respective single-key bucket is stitched together.

Abstract: A distributed system is adapted to manage the performance of distributed processes. In one aspect, multiple stripes associated with a data item are stored in a distributed storage. The stored stripes include one or more stripes of redundancy information for the data item. A distributed process including at least one task is performed. During performance of the distributed process, a determination is made as to whether to perform an accelerated data retrieval operation. Responsive to a determination to perform an accelerated data retrieval operation, at least one of the one or more stripes of redundancy information for the data item is requested from the distributed storage. Other stripes associated with the data item may also be requested from the distributed storage. After a sufficient subset of stripes associated with the data item is received, the data item is reconstructed using the subset.

Abstract: Accesses to a number of data blocks stored in a distributed storage are observed. Following observation of the accesses, the stored data blocks are redistributed. In one aspect, redistribution of the data blocks includes determining the access patterns for one or more of the data blocks based on the observed accesses, and determining the storage sizes for the one or more data blocks. Thereafter, based on the determined access patterns and determined storage sizes, the one or more data blocks are sorted. Subsequently, the one or more data blocks are redistributed or rebalanced across a number of storage devices of the distributed storage based on the sorting. In one aspect, the one or more data blocks are redistributed according to either a uniform distribution scheme or a proportional distribution scheme.