Abstract: A method includes receiving trace data representing access information about files stored in a large-scale distributed storage system, identifying file access patterns based on the trace data, receiving metadata information associated with the files stored in the large-scale distributed storage system, and generating a preferred storage parameter for each file based on the received metadata information and the identified file access patterns. The method also includes receiving, file reliability or accessibility information of a new file, determining whether the received file reliability or accessibility information of the new file matches information of a file group of the files in the large-scale distributed storage system, and when the file reliability or accessibility information of the new file matches the information of the file group, storing the new file in the large-scale distributed storage system using the preferred storage parameter associated with the file group.

Abstract: Methods, systems, and apparatus for automatic sharding and load balancing in a distributed data processing system. In one aspect, a method includes determining workload distribution for an application across worker computers and in response to determining a load balancing operation is required: selecting a first worker computer having a highest load measure relative to respective load measure of the other work computers; determining one or more move operations for a partition of data assigned to the first worker computer and a weight for each move operation; and selecting the move operation with a highest weight the selected move operation.

Abstract: A predictive method for scheduling of the operations is described. The predictive method utilizes data generated from computing an expected lifetime of the individual files or objects within the container. The expected lifetime of individual files or objects can be generated based on machine learning techniques. Operations such as garbage collection are scheduled at an epoch where computational efficiencies are realized for performing the operation.

Abstract: The technology is directed to cache admission control. One or more processors may categorize access requests for data items in a cache storage into a plurality of categories and collect information on the access requests over time. Based on the collected information, a utility value for caching data items in each category of the plurality of categories may be determined. Newly requested data items may be admitted into the cache storage in an order according to the corresponding utility values of their respective categories.

Abstract: Methods, systems, and apparatus for automatic sharding and load balancing in a distributed data processing system. In one aspect, a method includes determining workload distribution for an application across worker computers and in response to determining a load balancing operation is required: selecting a first worker computer having a highest load measure relative to respective load measure of the other work computers; determining one or more move operations for a partition of data assigned to the first worker computer and a weight for each move operation; and selecting the move operation with a highest weight the selected move operation.

Abstract: The technology is directed to cache admission control. One or more processors may categorize access requests for data items in a cache storage into a plurality of categories and collect information on the access requests over time. Based on the collected information, a utility value for caching data items in each category of the plurality of categories may be determined. Newly requested data items may be admitted into the cache storage in an order according to the corresponding utility values of their respective categories.

Abstract: A predictive method for scheduling of the operations is described. The predictive method utilizes data generated from computing an expected lifetime of the individual files or objects within the container. The expected lifetime of individual files or objects can be generated based on machine learning techniques. Operations such as garbage collection are scheduled at an epoch where computational efficiencies are realized for performing the operation.

Abstract: A method includes receiving trace data representing access information about files stored in a large-scale distributed storage system, identifying file access patterns based on the trace data, receiving metadata information associated with the files stored in the large-scale distributed storage system, and generating a preferred storage parameter for each file based on the received metadata information and the identified file access patterns. The method also includes receiving, file reliability or accessibility information of a new file, determining whether the received file reliability or accessibility information of the new file matches information of a file group of the files in the large-scale distributed storage system, and when the file reliability or accessibility information of the new file matches the information of the file group, storing the new file in the large-scale distributed storage system using the preferred storage parameter associated with the file group.

Abstract: A method includes receiving trace data representing access information about files stored in a large-scale distributed storage system, identifying file access patterns based on the trace data, receiving metadata information associated with the files stored in the large-scale distributed storage system, and generating a preferred storage parameter for each file based on the received metadata information and the identified file access patterns. The method also includes receiving, file reliability or accessibility information of a new file, determining whether the received file reliability or accessibility information of the new file matches information of a file group of the files in the large-scale distributed storage system, and when the file reliability or accessibility information of the new file matches the information of the file group, storing the new file in the large-scale distributed storage system using the preferred storage parameter associated with the file group.

Abstract: A method includes receiving trace data representing access information about files stored in a large-scale distributed storage system, identifying file access patterns based on the trace data, receiving metadata information associated with the files stored in the large-scale distributed storage system, and generating a preferred storage parameter for each file based on the received metadata information and the identified file access patterns. The method also includes receiving, file reliability or accessibility information of a new file, determining whether the received file reliability or accessibility information of the new file matches information of a file group of the files in the large-scale distributed storage system, and when the file reliability or accessibility information of the new file matches the information of the file group, storing the new file in the large-scale distributed storage system using the preferred storage parameter associated with the file group.

Abstract: A transactional memory system uses a volatile memory as primary storage for transactions. Data is selectively stored in a non-volatile memory to impart durability to the transactional memory system to allow the transactional memory system to be restored to a consistent state in the event of data loss to the volatile memory.

Abstract: A method includes receiving trace data representing access information about files stored in a large-scale distributed storage system, identifying file access patterns based on the trace data, receiving metadata information associated with the files stored in the large-scale distributed storage system, and generating a preferred storage parameter for each file based on the received metadata information and the identified file access patterns. The method also includes receiving, file reliability or accessibility information of a new file, determining whether the received file reliability or accessibility information of the new file matches information of a file group of the files in the large-scale distributed storage system, and when the file reliability or accessibility information of the new file matches the information of the file group, storing the new file in the large-scale distributed storage system using the preferred storage parameter associated with the file group.

Abstract: A method includes receiving trace data representing access information about files stored in a large-scale distributed storage system, identifying file access patterns based on the trace data, receiving metadata information associated with the files stored in the large-scale distributed storage system, and generating a preferred storage parameter for each file based on the received metadata information and the identified file access patterns. The method also includes receiving, file reliability or accessibility information of a new file, determining whether the received file reliability or accessibility information of the new file matches information of a file group of the files in the large-scale distributed storage system, and when the file reliability or accessibility information of the new file matches the information of the file group, storing the new file in the large-scale distributed storage system using the preferred storage parameter associated with the file group.

Abstract: A method includes receiving trace data representing access information about files stored in a large-scale distributed storage system, identifying file access patterns based on the trace data, receiving metadata information associated with the files stored in the large-scale distributed storage system, and generating a preferred storage parameter for each file based on the received metadata information and the identified file access patterns. The method also includes receiving, file reliability or accessibility information of a new file, determining whether the received file reliability or accessibility information of the new file matches information of a file group of the files in the large-scale distributed storage system, and when the file reliability or accessibility information of the new file matches the information of the file group, storing the new file in the large-scale distributed storage system using the preferred storage parameter associated with the file group.

Abstract: A method includes receiving trace data representing access information about files stored in a large-scale distributed storage system, identifying file access patterns based on the trace data, receiving metadata information associated with the files stored in the large-scale distributed storage system, and generating a preferred storage parameter for each file based on the received metadata information and the identified file access patterns. The method also includes receiving, file reliability or accessibility information of a new file, determining whether the received file reliability or accessibility information of the new file matches information of a file group of the files in the large-scale distributed storage system, and when the file reliability or accessibility information of the new file matches the information of the file group, storing the new file in the large-scale distributed storage system using the preferred storage parameter associated with the file group.

Abstract: Methods to determine and automatically recommend or adjust configuration parameters for storing files in large-scale distributed storage systems are disclosed. These methods may receive file metadata and trace data that allows the system to identify file access patterns. Additionally, the methods may receive information about distributed storage systems in a datacenter. This information can be used to choose storage parameters on a per-file basis for storing files.

Abstract: A hard disk drive includes a storage surface, a motor, a read/write had, and a number of regions into which the storage surface is divisible. Data is writable to and data is readable from the storage surface. The motor is to rotate the storage surface at a variable speed. The read/write head is to write data to and read data from the storage surface while the storage surface is rotated by the motor. Each region corresponds to a different speed at which the storage surface is rotated for the read/write head to write data to and read data from the region.

Abstract: A method for managing data items retrieved for storage in a prefetch memory buffer includes determining a probability that a first data item will be requested for retrieval. The method includes estimating a first request time at which the new data item will be requested. The method also includes determining a time differential for the first data item, wherein the time differential is determined based on current time and the first request time. The method includes calculating a first prefetch priority value for the first data item based on the first data item probability and the time differential. The method includes randomly comparing the first prefetch priority value of the first data item to the prefetch priority values of the one or more stored data items to identify at least one stored data item having a prefetch priority value lower than the first prefetch priority value.

Abstract: Systems and methods are discussed relating to allocation of memory from a fixed pool of fast memory within a data center having a data storage area equipped with that memory. Techniques include: receiving a request to write data in the storage area; identifying a file group associated with the write request; analyzing previous data activity traces associated with the file group; determining an available fast memory amount based on the total amount of fast memory in the fixed pool and a currently allocated amount of fast memory; determining a fast memory allocation for the file group based on the previous data activity traces, the available fast memory, and a fast memory constraint, the memory allocation including an allocation amount and a write probability; and providing information about the memory allocation to a file system of the data center, which writes the data based on the allocation amount and write probability.

Abstract: A system includes disk storage to provide differentiated storage QoS for a plurality of IO classes. Each IO class has a plurality of applications to it. A QoS controller collects IO statistics for each application and each class. The QoS controller adaptively determines an IO class assignment for each application based at least in part on the collected IO statistics.