Abstract: Embodiments are directed towards managing the distribution of tasks in a storage system. An execution path for tasks may be generated based on the type of the task and characteristics of the storage system such that the execution path includes storage computers in a storage system. The tasks may be provided to each storage computer in the execution path. A working set of intermediate results may be generated on the storage computer in the execution path. If there is more than one storage computer in the execution path, working sets may be iteratively communicated to a next storage computer in the execution path such that the next storage computer employs a previously generated working set to generate a next working set until each storage computer in the execution path has been employed to generate a working set. The results may be stored on the storage computers.

Abstract: Embodiments are directed to storing data in a storage system. Data portions may be provided from blocks of write data provided by client computers. A storage location identifier may be associated with each data portion based on the content of the each data portion. Data portions may be stored in a memory buffer that may correspond to the storage location identifiers. If an amount of data stored in the memory buffer exceeds a threshold value, or if a maximum latency time expires, the data portions are stored on a storage location that corresponds to the memory buffer, such that, the maximum latency time is selected to prevent a collision caused by simultaneously storing of data portions in the same storage location. A write acknowledgement message may be provided to the client computers indicating that the blocks of write data are stored to stable storage.

Abstract: Embodiments are directed to storing data in a storage system. Data portions may be provided from blocks of write data provided by client computers. A storage location identifier may be associated with each data portion based on the content of the each data portion. Data portions may be stored in a memory buffer that may correspond to the storage location identifiers. If an amount of data stored in the memory buffer exceeds a threshold value, or if a maximum latency time expires, the data portions are stored on a storage location that corresponds to the memory buffer, such that, the maximum latency time is selected to prevent a collision caused by simultaneously storing of data portions in the same storage location. A write acknowledgement message may be provided to the client computers indicating that the blocks of write data are stored to stable storage.

Abstract: Embodiments are directed towards storing data in a storage system. A data controller may obtain a write request and write data from a client computer. A write message may be generated and provided to a data coordinator computer. The data coordinator may communicate the write message to a plurality of L-node computers. The data coordinator may obtain write confirmation messages from the L-node computers that indicate that the write data is stored. If enough write confirmation messages are obtained to indicate that a quorum is reached, the data coordinator may communicate a save confirmation message to the data controller. The data controller may generate a write acknowledgement message based on the save confirmation message provided by the data coordinator. The data controller may provide the write acknowledgement message to the client computer that made the original write request.

Abstract: Embodiments are directed towards managing data storage that may experience a data failure. If a repair event is associated with a data storage failure, a new repair task may be generated and added to a task list. A priority value for each repair task in the task list may be determined based in part on the mean-time-to-data-loss (MTTDL) value associated with each repair task in the task list such that a lower MTTDL may indicate a higher priority value over a lower MTTDL. One or more repair tasks may be promoted to become active repair tasks based on the priority value the repair tasks such that the promoted repair tasks have a higher priority that than other repair tasks in the task list, if any. Each active repair task may be executed to repair one or more associated the storage failures.

Abstract: Embodiments are directed towards managing the distribution of tasks in a storage system. An execution path for tasks may be generated based on the type of the task and characteristics of the storage system such that the execution path includes storage computers in a storage system. The tasks may be provided to each storage computer in the execution path. A working set of intermediate results may be generated on the storage computer in the execution path. If there is more than one storage computer in the execution path, working sets may be iteratively communicated to a next storage computer in the execution path such that the next storage computer employs a previously generated working set to generate a next working set until each storage computer in the execution path has been employed to generate a working set. The results may be stored on the storage computers.

Abstract: Embodiments are directed towards managing the distribution of tasks in a storage system. An execution path for tasks may be generated based on the type of the task and characteristics of the storage system such that the execution path includes storage computers in a storage system. The tasks may be provided to each storage computer in the execution path. A working set of intermediate results may be generated on the storage computer in the execution path. If there is more than one storage computer in the execution path, working sets may be iteratively communicated to a next storage computer in the execution path such that the next storage computer employs a previously generated working set to generate a next working set until each storage computer in the execution path has been employed to generate a working set. The results may be stored on the storage computers.

Abstract: Embodiments are directed towards managing data storage that may experience a data failure. If a repair event is associated with a data storage failure, a new repair task may be generated and added to a task list. A priority value for each repair task in the task list may be determined based in part on the mean-time-to-data-loss (MTTDL) value associated with each repair task in the task list such that a lower MTTDL may indicate a higher priority value over a lower MTTDL. One or more repair tasks may be promoted to become active repair tasks based on the priority value the repair tasks such that the promoted repair tasks have a higher priority that than other repair tasks in the task list, if any. Each active repair task may be executed to repair one or more associated the storage failures.

Abstract: Embodiments are directed towards efficient erasure coding for storage systems. A smallest integer in a set may be determined based on a number of storage devices in the storage system such that the determined integer is greater or equal to the number of storage devices. The smallest integer in the set may be determined from the set {2N?1} where N is an integer based on the number of storage devices. A matrix may be generated based on a number of columns equal to the determined integer and a number of rows equal to the number of columns plus a number of global repair symbol devices and plus a number of virtual repair symbol devices. Each virtual repair symbol device enables an additional of level of protection against failures in the storage system.

Abstract: Embodiments are directed towards writing data to a shingle of storage devices enabling storage devices not in the shingle to be turned off. A shingle may be a sub-group of more than one storage devices selected from a larger group of storage devices. A plurality of data repair sets may be written to various subsets of the storage devices in the shingle. Each data repair set may include data block(s) and repair block(s)—with each block being stored on a different storage device in the shingle. When a condition is satisfied, the shingle may be modified to either remove at least one storage device from the shingle, add at least one other storage device to the shingle, or both. In various embodiments, the at least one other storage device may be a storage device in the group of storage devices that is not currently included in the shingle.

Abstract: Embodiments are directed towards a controller that provides individual network accessibility to a storage drive. The controller may include a first connector operative to couple with a storage-drive connector, a second connector operative to couple with a backplane connector of a multi-storage-drive chassis, memory, and processor. The controller may convert communication received through the first connector into an Ethernet protocol for output through the second connector, and convert communication received through the second connector into a storage-drive protocol for output through the first connector. A physical shape of the controller may fit adjacent to the storage-drive connector and occupy less space than is bounded by peripheral edges of an end of a separate housing of a storage drive coupled to the storage-drive connector. The controller may manage power provided to the storage drive and may coordinate with other controllers to manage power-up sequences of multiple storage drives.

Abstract: Embodiments are directed towards managing data storage that may experience a data failure. If a repair event is associated with a data storage failure, a new repair task may be generated and added to a task list. A priority value for each repair task in the task list may be determined based in part on the mean-time-to-data-loss (MTTDL) value associated with each repair task in the task list such that a lower MTTDL may indicate a higher priority value over a lower MTTDL. One or more repair tasks may be promoted to become active repair tasks based on the priority value the repair tasks such that the promoted repair tasks have a higher priority that than other repair tasks in the task list, if any. Each active repair task may be executed to repair one or more associated the storage failures.

Abstract: Embodiments are directed towards storing data in a storage system using an extensible data path. Data files may be provided to a caching tier in a storage system. If data files remain in the caching tier longer than a time limit, those data files may be removed from the caching tier and provided to a processing pipeline. The processing pipeline may be coupled to a capacity tier of the storage system. Filters to include in the processing pipeline may be determined based on the type of the data files. The data files may be updated based on applying each filter, such that each update corresponding to each filter may be cumulatively applied to each data file. Each updated data file may be stored in the capacity tier of the storage system after each filter in the processing pipeline has been applied.

Abstract: Embodiments are directed towards a controller that provides individual network accessibility to a storage drive. The controller may include a first connector operative to couple with a storage-drive connector, a second connector operative to couple with a backplane connector of a multi-storage-drive chassis, memory, and processor. The controller may convert communication received through the first connector into an Ethernet protocol for output through the second connector, and convert communication received through the second connector into a storage-drive protocol for output through the first connector. A physical shape of the controller may fit adjacent to the storage-drive connector and occupy less space than is bounded by peripheral edges of an end of a separate housing of a storage drive coupled to the storage-drive connector. The controller may manage power provided to the storage drive and may coordinate with other controllers to manage power-up sequences of multiple storage drives.

Abstract: Embodiments are directed towards a controller that provides individual network accessibility to a storage drive. The controller may include a first connector operative to couple with a storage-drive connector, a second connector operative to couple with a backplane connector of a multi-storage-drive chassis, memory, and processor. The controller may convert communication received through the first connector into an Ethernet protocol for output through the second connector, and convert communication received through the second connector into a storage-drive protocol for output through the first connector. A physical shape of the controller may fit adjacent to the storage-drive connector and occupy less space than is bounded by peripheral edges of an end of a separate housing of a storage drive coupled to the storage-drive connector. The controller may manage power provided to the storage drive and may coordinate with other controllers to manage power-up sequences of multiple storage drives.