Abstract: A system can receive a file in parts. The system can write a first part to a first temporary file that corresponds to the first part's length at a first location of the first temporary file based on the its part number. The system can for a second part having a same length as the first part, write the second part to the first temporary file at a second location of the first temporary file based on its part number. The system can, in response to receiving a third part that has a different length than the first part, write the third part to a second temporary file at a third location of the second temporary file based on its part number. The system can, after receiving the parts, write data of the second temporary file to an end of the first temporary file, and rename the temporary file.

Abstract: A system can receive a file in parts. The system can write a first part to a first temporary file that corresponds to the first part's length at a first location of the first temporary file based on the its part number. The system can for a second part having a same length as the first part, write the second part to the first temporary file at a second location of the first temporary file based on its part number. The system can, in response to receiving a third part that has a different length than the first part, write the third part to a second temporary file at a third location of the second temporary file based on its part number. The system can, after receiving the parts, write data of the second temporary file to an end of the first temporary file, and rename the temporary file.

Abstract: Implementations are provided herein for data deduplication, and more particularly, to post-process data deduplication on a large scale out storage system. Multiple techniques and implementations are disclosed that offer greater efficiency, higher performance, and more stability when performing post-process data deduplication at large scale. Disclosed implementations are based on a process for data deduplication involving four main phases: enumeration, commonality, sharing, and update. Multi-level hashing can be used to identify candidates for deduplication during the enumeration phase, providing a more efficient use of compute resources. In addition, datasets can be phase rotated through the post-process deduplication steps providing a more controllable deduplication environment as well as a more efficient use of resources.