Abstract: A method, system and apparatus for efficient storage of small files in a segment-based deduplication scheme by allocating multiple small files to a single data segment is provided. A mechanism for distinguishing between large files (e.g., files that are on the order of the size of a segment or larger) and smaller files, and starting a new segment at the beginning of a large file is also provided. A file attribute-based system for determining an identity of a small file at which to begin a new segment and then allocating subsequent small files to that segment and contiguous segments until a next small file having an appropriate attribute subsequently is encountered to begin a new segment is further provided. In one aspect of the present invention a filename hash is used for file attribute analysis to determine when a new segment should begin.

Abstract: Detection and proper deduplication of a re-started data stream in a segmentation analysis-based deduplication system are provided by retaining information about a previous data stream and using that information when performing segmentation of the re-started data stream. Information such as a segment size associated with a last data object received in the previous data stream and a record of how much data was present in the last segment associated with the previous data stream is retained. The retained segment size information is used to set a first data object segment size of the re-started data stream, and the size of last segment information is used to determine how much information should be put in the first segment associated with the re-started data stream in order to maintain proper alignment of the remainder of the segments for the first data object in the re-started data stream for deduplication.