Abstract: Example implementations relate to operations in a storage system. An example includes receiving a data unit to be stored in a persistent storage of a deduplication storage system, and identifying a container index that includes metadata regarding the data unit. The example also includes identifying a first container entity group (CEG) object that is recorded, in the container index, as a deduplicated storage location of the data unit. The example also includes, in response to a determination that that a validation state of the first CEG object is recorded as invalid in the container index, storing the data unit in a second CEG object. The example also includes modifying the container index to record the second CEG object as the deduplicated storage location of the data unit.

Abstract: Example implementations relate to deduplication operations in a storage system. An example includes detecting metadata changes associated with a container index of the deduplication storage system, and recording the metadata changes in multiple journals included in a journal group. The example also includes identifying, in the journal group, a first journal having a filled amount that is less than a fold threshold. The example also includes, determining whether the identified first journal is stagnant based on one or more journal metrics of the identified first journal, and in response to a determination that the identified first journal is stagnant based on the one or more journal metrics, modifying the container index to include each metadata change recorded in the identified first journal.

Abstract: Example implementations relate to deduplication operations in a storage system. An example includes initiating a housekeeping job to delete a backup item stored in a deduplication storage system; identifying a set of container indexes affected by the housekeeping job, the set of container indexes including at least a first container index; in response to a determination that a housekeeping work map includes a first work entry associated with the first container index, determining an ingest count of new data units added to a first container during an ingest of the backup item into the deduplication storage system, the first container associated with the first container index; and adding the determined ingest count to the cumulative counter included in the first work entry associated with the first container index.

Abstract: Example implementations relate to deduplication operations in a storage system. An example includes receiving a set of data units to be stored in persistent storage of a deduplication storage system, and recording the arrival order of the set of data units in a manifest. The example also includes, in response to determining that an average unit size of the set of data units is smaller than a minimum size threshold, identifying a first data unit that is smaller than the minimum size threshold. The example also includes generating an aggregated data unit including the first data unit and a second data unit, and, in response to determining that the size of the aggregated data unit is larger than a target size threshold, replacing, in the manifest, the first data unit and the second data unit with the aggregated data unit.

Abstract: Example implementations relate to deduplication operations in a storage system. An example includes receiving a data stream that includes multiple locality portions. The example also includes, upon determining that a first container index for a first portion has insufficient capacity to store metadata of a first set of new data units in the first portion, instantiating a first overflow container index associated with the first portion, and storing the metadata of the first set of new data units in the first overflow container index. The example also includes, upon determining that a second container index associated with a second portion has insufficient capacity to store metadata of a second set of new data units in the second portion, instantiating a second overflow container index associated with the second portion, and storing the metadata of the second set of new data units in the second overflow container index.

Abstract: Example implementations relate to deduplication operations in a storage system. An example includes receiving a data stream to be stored in persistent storage of a deduplication storage system, where the data stream includes multiple locality portions that each include multiple data units. The example also includes identifying a new data unit to be indexed by a particular container index that is associated with a locality portion including the new data unit. The example also includes, in response to a determination that the count of matching entries in the particular container index has reached a maximum number of matching entries, converting a first matching entry into a first nonmatching entry in the particular container index, and storing metadata of the new data unit in a new matching entry of the particular container index.

Abstract: Example implementations relate to deduplication operations in a storage system. An example includes receiving a stream of data units to be stored in a persistent storage of a deduplication storage system; loading a first container index into the memory to index one or more data units included in the stream; updating the first container index to indicate a storage location of a first data unit included in the stream; reading a manifest list in the first container index to identify a first manifest indexed by the first container index; and, while the first container index remains loaded in the memory to index the one or more data units, in response to a determination that the identified first manifest is included in the first backup item that has been marked for deletion, decrementing a reference count of a second data unit.

Abstract: Example implementations relate to deduplication operations in a storage system. An example includes receiving a stream of data units to be stored in a persistent storage of a deduplication storage system; loading a first container index into the memory to index one or more data units included in the stream; updating the first container index to indicate a storage location of a first data unit included in the stream; reading a manifest list in the first container index to identify a first manifest indexed by the first container index; and, while the first container index remains loaded in the memory to index the one or more data units, in response to a determination that the identified first manifest is included in the first backup item that has been marked for deletion, decrementing a reference count of a second data unit.

Abstract: Example implementations relate to deduplication operations in a storage system. An example includes generating a housekeeping work map to delete a backup item stored in a deduplication storage system; selecting a first work entry of the housekeeping work map, where the first work entry identifies a first container index and a first manifest; in response to a selection of the first work entry, loading the first container index into the memory, the first container index comprising a back-reference data structure; identifying, in the back-reference data structure, a back-reference entry indexed to the first manifest; determining, using the back-reference entry indexed to the first manifest, a first set of data units included in the first manifest and that are indexed in the first container index; and decrementing, in the first container index, a set of reference counts for the determined first set of data units.

Abstract: Example implementations relate to deduplication operations in a storage system. An example includes generating a housekeeping work map to delete a backup item stored in a deduplication storage system; selecting a first work entry of the housekeeping work map, where the first work entry identifies a first container index and a first manifest; in response to a selection of the first work entry, loading the first container index into the memory, the first container index comprising a back-reference data structure; identifying, in the back-reference data structure, a back-reference entry indexed to the first manifest; determining, using the back-reference entry indexed to the first manifest, a first set of data units included in the first manifest and that are indexed in the first container index; and decrementing, in the first container index, a set of reference counts for the determined first set of data units.

Abstract: Example implementations relate to deduplication operations in a storage system. An example includes initiating a housekeeping job to delete a backup item stored in a deduplication storage system; identifying a set of container indexes affected by the housekeeping job, the set of container indexes including at least a first container index; in response to a determination that a housekeeping work map includes a first work entry associated with the first container index, determining an ingest count of new data units added to a first container during an ingest of the backup item into the deduplication storage system, the first container associated with the first container index; and adding the determined ingest count to the cumulative counter included in the first work entry associated with the first container index.

Abstract: Example implementations relate to metadata operations in a storage system. An example includes detecting, by a storage controller of a deduplication storage system, a housekeeping operation to perform updates to data of a set of container indexes of stored deduplicated data; in response to a detection of the housekeeping operation, the storage controller identifying, among the set of container indexes, a plurality of container indexes associated with a particular journal group, the particular journal group comprising a plurality of journals to store updates to data of the plurality of container indexes, respectively; and maintaining, by the storage controller, the particular journal group loaded in memory until all of the updates of the housekeeping operation to the plurality of container indexes have been stored in the respective journals of the particular journal group.

Abstract: Example implementations relate to metadata operations in a storage system. An example includes receiving, by a storage controller of a deduplication storage system, a plurality of data streams to be stored in persistent storage of the deduplication storage system; identifying, by the storage controller, a set of journals in a first journal group that are modified during a first backup process; determining, by the storage controller, a count of the set of journals that are modified during the first backup process; comparing, by the storage controller, the determined count to a migration threshold; and migrating, by the storage controller, at least one journal of the set of journals to a second journal group based at least on a comparison of the determined count to the migration threshold.

Abstract: Example implementations relate to metadata operations in a storage system. An example includes receiving, by a storage controller of a deduplication storage system, a plurality of data streams to be stored in persistent storage of the deduplication storage system; identifying, by the storage controller, a set of journals in a first journal group that are modified during a first backup process; determining, by the storage controller, a count of the set of journals that are modified during the first backup process; comparing, by the storage controller, the determined count to a migration threshold; and migrating, by the storage controller, at least one journal of the set of journals to a second journal group based at least on a comparison of the determined count to the migration threshold.

Abstract: Example implementations relate to metadata operations in a storage system. An example includes detecting, by a storage controller of a deduplication storage system, a housekeeping operation to perform updates to data of a set of container indexes of stored deduplicated data; in response to a detection of the housekeeping operation, the storage controller identifying, among the set of container indexes, a plurality of container indexes associated with a particular journal group, the particular journal group comprising a plurality of journals to store updates to data of the plurality of container indexes, respectively; and maintaining, by the storage controller, the particular journal group loaded in memory until all of the updates of the housekeeping operation to the plurality of container indexes have been stored in the respective journals of the particular journal group.