Abstract: In general, in one aspect, a method for managing data in a data storage system includes receiving identifiers corresponding to different respective entries of a map stored in the data storage system, with a particular identifier corresponding to a particular entry of the map, the particular entry including a computed value corresponding to a particular portion of data stored in the data storage system and metadata indicating a location where the particular portion of data is stored in the data storage system, selecting, according to a first selection criterion, at least some of the identifiers for storage in a first portion of an index, and selecting, according to a second selection criterion, at least some of the identifiers for storage in a second portion of the index.

Abstract: In general, in one aspect, a method for managing data in a data storage system includes receiving identifiers corresponding to different respective entries of a map stored in the data storage system, with a particular identifier corresponding to a particular entry of the map, the particular entry including a computed value corresponding to a particular portion of data stored in the data storage system and metadata indicating a location where the particular portion of data is stored in the data storage system, selecting, according to a first selection criterion, at least some of the identifiers for storage in a first portion of an index, and selecting, according to a second selection criterion, at least some of the identifiers for storage in a second portion of the index.

Abstract: A method for constructing an index suitable for indexing a large set of records identified by long generally randomly distributed record names and for answering membership queries about the set, the method comprising assigning each different record a different record name, determining that a new record name is not already represented in the index by checking an entry in a first level index that is shorter than the complete new record name, and determining that a queried record name is already represented in the index by determining that the queried record name is represented in a second level index that contains enough information to reconstruct the complete queried record name.

Abstract: An index is constructed that is suitable for indexing a large set of records identified by randomly distributed record names and for answering membership queries about the set. The index has more than one level and the first-level index typically resides in RAM. Like a Bloom Filter, the first level index provides a non-membership indication with certainty and a membership indication with a controlled probability of error. Unlike a Bloom filter, entries in the first level index correspond uniquely to distinct records. Use is made of the statistical properties of a sorted list of random numbers in order to represent the first level index compactly.

Abstract: A method for constructing an index suitable for indexing a large set of records identified by long generally randomly distributed record names, and for answering membership queries about the set, the method comprising adding a new record to the set and assigning the new record a new record name using a process designed to produce names where at least a portion of each name is at least approximately random, determining that the new record name is not already represented in the index by checking a first level index that does not contain information sufficient to reconstruct the complete record names of records that have already been added to the index, abbreviating the new record name to form a new abbreviated name that is shorter than the new record name but that is sufficient to distinguish it from record names already represented in the index, combining the new abbreviated name with an abbreviated name of a record already represented in the first level index, to form a combined record name which is shorter t

Abstract: A method for constructing an index suitable for indexing a large set of records identified by long generally randomly distributed record names, and for answering membership queries about the set, the method comprising adding a new record to the set and assigning the new record a new record name using a process designed to produce names where at least a portion of each name is at least approximately random, determining that the new record name is not already represented in the index by checking a first level index, combining the new record name with record name information already represented in the index to form a combined record name which is shorter than the new record name, and adding the combined record name to the first level index to form a new first level index entry that represents the new record, wherein the first level index does not contain information sufficient to conclude that the new record name has been added to the index, wherein each different record in the set is assigned a different record

Abstract: A method for constructing an index suitable for indexing a large set of records identified by long generally randomly distributed record names and for answering membership queries about the set, the method comprising assigning each different record a different record name, determining that a new record name is not already represented in the index by checking an entry in a first level index that is shorter than the complete new record name, and determining that a queried record name is already represented in the index by determining that the queried record name is represented in a second level index that contains enough information to reconstruct the complete queried record name.