Abstract: A distributed storage system may store data object instances in persistent storage and may store keymap information for those data object instances in a distributed hash table on multiple computing nodes. Each data object instance may include a composite key containing a user key. The keymap information for each data object instance may map the user key to a locator and the locator to the data object instance. A request to store or retrieve keymap information for a data object instance may be routed to a particular computing node based on a consistent hashing scheme in which a hash function is applied to a portion of the composite key of the data object instance. Thus, related entries may be clustered on the same computing nodes. The portion of the key to which the hash function is applied may include a pre-determined number of bits or be identified using a delimiter.

Abstract: Systems and methods for providing object versioning in a storage system may support the logical deletion of stored objects. In response to a delete operation specifying both a user key and a version identifier, the storage system may permanently delete the specified version of an object having the specified key. In response to a delete operation specifying a user key, but not a version identifier, the storage system may create a delete marker object that does not contain object data, and may generate a new version identifier for the delete marker. The delete marker may be stored as the latest object version of the user key, and may be addressable in the storage system using a composite key comprising the user key and the new version identifier. Subsequent attempts to retrieve the user key without specifying a version identifier may return an error, although the object was not actually deleted.

Abstract: A distributed storage system may store data object instances in persistent storage and may store keymap information for those data object instances in a distributed hash table on multiple computing nodes. Each data object instance may include a composite key containing a user key. The keymap information for each data object instance may map the user key to a locator and the locator to the data object instance. A request to store or retrieve keymap information for a data object instance may be routed to a particular computing node based on a consistent hashing scheme in which a hash function is applied to a portion of the composite key of the data object instance. Thus, related entries may be clustered on the same computing nodes. The portion of the key to which the hash function is applied may include a pre-determined number of bits or be identified using a delimiter.

Abstract: Systems and methods for providing object versioning in a storage system may support the logical deletion of stored objects. In response to a delete operation specifying both a user key and a version identifier, the storage system may permanently delete the specified version of an object having the specified key. In response to a delete operation specifying a user key, but not a version identifier, the storage system may create a delete marker object that does not contain object data, and may generate a new version identifier for the delete marker. The delete marker may be stored as the latest object version of the user key, and may be addressable in the storage system using a composite key comprising the user key and the new version identifier. Subsequent attempts to retrieve the user key without specifying a version identifier may return an error, although the object was not actually deleted.

Abstract: A distributed storage system may store data object instances in persistent storage and may store keymap information for those data object instances in a distributed hash table on multiple computing nodes. Each data object instance may include a composite key containing a user key. The keymap information for each data object instance may map the user key to a locator and the locator to the data object instance. A request to store or retrieve keymap information for a data object instance may be routed to a particular computing node based on a consistent hashing scheme in which a hash function is applied to a portion of the composite key of the data object instance. Thus, related entries may be clustered on the same computing nodes. The portion of the key to which the hash function is applied may include a pre-determined number of bits or be identified using a delimiter.

Abstract: A method and apparatus for criterion-based retention of data object versions are disclosed. In the method and apparatus, a plurality of keys are sorted in accordance with an ordering scheme, whereby a key of the plurality of keys has an associated version of a data object and a timestamp. The key is inspected in accordance with the ordering scheme to determine based at least in part on the timestamp whether a criterion for performing an action on the associated version of the data object is satisfied. If the criterion is satisfied, a marker key is added to the plurality of keys, whereby the marker key precedes the inspected key according to the ordering scheme and indicates that the criterion is satisfied.

Abstract: Systems and methods for providing object versioning in a storage system may support the logical deletion of stored objects. In response to a delete operation specifying both a user key and a version identifier, the storage system may permanently delete the specified version of an object having the specified key. In response to a delete operation specifying a user key, but not a version identifier, the storage system may create a delete marker object that does not contain object data, and may generate a new version identifier for the delete marker. The delete marker may be stored as the latest object version of the user key, and may be addressable in the storage system using a composite key comprising the user key and the new version identifier. Subsequent attempts to retrieve the user key without specifying a version identifier may return an error, although the object was not actually deleted.

Abstract: A distributed storage system may store data object instances in persistent storage and may store keymap information for those data object instances in a distributed hash table on multiple computing nodes. Each data object instance may include a composite key containing a user key. The keymap information for each data object instance may map the user key to a locator and the locator to the data object instance. A request to store or retrieve keymap information for a data object instance may be routed to a particular computing node based on a consistent hashing scheme in which a hash function is applied to a portion of the composite key of the data object instance. Thus, related entries may be clustered on the same computing nodes. The portion of the key to which the hash function is applied may include a pre-determined number of bits or be identified using a delimiter.

Abstract: Windowed queries with event-based open times are described for use in data analytics for streaming data. A window semantic may be used to define windowed queries of an analytics application that, when executed against an incoming data stream, open a window at a time when a first event associated with a grouping key is received by a windowed query. The window remains open for a predefined duration specified in the windowed query, and query results may be generated based on any events associated with the grouping key that are received while the window remains open. After a lapse of the predefined duration, the window closes, and the query results may be output in association with the window.

Abstract: A distributed storage system may store data object instances in persistent storage and may store keymap information for those data object instances in a distributed hash table on multiple computing nodes. Each data object instance may include a composite key containing a user key. The keymap information for each data object instance may map the user key to a locator and the locator to the data object instance. A request to store or retrieve keymap information for a data object instance may be routed to a particular computing node based on a consistent hashing scheme in which a hash function is applied to a portion of the composite key of the data object instance. Thus, related entries may be clustered on the same computing nodes. The portion of the key to which the hash function is applied may include a pre-determined number of bits or be identified using a delimiter.

Abstract: Methods and apparatus for lifecycle support for storage objects are disclosed. A storage medium stores program instructions that when executed on a processor implement a storage lifecycle manager of a multi-tenant storage service. The lifecycle manager determines a lifecycle policy to be implemented for a storage object. The policy indicates lifecycle transitions of the object, including a transition from a first service level to a second service level. The lifecycle manager initiates modification operations corresponding to the storage object's data in accordance with the lifecycle policy, and verifies that the modification operations succeeded. The lifecycle manager provides, in response to a query, an indication of a current service level of the storage object.

Abstract: Methods and apparatus for lifecycle support for storage objects are disclosed. A storage medium stores program instructions that when executed on a processor implement a storage lifecycle manager of a multi-tenant storage service. The lifecycle manager determines a lifecycle policy to be implemented for a storage object. The policy indicates lifecycle transitions of the object, including a transition from a first service level to a second service level. The first service level and the second service level may each specify a maximum number of versions of the storage object. The lifecycle manager initiates modification operations corresponding to the storage object's data in accordance with the lifecycle policy, and verifies that the modification operations succeeded. The lifecycle manager provides, in response to a query, an indication of a current service level of the storage object.

Abstract: A customer of a resource provider environment can apply policies at the data object level that will live with a data object during its lifecycle, even as the object moves across trusted boundaries. A customer can classify data, causing tags and/or predicates to be applied to the corresponding data object. Each tag corresponds to a policy, with predicates relating to various actions that can be performed on the data. A chain of custody is maintained for each data object, such that any changes to the object, tags, or policies for the data can be determined, as may be required for various audit processes. The support of such policies also enables the resource provider environment to function as an intermediary, whereby a third party can receive the data along with the tags, policies, and chain of custody as long as the environment trusts the third party to receive the data object.

Abstract: Systems and methods for providing object versioning in a storage system may support the logical deletion of stored objects. In response to a delete operation specifying both a user key and a version identifier, the storage system may permanently delete the specified version of an object having the specified key. In response to a delete operation specifying a user key, but not a version identifier, the storage system may create a delete marker object that does not contain object data, and may generate a new version identifier for the delete marker. The delete marker may be stored as the latest object version of the user key, and may be addressable in the storage system using a composite key comprising the user key and the new version identifier. Subsequent attempts to retrieve the user key without specifying a version identifier may return an error, although the object was not actually deleted.

Abstract: A customer of a resource provider environment can apply policies at the data object level that will live with a data object during its lifecycle, even as the object moves across trusted boundaries. A customer can classify data, causing tags and/or predicates to be applied to the corresponding data object. Each tag corresponds to a policy, with predicates relating to various actions that can be performed on the data. A chain of custody is maintained for each data object, such that any changes to the object, tags, or policies for the data can be determined, as may be required for various audit processes. The support of such policies also enables the resource provider environment to function as an intermediary, whereby a third party can receive the data along with the tags, policies, and chain of custody as long as the environment trusts the third party to receive the data object.

Abstract: Methods and apparatus for a pipelined architecture for storage lifecycles are disclosed. A method includes identifying, during a transition candidate discovery iteration, storage objects of a multi-tenant service for which respective lifecycle transitions are to be scheduled, including a particular storage object identified as being a candidate for a particular lifecycle transition. The method includes generating transition job objects, each job comprising respective indications of one or more of the storage objects identified during the discovery iteration. The method includes assigning, during a transition execution iteration, resources to initiate storage operations for the particular lifecycle transition, and initiating, using the assigned resources, storage operations for the particular lifecycle transition.

Abstract: A method and apparatus for criterion-based retention of data object versions are disclosed. In the method and apparatus, a plurality of keys are sorted in accordance with an ordering scheme, whereby a key of the plurality of keys has an associated version of a data object and a timestamp. The key is inspected in accordance with the ordering scheme to determine based at least in part on the timestamp whether a criterion for performing an action on the associated version of the data object is satisfied. If the criterion is satisfied, a marker key is added to the plurality of keys, whereby the marker key precedes the inspected key according to the ordering scheme and indicates that the criterion is satisfied.

Abstract: Systems and methods for providing object versioning in a storage system may support the logical deletion of stored objects. In response to a delete operation specifying both a user key and a version identifier, the storage system may permanently delete the specified version of an object having the specified key. In response to a delete operation specifying a user key, but not a version identifier, the storage system may create a delete marker object that does not contain object data, and may generate a new version identifier for the delete marker. The delete marker may be stored as the latest object version of the user key, and may be addressable in the storage system using a composite key comprising the user key and the new version identifier. Subsequent attempts to retrieve the user key without specifying a version identifier may return an error, although the object was not actually deleted.

Abstract: A method and apparatus for managing keys pertaining to data objects are disclosed. In the method and apparatus, a plurality of keys that are associated with a plurality of data objects are retained, whereby the plurality of keys are capable of being listed in accordance with an ordering scheme. The ordering scheme is used to inspect the plurality of keys to determine whether a data object associated with a key of the plurality of keys satisfies a criterion. One or more actions are taken on the key or associated data object based at least in part on determining that the criterion is satisfied, whereby an order of taking the one or more actions is a reverse of an order by which the key is listed in accordance with the ordering scheme.

Abstract: A distributed storage system may store data object instances in persistent storage and may cache keymap information for those data object instances. The system may cache a latest symbolic key entry for some user keys of the data object instances. When a request is made for the latest version of stored data object instances having a specified user key, the latest version may be determined dependent on whether a latest symbolic key entry exists for the specified user key, and keymap information for the latest version may be returned. When storing keymap information, a flag may be set to indicate that a corresponding latest symbolic key entry should be updated. The system may delete a latest symbolic key entry for a particular user key from the cache in response to determining that no other requests involving the keymap information for data object instances having the particular user key are pending.