Abstract: Working set ratio estimations of data items in a sliding time window are determined to dynamically allocate storage for the data items. A working set ratio may be determined by accessing a fixed-size array that stores respective timestamps of last accesses of data items to determine which data items are useful to determine an estimate of a working set for the application within a range of time. The working set ratio is then determined from an estimated working set and an amount of computing resources allocated to the application by the estimated working set. The amount of the computing resources allocated to the application may then be automatically scaled according to the determine working set ratio.

Abstract: Systems and methods are described for providing updating of disk images supporting serverless code execution and controlled deployment of updated disk images. A disk image can be defined as a set of layers that represent a file system include code of a serverless function and other data used by the code. A function owner can designate one layer as containing software or other data subject to update. When a new version of the layer is obtained at a serverless compute system, the system can generate a new disk image containing the updated layer. The system can then gradually transition the function to the new disk image, by dividing calls to the function among two versions of the function—one using the prior disk image, and one using the new disk image. Performance data gained from the new version of the function can be used to control the gradual transition.

Abstract: Systems and methods are described for reclamation of computing resources in an on-demand code execution system. An on-demand code execution system may execute user-submitted code on virtual machine instances, which may be provisioned with quantities of various computing resources (memory, storage, processor time, etc.). These quantities of computing resources may be unused or underutilized depending on the resource requirements of the user-submitted code, or may become idle once the user-submitted code has completed execution. A resource reclamation system may thus reclaim these underutilized computing resources and reallocate them to other uses. The resource reclamation system may interact with a reclaimable resource identification process that executes within the virtual machine instance, which may identify unused or underused computing resources, claim them, and then allow the resource reclamation system to reallocate them.

Abstract: Systems and methods are provided for efficiently configuring an execution environment for an on-demand code execution system to handle a single request (or session) for a single user. Once the session or request is complete, the execution environment is reset, such as by having the hardware processor state, memory, and storage reset. In particular, prior to the execution of code, state of the execution environment of the host computing device is retrieved, such as hardware processor(s), memory, and/or storage state. Moreover, during execution of the code instructions, intermediate state can be gathered. Following the execution of the code, the execution environment is reset based on the saved state related to the hardware processor(s), memory, and/or storage. A subsequent code execution securely occurs in the execution environment and the execution environment is reset again, and so forth.

Abstract: Systems and methods are described for providing rapid access to data objects stored in a cache. Rather than storing data objects directly, each object can be broken into a number of parts via erasure coding, which enables the object to be generated from less than all parts. When servicing a request for the data object, a device can attempt to retrieve all parts, but begin to generate the data object as soon as a sufficient number of parts is retrieved, even if requests for other parts are outstanding. In this way, the data object can be retrieved without delay due to the slowest requests. For example, where one or more requests timeout, such as due to failure of cache devices, this timeout may have no effect on time required to retrieve the data object from the cache.

Abstract: Systems and methods are described for providing secure storage of data sets while enabling efficient deduplication of data. Each data set can be divided into fixed-length blocks. The plaintext of each block can be convergently encrypted, such as by using a hash of the plaintext as an encryption key, to result in block-level ciphertext that can be stored. If two data sets share blocks, the resulting block-level ciphertext can be expected to overlap, and thus duplicative block-level ciphertexts need not be stored. A manifest can be created to facilitate re-creation of the data set, which manifest identifies the block-level ciphertexts of the data set and a key by which each block-level ciphertext was encrypted. By use of block-level encryption, nearly identical data sets can be largely deduplicated, even if they are not perfectly identical.

Abstract: Systems and methods are provided for scoped credentials within secure execution environments executing within virtual machines instances in an on-demand code execution system. In the on-demand code execution system, the execution environments are reset after every request or session. By resetting the single execution environment after each request or session, security issues are addressed, such as side-channel attacks and persistent malware. Additionally, the use of scoped credentials improves security by limiting the access rights for each code execution request or session to the smallest atomic level for the request or session. Following the request or session, the scoped credential is invalidated.

Abstract: Systems and methods are described for providing rapid access to data sets used by serverless function executions. Rather than pre-loading an entire data set into an environment of a serverless function, which might incur large latencies, the environment is provided with a local access view of the data set, such as in the form of a read-only mount point. As blocks within the data set are requested, a local process can translate the requests into requests for corresponding network objects. The network objects are then retrieved, and the relevant portion of the object is made available to the environment. Network objects may be shared among multiple data sets, so a host device may include a cache enabling an object retrieved for a first environment to also be used to service requests from a second environment.

Abstract: Systems and methods are described for providing storage of encrypted data sets, deduplication of such data sets, and control of the redundancy of those data sets. A form of modified convergent encryption can be employed, whereby an encryption key for a data set is selected based on a combination of the plaintext of the data set and a salt value, with the salt value being selected from a number of permutations corresponding to a desired redundancy of the data set in a storage system. Accordingly, a given data set can result in a number of ciphertexts equal to the desired redundancy, and deduplication can occur by removing duplicative instances of individual ciphertexts. Salt values can be selected according to a variety of criteria, including user-based, time-based, and location-based criteria.

Abstract: Systems and methods are described for providing secure storage of data sets while enabling efficient deduplication of data. Each data set can be divided into fixed-length blocks. The plaintext of each block can be convergently encrypted, such as by using a hash of the plaintext as an encryption key, to result in block-level ciphertext that can be stored. If two data sets share blocks, the resulting block-level ciphertext can be expected to overlap, and thus duplicative block-level ciphertexts need not be stored. A manifest can be created to facilitate re-creation of the data set, which manifest identifies the block-level ciphertexts of the data set and a key by which each block-level ciphertext was encrypted. By use of block-level encryption, nearly identical data sets can be largely deduplicated, even if they are not perfectly identical.

Abstract: Systems and methods are described for providing storage of encrypted data sets, deduplication of such data sets, and control of the redundancy of those data sets. A form of modified convergent encryption can be employed, whereby an encryption key for a data set is selected based on a combination of the plaintext of the data set and a salt value, with the salt value being selected from a number of permutations corresponding to a desired redundancy of the data set in a storage system. Accordingly, a given data set can result in a number of ciphertexts equal to the desired redundancy, and deduplication can occur by removing duplicative instances of individual ciphertexts. Salt values can be selected according to a variety of criteria, including user-based, time-based, and location-based criteria.

Abstract: Systems and methods are provided to manage replicas of a virtualized block storage volume. The master replica of the virtualized block storage volume can heartbeat with each secondary replica of the virtualized block storage volume to provide an indication of the status of the master replica. Each secondary replica can reply to the heartbeat of the master replica. Each replica can be configured to request an updated replica configuration based on not receiving a heartbeat from one of the replicas. The master replica can request an updated replica configuration after a first time period without receiving a reply from one of the secondary replicas and each secondary replica can request an updated replica configuration after a second time period without receiving a communication from the master replica. Use of the heartbeat process between the master replica and the secondary replicas can increase system speed or reduce power consumption.

Abstract: Systems and methods are described for dynamically adjusting quantities of computing resources allocated to virtual machine instances in an on-demand code execution system. An on-demand code execution system may execute user-submitted code on virtual machine instances, which may be provisioned with quantities of various computing resources (memory, storage, processor time, etc.). Users may request that code be executed on virtual machine instances having a particular quantity of a particular computing resource, and a previously provisioned virtual machine instance have a different quantity of the resource than the quantity requested. A resource reclamation system may thus be used to dynamically adjust the quantity of computing resources without reprovisioning the virtual machine instance.

Abstract: Systems and methods are provided for efficiently configuring an execution environment for an on-demand code execution system to handle a single request (or session) for a single user. Once the session or request is complete, the execution environment is reset, such as by having the hardware processor state, memory, and storage reset. In particular, prior to the execution of code, state of the execution environment of the host computing device is retrieved, such as hardware processor(s), memory, and/or storage state. Moreover, during execution of the code instructions, intermediate state can be gathered. Following the execution of the code, the execution environment is reset based on the saved state related to the hardware processor(s), memory, and/or storage. A subsequent code execution securely occurs in the execution environment and the execution environment is reset again, and so forth.

Abstract: Systems and methods are described for simulated data object storage on a data storage system. The system may allow clients to store computed data objects, which are generated from a source data object based on a user-defined transformation. For example, computed data objects may be thumbnail images generated based on a full resolution image. When a request to store a computed data object is received, the system can predict a timing of a next request for the data object. If expected resource consumption associated with storing the data object until a next request exceeds expected resource consumption associated with generating the data object in response to the next request, the system can acknowledge the request to store the data object, but not actually store the data object. Instead, the system may generate the data object in response to the next request.

Abstract: A system may implement maintaining control plane data versions for a network-based service control plane. Various control plane actions may be performed which create new versions of control plane data that may be maintained for the control plane in a database. Some of these actions may be performed by multiple actors creating new versions of the same control plane data. For a particular control plane action, a new version number may be obtained to include in a new version of control plane data, and a conditional write request may be performed to insert the new version of control plane data at the database as part of an optimistic concurrency technique in order to maintain consistency for control plane data.