Abstract: In this disclosure, techniques are described for more efficiently sharing resources across multiple virtual machine instances. For example, techniques are disclosed for allowing additional virtual machine instances to be supported by a single computing system by more efficiently allocating memory to virtual machine instances by providing page swapping in a virtualized environment and/or predictive page swapping. In one embodiment, a virtual memory manager swaps pages predicatively in and/or out of a paging pool based on information from a central processing unit (“CPU”) scheduler. In one embodiment, the CPU scheduler provides scheduling information for virtual machine instances to the virtual memory manager, where the scheduling information allows the virtual memory manager to determine when a virtual machine is scheduled to become active or inactive. The virtual memory manager can then swap-in or swap-out memory pages.

Abstract: Systems and methods are disclosed which facilitate management of thin provisioned data storage. Specifically, portions of thinly provisioned data stores may be deallocated when they contain invalid data, such as data deleted by a user. A user may transmit notifications, which may include write requests corresponding to a defined bit pattern, to a provider of the data store (or to the data store itself) that data has been deleted. A management component may modify the data store, or metadata corresponding to the data store, to reflect the deletion. The management component may further monitor portions of the data store to determine whether individual portions contain entirely invalid data. If so, the portion may be deallocated from the thin provisioned data store, resulting in more efficient thin provisioning. Deallocation may be enabled even where deletion notifications from a user do not correspond directly to allocated storage portions.

Abstract: A connection request from a client is received at an access subsystem node of a distributed storage service. The connection request includes an attempt count parameter indicative of a number of times an establishment of a connection on behalf of the client has been attempted. A workload threshold level based on the attempt count parameter is identified for use in an acceptance decision for the connection request. In response to a determination that a local workload metric of the access subsystem node is below the workload threshold level, the connection is accepted.

Abstract: High-speed processing of packets to and from a virtualization environment can be provided while utilizing segmentation offload and other such functionality of hardware such as a network interface card. Virtualization information can be added to extension portions of protocol headers, for example, such that the payload portion is unchanged. The virtualization information can be hashed and added to the payload or stream at, or relative to, various segmentation boundaries, such that the virtualization or additional header information will only be added to a subset of the segmented data frames, thereby reducing the necessary overhead. Further, the hashing of the information can allow for reconstruction of the virtualization information upon desegmentation even in the event of packet loss.

Abstract: Methods and apparatus for supporting cached volumes at storage gateways are disclosed. A storage gateway appliance is configured to cache at least a portion of a storage object of a remote storage service at local storage devices. In response to a client's write request, directed to at least a portion of a data chunk of the storage object, the appliance stores a data modification indicated in the write request at a storage device, and asynchronously uploads the modification to the storage service. In response to a client's read request, directed to a different portion of the data chunk, the appliance downloads the requested data from the storage service to the storage device, and provides the requested data to the client.

Abstract: A node of a distributed storage service receives a read request from a client, directed to a particular physical page storing at least a portion of a file store object. The node transmits, to the client, a particular operation sequence number (OSN), obtained from a replicated state machine, indicative of an order in which the read request was processed at the node relative to other requests directed to the file store object. The node receives a write request from the client directed to the page. The write request includes the particular OSN. The node determines whether the page has been modified after the particular OSN was obtained. If the page has not been modified since the OSN was obtained, a modification indicated in the write request is performed; otherwise, the write request is rejected.

Abstract: Methods and apparatus for WAN-optimized cache population at a storage gateway appliance are disclosed. A read request is received at the appliance from a client. The request is directed to a particular data block of a data chunk of a storage object of a remote storage service. A method comprises, in response to the read request, initiating a first data transfer of the data chunk from the remote storage service to an intermediate device. The method includes determining, based on one or more criteria, contents of one or more other data transfers including a second data transfer from the intermediate device to the appliance, initiating the second data transfer to transmit the requested data block to the appliance, and providing the particular data block to the client from the appliance.

Abstract: Systems and method for the management of migrations of virtual machine instances are provided. A migration manager monitors the resource usage of a virtual machine instance over time in order to create a migration profile. When migration of a virtual machine instance is desired, the migration manager schedules the migration to occur such that the migration conforms to the migration profile.

Abstract: A node of a storage service is selected as a coordinator of a distributed transaction involving multiple page-level modifications. The coordinator identifies other nodes as members of a node chain collectively storing physical data pages at which proposed modifications are to be performed, including a decider node responsible for a decision to commit the transaction. The coordinator generates a transaction preparation message comprising a representation of an order of respective commit decisions associated with the proposed modifications, and transmits the message to a selected node of the chain for a sequential propagation along the chain. Each chain node performs a local commit analysis for its changes and stores a record of its intent to commit. If a decision to commit is reached at the decider, the proposed modifications are completed.

Abstract: Systems and methods for managing imaging of computing devices are provided. A device image manager enables selection of one or more device images on the basis of various computing device criteria. The device image manager may obtain hardware qualification information associated with one or more device images, and obtain device configuration information associated with a computing device. The device image manager may select a set of appropriate device images for the computing device based on the hardware qualification information, device configuration information, or various device image selection criteria.

Abstract: A directed acyclic graph (DAG) is generated to represent a namespace of a directory. In response to a request to create a new object with a specified name, a hash value bit sequence is computed for the name. A plurality of levels of the DAG are navigated using successive subsequences of the bit sequence to identify a candidate node for storing a new entry corresponding to the specified name. If the candidate node meets a split criterion, the new entry and at least a selected subset of entries of the candidate node's list of entries are distributed among a plurality of DAG nodes, including at least one new DAG node, using respective bit sequences obtained by applying the hash function for each distributed entry.

Abstract: A virtual tape system utilizes multiple virtual tape libraries. Some virtual elements of the virtual tape libraries are connected elements, such as virtual import/export slots, that logically connect two or more virtual tape libraries. Virtual media changers of the virtual tape libraries can be commanded, as if physical media changers, to virtually move virtual media, such as virtual tapes, within and among the virtual tape libraries. By moving a virtual medium to a connected element, the virtual medium can be virtually moved from one virtual tape library to another.

Abstract: In response to a rename request to change a file name at a storage service from a first name to a second name, a workflow comprising at least two atomic operations is initiated. In the first atomic operation, a lock is obtained on a first directory entry for the first name, and an intent record for the rename workflow is stored. In a second atomic operation, a pointer of a second directory entry for the second name is modified, and an indication of the pointer modification is stored. In a third set of operations, the intent record is deleted, the lock is released, and the first directory entry is deleted.

Abstract: A method and apparatus for staged execution pipelining and allocating resource to staged execution pipelines are provided. One or more execution pipelines are established, where each of the one or more execution pipelines includes one or more execution stages. Data is provided to the one or more execution pipelines for processing and resources are allocated to the execution pipeline.

Abstract: Virtual machines may migrate between heterogeneous sets of implementation resources in a manner that allows the virtual machines to efficiently and effectively adapt to new implementation resources. Furthermore, virtual machines may change types during migration without terminating the virtual machines. Migration templates may be established to manage migration of sets of virtual machines between sets of implementation resources and/or virtual machine types. Migration templates may be established based at least in part on information provided by migration agents added to the virtual machines under consideration for migration. The migration agents may detect and augment relevant virtual machine capabilities, as well as trigger reconfiguration of virtual machine components in accordance with migration templates.

Abstract: Some embodiments facilitate high performance packet-processing by enabling one or more processors that perform packet-processing to determine whether to enter an idle state or similar state. As network packets usually arrive or are transmitted in batches, the processors of some embodiments determine that more packets may be coming down a multi-stage pipeline upon receiving a first packet for processing. As a result, the processors may stay awake for a duration of time in anticipation of an incoming packet. Some embodiments keep track of the last packet that entered the first stage of the pipeline and compare that with a packet that the processor just processed in a pipeline stage to determine whether there may be more packets coming that need processing. In some embodiments, a processor may also look at a queue length of a queue associated with an upstream stage to determine whether more packets may be coming.

Abstract: A framework can be utilized with conventional networking components to enable those components to process packets of specific formats using conventional algorithms, such as algorithms for receive side coalescing (RCS) and TCP segmentation offloading (TSO). Format and flow information can be added to an opaque field or other portion of a packet, at an appropriate location or pre-configured offset. Placing information at a specific location or offset enables the networking hardware to quickly recognize a packet for processing. Packets can be segmented and coalesced using conventional algorithms on the networking hardware, enabling packets of various formats to be able to take advantage of various performance enhancements.

Abstract: Methods and apparatus for supporting cached volumes at storage gateways are disclosed. A storage gateway appliance is configured to cache at least a portion of a storage object of a remote storage service at local storage devices. In response to a client's write request, directed to at least a portion of a data chunk of the storage object, the appliance stores a data modification indicated in the write request at a storage device, and asynchronously uploads the modification to the storage service. In response to a client's read request, directed to a different portion of the data chunk, the appliance downloads the requested data from the storage service to the storage device, and provides the requested data to the client.

Abstract: A method and apparatus for staged execution pipelining and allocating resource to staged execution pipelines are provided. One or more execution pipelines are established, where each of the one or more execution pipelines includes one or more execution stages. Data is provided to the one or more execution pipelines for processing and resources are allocated to the execution pipeline.

Abstract: Attempts to update confirmation information or firmware for a hardware device can be monitored using a secure counter that is configured to monotonically adjust a current value of the secure counter for each update or update attempt. The value of the counter can be determined every time the validity of the firmware is confirmed, and this value can be stored to a secure location. At subsequent times, such as during a boot process, the actual value of the counter can be determined and compared with the expected value. If the values do not match, such that the firmware may be in an unexpected state, an action can be taken, such as to prevent access to, or isolate, the hardware until such time as the firmware can be validated or updated to an expected state.