Abstract: Systems and methods are disclosed which facilitate management of thin provisioned data storage. Specifically, portions of thinly provisioned data stores may be deallocated when they contain invalid data, such as data deleted by a user. A user may transmit notifications, which may include delete notifications, such as TRIM commands, to a provider of the data store (or to the data store itself) that data has been deleted. A management component may modify the data store, or metadata corresponding to the data store, to reflect the deletion. The management component may further monitor portions of the data store to determine whether individual portions contain entirely invalid data. If so, the portion may be deallocated from the thin provisioned data store, resulting in more efficient thin provisioning. Deallocation may be enabled even where deletion notifications from a user do not correspond directly to allocated storage portions.

Abstract: Disclosed are various embodiments for mitigating congestion in networks employing flow-based hashing to assign flows to routes. A flow of packets is sent from a source endpoint to a destination endpoint by way of a network. The flow of packets is associated with flow identification information. It is detected whether congestion is affecting the flow of packets in the network. A perturbation to the flow identification information for the flow of packets is effected in response to determining that congestion is affecting the flow of packets in the network.

Abstract: Disclosed are various embodiments for avoiding flow collisions in networks employing flow-based hashing to assign flows to routes. A flow of packets is sent from a source endpoint to a destination endpoint by way of a network. The flow is associated with flow identification information including, for example, a source port, a source network address, a destination port, and a destination address. It is determined whether to perturb the flow identification information for the flow to avoid flow collisions caused by flow-based hashing performed in the network. A state associated with the flow is updated to effect a perturbation to the flow identification information for the flow if the flow identification information for the flow is to be perturbed.

Abstract: When providing a user with native access to at least a portion of device hardware, the user can be prevented from modifying firmware and other configuration information by controlling the mechanisms used to update that information. In some embodiments, an asymmetric keying approach can be used to encrypt or sign the firmware. In other cases access can be controlled by enabling firmware updates only through a channel or port that is not exposed to the customer, or by mapping only those portions of the hardware that are to be accessible to the user. In other embodiments, the user can be prevented from modifying firmware by only provisioning the user on a machine after an initial mutability period wherein firmware can be modified, such that the user never has access to a device when firmware can be updated. Combinations and variations of the above also can be used.

Abstract: Some embodiments facilitate high performance packet-processing by enabling one or more processors that perform packet-processing to determine whether to enter an idle state or similar state. As network packets usually arrive or are transmitted in batches, the processors of some embodiments determine that more packets may be coming down a multi-stage pipeline upon receiving a first packet for processing. As a result, the processors may stay awake for a duration of time in anticipation of an incoming packet. Some embodiments keep track of the last packet that entered the first stage of the pipeline and compare that with a packet that the processor just processed in a pipeline stage to determine whether there may be more packets coming that need processing. In some embodiments, a processor may also look at a queue length of a queue associated with an upstream stage to determine whether more packets may be coming.

Abstract: Methods and apparatus for supporting cached volumes at storage gateways are disclosed. A storage gateway appliance is configured to cache at least a portion of a storage object of a remote storage service at local storage devices. In response to a client's write request, directed to at least a portion of a data chunk of the storage object, the appliance stores a data modification indicated in the write request at a storage device, and asynchronously uploads the modification to the storage service. In response to a client's read request, directed to a different portion of the data chunk, the appliance downloads the requested data from the storage service to the storage device, and provides the requested data to the client.

Abstract: A set of techniques for customizing resources for the workload of an application is described. The application can be received and hosted on a set of hardware and software resources of a service provider, which may include server instances, databases, load balancers and the like. Once the application is deployed and processing workload, it can be monitored to gather runtime trace information associated with the workload. This runtime trace information can be analyzed for potential optimizations and improvements to the set of resources or their configuration. Additionally, the user is enabled to input hardware/software specifications for the resources used to run the application. The service provider can use these specifications, along with the runtime trace information to optimize the resource stack used to execute the user's application.

Abstract: A method and apparatus for staged execution pipelining and allocating resource to staged execution pipelines are provided. One or more execution pipelines are established, where each of the one or more execution pipelines includes one or more execution stages. Data is provided to the one or more execution pipelines for processing and resources are allocated to the execution pipeline.

Abstract: Techniques for implementing a single-addressable virtual topology element (VTE) in a virtual topology. A VTE in a virtual topology may be distributed as multiple instantiated elements in a physical topology. However, the multiple instantiated elements are addressable as a single entity. Obtaining information associated with the VTE includes obtaining and aggregating information from each of the instantiated elements. Applying an overall configuration to the VTE includes determining a respective configuration for each instantiated element based on the overall configuration, and applying the respective configuration to each instantiated element.

Abstract: Techniques for implementing a virtual tap in a virtual topology are described. A request to tap a particular link between a set of virtual topology entities (VTEs) is received. The particular link between the set of VTEs is mapped to a link between a set of instantiated entities in a physical topology. One of the instantiated elements implements a tapping function. Additionally or alternatively, a tapping element is instantiated and the tapping element implements the tapping function. The tapping element is not one of the set of instantiated entities. The tapping function includes: applying a tap filter to a set of data; replicating the data that passes the tap filter; and transmitting the replicated data to a virtual endpoint. The replicated data may be referred to as “tapped data.” A user interface engine presents information regarding the particular link between the set of VTEs based on the tapped data.

Abstract: Systems and method for the management of migrations of virtual machine instances are provided. A migration manager monitors the resource usage of a virtual machine instance over time in order to create a migration profile. When migration of a virtual machine instance is desired, the migration manager schedules the migration to occur such that the migration conforms to the migration profile.

Abstract: A virtual tape library system is used to back up data from a client archive system expecting physical tape operations onto logical data containers and/or a metadata store of a storage service by emulating the physical tape operations. For example, a virtual tape library appliance is installed at a customer premise location to interface with a client archive system. The virtual tape library appliance provides virtual interfaces to appear as physical tape library subsystems, such as tape drives and media changing interfaces. However, these virtual interfaces are supported by logical data containers in a storage service and a metadata store. The virtual tape library system allows the client archive system to make requests to import new virtual tapes, export virtual tapes for archiving, store virtual tapes at a virtual location, load and eject virtual tapes into a virtual tape drive and operate on virtual tapes in a virtual tape drive.

Abstract: High-speed processing of packets to, and from, a virtualization environment can be provided while utilizing hardware-based segmentation offload and other such functionality. A hardware vendor such as a network interface card (NIC) manufacturer can enable the hardware to support open and proprietary stateless tunneling in conjunction with a protocol such as single root I/O virtualization (SR-IOV) in order to implement a virtualized overlay network. The hardware can utilize various rules, for example, that can be used by the NIC to perform certain actions, such as to encapsulate egress packets and decapsulate packets.

Abstract: Methods and apparatus for supporting cached volumes at storage gateways are disclosed. A storage gateway appliance is configured to cache at least a portion of a storage object of a remote storage service at local storage devices. In response to a client's write request, directed to at least a portion of a data chunk of the storage object, the appliance stores a data modification indicated in the write request at a storage device, and asynchronously uploads the modification to the storage service. In response to a client's read request, directed to a different portion of the data chunk, the appliance downloads the requested data from the storage service to the storage device, and provides the requested data to the client.

Abstract: Technologies are described herein for migrating network addresses between host computers in a hybrid network environment. Before migrating a network address between a source host computer in a first network and a target host computer in a second network, potentially affected host computers in the networks are notified of the upcoming migration. This may cause an address translation module of an affected host computer to enter a fast polling mode. The networking infrastructures of the first and second networks are then reconfigured to effect transfer of the network address. This may include updating overlay IP mappings for the network address being migrated to point to the target host computer. Finally, the affected host computers are notified that migration of the network address is complete, causing the address translation module of the affected host computer to stop fast polling and return to normal operation with update mappings.

Abstract: A method and apparatus for staged execution pipelining and allocating resource to staged execution pipelines are provided. One or more execution pipelines are established, where each of the one or more execution pipelines includes one or more execution stages. Data is provided to the one or more execution pipelines for processing and resources are allocated to the execution pipeline.

Abstract: Processes and systems are disclosed for selecting a producer system from a number of producer systems to lease to a consumer system. A leasing agent, in response to a request from the consumer system for access to a service at a producer system, can identify a producer system to lease to the lease requestor based, at least in part, on a selection weight associated with each producer system that the leasing agent is assigned. The selection weights can be modified based on status information associated with each of the producer systems. This status information may be obtain from the producer systems and/or from a consumer system that has previously accessed the producer system. The consumer system may provide the status information to the leasing agent as part of the consumer system's lease request.

Abstract: High-speed processing of packets to, and from, a virtualization environment can be provided while utilizing hardware-based segmentation offload and other such functionality. A hardware vendor such as a network interface card (NIC) manufacturer can enable the hardware to support open and proprietary stateless tunneling in conjunction with a protocol such as single root I/O virtualization (SR-IOV) in order to implement a virtualized overlay network. The hardware can utilize various rules, for example, that can be used by the NIC to perform certain actions, such as to encapsulate egress packets and decapsulate packets.

Abstract: Systems and methods are disclosed which facilitate management of thin provisioned data storage. Specifically, portions of thinly provisioned data stores may be deallocated when they contain invalid data, such as data deleted by a user. A user may transmit notifications, which may include delete notifications, such as TRIM commands, to a provider of the data store (or to the data store itself) that data has been deleted. A management component may modify the data store, or metadata corresponding to the data store, to reflect the deletion. The management component may further monitor portions of the data store to determine whether individual portions contain entirely invalid data. If so, the portion may be deallocated from the thin provisioned data store, resulting in more efficient thin provisioning. Deallocation may be enabled even where deletion notifications from a user do not correspond directly to allocated storage portions.

Abstract: Processes and systems are disclosed for selecting a producer system from a number of producer systems to lease to a consumer system. A leasing agent, in response to a request from the consumer system for access to a service at a producer system, can identify a producer system to lease to the lease requestor based, at least in part, on a selection weight associated with each producer system that the leasing agent is assigned. The selection weights can be modified based on status information associated with each of the producer systems. This status information may be obtain from the producer systems and/or from a consumer system that has previously accessed the producer system. The consumer system may provide the status information to the leasing agent as part of the consumer system's lease request.