Abstract: Entries in individual portions of a network map serve to establish inter-cluster routes that are formed from routing data structures at individual clusters, which routing data structures store connection parameters pertaining to access point nodes in a multi-cluster computing environment. Upon receiving a resource management request at one of the access points, the request parameters are used in combination with the network map to identify at least one target access point having a virtual machine for responding to the resource management request. The resource management request is routed to a next hop access point node on route(s) to target access points. If the next hop access point node is not the node to respond to the resource management request, then processing at the next hop node serves to route the resource management request to a further next hop access point node on a route to the target access point.

Abstract: Systems for low-latency data access in distributed computing systems. A method embodiment commences upon generating a first storage area in local storage of a first computing node. Access to the first storage area is provided through the first computing node. A second storage area is generated wherein the second storage area comprises a first set of metadata that comprises local storage device locations of at least some of the local storage areas of the first storage area. A set of physical access locations of the second storage area is stored to a database that manages updates to the second set of metadata pertaining to the second storage area. Accesses to the first storage area are accomplished by querying the database retrieve a location of the second set of metadata, and then accessing the first storage area through one or more additional levels of metadata that are node-wise collocated.

Abstract: A data storage site receives data from different data producer sites. Each of the data producer sites has a particular relationship to the data storage site, and each particular relationship carries corresponding data storage policies, constraints and commitments. When a data storage site receives a data storage request from a data producer, and that particular data is already present from a prior storage operation at the data storage site, the characteristics of the policies, constraints and commitments that were applied when that data was saved by the prior storage operation are reconciled with the policies, constraints and commitments of the requesting data producer. Deduplication logic reconciles different sets of policies, constraints and commitments such that the data can be effectively deduplicated by saving data-producer-specific metadata. Alternatively, the data can be effectively deduplicated by promoting the storage of the data so it covers a broader set of policies, constraints and commitments.

Abstract: Examples described herein may include management of a ROBO site. An example method includes detecting a first configuration of a first computing node cluster of a computing system over a first network, and detecting a second configuration of a second computing node cluster of the computing system over a second network. The example method further includes receiving a request to update a configuration of the computing system. The update includes an update of the first configuration of the first computing node cluster. The example method further includes determining whether the update of the first configuration of the first computing node cluster is compatible with the second configuration of the second computing node cluster, and in response to the update of the first configuration of the first computing node cluster being incompatible with the second configuration of the second computing node cluster, denying the request.

Abstract: A method and apparatus for data driven and cluster specific version/update control. The apparatus includes an automated multi-clusters management apparatus that interfaces with a plurality of remote clusters to provide data driven version/update control on a cluster by cluster basis. Generally, operation includes collection/identification of cluster specific data pertaining to software, hardware, and cluster requirements. The cluster specific data is later compared/analyzed against multi-cluster data pertaining to software releases, hardware characteristics, and known bugs/issues for each. The results of the comparison/analysis can then be ranked according to various metrics to different possible solutions and to differentiate the less desirable results from the more desirable results. Thus, the automated multi-cluster management apparatus provides for selection of versions/updates that is dependent on the cluster specific data.

Abstract: A virtual computing system deduplicates data. The system includes one or more processors, and one or more computer-readable storage media communicably coupled to the one or more processors. The one or more computer-readable storage media has instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to: scan extents for duplicate data chunks; calculate hash values for the extents at different chunk lengths; emit the hash values at the different chunk lengths; identify matching ones of the hash values; and select one or more candidates from among the matching ones of the hash values for dedup.

Abstract: Systems and methods for migrating a plurality of virtual machines (VMs) from a private cluster to a public cloud include identifying the plurality of VMs currently residing in the private cluster to be migrated to the public cloud. A communication graph indicative of communications involving the plurality of VMs is determined. A migration sequence for the plurality of VMs based on the communication graph is generated. The plurality of VMs is migrated from the private cluster to the public cloud according to the migration sequence.

Abstract: Systems and methods for scheduling storage management tasks over predicted user tasks in a distributed storage system. A method commences upon receiving a set of historical stimulus records that characterize management tasks that are run in the storage system. A corresponding set of historical response records comprising system metrics associated with execution of the system tasks is also received. A learning model is formed from the stimulus records and the response records and formatted to be used as a predictor. A set of forecasted user tasks is input as new stimulus records to the predictor to determine a set of forecasted system metrics that would result from running the forecasted user tasks. Management tasks are selected so as not to impact the forecasted user tasks. Management tasks can be selected based on non-contentions resource usage between historical management task resource usage and predictions of resource usage by the user tasks.

Abstract: Systems and methods for scheduling storage management tasks over predicted user tasks in a distributed storage system. A method commences upon receiving a set of historical stimulus records that characterize management tasks that are run in the storage system. A corresponding set of historical response records comprising system metrics associated with execution of the system tasks is also received. A learning model is formed from the stimulus records and the response records and formatted to be used as a predictor. A set of forecasted user tasks is input as new stimulus records to the predictor to determine a set of forecasted system metrics that would result from running the forecasted user tasks. Management tasks are selected so as not to impact the forecasted user tasks. Management tasks can be selected based on non-contentions resource usage between historical management task resource usage and predictions of resource usage by the user tasks.

Abstract: Embodiments serve to balance overall performance of a finite-sized caching system having a first cache of a first cache size and a second cache of a second cache size. A tail portion and a head portion of each of the caches are defined wherein incoming data elements are initially stored in a respective head portion and wherein evicted data elements are evicted from a respective tail portion. Performance metrics are defined wherein a performance metric includes a predicted miss cost that would be incurred when replacing an evicted data elements. A quantitative function is defined to include cache performance metrics and a cache reallocation amount. The cache performance metrics are evaluated periodically to determine a then-current cache reallocation amount. The caches can be balanced by increasing the first cache size by the cache reallocation amount and decreasing the second cache size by the cache reallocation amount.

Abstract: An architecture for implementing a distributed metadata system in a virtualization environment comprising a plurality of servers, wherein each server may access and operate upon the entirety of the metadata in the virtualization environment. In response to a request received at a server to operate upon the metadata, existing metadata associated with the request is read by the server. The server performs one or more updates based upon the retrieved metadata, and attempts to update the corresponding stored metadata. In some embodiments, attempting to update the stored metadata comprises reading the metadata again and comparing it to the originally retrieved metadata, and performing the update only if the newly retrieved metadata matches the originally retrieved metadata.

Abstract: Embodiments serve to balance overall performance of a finite-sized caching system having a first cache of a first cache size and a second cache of a second cache size. A tail portion and a head portion of each of the caches are defined wherein incoming data elements are initially stored in a respective head portion and wherein evicted data elements are evicted from a respective tail portion. Performance metrics are defined wherein a performance metric includes a predicted miss cost that would be incurred when replacing an evicted data elements. A quantitative function is defined to include cache performance metrics and a cache reallocation amount. The cache performance metrics are evaluated periodically to determine a then-current cache reallocation amount. The caches can be balanced by increasing the first cache size by the cache reallocation amount and decreasing the second cache size by the cache reallocation amount.

Abstract: Embodiments relate to supporting transaction data committed to a stable storage. Committed data in the cluster is stored in the persistent cache layer and replicated and stored in the cache layer of one or more secondary nodes. One copy is designated as a master copy and all other copies are designated as replica, with an exclusive write lock assigned to the master and a shared write lock extended to the replica. An acknowledgement of receiving the data is communicated following confirmation that the data has been replicated to each node designated to receive the replica. Managers and a director are provided to support management of the master copy and the replicas within the file system, including invalidation of replicas, fault tolerance associated with failure of a node holding a master copy, recovery from a failed node, recovered of the file system from a power failure, and transferring master and replica copies within the file system.

Abstract: Systems for memory management in virtual machines. A method embodiment forms a reconfigurable multi-level data structure by requesting reserved memory as well as loaned-out memory. The method configures the multi-level data structure to have a first area comprising reserved memory and having a second area comprising reconfigurable memory. The reserved memory portion of the data structure is populated with pointers that address one or more portions of the second area. During operation, the method receives a request to release some or all of the reconfigurable memory. Memory address pointers in the first portion of the multi-level data structure are modified to reflect the release of the reconfigurable memory. In a hypervisor-based virtual machine environment, the first area comprising reserved memory can be allocated via a hypervisor call. The second area comprising reconfigurable memory is allocated via a balloon memory driver call.

Abstract: An architecture for implementing a distributed metadata system in a virtualization environment comprising a plurality of servers, wherein each server may access and operate upon the entirety of the metadata in the virtualization environment. In response to a request received at a server to operate upon the metadata, existing metadata associated with the request is read by the server. The server performs one or more updates based upon the retrieved metadata, and attempts to update the corresponding stored metadata. In some embodiments, attempting to update the stored metadata comprises reading the metadata again and comparing it to the originally retrieved metadata, and performing the update only if the newly retrieved metadata matches the originally retrieved metadata.

Abstract: A method for managing storage allocation includes adaptively determining, by a storage device processor, a region width across disk spaces for a group of storage devices that is inversely proportional to a number of nodes sharing a particular storage device. An adaptive storage device allocation region of the particular storage device is created based on the determined region width across the disk spaces for the group of storage devices.

Abstract: Techniques are disclosed for peer-to-peer data transfers where a source device receives a request to read data words from a target device. The source device creates a first and second read command for reading a first portion and a second portion of a plurality of data words from the target device, respectively. The source device transmits the first read command to the target device, and, before a first read operation associated with the first read command is complete, transmits the second read command to the target device. The first and second portions of the plurality of data words are stored in a first and second portion a buffer memory, respectively. Advantageously, an arbitrary number of multiple read operations may be in progress at a given time without using multiple peer-to-peer memory buffers. Performance for large data block transfers is improved without consuming peer-to-peer memory buffers needed by other peer GPUs.

Abstract: Systems for high-performance computing. A storage control architecture is implemented by a plurality of nodes, where a node comprises combinations of executable containers that execute in cooperation with virtual machines running above a hypervisor. The containers run in a virtual machine above a hypervisor, and/or can be integrated directly into the operating system of a host node. Sensitive information such as credit card information may be isolated from the containers in a separate virtual machine that is configured to be threat resistant, and which can be accessed through a threat resistant interface module. One of the virtual machines of the node may be a node-specific control virtual machine that is configured to operate as a dedicated storage controller for a node. One of the virtual machines of the node may be a node-specific container service machine that is configured to provide storage-related and other support to a hosted executable container.

Abstract: Embodiments of the invention relate to supporting transaction data committed to a stable storage. Committed data in the cluster is stored in the persistent cache layer and replicated and stored in the cache layer of one or more secondary nodes. One copy is designated as a master copy and all other copies are designated as replica, with an exclusive write lock assigned to the master and a shared write lock extended to the replica. An acknowledgement of receiving the data is communicated following confirmation that the data has been replicated to each node designated to receive the replica. Managers and a director are provided to support management of the master copy and the replicas within the file system, including invalidation of replicas, fault tolerance associated with failure of a node holding a master copy, recovery from a failed node, recovered of the file system from a power failure, and transferring master and replica copies within the file system.

Abstract: A method for managing storage allocation includes adaptively determining, by a storage device processor, a region width across disk spaces for a group of storage devices that is inversely proportional to a number of nodes sharing a particular storage device. An adaptive storage device allocation region of the particular storage device is created based on the determined region width across the disk spaces for the group of storage devices.