Abstract: An illustrative embodiment disclosed herein is an apparatus including a processor having programmed instructions to detect an operation associated with a first object instance in a first cluster, determine that the first object instance is configured for replication to a second cluster, create an object information (info) metadata entry for the first object instance, create an object replication metadata entry for the first object instance, and simultaneously write the object info metadata entry and the object replication metadata entry to a first data structure and a second data structure, respectively.

Abstract: Methods, systems and computer program products for computer system data management. A leader node that is in a leader-follower relationship with a follower node is deployed to manage metadata across a plurality of interconnected computing nodes of the computer system. Upon a failure of any sort that results in a loss of operation or a disconnection of the leader node, the followers undertake steps to bring up a new leader. Such steps to bring up a new leader include electing a new leader to replace the lost or disconnected leader and initializing the data state of the new leader node by causing the leader to request incremental snapshots from followers. Additional metadata that might have changed since closing the incremental snapshots can also be retrieved from the followers. The metadata that might have changed since the time of closing the incremental snapshot is subjected to verification using a consensus algorithm.

Abstract: An illustrative embodiment disclosed herein is an apparatus including a processor having programmed instructions to maintain an object store including a primary cluster having one or more compute resources and one or more first storage resources, identify a secondary cluster having one or more second storage resources, select the secondary cluster to be added to the object store, allocate an available portion of the one or more second storage resources to the object store, and shard an object across the one or more second storage resources and the available portion of the one or more second storage resources.

Abstract: An illustrative embodiment disclosed herein is an apparatus including a processor with programmed instructions to receive a request to process storage data using a compute application, determine that a first location of the storage data is in a first storage resource in a first cluster of compute, storage, and network resources, determine that a second location of the compute application is in a first compute resource of a second cluster of compute, storage, and network resources, determine whether the first cluster includes a second compute resource that is compatible with the compute application, and respectively, either, based on a first compatibility determination, migrate the compute application from the first compute resource to the second compute resource, or, based on a second compatibility determination, replicate the storage data from the first storage resource to a second storage resource in the second cluster.

Abstract: An illustrated embodiment disclosed herein is an apparatus including a processor having programmed instructions to specify a first object as an executable, specify a second object as an environment on which to run the first object, specify a third object as data to be processed by the first object, and construct a composite object that associates the first, second, and third objects.

Abstract: Systems for low-latency data access in distributed computing systems. A method embodiment commences upon generating a first storage area in local storage of a first computing node. Access to the first storage area is provided through the first computing node. A second storage area is generated wherein the second storage area comprises a first set of metadata that comprises local storage device locations of at least some of the local storage areas of the first storage area. A set of physical access locations of the second storage area is stored to a database that manages updates to the second set of metadata pertaining to the second storage area. Accesses to the first storage area are accomplished by querying the database to retrieve a location of the second set of metadata, and then accessing the first storage area through one or more additional levels of metadata that are node-wise collocated.

Abstract: An approach for reduced size extent identifiers for a file system may be implemented by generating a full-size extent or file identifier and generating a smaller identifier from a portion of the full-size identifier. A check may be performed as to whether the smaller identifier is unique within a file system and if it is unique, the smaller identifier may be used in place of the full-size identifier. If not unique, the size of the smaller identifier may be increased. In some embodiments, the size of the smaller identifier is increased until a unique identifier if found.

Abstract: In some embodiments, a method receives a first acknowledgement message that acknowledges receipt of a first packet and determines whether the first acknowledgement message is a duplicate of a previous acknowledgement message that was sent previous to the first acknowledgement message. When the first acknowledgement message is the duplicate of the previous acknowledgement message, the method detects when a second acknowledge message is received that acknowledges receipt of a second packet. Then, the method determines that the second acknowledgement message is not the duplicate of the previous acknowledgement message and measures a metric based on the detecting of the first acknowledgement message and the second acknowledgement message.

Abstract: Systems for high-performance distributed computing. The systems include techniques for managing data and metadata across multiple nodes. A method embodiment commences by storing data at a node using a first storage mechanism that is local to the node. A first set of metadata is configured to identify a storage location for the stored data. The first set of metadata is stored using the same first storage mechanism that is local to the node. For accessing the first set of metadata, a second set of metadata is configured to identify a storage location for the first set of metadata. The second set of metadata is stored using a second storage mechanism that comprises a distributed metadata storage facility that stores metadata across multiple storage locations having at least one of the multiple storage locations that is not local to the node that stores data and metadata using the first storage mechanism.

Abstract: A data storage site receives data from different data producer sites. Each of the data producer sites has a particular relationship to the data storage site, and each particular relationship carries corresponding data storage policies, constraints and commitments. When a data storage site receives a data storage request from a data producer, and that particular data is already present from a prior storage operation at the data storage site, the characteristics of the policies, constraints and commitments that were applied when that data was saved by the prior storage operation are reconciled with the policies, constraints and commitments of the requesting data producer. Deduplication logic reconciles different sets of policies, constraints and commitments such that the data can be effectively deduplicated by saving data-producer-specific metadata. Alternatively, the data can be effectively deduplicated by promoting the storage of the data so it covers a broader set of policies, constraints and commitments.

Abstract: An illustrative embodiment disclosed herein is an apparatus including a processor having programmed instructions to request, from each of a plurality of services spanning multiple clusters, a plurality of time instances. A plurality of snapshots were taken during the plurality of time instances. The processor has programmed instructions to present, to a user, the plurality of time instances and a plurality of identifiers. Each of the plurality of identifiers corresponds to one of the plurality of services. The processor has programmed instructions to receive, from the user, a first selection of a first time instance of the plurality of time instances and a second selection of a subset of the plurality of identifiers. Each of the subset of identifiers corresponds to one of a subset of the plurality of services. The processor has programmed instructions to apply, to a target, a subset of the plurality of snapshots.

Abstract: A system and method for transmitting data from a local management system of a remote office branch office site in a virtual computing system to a central management system includes assigning, by the local management system, an initial priority to each piece of data, identifying data requested by the central management system since a previous data transmission, and updating the initial priority of each piece of data to an updated priority. The data requested by the central management system is prioritized higher than other data. The system and method also include combining, by the local management system, at least a portion of the data starting with data having a highest priority, into a data packet until the data packet achieves a computed data packet size, transmitting the data packet to the central management system, and updating the initial priority with the updated priority for future data transmissions.

Abstract: Examples of a distributed message service include a virtualized file system including a virtual disk configured to store messages for a message topic, and a broker logically allocated to the message topic. The broker is configured to cause a message directed to the message topic provided from a publisher to be stored at the virtual disk, and to route the message to a subscriber of the message topic that is registered with the broker. The distributed message service further includes an operating system configured to manage the virtualized file system and includes a message service configured to manage logical allocation of the broker and manage allocation of the virtual disk to the message topic.

Abstract: Systems for low-latency data access in distributed computing systems. A method embodiment commences upon generating a first storage area in local storage of a first computing node. Access to the first storage area is provided through the first computing node. A second storage area is generated wherein the second storage area comprises a first set of metadata that comprises local storage device locations of at least some of the local storage areas of the first storage area. A set of physical access locations of the second storage area is stored to a database that manages updates to the second set of metadata pertaining to the second storage area. Accesses to the first storage area are accomplished by querying the database retrieve a location of the second set of metadata, and then accessing the first storage area through one or more additional levels of metadata that are node-wise collocated.

Abstract: Systems for high-performance distributed computing. The systems include techniques for managing data and metadata across multiple nodes. A method embodiment commences by storing data at a node using a first storage mechanism that is local to the node. A first set of metadata is configured to identify a storage location for the stored data. The first set of metadata is stored using the same first storage mechanism that is local to the node. For accessing the first set of metadata, a second set of metadata is configured to identify a storage location for the first set of metadata. The second set of metadata is stored using a second storage mechanism that comprises a distributed metadata storage facility that stores metadata across multiple storage locations having at least one of the multiple storage locations that is not local to the node that stores data and metadata using the first storage mechanism.

Abstract: Systems for high-performance computing. A storage control architecture is implemented by a plurality of nodes, where a node comprises combinations of executable containers that execute in cooperation with virtual machines running above a hypervisor. The containers run in a virtual machine above a hypervisor, and/or can be integrated directly into the operating system of a host node. Sensitive information such as credit card information may be isolated from the containers in a separate virtual machine that is configured to be threat resistant, and which can be accessed through a threat resistant interface module. One of the virtual machines of the node may be a node-specific control virtual machine that is configured to operate as a dedicated storage controller for a node. One of the virtual machines of the node may be a node-specific container service machine that is configured to provide storage-related and other support to a hosted executable container.

Abstract: A method and apparatus for data driven and cluster specific version/update control. The apparatus includes an automated multi-clusters management apparatus that interfaces with a plurality of remote clusters to provide data driven version/update control on a cluster by cluster basis. Generally, operation includes collection/identification of cluster specific data pertaining to software, hardware, and cluster requirements. The cluster specific data is later compared/analyzed against multi-cluster data pertaining to software releases, hardware characteristics, and known bugs/issues for each. The results of the comparison/analysis can then be ranked according to various metrics to different possible solutions and to differentiate the less desirable results from the more desirable results. Thus, the automated multi-cluster management apparatus provides for selection of versions/updates that is dependent on the cluster specific data.

Abstract: Systems and methods for migrating a plurality of virtual machines (VMs) from a private cluster to a public cloud include identifying the plurality of VMs currently residing in the private cluster to be migrated to the public cloud. A communication graph indicative of communications involving the plurality of VMs is determined. A migration sequence for the plurality of VMs based on the communication graph is generated. The plurality of VMs is migrated from the private cluster to the public cloud according to the migration sequence.

Abstract: An illustrative embodiment disclosed herein is an apparatus including a processor having programmed instructions to update a versioned object by appending a latest version of the versioned object to a data structure, read a predetermined version identifier corresponding to the latest version from a pointer entry, and look up the latest version of the versioned object, in the data structure, using the predetermined version identifier corresponding to the latest version of the versioned object.

Abstract: An illustrative embodiment disclosed herein is an apparatus including a processor having programmed instructions to map a plurality of objects to a region and map the region to a plurality of vdisks. The plurality of objects are stored in the plurality of vdisks. The processor has programmed instructions to, responsive to mapping the plurality of objects to the region, identify a policy associated with the region. The policy specifies a policy action in response to satisfying a predetermined condition. The processor has programmed instructions to, responsive to satisfying the predetermined condition, access the plurality of vdisks and perform the policy action on the plurality of objects.