Abstract: A new snapshot of a storage volume is created by instructing computing nodes to suppress write requests. Once pending write requests from the computing nodes are completed, storage nodes create a new snapshot for the storage volume by allocating a new segment to the new snapshot and finalizes and performs garbage collection with respect to segments allocated to the previous snapshot. Subsequent write requests to the storage volume are then performed on the segments allocated to the new snapshot. A segment maps segments to a particular snapshot and metadata stored in the segment indicates storage volume addresses of data written to the segment. The snapshots may be represented by a storage manager in a hierarchy that identifies an ordering of snapshots and branches to clone snapshots.

Abstract: A workflow may include function calls to functions executed with respect to instances of elements. Health checks of elements of the workflow may be performed with reference to topology such that health checks of elements are omitted where a health check is performed by a parent in the topology. Batch processing of stages of a workflow may be performed with commencement of a stage being initiated based on completion percentage and execution time of a previous stage. Tasks of each stage may be performed by a pool of workers that are reused with pool size being selected based on expected completion time of each stage.

Abstract: A new snapshot of a storage volume is created by instructing computing nodes to suppress write requests. Once pending write requests from the computing nodes are completed, storage nodes create a new snapshot for the storage volume by allocating a new segment to the new snapshot and finalizes and performs garbage collection with respect to segments allocated to the previous snapshot. Subsequent write requests to the storage volume are then performed on the segments allocated to the new snapshot. An orchestration layer implements a multi-role application that is provisioned with virtualized storage and computation resources. A snapshot of the application may be created and used to rollback or clone the application. Clone snapshots of storage volumes may be gradually populated with data from prior snapshots to reduce loading on a primary snapshot.

Abstract: Snapshots are created of a unit of storage such that the snapshot includes all segments allocated to the unit of storage until a subsequent snapshot is created. A backup of a first snapshot may be performed by copying segments of the snapshot to backup storage. If a preceding snapshot has been backed up, those segments of the first snapshot are copied to backup storage without copying the segments of the preceding snapshot, otherwise all segments of the snapshot are copied. Metadata describing the snapshot may be retained but storage may be freed to be overwritten. The metadata may reference identifiers of segments copied to backup storage. Data stored in backup storage may also be labeled as individual segments or be associated with snapshot metadata in backup storage.

Abstract: A workflow may include function calls to functions executed with respect to instances of elements. Functions may be stored in a file store. When functions are edited and checked in to the file store, change to the function may be detected and workflows implicated by the change to the function are identified. Execution of the implicated workflows may then be invoked in response to detecting the change. Functions may have an exclusivity associated with them such that execution of a workflow includes executing function calls alone where required by the exclusivity or concurrently where permitted by the exclusivity. Check-in monitoring on a repository of element files (element definition and element functions) may be performed. In response to check-in of a file, workflows including elements referencing the file may be executed.

Abstract: Snapshots of storage volumes and containers of a bundled application may be created and used to rollback or clone the bundled application. Clone snapshots of storage volumes may be gradually populated with data from prior snapshots to reduce loading on a primary snapshot. Components of cloned applications may communicate with one another using addresses of these components in the parent application. Containers of the bundled application may communicate with an open virtual switch (OVS) that implements flows to implement translation between clone and parent addresses. Containers may be modified to execute operation-specific entrypoint functions prior to invoking an entrypoint of an application instance loaded in the containers.

Abstract: A bundled application includes a plurality of entities such as logical storage volumes, application instances, pods, clusters, and computing nodes that are dependent on one another. Dependencies of the bundled application on individual entities is determined and quantified. Impact of failure of an entity may be determined using the dependencies. Dependency may be determined with reference to redundancy among entities. Usage of an entity by other entities and potential redistribution may be determined.

Abstract: A new snapshot of a storage volume is created by instructing computing nodes to suppress write requests. An orchestration layer implements a multi-role application that is provisioned with virtualized storage and computation resources. A snapshot of the application may be created and used to rollback or clone the application. Clones snapshots of storage volumes may be gradually populated with data from prior snapshots to reduce loading on a primary snapshot. A multi-role application may be orchestrated using multiple orchestration approaches and objects of the multi-role application may be discovered and added to an application definition. The application definition may be used to create snapshots of the application and the snapshot may be used to rollback, clone, backup, or migrate the application.

Abstract: Systems and methods are disclosed for generating traffic statistics for a datacenter. Distributed, parallel processing may be used to generate traffic statistics from data sets about traffic in a datacenter. To reduce data sets from which such statistics are derived to manageable sizes and relevant processing times for distributed, parallel processing, traffic agents may be provided at end hosts in the datacenter. The traffic agents may summarize data traffic over large numbers of packets in terms of the various sockets over which they are transmitted. Reports on the various sockets may be sent by the various flow agents that monitor them to an analytic controller. The analytic controller may aggregate, provide flow-path information for, further reduce, and/or provision the resultant data for distributed parallel processing.

Abstract: Systems and methods are disclosed for reducing latency in processing data sets in a distributed fashion. A job-queue operable for queuing data-processing jobs run on multiple nodes in a cluster may be communicatively coupled to a job analyzer. The job analyzer may be operable to read the data-processing jobs and extract information characterizing those jobs in ways that facilitate identification of resources in the cluster serviceable to run the data-processing jobs and/or data to be processed during the running of those jobs. The job analyzer may also be coupled to a resource warmer operable to warm-up a portion of the cluster to be used to run a particular data-processing job prior to the running of the job. In some embodiments, mappers and/or reducers may be extracted from the jobs and converted into compute node identifiers and/or data units identifying blocks for processing, informing the warm-up operations of the resource warmer.

Abstract: Systems and methods are disclosed for scheduling jobs processed in a distributed fashion to realize unharnessed efficiencies latent in the characteristics of the jobs and distributed processing technologies. A job store may be communicatively coupled to a job analyzer. The job analyzer may be operable to read information characterizing a job to identify multiple data blocks to be processed during the job at multiple locations in a cluster of nodes. A scheduling module may use information about the multiple data blocks, their storage locations, their status with respect to being provisioned to processing logic, data blocks to be processed by other jobs, data blocks in cache that have been pre-fetched for a prior job, quality-of-services parameters, and/or job characteristics, such as job size, to schedule the job in relation to other jobs.

Abstract: Systems and methods are disclosed for classifying traffic flows. A traffic agent operable to collect classification information for one or more traffic flows may be deployed at an end host communicatively coupled to a data-center network. The traffic agent, deployed in a user space independent of the operating system, may compare the classification information for a given traffic flow to a metric value. Where the classification information achieves a certain threshold indicated by the metric value, the traffic agent may classify the traffic flow as an elephant flow. In some examples, a library may be included with the traffic agent that may include a modified send function. The modified send function may provide classification information to the traffic agent indexed to the traffic flow for which it is called so that the traffic agent may analyze the classification information to potentially provide a classification for the traffic flow.

Abstract: Systems and methods are disclosed for interacting with a file system. The file system is operable to reside in user space of a computing system. A module, also within user space, may provide a messaging service supporting requests from an application to the file system. By bypassing a System-Call Interface (SCI) of the computing system's kernel space, the module may support requests from the application to the file system with enhanced efficiency and/or customizable features not provided by the SCI. In some examples, the module may include a library in an independent layer within user space and below the application, allowing the library to provide an application-independent messaging service for different applications. Furthermore, in some examples, the module may include a segment of memory, within user space, shared between the application and the file system for passing data involved in requests and/or responses to and/or from the file system.