Abstract: Mechanisms are provided for maintaining variable size deduplication segments using fixed size extents. Variable size segments are identified and maintained in a datastore suitcase. Duplicate segments need not be maintained redundantly but can be managed by updating reference counts associated with the segments in the datastore suitcase. Segments are maintained using fixed size extents. A minor increase in storage overhead removes the need for inefficient recompaction when a segment is removed from the datastore suitcase. Fixed size extents can be reallocated for storage of new segments.

Abstract: A clustered storage system may include potentially many different nodes, each including a storage driver and a kernel module. A node may mount a virtual storage volume for the use of a container application at the node. The storage driver and kernel module may receive a request from a storage driver at a different node and respond by indicating whether the virtual storage volume is in active use. In this way, the clustered storage system may safely but forcibly unmount a virtual storage volume having a failed or hanging mount point so that the volume may be mounted on a different node.

Abstract: A container orchestrator may communicate with a scheduler agent on a compute node to mount and/or unmount a virtual storage volume at the compute node. The scheduler agent may perform operations such as creating and/or deleting a mount path, mounting and/or unmounting the virtual storage volume, and/or marking the mount path as writable and/or read-only in order to eliminate cross-node race conditions.

Abstract: Two servers implemented as containerized applications may manage the storage of data and the access of that data by compute jobs in a distributed system. A metadata server may distribute data on ingress and assign files to particular storage volumes. The metadata server may then provide a lookup function for files and be configured to distribute a file to other volumes when necessary.

Abstract: A clustered storage system may include potentially many different nodes, each including a storage driver and a scheduler agent. A node may mount a virtual storage volume for the use of a container application at the node. When a master node receives a request to instantiate a new containerized application, a scheduler at the master node may determine prioritization information for the new containerized application. The prioritization information may prioritize one or more nodes at which a virtualized storage volume employed by the application container instance is mounted.

Abstract: Clustered containerized applications are implemented with scalable provisioning. Methods include receiving a data storage request to store one or more data values in a storage volume implemented across a storage node cluster, the storage node cluster including a plurality of storage nodes including one or more storage devices having storage space allocated for storing data associated with the storage volume. Methods may further include identifying a cluster hierarchy associated with the storage node cluster, the cluster hierarchy identifying storage characteristics of the plurality of storage nodes, the cluster hierarchy also identifying physical location information for the plurality of storage nodes, the physical location information indicating node-to-node proximity on a network graph.

Abstract: Systems, devices, and methods are disclosed herein for containerized scalable storage applications. Methods may include instantiating an application instance based on a plurality of application instance parameters, the application instance being configured to utilize a plurality of storage volumes implemented in a storage cluster. Methods may also include enumerating a plurality of unattached storage volumes included in the cluster associated with the application instance, the plurality of unattached storage volumes having a plurality of underlying physical storage devices, and the plurality of unattached storage volumes being identified based on a plurality of application instance parameters. The methods may further include attaching at least some of the plurality of unattached storage volumes to the application instance, wherein the attaching enables the application instance to access data stored in the attached storage volumes.

Abstract: A clustered storage system may include potentially many different nodes, each including a storage driver and a kernel module. A node may mount a virtual storage volume for the use of a container application at the node. The storage driver and kernel module may receive a request from a storage driver at a different node and respond by indicating whether the virtual storage volume is in active use. In this way, the clustered storage system may safely but forcibly unmount a virtual storage volume having a failed or hanging mount point so that the volume may be mounted on a different node.

Abstract: Mechanisms are provided for adjusting a configuration of data stored in a storage system. According to various embodiments, a storage module may be configured to store a configuration of data. A processor may be configured to identify an estimated performance level for the storage system based on a configuration of data stored on the storage system. The processor may also be configured to transmit an instruction to adjust the configuration of data on the storage system to meet the service level objective when the estimated performance level fails to meet a service level objective for the storage system.

Abstract: Techniques and mechanisms are provided for migrating data blocks around a cluster during node addition and node deletion. Migration requires no downtime, as a newly added node is immediately operational while the data blocks are being moved. Blockmap files and deduplication dictionaries need not be updated.

Abstract: Mechanisms are provided for adjusting a configuration of data stored in a storage system. According to various embodiments, a storage module may be configured to store a configuration of data. A processor may be configured to identify an estimated performance level for the storage system based on a configuration of data stored on the storage system. The processor may also be configured to transmit an instruction to adjust the configuration of data on the storage system to meet the service level objective when the estimated performance level fails to meet a service level objective for the storage system.

Abstract: Techniques and mechanisms described herein facilitate the execution of a software program container having a plurality of layers. Each layer may include a plurality of files. For instance, a base layer may include a version of an operating system, while an upper layer may include a software program configured to run within the operating system. Different layers may store, create, or modify the same file. However, the precedence of the version of the data for the file may be based on an ordering of the layers. For example, the version of the file in a higher layer should supersede the version of the file in a lower layer. In some implementations, a privileged storage container that runs atop a virtualization layer may manage storage resources for other containers. The privileged storage container may store each file in a software program container as a snapshotable object.

Abstract: Disclosed herein are systems, methods, and devices for the implementation of clustered containerized software applications. Methods may include identifying, using a processor of a first storage container node, a storage container node cluster including a plurality of storage container nodes, and sending a packet to at least a second storage container node of the plurality of storage container nodes. The methods may also include receiving a response from the second storage container node, the response including a reply to the packet, and generating a distance map based, at least in part, on the received response, the distance map characterizing a plurality of distances between the plurality of storage container nodes and the first storage container node. The methods may further include identifying at least one additional storage container node based, at least in part, on the generated distance map.

Abstract: Techniques and mechanisms described herein facilitate the execution of a software program container having a plurality of layers. Each layer may include a plurality of files. For instance, a base layer may include a version of an operating system, while an upper layer may include a software program configured to run within the operating system. Different layers may store, create, or modify the same file. However, the precedence of the version of the data for the file may be based on an ordering of the layers. For example, the version of the file in a higher layer should supersede the version of the file in a lower layer. In some implementations, a privileged storage container that runs atop a virtualization layer may manage storage resources for other containers. The privileged storage container may store each file in a software program container as a snapshotable object.

Abstract: Techniques and mechanisms are provided for migrating data blocks around a cluster during node addition and node deletion. Migration requires no downtime, as a newly added node is immediately operational while the data blocks are being moved. Blockmap files and deduplication dictionaries need not be updated.

Abstract: Mechanisms are provided for maintaining variable size deduplication segments using fixed size extents. Variable size segments are identified and maintained in a datastore suitcase. Duplicate segments need not be maintained redundantly but can be managed by updating reference counts associated with the segments in the datastore suitcase. Segments are maintained using fixed size extents. A minor increase in storage overhead removes the need for inefficient recompaction when a segment is removed from the datastore suitcase. Fixed size extents can be reallocated for storage of new segments.

Abstract: Techniques and mechanisms described herein facilitate the execution of a software program container having a plurality of layers. Each layer may include a plurality of files. For instance, a base layer may include a version of an operating system, while an upper layer may include a software program configured to run within the operating system. Different layers may store, create, or modify the same file. However, the precedence of the version of the data for the file may be based on an ordering of the layers. For example, the version of the file in a higher layer should supersede the version of the file in a lower layer. In some implementations, a privileged storage container that runs atop a virtualization layer may manage storage resources for other containers. The privileged storage container may store each file in a software program container as a snapshotable object.

Abstract: According to various embodiments, techniques and mechanisms described herein may facilitate the resynchronization of storage container nodes within a storage volume. In some implementations, a virtual storage volume may be created by aggregating storage resources from two or more storage container nodes. Each storage container node may include a privileged storage container that runs atop a virtualization layer. For redundancy, a virtual storage volume may store the same data on two or more of the storage nodes that make up the volume. However, the data may become out-of-sync, for instance if one or more of the nodes fails during the execution of a storage operation. Data may be resynchronized after a node failure by designating data as source data for resynchronization based on comparing metadata across nodes in view of data integrity guarantees.

Abstract: A system provides file aware block level deduplication in a system having multiple clients connected to a storage subsystem over a network such as an Internet Protocol (IP) network. The system includes client components and storage subsystem components. Client components include a walker that traverses the namespace looking for files that meet the criteria for optimization, a file system daemon that rehydrates the files, and a filter driver that watches all operations going to the file system. Storage subsystem components include an optimizer resident on the nodes of the storage subsystem. The optimizer can use idle processor cycles to perform optimization. Sub-file compression can be performed at the storage subsystem.

Abstract: Mechanisms are provided for maintaining variable size deduplication segments using fixed size extents. Variable size segments are identified and maintained in a datastore suitcase. Duplicate segments need not be maintained redundantly but can be managed by updating reference counts associated with the segments in the datastore suitcase. Segments are maintained using fixed size extents. A minor increase in storage overhead removes the need for inefficient recompaction when a segment is removed from the datastore suitcase. Fixed size extents can be reallocated for storage of new segments.