Abstract: A computer implemented method includes maintaining information indicative of whether a data replication relationship between a dataset associated with the local CG (CG1) and a mirror copy of the dataset stored on a remote CG (CG2) of a remote distributed storage system is in an in-synchronization (InSync) state or an out-of-synchronization (OOS) state, determining whether a CG storage expansion process is in progress, and in response to determining the OOS state and whether the primary storage site has a failure, initiating an automatic unplanned failover (AUFO) workflow without manual intervention on the original volumes in the CG1 of a first storage cluster of the primary storage site and the original volumes in the CG2 of a second storage cluster when the CG storage expansion process is in progress with a new source volume to be a member of CG1 and a new destination volume to be a member of CG2.

Abstract: The disclosure describes artificial intelligence (AI) data platform that utilizes snapshots obtained from a storage node to update a vector database. The AI data platform compares snapshots to generate differential snapshots that identify changed data in storage volumes. The AI data platform uses the differential snapshots to update vector embeddings in a vector database for retrieval-augmented generation (RAG) workflows.

Abstract: The disclosure describes a node management service that determines to deploy a primary workload to a new instance in a compute cluster. The node management service projects an allocatable capacity for the new instance based on historical capacity data. The allocatable capacity is the amount of compute resources available for running the primary workload in the new instance after deployment of a supporting workload to the new instance. The node management service associates the primary workload with the new instance upon determining that the projected allocatable capacity is sufficient for running the primary workload.

Abstract: Techniques are provided for implementing a distributed control plane to facilitate communication between a container orchestration platform and a distributed storage architecture. The distributed storage architecture hosts worker nodes that manage distributed storage that can be made accessible to applications within the container orchestration platform through the distributed control plane. However, the worker nodes may support an imperative model of program commands, but the container orchestration platform and applications therein utilize a declarative model of programming commands not supported by the distributed storage architecture. Accordingly, the distributed control plane is configured with control plane controllers that are paired with the worker nodes and are configured to reformat commands between the imperative model and the declarative model.

Abstract: Techniques are provided for combining data block and checksum block I/O into a single I/O operation. Many storage systems utilize checksums to verify the integrity of data blocks stored within storage devices managed by a storage stack. However, when a storage system reads a data block from a storage device, a corresponding checksum must also be read to verify integrity of the data in the data block. This results in increased latency because two read operations are being processed through the storage stack and are being executed upon the storage device. To reduce this latency and improve I/O operations per second, a single combined I/O operation corresponding to a contiguous range of blocks including the data block and the checksum block is processed through the storage stack instead of two separate I/O operations. Additionally, I/O operation may be combined into a single request that is executed upon the storage device.

Abstract: Systems and methods for enhancing container security are provided by reducing the attack surface. In one example, the exposure of containers to potential security vulnerabilities is reduced by identifying dynamically loaded symbols by an application via performance of static symbol analysis by examining a section of an executable to identify dynamically loaded symbols corresponding to functions contained within shared libraries. Based on a given shared library's usage of functions within standard libraries and a known mapping between functions of standard libraries and system calls, those system calls potentially accessed by the application may be identified and a security policy may be generated and configured for enforcement by a kernel security module to limit system call usage accordingly. Thereafter, the security policy enforced by the kernel security module may be refined based on performance of dynamic symbol analysis to identify system calls that are actually called by the application during runtime.

Abstract: Systems and methods are provided for data migration including cloning a multiple-logical unit number (LUN) volume into a plurality of cloned single-LUN volumes on a first storage node of a computing system; creating a plurality of new volumes on a second storage node of the computing system based at least in part on the plurality of cloned single-LUN volumes; selectively copying snapshot data from the plurality of cloned single-LUN volumes to the plurality of new volumes; establishing snapshot mirror relationships between the plurality of cloned single-LUN volumes and the plurality of new volumes; synchronizing the plurality of cloned single-LUN volumes and the plurality of new volumes; and performing a migration of data logical interface failovers (LIFs) from the plurality of cloned single-LUN volumes to the plurality of new volumes.

Abstract: The disclosure describes systems, devices, and methods for managing multi-part files and attributes thereof in data storage environments. In an example implementation, a method of operating multiple storage controllers in a data storage environment is provided. In performing the method, a data element of a storage controller of the multiple storage controllers receives an input/output request from a network element of a one of the multiple storage controllers corresponding to data to a multi-part file stored across multiple storage devices in the data storage environment. The input/output request includes current attributes of the multi-part file. The data element executes the input/output request with respect to a part of the multi-part file corresponding to the storage controller and determines whether to delegate attribute authority to the network element based on a set of criteria.

Abstract: Techniques are provided for volume group backup, volume group restore, and volume group garbage collection for volume groups backed up to an object store. A volume group workflow is implemented to orchestrate individual consistent volume workflows that are separately and individually implemented by nodes hosting constituent volumes of a volume group. The volume group workflow and the individual consistent volume workflows are performed to back up the volume group to the object store, restore a volume group backup from the object store to a restore destination, and/or perform garbage collection on slots of objects storing data unique to a volume group backup to delete.

Abstract: Methods and systems for managing storage is provided. One method includes initializing, a storage service user interface (“SSUI”) within a cloud manager user interface (“CMUI”), the SSUI enables a storage service operation selected from a backup operation, a restore operation and a cloning operation associated with a storage object stored at a cloud volume presented by a cloud layer, the CMUI presented by a cloud provider; transmitting, by the SSUI, an authorization token to a Software As a Service (“SaaS”) layer for authenticating deployment of the SSUI; upon validating the authorization token, initializing a SSUI agent to interface with a deployed storage micro-service layer offered by the cloud layer to execute the storage service operation; transmitting, by the SSUI, an application programming interface (API) request for the SSUI agent for executing the storage service operation; and executing, by the deployed storage micro-service layer, the storage service operation.

Abstract: Systems and methods for multiple device consumption of shared namespaces of ephemeral storage devices by a consumer of a virtual storage system are provided. In an example, multiple namespaces of respective ephemeral storage devices are shared among multiple consumers of a virtual storage system by creating multiple partitions within each of the namespaces for use by respective consumers of the multiple consumers. Corresponding partitions of respective shared namespace may then be treated as a stripe set to facilitate multiple device consumption for a subsystem (e.g., operation log journaling) of the virtual storage system by striping data associated with input/output (I/O) requests of a consumer (e.g., a journaling driver) across one or more stripe units of one or more stripes within the stripe set.

Abstract: A monitoring system can generate compressed storage system monitoring data segments using monitoring data obtained from a storage system. The monitoring system can obtain compression information for a compressed segment generated from a segment of storage system monitoring data and generate a predicted portion of storage system monitoring data using the compression information. The monitoring system can obtain an additional portion of storage system monitoring data, the additional portion contiguous to the segment of storage system monitoring data, and determine that the predicted portion matches the additional portion and combining the additional portion and the compressed segment. In response to a user query, the monitoring system can perform at least one of: reconstructing and providing the additional portion using the compressed segment; or providing the compressed segment for reconstruction of the additional portion.

Abstract: Various embodiments of the present technology generally relate to systems and methods for providing a resource distribution engine. In an example, a method includes identifying, by an instance management service, a request from a cluster orchestrator to acquire computing resources associated with deployment of an application. The method may also include determining, by the instance management service, markets containing reclaimable resources suitable for the computing resources, and performing, by a resource distribution engine of the instance management service, a resource distribution process to determine a resource distribution of the reclaimable resources within the markets. The resource distribution of the reclaimable resources may include a distribution of instance type of the reclaimable resources across the markets. The method may also include securing, by the instance management service, the reclaimable resources identified by the resource distribution as provisioned reclaimable resources.

Abstract: Systems and methods for use of information relating to units of storage space movement and ownership to facilitate performance of disaggregated storage system workflows are provided. In various examples, the unit of disaggregated storage space for purposes of movement and ownership is an allocation area (AA). AA ownership information relating to AAs owned by dynamically extensible file systems (DEFSs) of a cluster may be maintained within two different persistent data sources, including an AA label region and DEFS AA owner metafiles. A node-scoped cache may be implemented to cache the AA ownership information in-memory for DEFSs hosted by a given node to provide a multiprocessing (MP)-safe cache for AA ownership information, a mechanism to coordinate between AA movement and other subsystems needing to know whether an AA is “local” or “remote,” and a mechanism to serialize access and Input/Output (IO) operations to the persistent AA ownership information data sources.

Abstract: Systems and methods for use of information relating to units of storage space movement and ownership to facilitate performance of disaggregated storage system workflows are provided. In various examples, the unit of disaggregated storage space for purposes of movement and ownership is an allocation area (AA). AA ownership information relating to AAs owned by dynamically extensible file systems (DEFSs) of a cluster may be maintained within two different persistent data sources, including an AA label region and DEFS AA owner metafiles. A node-scoped cache may be implemented to cache the AA ownership information in-memory for DEFSs hosted by a given node to provide a multiprocessing (MP)-safe cache for AA ownership information, a mechanism to coordinate between AA movement and other subsystems needing to know whether an AA is “local” or “remote,” and a mechanism to serialize access and Input/Output (IO) operations to the persistent AA ownership information data sources.

Abstract: Systems and methods for use of information relating to units of storage space movement and ownership to facilitate performance of disaggregated storage system workflows are provided. In various examples, the unit of disaggregated storage space for purposes of movement and ownership is an allocation area (AA). AA ownership information relating to AAs owned by dynamically extensible file systems (DEFSs) of a cluster may be maintained within two different persistent data sources, including an AA label region and DEFS AA owner metafiles. A node-scoped cache may be implemented to cache the AA ownership information in-memory for DEFSs hosted by a given node to provide a multiprocessing (MP)-safe cache for AA ownership information, a mechanism to coordinate between AA movement and other subsystems needing to know whether an AA is “local” or “remote,” and a mechanism to serialize access and Input/Output (IO) operations to the persistent AA ownership information data sources.

Abstract: Systems and methods for performing space balancing heuristics within a distributed storage system that makes use of disaggregated storage are provided. In various examples described herein, the unit of storage space assignment used to associate blocks of storage space within a storage pod with a given dynamically extensible file system (DEFS) is an AA, representing a large chunk of one or more gigabytes (GB). The use of AAs described herein allows disk space to be used more elastically across individual nodes of a storage cluster. Each DEFS periodically reports its AA usage information by persisting it to a respective heuristic block within the storage pod. An AA director operable within the cluster periodically monitors the reported AA usage information, performs a heuristic algorithm to determine donor and recipient DEFSs at a given instance, and triggers AA movement accordingly by matching one or more donor DEFSs to each recipient DEFS.

Abstract: Methods and systems for a networked storage environment are provided. One method includes generating a first and a second batch of entries in response to scanning a source file system, each batch of entries associated with one or more directories of the source file system and indicating a path to a file associated with each entry; determining, by a first worker process, a first checksum for data associated with the first batch of entries loaded in a first buffer; appending, the first buffer contents processed by the first worker process to a first archive file; and generating an archive data structure having a manifest file storing metadata for the first batch and a second batch entries with a first checksum determined by the first worker process and a second checksum determined by a second worker process, and data from the first archive file and a second archive file.

Abstract: The disclosure describes systems, devices, and methods for managing data storage environments. In an example implementation, a method of operating a controller in a data storage environment is provided. In performing the method, the controller identifies a change to a layout of drives in the data storage environment, and in response to identifying the change, takes a lock on instances of layout metadata stored on the drives. The controller then updates the instances of the layout metadata to reflect the change to the layout and releases the lock.

Abstract: The disclosure describes a workflow management service that dynamically provisions workflow documents across multiple accounts to avoid reaching document limits imposed by a compute provider. The workflow management service initially allocates workflow documents to a central account, which manages workflow execution and distribution. The workflow management service also provisions shard accounts dedicated to hosting and executing workflow documents. When the number of workflow documents in the central account or an existing shard account approaches the compute provider's imposed limit, the workflow management service provisions a new shard account and redirects future workflow documents to the new shard account.