Abstract: A multi-node, multi-container cluster system that generates, aggregates, and manages log files from services and components to be used for audit logs used by an auditor. Logs are collected from all components of the system and aggregated using a central persistent volume for non-critical applications and additional dedicated persistent volumes for critical tasks. Audit logs are formatted to provide a consistent logging format in conformance with relevant corporate and government requirements. The audit logs are transferred to an audit log management system that is configured to securely store the audit logs in accordance with security requirements for availability, protection against tampering, and secure access.

Abstract: A multi-node, multi-container cluster system that generates, aggregates, and manages log files from services and components to be used for audit logs used by an auditor. Logs are collected from all components of the system and aggregated using a central persistent volume for non-critical applications and additional dedicated persistent volumes for critical tasks. Audit logs are formatted to provide a consistent logging format in conformance with relevant corporate and government requirements. Log files are stored persistently outside of pods to ensure high availability for any audit log management needs. Audit logs can be exported to external, secure syslog server, and can be made subject to defined retention policies, as needed.

Abstract: A process of recovering from a failure of a solid-state device (SSD) storing a persistent volume (PV) for logging in a cluster network. The PV is recreated on a different SSD that has spare capacity and IOPS resources, and log information is redirected from an application to the relocated PV on the different SSD. Upon replacement of the original failed SSD, the PV can be copied from the different SSD to the new replacement SSD, or the replacement SSD can be left empty, and the relocated PV on the different SSD can continue to be used. This process provides a high degree of redundancy with regard to logging so that a failure of single SSD does not bring down the node or the entire cluster network.

Abstract: A multi-node, multi-container cluster system that generates, aggregates, and manages log files from services and components to be used for audit logs and to debug and perform other serviceability tasks provided by a vendor of the cluster system. Logs are collected from all components of the system and aggregated using a central persistent volume for non-critical applications and additional dedicated persistent volumes for critical tasks. The logs are implemented to minimize the risk of losing log lines and reduce resource overheads. Logs are formatted to provide a consistent logging format utilizing industry standard data formats, such as timestamps. Log files are stored persistently outside of pods to ensure high availability for any audit log management needs.

Abstract: A multi-node, multi-container cluster system that generates, aggregates, and manages log files from services and components to be used for audit logs and to debug and perform other serviceability tasks provided by a vendor of the cluster system. Logs are collected from all components of the system and aggregated using a central persistent volume for non-critical applications and additional dedicated persistent volumes for critical tasks. The logs are implemented to minimize the risk of losing log lines and reduce resource overheads. Logs are formatted to provide a consistent logging format utilizing industry standard data formats, such as timestamps. Log files are stored persistently outside of pods to ensure high availability for any audit log management needs.

Abstract: A multi-node, multi-container cluster system that generates, aggregates, and manages log files from services and components to be used for audit logs and to debug and perform other serviceability tasks provided by a vendor of the cluster system. Logs are collected from all components of the system and aggregated into a consistent format for user analysis and further audit log management. Logs are stored in a central persistent volume (PV) for non-critical components and additional dedicated persistent volumes for critical components, where critical components require at least one of node affinity or log perseverance. A PV mounter pod is used to mount the dedicated PVs to respective components.

Abstract: A single multi-container service pod that provides a centralized cluster-aware debugger and a centralized set of debugging tools where the service pod contains one or more containers for applications executed by the cluster system. Each debugging tool of the set is associated with a respective application, such as a dump utility, a connection checker, and a log search utility. A vendor providing support for the cluster network, executes a corresponding debugging tool of the set of debugging tools to address a problem encountered by a node executing an application. Debugging the problem from the service pod impacts a debugged application on the other pods, and the service pod can trigger the debugging tools on other applications on the nodes.

Abstract: Providing scalable support bundle processing for cluster filesystem serviceability by collecting system statistics and log information for each node in a cluster system, and combining the collected information to form a support bundle for the cluster system that can then be transmitted to a vendor providing support. A support bundle worker component is provided in each node and a support bundle controller is provided in a controller node, which issues a collection command to each node causing respective worker bundle components on the nodes to temporarily store data for the support bundle on each node prior to transmission as a single dataset to the vendor. A support bundle can be generated automatically upon occurrence of a node or system error condition to notify and provide information to a user or system vendor.

Abstract: A single multi-container service pod that provides a centralized cluster-aware debugger to process support bundle information including log files for around an error event. Debugging support tools include a collector collecting system statistics, system information, and logs for each node to form collected data processed by debugging tools to rectify the error. Log files are generated for the entire cluster system from the logs for each node. A loader processes the log files by decompression and normalization of names and timestamps, and loads the processed log files into a search engine through which support personnel can search for specific logs around the error to quickly find and diagnose component/service issues for debugging using the debugging tools.

Abstract: Embodiments are described for providing backup resources in a multi-tier network, by providing a plurality of different storage devices, each with different operating characteristics and costs for storage of data assets, and organizing the storage devices into a plurality of tiers based on respective operating characteristics, such as based on performance versus cost parameters. Certain Service Level Agreements (SLAs) dictating storage requirements for each of the data assets to a backup program are provided to the backup system, and a mapper maps each SLA to one or more tiers of the plurality of tiers based on the storage requirements of a respective SLA to the operating characteristics of each tier.

Abstract: A system receives a stream of fingerprints, which correspond to segments of a data file, from a source of the data file, and identifies fingerprints in the stream of fingerprints as fingerprints formatted by a first fingerprint format or a second fingerprint format. If a fingerprint, in the stream of fingerprints, does not duplicate any fingerprint in a fingerprint cache, the system stores the non-duplicate fingerprint in a first storage tier corresponding to the first fingerprint format or a second storage tier corresponding to the second fingerprint format. The system requests a segment of the data file, which corresponds to the non-duplicate fingerprint, from the source of the data file. The system receives the requested segment of the data file from the source of the data file, and stores the received segment of the data file in destination storage.

Abstract: A source site includes a controller, a set of source worker nodes, and a message queue connected between the controller and source worker nodes. The message queue receives messages and stores the messages for retrieval. A destination site includes a set of destination worker nodes. Tasks are generated to replicate changes to objects at the source site to the destination site. The controller pushes messages corresponding to the tasks onto the message queue. A source worker node retrieves a message corresponding to a task from the message queue for processing in conjunction with a destination worker node. The message is indicated as having been retrieved from the message queue.

Abstract: Files in a file system are assigned to front-end instances. The files are represented by segment trees. Similarity groups are assigned to back-end instances. The similarity groups include segments at a lower-level of the segment trees. The front-end instances are responsible for operations involving an upper-level of the segment trees. The back-end instances are responsible for operations involving the lower-level of the segment trees. Upon a failure of an instance, an attempt is made to restart the failed instance. If the restart is successful, file and similarity group assignments are retained. If the restart is not successful, file assignments are redistributed to other front-end instances when the failed instance is a front-end instance, and similarity group assignments are redistributed to other back-end instances when the failed instance is a back-end instance.

Abstract: Embodiments are described for migrating data and load balancing in a multi-domain network having a clustered network partitioned into a number of separate domains. Each domain contains resources like processors and memory, as well as deduplication and compression services. A single global namespace for the network is presented by combining all of the namespaces in the domains. A multi-domain manager component with an analytics engine facilitates analytics-driven migration of data among nodes in different domains, and a load balancer component monitors environmental operating conditions of each node in the clustered network, detects nodes having operating conditions exceeding defined tolerances, and migrates data and processes from nodes exceeding the defined tolerances to other nodes not exceeding the defined tolerances.

Abstract: A distributed and deduplicating file system that stores backup metadata to object storage is described. A node, in a cluster of nodes, receives backup data associated with a write operation to back up a data file, and identifies metadata associated with deduplicated data by deduplicating the backup data. If the access frequency associated with the metadata is greater than or equal to an access threshold, and the size of the metadata is less than or equal to a storage threshold, the cluster writes the metadata to each of the memory devices corresponding to the nodes in the cluster. If the access frequency associated with the metadata is less than the access threshold, or the size of the metadata is more than the storage threshold, the node writes information which includes the metadata to an object storage that stores data objects for the nodes in the cluster.

Abstract: Mapping information identifies ranges of files, a set of front-end microservices, and assignments of the ranges to the front-end microservices. Each front-end microservice is thereby responsible for a range of files. The files are represented by segment trees and the front-end microservices handle operations involving an upper-level of the segment trees. A file system request on a file is directed to a particular front-end microservice that is responsible for handling a particular range of files within which the file falls according to the mapping information. An indication is received from a container orchestration service that a number of front-end microservices has changed. The mapping information is updated based on the change in the number of front-end microservices.

Abstract: Embodiments are described for a system and method facilitating deduplication in a multi-tier storage system in which a file can have different portions written to different tiers. A process partition the data space of each tier to a number of similarity groups and distributes the similarity groups across file system services in a cluster. The distribution is done in such a way that for a given similarity group ID, the same file system service owns the similarity groups of every tier. This allows for efficient checks for deduplication as it can be done local to a node rather than requiring remote procedure calls.

Abstract: Systems and methods for cleaning a storage system. A deduplicated storage system is cleaned by identifying structures that include dead or unreferenced segments. This includes processing recipes to identify the segments that are no longer part of a live object recipe. Then, the dead segments are removed. This is accomplished by copying forward the live segments and then deleting, as a whole, the structure that included the dead segments.

Abstract: Embodiments are described for migrating data and load balancing in a multi-domain network having a clustered network partitioned into a number of separate domains. Each domain contains resources like processors and memory, as well as deduplication and compression services. A single global namespace for the network is presented by combining all of the namespaces in the domains. A multi-domain manager component with an analytics engine facilitates analytics-driven migration of data among nodes in different domains, and a load balancer component monitors environmental operating conditions of each node in the clustered network, detects nodes having operating conditions exceeding defined tolerances, and migrates data and processes from nodes exceeding the defined tolerances to other nodes not exceeding the defined tolerances.

Abstract: Embodiments are described for providing backup resources in a multi-tier network, by providing a plurality of different storage devices, each with different operating characteristics and costs for storage of data assets, and organizing the storage devices into a plurality of tiers based on respective operating characteristics, such as based on performance versus cost parameters. Certain Service Level Agreements (SLAs) dictating storage requirements for each of the data assets to a backup program are provided to the backup system, and a mapper maps each SLA to one or more tiers of the plurality of tiers based on the storage requirements of a respective SLA to the operating characteristics of each tier.