Abstract: Described in detail herein are systems and methods for single instancing blocks of data in a data storage system. For example, the data storage system may include multiple computing devices (e.g., client computing devices) that store primary data. The data storage system may also include a secondary storage computing device, a single instance database, and one or more storage devices that store copies of the primary data (e.g., secondary copies, tertiary copies, etc.). The secondary storage computing device receives blocks of data from the computing devices and accesses the single instance database to determine whether the blocks of data are unique (meaning that no instances of the blocks of data are stored on the storage devices). If a block of data is unique, the single instance database stores it on a storage device. If not, the secondary storage computing device can avoid storing the block of data on the storage devices.

Abstract: A scalable approach is disclosed for processing auxiliary-copy jobs in a storage management system by using distributed media agent resources instead of a centralized storage manager. Enhanced media agents coordinate and control auxiliary-copy jobs and tap the storage manager to reserve data streams and provide job-specific metadata on demand. An enhanced storage manager may initially select a media agent as “coordinator” to coordinate auxiliary-copy jobs with any number of other media agents, which act as “controllers.” A coordinator media agent is generally responsible for obtaining data stream reservation information from the storage manager and assigning auxiliary-copy jobs to respective controller media agents, based on the components involved in the respective reserved data streams.

Abstract: An illustrative approach to managing snapshots streamlines how and when snapshots are generated in a storage management system, such that fewer snapshots may be generated without diminishing the scope of data protection. A novel unified-snapshot storage policy may govern snapshots for any number of subclients. A unified-snapshot job based on the unified-snapshot storage policy enables the illustrative storage management system to automatically discover relevant components and generate at most one snapshot per target logical unit number (“LUN”) in a storage array. Each snapshot may comprise the data of any number of subclients and/or clients in the storage management system. Accordingly, one unified-snapshot job may yield a minimum but sufficient number of snapshots comprising data of all subclients associated with the governing unified-snapshot storage policy. An enhanced storage manager may manage the unified-snapshot jobs.

Abstract: A distributed, deduplicated storage system according to certain embodiments is arranged in a parallel configuration including multiple deduplication nodes. Deduplicated data is distributed across the deduplication nodes. The deduplication nodes can be networked together and communicate with one another according using a light-weight, customized communication scheme (e.g., a scheme based on FTP or HTTP). In some cases, deduplication management information including deduplication signatures and/or other metadata is stored separately from the deduplicated data in deduplication management nodes, improving performance and scalability.

Abstract: Data storage operations, including content-indexing, containerized deduplication, and policy-driven storage, are performed within a cloud environment. The systems support a variety of clients and cloud storage sites that may connect to the system in a cloud environment that requires data transfer over wide area networks, such as the Internet, which may have appreciable latency and/or packet loss, using various network protocols, including HTTP and FTP. Methods are disclosed for content indexing data stored within a cloud environment to facilitate later searching, including collaborative searching. Methods are also disclosed for performing containerized deduplication to reduce the strain on a system namespace, effectuate cost savings, etc. Methods are disclosed for identifying suitable storage locations, including suitable cloud storage sites, for data files subject to a storage policy.

Abstract: Described in detail herein are systems and methods for single instancing blocks of data in a data storage system. For example, the data storage system may include multiple computing devices (e.g., client computing devices) that store primary data. The data storage system may also include a secondary storage computing device, a single instance database, and one or more storage devices that store copies of the primary data (e.g., secondary copies, tertiary copies, etc.). The secondary storage computing device receives blocks of data from the computing devices and accesses the single instance database to determine whether the blocks of data are unique (meaning that no instances of the blocks of data are stored on the storage devices). If a block of data is unique, the single instance database stores it on a storage device. If not, the secondary storage computing device can avoid storing the block of data on the storage devices.

Abstract: Techniques disclosed herein are used for restoring deduplicated backup data from cloud-based storage and from multi-node replicated files systems, and they also may be used in more traditional data storage technologies. Pre-restore steps include analysis of deduplication indexes to identify data segments that are stored consecutively on storage media. Reading data in aggregate runs of consecutively stored data segments reduces interactions with storage media that hosts the deduplicated data and speeds up retrieval. Parallel reads from multiple storage devices in multi-node replicated file systems also speed up retrieval. An illustrative enhanced media agent pre-fetches data (stored in deduplicated form) in anticipation of read requests that are expected in the restore operation.

Abstract: A distributed, deduplicated storage system according to certain embodiments is arranged in a parallel configuration including multiple deduplication nodes. Deduplicated data is distributed across the deduplication nodes. The deduplication nodes can be networked together and communicate with one another according using a light-weight, customized communication scheme (e.g., a scheme based on FTP or HTTP). In some cases, deduplication management information including deduplication signatures and/or other metadata is stored separately from the deduplicated data in deduplication management nodes, improving performance and scalability.

Abstract: An information management system provides a data deduplication system that uses a primary table, a deduplication chunk table, and a chunk integrity table to ensure that a referenced deduplicated data block is only verified once during the data verification of a backup or other replication operation. The data deduplication system may reduce the computational and storage overhead associated with traditional data verification processes. The primary table, the deduplication chunk table, and the chunk integrity table, all of which may be stored in a deduplication database, can also ensure synchronization between the deduplication database and secondary storage devices.

Abstract: Systems and methods for constructing a database in an information management system. The systems and methods may include using metadata stored in non-production storage devices to construct an information management system database. In some implementations the methods are used to consolidate multiple information management systems. In other implementations, the systems and methods are used for reviewing content of archived storage media.

Abstract: An information management system implements a method for determining whether to initiate a failover of a storage manager in the system. Nodes in the system may be assigned as monitoring nodes configured to communicate with a storage manager. Furthermore, each monitoring node may be configured to transmit a status inquiry message to each other monitoring node, and each monitoring node may receive a status inquiry message from each other monitoring node. Each monitoring node may also generate a session message comprising status information, and a plurality of failover nodes may receive the generated session messages. A failover node may then determine whether a failover condition has occurred by applying a plurality of failover rules to the status information of the transmitted session message. Where the failover node determines that a failover condition has occurred, the failover node may activate an offline storage manager to replace the active storage manager.

Abstract: Presented herein are systems and methods for backing up and restoring related backup data stored in different cloud storage tiers in a cloud storage environment. Backup data generated during a backup job is stored to different cloud storage tiers based on the type of data and a storage policy comprising designations of a first cloud storage tier and a second cloud storage tier. Backup metadata and index data is stored in the designated first cloud storage tier and backup payload data is stored in the designated second cloud storage tier. Designations of the storage tiers are based on the attributes associated with each tier of cloud storage. During restore, the backup payload data is recalled from the second storage tier into the first storage tier where it is used to restore to the destination client or customer.

Abstract: A deduplicated storage system is provided according to certain embodiments that uses one or more mechanisms to update the deduplication database and remove records corresponding to data blocks that have been or will be erased from the secondary copies, without using or tracking reference counting values. Some embodiments described herein use a secondary table to identify the corresponding records from the primary table that can be removed and/or moved to another table for storing “zero-reference” data blocks. In other embodiments, the system will then traverse the “zero-reference” table and remove those primary data blocks from secondary storage devices.

Abstract: An improved information management system that implements a staging area or cache to temporarily store primary data in a native format before the primary data is converted into secondary copies in a secondary format is described herein. For example, the improved information management system can include various media agents that each include one or more high speed drives. When a client computing device provides primary data for conversion into secondary copies, the primary data can initially be stored in the native format in the high speed drive(s). If the client computing device then submits a request for the primary data, the media agent can simply retrieve the primary data from the high speed drive(s) and transmit the primary data to the client computing device. Because the primary data is already in the native format, no conversion operations are performed by the media agent, thereby reducing the restore delay.

Abstract: Techniques for providing shared access to, e.g., a small computer system interface (SCSI) storage device in a computer network include providing an operational mode on SCSI interfaces with a first media agent and a second media agent such that, in response to inquiry messages on the SCSI interfaces, the SCSI storage device appears as a SCSI target device to the first media agent and the second media agent and mapping data operations between the first media agent and the SCSI storage device and the second media agent and the SCSI storage device to logically unique channel numbers for the first media agent and the second media agent to perform data storage operations over their respective SCSI interfaces by concurrently sharing the SCSI storage device.

Abstract: A deduplicated storage system is provided according to certain embodiments that uses one or more mechanisms to update the deduplication database and remove records corresponding to data blocks that have been or will be erased from the secondary copies, without using or tracking reference counting values. Some embodiments described herein use a secondary table (for tracking archive file contents) and a bitmap to mark which primary records are present in the secondary table. In another embodiment, once the marking phase is completed, the deduplication system uses the marked-up bitmap to identify the corresponding records from the primary table that can be moved to another table for storing “zero-reference” data blocks. In other embodiments, the system will then traverse the “zero-reference” table and remove those primary data blocks from secondary storage devices.

Abstract: Data storage operations, including content-indexing, containerized deduplication, and policy-driven storage, are performed within a cloud environment. The systems support a variety of clients and cloud storage sites that may connect to the system in a cloud environment that requires data transfer over wide area networks, such as the Internet, which may have appreciable latency and/or packet loss, using various network protocols, including HTTP and FTP. Methods are disclosed for content indexing data stored within a cloud environment to facilitate later searching, including collaborative searching. Methods are also disclosed for performing containerized deduplication to reduce the strain on a system namespace, effectuate cost savings, etc. Methods are disclosed for identifying suitable storage locations, including suitable cloud storage sites, for data files subject to a storage policy.

Abstract: According to certain aspects, a method can include receiving, in response to an indication that a data storage database is being restored to a second time before a first time such that the data storage database comprises a plurality of first archive file identifiers associated at the second time, a first instruction from a data storage computer, where the first instruction instructs a media agent to stop scheduled secondary storage operations associated with a deduplication database, and where the deduplication database comprises a plurality of second archive file identifiers; determining at least one second archive file identifier in the plurality of second archive file identifiers that does not correlate with any first archive identifier in the plurality of first archive file identifiers; and, for each of the at least one second archive identifier, instructing the deduplication database to prune an entry associated with the respective second archive file identifier.

Abstract: A deduplicated storage system is provided according to certain embodiments that uses one or more mechanisms to update the deduplication database and remove records corresponding to data blocks that have been or will be erased from the secondary copies, without using or tracking reference counting values. Some embodiments described herein use a secondary table (for tracking archive file contents) and a bitmap to mark which primary records are present in the secondary table. In another embodiment, once the marking phase is completed, the deduplication system uses the marked-up bitmap to identify the corresponding records from the primary table that can be moved to another table for storing “zero-reference” data blocks. In other embodiments, the system will then traverse the “zero-reference” table and remove those primary data blocks from secondary storage devices.

Abstract: A high availability distributed, deduplicated storage system according to certain embodiments is arranged to include multiple deduplication database media agents. The deduplication database media agents store signatures of data blocks stored in secondary storage. In addition, the deduplication database media agents are configured as failover deduplication database media agents in the event that one of the deduplication database media agents becomes unavailable.