Abstract: The disclosed techniques enable push-based piggybacking of a source-driven logical replication system. Logical replication of a data set (e.g., a snapshot) from a source node to a destination node can be achieved from a source-driven system while preserving the effects of storage efficiency operations (deduplication) applied at the source node. However, if missing data extents are detected at the destination, the destination has an extent pulling problem as the destination may not have knowledge of the physical layout on the source-side and/or mechanisms for requesting extents. The techniques overcome the extent pulling problem in a source-driven replication system by introducing specific protocols for obtaining missing extents within an existing replication environment by piggybacking data pushes from the source.

Abstract: A technique to name data is disclosed to allow preservation of storage efficiency over a link between a source and a destination in a replication relationship as well as in storage at the destination. The technique allows the source to send named data to the destination once and refer to it by name multiple times in the future, without having to resend the data. The technique also allows the transmission of data extents to be decoupled from the logical containers that refer to the data extents. Additionally, the technique allows a replication system to accommodate different extent sizes between replication source and destination while preserving storage efficiency.

Abstract: The disclosed techniques enable push-based piggybacking of a source-driven logical replication system. Logical replication of a data set (e.g., a snapshot) from a source node to a destination node can be achieved from a source-driven system while preserving the effects of storage efficiency operations (deduplication) applied at the source node. However, if missing data extents are detected at the destination, the destination has an extent pulling problem as the destination may not have knowledge of the physical layout on the source-side and/or mechanisms for requesting extents. The techniques overcome the extent pulling problem in a source-driven replication system by introducing specific protocols for obtaining missing extents within an existing replication environment by piggybacking data pushes from the source.

Abstract: The disclosed techniques enable push-based piggybacking of a source-driven logical replication system. Logical replication of a data set (e.g., a snapshot) from a source node to a destination node can be achieved from a source-driven system while preserving the effects of storage efficiency operations (deduplication) applied at the source node. However, if missing data extents are detected at the destination, the destination has an extent pulling problem as the destination may not have knowledge of the physical layout on the source-side and/or mechanisms for requesting extents. The techniques overcome the extent pulling problem in a source-driven replication system by introducing specific protocols for obtaining missing extents within an existing replication environment by piggybacking data pushes from the source.

Abstract: A technique to name data is disclosed to allow preservation of storage efficiency over a link between a source and a destination in a replication relationship as well as in storage at the destination. The technique allows the source to send named data to the destination once and refer to it by name multiple times in the future, without having to resend the data. The technique also allows the transmission of data extents to be decoupled from the logical containers that refer to the data extents. Additionally, the technique allows a replication system to accommodate different extent sizes between replication source and destination while preserving storage efficiency.

Abstract: A technique to name data is disclosed to allow preservation of storage efficiency over a link between a source and a destination in a replication relationship as well as in storage at the destination. The technique allows the source to send named data to the destination once and refer to it by name multiple times in the future, without having to resend the data. The technique also allows the transmission of data extents to be decoupled from the logical containers that refer to the data extents. Additionally, the technique allows a replication system to accommodate different extent sizes between replication source and destination while preserving storage efficiency.

Abstract: The disclosed techniques enable push-based piggybacking of a source-driven logical replication system. Logical replication of a data set (e.g., a snapshot) from a source node to a destination node can be achieved from a source-driven system while preserving the effects of storage efficiency operations (deduplication) applied at the source node. However, if missing data extents are detected at the destination, the destination has an extent pulling problem as the destination may not have knowledge of the physical layout on the source-side and/or mechanisms for requesting extents. The techniques overcome the extent pulling problem in a source-driven replication system by introducing specific protocols for obtaining missing extents within an existing replication environment by piggybacking data pushes from the source.

Abstract: The disclosed techniques enable push-based piggybacking of a source-driven logical replication system. Logical replication of a data set (e.g., a snapshot) from a source node to a destination node can be achieved from a source-driven system while preserving the effects of storage efficiency operations (deduplication) applied at the source node. However, if missing data extents are detected at the destination, the destination has an extent pulling problem as the destination may not have knowledge of the physical layout on the source-side and/or mechanisms for requesting extents. The techniques overcome the extent pulling problem in a source-driven replication system by introducing specific protocols for obtaining missing extents within an existing replication environment by piggybacking data pushes from the source.

Abstract: A technique to name data is disclosed to allow preservation of storage efficiency over a link between a source and a destination in a replication relationship as well as in storage at the destination. The technique allows the source to send named data to the destination once and refer to it by name multiple times in the future, without having to resend the data. The technique also allows the transmission of data blocks to be decoupled from the logical containers that refer to the data blocks. The technique creates a name-to-data mapping at the destination, which maps source data block names to destination data block addresses. An invalidation of the data block namespace can be detected, in which case appropriate steps can be taken to rebuild the name-to-data mapping.

Abstract: The disclosed techniques enable push-based piggybacking of a source-driven logical replication system. Logical replication of a data set (e.g., a snapshot) from a source node to a destination node can be achieved from a source-driven system while preserving the effects of storage efficiency operations (deduplication) applied at the source node. However, if missing data extents are detected at the destination, the destination has an extent pulling problem as the destination may not have knowledge of the physical layout on the source-side and/or mechanisms for requesting extents. The techniques overcome the extent pulling problem in a source-driven replication system by introducing specific protocols for obtaining missing extents within an existing replication environment by piggybacking data pushes from the source.

Abstract: According to an embodiment of the invention, an agent scans a storage server, and collects information about files stored on the storage server. The agent then summarizes the information into a summary, and stores the summary on a database server. The summaries can be accessed by a system administrator using a graphical user interface (GUI). The summaries contain information about the use of the storage server, and highlight potential problem areas. By having an agent, rather than a multi-appliance management application (MMA) scan the storage server and create the summary, the MMA is able to perform other tasks.

Abstract: The disclosed techniques enable push-based piggybacking of a source-driven logical replication system. Logical replication of a data set (e.g., a snapshot) from a source node to a destination node can be achieved from a source-driven system while preserving the effects of storage efficiency operations (deduplication) applied at the source node. However, if missing data extents are detected at the destination, the destination has an extent pulling problem as the destination may not have knowledge of the physical layout on the source-side and/or mechanisms for requesting extents. The techniques overcome the extent pulling problem in a source-driven replication system by introducing specific protocols for obtaining missing extents within an existing replication environment by piggybacking data pushes from the source.

Abstract: A technique to name data is disclosed to allow preservation of storage efficiency over a link between a source and a destination in a replication relationship as well as in storage at the destination. The technique allows the source to send named data to the destination once and refer to it by name multiple times in the future, without having to resend the data. The technique also allows the transmission of data extents to be decoupled from the logical containers that refer to the data extents. Additionally, the technique allows a replication system to accommodate different extent sizes between replication source and destination while preserving storage efficiency.

Abstract: A system and method are described for storing and accessing data using a plurality of probabilistic data structures. In one embodiment, a plurality of probabilistic data structures are identified, each probabilistic data structure being associated with at least one other probabilistic data structure. The plurality of probabilistic data structures each correspond to a portion of a storage device storing a dataset. The dataset may include a plurality of values, such as a plurality of data blocks. A plurality of keys may be generated from the plurality of values, such as a plurality of fingerprints. Each key may be inserted into the probabilistic data structures which correspond to the portion of the storage device storing the value the key was generated from. The plurality of probabilistic data structures are stored in a memory and may be queried with an item to determine if the item exists in the plurality of probabilistic data structures.

Abstract: A system and method are described for providing variable length deduplication on a fixed block file system. In one embodiment, data comprising a plurality of data items is received. A determination is made if a segment of the data items exists on a storage volume. If the segment of data items does not exist on the storage volume, the segment of data items is stored on the storage volume. If the segment of data items exists on the storage volume, the segment of data items is deduplicated with the storage volume. A logical identifier describing a location of the segment in the data stream and a physical identifier describing a physical location of the segment are determined. The logical identifier, the physical identifier and the length of the segment is stored in a data structure such that the segment of data items can be accessed using the logical identifier.

Abstract: According to an embodiment of the invention, a filer or other storage server is coupled to a network to store files for users of the network. Agents are coupled to the filer, and performs a scan or file walk for a Multi-Appliance Management Application (MMA) which is coupled to the filer and can monitor and manage the filer. A directory structure of the filer may be divided into two or more paths, each of which may be scanned by an independent agent. The results of the scan, or file walk, are saved to a database server. In this way, multiple agents may be used to scan a single storage server, and the amount of time required for the scan is reduced.

Abstract: A storage system includes a storage operating system configured to provide two-step data deduplication in which, during a data storage operation and prior to committing data to a storage medium of the storage system, context-aware data deduplication is performed on the data to obtain context-aware deduplicated data, and, subsequent to committing the context-aware deduplicated data to the storage medium, a context-independent data deduplication operation is performed on the context-aware deduplicated data to obtain context-independent deduplicated context-aware deduplicated data, for commitment to the storage medium.

Abstract: According to an embodiment of the present invention, a filer or other storage server is coupled to a network to store files for users of the network. An agent is coupled to the filer, and performs a scan or file walk for a Multi-Appliance Management Application (MMA) which is coupled to the filer and can monitor and manage the filer. The agent assigns identification (ID) numbers to the directories while scanning them. The ID numbers are assigned in a depth first search (DFS) order to reduce the amount of resources required for specific queries that may later be required. Several types of queries, including determining the parent of a node, determining all of the children of a node, determining the immediate children of a node, and determining all of the ancestors of a node may be easily accomplished using the ID numbers.

Abstract: According to an embodiment of the invention, a directory on a storage server is scanned. A number of children of the directory is determined, and the number is added to a reference count. A child is scanned to collect information about the child, and the information is combined into a summary of the directory. After the child is scanned, the reference count is reduced by one. According to another embodiment of the invention, the storage server is scanned by an agent separate from the storage server.

Abstract: According to an embodiment of the invention, an agent scans a storage server, and collects information about files stored on the storage server. The agent then summarizes the information into a summary, and stores the summary on a database server. The summaries can be accessed by a system administrator using a graphical user interface (GUI). The summaries contain information about the use of the storage server, and highlight potential problem areas. By having an agent, rather than a multi-appliance management application (MMA) scan the storage server and create the summary, the MMA is able to perform other tasks.