AUTOMATED CLOUD RECOVERY TO PROVIDE A FULL USABLE APPLICATION IMAGE
A storage array creates snapshots of each of a plurality of devices of a storage group associated with a production device on which active application data is logically stored. Metadata that indicates associations between the snapshots and the devices is stored on cloud storage with the snapshots as a snapset object. A program running on a management station used the snapset metadata to automatically create new devices on which to recover the snapshots on a selected storage array and uses the snapset metadata to cause the snapshots to be automatically written from the cloud storage system to the new devices on the selected storage array.
Latest EMC IP HOLDING COMPANY LLC Patents:
- Persistent memory tiering supporting fast failover in a deduplicated file system
- System and method for storage optimization using machine learning- driven application and workload identification
- Generating and morphing a collection of databases that collectively has desired dedupability, compression, clustering and commonality
- Edge function bursting
- System and method for a content-aware and context-aware compression algorithm selection model for a file system
The subject matter of this disclosure is generally related to electronic data storage and more particularly to recovery of snapped application images.
BACKGROUNDHigh capacity data storage systems such as storage area networks (SANs) are used to maintain large data sets and contemporaneously support multiple users. A storage array, which is an example of a SAN, includes a network of interconnected compute nodes that manage access to data stored on arrays of drives. The data is typically used by “host applications” that run on servers known as “hosts.” The drives are not discoverable by the hosts, so the compute nodes create a logical production volume that is discoverable by the hosts. Examples of host applications may include, but are not limited to, databases and other software for email, accounting, manufacturing, inventory control, and a wide variety of other business processes. Separate logical production volumes may be created for each host application supported by the storage array.
A variety of techniques may be implemented by the storage system to avoid data loss, maintain data availability, and protect against data corruption. One such technique is creation of point-in-time copies of a data set. Creating a complete copy of a large data set requires a significant amount of time and resources so it is common practice to create smaller incremental updates known as snapshots or “snaps.” Each snap only represents the changes made to the data set since some prior point in time, e.g. and without limitation since creation of the previous snap. Consequently, snaps take less time and resources to generate than complete copies. Snaps enable recreation of a prior point in time state of the data set.
SUMMARYIn accordance with some aspects a method for creating and recovering an application image comprises creating, with a storage array, a snapshot of each of a plurality of devices of a storage group associated with a production device on which active application data is logically stored, thereby creating a plurality of the snapshots, each of the snapshots representing device state at a same point in time, creating metadata that indicates associations between ones of the snapshots and ones of the devices on which the active application data is stored, and creating a snapset comprising all the snapshots and the metadata, storing the snapset on a cloud storage system, and with a third network node using the snapset metadata to automatically create new devices on which to recover the snapshots on a selected storage array and using the snapset metadata to cause the snapshots to be automatically written from the cloud storage system to the new devices on the selected storage array.
In accordance with some aspects a storage system comprises: a storage array configured to create a snapshot of each of a plurality of devices of a storage group associated with a production device on which active application data is logically stored, thereby creating a plurality of the snapshots, each of the snapshots representing device state at a same point in time, the storage array further configured to create metadata that indicates associations between ones of the snapshots and ones of the devices on which the active application data is stored and create a snapset comprising all the snapshots and the metadata; a cloud storage system configured to store the snapset; and a third network node configured to use the snapset metadata to automatically create new devices on which to recover the snapshots on a selected storage array and use the snapset metadata to cause the snapshots to be automatically written from the cloud storage system to the new devices on the selected storage array.
In accordance with some implementations a computer-readable storage medium stores instructions that when executed by a computer cause the computer to perform a method for using a computer system to create and recover an application image, the method comprising, with a storage array: creating a snapshot of each of a plurality of devices of a storage group associated with a production device on which active application data is logically stored, thereby creating a plurality of the snapshots, each of the snapshots representing device state at a same point in time; creating metadata that indicates associations between ones of the snapshots and ones of the devices on which the active application data is stored; and creating a snapset comprising all the snapshots and the metadata; with a cloud storage system: storing the snapset; with a third network node: using the snapset metadata to automatically create new devices on which to recover the snapshots on a selected storage array; and using the snapset metadata to cause the snapshots to be automatically written from the cloud storage system to the new devices on the selected storage array.
All examples, aspects and features mentioned in this document can be combined in any technically possible way. Other aspects, features, and implementations may become apparent in view of the detailed description and figures.
The terminology used in this disclosure is intended to be interpreted broadly within the limits of subject matter eligibility. The terms “disk” and “drive” are used interchangeably herein and are not intended to refer to any specific type of non-volatile storage media. The terms “logical” and “virtual” are used to refer to features that are abstractions of other features, e.g. and without limitation abstractions of tangible features. The term “physical” is used to refer to tangible features that possibly include, but are not limited to, electronic hardware. For example, multiple virtual computers could operate simultaneously on one physical computer. The term “logic” is used to refer to special purpose physical circuit elements, firmware, software, computer instructions that are stored on a non-transitory computer-readable medium and implemented by multi-purpose tangible processors, and any combinations thereof. Aspects of the inventive concepts are described as being implemented in a data storage system that includes host servers and a storage array. Such implementations should not be viewed as limiting. Those of ordinary skill in the art will recognize that there are a wide variety of implementations of the inventive concepts in view of the teachings of the present disclosure.
Some aspects, features, and implementations described herein may include machines such as computers, electronic components, optical components, and processes such as computer-implemented procedures and steps. It will be apparent to those of ordinary skill in the art that the computer-implemented procedures and steps may be stored as computer-executable instructions on a non-transitory computer-readable medium. Furthermore, it will be understood by those of ordinary skill in the art that the computer-executable instructions may be executed on a variety of tangible processor devices, i.e. physical hardware. For practical reasons, not every step, device, and component that may be part of a computer or data storage system is described herein. Those of ordinary skill in the art will recognize such steps, devices, and components in view of the teachings of the present disclosure and the knowledge generally available to those of ordinary skill in the art. The corresponding machines and processes are therefore enabled and within the scope of the disclosure.
Cloud storage 120 and the storage array 100 are distinct types of storage systems. Cloud storage exhibits greater data access latency than the storage array and may be unsuitable for active data sets. Cloud storage is used to reduce per-bit storage costs in situations where high-performance capabilities are not required, e.g. data backup and storage of inactive or infrequently accessed data. The cloud storage gateway 122 enables the storage array 100 to use the data storage resources of cloud storage 120 by functioning as an intermediary device. More particularly, the cloud storage gateway converts IOs between a form used by the storage array and a form used by storage servers of the cloud storage. Cloud storage and the storage array are configured to utilize different protocols for IOs. For example, and without limitation, the storage array may utilize a transport layer protocol such as Fibre Channel, iSCSI (internet small computer system interface) or NAS (Network-Attached Storage) protocols such as NFS (Network File System), SMB (Server Message Block), CIFS (Common Internet File System) and AFP (Apple Filing Protocol). In contrast, the cloud storage may utilize any of a variety of different non-standard and provider-specific APIs (Application Programming Interfaces) such as AWS (Amazon Web Services), Dropbox, OpenStack, Google Drive/Storage APIs based on, e.g., JSON (JavaScript Object Notation). In response to receipt of an iSCSI format IO command from the storage array, the cloud storage gateway converts that IO into a format, for example and without limitation OpenStack, that can be processed by cloud storage, thereby providing a corresponding OpenStack IO. The cloud storage gateway can also convert messages from the cloud storage format to the host/storage array format.
Referring to
A cloud tethering system (CTS) 250 running on the storage array 100 automatically generates snapsets with metadata in accordance with predefined rules. The rules collectively form a cloud protection policy that may indicate, for example, and without limitation, frequency, retention period, and destination of snapsets for a specific storage group. An example of a cloud protection policy could be frequency=30 days, retention=1 year, cloud repository=Amazon_S3_Object_Store. In accordance with the example cloud protection policy a new snapset is taken every 30 days and retained in the cloud provider Amazon_S3_Object_Store for a year.
An application programming interface (API) 252 running on the storage array 100 enables the management station 130 to control the storage array. For example, the API 252 may be responsive to representational state transfer (REST) operations to generate commands that cause the compute nodes to perform various operations. As will be described in greater detail below, the API 252 enables the snapset recovery program 128 to cause the storage array to perform operations associated with automated snapset recovery.
Referring to
Although advantages should not be viewed as limiting the invention, the described snapset object, metadata and automated snapset recovery technique can be accomplished in less time and with less chance of error than prior techniques. Recovering application images from heterogenous cloud repositories has previously been a multi-step process requiring numerous error-prone admin inputs. Typical steps include tracking of multiple volumes and corresponding snaps associated with an application image, tracking the snapshot image that has the mount point, tracking the number of volumes in the application image, tracking the size of the all the volumes in the application image and creating the exact sizes of target volumes, provisioning storage for the space required for recovery, pairing a volume to a corresponding snapshot image, generating a recovery job for every snapshot on a volume by volume basis, and tracking the progress of all the recovery jobs. The complexity and likelihood of error increases as the number of snapped volumes associated with the application image increases. Further, volumes for which recovery has completed are without practical use until the entire application image is recovered.
The described snapset object, metadata and automated snapset recovery technique is also advantageously node independent. In other words, an application image can be recovered to any storage array because the snapset metadata provides the information required by the snapset recovery program to manage recovery. The technique is not reliant on the snapset metadata being maintained in the original storage array and the application image is not limited to being recovered on the original storage array.
Specific examples have been presented to provide context and convey inventive concepts. The specific examples are not to be considered as limiting. A wide variety of modifications may be made without departing from the scope of the inventive concepts described herein. Moreover, the features, aspects, and implementations described herein may be combined in any technically possible way. Accordingly, modifications and combinations are within the scope of the following claims.
Claims
1. A method for creating and recovering an application image, comprising:
- with a storage array: creating a snapshot of each of a plurality of devices of a storage group associated with a production device on which active application data is logically stored, thereby creating a plurality of the snapshots, each of the snapshots representing device state at a same point in time; creating metadata that indicates associations between ones of the snapshots and ones of the devices on which the active application data is stored; and creating a snapset comprising all the snapshots and the metadata;
- with a cloud storage system: storing the snapset;
- with a third network node: using the snapset metadata to automatically create new devices on which to recover the snapshots on a selected storage array; and using the snapset metadata to cause the snapshots to be automatically written from the cloud storage system to the new devices on the selected storage array.
2. The method of claim 1 wherein creating metadata comprises including storage array ID, storage group UUID, snapshot sizes, snapshot names, snapshot IDs, snapset ID, volume WWNs, and a timestamp.
3. The method of claim 1 wherein creating metadata comprises including a storage group nice name, user provider mount point marker, and original source volume number.
4. The method of claim 1 comprising the third network node sending a command to the cloud storage system to get a catalog of all cloud snapsets stored on the cloud storage system.
5. The method of claim 1 comprising the third network node automatically sending a command to the selected storage array to create a new storage group for the new devices.
6. The method of claim 5 comprising the third network node automatically sending a command to the selected storage array to create the new devices as protected recovery devices that equal in number the devices of the storage group associated with the production device on which active application data is logically stored.
7. The method of claim 6 comprising the third network node automatically sending a command to the selected storage array to unprotect the new devices after the snapshots are written from the cloud storage system to the new devices on the selected storage array.
8. A storage system, comprising:
- a storage array configured to create a snapshot of each of a plurality of devices of a storage group associated with a production device on which active application data is logically stored, thereby creating a plurality of the snapshots, each of the snapshots representing device state at a same point in time, the storage array further configured to create metadata that indicates associations between ones of the snapshots and ones of the devices on which the active application data is stored and create a snapset comprising all the snapshots and the metadata;
- a cloud storage system configured to store the snapset; and
- a third network node configured to use the snapset metadata to automatically create new devices on which to recover the snapshots on a selected storage array and use the snapset metadata to cause the snapshots to be automatically written from the cloud storage system to the new devices on the selected storage array.
9. The storage system of claim 8 wherein the metadata comprises storage array ID, storage group UUID, snapshot sizes, snapshot names, snapshot IDs, snapset ID, volume WWNs, and a timestamp.
10. The storage system of claim 8 wherein the metadata comprises a storage group nice name, user provider mount point marker, and original source volume number.
11. The storage system of claim 8 wherein the third network node is configured to send a command to the cloud storage system to get a catalog of all cloud snapsets stored on the cloud storage system.
12. The storage system of claim 8 wherein the third network node is configured to automatically send a command to the selected storage array to create a new storage group for the new devices.
13. The storage system of claim 12 wherein the third network node is configured to automatically send a command to the selected storage array to create the new devices as protected recovery devices that equal in number the devices of the storage group associated with the production device on which active application data is logically stored.
14. The storage system of claim 13 wherein the third network node is configured to automatically send a command to the selected storage array to unprotect the new devices after the snapshots are written from the cloud storage system to the new devices on the selected storage array.
15. A computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for using a computer system to create and recover an application image, the method comprising:
- with a storage array: creating a snapshot of each of a plurality of devices of a storage group associated with a production device on which active application data is logically stored, thereby creating a plurality of the snapshots, each of the snapshots representing device state at a same point in time; creating metadata that indicates associations between ones of the snapshots and ones of the devices on which the active application data is stored; and creating a snapset comprising all the snapshots and the metadata;
- with a cloud storage system: storing the snapset;
- with a third network node: using the snapset metadata to automatically create new devices on which to recover the snapshots on a selected storage array; and using the snapset metadata to cause the snapshots to be automatically written from the cloud storage system to the new devices on the selected storage array.
16. The computer-readable storage medium of claim 15 wherein creating metadata comprises including storage array ID, storage group UUID, snapshot sizes, snapshot names, snapshot IDs, snapset ID, volume WWNs, and a timestamp.
17. The computer-readable storage medium of claim 15 wherein creating metadata comprises including a storage group nice name, user provider mount point marker, and original source volume number.
18. The computer-readable storage medium of claim 15 comprising the third network node sending a command to the cloud storage system to get a catalog of all cloud snapsets stored on the cloud storage system.
19. The computer-readable storage medium of claim 15 comprising the third network node automatically sending a command to the selected storage array to create a new storage group for the new devices.
20. The computer-readable storage medium of claim 19 comprising the third network node automatically sending a command to the selected storage array to create the new devices as protected recovery devices that equal in number the devices of the storage group associated with the production device on which active application data is logically stored.
Type: Application
Filed: Oct 30, 2020
Publication Date: May 5, 2022
Applicant: EMC IP HOLDING COMPANY LLC (Hopkinton, MA)
Inventors: Francisco Aquino (Marlborough, MA), Kenneth Byrne (Knockraha), Warren Fleury (Ballincollig), Thiago Santos (Cork City), Deepak Vokaliga (Hopkinton, MA)
Application Number: 17/084,691