Staggered writing for data storage systems
The present invention relates to write operations in data storage systems. In an embodiment, the invention relates to a method of writing data across a plurality of disk arrays, including writing a first copy of the data as a first plurality of stripes corresponding to the plurality of disk arrays and writing a second copy of the data as a second plurality of stripes corresponding to the plurality of disk arrays, wherein the second copy of the data is staggered with respect to the first copy on the plurality of disk arrays. In another embodiment, the invention relates to a data storage system including a plurality of disk arrays and a host, coupled to the plurality of disk arrays, with a program that instructs the host to write a first copy of the data as a first plurality of stripes that correspond to the plurality of disk arrays and to write a second copy of the data as a second plurality of stripes that correspond to the plurality of disk arrays, wherein the second copy of the data is staggered with respect to the first copy on the plurality of disk arrays.
Latest Pillar Data Systems, Inc. Patents:
- Systems and methods of searching for and determining modified blocks in a file system
- Methods of determining and searching for modified blocks in a file system
- Storage device with instructions for performing methods of processing I/O requests
- Systems and methods of asynchronous data replication
- Snapshots of file systems in data storage systems
The present invention relates to write operations in data storage systems. This application incorporates herein by reference as follows:
U.S. application Ser. No. 10/677,660, Systems and Methods of Multiple Access Paths to Single Ported Storage Devices, filed on Oct. 1, 2003
U.S. application Ser. No. 10/696,327, Data Replication in Data Storage Systems, filed on Oct. 28, 2003, now U.S. Patent No. 7,143,122 B2 and
U.S. application Ser. No. 10/837,322, Guided Configuration of Data Storage Systems, filed on Apr. 30, 2004, now U.S. Patent No. 7,216,192 B2.
BACKGROUNDToday, many data storage systems use redundant array of inexpensive disks (RAID) technology. Chen et al., RAID: High-Performance, Reliable Secondary Storage (1994) gives an overview of RAID levels 0-6 and is incorporated herein by reference.
In RAID 0, an array controller stripes data across the disk array and accesses data in parallel which increases I/O performance. RAID 0 has good write performance because it has no need to update redundant data, but has decreased reliability because there is no data redundancy. Loss of a single disk in the array results in data loss. Also, the data distributed on N identical disks of a disk array has 1/N the reliability of a single disk.
In RAID 1, when an array controller writes data to a disk, it also writes the data to a redundant disk. The array controller can read data from either disk, and if one disk fails, the array controller can read the data from the other disk increasing reliability but RAID 1 doubles the storage required so is expensive.
In RAID 5, an array controller writes the data and the parity over the disk array. The array controller stores the parity of the data as blocks associated with a set of data blocks when it writes data, and updates that parity data as data is changed by further write operations. If one disk fails, its data can be reconstructed by reading the remaining disks and the parity disk. This permits all of the disks to service a read request, but write requests tend to be less efficient, because they must read-modify-write the parity of the data.
SUMMARY OF THE INVENTIONThe present invention relates to write operations in data storage systems. In an embodiment, the invention relates to a method of writing data across a plurality of storage devices (e.g., disk arrays), comprising writing a first copy of the data as a first plurality of stripes corresponding to the plurality of disk arrays and writing a second copy of the data as a second plurality of stripes corresponding to the plurality of disk arrays, wherein the second copy of the data is staggered with respect to the first copy on the plurality of disk arrays. In another embodiment, the invention relates to a data storage system comprising a plurality of storage devices (e.g., disk arrays) and a host, coupled to the plurality of disk arrays, with a program instructing the host to write a first copy of the data as a first plurality of stripes that correspond to the plurality of disk arrays and to write a second copy of the data as a second plurality of stripes that correspond to the plurality of disk arrays, wherein the second copy of the data is staggered with respect to the first copy on the plurality of disk arrays. In either embodiment, the invention can be extended to N copies.
The following description includes the best mode of carrying out the invention, illustrates the principles of the invention, and should not be taken in a limiting sense. The scope of the invention is determined by reference to the claims. Each part or step is assigned its own number in the specification and drawings.
Each host runs an operating system such as Linux, UNIX, a Microsoft OS, or another suitable operating system. Tanenbaum, Modern Operating Systems (2001) describes operating systems in detail and is incorporated herein by reference.
Each host includes a bus adapter 22 between the CPU-memory bus 14 and an interface bus 24, which in turn interfaces with network adapters 17, 25 and 26. The first host 18 communicates through the network adapter 25 over a link 40 with a second host 19. The first host 18 can also communicate through the network adapter 17 over link 28 with the local area network (LAN) 30. The first host 18 also communicates through the network adapter 26 over a link 21 with a storage interconnect network 29. Similarly, the second host 19 communicates over links 38 and 39 with the LAN 30 and the storage interconnect network 29, respectively. The storage interconnect network 29 also communicates over links 32, 34, and 36 with the data storage subsystems 44, 46, and 48, respectively.
In sum, the hosts 18, 19 and 20 communicate with each other, the LAN 30 and storage interconnect network 29 and data storage subsystems 44, 46, and 48.
The LAN 30 and the storage interconnect network 29 can be separate networks as illustrated or combined in a single network, and may be any suitable known bus, SAN, LAN, or WAN technology such as Fibre Channel, SCSI, InfiniBand, or Ethernet, and the type of interconnect is not essential to the invention. See Kembel, The FibreChannel Consultant, A Comprehensive Introduction (1998), Kembel, The FibreChannel Consultant, Arbitrated Loop (1996-1997) The FibreChannel Consultant, Fibre Channel Switched Fabric (2001), Clark, Designing Storage Area Networks (2003), Clark, IP SANs: A Guide to iSCSI, iFCP, and FCIP Protocols for Storage Area Networks (2002) and Clark, Designing Storage Area Networks (1999), which are incorporated herein by reference.
U.S. application Ser. No.10/677,560, Systems and Methods of Multiple Access Paths to Single Ported Storage Devices, filed on Oct. 1, 2003 describes suitable data storage subsystems, each containing a disk array, and is incorporated by reference herein. In an alternative embodiment, any other suitable array of storage devices can replace the disk arrays (e.g. an array of tape drives or an array of nonvolatile semiconductor memory). Massiglia, The RAID Book: A Storage System Technology Handbook (6th Edition, 1997) describing RAID technology is incorporated herein by reference.
A host may access secondary storage devices (e.g., disk drives) through a VLUN (virtual logical unit) that abstracts the storage device(s) as a linear array of fixed-size blocks. A logical block address (LBA) identifies each fixed-sized block. The data storage system constructs a VLUN from all or parts of several physical storage devices such as disk drives. To make a large VLUN, a data storage system may concatenate space allocated from several storage devices. To improve performance, the data storage system maps adjacent regions of VLUN space onto different physical storage devices (striping). To improve reliability, the system holds multiple copies of a VLUN on different storage devices (mirroring).
Users request write and read operations of the data storage system 100. In operation, a user requests an I/O operation of one of the hosts 18,19, or 20 which will transmit a command (or I/O request) on the LAN 30 or the storage interconnect network 29 to one of the data storage subsystems 44, 46, or 48.
If a write is received, the data storage subsystem 44 can use a write-through scheme and not acknowledge the write until the data is written to nonvolatile memory (e.g., disk array 43). This ensures data consistency between the host and data storage subsystem.
In a write-back scheme, the data storage subsystem 44 can acknowledge the write before data is written to a disk array 43 as long as the data is stored in another form of nonvolatile memory (e.g., battery backed RAM) until written to the disk array to ensure data consistency in the event of a power failure, etc.
Each stripe is made of strips written on corresponding disks. For example, stripe A1 is written across the six disks of disk array 1. In RAID 5, the array controller writes stripe A1 made of strips A11-A16 on the disks 51-56. Strips A11-A15 contain the data and strip A16 the parity of strips A11-A15. In an embodiment, the stripe A1 is 768 KB and each strip is 128 KB. In RAID 5, if any disk in the disk array fails, the data can be recovered from the other disks.
If any disk array fails (e.g., two disks in the disk array fail or the array controller fails), the host can access the data from another disk array. To support fault tolerance, when writing data to the VLUN, the host writes the data to all of the disk arrays. When reading data from the VLUN, any copy can be used.
The host writes the first copy of A-C as stripe A1 to disk array 1, stripe B1 to disk array 2, and stripe C1 to disk array 3. Next, the host writes the second copy of A-C as stripe C2 to disk array 1, stripe A2 to disk array 2, and stripe B2 to disk array 3. The host staggers the second copy with respect to the first copy so that two copies of any given stripe (e.g., A, B, or C) do not reside on the same disk array. Thus, the host staggers the second copy of stripe A:A1 is stored on disk array 1, while A2 is stored on disk array 2. This staggering writing of the stripes completes the first stripe set, i.e., stripes A-C.
Subsequent sets of stripes can be staggered in the same pattern. For example, the host can write the second stripe set, i.e., stripes D-F by writing stripe D1 to disk array 1, stripe E1 to disk array 2, and stripe F1 to disk array 3, and staggering stripe F2 to disk array 1, stripe D2 to disk array 2, and stripe E2 to disk array 3. This again avoids storing two copies of a stripe (e.g., D, E, or F) on the same disk array. This is writing in round robin across the plurality of disk arrays. This striping supports the ability to write full stripes under RAID 5.
The extent lists contain information about the stripes and the space allocated on the disk arrays as detailed in
In
In the embodiment of
As in
Also as in
The disk arrays 1-3 of
The extent lists allow the host to write the stripes at noncontiguous locations on the disk arrays to overcome fragmentation. Each extent on the extent list contains the starting LBAs of each stripe on the disk array to show where the host has allocated space for the VLUN.
Tanenbaum, Modern Operating Systems (2001), incorporated herein by reference, describes techniques for tracking of free space on secondary storage devices such as hard disk drives on pages 413-414. In an embodiment, a bit map tracks the allocation status of the stripes on each disk array. The host will set a bit of the bit map to 1 when it allocates a corresponding stripe and set the bit to 0 when it deallocates the corresponding stripe. Another embodiment uses linked lists of disk blocks.
As shown in
The identifiers 1-12 are to help the reader find the stripes in the disk arrays rather than indicate the writes be done in a certain order (e.g., ascending, descending, or in sequential steps). The write operation can extend to three or more copies.
Each disk array has its own extent list. The extent list among other things indicates the range of blocks allocated for the VLUN. Each extent list includes extents such as extents 1-M. Each extent contains forward and backward pointers so the host can search and maintain the extent list. Each extent also includes the starting LBA and length of the extent to indicate the contiguous space allocated for the VLUN on the disk array. Finally, the disk array ID identifies the disk array containing the allocation. In an alternative embodiment, the extent list could be implemented by a different data structure such an array, a singly linked list, a tree, or a hash table. Cormen, Introduction to Algorithms (2nd Ed. 2001) describes the details of such data structures and is incorporated herein by reference.
The method begins at step 200. At step 201, the user will enter the value of N to define the number of copies of the block that will be written to the disk arrays. At step 202, the host assigns zero to a variable named Copy that increments each time a copy of the block is written to a disk array. At step 204, the host calculates the VLUN data index by dividing the VLUN block by the VLUN data size, both in units of blocks addressable as LBAs. The integer math result is the VLUN data index of the VLUN data A, B, C, D, E, or F where the VLUN block resides. As shown in
One embodiment of the method uses the extent lists as described in steps 213-214. In an alternative embodiment, the method uses the extent lists illustrated in
Claims
1. A method of writing data across a plurality of disk arrays in a data storage system, comprising:
- writing a first copy of the data as a first plurality of stripes corresponding to the plurality of disk arrays; and
- writing a second copy of the data as a second plurality of stripes corresponding to the plurality of disk arrays, wherein the second copy of the data is staggered with respect to the first copy of the data so that no two copies of any stripe are written to one of the plurality of disk arrays.
2. The method of claim 1, wherein the second copy is staggered from the first copy by one disk array.
3. The method of claim 1, further comprising the step of writing the data of a VLUN to the plurality of disk arrays.
4. The method of claim 1, further comprising the step of writing the data of a VLUN to noncontiguous free space on the plurality of disk arrays.
5. The method of claim 4, further comprising the step of describing the noncontiguous free space allocated on each of the plurality of disk arrays in an extent list.
6. The method of claim 5, wherein the extent list includes a plurality of extents, each extent including a starting LBA, a length of extent, an offset, and a disk array ID.
7. The method of claim 1, wherein the writing of the first and second copy of the data includes writing in round robin across the plurality of disk arrays.
8. The method of claim 1, wherein the writing of the data includes writing the data in fixed-size blocks.
9. The method of claim 1, further comprising implementing RAID 5 in the plurality of disk arrays.
10. The method of claim 9, further comprising writing the data from a VLUN as a multiple of the RAID 5 stripe length.
11. The method of claim 1, further comprising writing a third copy of the data as a third plurality of stripes that correspond to the plurality of disk arrays, wherein the third copy of the data is staggered with respect to the first copy and the second copy of the data on the plurality of disk arrays.
12. A date storage system comprising:
- a plurality of disk arrays in data storage subsystems; and
- a host, coupled to the plurality of disk arrays, programmed to instruct the host to write a first copy of the data as a first plurality of stripes that correspond to the plurality of disk arrays and to write a second copy of the data as a second plurality of stripes that correspond to the plurality of disk arrays, wherein the second copy of the data is staggered with respect to the first copy of the data on the plurality of disk arrays.
13. The data storage system of claim 12, further comprising a VLUN containing data accessible to the host.
14. The data storage system of claim 13, further comprising a VLUN geometry map accessible to the host to indicate the space allocated for the first and second copies of the data on the plurality of disk arrays.
15. The data storage system of claim 14, wherein the VLUN geometry map includes extent lists.
16. The data storage system of claim 15, wherein the extent list includes a plurality of extents, wherein each extent includes a starting LBA, a length of extent, an offset, and a disk array ID.
17. The data storage system of claim 12, wherein no two copies of any stripe are written to one of the plurality of disk arrays.
18. The data storage system of claim 12, wherein the plurality of disk arrays implement RAID 5.
19. The data storage system of claim 18, wherein the date includes RAID 5 stripes made up of multiple strips.
20. The data storage system of claim 12, wherein the host is programmed to write the first and second copy of the data in round robin across the plurality of disk arrays.
21. The data storage system of claim 12, wherein the host is programmed to write the first and second copy of the data in fixed-size blocks.
22. The data storage system of claim 12, wherein the host is further programmed to write a third copy of the data as a third plurality of stripes that correspond to the plurality of disk arrays, wherein the third copy of the data is staggered with respect to the first copy and the second copy of the data on the plurality of disk arrays.
23. The data storage system of claim 22, wherein the host is further programmed to write four or more copies of the data as stripes that correspond to the plurality of disk arrays, wherein each copy of the data is staggered with respect to the other copies of the data on the plurality of disk arrays.
5331646 | July 19, 1994 | Krueger et al. |
5459853 | October 17, 1995 | Best et al. |
5799174 | August 25, 1998 | Muntz et al. |
6067635 | May 23, 2000 | DeKoning et al. |
6076143 | June 13, 2000 | Blumenau |
6118790 | September 12, 2000 | Bolosky et al. |
6170037 | January 2, 2001 | Blumenau |
6240243 | May 29, 2001 | Chen et al. |
6332177 | December 18, 2001 | Humlicek |
6397292 | May 28, 2002 | Venkatesh et al. |
6425052 | July 23, 2002 | Hashemi |
6535869 | March 18, 2003 | Housel, III |
6907505 | June 14, 2005 | Cochran et al. |
20020073276 | June 13, 2002 | Howard et al. |
20030101321 | May 29, 2003 | Ohran |
20030172149 | September 11, 2003 | Edsall et al. |
20030188097 | October 2, 2003 | Holland et al. |
- Jim Senicka, Brian Duffy, Veritas-vx, Stupid Striping Question, Dec. 28, 2000, pp. 1-2, Veritas-vx maillist, US.
- Peter M. Chen, Edward K. Lee, Garth A. Gibson, Randy H. Katz, David A. Patterson, Raid: HIgh-Performance, Reliable Secondary Storage, Jun. 1994, pp. 145-185. ACM Computing Surveys. vol. 26. No. 2. US.
Type: Grant
Filed: Oct 27, 2004
Date of Patent: May 27, 2008
Patent Publication Number: 20060087940
Assignee: Pillar Data Systems, Inc. (San Jose, CA)
Inventors: Michael Allan Brewer (Boulder, CO), Robert Arthur Ellis (Nederland, CO)
Primary Examiner: Dieu-Minh Le
Attorney: Robert Moll
Application Number: 10/975,290
International Classification: G06F 11/00 (20060101);