STORAGE SYSTEM CONFIGURED TO GUARANTEE SUFFICIENT CAPACITY FOR A DISTRIBUTED RAID REBUILD PROCESS
A storage system comprises a plurality of storage devices, and is configured to establish a redundant array of independent disks (RAID) arrangement comprising a plurality of stripes, with each of the plurality of stripes comprising a plurality of blocks, the blocks being distributed across multiple ones of the storage devices. In conjunction with establishment of the RAID arrangement, the storage system is further configured, for each of the plurality of stripes, to designate multiple ones of the storage devices as respective spare devices for that stripe, and for each of the storage devices, to determine numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices in each of multiple spare levels and for each of multiple failure combinations. A particular number of spare blocks is reserved for each of the storage devices using the determined numbers.
The present application is a continuation-in-part of U.S. patent application Ser. No. 16/852,632, filed Apr. 20, 2020 and entitled “Storage System Configured to Guarantee Sufficient Capacity for a Distributed Raid Rebuild Process,” which is incorporated by reference herein in its entirety.
FIELDThe field relates generally to information processing systems, and more particularly to storage in information processing systems.
BACKGROUNDIn many storage systems, data is distributed across multiple storage devices in accordance with redundant array of independent disks (RAID) arrangements. Some RAID arrangements allow a certain amount of lost data to be rebuilt using parity information, typically in response to a storage device failure or other type of failure within the storage system. For example, a RAID 6 arrangement uses “dual parity” and can recover from simultaneous failure of two storage devices of the storage system. These and other RAID arrangements provide redundancy for stored data, with different types of RAID arrangements providing different levels of redundancy. Storage systems that utilize such RAID arrangements are typically configured to perform a rebuild process after detection of a storage device failure, and once the rebuild process is completed, the storage system can sustain additional failures. Conventional RAID techniques of this type can be problematic in certain situations.
For example, in distributed RAID arrangements, in which the total number of blocks per stripe is less than the total number of storage devices across which the blocks are distributed, it is unduly difficult under conventional practice to ensure that there is sufficient available capacity to rebuild the blocks of one or more failed storage devices. Conventional approaches typically require the performance of complex calculations in order to determine a particular allocation of spare capacity across the multiple storage devices, or instead allocate excessive amounts of spare capacity to each device, which wastes storage device capacity and therefore degrades storage system performance.
A need therefore exists for an improved approach that can guarantee sufficient rebuild capacity without the need for complex calculations or wasted capacity, and yet is readily applicable to a wide variety of different RAID arrangements.
SUMMARYIllustrative embodiments provide techniques for guaranteeing sufficient capacity for a distributed RAID rebuild process in a storage system. For example, some embodiments provide RAID arrangements in which spare blocks are reserved for storage devices in a manner that ensures that a rebuild process initiated responsive to a failure of one or more of the storage devices is guaranteed to have sufficient available storage device capacity to complete the rebuilding of the blocks of the one or more failed storage devices. Reserved capacity in some embodiments is shared at the storage device level, such that, for example, spare blocks reserved on one storage device can be used to rebuild blocks of different ones of the other storage devices in the event that one of those other storage devices were to fail.
Advantageously, such techniques in illustrative embodiments herein can guarantee sufficient capacity to complete the rebuild process, while also being computationally simple, as well as highly efficient in that only at or near minimum amounts of capacity need to be reserved for rebuild within the storage system.
Moreover, the disclosed techniques in illustrative embodiments overcome additional drawbacks of conventional approaches. For example, some embodiments can be used for any type of RAID arrangement, including those involving combinations of multiple instances of distinct RAID approaches, such as a mixture of multiple distinct RAID types (e.g., RAID 1 and RAID 6) over the same set of storage devices, or a mixture of multiple stripe sets of different instances of one RAID type (e.g., two separate instances of RAID 5) over the same set of storage devices. Illustrative embodiments disclosed herein provide efficient mechanisms to guarantee sufficient capacity in these and other situations in which allocation of blocks to stripes cannot be predetermined, and in numerous other demanding RAID arrangements, including those involving parity RAID techniques and/or non-parity RAID techniques.
In one embodiment, a storage system comprises a plurality of storage devices. The storage system is further configured to establish a RAID arrangement comprising a plurality of stripes, with each of the plurality of stripes comprising a plurality of blocks, the blocks being distributed across multiple ones of the storage devices. In conjunction with establishment of the RAID arrangement, the storage system is further configured, for each of the plurality of stripes, to designate multiple ones of the storage devices as respective spare devices for that stripe, and for each of the storage devices, to determine numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices in each of multiple spare levels and for each of multiple failure combinations. A particular number of spare blocks is reserved for each of the storage devices based at least in part on the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices across the multiple spare levels and the multiple failure combinations.
In some embodiments, the RAID arrangement comprises a distributed RAID arrangement in which a total number of blocks per stripe is less than a total number of the storage devices over which the blocks of the plurality of stripes are distributed.
For example, the RAID arrangement can comprise at least one parity RAID arrangement supporting recovery from a failure of at least one of the plurality of storage devices, such as a RAID 5 arrangement supporting recovery from a failure of a single one of the plurality of storage devices, or a RAID 6 arrangement supporting recovery from simultaneous failure of up to two of the storage devices. In these and other parity RAID arrangements, each of the stripes illustratively comprises a plurality of data blocks and one or more parity blocks. Various combinations of parity RAID and/or non-parity RAID can also be used.
The storage system is illustratively configured to store stripe metadata indicating for each of the stripes the multiple ones of the storage devices that are designated as respective spare devices for that stripe. For example, the stripe metadata indicating for each of the stripes the multiple ones of the storage devices that are designated as respective spare devices for that stripe illustratively comprises an allocation table, the allocation table comprising a plurality of entries for respective ones of the stripes with each such entry comprising identifiers of the multiple storage devices that are designated as respective spare devices for that stripe.
Additionally or alternatively, the storage system is illustratively configured to store stripe metadata indicating for each of the storage devices the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices in each of multiple spare levels and for each of multiple failure combinations. For example, the stripe metadata indicating for each of the storage devices the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices in each of multiple spare levels and for each of multiple failure combinations illustratively comprises a reserve table, the reserve table comprising a plurality of entries for respective ones of the storage devices with each such entry comprising the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices in each of multiple spare levels and for each of multiple failure combinations.
Numerous other types and arrangements of one or more tables or other stripe metadata can be used in other embodiments.
In some embodiments, reserving a particular number of spare blocks for each of the storage devices based at least in part on the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices across the multiple spare levels and the multiple failure combinations comprises summing the determined numbers for the storage device across the spare levels to generate resulting sums for respective ones of the failure conditions, determining a highest one of the resulting sums across the respective ones of the failure conditions, and reserving a particular number of spare blocks for the storage device that is greater than or equal to the highest one of the resulting sums. Other techniques can be used in other embodiments to determine the particular number of spare blocks for a given storage device based at least in part on the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices across the multiple spare levels and the multiple failure combinations.
In some embodiments, designating multiple ones of the storage devices as respective spare devices for a given one of the stripes comprises designating at least first and second storage devices as respective first and second spare devices for the given stripe, with the first and second storage devices corresponding to respective first and second spare levels of the multiple spare levels. A first failure impacting the given stripe results in utilization of a spare block of the first spare device and a second failure impacting the given stripe results in utilization of a spare block of the second spare device.
The failure combinations in some embodiments comprise respective different sets of two or more of the storage devices.
In some embodiments, different ones of the storage devices are implemented on different ones of a plurality of storage nodes of a distributed storage system. In such an arrangement, the failure combinations may comprise, for example, different combinations of one of the storage devices and one of the nodes.
In some embodiments, the storage system is further configured to detect a failure of at least one of the storage devices, and responsive to the detected failure, to initiate a rebuild process to reconstruct blocks of the one or more failed storage devices utilizing the blocks of other ones of the storage devices. The rebuild process illustratively utilizes at least a subset of the reserved spare blocks of respective ones of the non-failed storage devices.
The reserved spare blocks in illustrative embodiments are sufficient in number to guarantee that a rebuild process initiated responsive to a failure of one or more of the storage devices has sufficient available storage device capacity to reconstruct the blocks of the one or more failed storage devices.
The storage system in some embodiments is implemented as a distributed storage system comprising a plurality of storage nodes, each storing data in accordance with a designated RAID arrangement, although it is to be appreciated that a wide variety of other types of storage systems can be used in other embodiments.
These and other illustrative embodiments include, without limitation, apparatus, systems, methods and processor-readable storage media.
Illustrative embodiments will be described herein with reference to exemplary information processing systems and associated computers, servers, storage devices and other processing devices. It is to be appreciated, however, that these and other embodiments are not restricted to the particular illustrative system and device configurations shown. Accordingly, the term “information processing system” as used herein is intended to be broadly construed, so as to encompass, for example, processing systems comprising cloud computing and storage systems, as well as other types of processing systems comprising various combinations of physical and virtual processing resources. An information processing system may therefore comprise, for example, at least one data center or other cloud-based system that includes one or more clouds hosting multiple tenants that share cloud resources. Numerous different types of enterprise computing and storage systems are also encompassed by the term “information processing system” as that term is broadly used herein.
The host devices 101 illustratively comprise servers or other types of computers of an enterprise computer system, cloud-based computer system or other arrangement of multiple compute nodes associated with one or more users.
For example, the host devices 101 in some embodiments illustratively provide compute services such as execution of one or more applications on behalf of each of one or more users associated with respective ones of the host devices. Such applications illustratively generate input-output (IO) operations that are processed by the storage system 102. The term “input-output” as used herein refers to at least one of input and output. For example, IO operations may comprise write requests and/or read requests directed to logical addresses of one or more logical storage volumes of the storage system 102. These and other types of IO operations are also generally referred to herein as IO requests.
The storage system 102 illustratively comprises processing devices of one or more processing platforms. For example, the storage system 102 can comprise one or more processing devices each having a processor and a memory, possibly implementing virtual machines and/or containers, although numerous other configurations are possible.
The storage system 102 can additionally or alternatively be part of cloud infrastructure such as an Amazon Web Services (AWS) system. Other examples of cloud-based systems that can be used to provide at least portions of the storage system 102 include Google Cloud Platform (GCP) and Microsoft Azure.
The host devices 101 and the storage system 102 may be implemented on a common processing platform, or on separate processing platforms. The host devices 101 are illustratively configured to write data to and read data from the storage system 102 in accordance with applications executing on those host devices for system users.
The term “user” herein is intended to be broadly construed so as to encompass numerous arrangements of human, hardware, software or firmware entities, as well as combinations of such entities. Compute and/or storage services may be provided for users under a Platform-as-a-Service (PaaS) model, an Infrastructure-as-a-Service (IaaS) model and/or a Function-as-a-Service (FaaS) model, although it is to be appreciated that numerous other cloud infrastructure arrangements could be used. Also, illustrative embodiments can be implemented outside of the cloud infrastructure context, as in the case of a stand-alone computing and storage system implemented within a given enterprise.
The network 104 is assumed to comprise a portion of a global computer network such as the Internet, although other types of networks can be part of the network 104, including a wide area network (WAN), a local area network (LAN), a satellite network, a telephone or cable network, a cellular network, a wireless network such as a WiFi or WiMAX network, or various portions or combinations of these and other types of networks. The network 104 in some embodiments therefore comprises combinations of multiple different types of networks each comprising processing devices configured to communicate using Internet Protocol (IP) or other communication protocols.
As a more particular example, some embodiments may utilize one or more high-speed local networks in which associated processing devices communicate with one another utilizing Peripheral Component Interconnect express (PCIe) cards of those devices, and networking protocols such as InfiniBand, Gigabit Ethernet or Fibre Channel. Numerous alternative networking arrangements are possible in a given embodiment, as will be appreciated by those skilled in the art.
The storage system 102 comprises a plurality of storage devices 106 configured to store data of a plurality of storage volumes. The storage volumes illustratively comprise respective logical units (LUNs) or other types of logical storage volumes. The term “storage volume” as used herein is intended to be broadly construed, and should not be viewed as being limited to any particular format or configuration.
References to “disks” in this embodiment and others disclosed herein are intended to be broadly construed, and are not limited to hard disk drives (HDDs) or other rotational media. For example, at least portions of the storage devices 106 illustratively comprise solid state drives (SSDs). Such SSDs are implemented using non-volatile memory (NVM) devices such as flash memory. Other types of NVM devices that can be used to implement at least a portion of the storage devices 106 include non-volatile random access memory (NVRAM), phase-change RAM (PC-RAM), magnetic RAM (MRAM), resistive RAM, spin torque transfer magneto-resistive RAM (STT-MRAM), and Intel Optane™ devices based on 3D XPoint™ memory. These and various combinations of multiple different types of NVM devices may also be used. For example, HDDs can be used in combination with or in place of SSDs or other types of NVM devices in the storage system 102.
It is therefore to be appreciated that numerous different types of storage devices 106 can be used in storage system 102 in other embodiments. For example, a given storage system as the term is broadly used herein can include a combination of different types of storage devices, as in the case of a multi-tier storage system comprising a flash-based fast tier and a disk-based capacity tier. In such an embodiment, each of the fast tier and the capacity tier of the multi-tier storage system comprises a plurality of storage devices with different types of storage devices being used in different ones of the storage tiers. For example, the fast tier may comprise flash drives while the capacity tier comprises HDDs. The particular storage devices used in a given storage tier may be varied in other embodiments, and multiple distinct storage device types may be used within a single storage tier. The term “storage device” as used herein is intended to be broadly construed, so as to encompass, for example, SSDs, HDDs, flash drives, hybrid drives or other types of storage devices.
In some embodiments, the storage system 102 illustratively comprises a scale-out all-flash distributed content addressable storage (CAS) system, such as an XtremIO™ storage array from Dell EMC of Hopkinton, Mass. A wide variety of other types of distributed or non-distributed storage arrays can be used in implementing the storage system 102 in other embodiments, including by way of example one or more VNX©, VMAX©, Unity™ or PowerMax™ storage arrays, commercially available from Dell EMC. Additional or alternative types of storage products that can be used in implementing a given storage system in illustrative embodiments include software-defined storage, cloud storage, object-based storage and scale-out storage. Combinations of multiple ones of these and other storage types can also be used in implementing a given storage system in an illustrative embodiment.
The term “storage system” as used herein is therefore intended to be broadly construed, and should not be viewed as being limited to particular storage system types, such as, for example, CAS systems, distributed storage systems, or storage systems based on flash memory or other types of NVM storage devices. A given storage system as the term is broadly used herein can comprise, for example, any type of system comprising multiple storage devices, such as network-attached storage (NAS), storage area networks (SANs), direct-attached storage (DAS) and distributed DAS, as well as combinations of these and other storage types, including software-defined storage.
In some embodiments, communications between the host devices 101 and the storage system 102 comprise Small Computer System Interface (SCSI) or Internet SCSI (iSCSI) commands. Other types of SCSI or non-SCSI commands may be used in other embodiments, including commands that are part of a standard command set, or custom commands such as a “vendor unique command” or VU command that is not part of a standard command set. The term “command” as used herein is therefore intended to be broadly construed, so as to encompass, for example, a composite command that comprises a combination of multiple individual commands. Numerous other commands can be used in other embodiments.
For example, although in some embodiments certain commands used by the host devices 101 to communicate with the storage system 102 illustratively comprise SCSI or iSCSI commands, other embodiments can implement IO operations utilizing command features and functionality associated with NVM Express (NVMe), as described in the NVMe Specification, Revision 1.3, May 2017, which is incorporated by reference herein. Other storage protocols of this type that may be utilized in illustrative embodiments disclosed herein include NVMe over Fabric, also referred to as NVMeoF, and NVMe over Transmission Control Protocol (TCP), also referred to as NVMe/TCP.
The host devices 101 are configured to interact over the network 104 with the storage system 102. Such interaction illustratively includes generating IO operations, such as write and read requests, and sending such requests over the network 104 for processing by the storage system 102. In some embodiments, each of the host devices 101 comprises a multi-path input-output (MPIO) driver configured to control delivery of IO operations from the host device to the storage system 102 over selected ones of a plurality of paths through the network 104. The paths are illustratively associated with respective initiator-target pairs, with each of a plurality of initiators of the initiator-target pairs comprising a corresponding host bus adaptor (HBA) of the host device, and each of a plurality of targets of the initiator-target pairs comprising a corresponding port of the storage system 102.
The MPIO driver may comprise, for example, an otherwise conventional MPIO driver, such as a PowerPath© driver from Dell EMC. Other types of MPIO drivers from other driver vendors may be used.
The storage system 102 in this embodiment stores data across the storage devices 106 in accordance with at least one RAID arrangement 107 involving multiple ones of the storage devices 106. The RAID arrangement 107 in the present embodiment illustratively comprises at least one RAID group. The RAID group illustratively comprises storage devices that each have the same capacity. Alternatively, the RAID group may comprise mixed-capacity storage devices, such as one or more storage devices having relatively low capacities and one or more storage devices having relatively high capacities. Such a RAID group is also referred to herein as a mixed-capacity RAID group. There may be multiple distinct RAID groups within the storage system 102, each involving a different subset of the storage devices 106. The term “RAID group” as used herein is intended to be broadly construed, so as to encompass, for example, a set of storage devices that are part of a given RAID arrangement, such as at least a subset of the storage devices 106 that are part of the RAID arrangement 107. A given such RAID group comprises a plurality of stripes, each containing multiple stripe portions distributed over multiple ones of the storage devices 106 that are part of the RAID group.
In the particular illustrative RAID arrangement 107 shown in the figure, the example RAID group more specifically comprises a subset of the storage devices 106 that includes n of the storage devices 106 individually denoted 1 through n, respectively, with each such storage device being assumed for simplicity of illustration to have the same storage capacity. For example, the storage devices 1 through n may comprise respective SSDs each having a 500 GigaByte (GB) capacity, a 1 TeraByte (TB) capacity, or another capacity, although these and other particular storage device capacities referred to herein should not be viewed as limiting in any way. It is to be appreciated, however, that the disclosed techniques can be readily extended to other types of RAID groups, such as mixed-capacity RAID groups, each including a variety of different SSDs having different capacities. Illustrative embodiments can therefore include more than two sets of storage devices, such as three, four or more different sets of storage devices, with the storage devices of a given such set each having a storage capacity that is different than that of the storage devices of the other sets. One or more of such sets can alternatively include only a single storage device rather than multiple storage devices.
The RAID arrangement 107 can comprise, for example, a RAID 5 arrangement supporting recovery from a failure of a single one of the plurality of storage devices, a RAID 6 arrangement supporting recovery from simultaneous failure of up to two of the storage devices, or another type of RAID arrangement. For example, some embodiments can utilize RAID arrangements with redundancy higher than two.
The term “RAID arrangement” as used herein is intended to be broadly construed, and should not be viewed as limited to RAID 5, RAID 6 or other parity RAID arrangements. For example, a RAID arrangement in some embodiments can comprise combinations of multiple instances of distinct RAID approaches, such as a mixture of multiple distinct RAID types (e.g., RAID 1 and RAID 6) over the same set of storage devices, or a mixture of multiple stripe sets of different instances of one RAID type (e.g., two separate instances of RAID 5) over the same set of storage devices. As will be described in more detail below, illustrative embodiments disclosed herein provide efficient mechanisms to guarantee sufficient capacity in these and other situations in which allocation of blocks to stripes cannot be predetermined, and in numerous other demanding RAID arrangements, including those involving parity RAID techniques and/or non-parity RAID techniques.
The RAID arrangement 107 is established by a storage controller 108 of the storage system 102. The storage devices 106 in the context of the RAID arrangement 107 and other RAID arrangements herein are also referred to as “disks” or “drives.” A given such RAID arrangement may also be referred to in some embodiments herein as a “RAID array.”
The RAID arrangement 107 in this embodiment illustratively includes an array of n different “disks” denoted 1 through n, each a different physical storage device of the storage devices 106. Multiple such physical storage devices are typically utilized to store data of a given LUN or other logical storage volume in the storage system 102. For example, data pages or other data blocks of a given LUN or other logical storage volume can be “striped” along with its corresponding parity information across multiple ones of the disks in the RAID arrangement 107 in accordance with RAID 5 or RAID 6 techniques.
A given RAID 5 arrangement defines block-level striping with single distributed parity and provides fault tolerance of a single drive failure, so that the array continues to operate with a single failed drive, irrespective of which drive fails. For example, in a conventional RAID 5 arrangement, each stripe includes multiple data blocks as well as a corresponding p parity block. The p parity blocks are associated with respective row parity information computed using well-known RAID 5 techniques. The data and parity blocks are distributed over the disks to support the above-noted single distributed parity and its associated fault tolerance.
A given RAID 6 arrangement defines block-level striping with double distributed parity and provides fault tolerance of up to two drive failures, so that the array continues to operate with up to two failed drives, irrespective of which two drives fail. For example, in a conventional RAID 6 arrangement, each stripe includes multiple data blocks as well as corresponding p and q parity blocks. The p and q parity blocks are associated with respective row parity information and diagonal parity information computed using well-known RAID 6 techniques. The data and parity blocks are distributed over the disks to collectively provide a diagonal-based configuration for the p and q parity information, so as to support the above-noted double distributed parity and its associated fault tolerance.
In such RAID arrangements, the parity blocks are typically not read unless needed for a rebuild process triggered by one or more storage device failures.
These and other references herein to RAID 5, RAID 6 and other particular RAID arrangements are only examples, and numerous other RAID arrangements can be used in other embodiments.
As mentioned previously, conventional RAID techniques of this type can be problematic. For example, in distributed RAID arrangements, in which the total number of blocks per stripe is less than the total number of storage devices across which the blocks are distributed, it is unduly difficult under conventional practice to ensure that there is sufficient available capacity to rebuild the blocks of one or more failed storage devices.
Conventional approaches typically require the performance of complex calculations in order to determine a particular allocation of spare capacity across the multiple storage devices, or instead allocate excessive amounts of spare capacity to each device, which wastes storage device capacity and therefore degrades storage system performance.
The storage system 102 overcomes these and other drawbacks of conventional practice by implementing techniques for guaranteeing sufficient capacity for a distributed RAID rebuild process. For example, as will be described in more detail below, the storage system 102 is configured to guarantee sufficient rebuild capacity, without the need for complex calculations or wasted capacity, for a wide variety of different RAID arrangements. Such embodiments provide RAID arrangements in which spare blocks are reserved for storage devices in a manner that ensures that a rebuild process initiated responsive to a failure of one or more of the storage devices is guaranteed to have sufficient available storage device capacity to complete the rebuilding of the blocks of the one or more failed storage devices. Reserved capacity in some embodiments is shared at the storage device level, such that, for example, spare blocks reserved on one storage device can be used to rebuild blocks of different ones of the other storage devices in the event that one of those other storage devices were to fail.
Advantageously, such techniques in illustrative embodiments herein can guarantee sufficient capacity to complete the rebuild process, while also being computationally simple, as well as highly efficient in that only at or near minimum amounts of capacity need to be reserved for rebuild within the storage system.
The storage controller 108 of storage system 102 comprises stripe configuration logic 112, parity computation logic 114, and device rebuild logic 116. The stripe configuration logic 112 determines an appropriate stripe configuration and a distribution of stripe portions across the storage devices 106 for a given RAID arrangement, including allocation of spare devices for each stripe and reservation of spare blocks for each device in conjunction with guaranteeing sufficient rebuild capacity. The parity computation logic 114 performs parity computations of various RAID arrangements, such as p parity computations of RAID 5, and/or p and q parity computations of RAID 6, using well-known techniques. The device rebuild logic 116 is configured to control the performance of a RAID rebuild process in the storage system 102, as will be described in more detail elsewhere herein.
As noted above, the storage system 102 in this embodiment implements functionality for guaranteeing sufficient capacity for RAID rebuild. This illustratively includes the performance of a process for guaranteeing sufficient capacity for RAID rebuild in the storage system 102, such as the example process to be described below in conjunction with
References herein to “guaranteeing sufficient capacity for RAID rebuild” are intended to be broadly construed, so as to encompass various types of RAID arrangements in which spare blocks are reserved for particular storage devices 106 in accordance with establishment of the RAID arrangement in the storage system 102.
In operation, the storage controller 108 via its stripe configuration logic 112 establishes a RAID arrangement comprising a plurality of stripes each having multiple portions distributed across multiple ones of the storage devices 106. Examples include the RAID arrangement 107, and the additional RAID arrangements to be described below in conjunction with
The stripe portions of each of the stripes illustratively comprise a plurality of data blocks and one or more corresponding parity blocks. The data and parity blocks are also referred to herein as “chunklets” of a RAID stripe, and such blocks or chunklets are examples of what are more generally referred to herein as “stripe portions.” In the case of RAID 5, the parity blocks or parity chunklets illustratively comprise row parity or p parity blocks, and are generated by parity computation logic 114 using well-known RAID 5 techniques. In the case of RAID 6, the parity blocks or parity chunklets illustratively comprise row parity or p parity blocks and diagonal parity or q parity blocks, and are generated by parity computation logic 114 using well-known RAID 6 techniques.
The storage controller 108 utilizes its stripe configuration logic 112 to establish a RAID arrangement comprising a plurality of stripes, with each of the plurality of stripes comprising a plurality of data blocks and one or more corresponding parity blocks, the data blocks and parity blocks being distributed across multiple ones of the storage devices 106 of a RAID group. It is to be appreciated, however, that non-parity RAID arrangements, or combinations of non-parity and parity RAID arrangements, can also be used.
Accordingly, in certain portions of the following description of illustrative embodiments, the term “blocks” will be used to refer generally to both data blocks and parity blocks. A RAID arrangement can therefore more generally comprise a plurality of stripes, with each of the plurality of stripes comprising a plurality of blocks, and the blocks being distributed across multiple ones of the storage devices.
In conjunction with establishment of the RAID arrangement, the storage controller 108 is further configured, for each of the plurality of stripes, to designate a particular one of the storage devices as a spare storage device for that stripe, and for each of the storage devices, to determine numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices. The storage controller 108 reserves a particular number of spare blocks for each of the storage devices based at least in part on the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices.
The RAID arrangement in some embodiments comprises a distributed RAID arrangement in which a total number of blocks per stripe is less than a total number of the storage devices over which the blocks of the plurality of stripes are distributed. Distributed RAID generally refers to a type of RAID in which the width of the RAID stripe in blocks is smaller than the total number of storage devices over which the blocks are distributed. An important advantage of distributed RAID relative to other types of RAID is a shorter rebuild time. For example, in distributed RAID, spare blocks are illustratively distributed over all of the storage devices that store blocks of the RAID stripes, which reduces rebuild time as the writes performed in conjunction with rebuild are spread over all of those storage devices. Such distributed RAID arrangements can include parity RAID arrangements, non-parity RAID arrangements, or possibly combinations of multiple different RAID types.
In some embodiments, the storage system 102 is further configured to store stripe metadata indicating for each of the stripes the particular one of the storage devices that is designated as a spare storage device for that stripe. For example, the stripe metadata indicating for each of the stripes the particular one of the storage devices that is designated as a spare storage device for that stripe illustratively comprises an allocation table, with the allocation table comprising a plurality of entries for respective ones of the stripes with each such entry comprising an identifier of the storage device that is designated as a spare storage device for that stripe. An example of an allocation table is shown in
Additionally or alternatively, the storage system 102 is further configured to store stripe metadata indicating for each of the storage devices the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices. For example, the stripe metadata indicating for each of the storage devices the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices comprises a reserve table, with the reserve table comprising a plurality of entries for respective ones of the storage devices with each such entry comprising the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices. An example of a reserve table is shown in
In some embodiments, reserving a particular number of spare blocks for each of the storage devices based at least in part on the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices comprises reserving a particular number of spare blocks that is equivalent to a highest one of the determined numbers of the stripes. In such an arrangement, the number of spare blocks reserved for a given one of the storage devices is the maximum number of the determined numbers of stripes for which that storage device is designated as a spare device for respective ones of the other storage devices. Alternative techniques can be used to reserve particular numbers of spare blocks for respective storage devices based at least in part on their respective sets of determined numbers in other embodiments.
The storage system 102 is illustratively further configured to detect a failure of at least one of the storage devices, and responsive to the detected failure, to initiate a rebuild process to reconstruct blocks of the one or more failed storage devices utilizing the blocks of other ones of the storage devices. The rebuild process utilizes at least a subset of the reserved spare blocks of respective ones of the non-failed storage devices. The reserved spare blocks are sufficient in number to guarantee that a rebuild process initiated responsive to a failure of one or more of the storage devices has sufficient available storage device capacity to reconstruct the blocks of the one or more failed storage devices.
In some embodiments, the reserved spare blocks of a given one of the storage devices are available for utilization in reconstructing one of at least first and second other ones of the storage devices in a rebuild process initiated responsive to a failure of said one of the at least first and second other ones of the storage devices. Each of the storage devices is designated as a spare device for multiple other ones of the storage devices.
In the case of parity RAID arrangements, the storage controller 108 detects a failure of at least one of the storage devices of the RAID arrangement, and responsive to the detected failure, reconstructs data blocks of that storage device utilizing the data blocks and parity blocks stored on other ones of the storage devices, with the reconstructed data blocks being stored in respective ones of the available spare blocks.
This reconstruction also utilizes what is more generally referred to herein as a “rebuild process” to reconstruct the data blocks of the failed storage device based on data blocks and parity blocks of the remaining storage devices of the RAID arrangement. The failure illustratively comprises a full or partial failure of one or more of the storage devices 106 in a RAID group of the RAID arrangement 107. A “remaining storage device” as that term is broadly used herein refers to a storage device that is not currently experiencing a failure. Thus, all of the storage devices of the RAID group other than the one or more storage devices for which a failure was detected are considered remaining storage devices of the RAID group. Such remaining storage devices are also referred to herein as “surviving storage devices,” as these storage devices have survived the one or more detected failures.
The storage system 102 illustratively rebuilds stripe portions impacted by the one or more detected failures by reconstruction of impacted data blocks and parity blocks using non-impacted data blocks and parity blocks, using well-known techniques, such as the RAID 5 or RAID 6 techniques mentioned previously. This rebuild process continues until all of the stripe portions of the impacted stripes are fully rebuilt.
Numerous other types of RAID implementations can be used in illustrative embodiments herein, as will be appreciated by those skilled in the art, possibly using error correcting codes such as Reed Solomon codes or other types of codes that are known to those skilled in the art. The term “parity” as used herein is therefore intended to be broadly construed, so as to encompass these and other types of information suitable for use in recovering from at least one failure in at least one storage device.
Additional details regarding examples of techniques for storing data in RAID arrays such as the RAID arrangement 107 of the
In some embodiments, establishing a RAID arrangement in which sufficient rebuild capacity is guaranteed comprises executing an algorithm to determine an appropriate distribution of reserve capacity across multiple storage devices, and to utilize portions of that reserve capacity in a rebuild process triggered by a device failure.
An example of an algorithm that guarantees sufficient capacity for the RAID rebuild in the manner described above, without placing any assumptions on the assignment of blocks to stripes and using only a near minimal amount of reserved storage space, illustratively includes the following steps.
1. For each stripe, when the stripe is allocated determine which storage device will serve as its spare. This does not actually reserve any capacity. An identifier of the spare device is stored as part of stripe metadata for its corresponding stripe.
2. Maintain a table that indicates for each storage device how many times it serves as a spare for each other storage device.
3. For each storage device, the amount of reserve capacity is determined as the maximal amount of reserve capacity it provides for any other storage device.
4. When a storage device fails, each stripe allocates for use in a rebuild process at least one spare block from its spare device as identified in the stripe metadata.
Such an algorithm is illustratively executed by the stripe configuration logic 112, the parity computation logic 114 and the device rebuild logic 116 of the storage controller 108 in storage system 102.
These and other algorithms disclosed herein can provide guaranteed sufficient reserve capacity for rebuild, without regard to the particular RAID type being used (e.g., different RAID stripes may have been allocated for different RAID types) or whether multiple instances of one or more RAID types are used. Reserved capacity is shared at the device level, such that, for example, the reserved capacity of a first device can be used by a second device if that device fails, by a third device if that device fails, and so on. The disclosed algorithms are also computationally very simple, illustratively involving designation of spare devices for respective allocated stripes and reserving of spare blocks based on numbers of times each device serves as a spare for respective other devices, followed by selection of spare blocks from spare devices as part of a rebuild process when a device fails.
It is to be appreciated that this particular algorithm, like others described herein, is presented by way of illustrative example only, and can be varied in other embodiments. For example, certain steps can be performed at least in part in parallel with other steps in other embodiments. Also, additional or alternative steps can be used in other embodiments, as well as different RAID arrangements.
As indicated previously, the above-described techniques relating to guaranteeing sufficient capacity for RAID rebuild in the storage system 102 are illustratively implemented at least in part by the storage controller 108, utilizing its stripe configuration logic 112 and parity computation logic 114. A rebuild process utilizing data blocks and parity blocks to recover from one or more storage device failures is illustratively implemented at least in part by the storage controller 108, utilizing its device rebuild logic 116.
The storage controller 108 and the storage system 102 may further include one or more additional modules and other components typically found in conventional implementations of storage controllers and storage systems, although such additional modules and other components are omitted from the figure for clarity and simplicity of illustration.
The storage system 102 in some embodiments is implemented as a distributed storage system, also referred to herein as a clustered storage system, comprising a plurality of storage nodes. Each of at least a subset of the storage nodes illustratively comprises a set of processing modules configured to communicate with corresponding sets of processing modules on other ones of the storage nodes. The sets of processing modules of the storage nodes of the storage system 102 in such an embodiment collectively comprise at least a portion of the storage controller 108 of the storage system 102. For example, in some embodiments the sets of processing modules of the storage nodes collectively comprise a distributed storage controller of the distributed storage system 102. A “distributed storage system” as that term is broadly used herein is intended to encompass any storage system that, like the storage system 102, is distributed across multiple storage nodes.
It is assumed in some embodiments that the processing modules of a distributed implementation of storage controller 108 are interconnected in a full mesh network, such that a process of one of the processing modules can communicate with processes of any of the other processing modules. Commands issued by the processes can include, for example, remote procedure calls (RPCs) directed to other ones of the processes.
The sets of processing modules of a distributed storage controller illustratively comprise control modules, data modules, routing modules and at least one management module. Again, these and possibly other modules of a distributed storage controller are interconnected in the full mesh network, such that each of the modules can communicate with each of the other modules, although other types of networks and different module interconnection arrangements can be used in other embodiments.
The management module of the distributed storage controller in this embodiment may more particularly comprise a system-wide management module. Other embodiments can include multiple instances of the management module implemented on different ones of the storage nodes. It is therefore assumed that the distributed storage controller comprises one or more management modules.
A wide variety of alternative configurations of nodes and processing modules are possible in other embodiments. Also, the term “storage node” as used herein is intended to be broadly construed, and may comprise a node that implements storage control functionality but does not necessarily incorporate storage devices.
Communication links may be established between the various processing modules of the distributed storage controller using well-known communication protocols such as TCP/IP and remote direct memory access (RDMA). For example, respective sets of IP links used in data transfer and corresponding messaging could be associated with respective different ones of the routing modules.
Each storage node of a distributed implementation of storage system 102 illustratively comprises a CPU or other type of processor, a memory, a network interface card (NIC) or other type of network interface, and a subset of the storage devices 106, possibly arranged as part of a disk array enclosure (DAE) of the storage node. These and other references to “disks” herein are intended to refer generally to storage devices, including SSDs, and should therefore not be viewed as limited to spinning magnetic media.
A RAID group in some embodiments is established for a particular one of the storage nodes of a distributed implementation of storage system 102. The storage devices associated with the particular one of the storage nodes are illustratively part of a DAE of that storage node, although other storage device arrangements are possible. Each such storage device illustratively comprises an SSD, HDD or other type of storage drive. Similar arrangements can be implemented for each of one or more other ones of the storage nodes, although distributed implementations using multiple storage nodes are not required.
The storage system 102 in the
The term “processing platform” as used herein is intended to be broadly construed so as to encompass, by way of illustration and without limitation, multiple sets of processing devices and associated storage systems that are configured to communicate over one or more networks. For example, distributed implementations of the system 100 are possible, in which certain components of the system reside in one data center in a first geographic location while other components of the system reside in one or more other data centers in one or more other geographic locations that are potentially remote from the first geographic location. Thus, it is possible in some implementations of the system 100 for the host devices 101 and the storage system 102 to reside in different data centers. Numerous other distributed implementations of the host devices and the storage system 102 are possible.
Additional examples of processing platforms utilized to implement host devices 101 and storage system 102 in illustrative embodiments will be described in more detail below in conjunction with
It is to be appreciated that these and other features of illustrative embodiments are presented by way of example only, and should not be construed as limiting in any way.
Accordingly, different numbers, types and arrangements of system components such as host devices 101, storage system 102, network 104, storage devices 106, RAID arrangement 107, storage controller 108, stripe configuration logic 112, parity computation logic 114, and device rebuild logic 116 can be used in other embodiments.
It should be understood that the particular sets of modules and other components implemented in the system 100 as illustrated in
The operation of the information processing system 100 will now be described in further detail with reference to the flow diagram of the illustrative embodiment of
The process as illustrated in
In step 200, the storage system 102 allocates stripes for a distributed RAID arrangement comprising at least one RAID group such as that illustrated in the RAID arrangement 107 involving storage devices 106 of
In step 202, the storage system 102 designates, for each of the stripes of the RAID arrangement, a particular one of the storage devices as a spare device. Different ones of the storage devices are illustratively designated as spare devices for respective different ones of the stripes, and each of the storage devices is illustratively designated as a spare device for multiple distinct ones of the stripes.
In step 204, the storage system 102 determines, for each of the storage devices of the RAID arrangement, numbers of stripes for which that storage device is designated as a spare device for respective other ones of the storage devices.
In step 206, the storage system 102 reserves a number of spare blocks for each of the storage devices that is equivalent to the highest number of its determined number of stripes. The reserved numbers of spare blocks for the storage devices are such that sufficient rebuild capacity is guaranteed.
Stripe arrangements of the type utilized in the
The steps are shown in sequential order for clarity and simplicity of illustration only, and certain steps can at least partially overlap with other steps. For example, the determination of numbers of stripes for which each device is designated as a spare for respective other devices in step 204 and the reserving of numbers of spare blocks for respective devices in step 206 can be performed at least in part in parallel. For instance, steps 204 and 206 can be separately iterated for each device.
Different instances of the process of
The particular processing operations and other system functionality described in conjunction with the flow diagram of
Functionality such as that described in conjunction with the flow diagram of
For example, a storage controller such as storage controller 108 in storage system 102 that is configured to perform the steps of the
Additional illustrative embodiments will now be described with reference to
Referring initially to
The distribution of blocks over the storage devices in the example of
Since the maximal number of utilized blocks per device is 8, when a device fails up to 8 blocks must be rebuilt to other devices. After the failure of a device there are 4 remaining devices, and therefore it may appear that it is sufficient to simply reserve two spare blocks on each device using a static algorithm. However, since each stripe may have its own restrictions of which devices can be considered for its spare (e.g., the spare may not be a device that is already in use by the stripe), solving this problem for the more general case is extremely difficult computationally. A simple static algorithm that uses any eligible device for a spare may not be able to fulfill the needs of all the stripes, leading to potential insufficient capacity to complete the rebuild. Such a scenario is illustrated in
In the
Such insufficient capacity issues are addressed in illustrative embodiments using the techniques disclosed herein.
Accordingly, each stripe is assigned a particular one of the devices as its spare device. For each stripe, as three of the devices are already used for blocks, there are only two candidate spare devices. The logic of stripe allocation is not restricted in this regard, and so in some embodiments the same or similar mechanism used to allocate blocks of the stripes to the storage devices can be used to allocate spare storage devices for the respective stripes. In some embodiments, in order to minimize the rebuild time, the spare allocation logic should attempt to maintain an even distribution of spare capacity, and an even number of device dependencies, although numerous other arrangements can be used in allocating the spare devices.
Considering device D1, that device is designated as a spare for a stripe block of device D2 twice, for D3 once, for D4 twice, and for D5 once, as is apparent from the allocation table of
Similarly, for device D2, that device is designated as a spare for a stripe block of device D1 three times, for D3 twice, for D4 once, and for D5 three times, as is again apparent from the allocation table of
As another example, for device D3, that device is designated as a spare for stripe blocks of device D1 twice, for D2 three times, for D4 twice, and for D5 twice, as is once again apparent from the allocation table of
The maximum number of blocks is similarly determined for devices D4 and D5, from the allocation table of
Spare blocks are reserved for the different devices using the numbers in the “blocks” column of the reserve table. In other words, total numbers of spare blocks given by 2, 3, 3, 2 and 3 are reserved on the respective storage devices D1, D2, D3, D4 and D5. Although it may appear that the required spare blocks in this example (3 in some cases) are much higher (50% more) than what is theoretically required (only 2), that is primarily due to fact that the present example uses a very low number of stripes for clarity and simplicity of illustration. It is expected that numerous practical implementations will typically contain a much larger number of stripes, and therefore a much lower percentage increase in required spare capacity relative to the theoretical minimum associated with use of a static approach, while also serving to guarantee sufficient capacity to complete the rebuild process.
As noted above, use of a simple static algorithm led to insufficient capacity to complete the rebuild in
Although these examples use stripes and blocks as allocation units, this is by way of illustrative example only, and different allocation units can be used in other embodiments. For example, an allocation unit can be larger than a single RAID stripe, depending on various aspects of the RAID design. Illustrative embodiments can be adapted in a straightforward manner for use with these and other such larger allocation units.
Also, the examples above are given for RAID 5 but illustrative embodiments are not restricted to RAID 5, RAID 6 or other types of parity RAID. Any RAID arrangement or combination of RAID arrangements may be used.
Accordingly, it is to be appreciated that the particular RAID arrangements illustrated in
Additional illustrative embodiments will now be described with reference to
The embodiments previously described with reference to
The embodiments to be described in conjunction with
In the context of multiple failures, the use of multiple spare devices per stripe is relevant also for RAID arrangements that support recovery from only a single failure. For example, in a RAID arrangement that supports recovery from only a single failure (e.g., RAID 5), a second failure can illustratively occur after completion of a rebuild process for a first failure. However, in a RAID arrangement that supports recovery from multiple failures (e.g., RAID 6) the first and second failures may be concurrent.
Illustrative embodiments to be described below provide support for multiple failures while still achieving a near minimal amount of reserved capacity. In these embodiments, each stripe is allocated multiple spare devices.
As will become more apparent, such an arrangement provides a number of significant advantages. For example, this arrangement can provide support for multiple device failures, as well as node failures where each node contains multiple devices, while still guaranteeing sufficient spare capacity to complete rebuild. Also, capacity can be reserved only for stripes that suffer multiple failures. Illustrative embodiments can therefore provide adjustable spare capacity based on required resiliency level (e.g., a single device failure, multiple device failures, a single device failure and a single node failure, and multiple node failures). As in other embodiments herein, the use of shared spare capacity reduces the amount of storage space required to guarantee sufficient spare capacity to complete rebuild.
Referring now to the flow diagram of
In step 700, stripes are allocated for a distributed RAID arrangement across multiple storage devices. It is therefore assumed in the present embodiment that the RAID arrangement comprises a distributed RAID arrangement in which a total number of blocks per stripe is less than a total number of the storage devices over which the blocks of the plurality of stripes are distributed.
In step 702, for each stripe, multiple ones of the storage devices are designated as respective spare devices. For example, first and second spare devices may be designated for each of the storage devices of the RAID arrangement. More than two spare devices may be designated in other embodiments.
In step 704, for each device, the process determines numbers of stripes for which that device is designated as a spare device for respective other ones of the devices in each of multiple spare levels and for each of multiple failure combinations. Such determinations illustratively involve the use of multiple counters to maintain respective counts of the numbers of stripes for each device for which that device is designated as a spare device relative to other ones of the devices, although a wide variety of other arrangements are possible. For example, particular determined numbers shown in one or more example tables presented herein can be maintained using respective counters.
In step 706, a number of spare blocks is reserved for each device based at least in part on the determined numbers, for example, by summing the determined numbers across the spare levels and taking a maximum of the resulting sums across the multiple failure combinations.
As in the flow diagram of
In the
For example, the stripe metadata indicating for each of the stripes the multiple ones of the storage devices that are designated as respective spare devices for that stripe illustratively comprises an allocation table. The allocation table illustratively comprises a plurality of entries for respective ones of the stripes with each such entry comprising identifiers of the storage devices that are designated as respective spare devices for that stripe.
Additionally or alternatively, the storage system 102 may be further configured to store stripe metadata indicating for each of the storage devices the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices in each of multiple spare levels and for each of multiple failure combinations.
For example, the stripe metadata indicating for each of the storage devices the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices in each of multiple spare levels and for each of multiple failure combinations comprises a reserve table. The reserve table illustratively comprises a plurality of entries for respective ones of the storage devices with each such entry comprising the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices in each of multiple spare levels and for each of multiple failure combinations. As indicated above, counters can be used in some embodiments to provide respective ones of the determined numbers.
Reserving a particular number of spare blocks for each of the storage devices based at least in part on the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices across the multiple spare levels and the multiple failure combinations illustratively comprises summing the determined numbers for the storage device across the spare levels to generate resulting sums for respective ones of the failure conditions, determining a highest one of the resulting sums across the respective ones of the failure conditions, and reserving a particular number of spare blocks for the storage device that is greater than or equal to the highest one of the resulting sums.
In some embodiments, designating multiple ones of the storage devices as respective spare devices for a given one of the stripes comprises designating at least first and second storage devices as respective first and second spare devices for the given stripe. The first and second storage devices correspond to respective first and second spare levels of the multiple spare levels. A first failure impacting the given stripe results in utilization of a spare block of the first spare device and a second failure impacting the given stripe results in utilization of a spare block of the second spare device.
The failure combinations in some embodiments comprise respective different sets of two or more of the storage devices. Numerous other types of failure combinations can be used in other embodiments.
For example, in some embodiments, different ones of the storage devices are implemented on different ones of a plurality of nodes of a distributed storage system. In such an embodiment, the failure combinations may comprise different combinations of one of the storage devices and one of the nodes.
An example algorithm in an illustrative embodiment with multiple spare devices per storage device will now be described in further detail with reference to
Step 1. Allocate spare devices to each stripe in conjunction with allocation of the stripe, and reserve sufficient spare capacity on those spare devices. This step illustratively utilizes an allocation table and a reserve table, examples of which are shown in respective
In the allocation table, for each stripe, when the stripe is allocated, N devices are allocated to serve as its spare devices, where N is the maximal number of supported failures. As in other embodiments, this does not actually reserve any capacity. The spare device IDs are stored as part of the stripe metadata. It should be noted that the variable N in this context is used in a different manner than in one or more other contexts herein.
In the reserve table, for each spare device, information is stored regarding the combinations of source devices that when failed lead to allocation of spare capacity from the spare device. This illustratively involves performing the following operations (a) through (c):
(a) For each spare device, calculate how much spare capacity it requires at each of at least two spare levels, for each of the failure combinations, including how much capacity as a first spare, how much capacity as a second spare, and so on. Each spare device has its spare capacity tracked and accumulated separately at each spare level. For stripes in which the current device is the first spare device, a single failure is tracked for that device. For stripes in which the current device is the second spare device, combinations of two failures that lead to the current device being used as a spare device are tracked for that device. This is continued for all the spare levels.
(b) For each spare device, the spare capacity required by a particular failure combination is the summation of the spare capacities for that spare device over all of the spare levels.
(c) The spare capacity required for each spare device is the maximal value over all of the possible failure combinations. This is the amount of spare capacity that is reserved for that spare device.
Step 2. Allocate spare blocks from the spare devices when one or more devices fail, for use in rebuild of the corresponding stripes. When a device fails, each impacted stripe allocates a block from a spare device as indicated in its stripe metadata, in accordance with the established ordering of multiple spare devices. The first failure will use the first spare device, the second failure will use the second spare device, and so on. This step is illustrated by a spare block allocation table, an example of which is shown in
Each of Step 1 and Step 2 will be described in further detail below, using the tables of
The allocation of spare devices in Step 1 will first be described in more detail with reference to
The logic of stripe allocation is not restricted and may use the same allocation mechanism as that used by the data blocks. In order to minimize the rebuild time, the spare allocation logic is illustratively configured to provide, to the greatest extent possible within a given implementation, an even distribution of spare capacity, an even number of device dependencies within stripes, and an even distribution of devices as spares to the same stripe.
Step 1 in the present embodiment more particularly proceeds in the following manner. When allocating spare devices for a stripe, for all failure combinations that affect this stripe, increment the number of spare blocks a first spare device requires for this failure combination. For the first spare device, a row is counted only once, even if two data blocks are lost in the failure combination. For a second spare device, the spare is counted if either two data blocks are lost or if one data block is lost and S1 is failed as well.
Referring now to the allocation table of
The output of the above-described Step 1 in some embodiments is in the form of a reserve table, illustratively arranged as shown in
The content of the reserve table of
Considering by way of example the case where devices D1 and D2 have failed, the required spare capacity for another device, illustratively device D4, is determined as follows. As a first level spare, D4 must contain four spare blocks corresponding to rows (4, 7, 13, 19) in the allocation table of
When allocating a new stripe, for example, the stripe corresponding to row 20 in the allocation table of
1. Single—Add to S1 (which is D6):
-
- (a) D2, D3, D4
2. Dual—Add to S2 (which is D5):
-
- (a) 2 Data: {D2, D3}, {D3, D4}, {D2, D4}
- (b) 1 Data and S1: {D2, D6}, {D3, D6}, {D4, D6}
The total number of spares required per failure combination is given in the reserve table of
Dual Failure Spare=First Spare Level+Second Spare Level
The amount of spare capacity to reserve per device is the maximum of each column in the “Dual Failure Spare” column group. Accordingly, for devices D1, D2, D3, D4, D5 and D6, the corresponding numbers of spare blocks reserved are 6, 5, 6, 7, 7 and 7, respectively, as shown in the MAX line of the “Dual Failure Spare” column group.
Note that if it is desirable to reserve capacity only for those stripes that suffer a dual failure, the maximum of the “Second Spare Level” column group will be the end result of the spare capacity reservation process.
Step 2 of the example algorithm will now be described in more detail.
The allocation of spare blocks in Step 2 responsive to a failure more particularly proceeds in the following manner.
A spare block must be allocated when a device fails. The spare device is determined using stripe metadata, shown in the “Spare” column of the allocation table of
Referring to row 1, only one device failed in this stripe, therefore only the first spare device S1 is used, which is device D1. In row 4, two spare devices are required and hence both spare devices S1 and S2, which are D4 and D3, are used. If the first spare device is failed, the second spare device is used even though only one spare device is needed. This can be seen in row 3 where D5 is the first spare device S1 but that device is failed and so D1 is used.
The algorithm described above provides efficiency advantages over other algorithms. For example, in a static algorithm that allocates dedicated spare stripes, the number of spare blocks required is given by:
SPARE_STRIPES=SPARE_BLOCKS_PER_STRIPE*STRIPES
In the above example of
The number of spare blocks required in the algorithm described above is given by:
SPARE_STRIPES=SPARE_BLOCKS_PER_STRIPE*STRIPES*RAID_WIDTH/DEVICES*DISTRIBUTION_EVENNESS_FACTOR
The first two parameters are the same as in the previous calculation. Next the number is multiplied by the RAID width as a counter may be incremented for each device used by the stripe. Such counters are illustratively distributed and shared over all the devices, and so the formula divides by the number of devices. DISTRIBUTION_EVENNESS_FACTOR addresses the ability of the algorithm to spread out the spare capacity evenly. For an ideal implementation of the algorithm this factor will be 1. It is clear that this algorithm benefits more when the RAID width is relatively small as compared to the number of devices.
Another advantageous feature of this embodiment is that the remaining capacity is not wasted. It can still be used for another failure, although there will no longer be a guarantee of sufficient capacity. The total number of reserved blocks is 38. This is better than the 40 blocks required by the static algorithm, but the improvement is limited because the relatively small number of stripes in this example make it difficult to achieve an even distribution of spare capacity. However, even with only 20 stripes in this simplified example, a small gain is nonetheless achieved relative to a static algorithm. It is expected that larger gains will be achieved in RAID arrangements involving larger numbers of stripes.
The manner in which illustrative embodiments support node failures will now be described in further detail with reference to
In addition to supporting multiple device failures, the algorithm described above also supports RAID arrangements that are spread across nodes of a distributed storage system, such as RAID arrangements in which each block is on a different node. Such scenarios are common when the devices are accessible only from a single node, for example, in software-defined storage environments.
When allocating a new stripe, for example, the stripe corresponding to row 20 in the allocation table of
1. Single—Add to S1 (which is D6):
-
- (a) D1/N1, D4/N2
2. Dual—Add to S2 (which is D3):
-
- (a) 2 Data: {N1, D4}, {N2, D1}
- (b) 1 Data and S1: {D1, N3}, {D4, N3}, {N1, D6}, {N2, D6}
The total number of spares required per failure combination is given in the reserve table of
Dual Failure Spare=First Spare Level+Second Spare Level
The amount of spare capacity to reserve per device is the maximum of each column in the “Dual Failure Spare” column group. Accordingly, for devices D1, D2, D3, D4, D5 and D6, the corresponding numbers of spare blocks reserved are 6, 7, 7, 7, 7 and 6, respectively, as shown in the MAX line of the “Dual Failure Spare” column group.
As in other examples disclosed herein, the examples of
Additionally or alternatively, other aspects of the illustrative embodiments can be varied. For example, although the examples of
Moreover, the above-described algorithms and corresponding operations associated with guaranteeing sufficient capacity for RAID rebuild are presented by way of illustrative example only, and should not be viewed as limiting in any way. Additional or alternative operations can be used in other embodiments.
Again, references to “disks” in the context of RAID herein are intended to be broadly construed, and should not be viewed as being limited to disk-based storage devices. For example, the disks may comprise SSDs, although it is to be appreciated that many other storage device types can be used.
Illustrative embodiments of a storage system with functionality for guaranteeing sufficient capacity for RAID rebuild in a storage system as disclosed herein can provide a number of significant advantages relative to conventional arrangements.
For example, some embodiments disclosed herein provide RAID arrangements in which spare blocks are reserved for storage devices in a manner that ensures that a rebuild process initiated responsive to a failure of one or more of the storage devices is guaranteed to have sufficient available storage device capacity to complete the rebuilding of the blocks of the one or more failed storage devices.
Advantageously, such techniques in illustrative embodiments herein can guarantee sufficient capacity to complete the rebuild process, while also being computationally simple, as well as highly efficient in that only at or near minimum amounts of capacity need to be reserved for rebuild within the storage system.
Moreover, the disclosed techniques in illustrative embodiments overcome additional drawbacks of conventional approaches. For example, some embodiments can be used for any type of RAID arrangement, including those involving combinations of multiple instances of distinct RAID approaches, such as a mixture of multiple distinct RAID types over the same set of storage devices, or a mixture of multiple stripe sets of different instances of one RAID type over the same set of storage devices.
Illustrative embodiments disclosed herein provide efficient mechanisms to guarantee sufficient capacity in these and other situations in which allocation of blocks to stripes cannot be predetermined, and in numerous other demanding RAID arrangements, including those involving parity RAID techniques and/or non-parity RAID techniques.
Accordingly, illustrative embodiments provide advantages over conventional techniques that utilize static allocation policies or “best efforts” policies. These and other conventional techniques cannot guarantee sufficient capacity for a wide range of different RAID arrangements or combinations of such arrangements.
It is to be appreciated that the particular advantages described above and elsewhere herein are associated with particular illustrative embodiments and need not be present in other embodiments. Also, the particular types of information processing system features and functionality as illustrated in the drawings and described above are exemplary only, and numerous other arrangements may be used in other embodiments.
Illustrative embodiments of processing platforms utilized to implement host devices and storage systems with functionality for guaranteeing sufficient capacity for RAID rebuild in a storage system will now be described in greater detail with reference to
The cloud infrastructure 1500 further comprises sets of applications 1510-1, 1510-2, . . . 1510-L running on respective ones of the VMs/container sets 1502-1, 1502-2, . . . 1502-L under the control of the virtualization infrastructure 1504. The VMs/container sets 1502 may comprise respective VMs, respective sets of one or more containers, or respective sets of one or more containers running in VMs.
In some implementations of the
A hypervisor platform may be used to implement a hypervisor within the virtualization infrastructure 1504. Such a hypervisor platform may comprise an associated virtual infrastructure management system. The underlying physical machines may comprise one or more distributed processing platforms that include one or more storage systems.
In other implementations of the
As is apparent from the above, one or more of the processing modules or other components of system 100 may each run on a computer, server, storage device or other processing platform element. A given such element may be viewed as an example of what is more generally referred to herein as a “processing device.” The cloud infrastructure 1500 shown in
The processing platform 1600 in this embodiment comprises a portion of system 100 and includes a plurality of processing devices, denoted 1602-1, 1602-2, 1602-3, . . . 1602-K, which communicate with one another over a network 1604.
The network 1604 may comprise any type of network, including by way of example a global computer network such as the Internet, a WAN, a LAN, a satellite network, a telephone or cable network, a cellular network, a wireless network such as a WiFi or WiMAX network, or various portions or combinations of these and other types of networks.
The processing device 1602-1 in the processing platform 1600 comprises a processor 1610 coupled to a memory 1612.
The processor 1610 may comprise a microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), graphics processing unit (GPU) or other type of processing circuitry, as well as portions or combinations of such circuitry elements.
The memory 1612 may comprise random access memory (RAM), read-only memory (ROM), flash memory or other types of memory, in any combination. The memory 1612 and other memories disclosed herein should be viewed as illustrative examples of what are more generally referred to as “processor-readable storage media” storing executable program code of one or more software programs.
Articles of manufacture comprising such processor-readable storage media are considered illustrative embodiments. A given such article of manufacture may comprise, for example, a storage array, a storage disk or an integrated circuit containing RAM, ROM, flash memory or other electronic memory, or any of a wide variety of other types of computer program products. The term “article of manufacture” as used herein should be understood to exclude transitory, propagating signals. Numerous other types of computer program products comprising processor-readable storage media can be used.
Also included in the processing device 1602-1 is network interface circuitry 1614, which is used to interface the processing device with the network 1604 and other system components, and may comprise conventional transceivers.
The other processing devices 1602 of the processing platform 1600 are assumed to be configured in a manner similar to that shown for processing device 1602-1 in the figure.
Again, the particular processing platform 1600 shown in the figure is presented by way of example only, and system 100 may include additional or alternative processing platforms, as well as numerous distinct processing platforms in any combination, with each such platform comprising one or more computers, servers, storage devices or other processing devices.
For example, other processing platforms used to implement illustrative embodiments can comprise converged infrastructure such as VxRail™, VxRack™, VxRack™ FLEX, VxBlock™ or Vblock© converged infrastructure from Dell EMC.
It should therefore be understood that in other embodiments different arrangements of additional or alternative elements may be used. At least a subset of these elements may be collectively implemented on a common processing platform, or each such element may be implemented on a separate processing platform.
As indicated previously, components of an information processing system as disclosed herein can be implemented at least in part in the form of one or more software programs stored in memory and executed by a processor of a processing device. For example, at least portions of the functionality for guaranteeing sufficient capacity for RAID rebuild in a storage system of one or more components of a storage system as disclosed herein are illustratively implemented in the form of software running on one or more processing devices.
It should again be emphasized that the above-described embodiments are presented for purposes of illustration only. Many variations and other alternative embodiments may be used. For example, the disclosed techniques are applicable to a wide variety of other types of information processing systems, host devices, storage systems, storage devices, RAID arrangements, storage controllers, stripe configuration logic, parity computation logic, device rebuild logic and other components. Also, the particular configurations of system and device elements and associated processing operations illustratively shown in the drawings can be varied in other embodiments. Moreover, the various assumptions made above in the course of describing the illustrative embodiments should also be viewed as exemplary rather than as requirements or limitations of the disclosure. Numerous other alternative embodiments within the scope of the appended claims will be readily apparent to those skilled in the art.
Claims
1. An apparatus comprising:
- a storage system comprising a plurality of storage devices;
- the storage system being configured:
- to establish a redundant array of independent disks (RAID) arrangement comprising a plurality of stripes, with each of the plurality of stripes comprising a plurality of blocks, the blocks being distributed across multiple ones of the storage devices; and
- in conjunction with establishment of the RAID arrangement:
- for each of the plurality of stripes, to designate multiple ones of the storage devices as respective spare devices for that stripe;
- for each of the storage devices, to determine numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices in each of multiple spare levels and for each of multiple failure combinations; and
- to reserve a particular number of spare blocks for each of the storage devices based at least in part on the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices across the multiple spare levels and the multiple failure combinations.
2. The apparatus of claim 1 wherein the RAID arrangement comprises a distributed RAID arrangement in which a total number of blocks per stripe is less than a total number of the storage devices over which the blocks of the plurality of stripes are distributed.
3. The apparatus of claim 1 wherein the RAID arrangement comprises at least one parity RAID arrangement supporting recovery from a failure of at least one of the plurality of storage devices and wherein a given one of the stripes comprises a plurality of data blocks and one or more parity blocks.
4. The apparatus of claim 1 wherein the storage system is further configured to store stripe metadata indicating for each of the stripes the multiple ones of the storage devices that are designated as respective spare devices for that stripe.
5. The apparatus of claim 4 wherein the stripe metadata indicating for each of the stripes the multiple ones of the storage devices that are designated as respective spare devices for that stripe comprises an allocation table, the allocation table comprising a plurality of entries for respective ones of the stripes with each such entry comprising identifiers of the storage devices that are designated as respective spare devices for that stripe.
6. The apparatus of claim 1 wherein the storage system is further configured to store stripe metadata indicating for each of the storage devices the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices in each of multiple spare levels and for each of multiple failure combinations.
7. The apparatus of claim 6 wherein the stripe metadata indicating for each of the storage devices the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices in each of multiple spare levels and for each of multiple failure combinations comprises a reserve table, the reserve table comprising a plurality of entries for respective ones of the storage devices with each such entry comprising the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices in each of multiple spare levels and for each of multiple failure combinations.
8. The apparatus of claim 1 wherein reserving a particular number of spare blocks for each of the storage devices based at least in part on the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices across the multiple spare levels and the multiple failure combinations comprises:
- summing the determined numbers for the storage device across the spare levels to generate resulting sums for respective ones of the failure conditions;
- determining a highest one of the resulting sums across the respective ones of the failure conditions; and
- reserving a particular number of spare blocks for the storage device that is greater than or equal to the highest one of the resulting sums.
9. The apparatus of claim 1 wherein designating multiple ones of the storage devices as respective spare devices for a given one of the stripes comprises designating at least first and second storage devices as respective first and second spare devices for the given stripe, the first and second storage devices corresponding to respective first and second spare levels of the multiple spare levels, and wherein a first failure impacting the given stripe results in utilization of a spare block of the first spare device and a second failure impacting the given stripe results in utilization of a spare block of the second spare device.
10. The apparatus of claim 1 wherein the failure combinations comprise respective different sets of two or more of the storage devices.
11. The apparatus of claim 1 wherein different ones of the storage devices are implemented on different ones of a plurality of nodes of a distributed storage system.
12. The apparatus of claim 11 wherein the failure combinations comprise different combinations of one of the storage devices and one of the nodes.
13. The apparatus of claim 1 wherein the storage system is further configured:
- to detect a failure of at least one of the storage devices; and
- responsive to the detected failure, to initiate a rebuild process to reconstruct blocks of the one or more failed storage devices utilizing the blocks of other ones of the storage devices;
- wherein the rebuild process utilizes at least a subset of the reserved spare blocks of respective ones of the non-failed storage devices.
14. The apparatus of claim 1 wherein the reserved spare blocks are sufficient in number to guarantee that a rebuild process initiated responsive to a failure of one or more of the storage devices has sufficient available storage device capacity to reconstruct the blocks of the one or more failed storage devices.
15. A method for use in a storage system comprising a plurality of storage devices, the method comprising:
- establishing a redundant array of independent disks (RAID) arrangement comprising a plurality of stripes, with each of the plurality of stripes comprising a plurality of blocks, the blocks being distributed across multiple ones of the storage devices; and
- in conjunction with establishment of the RAID arrangement:
- for each of the plurality of stripes, designating multiple ones of the storage devices as respective spare devices for that stripe;
- for each of the storage devices, determining numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices in each of multiple spare levels and for each of multiple failure combinations; and
- reserving a particular number of spare blocks for each of the storage devices based at least in part on the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices across the multiple spare levels and the multiple failure combinations.
16. The method of claim 15 wherein reserving a particular number of spare blocks for each of the storage devices based at least in part on the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices across the multiple spare levels and the multiple failure combinations comprises:
- summing the determined numbers for the storage device across the spare levels to generate resulting sums for respective ones of the failure conditions;
- determining a highest one of the resulting sums across the respective ones of the failure conditions; and
- reserving a particular number of spare blocks for the storage device that is greater than or equal to the highest one of the resulting sums.
17. The method of claim 15 wherein designating multiple ones of the storage devices as respective spare devices for a given one of the stripes comprises designating at least first and second storage devices as respective first and second spare devices for the given stripe, the first and second storage devices corresponding to respective first and second spare levels of the multiple spare levels, and wherein a first failure impacting the given stripe results in utilization of a spare block of the first spare device and a second failure impacting the given stripe results in utilization of a spare block of the second spare device.
18. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by a processor of a storage system comprising a plurality of storage devices, causes the storage system:
- to establish a redundant array of independent disks (RAID) arrangement comprising a plurality of stripes, with each of the plurality of stripes comprising a plurality of blocks, the blocks being distributed across multiple ones of the storage devices; and
- in conjunction with establishment of the RAID arrangement:
- for each of the plurality of stripes, to designate multiple ones of the storage devices as respective spare devices for that stripe;
- for each of the storage devices, to determine numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices in each of multiple spare levels and for each of multiple failure combinations; and
- to reserve a particular number of spare blocks for each of the storage devices based at least in part on the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices across the multiple spare levels and the multiple failure combinations.
19. The computer program product of claim 18 wherein reserving a particular number of spare blocks for each of the storage devices based at least in part on the determined numbers of the stripes for which that storage device is designated as a spare device for respective ones of the other storage devices across the multiple spare levels and the multiple failure combinations comprises:
- summing the determined numbers for the storage device across the spare levels to generate resulting sums for respective ones of the failure conditions;
- determining a highest one of the resulting sums across the respective ones of the failure conditions; and
- reserving a particular number of spare blocks for the storage device that is greater than or equal to the highest one of the resulting sums.
20. The computer program product of claim 18 wherein designating multiple ones of the storage devices as respective spare devices for a given one of the stripes comprises designating at least first and second storage devices as respective first and second spare devices for the given stripe, the first and second storage devices corresponding to respective first and second spare levels of the multiple spare levels, and wherein a first failure impacting the given stripe results in utilization of a spare block of the first spare device and a second failure impacting the given stripe results in utilization of a spare block of the second spare device.
Type: Application
Filed: Jun 16, 2020
Publication Date: Oct 21, 2021
Inventor: Yosef Shatsky (Karnei Shomron)
Application Number: 16/902,890
