9/13/24, 10:38 AM Distributed RAID array properties - IBM Documentation
Distributed RAID array properties
Last Updated: 2024-06-14
Distributed RAID arrays contain rebuild areas that are used to maintain redundancy after a drive fails.
This is true of all distributed RAID arrays, except for distributed RAID 1 arrays with only 2 drives. If not enough drives are available on the system, you cannot
configure a distributed RAID array. Distributed RAID arrays solve rebuild bottlenecks in non distributed RAID array configurations because rebuild areas are
distributed across all the drives in the array.
The rebuild write workload is spread across all the drives rather than just the single spare drive, which results in faster rebuilds on an array. Distributed RAID
arrays remove the need for separate drives that are idle until a failure occurs. Instead of allocating one or more drives as spares, the spare capacity is distributed
over specific rebuild areas across all the member drives. Data can be copied faster to the rebuild area and redundancy is restored much more rapidly. Additionally,
as the rebuild progresses, the performance of the pool is more uniform because all of the available drives are used for every volume extent. After the failed drive is
replaced, data is copied back to the drive from the distributed rebuild areas. Unlike hot-spare drives, read/write requests are processed on other parts of the drive
that are not being used as rebuild areas. The number of rebuild areas is based on the width of the distributed RAID array and on the technology of the member
drives.
For example, a distributed RAID array that uses RAID 6 drives can handle two concurrent drive failures. After the two failed drives are rebuilt, the array can, once
again, tolerate another two drive failures. If all of the rebuild areas are used, the array becomes degraded on the next drive failure.
The effective capacity of a distributed RAID 1 array that is made from the same number of underlying drives as a RAID 5 or RAID 6 array is in general less than the
effective capacity of the latter. However, while capacity-wise, distributed RAID 5 and 6 are generally better, performance-wise, RAID 1 may be better for certain
workloads.
Supported RAID levels
The system supports the following RAID levels for distributed RAID arrays:
Distributed RAID 1
Distributed RAID 1 arrays consist of two mirrored strips, which are distributed across all member drives. Distributed RAID 1 arrays can tolerate a single failed
member drive when a rebuild area is in place. If a member drive fails in a distributed RAID 1 array contains two member drives, or the rebuild area is no longer
available, the array becomes degraded. Unlike distributed RAID 6, distributed RAID 1 does not contain any parity strips. In order to restore redundancy for
degraded distributed storage arrays, the rebuild-in-place process is used, reconstructing the data directly back into the replaced member drive. Degraded
distributed RAID arrays with only two member drives use the rebuild-in-place process to restore redundancy, copying the data directly back into the replaced
member drive.
https://www.ibm.com/docs/en/flashsystem-5x00/8.7.x?topic=configurations-distributed-raid-array-properties 1/12
9/13/24, 10:38 AM Distributed RAID array properties - IBM Documentation
Distributed RAID 6
Distributed RAID 6 arrays stripe data over the member drives with two parity strips on every stripe. Distributed RAID 6 arrays can tolerate any two concurrent
member drive failures. In order to restore redundancy for degraded distributed RAID arrays, the rebuild-in-place process is used for reconstructing the data (or
parity) directly back into the replaced member drive.
Example of a distributed RAID array with no rebuild areas
Note: While this example is showing a 2 member drive storage array (with distributed RAID 1), the same rebuild-in-place process occurs in any
distributed storage arrays if the member drives have failed to an extent that no rebuild area is available. In these cases, the data is reconstructed, not
copied, onto the replaced drive.
Figure 1. Two member drive storage array with no rebuild area
https://www.ibm.com/docs/en/flashsystem-5x00/8.7.x?topic=configurations-distributed-raid-array-properties 2/12
9/13/24, 10:38 AM Distributed RAID array properties - IBM Documentation
shows an example of a distributed RAID array with two member drives and no rebuild area; both of the drives in the array are active.
1 (Minimum) two active drives and stripe width
2 Pack, with depth of two strips
Figure 2. Two member drive storage array with no rebuild area
https://www.ibm.com/docs/en/flashsystem-5x00/8.7.x?topic=configurations-distributed-raid-array-properties 3/12
9/13/24, 10:38 AM Distributed RAID array properties - IBM Documentation
Figure 3 shows a distributed RAID array with two member drives that contains a failed drive. To recover data, the data copies the strip from the active drive to the
new or previously failed drive.
1 Active drive
2 Failed drive
Figure 3. Two member drive storage array with a failed drive
https://www.ibm.com/docs/en/flashsystem-5x00/8.7.x?topic=configurations-distributed-raid-array-properties 4/12
9/13/24, 10:38 AM Distributed RAID array properties - IBM Documentation
Figure 4 shows the rebuild-in-place process, with the new data being copied directly into the replaced drive.
1 Active drive
2 Data being copied directly into the replaced drive
Figure 4. Rebuild-in-place process on the two member drive
https://www.ibm.com/docs/en/flashsystem-5x00/8.7.x?topic=configurations-distributed-raid-array-properties 5/12
9/13/24, 10:38 AM Distributed RAID array properties - IBM Documentation
Example of a distributed RAID array with rebuild areas
Figure 5 shows an example of a distributed RAID array that is configured with RAID 6; all of the drives in the array are active. The rebuild areas are distributed
across all of the drives and the drive count includes all of the drives.
1 An active drive
2 Rebuild areas, which are distributed across all drives.
https://www.ibm.com/docs/en/flashsystem-5x00/8.7.x?topic=configurations-distributed-raid-array-properties 6/12
9/13/24, 10:38 AM Distributed RAID array properties - IBM Documentation
3 Drive count, which includes all drives.
4 Stripes of data (two stripes are shown)
5 Stripe width
6 Pack, which equals the drive count that is multiplied by stripe width.
7 Additional packs in the array (not shown)
Figure 5. Distributed RAID array (RAID 6 level)
https://www.ibm.com/docs/en/flashsystem-5x00/8.7.x?topic=configurations-distributed-raid-array-properties 7/12
9/13/24, 10:38 AM Distributed RAID array properties - IBM Documentation
Figure 6 shows a distributed RAID array that contains a failed drive. To recover data, the data is read from multiple drives. The recovered data is then written to the
rebuild areas, which are distributed across all of the drives in the array. The remaining rebuild areas are distributed across all drives.
1 Failed drive
2 Rebuild areas, which are distributed across all drives.
3 Remaining rebuild areas rotate across each remaining drive.
4 Additional packs in the array (not shown)
https://www.ibm.com/docs/en/flashsystem-5x00/8.7.x?topic=configurations-distributed-raid-array-properties 8/12
9/13/24, 10:38 AM Distributed RAID array properties - IBM Documentation
Figure 6. Distributed RAID array (RAID 6 level) with a failed drive
Array width
The array width, which is also referred to as the drive count, indicates the total number of drives in a distributed RAID array. This total includes the number of
drives that are used for data capacity and parity, (if there are parity strips, like in distributed RAID 6), and the rebuild area (if existing) that is used to restore
redundancy of the data.
https://www.ibm.com/docs/en/flashsystem-5x00/8.7.x?topic=configurations-distributed-raid-array-properties 9/12
9/13/24, 10:38 AM Distributed RAID array properties - IBM Documentation
Rebuild area
The rebuild area is the drive capacity that is reserved within a distributed RAID array to reconstruct data after a drive failure; it provides no usable capacity. Unlike
a non distributed RAID array, the rebuild area is distributed across all of the drives in the array. As data is rebuilt during the copyback process, the rebuild area
contributes to the performance of the distributed RAID array because all of the member drives perform I/O requests.
Note: Distributed RAID 1 systems with only two member drives do not contain a rebuild area.
Rebuild-in-place
A distributed RAID array process that takes place when no rebuild area is available, which causes the array to become degraded. The rebuild-in-place process
restores data redundancy by copying or reconstructing the data directly back into the replaced member drive by using the original data distribution.
Note: Not all distributed RAID array storage systems might support the rebuild-in-place process. For more information, see Supported RAID levels.
Stripe and stripe width
A stripe, which can also be referred to as a redundancy unit, is the smallest amount of data that can be addressed. Distributed RAID array strip size is 256 KiB.
The stripe width indicates the number of strips of data that can be written at one time when data is rebuilt after a drive fails. This value is also referred to as the
redundancy unit width. In Figure 5, the stripe width of the array is 5.
Drive class
To enhance performance of a distributed RAID array, all of the drives must come from the same, or superior, drive class. Each drive class is identified by its
drive_class_id. The system uses the following information to determine the drive class of each drive:
Block size
Indicates the block size of the drive class. Valid block size is either 512 or 4096 bytes.
Capacity
Indicates the capacity of the drive class.
I/O group
Indicates the I/O group name that is associated with the drive class.
https://www.ibm.com/docs/en/flashsystem-5x00/8.7.x?topic=configurations-distributed-raid-array-properties 10/12
9/13/24, 10:38 AM Distributed RAID array properties - IBM Documentation
RPM speed
Indicates the speed of the drive class. Valid RPM speed can be 7.2 K, 10 K, or 15 K. For SSDs, this value is blank.
Technology
Indicates the technology for the drive class. The following technology types are supported:
Storage Class Memory
Storage Class Memory drives use persistent memory technologies that improve endurance and reduce latency of current flash storage device technologies.
All SCM drives use NVMe architecture.
Tier 0 flash
Tier 0 flash drives are high-performance flash drives that process read and write operations and provide faster access to data than Enterprise or nearline
drives. For most Tier 0 flash drives, the system monitors their wear level and issues warnings when the drive is nearing replacement. Some NVMe-attached
drives, such as IBM FlashCore® Module drives, are considered Tier 0 flash drives.
Tier 1 flash
Tier 1 flash drives are lower-cost flash drives, typically with larger capacities, but slightly lower performance and write endurance characteristics. The
system monitors their wear level and issues warnings when the drive is nearing replacement.
Enterprise disks
Enterprise disks are disk drives that are optimized for performance.
Nearline disks
Nearline disks are disk drives that are optimized for capacity.
To replace a failed member drive in the distributed RAID array, the system can use another drive that has the same drive class as the failed drive. The system can
also select a drive from a superior drive class. For example, two drive classes can contain drives of the same technology type but different data capacities. In this
case, the superior drive class is the drive class that contains the higher capacity drives.
To display information about all of the drive classes that are available on the system, use the lsdriveclass command. Example output from the lsdriveclass
command shows four drive classes on the system. Drive class 209 contains drives with a capacity of 278.9 GB; drive class 337 contains drives with a capacity of
558.4 GB. Although the drives have the same RPM speed, technology type, and block size, drive class 337 is considered to be superior to drive class 209.
Example output from the lsdriveclass command
id RPM capacity IO_group_id IO_group_name tech_type block_size candidate_count superior_count total_count transport_protocol compressed
0 7.0TB 0 io_grp0 tier0_flash 4096 6 6 6 nvme no
1 20.0TB 0 io_grp0 tier0_flash 512 2 2 8 nvme yes
2 744.7GB 0 io_grp0 tier0_flash 512 2 2 2 sas no
3 1.7TB 0 io_grp0 tier1_flash 512 2 2 2 sas no
https://www.ibm.com/docs/en/flashsystem-5x00/8.7.x?topic=configurations-distributed-raid-array-properties 11/12
9/13/24, 10:38 AM Distributed RAID array properties - IBM Documentation
Slow write priority settings
When a redundant array is doing read/write I/O operations, the performance of the array is bound by the performance of the slowest member drive. If the SAS
network is unstable or if too much work is being driven to the array when drives do internal ERP processes, performance to member drives can be far worse than
usual. In this situation, arrays that offer redundancy can accept a short interruption to redundancy to avoid writing to, or reading from, the slow component. Write
operations that are mapped to a poorly performing drive are committed to the other copy or parity, and are then completed with good status (assuming no other
failures). When the member drive recovers, the redundancy is restored by a background process of writing the strips that were marked out of sync while the
member was slow.
This technique is governed by the setting of the slow_write_priority attribute of the distributed RAID array, which defaults to latency when the array is
created. When set to latency, the array is allowed to become out of sync in an attempt to smooth poor member performance. You can use the charray
command to change the slow_write_priority attribute to redundancy. When set to redundancy, the array is not allowed to become out of sync. However,
the array can avoid suffering read performance loss by returning reads to the slow component from redundant paths.
When the array uses latency mode or attempts to avoid reading a component that is in redundancy mode, the system evaluates the drive regularly to assess
when it becomes a reliable part of the system again. If the drive never offers good performance or causes too many performance failures in the array, the system
fails the hardware to prevent ongoing exposure to the poor-performing drive. The system fails the hardware only if it cannot detect another explanation for the bad
performance from the drive.
Distributed RAID array member drive replacement
If the fault LED light on a drive is lit, the drive is marked as failed and is no longer used in the distributed RAID array. When the system detects that a failed drive
was replaced, it automatically removes the failed hardware from the array configuration. If the new drive is suitable (for example, in the same drive class), the
system begins a copyback operation to make a rebuild area available again in the distributed RAID array.
Parent topic:
Array configurations
https://www.ibm.com/docs/en/flashsystem-5x00/8.7.x?topic=configurations-distributed-raid-array-properties 12/12
