Configuring raid -- md0p2 or md1?

Forum: LXer Meta ForumTotal Replies: 3
Author Content
penguinist

Aug 05, 2016
10:18 AM EST
Hello fellow LXers. Someone here certainly has the answer to my question of the day.

I'm upgrading my archival storage which will consist of a raid5 assembly of four drives organized as two equal size partitions. In the past I've done this in two different ways:

Alternative 1: Partition the raw drives each with two equal sized partitions and then set up two independent raid5 arrays on these partitions using:

mdadm --create /dev/md0 --level=5 --raid-devices=4 /dev/sd[bcde]1
mdadm --create /dev/md1 --level=5 --raid-devices=4 /dev/sd[bcde]2
This gives two independent arrays /dev/md0 and /dev/md1 which can then be formated, mounted and used.

Alternative 2: Partition the raw drives each with one large partition and then set up a single raid5 array on them using:

mdadm --create /dev/md0 --level=5 --raid-devices=4 /dev/sd[bcde]1
Then partition that /dev/md0 array into two parts. This gives a single array with two partitions, /dev/md0p1 and /dev/md0p2 which can then be formated, mounted and used.

So the question is, does one of these alternatives have advantages over the other or are they equivalent in the end?

I've "googled" this question using my favorite search engine and have come up with little opinion on this one way or the other.
jdixon

Aug 05, 2016
11:20 AM EST
> So the question is, does one of these alternatives have advantages over the other or are they equivalent in the end?

That's a really good question. Unless there's a speed difference involved (which you'd have to test by benchmarking both configurations), I'd think they would be equivalent. I'd be please to be corrected by a more knowledgeable reader. As with you a search doesn't turn up much information.
mbaehrlxer

Aug 06, 2016
12:37 PM EST
smaller raid partitions complete faster if you ever need to replace a disk and resync.

normally, it doesn't matter, but if you need to reboot during the resync then any incomplete resync will start over.

if you have multiple raid partitions, then any partition that completed the resync before the reboot will be fine, and you'll only need to start over with the ones not yet completed.

in my case i have a single btrfs partition that is spread over 4 md raid devices on two 3TB disks.

greetings, eMBee.
penguinist

Aug 09, 2016
9:23 AM EST
We have some test results. First please permit me to complain about how long it takes to run performance experiments on a 12TB array. Basically you get one experiment per day when you account for the time it takes to build a raid5 array. Ok I have that off my shoulders, now let's get to the results.

First test:

Partition four 4TB drives with single partitions and build a 12 TB array, then partition that into two equal 6TB parts and format them with ext4. This gives us a /dev/md0p1 and a /dev/md0p2. For those interested in the details, here are the commands we used to accomplish this:

parted /dev/sdb mklabel gpt mkpart primary 1M 4001GB
parted /dev/sdc mklabel gpt mkpart primary 1M 4001GB
parted /dev/sdd mklabel gpt mkpart primary 1M 4001GB
parted /dev/sde mklabel gpt mkpart primary 1M 4001GB
mdadm --create /dev/md0 --level=5 --raid-devices=4 /dev/sd[bcde]1
# now we have a big array and we proceed to partition it into two pieces
parted /dev/md0 mklabel gpt
parted /dev/md0 mkpart primary 3072s  11720659967s
parted /dev/md0 mkpart primary 11720659968s 23441313791s
mkfs.ext4 /dev/md0p1
mkfs.ext4 /dev/md0p2
mount /dev/md0p1 /data
mount /dev/md0p2 /back


Running the bonnie++ performance benchmark on this gives us these results:

bonnie++ -b -d /data -s 128G -n 0 -m after_init -u root
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
after_init     128G   808  96 262866  40 150010  25  3354  83 422299  38 130.1  15
Latency             14473us     354ms     756ms   27122us     115ms     464ms


So we have these results:

Write at 263 MB/s, Read at 422 MB/s



Second test:

Partition four 4TB drives each with two partitions and build two 6TB arrays from them, then format those arrays with ext4. This gives us a /dev/md0 and a /dev/md1. For those interested in the details, here are the commands we used to accomplish this:

parted /dev/sdb --script mklabel gpt mkpart primary 2048s 3907018751s mkpart primary 3907018752s 7814033407s unit s p
parted /dev/sdc --script mklabel gpt mkpart primary 2048s 3907018751s mkpart primary 3907018752s 7814033407s unit s p
parted /dev/sdd --script mklabel gpt mkpart primary 2048s 3907018751s mkpart primary 3907018752s 7814033407s unit s p
parted /dev/sde --script mklabel gpt mkpart primary 2048s 3907018751s mkpart primary 3907018752s 7814033407s unit s p
mdadm --create /dev/md0 --level=5 --raid-devices=4 /dev/sd[bcde]1
mdadm --create /dev/md1 --level=5 --raid-devices=4 /dev/sd[bcde]2
mkfs.ext4 /dev/md0
mkfs.ext4 /dev/md1
mount /dev/md0 /data
mount /dev/md1 /back


Running the bonnie++ performance benchmark on this gives us these results:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
after_init     128G  1018  95 280598  42 156776  25  4427  87 480970  35 146.1   8
Latency             13782us     254ms     672ms   18787us     111ms     517ms


So we have these results:

Write at 281 MB/s, Read at 481 MB/s



Conclusions:

The smaller array configuration (prepartition the raw drives) wins the performance race by a noticeable margin resulting in 6.7% improvement with writes and 13.9% improvement with reads.

Notice that in both tests, the mdadm defaults were used with no attempt to optimize or fine tune the parameters. (I'll leave those tests to the next investigator.)

Posting in this forum is limited to members of the group: [Editors, MEMBERS, SITEADMINS.]

Becoming a member of LXer is easy and free. Join Us!