1. Introduction

Welcome astute reader.

The IT industry has a continual balance between security and usability. Within this balance, performance usually affects usability. In the realm of protecting "Data at Rest" (i.e. encryption) one may find three factors affecting performance, and therefore usabilty: The harddrive, CPU and RAM. Of these, the harddrive will always prove to be a bottleneck (yes, even with an Solid State Drive, SSD).

Free Software has a rather elegant solution for protecting Data at Rest, called Linux Unified Key Setup (LUKS). In the spirit of "Freedom 0: The freedom to run the program for any purpose." please enjoy my contribution to our collective knowledgebase of a solution to the fascinating problem of "how can we speed up encyption"?

(C) 2014 Justin Mitchell Bennett. Licenced under the GNU FDL.

2. Definitions and Baselines

I used an Acer Aspire V5-171 as the platform for creating a hybrid RAID 1 (RAM drive & SATA HDD). I noted, after about a year of use, that Fedora 19 rarely used over 1 GB of RAM, with the exception of running a Windows 7 virtual machine through Virtual Machine Manager (libvirt/QEMU). As the system allowed for a max of 16GB DDR3, when I upgraded the modules, I factored I could comfortably utilize about 10GB of RAM for a RAM drive to run Fedora in.

I did find limitation however, in how much hard drive space Fedora uses for root /. I tend to partition / (root), /home, and swap on the systems I build. However, with this setup, root (/) tended to run 11-16GB, with squid, ndjbdns, LibreOffice, Android Development Tool, Eclipse, etc. With Fedora, /usr tends to be the largest directory under root. For reasons I won't go into, /usr cannot currently be mounted as a separate partition in Fedora 20; which actually proves useful, as it houses both /bin and /sbin, as well as Xorg. Unless you've already got a somewhat streamlined Fedora installation, creating separate /opt & /var partitions is *highly* recommended.

2A. Partitions

This guide presumes a prerequisite knowledge of partitions, (i.e. creating them and the benefit of multiple partitions). Additionally, some familiarity with EUFI and GPT, as they affect a Fedora 20 installation, (i.e. creating a /boot/efi & /boot partition). If working with BIOS and/or MBR please adjust your partition strategy accordingly. Additionally, as shrinking/growing partitions can be a huge pain, this guide has been written from the perspective of a fresh Fedora installation or re-installation.

2B. Logical Volume Management (LVM)

This guide also presumes a familiarity with Logical Volume Management. As a minimum, you must have an understanding of Physical Volumes (pv), Volume Groups (vg) & Logical Volumes (lv) as they pertain to LVM. Further reading: RHEL Logical Volume Manager Administration

2C. RAID 1

As used in this guide the reference to RAID 1 (mirror) should be understood as RAID 1 being implemented through LVM2. Development of RAID continues at Red Hat, and of interest specific to this guide, we'll utilize lvchange --writemostly. For reference:

Fedora 20 LVM 2 changelog:
2.02.99-1	Peter Rajnoha	
	25 Jul 2013

- Add lvcreate/lvchange --[raid]{min|max}recoveryrate for raid LVs.
- Add lvchange --[raid]writemostly/writebehind support for RAID1
- Add lvchange --[raid]syncaction for scrubbing of RAID LVs.

2D. Linux Unified Key Setup (LUKS)

In the simpliest terms, LUKS extends cryptsetup functionality with an aim to conform to TKS1. TKS1 uses a hashed key for performing data de/encryption through an encryption algorithm. In this guide, specifically, AES encryption. If it's good enough to secure US Secret classification of material, it should be secure enough to secure your Data At Rest. Further reading at Cryptsetup

2E. Fedora Live USB image

I utilized two USB sticks for this setup. At a minimum, create a Fedora USB media. As I prefer MATE Desktop to Gnome, I additionally created a net-install image. Additionally, once booted from the Fedora USB media, install gparted.

2F. CPU (AES-NI), RAM, and HDD benchmarks:

(Note, for these tests, I booted off the Fedora USB media)

Newer Intel(R) processors include the AES-NI instruction set. To test whether your processor has such instructions type

cat /proc/cpuinfo | grep aes
For brevity, I've skipped posting the output here.

Next, test the CPU throughput of various encryption algorithms. Type

cryptsetup benchmark
My output yields:

# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1       386073 iterations per second
PBKDF2-sha256     209380 iterations per second
PBKDF2-sha512     140034 iterations per second
PBKDF2-ripemd160  308404 iterations per second
PBKDF2-whirlpool  164870 iterations per second
#  Algorithm | Key |  Encryption |  Decryption
     aes-cbc   128b   434.6 MiB/s  1528.2 MiB/s
 serpent-cbc   128b    63.3 MiB/s   212.4 MiB/s
 twofish-cbc   128b   133.2 MiB/s   250.6 MiB/s
     aes-cbc   256b   316.0 MiB/s  1140.8 MiB/s
 serpent-cbc   256b    63.7 MiB/s   213.1 MiB/s
 twofish-cbc   256b   132.8 MiB/s   250.2 MiB/s
     aes-xts   256b  1264.3 MiB/s  1266.8 MiB/s
 serpent-xts   256b   218.7 MiB/s   207.4 MiB/s
 twofish-xts   256b   241.3 MiB/s   247.5 MiB/s
     aes-xts   512b   994.2 MiB/s   986.7 MiB/s
 serpent-xts   512b   218.8 MiB/s   208.9 MiB/s
 twofish-xts   512b   243.7 MiB/s   247.7 MiB/s

Of note, AES-XTS with a 256bit key shows the highest throughput. By default, Fedora (through Anaconda) will install an AES-XTS 512 bit key LUKS partition using a SHA1 hash. I chose to reduce the AES key size from 512 to 256 for gaining throughput (~1000MiB/s vs ~1260MiB/s), and increased the key hash to SHA512, SHA1 having documented weakness.

On my system, I have 16GB DDR3L 1600MHz PC3 12800 memory. To display the current memory, type

type sudo dmidecode --type 17 | less

From that, look for the following:

Size: 8192 MB
Form Factor: SODIMM
Type: DDR3
Speed: 1600 MHz
Configured Clock Speed: 1600 MHz

(Sidenote - From the factory, ACER shipped 6GB of RAM, however, the 2GB stick had been DDR3 1333MHz, slowing the other stick, a 4GB 1600MHz to 1333Mhz)

Presuming you have enough RAM, create a RAM disk of 1GB for testing throughput with:

sudo modprobe brd rd_nr=1 rd_size-1048576 max_part=0

Now test its read speed through hdparm:

sudo hdparm -t /dev/ram0
/dev/ram0:
 Timing buffered disk reads: 5050 MB in  3.00 seconds = 1683.23 MB/sec

After such testing, the RAM disk can be removed with

sudo rmmod brd

For comparison, do a test against the hard drive. In this case, an upgraded Travelstar 0J42251, /dev/sda3 has been fully partitioned as a LVM2 Physical Partition (sda1=/boot/efi @ 256MiB, sda2=/boot @ 512 MiB):

(Sidenote - Though a SATA 3 6GB/s drive, For unknown reasons, Acer released a BIOS update that capped the motherboard at SATA 2, though the chipset specifies SATA 3 capability)

sudo hdparm -t /dev/sda3
/dev/sda3:
 Timing buffered disk reads: 302 MB in  3.00 seconds = 100.51 MB/sec

In creating a RAID 1 mirror, we're trying to push our encryption throughput for Fedora (OS) from 100.51 MB/sec (HDD speed) to 1260 Mib/~1321 MB/s placing the encryption bottleneck at the CPU.

3. LVM or Creating the PV, VG & LGs

3A. Physical Volume (pv)

I'm not entirely sure why Fedora no longer maintains an LVM GUI. Given some of the added complexity of the underlying LVM, I do understand how the GUI falls very short on harnessing some of the advanced features of LVM, namely on the fly RAID 1 creation.

I chose the following partition scheme: sda1 /boot/efi 256 MiB, sda2 /boot 512 MiB, sda3 LVM2 all remaining space (~930.76 MiB)

pvcreate /dev/sda3

3B. Volume Group (vg)

Next I created the Volume Group. Fedora (anaconda) names their vg Fedora_[Hostname]. My hostname being gintonbo (Japanese for silver dragonfly):

vgcreate fedora_gintonbo /dev/sda3

3C. Logical Volumes

The remaining partitions in the system (/, /opt, /var, with a different command for /home):

lvcreate -L 16G -n lv_root fedora_gintonbo
lvcreate -L 16G -n lv_swap fedora_gintonbo
lvcreate -L 16G -n lv_opt fedora_gintonbo
lvcreate -L 16G -n lv_var fedora_gintonbo 

Having plenty of space remaining, I used 90% of it for lv_home:

lvcreate -l 90%FREE -n lv_home fedora_gintonbo

From the Cryptsetup FAQ 2.19 "How can I wipe a device with crypto-grade randomness?" next prewipe the drives. *Warning* yes, this will take a while, have a beverage or two of choice handy for this step:

cryptsetup open --type plain -d /dev/urandom /dev/fedora_gintonbo/lv_root to_be_wiped
cat /dev/zero > /dev/mapper/to_be_wiped
cryptsetup close /dev/fedora_gintonbo/lv_root

Repeat for the remaining Logical Volumes create, and it's better to save /dev/fedora_[hostname]/lv_home for last, as it will take the longest!

3D. RAM disk Extents

Either while waiting for cryptsetup to finish or upon completion of filling the partitions with cryptographic randomness, create the RAM disk for RAID. Kernel module brd uses K, so 1048576K=1GiB, and therefore times 10 (for 10GiB) = 10485760. However, the lv_root partition created earlier had been 16GiB. We're going to remove that partition in the next step, and reduce it down to 10GiB, however, math at such powers of 10 versus powers of 2 becomes fuzzy very quickly. Better to oversize your lv_root and shrink it, rather than trying to guess how much more RAM you'll need.

Kernel Module brd create the Ram Drive, 10GB this time:

sudo modprobe brd rd_nr=1 rd_size-10485760 max_part=0

Create a pv with the Ram disk:

pvcreate /dev/ram0

Create a temporary VG for getting total extents for /dev/ram0

vgcreate testvg

Next create a Logical Volume utilizing 100% of the RAM drive:

lvcreate 100%FREE -n lv_test testvg

Now we're going to find out how many *extents* are on the drive:

vgs -o +vg_free_count,vg_extent_count

On the RAM disk this gave 2559 total extents.

We can now remove the LV with

lvremove /dev/testvg/lv_test

Then remove the VG

vgremove testvg 

Next remove the PV

pvremove /dev/ram0

Lastly, unload the RAM disk

sudo rmmod brd

Please note, LVMs do not expect RAM disks to be PVs, and are VERY sensitive to being unable to locate a PV (think powerloss). Always cleanly shutdown your RAM disk, being unable to boot later due to a RAM wipe will not be fun. You have been warned.

3E. lv_root Remove & Recreate Through Extents

From the above, we came up with 2559 extents, with which we'll now match the HDD size to. In similar fashion as above, remove the lv_root:

lvremove /dev/fedora_[hostname]/lv_root

Then recreate it, specifying extents, rather than MB:

lvcreate -l2559 -n lv_root fedora_[hostname]

4. LUKS AES Encryption & Ext4 File System

At this point, you should have five (5) LVs on the system, lv_root ~10GiB, lv_opt 16GiB, lv_var 16 Gib, lv_swap 16 Gib, and lv_home ~768GiB.

From the cryptsetup benchmark tests, aes-xts-plain had the highest throughput, when using a 256 bit key. Additionally, instead of SHA1 we'll utilize SHA512, and specify spending 5 sec (-i 5000) on hash iteration. When mounted through the mapper, Fedora mounts LUKS encrypted partitions at /dev/mapper/luks-[UUID]. However, as we'll be manually mounting the partition through the mapper, we'll need to utilize cryptsetup luksUUID [dev] for obtaining the UUID.

For each of the five LVs, you'll repeat the following steps. I'll illustrate two of them, to show what changes:

cryptsetup luksFormat -y -c aes-xts-plain -s 256 -h SHA512 -i 5000 --use-urandom /dev/mapper/fedora_gintonbo-lv_root
cryptsetup luksUUID /dev/mapper/fedora_gintonbo-lv_root
cryptsetup luksOpen /dev/mapper/fedora_gintonbo-lv_root luks-[UUID] 
mkfs.ext4 -m1 -L gintonbo-root /dev/mapper/luks-[UUID]

lv_home

cryptsetup luksFormat -y -c aes-xts-plain -s 256 -h SHA512 -i 5000 --use-urandom /dev/mapper/fedora_gintonbo-lv_home
cryptsetup luksUUID /dev/mapper/fedora_gintonbo-lv_home
cryptsetup luksOpen /dev/mapper/fedora_gintonbo-lv_home luks-[UUID] 
mkfs.ext4 -m1 -L gintonbo-home /dev/mapper/luks-[UUID]

Again, as I had a new HDD, I'd geared the LVMs, LUKS and EXT4 partions toward a fresh Fedora install. Partitioning the system this way also worked seemlessly with Anaconda, only having to unlock the drives.

5. RAM disk & (LVM) RAID 1

Size matching the root partition to the RAM disk had been the hardest part of this endeavor. With the partitions setup in the last step, I'd then installed Fedora 20 through a Net Install with MATE. Because the root partition had been expressed through extents size (2559 = ~10GiB) the RAM disk as RAID member becomes trivially easy.

5A. To create the RAM disk and add it to the LVM RAID 1:

sudo modprobe brd rd_nr=1 rd_size=10485760 max_part=0
sudo pvcreate /dev/ram0
sudo vgextend fedora_gintonbo /dev/ram0
(Optionally display your LVs at this point) sudo lvs -a -o name,copy_percent,devices fedora_gintonbo
sudo lvconvert --type raid1 -m1 fedora_gintonbo/lv_root

To watch the percentage of mirror building

sudo lvs -a -o name,copy_percent,devices fedora_gintonbo

Lastly, change the HDD to writemostly, so reads occur from RAM disk:

sudo lvchange --writemostly /dev/sda3:y fedora_gintonbo/lv_root

5B. To remove the RAM disk from LVM RAID 1 (prior to system powerloss event!)

sudo lvchange --writemostly /dev/sda3:n fedora_gintonbo/lv_root
sudo lvconvert -m0 fedora_gintonbo/lv_root /dev/ram0
sudo vgreduce fedora_gintonbo /dev/ram0
sudo rmmod /dev/ram0

6. Systemd Unit file

As the finishing touch, have systemd manage RAM disk & RAID 1 creation/removal. We'll create a systemd unit called 2Ro2L (RAMdisk RAID on LVM & LUKS) and two scripts:

sudo pluma /lib/systemd/system/2Ro2L.service 
[Unit]
Description=Enable RAMdisk RAID 1 on LVM & LUKS
After=multi-user.target

[Service]
RemainAfterExit=yes
ExecStart=/usr/sbin/2Ro2Lstart
ExecStop=/usr/sbin/2Ro2Lstop
Type=oneshot

[Install]
WantedBy=multi-user.target
sudo chmod 644 /lib/systemd/system/2RoRL.service

Then create the start and stop scripts:

sudo pluma /usr/sbin/2Ro2Lstart
#!/bin/sh

# create the RAM disk
modprobe brd rd_nr=1 rd_size=10485760 max_part=0

# create an LVM Physical Volume on created RAMdisk:
pvcreate /dev/ram0

# Extend Volume Group to include above pv
vgextend fedora_gintonbo /dev/ram0

# Convert the Logical Volume to a RAID 1 configuration
lvconvert --type raid1 -m1 fedora_gintonbo/lv_root

# Change the Logical Volume RAID properties
lvchange --writemostly /dev/sda3:y fedora_gintonbo/lv_root
sudo pluma /usr/sbin/2Ro2Lstop
#!/bin/sh

# Remove Logical Volume RAID properties
lvchange --writemostly /dev/sda3:n fedora_gintonbo/lv_root

# Remove the RAMdisk from the RAID
lvconvert -m0 fedora_gintonbo/lv_root /dev/ram0

# Remove the RAMdisk from the Volume Group
vgreduce fedora_gintonbo /dev/ram0

# Remove the RAMdisk altogether
rmmod brd
sudo chmod 755 /usr/sbin/2Ro2Ls*

Start the service, stop the service, and if all goes well, enable the service for multiuser target

sudo systemctl start 2Ro2L.service
sudo systemctl stop 2Ro2L.service

If those ran without error, than enable the service:

sudo systemctl enable 2Ro2L.service
Special thanks to Sergey Davidoff, FedoraZram for the systemd unit information!
Share this page:

1 Comment(s)