ReaR: Backup and recover your Linux server with confidence

Backing up and recovering a Linux server is a common activity for system administrators throughout their career. While many ad-hoc utilities help sysadmins manage backups, this article focuses on the ReaR - Relax and Recover tool. Yes, you read it right—a tool to help you have peace of mind (relax) when it comes to performing mundane backup and recovery operations.

ReaR is an open-source administration tool to create disaster recovery archives and bootable ISO images of a Linux server. By integrating with your existing backup solution, ReaR can be used as a backup and restore utility, as well.

How it works

ReaR creates a bootable image consisting of the latest state of the server to be backed up. It also gives the system administrator the flexibility to select files for backup. It is highly recommended to run ReaR pro-actively per your existing backup schedules so that you have the up-to-date server state.

While there is a lot of documentation already out there on the Internet, this article covers ReaR's functionality with supporting screenshots.

Throughout this post, I refer to the server to be backed up as production and the external NFS server as storage.

The production and storage servers each have Red Hat Enterprise Linux 8.2 (RHEL) deployed. Adjust the instructions to match your operating system version. For example, replace dnf with yum for older Red Hat-based systems.

Configure the destination server as a backup target

ReaR supports NFS, CIFS (Samba), USB, rsync, and local destinations as backup targets. In this article, server storage is configured as an NFS server to be used as the backup target for the ReaR utility deployed on the production server.

[root@storage ~]# hostname
storage.mj0vy.org

[root@storage ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.2 (Ootpa)

[root@storage ~]# cat /etc/exports.d/rear.exports
/backups   192.168.122.0/24(rw,no_root_squash)

[root@storage ~]# exportfs -s
/backups  192.168.122.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)

[root@storage ~]# ip a s eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:d4:04:e4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.44/24 brd 192.168.122.255 scope global dynamic noprefixroute eth0
       valid_lft 3123sec preferred_lft 3123sec
    inet6 fe80::5054:ff:fed4:4e4/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

Deploy ReaR on the server to be backed up

On the production server, install the rear, genisoimage, and syslinux packages. In RHEL, these packages are part of the base repository.

[root@production ~]# hostname
production.mj0vy.org

[root@production ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.2 (Ootpa)

[root@production ~]# dnf install rear genisoimage syslinux

[root@production ~]# ip a s eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:5f:f2:e0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.19/24 brd 192.168.122.255 scope global dynamic noprefixroute eth0
       valid_lft 2129sec preferred_lft 2129sec
    inet6 fe80::5054:ff:fe5f:f2e0/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

Configure ReaR on the server to be backed up

Edit the /etc/rear/local.conf file with the following configurations:

OUTPUT
Define the bootable image type among USB, ISO, RAWDISK

OUTPUT_URL
Backup Target among NFS, CIFS, FILE, FTP, HTTP, RSYNC

BACKUP
Which backup method to use: External third-party or internal ReaR mechanism?

BACKUP_URL
Backup target location when using the internal backup method.

BACKUP_PROG_EXCLUDE
Directories to be excluded while performing the backup. Examples are /media and /var/tmp

[root@production ~]# cat /etc/rear/local.conf
OUTPUT=ISO
OUTPUT_URL=nfs://192.168.122.44/exports
BACKUP=NETFS
BACKUP_URL=nfs://192.168.122.44/exports
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash')

Execute ReaR on the server to be backed up

ReaR accepts different options based on whether you need to create a rescue media and/or backup the system as a tar file.

rear mkbackup - create the rescue media and backup the system.
rear mkbackuponly - create only the backup tar file.
rear mkrescue - create only the rescue media.

Image

Backup the production server with an ISO rescue media

[root@production ~]# rear -v -d mkbackup
Relax-and-Recover 2.4 / Git
Using log file: /var/log/rear/rear-production.log
Using backup archive '/tmp/rear.gGxFXV4S0pcfEmD/outputfs/production/backup.tar.gz'
Creating disk layout
Using guessed bootloader 'GRUB' (found in first bytes on /dev/vda)
Creating root filesystem layout
Adding net.ifnames=0 to KERNEL_CMDLINE
Handling network interface 'eth0'
eth0 is a physical device
Handled network interface 'eth0'
To log into the recovery system via ssh set up /root/.ssh/authorized_keys or specify SSH_ROOT_PASSWORD
Copying logfile /var/log/rear/rear-production.log into initramfs as '/tmp/rear-production-partial-2020-07-30T02:40:30+05:30.log'
Copying files and directories

After some minutes, the files stored in local directories are copied over the network to the remote NFS target location.

Copying files and directories
Copying binaries and libraries
Copying kernel modules
Copying all files in /lib*/firmware/
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (200298197 bytes) in 20 seconds
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-production.iso (202M)
Copying resulting files to nfs location
Saving /var/log/rear/rear-production.log as rear-production.log to nfs location
Creating tar archive '/tmp/rear.gGxFXV4S0pcfEmD/outputfs/production/backup.tar.gz'
Archived 1037 MiB [avg 11304 KiB/sec]

Inspect the storage server while ReaR is running on the production server

[root@storage ~]# ls -lh /backups/production/
total 1.4G
-rw-------. 1 root root 3.3M Jul 30 02:42 backup.log
-rw-------. 1 root root 1.2G Jul 30 02:42 backup.tar.gz
-rw-------. 1 root root  202 Jul 30 02:41 README
-rw-------. 1 root root 202M Jul 30 02:41 rear-production.iso
-rw-------. 1 root root 712K Jul 30 02:41 rear-production.log
-rw-------. 1 root root    0 Jul 30 02:42 selinux.autorelabel
-rw-------. 1 root root  277 Jul 30 02:41 VERSION

The hostname of the server to be backed up is suffixed in the ISO file created in the remote target location.

• rear-production.iso - Bootable ISO Image, used during recovery
• backup.tar.gz - TAR file containing the files from the production server

The ReaR command finishes with the suggestion to remove the directory created during the mkbackup operation.

Exiting rear mkbackup (PID 21370) and its descendant processes
Running exit tasks
You should also rm -Rf /tmp/rear.QeZDyfY54KuYq65

Restore the server using the bootable ISO image created by ReaR

The production server is restored using the rear-production.iso image. Boot the computer with the CD-ROM as the first boot device and choose "Recover production" or "Automatic Recover production."

Image

Selecting "Automatic Recover production" automates the recovery process, assuming there are no errors.

Image

This article focusses on the "Recover <hostname>" option to manually recover the production server.

Image

The system boots to the ReaR banner detailing your backup and output type.

Image

Log in as the root user without being prompted for a password.

Image

As suggested by the login banner, run rear recover to restore the system by connecting to the storage server. From there, retrieve the backup.tar.gz and restore it to the right destination with appropriate permissions.

RESCUE production: ~ # rear -v -d recover

Image

Inspecting /var/log/rear/rear-production.log gives you details of how the restoration works. The relevant information from the log file is referenced below:

2020-07-31 06:58:38.091131490 Started RPC portmapper 'rpcbind'.
2020-07-31 06:58:38.102814819 RPC portmapper 'rpcbind' available.
2020-07-31 06:58:38.124144303 Started rpc.statd.
2020-07-31 06:58:38.635988990 Mounting with 'mount -v -t nfs -o rw,noatime 192.168.122.44:/backups /tmp/rear.2b0Q3MJJ2gwdEvq/outputfs'
mount.nfs: timeout set for Fri Jul 31 07:00:38 2020
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.122.44,clientaddr=192.168.122.19'
2020-07-31 06:58:38.701408122 Using backup archive '/tmp/rear.2b0Q3MJJ2gwdEvq/outputfs/production/backup.tar.gz'

Boot the server from the hard disk after the restore operation

The server can now be rebooted. Choose the hard disk as the boot device. During the restore process, the server connects to the NFS share and then transfers the backup.tar.gz file to your local directory. Confirm this by checking the /var/log/rear/recover/restore/recover.backup.tar.gz.587.restore.log file when the restore operation is finished.

2020-07-31 06:59:10.851011181 dd if=/tmp/rear.2b0Q3MJJ2gwdEvq/outputfs/production/backup.tar.gz | cat | tar --block-number --totals --verbose --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls --gzip -C /mnt/local/ -x -f -

Next, SELinux relabeling is done. The restoration time varies depending upon the size of the backup.tar.gz file.

Eventually, the server boots to the login screen.

Image

Login with the same credentials and verify that the server state is restored.

That's a wrap

ReaR is an integral part of many Linux-based backup solutions. OpenStack and the Red Hat OpenStack Platform use ReaR for the undercloud and control plane backup and restore. Watch for future articles in this space addressing the patching and rollback options for applications and operating systems.

For more on using ReaR in a Red Hat Enterprise Linux production environment, be sure to consult this solution in the Red Hat Customer Portal.

[ Free online course: Red Hat Enterprise Linux technical overview. ]