How to Install GlusterFS Scalable Network Filesystem on Debian 11

GlusterFS or Gluster File System, is a free and open-source distributed file system developed by RedHat. GlusterFS is a scalable file system formed from several servers into one entity file system that allows users to connect and mount the GlusterFS volume.

GlusterFS is a modern file system that can handle petabytes of data. Also, it's easy to install and maintain, and it's also easy to scale the file system.

In this tutorial, you will install GlusterFS - distributed and scalable network filesystem - on Debian 11 servers. You'll set up GlusterFS volume that automatically replicates data to multiple servers and set up the high-availability filesystem. In addition to that, you'll also learn how to use the 'parted', a Linux partitioning tool for setting up additional disks on Debian servers. Lastly, you'll verify the data replication on GlusterFS between multiple Debian servers and verify the hight-availability too.

Prerequisites

To complete this tutorial, you must have the following requirements:

  • Two or three Debian 11 servers.
  • A non-root user with sudo/administrator root privileges.

This example uses three Debian 11 servers with the following details:

Hostname    IP Address
--------------------------
node1       192.168.5.50
node2       192.168.5.56
node3       192.168.5.57

That's it. If these requirements are ready, start the GlusterFS installation.

Setup Hostname and FQDN

The first step you must do is set up hostname and fqdn for all your Debian servers that will be used by GlusterFS. You can set up the hostname via the 'hostnamectl' command and set up FQDN (Fully Qualified Domain Name) via the '/etc/hosts' file.

To set up the hostname, enter the following command on each server.

# run on node1
sudo hostnamectl set-hostname node1.home.lan

# run on node2
sudo hostnamectl set-hostname node2.home.lan

# run on node3
sudo hostnamectl set-hostname node3.home.lan

Next, open the '/etc/hosts' file on each server using your preferred editor. For this entire tutorial, you'll be using nano editor.

sudo nano /etc/hosts

Add the following lines to the file. When you have an IP address referring to multiple hostnames, replace it with the following lines.

192.168.5.50  node1.home.lan  node1
192.168.5.56  node2.home.lan  node2
192.168.5.57  node3.home.lan  node3

Press Ctrl+x to exit, input y to confirm, then press ENTER to proceed.

Lastly, verify the fqdn on each server by issuing the following command.

hostname -f
cat /etc/hosts

You'll receive an output similar to this - On node1, the fqdn should be node1.home.lan, on node2 is node2.home.lan, and on node3 should be node3.home.lan.

setup hostname and fqdn

Setting up Disk Partition

It's recommended to use a specific drive/disk for the GlusterFS deployment. In this example, each Debian server has an additional disk '/dev/vdb' that will be used for the GlusterFS installation. And in this step, you'll learn how to set new a new disk on a Linux system via the terminal and the 'fdisk' command.

To start, issue the following 'fdisk' command to verify the list of available disks on your Debian server.

sudo fdisk -l

You'll receive an output similar to this - There are two disks available on the 'node1', the '/dev/vda' where the operating system is installed, and '/dev/vdb' which is not yet configured.

list disks

To start partitioning the '/dev/vdb' disk, enter the following fdisk command. You should get the new prompt of the fdisk tool and connect to the '/dev/vdb' disk.

sudo fdisk /dev/vdb
  • To create a new partition, enter the 'n' command.
  • Select the partition type you want to create. 'p' for primary and 'e' for extended. In this example, you'll create a primary partition, so input 'p'.
  • Within the disk '/dev/vdb', how many partitions do you want to create? In this example, you'll create one partition only, so input '1'.
  • Choose the first sector for the new partition. You can leave it as default, so press ENTER to continue.
  • The last sector is where you define how much size of the partition you want to create. This example will be around 5GB, so enter '+5GB'.
  • Lastly, enter 'w' to confirm and apply the changes you've made on the '/dev/vdb' disk.

The output 'The partition table has been altered' confirms that the changes are applied to your disk.

create partition fdisk

With this, your new partition is created, but still unusable. Because you need to format your new partition to specific filesystem formats. You can enter the following fdisk command to ensure that the new partition on the '/dev/vdb' disk is created.

sudo fdiks -l

You'll receive an output similar to this - On the '/dev/vdb' section, you will see the new partition '/dev/vdb1' is created with the size '4.7GB'.

list partition dev/vdb

Next, issue the following command to format your new partition '/dev/vdb1'. In this example, you'll format the partition to ext4 filesystem format.

sudo mkfs -t ext4 /dev/vdb1

You'll then receive an output similar to this - The new partition '/dev/vda1' is formatted as an ext4 filesystem.

formating partition

Setup Auto-Mount Partition

In this step, you'll set up auto-mount the new partition '/dev/vdb1' via the '/etc/fstab' file. You'll also create a new directory that will be used to store data in GlusterFS.

First, create a new target directory that will be used to mount the new partition '/dev/vdb1'.

# run on node1
mkdir -p /data/node1

# run on node2
mkdir -p /data/node2

# run on node3
mkdir -p /data/node3

Open the config file '/etc/fstab' using the below nano editor command.

sudo nano /etc/fstab

Add the following line to the file. With this, the new partition '/dev/vdb1' will be mounted automatically on system startup.

# for node1
/dev/vdb1    /data/node1    ext4    defaults    0    1

# for node2
/dev/vdb1    /data/node2    ext4    defaults    0    1

# for node3
/dev/vdb1    /data/node3    ext4    defaults    0    1

Next, run the below command to mount the new partition that you've configured via the '/etc/fstab' file. If no error, you're ready to go.

sudo mount -a

Lastly, run the below command to create a new directory 'brick0' on the newly mounted partition for each server.

# run on node1
mkdir -p /data/node1/brick0

# run on node2
mkdir -p /data/node2/brick0

# run on node3
mkdir -p /data/node3/brick0

Installing GlusterFS Server

In this step, you'll install the GlusterFS package on Debian servers that will be used for the GlusterFS cluster. So, be sure to run the following commands on the node1, node2, and node3 servers.

Run the below apt command to install basic dependencies on your system. Input y when prompted and press ENTER to proceed.

sudo apt install gnupg2 apt-transport-https software-properties-common

Output:

install dependencies

Download the GPG key for the GlusterFS repository via the curl command. Then, convert the new key to the file '/usr/share/keyrings/glusterfs-archive-keyring.gpg'.

curl https://download.gluster.org/pub/gluster/glusterfs/10/rsa.pub | gpg --dearmor > /usr/share/keyrings/glusterfs-archive-keyring.gpg

Next, run the below command to create new environment variables and add the GlusterFS repository to your system.

DEBID=$(grep 'VERSION_ID=' /etc/os-release | cut -d '=' -f 2 | tr -d '"')
DEBVER=$(grep 'VERSION=' /etc/os-release | grep -Eo '[a-z]+')
DEBARCH=$(dpkg --print-architecture)

echo "deb [signed-by=/usr/share/keyrings/glusterfs-archive-keyring.gpg] https://download.gluster.org/pub/gluster/glusterfs/LATEST/Debian/${DEBID}/${DEBARCH}/apt ${DEBVER} main" | sudo tee /etc/apt/sources.list.d/gluster.list

Output:

add repo

Once the GPG key and repository are added, run the below apt command to update and refresh your package index.

sudo apt update

Output:

update repo

Now install the GLusterFS server package by entering the following command. When prompted for confirmation, input y and press ENTER.

sudo apt install glusterfs-server

Output:

install glusterfs

After GlusterFS installed, run the below systemctl command utility to start and enable the GLusterFS service. With this, the GlusterFS should be running on all your servers and it's now enabled, which means the GlusterFS service will start automatically upon the system startup.

sudo systemctl start glusterd
sudo systemctl enable glusterd

Lastly, run the below systemctl command to verify the GlusterFS service and ensure that the service is running and enabled.

sudo systemctl status glusterd

If successful, you should receive an output like this - The output 'active (running)' confirms that the GlusterFS is running and the output '...; enabled;' confirms that the GlusterFS is enabled and will start automatically upon the system bootup.

start verify glusterfs service

At this point, you've now prepared a new disk/partition and installed the GLusterFS server. Next, you'll start creating and initializing the GLusterFS cluster.

Initializing Storage Pool

In this step, you'll set up the GlusterFS cluster with three Debian servers, node1, node2, and node3. You'll initialize the GlusterFS cluster from 'node1' and add the other nodes 'node2' and 'node3' to the cluster.

Before you start, ensure that each server is accessible via the hostname or fqdn. You can verify this by issuing the ping command to each server.

ping node2.home.lan
ping node3.home.lan

On the 'node1' terminal, run the below gluster command to initialize the GlusterFS cluster with the cluster member node2 and node3.

sudo gluster peer probe node2.home.lan
sudo gluster peer probe node3.home.lan

The output 'peer probe: success' confirms that the GlusterFS cluster initialization is successful.

initilize cluster storage pool

Next, move to the 'node2' terminal and issue the following command to verify the GlusterFS cluster status.

sudo gluster peer status

You should receive an output similar to this - On the 'node2', you can see the GlusterFS cluster with two peers, node1 and node3 with the status connected.

status node2

Now move to the 'node3' terminal and run the below command to verify the GLusterFS cluster status.

sudo gluster peer status

Output - On the 'node3', you can see the GlusterFS cluster with two peers, node1 and node2 with the status connected.

status node3

Additionally, you can also verify the list of pools on the GlusterFS cluster via the below command. You can run this on the node1, node2, or node3 server.

sudo gluster pool list

With this, you've now successfully initialized the GlusterFS cluster with three Debian servers. In the next step, you'll learn how to create volume on GlusterFS and how to mount the GlusterFS volume from the client machine.

Creating Replicated Volume

On GlusterFS, there are multiple types of volumes that you can create, this includes Distributed volume, Replicated volume, Distributed Replicated volume, Dispersed volume, and Distributed Dispersed volume. Check the official Documentation of GlusterFS to get details about each volume type.

In this step, you'll create a new GlusterFS volume with the type Replicated with three different GlusterFS servers. With this, your data will automatically be copied from one server to another inside the storage pool and GlusterFS cluster.

Run the below gluster command to create a new replicated volume on GlusterFS. In this example, you'll create a new volume 'testVolume' with the type Replicated between three servers node1, node2, and node3.

sudo gluster volume create testVolume replica 3 node1.home.lan:/data/node1/brick0 node2.home.lan:/data/node2/brick0 node3.home.lan:/data/node3/brick0

The output 'volume create: testVolume: success: ...' confirms that the new volume 'testVolume' is created.

Next, you must start the 'testVolume' before you can use it by issuing the following command.

sudo gluster volume start testVolume

The output 'volume start: testVolume: success' confirms that the 'testVolume' is started and ready.

Output:

create and start volume

Lastly, you can now verify details information about the 'testVolume' via the gluster command below.

sudo gluster volume info

You'll receive an output similar to this - The 'testVolume' with the type 'Replicated', default transfer is 'tcp', and three different servers node1, node2, and node3.

Output:

verify volume

With this, you've now initialized the GlusterFS cluster with three different servers and created a Replicated volume 'testVolume' on top of it. In the next step, you'll learn how to set up the client machine and munt the GLusterFS volume.

Mount GlusterFS Volume on Client

In this step, you'll learn how to mount a GlusterFS volume on the client machine, this example uses an Ubuntu/Debian-based machine with the hostname 'client'. Now, you'll mount the GlusterFS volume 'testVolume' to the client machine and set up auto-mount via '/etc/fstab' file.

First, open the '/etc/hosts' file using the following nano editor command.

sudo nano /etc/hosts

Input the following lines to the file and be sure to change the details of IP addresses and hostnames with the GLusterFS server.

192.168.5.50  node1.home.lan  node1
192.168.5.56  node2.home.lan  node2
192.168.5.57  node3.home.lan  node3

Save the file and exit the editor when finished.

Next, run the below apt command to install the 'glusterfs-client' package. To mount the GlusterFS volume, you must install the 'glusterfs-client' package on your client machine.

sudo apt install glusterfs-client

Input y when prompted and press ENTER to proceed.

install glusterfs client

After the glusterfs-client is installed, create a new directory '/data' that will be used as the target mount directory for the GlusterFS volume.

mkdir /data

Mount the GlusterFS volume 'testVolume' to the '/data' directory via the mount command below.

sudo mount.glusterfs node1.home.lan:/testVolume /data

Verify the list of mounted disks on your system via the df command below.

sudo df -h

If successful, you should see the GlusterFS volume 'testVolume' mounted to the '/data' directory.

Output:

mount glusterfs

Next, you'll set up auto-mount of the GlusterFS volume via the '/ect/fstab' file.

Open the '/etc/fstab' file using the following nano editor command.

sudo nano /etc/fstab

Add the following lines to the file. With this, the GlusterFS volume 'testVolume' will automatically be mounted upon boot.

node1.home.lan:/testVolume /data glusterfs defaults,_netdev 0 0

Save and exit the file when finished.

Now you've finished the client configuration to mount the GlusterFS volume and configured the auto-mount of GlusterFS volume via the '/etc/fstab' file. In the next step, you'll verify the replication and high-available of the GLusterFS cluster.

Test Replication and High-Availability

On the client machine, move the working directory to '/data' and create new files using the below command. This will create new files '1-15.md'.

cd /data
touch file{1..15}.md

Issue the ls command to get the list of files in the current directory.

ls

Output:

add new files

Next, move to the 'node1' terminal and go to the directory '/data/node1/brick0'. Then, issue the ls command to check the list of files and directories.

cd /data/node1/brick0
ls

You should see files '1-15.md' available on node1.

Output:

data replicated node1

Move to the 'node2' terminal and go to the directory '/data/node2/brick0'. Then, issue the ls command to check the list of files and directories.

cd /data/node2/brick0
ls

You should see files '1-15.md' available on node2.

Output:

data replicated node2

Move to the 'node3' terminal and go to the directory '/data/node3/brick0'. Then, issue the ls command to check the list of files and directories.

cd /data/node3/brick0
ls

You should see files '1-15.md' available on node3.

Output:

data replicated node3

With this, the data that you've made from the client machine is replicated to multiple servers on the GlusterFS server.

Next, to the High-Availability of GLusterFS, you can turn off or shut down the 'node1' and verify that the client machine is still connected to the GlusterFS cluster.

Move to the 'node1' terminal and run the below command to shut down the server.

sudo poweroff

Next, go to the 'node2' terminal and run the below command to check the GLusterFS cluster status.

sudo gluster peer status

You should receive an output like this - The node1 state is 'Disconnected'.

test ha

Move to the client terminal and run the below command to ensure that you're still connected to the GlusterFS cluster.

cd /data
ls

Output:

test ha

With this, the High-Availability of the GlusterFS cluster is working.

Conclusion

You've installed GlusterFS Cluster with three Debian 11 servers in this tutorial. You've also configured a new disk/partition on Linux via fdisk and configured auto-mount of Linux disk/partition via /etc/fstab file. In addition, you've learned how to create Replicated volume on GlusterFS and set up a Debian/Ubuntu-based client machine to mount the GlusterFS volume.

With this in mind, you can now leverage your GlusterFS cluster by adding more disks and servers to have a high-availability network file system accessible from your Networks. You can learn more about GlusterFS administration from the GlusterFS official Documentation.

Share this page:

1 Comment(s)