Can Glusterfs Continue Working With One Node Down

In one of my earlier posts on GlusterFS, we went through the steps on how to setup a Distributed Storage Volume, where the end result was to have scalable storage, where size was the requirement.

What will we be doing today with GlusterFS?

Today, we will be going through the steps on how to setup a Replicated Storage Volume with GlusterFS, where we will have 3 GlusterFS Nodes, and using the replication factor of 3.

Replication Factor of 3:

In other words, having 3 copies of our data and in our case, since we will have 3 nodes in our cluster, a copy of our data will reside on each node.

What about Split-Brain:

In Clustering, we get the term Split-Brain, where a node dies or leaves the cluster, the cluster reforms itself with the available nodes and then during this reformation, instead of the remaining nodes staying with the same cluster, 2 subset of cluster are created, and they are not aware of each other, which causes data corruption, here's a great resource on Split-Brain

To prevent Split-Brain in GlusterFS, we can setup a Arbiter Volume. In a Replica Count of 3 and Arbiter count of 1: 2 Nodes will hold the replicated data, and the 1 Node which will be the Arbiter node, will only host the file/directory names and metadata but not any data. I will write up an article on this in the future.

Getting Started:

Let's get started on setting up a 3 Node Replicated GlusterFS. Each node will have an additional drive that is 50GB in size, which will be part of our GlusterFS Replicated Volume. I will also be using Ubuntu 16.04 as my linux distro.

Preparing DNS Resolution:

I will install GlusterFS on each node, and in my setup I have the following DNS entries:

• gfs01 (10.0.0.2)
• gfs02 (10.0.0.3)
• gfs03 (10.0.0.4)

Preparing our Secondary Drives:

I will be formatting my drives with XFS. Listing our block volumes:

1 2 3 4

$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vdb 253:16 0 46.6G 0 disk vda 253:0 0 18.6G 0 disk /

Creating the FileSystem with XFS, which we will be running on each node:

1	$ mkfs.xfs /dev/vdb

Then creating the directories where our bricks will reside, and also add an entry to our /etc/fstab so that our disk gets mounted when the operating system boots:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

# node: gfs01 $ mkdir /gluster/bricks/1 -p $ echo '/dev/vdb /gluster/bricks/1 xfs defaults 0 0' >> /etc/fstab $ mount -a $ mkdir /gluster/bricks/1/brick # node: gfs02 $ mkdir /gluster/bricks/2 -p $ echo '/dev/vdb /gluster/bricks/2 xfs defaults 0 0' >> /etc/fstab $ mount -a $ mkdir /gluster/bricks/2/brick # node: gfs03 $ mkdir /gluster/bricks/3 -p $ echo '/dev/vdb /gluster/bricks/3 xfs defaults 0 0' >> /etc/fstab $ mount -a $ mkdir /gluster/bricks/3/brick

After this has been done, we should see that the disks are mounted, for example on node: gfs01:

1 2 3 4

$ df -h Filesystem Size Used Avail Use% Mounted on /dev/vda 18G 909M 17G 3% / /dev/vdb 47G 80M 47G 1% /gluster/bricks/1

Installing GlusterFS on Each Node:

Installing GlusterFS, repeat this on all 3 Nodes:

1 2 3

$ apt update && sudo apt upgrade -y $ apt install xfsprogs attr glusterfs-server glusterfs-common glusterfs-client -y $ systemctl enable glusterfs-server

In order to add the nodes to the trusted storage pool, we will have to add them by using gluster peer probe. Make sure that you can resolve the hostnames to the designated IP Addresses, and that traffic is allowed.

1 2 3

$ gluster peer probe gfs01 $ gluster peer probe gfs02 $ gluster peer probe gfs03

Now that we have added our nodes to our trusted storage pool, lets verify that by listing our pool:

1 2 3 4 5

$ gluster pool list UUID Hostname State f63d0e77-9602-4024-8945-5a7f7332bf89 gfs02 Connected 2d4ac6c1-0611-4e2e-b4af-9e4aa8c1556d gfs03 Connected 6a604cd9-9a9c-406d-b1b7-69caf166a20e localhost Connected

Great! All looks good.

Create the Replicated GlusterFS Volume:

Let's create our Replicated GlusterFS Volume, named gfs:

1 2 3 4 5 6 7

$ gluster volume create gfs \ replica 3 \ gfs01:/gluster/bricks/1/brick \ gfs02:/gluster/bricks/2/brick \ gfs03:/gluster/bricks/2/brick volume create: gfs: success: please start the volume to access data

Now that our volume is created, lets list it to verify that it is created:

1 2	$ gluster volume list gfs

Now, start the volume:

1 2	$ gluster volume start gfs volume start: gfs: success

View the status of our volume:

1 2 3 4 5 6 7

$ gluster volume status gfs Status of volume: gfs Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick gfs01:/gluster/bricks/1/brick 49152 0 Y 6450 Brick gfs02:/gluster/bricks/2/brick 49152 0 Y 3460 Brick gfs03:/gluster/bricks/3/brick 49152 0 Y 3309

Next, view the volume inforation:

1 2 3 4 5 6 7 8 9 10 11 12

$ gluster volume info gfs Volume Name: gfs Type: Replicate Volume ID: 6f827df4-6df5-4c25-99ee-8d1a055d30f0 Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: gfs01:/gluster/bricks/1/brick Brick2: gfs02:/gluster/bricks/2/brick Brick3: gfs03:/gluster/bricks/3/brick

Security:

From a GlusterFS level, it will allow clients to connect by default. To authorize these 3 nodes to connect to the GlusterFS Volume:

1	$ gluster volume set gfs auth.allow 10.0.0.2,10.0.0.3,10.0.0.4

Then if you would like to remove this rule:

1	$ gluster volume set gfs auth.allow *

Mount the GlusterFS Volume to the Host:

Mount the GlusterFS Volume to each node, so we will have to mount it to each node, and also append it to our /etc/fstab file so that it mounts on boot:

1 2	$ echo 'localhost:/gfs /mnt glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab $ mount.glusterfs localhost:/gfs /mnt

Verify the Mounted Volume:

Check the mounted disks, and you will find that the Replicated GlusterFS Volume is mounted on our /mnt partition.

1 2 3 4 5

$ df -h Filesystem Size Used Avail Use% Mounted on /dev/vda 18G 909M 17G 3% / /dev/vdb 47G 80M 47G 1% /gluster/bricks/1 localhost:/gfs 47G 80M 47G 1% /mnt

You will note that GlusterFS Volume has a total size of 47GB usable space, which is the same size as one of our disks, but that is because we have a replicated volume with a replication factor of 3: (47 * 3 / 3)

Now we have a Storage Volume which has 3 Replicas, one copy on each node, which allows us Data Durability on our Storage.

butlerhanted.blogspot.com

Source: https://blog.ruanbekker.com/blog/2019/03/05/setup-a-3-node-replicated-storage-volume-with-glusterfs/

Share this post