High Availability of gluster block device

If you haven’t gone through the above “chapter 1” yet, I recommend to check it first.


I hope your first gluster based block volume creation was Easy Like Sunday Morning with the help of Starting with gluster-block. In this blog lets try to understand, how to achieve high availability (HA) for block volume using gluster-block

The Idea is to have multiple paths from initiator/client to the same block volume hosted in multiple server nodes, hence if one server node goes faulty, one among remaining servers will be able to serve IO.


  • 3 Fedora-29 nodes for servers
  • 1 Fedora-29 node as client
  • Open 24007(for glusterd) 24010(gluster-blockd) 3260(iscsi targets) 111(rpcbind) ports and glusterfs service in your firewall on all the server nodes

Creating replica block hosting volume

# On all the 3 server nodes, start glusterd
[server ~]# systemctl start glusterd

# From <Host-IP1> server node, create a trusted storage pool
[server ~]# gluster peer probe <Host-IP2>
[server ~]# gluster peer probe <Host-IP3>

# create the replica 3 hosting volume
[server ~]# gluster volume create hosting-volume replica 3 <Host-IP1>:/brick <Host-IP2>:/brick <Host-IP3>:/brick force

# set the block profile on replica 3 volume to tune it with block recommendations
[server ~]# gluster volume set hosting-volume group gluster-block

# start the block hosting volume
[server ~]# gluster volume start hosting-volume

Creating the HA 3 block volume

# on all the 3 server nodes, start gluster-blockd
[server ~]# systemctl start gluster-blockd

# lets now create HA 3 block volume from the above created hosting-volume
[server ~]# gluster-block create hosting-volume/block-volume ha 3 <Host-IP1>,<Host-IP2>,<Host-IP3> 1GiB –json-pretty

Attaching the block volume on client

# Install the initiator and multipath utils
[client ~]# dnf install iscsi-initiator-utils device-mapper-multipath
[client ~]# systemctl start iscsid.service

# below we set multipathing in Active/Passive mode
[client ~]# modprobe dm_multipath
[client ~]# mpathconf –enable

# please add the below configuration at the end of
# /etc/multipath.conf file
# (please note that we need at least tcmu-runner >= 1.4.0)

devices {
    device {
            vendor "LIO-ORG"
            user_friendly_names "yes" # names like mpatha
            path_grouping_policy "failover" # one path per group
            hardware_handler "1 alua"
            path_selector "round-robin 0"
            failback immediate
            path_checker "tur"
            prio "alua"
            no_path_retry 120
            rr_weight "uniform"

# restart/reload the multipathd, to take changes into effect
[client ~]# systemctl restart multipathd

# note the available devices
[client ~]# lsblk

# discovery … & login … to the block volume
[client ~]# iscsiadm -m discovery -t st -p <Host-IP1> -l

# note the mapper device mapping to 3 paths
[client ~]# lsblk

# check the multipath mappings
[client ~]# multipath -ll

# format the mapper device
[client ~]# mkfs.xfs /dev/mapper/mpathX

# now mount it
[client ~]# mount /dev/mapper/mpathX /mnt/

Learn how to gracefully logout and delete the block volume:

Demo (More detailed)

Watch this demo, to see how multipath switch happens when a node goes faulty and the volume is still consistent and made available for applications.


  • Please join our mailing list
  • To ask a question or start a discussion, you can also raise an issue
  • IRC: #gluster-devel on Freenode

https://github.com/gluster/gluster-block/issues/215 (Ubuntu) 


My next blog will talk about ‘How to resize Block-Volume’

Thanks for your time!

One thought on “High Availability of gluster block device

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s