Stage 5. Redundant, Distributed and Persistent storage for Swarm with GlusterFS
Obsolete. See http://www.disasterproject.com/2016/07/stage-6-redesign-of-project-using.html for Docker 1.12 and next.
In this stage we can raise containers with ldap services, but data in containers don't are permanent and ldap database is loose when a container is stopped.
We need persistent storage and this can be ever the same. This is a important problem because the ldap containers can be raised in any host and the data of the ldap service must be persistent across sessions.
Configuring DNS resolution
Name resolution is critical in cluster environment. In this lab design have three sources for name resolution: Weave, Consul and external.
Do it easy with dnsmasq, three configuration files define how to access to the sources:
- Resolve .consul domains (for example for hosts) in the consul dns service in 8600 port in each host: server=/consul/127.0.0.1#8600
- .weave.local domains (usable for containers in weave network) to the weave dns service in 5353 port in each host: server=/weave.local/127.0.0.1#5353
- All the other names must be resolve by a external source (using the gateway): server=/*/GWIP
Distributed Storage with GlusterFS
GlusterFS is a filesystem developed to support cluster environments. Is really powerful and could be deployed easily in our Lab. The mission of glusterfs is give us a coherent filesystem usable for each server without problems.
Gluster can be defined in four modes: replication, distribution and disperse, each mode has it own features and can be mixed with the other modes (more information in glusterfs documentation).
- Replication: A file can be saved in n different storage peers depending in the number of replicas we need. It give us protection about storage lost.
- Distribution: File A and B can be saved in different storage peers.
- Stripped: File A can be broken in pieces and save in different storage peers. The main goal is make faster access to information.
- Disperse: The files are broken in pieces, add recovery information. The volumen has a redundancy value which define how many peers can be lost.
We will choose a disperse configuration where we can lost two peers. This give us a folder in each server called /persistent-storage to be published in the docker containers.
Changes in the Vagrant file and commands
After launch Vagrant up, there are some steps to activate the distributed storage and must be executed in the first master node:
- Add the other peers: sudo gluster peer probe PEER
- Create the volume in disperse mode, see the disperse and redundancy values: sudo gluster volume create persistent-storage disperse 3 redundancy 2 transport tcp ...
- Activate the volume: sudo gluster volume start persistent-storage
And then we can mount the shared folders (mount /persistent-storage)
Snapshot with Vagrant
Vagrant can save the actual state of the virtual machines in snapshots, with all the infrastructure deployed we use snapshots to make tests faster like see in the next figure:
Stage 5 Command Line ExecutionThese are the steps to raise the virtual machines and the video with full execution.
$ vagrant box remove elasticmmldap/base_docker $ vagrant box remove elasticmmldap/base_swarm $ vagrant box remove elasticmmldap/base_weave $ if [ -d base_docker ];then rm -rf base_docker;fi $ git clone -b base_docker https://github.com/aescanero/elasticmmldap base_docker $ cd base_docker ~/base_docker$ vagrant up ~/base_docker$ vagrant halt ~/base_docker$ vagrant package ~/base_docker$ vagrant box add package.box --name elasticmmldap/base_docker ~/base_docker$ vagrant destroy -f ~/base_docker$ cd .. $ if [ -d base_swarm ];then rm -rf base_swarm;fi $ git clone -b base_swarm https://github.com/aescanero/elasticmmldap base_swarm $ cd base_swarm ~/base_swarm$ vagrant up ~/base_swarm$ vagrant halt ~/base_swarm$ vagrant package ~/base_swarm$ vagrant box add package.box --name elasticmmldap/base_swarm ~/base_swarm$ vagrant destroy -f ~/base_swarm$ cd .. $ if [ -d base_weave ];then rm -rf base_weave;fi $ git clone -b base_weave https://github.com/aescanero/elasticmmldap base_weave $ cd base_weave ~/base_weave$ vagrant up ~/base_weave$ vagrant halt ~/base_weave$ vagrant package ~/base_weave$ vagrant box add package.box --name elasticmmldap/base_weave ~/base_weave$ vagrant destroy -f ~/base_weave$ cd .. $ git clone https://github.com/aescanero/elasticmmldap elasticmmldap $ cd elasticmmldap ~/elasticmmldap$ vagrant up ~/elasticmmldap$ for i in swarm-master-2 swarm-node-1 swarm-node-2 swarm-node-3; do vagrant ssh swarm-master-1 -c "sudo gluster peer probe $i";done ~/elasticmmldap$ vagrant ssh swarm-master-1 -c "sudo gluster volume create persistent-storage disperse 3 redundancy 2 transport tcp swarm-master-1:/glusterfs swarm-master-2:/glusterfs swarm-node-1:/glusterfs swarm-node-2:/glusterfs swarm-node-3:/glusterfs force" ~/elasticmmldap$ vagrant ssh swarm-master-1 -c "sudo gluster volume start persistent-storage" ~/elasticmmldap$ for i in swarm-master-1 swarm-master-2 swarm-node-1 swarm-node-2 swarm-node-3; do vagrant ssh $i -c "sudo mount /persistent-storage";done
A video with the execution:
To "save" the lab in snapshots to recover later:
~/elasticmmldap$ for i in swarm-master-1 swarm-master-2 swarm-node-1 swarm-node-2 swarm-node-3; do vagrant push $i;done
All the code of this lab in: https://github.com/aescanero/elasticmmldap