Docker Swarm Raspberry Pi Cluster

Raspberry Pi Cluster running Docker Swarm

Raspberry Pi Cluster running Docker Swarm

Cluster Hosts

• 4 x Raspberry Pi 4 8GB
• Raspberry Pi OS Lite
• Debian version: 11 (bullseye)
• Samsung Bar Plus 128GB 400MB/s USB 3.1 Flash storage
• UFW firewall
• KeepAlived
• GlusterFS
• Syslog
• SNMP
• Cockpit
• Docker

Docker Swarm

• Portainer
• Traefik

Configuration Steps

Prepare Image on SD card

1. Raspberry Pi Imager to write image to SD card on Windows
2. Setup ssh, hostname and user name options in the advance config
3. Repeat for each Pi or sd card

Configure Raspberry Pi

1. Install SD cards, reboot Pi’s, wait to see them on the network. You can use your routers dhcp list or a Network Scanner to scan your network for the devices

2. Use Putty to ssh into the pi

3. Set Static IP

   sudo nano /etc/dhcpcd.conf
   
   static ip_address=192.168.1.14
   static routers=192.168.1.1
   static domain_name_servers=192.168.1.1

4. Reboot

5. Set up ssh Keys *Optional

   ssh-keygen -t rsa
   ssh-copy-id clusteradmin@pi-cluster-1.local
   ssh-copy-id clusteradmin@pi-cluster-2.local
   ssh-copy-id clusteradmin@pi-cluster-3.local
   ssh-copy-id clusteradmin@pi-cluster-4.local

6. Install Firewall https://www.addictedtotech.net/how-to-install-and-configure-ufw-firewall-on-a-raspberry-pi-4/

   sudo apt update
   sudo apt install ufw -y

7. Add Firewall Rules

   sudo ufw allow ssh
   sudo ufw allow 2376/tcp comment "docker swarm"
   sudo ufw allow 2377/tcp comment "docker swarm"
   sudo ufw allow 7946/tcp comment "docker swarm"
   sudo ufw allow 7946/udp comment "docker swarm"
   sudo ufw allow 4789/udp comment "docker swarm"
   sudo ufw allow 9789/udp comment "docker swarm"

   From https://florianmuller.com/build-a-raspberry-pi-4-docker-swarm-cluster-with-four-nodes-and-deploy-traefik-with-portainer

   sudo ufw enable

   From https://www.addictedtotech.net/how-to-install-and-configure-ufw-firewall-on-a-raspberry-pi-4/

8. Update the Pi

   sudo apt update
   sudo apt full-upgrade
   sudo rpi-eeprom-update -a
   sudo reboot

   From https://github.com/raspberrypi/firmware/issues/1559

9. Install Docker, cifs and others

   for host in pi-cluster-1.local pi-cluster-2.local pi-cluster-3.local pi-cluster-4.local; do ssh clusteradmin@$host curl -sSL https://get.docker.com | sh; done

   or on each node:

   sudo curl -sSL https://get.docker.com | sh
   sudo usermod -aG docker clusteradmin

   On each node:

   sudo apt-get install samba-common smbclient samba-common-bin smbclient cifs-utils

10. Configure Docker Swarm

    sudo docker swarm init --advertise-addr 192.168.1.11

    Output: Swarm initialized: current node (ddddddd) is now a manager.

    To add a worker to this swarm, run the following command:

    docker swarm join –token SWMTKN-token 192.168.1.11:2377

    To add a manager to this swarm, run ‘docker swarm join-token manager’ and follow the instructions.

    sudo docker swarm join --token SWMTKN-token 192.168.1.11:2377

11. Install GlusterFS From https://florianmuller.com/setup-a-scalable-high-availability-glusterfs-network-filesystem-with-docker-swarm-on-ubuntu-server-20-04-lts &&

    sudo apt install software-properties-common glusterfs-server -y
    sudo systemctl start glusterd
    sudo systemctl enable glusterd
    sudo ufw allow 24007/tcp
    sudo ufw allow 24008/tcp
    sudo ufw allow 24009/tcp
    sudo ufw allow 24010/tcp
    sudo ufw allow 49152/tcp
    sudo ufw allow 49153/tcp
    sudo ufw allow 38465:38467/tcp
    sudo ufw allow 111
    sudo ufw allow 49153:60999/tcp

    From https://florianmuller.com/setup-a-scalable-high-availability-glusterfs-network-filesystem-with-docker-swarm-on-ubuntu-server-20-04-lts

    gluster peer probe pi-cluster-1; gluster peer probe pi-cluster-2; gluster peer probe pi-cluster-3; gluster peer probe pi-cluster-4;

    On each node:

    sudo mkdir -p /gluster/dockerclusterfiles
    sudo gluster volume create cluster-gfs replica 4 pi-cluster-1:/gluster/dockerclusterfiles pi-cluster-2:/gluster/dockerclusterfiles pi-cluster-3:/gluster/dockerclusterfiles pi-cluster-4:/gluster/dockerclusterfiles force
    sudo gluster volume add-brick cluster-gfs replica 4 pi-cluster-1:/gluster/dockerclusterfiles pi-cluster-2:/gluster/dockerclusterfiles pi-cluster-3:/gluster/dockerclusterfiles pi-cluster-4:/gluster/dockerclusterfiles force
    	sudo gluster volume start cluster-gfs

    On each node:

    sudo mkdir -p /mnt/docker
    
    sudo chown -R clusteradmin:clusteradmin /mnt/docker/
    sudo chown root:docker /mnt/docker/
    
    Sudo -s
    echo 'localhost:/cluster-gfs /mnt/docker glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab

    From https://florianmuller.com/setup-a-scalable-high-availability-glusterfs-network-filesystem-with-docker-swarm-on-ubuntu-server-20-04-lts https://stanislas.blog/2018/10/how-to-mount-local-glusterfs-volume-boot-fstab-systemd-fix/

    localhost:/gv0 /srv glusterfs defaults,_netdev 0 0
    localhost:/gv0 /srv glusterfs defaults,_netdev,noauto,x-systemd.automount 0 0
    localhost:/cluster-gfs /mnt/docker glusterfs defaults,_netdev,noauto,x-systemd.automount,backupvolfile-server=localhost 0 0

    From https://stanislas.blog/2018/10/how-to-mount-local-glusterfs-volume-boot-fstab-systemd-fix/

12. Traefik Install

    mkdir /mnt/docker/traefik 
    mkdir /mnt/docker/traefik/data 
    mkdir /mnt/docker/traefik/logs
    
    touch /mnt/docker/traefik/config.yml 
    touch /mnt/docker/traefik/traefik-swarm-compose.yml 
    touch /mnt/docker/traefik/logs/traefik.log 
    touch /mnt/docker/traefik/data/traefik.yml 
    touch /mnt/docker/traefik/data/acme.json 
    sudo chmod 600 /mnt/docker/traefik/data/acme.json
    
    docker network create --driver=overlay proxy
    
    sudo apt update && sudo apt install apache2-utils -y
    
    echo $(htpasswd -nb htpadmin XXXXXXXX ) | sed -e s/\$/\$\$/g

    From https://florianmuller.com/build-a-raspberry-pi-4-docker-swarm-cluster-with-four-nodes-and-deploy-traefik-with-portainer

13. Keepalived On each node:

    sudo apt-get install keepalived
    
    sudo lsmod |grep -P '^ip_vs\s'||(echo "modprobe ip_vs">>/etc/modules &&modprobe ip_vs)

    From https://github.com/matthewpayne/keepalived-pi-cluster

    sudo ufw allow to 224.0.0.18 comment 'VRRP Broadcast' 
    sudo ufw allow from 192.168.1.11 comment 'VRRP Router'
    sudo ufw allow from 192.168.1.12 comment 'VRRP Router'
    sudo ufw allow from 192.168.1.13 comment 'VRRP Router'
    sudo ufw allow from 192.168.1.14 comment 'VRRP Router'
    sudo ufw allow from 192.168.1.10 comment 'VRRP Router'
    sudo ufw allow from 192.168.1.9 comment 'VRRP Router'

    From https://24htech.asia/how-to-setup-highly-available-nginx-with-keepalived-in-linux-s277046.html

    Set Priority

    sudo docker node update pi-cluster-4 --label-add KEEPALIVED_PRIORITY=100
    sudo docker node update pi-cluster-3 --label-add KEEPALIVED_PRIORITY=200
    sudo docker node update pi-cluster-2 --label-add KEEPALIVED_PRIORITY=300
    sudo docker node update pi-cluster-1 --label-add KEEPALIVED_PRIORITY=300

    From https://hub.docker.com/r/lolhens/keepalived-swarm

    Deploy the swarm:

    keepalivedsblob/main/keepalived.yml

    From:https://4devnotes.wordpress.com/2019/01/30/docker-swarm-with-keepalive/

    /etc/keepalived/keepalived.conf
    sudo systemctl enable keepalived.service
    sudo systemctl start keepalived.service

14. Syslog Rsyslog

    sudo apt-get install rsyslog
    sudo nano /etc/rsyslog.conf
    
    	Comment out:
    	#*.emerg                                :omusrmsg:*
    	Add:
    	*.* @@192.168.1.x:514

    From https://rubysash.com/operating-system/linux/setup-rsyslog-client-forwarder-on-raspberry-pi/

    sudo service rsyslog restart
    sudo service rsyslog stop
    
    TEST: 
    logger -p daemon.emerg "DANGER WILL ROBINSON!!!"

    From https://rubysash.com/operating-system/linux/setup-rsyslog-client-forwarder-on-raspberry-pi/

15. Cockpit

    sudo apt install cockpit
    sudo ufw allow 9090/tcp 
    
    systemctl status cockpit.socket

    From https://opensource.com/article/21/5/raspberry-pi-cockpit

16. SNMP

    sudo apt-get install bc snmpd snmp snmp-mibs-downloader

    From https://community.librenms.org/t/raspberry-pi-monitoring-issue/2616

    sudo ufw allow 161/udp 
    sudo nano /etc/snmp/snmpd.conf
    
    Udp:<IP>:161
    
    rocommunity  public default -V systemonly
    rocommunity6 public default -V systemonly
    
    extend .1.3.6.1.4.1.2021.7890.1 distro /usr/local/bin/distro
    extend cpuTemp0 /usr/local/bin/cputemp
    extend raspberry /usr/bin/sudo /bin/sh /etc/snmp/raspberry.sh
    extend docker /etc/snmp/docker-stats.sh
    
    # grant access to the right information, by default it is too restrictive
    view systemonly included .1.3.6.1.2
    view systemonly included docker
    view systemonly included raspberry
    view systemonly included cpuTemp0
    
    
    # grant access to distro information
    view systemonly included .1.3.6.1.4.1.2021.7890.1
    
    # grant access to the temp information
    view systemonly included .1.3.6.1.4.1.8072.1.3.2

    From https://github.com/smford/rpi-snmpd-configuration

    sudo systemctl restart snmpd

    Testing:

    snmpwalk -c public 192.168.1.11 -v2c
    snmpwalk -c public 192.168.1.14 -v2c NET-SNMP-EXTEND-MIB::nsExtendObjects

    From https://bigdanzblog.wordpress.com/2015/01/03/installing-snmp-onto-a-raspberry-pi/

Mark is a tech savy IT profesional