Kubernetes on Raspberry Pi with kubeadm: Part 1: Docker, kubeadm, kubelet, kubectl

For this article we will be starting the process of installing Kubernetes on a Raspberry Pi. To keep things in easy as possible I have split this article in two parts. The first the TL;DR version to get you up and running as quick as possible and the second provides additional context around WHY you are doing what you are doing. All instructions assume you are running Ubuntu on Raspberry Pi 3 or 4.

TL;DR

1. Install Docker.

$ sudo apt install docker.io -y

2. [OPTIONAL] Add your user to the Docker group.

$ sudo groupadd docker
$ sudo usermod -aG docker $USER

Log out and back in for this to take effect.

3. Check cgroups config. If your output matches the below move on to step 4. If your driver is already set to systemd and these warnings are not present proceed to step 5.

$ sudo docker info | grep -i cgroup
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support

4. Configure cgroups.

a. Update the driver

$ sudo mkdir /etc/docker
$ cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

b. Enable cgroups

$ sudo sed -i '$ s/$/ cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1/' /boot/firmware/cmdline.txt

5. Configure Bridged traffic.

$ cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

$ cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
$ sudo sysctl --system

6. Install Kubeadm, Kubelet, Kubectl

$ sudo apt-get update
$ sudo apt-get install -y apt-transport-https ca-certificates curl
$ sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

$ echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

$ sudo apt-get update
$ sudo apt-get install -y kubelet kubeadm kubectl
$ sudo apt-mark hold kubelet kubeadm kubectl

Deep[ish] Dive

So what exactly just happened? Let’s go through each step again.

1. Install Docker. This is pretty straight forward and we are simply using the apt package manager to install Docker as your container runtime. While Kubernetes does support other runtimes (CRI-O, containerd, etc), Docker is a pretty simple starting point as most people are already familiar with it.

Keep in mind that Docker has been deprecated as of Kubernetes v1.20 and it will be completely removed in a later release. 

$ sudo apt install docker.io -y

2. [OPTIONAL] Warning: The docker group grants privileges equivalent to the root user.

Add your user to the Docker group. If you want to be able to run docker commands without the need to prefix them with sudo then proceed. Be aware of the security implications this may have as noted in the docker documentation. 

$ sudo groupadd docker
$ sudo usermod -aG docker $USER

3. Here is where things will get a little more fun…ish…well, interesting at least maybe not fun. We are checking to see what cgroups driver is being used and how it is configured. If you aren’t familiar with cgroups the wiki page gives a nice quick overview. The general idea is that cgroups can be used to isolate resources. Similar [not the same] to how you can increase the RAM in a virtual machine by giving it more memory from what is available on the host. 

$ sudo docker info | grep -i cgroup
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support

As noted in the Kubernetes docs, cgroupfs is not recommended for us with kubeadm (the tool we will be using to install our Kuberentes cluster). Spoiler, this is because kubeadm manages the kubelet as a systemd service.

4. Configure cgroups

a. Docker uses the daemon.json to configure the Docker daemon. We will use a Here doc to write this JSON out to the file. We create the directory first in case it does not already exist. This configuration will tell the Docker daemon to:

  • Change the cgroupdriver to systemd
  • Use an overlay2 storage driver
  • Use the json-file log-driver for logging and pass it the option max-size
$ sudo mkdir /etc/docker
$ cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

b. We will now tell the OS which cgroups we want to enable. We do this by updating the cmdline.txt (used to update kernel parameters).

  • Cpusets provides a Linux kernel mechanism to constrain CPUs
  • Memory provides a Linux kernel mechanism to constrain memory
  • Enable swap accounting (monitor/restrict swap usage)

The sed utility is used to quickly edit the cmdline.txt in place and appended the necessary parameters. It is important that these get appended to the end of the existing line and not on their own line. This is handled with the below where

  • The first $ signifies the last line of the file
  • s/ is used to match a pattern against a regular expression
  • /$/ matches the end of the line
  • cgroup_enable, cgroup_memory, swapaccount are the parameters we are adding
  • /boot/firmware/cmdline.txt is the file we are editing
$ sudo sed -i '$ s/$/ cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1/' /boot/firmware/cmdline.txt

5. Here we will first check that the br_netfilter module is loaded. This module is used to filter bridged traffic and send it to iptables for processing. sysctl is used to load the values from all system directories.

$ cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

$ cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
$ sudo sysctl --system

6. Now the easy part, we use apt-get to install a few pre-reqs

  • apt-transport-https for to connect to https repos.
  • ca-certificates is a set of CA certs from Mozilla. It is the same CA list used in Firefox
  • curl for command line requests

We then add a new repo to allow of downloading packages from kubernetes.io. Next we install the kubelet, kubectl and kubeadm CLI tools. Finally, we set them to hold to prevent them from being accidentally updated when automatically.

$ sudo apt-get update
$ sudo apt-get install -y apt-transport-https ca-certificates curl
$ sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

$ echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

$ sudo apt-get update
$ sudo apt-get install -y kubelet kubeadm kubectl
$ sudo apt-mark hold kubelet kubeadm kubectl