Cluster computing on the Raspberry Pi with Kubernetes

No readers like this yet.
Kurt Stam's LEGO-encased Raspberry Pi cluster

Photo by Kurt Stam CC BY-SA 4.0

Ever wanted to make your very own cloud? Now you can! All it takes is some cheap open source hardware and open source software. For about $200, I was able to set up four Raspberry Pi 2s with the Kubernetes cloud operating system using Fabric8.

Fabric8 is an open source DevOps and integration platform that works out of the box on any Kubernetes or OpenShift environment and provides continuous delivery, management, ChatOps, and a Chaos Monkey. In short, it's the best framework there is to develop and manage your microservices on. (Full disclosure: I work on the Red Hat Fabric8 project.)

Never before was there a better match between a software architecture and the hardware it runs on. The Pis are completely silent (no fan) and have a quite powerful quad-core CPU, while the microservices architecture makes each process relatively small so it can run on the 1GB RAM memory constraint of each node. You can simply add more Pis if you need more computing power. Lastly, it's just plain fun to play with the hardware instead of logging into a remote virtual machine at Amazon.

1. Required hardware

To build this four-Pi setup I used:

  • 4 Raspberry Pi 2s
  • 4 16GB MicroSD cards (Class 10)
  • 1 60W power supply with USB outlets
  • 4 short USB to Micro USB cables (for powering the Pis)
  • 4 short Cat 5 network cables
  • 1 longer Cat 5 network cable to hook into your network
  • 1 network hub (Mine is an old five-port, 10/100MBps I dusted off)
  • LEGOs (Trust me, it feels good to build your own!)

It's important to get Class 10 MicroSD cards, as this is the one component that greatly affects the speed of the system. You can easily get away with a smaller 20W power supply for four Pis, but I'm planning on adding more later so I got a bigger one. The network port on the Pi is 100MBps, so you can use your old hub.

2. Flashing the MicroSD cards

Now we need to get the operating system onto the MicroSD Cards. I'm using a Hypriot image. I tested a bunch of others, including Pidora, Redsleeve/CentOS, and Arch Linux, but concluded that Hypriot is the best for me at this point (mostly because of the native Docker support and the image being small). You can download the latest Hypriot image here.

  1. Insert the memory card in the card reader. Do not mount it. If it automounts, unmount the card. Then write the image to the card. On my OS X machine, the command for doing so is:
    sudo dd bs=1m if=hypriot-rpi-20151103-224349.img of=/dev/diskn
  2. Replace 'n' with the number of disk found using the disk utility: On Mac OS X 10.8.x Mountain Lion, "Verify Disk" (before unmounting) will display the BSD name as "/dev/disk2s1" or similar. Please double check or else you maybe formatting a disk you did not intend to format!
  3. Wait. This may take 10 minutes or so.
  4. Do this for all four MicroSD cards. You can work on your LEGO casing while waiting.

3. Power up and add DNS

You can now power up the first Pi. By default, use root/hypriot to log in. You will need to set up DNS for each of the Pis. If you don't want to hook up a monitor and keyboard to figure out its IP and MAC address, then you can use nmap. If you have a DNS service, provision it there. Otherwise, name the Pis in /etc/hosts. I named my Pis rpi-master, rpi-node-1, rpi-node-2, and rpi-node-3. After a quick reboot of each Pi, you'll see the names reflected in the cmd prompt. Note that if you skip the DNS step, your Kubernetes cluster will not work. I added the following to my /etc/hosts files on all machines:

192.168.1.9     rpi-master
192.168.1.21    rpi-node-1
192.168.1.18    rpi-node-2
192.168.1.23    rpi-node-3

4. Add swap (optional)

I added 1GB of swap space to my rpi-master because it runs a few more Kubernetes components. Some people seem concerned that frequent writes to the MicroSD card will make it fail quickly. I therefore decided to set the swappiness such that it only uses the swap as a last resort.

dd if=/dev/zero of=/swap/swapfile bs=1M count=1024
mkswap /swap/swapfile
swapon /swap/swapfile

Now you can check with top that you have a 1GB of swap space.

KiB Mem: 947468 total, 913096 used, 34372 free, 69884 buffers
KiB Swap: 1048572 total, 484 used, 1048088 free, 667248 cached

To make the swap permanent between reboots, add

/swap/swapfile none swap sw 0 0

to your /etc/fstab file. Finally, in /etc/sysctl.conf, I've set the swappiness to 1:

vm.swappiness = 1

"1" means it will only use swap when RAM use exceeds 99%.

5. Install Kubernetes

These instructions are to install the plain vanilla Kubernetes from Google. I had some issues compiling OpenShift 3 due to some 64-bit dependencies, and I will get back to that later. I would like go support both. All the code I'm using is checked in under https://github.com/Project31. If you want to compile your own Kubernetes binaries, check out my blog post about it.

5.1. Install Kubernetes master

The Kubernetes master runs the Kubernetes REST API, a scheduler, kubernetes-proxy, and a replication controller. It also uses etcd as a key-value store that is replicated with between other Kubernetes masters. For more details, please see the Kubernetes documentation.


Diagram of Kubernetes master components
Kubernetes master components.

To install these components onto the master, log in to the master node and run:

git clone git@github.com:Project31/kubernetes-installer-rpi.git
cd kubernetes-install-rpi
./build-master.sh

Verify the running Docker containers using docker ps:

CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES
d598e486daf5 hyperkube "/hyperkube proxy --m" ... ... k8s_kube-proxy...
026c19a67f86 hyperkube "/hyperkube scheduler" ... ... k8s_kube-scheduler...
8f615b87cfda hyperkube "/hyperkube controlle" ... ... k8s_kube-controller-manager....
a737d9927c03 hyperkube "/hyperkube apiserver" ... ... k8s_kube-apiserver....
0207a21ce18d etcd "etcd --data-dir=/var" ... ... k8s_etcd...
a4174bf7cb98 .../pause:0.8.0 "/pause" ... ... k8s_POD...

and the now set export KUBERNETES_MASTER=http://rpi-master:8080 so your kubectl can connect the Kubernetes REST API on the master. Now let take a look at the running pods using kubectl get pods

NAME READY STATUS RESTARTS AGE
kube-controller-rpi-master 5/5 Running 0 10s

We can see that the kube-controller-rpi-master pod contains runs five Docker containers with the Kubernetes services mentioned above.

5.2. Install Kubernetes node

The Kubernetes node only runs the Kubernetes proxy and the pods.


Kubernetes client components diagram
Kubernetes client components.

To install these components on a node, log in to the node and run:

git clone git@github.com:Project31/kubernetes-installer-rpi.git
cd kubernetes-install-rpi

Now edit the kube-procy.yaml and set --master=http://rpi-master:8080 to your Kubernetes master. Then edit the kubelet.service file and set your master's Kubernetes REST endpoint there as well (which in my case is http://192.168.1.9:8080).

Now you can run the install:

./build-worker.sh

And verify our proxy came up using docker ps:

CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES
cf4a9a2d7f35 hyperkube "/hyperkube proxy --m" 40 seconds ago Up 37 seconds k8s_kube-proxy...
d9f8f937df4d gcr.io/go... "/pause" 43 seconds ago Up 40 seconds k8s_POD.e4cc..

The proxy is running! Set export KUBERNETES_MASTER=http://rpi-master:8080 so your kubectl can connect the Kubernetes REST API on the master. Now verify the nodes are all registered:

kubectl get nodes
NAME LABELS STATUS
rpi-master kubernetes.io/hostname=rpi-master Ready
rpi-node-1 kubernetes.io/hostname=rpi-node-1 Ready
rpi-node-2 kubernetes.io/hostname=rpi-node-2 Ready
rpi-node-3 kubernetes.io/hostname=rpi-node-3 Ready

Yay, it worked!

6. Open Docker for remote connections

We need to fix up the Docker configuration on the master so that it accepts remote connections and we can deploy something to it. Open the /etc/default/docker file for editing and set the DOCKER_OPTS:

DOCKER_OPTS="-H tcp://192.168.1.9:2375 -H unix:///var/run/docker.sock --storage-driver=overlay -D"

Specify the IP address of machine, or you can user '0.0.0.0' to bind to all interfaces. Now we can remotely push Docker images to the master.

7. Deploy a simple service

Let's deploy a simple service and scale to two pods to make sure things are working correctly:

kubectl -s http://localhost:8080 run httpd --image=hypriot/rpi-busybox-httpd --port=80
kubectl scale --replicas=2 rc httpd
kubectl get pods -o wide

We see that even though we executed the command on the master, it started a pod on node-2 and node-3:

NAME READY STATUS RESTARTS AGE NODE
httpd-4v1qw 1/1 Running 0 9m rpi-node-2
httpd-qxcxu 0/1 Pending 0 16s rpi-node-3
kube-controller-rpi-master 5/5 Running 0 1d rpi-master
kube-system-rpi-node-1 1/1 Running 0 33m rpi-node-1
kube-system-rpi-node-2 1/1 Running 0 43m rpi-node-2
kube-system-rpi-node-3 1/1 Running 0 53m rpi-node-3

8. Conclusion

First of all, it works! The Kubernetes master may not have much memory left to run services, but the nodes, on the other hand, have almost an entire 1GB available for Docker containers. If you ever need more computing power, you can simply add nodes. The extremely low price of this hardware platform, combined with the Kubernetes cloud OS and the Fabric8 DevOps and iPaaS capabilities make this truly disruptive technology. It really is a cloud in a box for just $200.

Kurt Stam
Kurt Stam has been working in the enterprise integration space for the two decades. He has designed and implemented integration solutions for high-volume distributed systems in the telecommunication, financial, and travel industries. He is a core middleware developer at Red Hat where he worked on JBossESB, Apache jUDDI, S-RAMP and SOA Governance. Currently he is working on the Fabric8 project.

13 Comments

Hi Luxas! Always good to hear from you. I hope we will do more work together. My next focus is to get OpenShift 3 (k8s implementation) running on ARM. I can't wait to get my hands on some of these http://www.pine64.com boards, which are 64 bit, which OpenShift requires.

In reply to by luxas (not verified)

I'm waiting for my Pine64 too. It's really a great product with such a low price.

Will look into origin a bit after I've released a new Kubernetes on ARM version with v1.2.0 k8s. Just a quick look on origin was promising. It shouldn't be that hard to get the binary working.

On the other hand though, it will take some time to convert all addons/internal services/procedures that openshift origin has.

In reply to by KurtStam

Seems like a very interestesting thing to do, any pictures of the actual setup.

Trying to get my brain around this. Can a single container use compute resources from multiple nodes? If so, how does the storage part work?

No it cannot, but you can start multiple containers running the same service/application, if you need to handle more load. Containers should be kept stateless and immutable. If you need (shared) storage you can use Kubernetes Volumes.

In reply to by Zach

Can the PIs be a mix of Pi2 and Pi3?

If you install 32 bit binaries and RPi docker images on both then yes. You probably want to run 64 bit binaries on the RPi 3 though, and then you cannot put them into the same cluster.

In reply to by Neil A (not verified)

Hi Kurt, very nice work! I run some rpis on hypriot images and want to dig deeper into k8s. I followed your description but ran into issues when I run and start the master. The hyperkube apiserver and the etcd container are only up for a second and then gone. With command "kubectl get pods" I get the error: couldn't read version from server: Get http://rpi-master:8080/api...

Is there something I missed here?
Thanx in advance,
Dennis

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.