Building 28-Core Raspberry Pi Cluster from Scratch: Release The Kraken

This is the second installment of my 'How I built' trilogy detailing the build process of all 3 of my clusters. Read the first here . Octo pains In October 2018, more than a year after , I had officially outgrown the cluster. I started noticing the performance bottlenecks started when I tried to run Wordpress on it, where a single page load of a newly installed Wordpress blog takes about Nevertheless, all that was unsurprising given the measly single-core 700MHz processor on each node. building my first cluster Octopi 10 seconds! At that point in time, I've graduated from university then and secured a job as a Data Scientist + Full-Stack Engineer at a startup and I thought to myself: What better a time to build myself a new cluster? Kraken The second cluster was named . Kraken The (/ˈkrɑːkən/)[1] is a legendary cephalopod-like sea monster of gigantic size in Scandinavian folklore. ( ) kraken Wikipedia The Kraken is basically a much bigger and monstrous octopus, hence the name, symbolizing an evolution of the first cluster. It was built out of 7 Raspberry Pi 3Bs powered by a single USB charger. With my first pay check, I had initially intended to build a cluster of 8 nodes but once again I was unable to realize this idea. The maximum number of ports on a consumer-grade network switch was 8, enough for 7 nodes and a cable to the router. finally cursed | Item | TP-Link 8-port Switch | TP-Link 16-Port Smart Switch | |-------------|-----------------------|------------------------------| | Cost (SGD)* | S$26.46 | S$195.03 | | Cost (USD) | $18.99 | $139.97 | SGD/USD exchange rate is 0.71770 as of the time of writing * An alternative was to use commercial-grade network switches with 16 ports but that was obviously was out of the question (and budget). Cluster Specifications | | 7 × Raspberry Pi 3 Model B | Overall | |---------|----------------------------|-------------------------| | CPU | 4 × Cortex-A53 1.2 GHz | 28 × Cortex-A53 1.2 GHz | | RAM | 1GB | 7GB | | USB | 4 × USB 2.0 | 28 × USB 2.0 | | Storage | 16GB MicroSD Class 10 | 128GB* Class 10 | Comprised of 1 × 32GB, 6 × 16GB * The Kraken cluster is born! (No that's not my ceiling) The total damage my wallet sustained from Kraken was or in , which was virtually all I had left for the month after subtracting expenses and education loan repayments. S$626.63 $449.73 USD Parts list | Item | Qty | Cost (SGD) | Cost (USD) * | |--------------------------------------------------|-----|------------|--------------| | Raspberry Pi 3 Model B | 7 | S$420.00 | $301.43 | | Anker PowerPort 10 60W 10-port USB Charger | 1 | S$32.58 | $23.38 | | TP-Link TL-SF1008D 8-Port Ethernet Switch | 1 | S$20.00 | $14.35 | | Cat 5e 30cm cable | 7 | S$3.50 | $2.51 | | Cat 5e 1m cable | 1 | S$2.00 | $1.44 | | MicroUSB charging cable | 7 | S$21.00 | $15.07 | | 16GB Class 10 MicroSD Card | 7 | S$70.00 | $50.24 | | 32GB Class 10 MicroSD Card | 1 | S$16.00 | $11.48 | | ^ GeauxRobot Raspberry Pi 7-Layer Dog Bone Stack | 1 | S$41.55 | $29.82 | | Total | | S$626.63 | $449.73 | SGD/USD exchange rate is 0.71770 as of the time of writing ^ Optional but handsome * Something noteworthy was that I chose a 32GB MicroSD card as storage for the first node as I had intended it to be the m̶a̶s̶t̶e̶r̶ main node of a Docker swarm setup and anticipated that I'd need additional storage for building and deploying Docker images. Here you can see Kraken (top) and Octopi (bottom) co-existing, in a photo shot with a potato Kraken gigabit upgrade One year later in September 2019, I found myself regularly hitting the bandwidth limit of 100Mbps on the built-in ethernet ports of Raspberry Pi 3Bs, especially when doing large file transfers in or out of the machines, where transfer speeds hover around a sad . 8MB/s I looked around and found Jeff Geerling's where I learnt that I could use USB Gigabit ethernet adapters to increase the bandwidth to slightly over 200Mbps and so I bought a bunch of cheap Chinese USB Gigabit ethernet adapters and a gigabit switch and got down to upgrading. blog post Additional parts list | Item | Qty | Cost (SGD) | Cost (USD) * | |-----------------------------------------------|-----|------------|--------------| | TP-Link TL-SG108 8-Port Gigabit Switch | 1 | S$36.00 | $26.56 | | USB 3.0 Gigabit RJ45 Ethernet Network Adapter | 7 | S$45.36 | $32.55 | | Total | | S$82.36 | $59.11 | SGD/USD exchange rate is 0.71770 as of the time of writing * After the upgrade, this is how Kraken looked like: This upgrade brings the total cost of the cluster to or . S$708.99 $508.84 USD Benchmarks Running iperf before the upgrade, we see a maximum bandwidth of . 93.1 Mbps ~ ❯ iperf -c 192.168.3.11 ------------------------------------------------------------ Client connecting to 192.168.3.11, TCP port 5001 TCP window size: 129 KByte (default) ------------------------------------------------------------ [ 4] 192.168.3.71 port 57041 connected with 192.168.3.11 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 111 MBytes 93.1 Mbits/sec local Running iperf after installing the gigabit adapter, we see a maximum bandwidth of ! 224 Mbps ~ ❯ iperf -c 192.168.3.11 ------------------------------------------------------------ Client connecting to 192.168.3.11, TCP port 5001 TCP window size: 145 KByte (default) ------------------------------------------------------------ [ 4] 192.168.3.71 port 57298 connected with 192.168.3.11 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 268 MBytes 224 Mbits/sec local With that simple mod, I've gained . However, these are still only speeds as typical usage scenarios involve writing data received from network to disk, but iperf only receives data from the network and . 131Mbps per node theoretical does not write data to disk Some caveats Sustained full utilization of this newly available bandwidth is , even in web serving. It will mostly aid in transferring large assets like images faster for the first load, after which the images will be cached by users browsers. highly unlikely in typical usage Additionally, the actual bandwidth is still constrained by the notorious shared USB 2.0 bus in the Raspberry Pi models 1 to 3. For the uninitiated, the 480Mbps theoretical unidirectional bandwidth in a single USB 2.0 bus is shared between the Ethernet Port, the SD card slot and all USB ports. Bandwidth distribution may look like this: | Operation | Read bandwidth | Write bandwidth | |----------------------------|----------------|-----------------| | Actual available bandwidth | 200Mbps | 200Mbps | | SD Card | -80Mbps | -80Mbps | | Ethernet disconnected | -0Mbps | -0Mbps | | No USB usage | -0Mbps | -0Mbps | | Available for GbE | 120Mbps | 120Mbps | Even though the tabulated numbers give the impression that there is no performance gain from this upgrade, they represent the worst case scenario where typically, one would expect mostly reads and few writes in web-servers. Bandwidth distribution should typically look like this in real-world: | Operation | Read bandwidth | Write bandwidth | |----------------------------|----------------|-----------------| | Actual available bandwidth | 300Mbps | 100Mbps | | SD Card | -100Mbps | -50Mbps | | Ethernet disconnected | -0Mbps | -0Mbps | | No USB usage | -0Mbps | -0Mbps | | Available for GbE | 200Mbps | 50Mbps | Should you build this cluster? This build is for you if you: You wish to learn Docker and its associated frameworks You do not want to spend a fortune on procuring multiple Raspberry Pi 4Bs to set up a cluster Want to explore Kubernetes This build is not for you if you: Are already familiar with the Docker ecosystem Are looking for high-performance home setups My two cents I highly recommend building this cluster for anyone who's keen on getting into Docker and Kubernetes for 2 main reasons. Firstly, this cluster is compatible with the latest versions of officially supported Docker images. Raspberry Pi 3Bs run on the CPU architecture, which happens to be the current lowest common denominator among Arm processors today. armv7 Latest Arm processors ( ) are backward-compatible with all code written and compiled on . In contrast, processors are not backward-compatible with processors (Raspberry Pi 1 and 2), so they are essentially in the process of being phased out from the community. arm64 armv7 arm64 armv6 Secondly, this cluster would be perfect for most but the most bandwidth demanding applications, from hosting your own blog, file sync service, media library managers, link-shorteners, note-taking applications and so on. The only time it would not perform as intended is when your application demands heavy sustained writes, such as when encoding videos, given the USB 2.0 bus bottlenecks in the Raspberry Pi 3, which is resolved only in Raspberry Pi 4. All in all, building a Raspberry Pi 3 cluster is, in my opinion, the most cost-effective way of learning Docker and clustering in general, and will remain so in the foreseeable future. So if you are just getting into Docker, I'd highly recommend this build. What's next? In my next post in this series, I shall introduce the cluster which I have built specifically for high I/O applications, capable of real-time video transcoding and streaming of HEVC -> h264 videos. Leviathan Originally published at https://ikarus.sg on July 6, 2020.