In my , I started the overview of my guides (link to the project at the end of this article) with the ones that explain how to setup the virtualizing platform Proxmox VE in a single common PC. In this piece I'll tackle those that deal with creating the cluster itself with Debian virtual machines and K3s. previous article Chapter 02. Forging the cluster Setting up a cluster demands considering a number of aspects: from the internal networking between the cluster nodes to the resources you're going to assign to each node, among several other details. My guides cover all of these problems, and a bit more. G017 - Virtual Networking ~ Network configuration The initial Proxmox VE setup has its networking wrapped around a default SDN (Software Defined Network) that comes with just one virtual Linux Bridge. This configuration is enough for giving connectivity to the Proxmox VE host itself and any VMs you create in it, but you cannot use it for isolating the internal networking of a cluster. You need to create another Linux Bridge, this one isolated from external connections completely. This guide explains how to do this rather easy procedure in Proxmox VE, plus a little extra explanation about managing these Linux Bridges with shell commands. Alternative network setup with OVS There's an alternative configuration you might like to try. Instead of using Linux Bridges, you could use the package to setup the whole software-based networking in your Proxmox VE setup. I explain how to setup a basic configuration in the guide , but bear in mind that OVS is a more complex system that also demands more resources to run. Open vSwitch G910 - Appendix 10 ~ Setting up virtual network with Open vSwitch G018 - K3s cluster setup 01 ~ Requirements and arrangement Here is when you have to make certain important decisions. What kind of cluster you want to run? What is the most adequate Kubernetes distribution to use? Which apps or services you'll deploy later in the cluster? You have to answer this questions, but taking into account the constraints of your host system. This guide does this, first reviewing the hardware requirements of each software, then proposing a particular cluster setup that can fit into the host system. In particular, my guide proposes Rancher K3s as the Kubernetes distribution to use, since is designed to be as lightweight as possible. Also, it takes a look at the requirements of Nextcloud, Gitea and a monitoring stack that includes Prometheus and Grafana; all of them will be services deployed in the cluster later. And, finally, the K3s cluster arrangement that I put forward here is one made up of one master (in K8s jargon)/server (for K3s) node and two worker (in K8s)/agent (for K3s) nodes. G019 - K3s cluster setup 02 ~ Storage setup At this point, in your Proxmox VE host there's just a very elemental storage setup. You need to organize its free space to make it useful for your purposes, which means you have to configure it in such a way that Proxmox VE can make use of it properly. This guide explains you exactly this, by using a combination of LVM and Proxmox VE configuration that enables different storage spaces for different purposes: from storing your VMs images and templates, to storing backups. G020 - K3s cluster setup 03 ~ Debian VM creation In my proposed setup, I use three small Debian-based virtual machines. To build each of them I use what in Proxmox VE are called VM templates, which I create in a two-stage approach. The first stage is the creation and configuration of a regular Debian VM that can be used for anything. In the G020 guide I start this first stage with the creation of the VM, set with a particular hardware configuration that has —among other details— , and the installation of a barebones Debian system in it. two network cards G021 - K3s cluster setup 04 ~ Debian VM configuration After installing Debian into the virtual machine, you still have to configure the VM properly although in a generic way so it can be useful for almost anything. This is a procedure very similar to the one applied to the Proxmox VE host, since it implies installing certain packages through apt, system hardening and sysctl adjustments, among other things. All of this is explained in this guide. G022 - K3s cluster setup 05 ~ Connecting the VM to the NUT server This guide is all about connecting the Debian VM to the UPS unit through the NUT software. This implies turning the NUT setup in the Proxmox VE host into a NUT server, and enabling a NUT client in the VM. I also explain how to setup the NUT software in such a way that can shutdown the VM when it detects a situation that requires a controlled emergency shutdown, such as when the UPS kicks in due to power loss. Of course, if you don't have an UPS unit (hint, you should if your Proxmox VE host is a regular PC) you can skip this guide. G023 - K3s cluster setup 06 ~ Debian VM template and backup With the Debian VM completely configured, you can turn it into a VM template. It's a very simple procedure done through the Proxmox VE web console, and turns the VM into a frozen-in-time compressed image that you can clone to create other VMs. The main advantage of this is mainly the smaller chunck of space it takes up from your storage capacity, and the main disadvantage is that its frozen nature means that it'll get old over time. You cannot turn a VM template back into a regular VM, at least not through the Proxmox VE web console, so you may very well keep the VM as is if you happen to have plenty of storage space. This guide also shows how to make a backup of a VM manually through the Proxmox VE web console, a process that it's too rather easy to do. G024 - K3s cluster setup 07 ~ K3s node VM template setup With the generic Debian VM available, you can do the second stage for creating a Kubernetes node VM template. The guide explains all that is required to clone a VM from the Debian VM template and make it ready to become a K3s node. If you're wondering what's the need for further specific configuration, know that a Kubernetes node in general, and a K3s one in particular, has certain particularities that must be taken into account in the supporting system. Things like disabling swap or enabling the second network card to later facilitate the internal cluster communication through the isolated Linux Bridge (prepared earlier in Proxmox VE), are things that you must leave prepared in a more specialized VM template. G024 G025 - K3s cluster setup 08 ~ K3s Kubernetes cluster setup At last, with the K3s node VM template ready, you can start creating the VMs you'll use as nodes in your K3s Kubernetes cluster. Remember that the cluster I propose is made up of three nodes: one is the master/server, while the remaining two are the workers/agents. Each node has its own particularities to attend to, such as requiring unique different hostnames or having certain ports enabled in the Proxmox VE firewall. All of these and more details are covered by this guide, plus the proper preconfiguration and installation of the K3s software in each of the VMs. At the end of this walkthrough, I also give some basic indications about how to monitor your K3s cluster with the command and the K3s log files. G025 kubectl Deploying a cluster with two or more master/server nodes If you want an even more complete K8s experience, you might like to try building a cluster with more than one master/server node. The differences in the K3s configuration are small in length but have very significant implications. The very first thing to know is that, while a single-server K3s cluster uses just a database, a multi-server one requires a full database engine like (the default database use in a standard Kubernetes installation). This has implications in aspects such as performance or firewalling, things that are covered in detail in my extra guide . sqlite etcd G908 - Appendix 08 ~ K3s cluster with two or more server nodes G026 - K3s cluster setup 09 ~ Setting up a kubectl client for remote access You don't want to be monitoring your cluster from your master node, it's not safe at all. You want to use another client system from which you can launch commands remotely against your server node. This guide explains how to install the official in a Debian-based Linux system, and how to stablish connection with your K3s cluster. It also tells you about a tool for validating Kubernetes manifest. kubectl G026 kubectl kubeval G027 - K3s cluster setup 10 ~ Deploying the MetalLB load balancer K3s comes with embedded services that provide all the essential functionality required to run a cluster. Yet in my guide I disable in the K3s configuration two particular services, the and the default . In the first case is due to the need of adjusting the metrics-server configuration, something that it's better done by deploying it later. The second service is disabled because there's a better load balancer that can be deployed in your cluster: MetalLB. The guide shows you how to deploy MetalLB with a Kustomize project, and what is a fitting configuration for this service in your cluster. G025 metrics-server load balancer G027 is an official Kubernetes tool, embedded in the command since a while ago, that allows you to compose deployments as coherent projects. In other words, you can consider it an official alternative to use Helm charts or similar tools. Kustomize kubectl Adjusting the configuration for the newer version of MetalLB Like any other software, MetalLB updated frequently. And since its version onwards it has changed how it can be configured. I've amended my guide with the appendix guide , where I explain how to do the MetalLB configuration in the new way. Stick to the configuration explained in the guide only if, for some reason, you need or want to work with an older version of MetalLB. 0.13.0 G027 G912 - Appendix 12 ~ Adapting MetalLB config to CR G027 G028 - K3s cluster setup 11 ~ Deploying the metrics-server service The default is the other service that was configured as disabled in your K3s cluster. The guide explains how to deploy it with a Kustomize project, while also indicating the small configuration change that demanded deploying this service later. This service is the one that makes the works, giving you stats of the CPU and RAM resources consumed by the nodes and pods deployed in your cluster. metrics-server G028 kubectl top G029 - K3s cluster setup 12 ~ Setting up cert-manager and wildcard certificate At this point you'll have a fully functional cluster, but there's still a service that is very common to have in a Kubernetes setup: the . With this service you can manage certificates, such as the one you'll need later to enable secure connections to other services, such as Nextcloud or Gitea, you'll see deployed in later guides. In this guide you'll see how to deploy cert-manager and also how to use to generate a self-signed certificate you can use later. cert-manager G029 There's a particular functionality that cert-manager doesn't provide, which is syncing a certificate's secret between different namespaces. To help you with this task, there's another software called that you can deploy in your cluster to take care of this in a more automated way. The guide also explains you the setup and use of this other utility service. Reflector G030 - K3s cluster setup 13 ~ Deploying the Kubernetes Dashboard One way to see what's going on in your cluster is to use commands. A more convenient one is by looking through graphical interfaces, and K8s also has its own official one known as the . My explains you how to deploy and access it from your client system, and also hints you about how to navigate through this dashboard. kubectl Kubernetes Dashboard G030 kubectl G031 - K3s cluster setup 14 ~ Enabling the Traefik dashboard One of the embedded services that come included in any K3s installation is the Traefik ingress controller. In the cluster setup explained in my guides, it's running just fine but know that it has also its own graphic dashboard. With the guide you can see the basic way of enabling access to it but in a secure manner, since the dashboard itself doesn't come with any kind of login control. G031 In the next chapter With your K3s cluster fully ready and able, its time to give it something to do for real. What about deploying some useful servers like Nextcloud or Gitea, or even a Prometheus-based monitoring stack? In my guides I did just that, and I'll tell you about it in my next article. Link to the guide series Small homelab K8s cluster on Proxmox VE

Building a Small K8s Cluster on a Single PC - Chapter 2 - Forging the Cluster.

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Building a Small K8s Cluster on a Single PC - Chapter 1 - A Host is Rising.

Building a Small K8s Cluster on a Single PC - Chapter 1 - A Host is Rising.

Building Elixir Applications as systemd Services

Install Deb-Multimedia Repositories on Debian and Kali Linux

Launching a Python Webserver for Mobile Development

Building a Small K8s Cluster on a Single PC - Chapter 1 - A Host is Rising.

Building a Small K8s Cluster on a Single PC - Chapter 1 - A Host is Rising.

Building Elixir Applications as systemd Services

Install Deb-Multimedia Repositories on Debian and Kali Linux

Launching a Python Webserver for Mobile Development

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps