Improve Kubernetes Network Security with Cilium

The Promise of Microservices There are two main benefits of using a microservices architecture. Enable distributed teams to work independently on parts of a distributed system, thus making Conway’s Law work in our favor. Tame distributed systems complexity by offloading more and more cross-cutting concerns from application code to the underlying infrastructure. In a world where microservices architecture for server-side workloads is the dominating paradigm and more and more compute runs on Kubernetes, we have a chance to truly fulfill the microservices technical and organizational promise. This article will show how to, using Cilium, tackle the authorization concern and move to push it to the underlying platform from the application code. You will find this article helpful if you are a developer working with Kubernetes, Kubernetes administrator or simply curious to learn about modern cloud-native patterns. What is Cilium Cilium is open-source software for providing, securing and observing network connectivity between container workloads — cloud-native, and fueled by the revolutionary Kernel technology eBPF. Source: https://cilium.io/ If you want to try Cilium yourself, check out their excellent Interactive Tutorial What is eBPF eBPF is a revolutionary technology with origins in the Linux kernel that can run sandboxed programs in an operating system kernel. It is used to safely and efficiently extend the capabilities of the kernel without requiring to change kernel source code or load kernel modules. To learn more about eBPF, visit Introduction to eBPF Here is a great video with Liz Rice explaining eBPF in detail. Below diagram shows how eBPF works on a high level Use Case: Granular Authorization Control Let’s imagine a scenario where your REST API consists of multiple endpoints exposing flight booking related resources. This REST API is deployed to a managed K8s cluster, let’s say GKE (Google Kubernetes Engine) and is often accessed by other microservices running in the cluster as well as some external services. From a security point of view you want to follow Zero Trust Security and the Principle of Least Privilege and to achieve this you need to tightly control and verify access to your API and only expose those endpoints that are essential for calling services and not more. Kubernetes Network Policies can take us halfway there. Network Policies Kubernetes network policies define network traffic rules for pods running in a cluster. We are going to focus on Cilium and show how it can provide enhanced and more powerful policies The below diagram shows more information about network policies. However, there is one problem. Our flight booking service exposes multiple REST endpoints and Kubernetes Network policies work only on IP:PORT combination. This means that each service running in the cluster will have access to all endpoints even if it doesn’t need it. This clearly violates the Principle of least privilege. Network Policies Improved Cilium addresses this issue by introducing a CRD CiliumNetworkPolicy which adds the missing functionality and enables us to decoratively create rules governing access to various endpoints of our API. As a side note, from architectural point of view, the same could be achieved with an API Management Gateway such as KONG, but this is a different approach and works only with HTTP services, whereas Cilium being a lower level solution supports Kafka, Databases and more. Here is a sample CiliumNetworkPolicy YAML file strictly allowing only traffic from pods with selected labels to use GET verb on the /flights resource. apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "readflights" spec: description: "Allow HTTP GET /flights from env=prod, app=flights_board to app=flights_service" endpointSelector: matchLabels: app: flights_service ingress: - fromEndpoints: - matchLabels: env: prod app: flights_board toPorts: - ports: - port: "80" protocol: TCP rules: http: - method: "GET" path: "/flights" Cilium also supports DNS Policy where we can for example allow only incoming external traffic from a load balancer address or pattern matching to a name of service hosted externally. What are the benefits? Setting rules on the HTTP level (Layer 7) enables offloading of the authorization concerns for APIs to Kubernetes instead of encoding the rules in the application itself. The benefits of this approach are: ability to change the authorization rules independently from application code base development possibility to separate the application code pipeline from the rules pipeline enabling teams to collaborate ability to deploy another instance of the same API in a container image, but with different labels and rules which may depend on namespaces or different conditions standardized and centrally controlled security aspect Demo Scenario You can follow along without installing anything on your local machine. This demo will show how to secure access to our example flight booking service using CiliumNetworkPolicy. Prerequisites We are going to use a clean Ubuntu 20.04 instance on Katacoda, so no need to install anything locally. #1 Spin up Katacoda environment Activate Ubuntu 20.04 Playground on Katacoda and follow the steps below. #2 Install k3s on a Ubuntu instance We are going to use a small and fast Kubernetes distribution from Rancher called k3s. This will enable us to spin up a fresh Kubernetes cluster very fast and proceed with the next steps. curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='--flannel-backend=none --disable-network-policy' sh - Set KUBECONFIG environmental variable to point to k3s config file, so we can talk to the cluster via kubectl which is already pre-installed on the Katacoda environment. export KUBECONFIG=/etc/rancher/k3s/k3s.yaml #3 Install the Cilium CLI If you need help installing Cilium, please refer to their excellent documentation. curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz{,.sha256sum} sha256sum --check cilium-linux-amd64.tar.gz.sha256sum sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin rm cilium-linux-amd64.tar.gz{,.sha256sum} #4 Install Cilium on the cluster cilium install Cilium can take a moment to activate so we will use this command to Wait for Cilium to fully start. cilium status --wait #5 Deploy sample go API Let’s deploy a minimalistic Go REST API where we can easily test CiliumNetworkPolicy in action. kubectl apply -f https://raw.githubusercontent.com/Piotr1215/go-sample-api/master/k8s/deployment.yaml To see the API open port 31234 in the Katacoda terminal The API has 3 simple GET endpoints / returns “HOME Page” /version returns “VERSION Page” /about returns “ABOUT Page” #6 Check service connectivity Create a test BusyBox pod and check connectivity to go-api service kubectl run -it --rm debug \ --image=radial/busyboxplus:curl \ --restart=Never \ -- curl -w "\n" http://go-api-svc Let’s break down this command: kubectl run - starts a new pod the -it flag ensures that we can interact with the pod and send commands to the container running inside --rm instructs Kubernetes to remove the pod right after it exits curl -w "\n" http://go-api-svc calls a go-api service taking advantage of Kubernetes DNS and service discovery mechanism After running this command you should see HOME Page returned to the terminal. #7 Apply Network Policy Let’s apply policy that allows only traffic from pods with label app:version_ready to GET endpoint of the go-api pod. apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "readflights" spec: description: "Allow HTTP GET /version from app=version_reader to type=service" endpointSelector: matchLabels: type: service ingress: - fromEndpoints: - matchLabels: app: version_reader toPorts: - ports: - port: "8080" protocol: TCP rules: http: - method: "GET" path: "/version"kubectl apply -f https://raw.githubusercontent.com/Piotr1215/go-sample-api/master/k8s/cilium-policy.yaml #8 Check if the connectivity works If our policy works correctly, we shouldn’t be able to access the service any longer. kubectl run -it --rm debug \ --image=radial/busyboxplus:curl \ --restart=Never \ --timeout=15s \ -- curl -w "\n" http://go-api-svc The above command will result in a timeout. If you don’t want to wait for a timeout, you can create a new terminal session with the + icon on the top. #9 Identity Aware Policy In order to grant access to a pod to the /version endpoint, we have to label it appropriately with app=version_reader. This will enable the identity-aware policy, where instead of targeting pods via their IP:PORT combo, we can use K8s labels too. Leveraging labels instead of IPs works well with the ephemeral nature of Kubernetes infrastructure. kubectl run -it --rm debug2 \ --image=radial/busyboxplus:curl \ --labels app=version_reader \ --restart=Never \ -- curl -w "\n" http://go-api-svc/version This should print out VERSION Page. Let’s try to access the /about endpoint from the same pod. Will it work? kubectl run -it --rm debug2 \ --image=radial/busyboxplus:curl \ --labels app=version_reader \ --timeout=15s \ --restart=Never \ -- curl -w "\n" http://go-api-svc/about Conclusion We have just scratched the surface of what Cilium is capable of, but I believe that focusing on a practical use case helps us learn actual skills with Cilium, rather than learning about Cilium. The current trend with the cloud-native ecosystem is to embrace eBPF in more and more scenarios. This technology is currently used by Google, Facebook, SUSE, AWS and many others as a powerful and flexible low-level solution that addresses a much better set of existing challenges. Cilium bridges the abstraction gap between low-level eBFP primitives and end-users and IMO is one of the most promising cloud-native projects. Also Published Here The Promise of Microservices There are two main benefits of using a microservices architecture. Enable distributed teams to work independently on parts of a distributed system, thus making Conway’s Law work in our favor. Tame distributed systems complexity by offloading more and more cross-cutting concerns from application code to the underlying infrastructure. Enable distributed teams to work independently on parts of a distributed system, thus making Conway’s Law work in our favor. Conway’s Law Conway’s Law Tame distributed systems complexity by offloading more and more cross-cutting concerns from application code to the underlying infrastructure. offloading more and more cross-cutting concerns from application code to the underlying infrastructure. In a world where microservices architecture for server-side workloads is the dominating paradigm and more and more compute runs on Kubernetes, we have a chance to truly fulfill the microservices technical and organizational promise. This article will show how to, using Cilium, tackle the authorization concern and move to push it to the underlying platform from the application code. You will find this article helpful if you are a developer working with Kubernetes, Kubernetes administrator or simply curious to learn about modern cloud-native patterns. What is Cilium Cilium is open-source software for providing, securing and observing network connectivity between container workloads — cloud-native, and fueled by the revolutionary Kernel technology eBPF. Source: https://cilium.io/ https://cilium.io/ https://cilium.io/ If you want to try Cilium yourself, check out their excellent Interactive Tutorial If you want to try Cilium yourself, check out their excellent Interactive Tutorial If you want to try Cilium yourself, check out their excellent Interactive Tutorial Interactive Tutorial Interactive Tutorial What is eBPF eBPF is a revolutionary technology with origins in the Linux kernel that can run sandboxed programs in an operating system kernel. It is used to safely and efficiently extend the capabilities of the kernel without requiring to change kernel source code or load kernel modules. To learn more about eBPF, visit Introduction to eBPF eBPF is a revolutionary technology with origins in the Linux kernel that can run sandboxed programs in an operating system kernel. It is used to safely and efficiently extend the capabilities of the kernel without requiring to change kernel source code or load kernel modules. To learn more about eBPF, visit Introduction to eBPF eBPF is a revolutionary technology with origins in the Linux kernel that can run sandboxed programs in an operating system kernel. It is used to safely and efficiently extend the capabilities of the kernel without requiring to change kernel source code or load kernel modules. To learn more about eBPF, visit Introduction to eBPF Introduction to eBPF Introduction to eBPF Here is a great video with Liz Rice explaining eBPF in detail. great video great video Below diagram shows how eBPF works on a high level Use Case: Granular Authorization Control Let’s imagine a scenario where your REST API consists of multiple endpoints exposing flight booking related resources. This REST API is deployed to a managed K8s cluster, let’s say GKE (Google Kubernetes Engine) and is often accessed by other microservices running in the cluster as well as some external services. From a security point of view you want to follow Zero Trust Security and the Principle of Least Privilege and to achieve this you need to tightly control and verify access to your API and only expose those endpoints that are essential for calling services and not more. Zero Trust Security Zero Trust Security Principle of Least Privilege Principle of Least Privilege Kubernetes Network Policies can take us halfway there. Network Policies Kubernetes network policies define network traffic rules for pods running in a cluster. network policies network policies We are going to focus on Cilium and show how it can provide enhanced and more powerful policies We are going to focus on Cilium and show how it can provide enhanced and more powerful policies We are going to focus on Cilium and show how it can provide enhanced and more powerful policies Cilium Cilium The below diagram shows more information about network policies. However, there is one problem. Our flight booking service exposes multiple REST endpoints and Kubernetes Network policies work only on IP:PORT combination. This means that each service running in the cluster will have access to all endpoints even if it doesn’t need it. This clearly violates the Principle of least privilege. This clearly violates the Principle of least privilege. Network Policies Improved Cilium addresses this issue by introducing a CRD CiliumNetworkPolicy which adds the missing functionality and enables us to decoratively create rules governing access to various endpoints of our API. CRD CRD CiliumNetworkPolicy As a side note, from architectural point of view, the same could be achieved with an API Management Gateway such as KONG, but this is a different approach and works only with HTTP services, whereas Cilium being a lower level solution supports Kafka, Databases and more. As a side note, from architectural point of view, the same could be achieved with an API Management Gateway such as KONG, but this is a different approach and works only with HTTP services, whereas Cilium being a lower level solution supports Kafka, Databases and more. As a side note, from architectural point of view, the same could be achieved with an API Management Gateway such as KONG , but this is a different approach and works only with HTTP services, whereas Cilium being a lower level solution supports Kafka, Databases and more. KONG KONG Here is a sample CiliumNetworkPolicy YAML file strictly allowing only traffic from pods with selected labels to use GET verb on the /flights resource. apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "readflights" spec: description: "Allow HTTP GET /flights from env=prod, app=flights_board to app=flights_service" endpointSelector: matchLabels: app: flights_service ingress: - fromEndpoints: - matchLabels: env: prod app: flights_board toPorts: - ports: - port: "80" protocol: TCP rules: http: - method: "GET" path: "/flights" apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "readflights" spec: description: "Allow HTTP GET /flights from env=prod, app=flights_board to app=flights_service" endpointSelector: matchLabels: app: flights_service ingress: - fromEndpoints: - matchLabels: env: prod app: flights_board toPorts: - ports: - port: "80" protocol: TCP rules: http: - method: "GET" path: "/flights" Cilium also supports DNS Policy where we can for example allow only incoming external traffic from a load balancer address or pattern matching to a name of service hosted externally. DNS Policy DNS Policy What are the benefits? Setting rules on the HTTP level (Layer 7) enables offloading of the authorization concerns for APIs to Kubernetes instead of encoding the rules in the application itself. The benefits of this approach are: ability to change the authorization rules independently from application code base development possibility to separate the application code pipeline from the rules pipeline enabling teams to collaborate ability to deploy another instance of the same API in a container image, but with different labels and rules which may depend on namespaces or different conditions standardized and centrally controlled security aspect ability to change the authorization rules independently from application code base development possibility to separate the application code pipeline from the rules pipeline enabling teams to collaborate ability to deploy another instance of the same API in a container image, but with different labels and rules which may depend on namespaces or different conditions standardized and centrally controlled security aspect Demo Scenario You can follow along without installing anything on your local machine. This demo will show how to secure access to our example flight booking service using CiliumNetworkPolicy. Prerequisites We are going to use a clean Ubuntu 20.04 instance on Katacoda, so no need to install anything locally. #1 Spin up Katacoda environment Activate Ubuntu 20.04 Playground on Katacoda and follow the steps below. Ubuntu 20.04 Playground Ubuntu 20.04 Playground #2 Install k3s on a Ubuntu instance We are going to use a small and fast Kubernetes distribution from Rancher called k3s . This will enable us to spin up a fresh Kubernetes cluster very fast and proceed with the next steps. k3s k3s curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='--flannel-backend=none --disable-network-policy' sh - curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='--flannel-backend=none --disable-network-policy' sh - Set KUBECONFIG environmental variable to point to k3s config file, so we can talk to the cluster via kubectl which is already pre-installed on the Katacoda environment. kubectl export KUBECONFIG=/etc/rancher/k3s/k3s.yaml export KUBECONFIG=/etc/rancher/k3s/k3s.yaml #3 Install the Cilium CLI If you need help installing Cilium, please refer to their excellent documentation. If you need help installing Cilium, please refer to their excellent documentation. If you need help installing Cilium, please refer to their excellent documentation . excellent documentation excellent documentation curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz{,.sha256sum} sha256sum --check cilium-linux-amd64.tar.gz.sha256sum sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin rm cilium-linux-amd64.tar.gz{,.sha256sum} curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz{,.sha256sum} sha256sum --check cilium-linux-amd64.tar.gz.sha256sum sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin rm cilium-linux-amd64.tar.gz{,.sha256sum} #4 Install Cilium on the cluster cilium install cilium install Cilium can take a moment to activate so we will use this command to Wait for Cilium to fully start. cilium status --wait cilium status --wait #5 Deploy sample go API Let’s deploy a minimalistic Go REST API where we can easily test CiliumNetworkPolicy in action. kubectl apply -f https://raw.githubusercontent.com/Piotr1215/go-sample-api/master/k8s/deployment.yaml kubectl apply -f https://raw.githubusercontent.com/Piotr1215/go-sample-api/master/k8s/deployment.yaml To see the API open port 31234 in the Katacoda terminal 31234 The API has 3 simple GET endpoints / returns “HOME Page” /version returns “VERSION Page” /about returns “ABOUT Page” / returns “HOME Page” /version returns “VERSION Page” /about returns “ABOUT Page” #6 Check service connectivity Create a test BusyBox pod and check connectivity to go-api service BusyBox BusyBox kubectl run -it --rm debug \ --image=radial/busyboxplus:curl \ --restart=Never \ -- curl -w "\n" http://go-api-svc kubectl run -it --rm debug \ --image=radial/busyboxplus:curl \ --restart=Never \ -- curl -w "\n" http://go-api-svc Let’s break down this command: kubectl run - starts a new pod the -it flag ensures that we can interact with the pod and send commands to the container running inside --rm instructs Kubernetes to remove the pod right after it exits curl -w "\n" http://go-api-svc calls a go-api service taking advantage of Kubernetes DNS and service discovery mechanism kubectl run - starts a new pod kubectl run the -it flag ensures that we can interact with the pod and send commands to the container running inside -it --rm instructs Kubernetes to remove the pod right after it exits --rm curl -w "\n" http://go-api-svc calls a go-api service taking advantage of Kubernetes DNS and service discovery mechanism curl -w "\n" http://go-api-svc After running this command you should see HOME Page returned to the terminal. HOME Page #7 Apply Network Policy Let’s apply policy that allows only traffic from pods with label app:version_ready to GET endpoint of the go-api pod. apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "readflights" spec: description: "Allow HTTP GET /version from app=version_reader to type=service" endpointSelector: matchLabels: type: service ingress: - fromEndpoints: - matchLabels: app: version_reader toPorts: - ports: - port: "8080" protocol: TCP rules: http: - method: "GET" path: "/version"kubectl apply -f https://raw.githubusercontent.com/Piotr1215/go-sample-api/master/k8s/cilium-policy.yaml apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "readflights" spec: description: "Allow HTTP GET /version from app=version_reader to type=service" endpointSelector: matchLabels: type: service ingress: - fromEndpoints: - matchLabels: app: version_reader toPorts: - ports: - port: "8080" protocol: TCP rules: http: - method: "GET" path: "/version"kubectl apply -f https://raw.githubusercontent.com/Piotr1215/go-sample-api/master/k8s/cilium-policy.yaml #8 Check if the connectivity works If our policy works correctly, we shouldn’t be able to access the service any longer. kubectl run -it --rm debug \ --image=radial/busyboxplus:curl \ --restart=Never \ --timeout=15s \ -- curl -w "\n" http://go-api-svc kubectl run -it --rm debug \ --image=radial/busyboxplus:curl \ --restart=Never \ --timeout=15s \ -- curl -w "\n" http://go-api-svc The above command will result in a timeout. If you don’t want to wait for a timeout, you can create a new terminal session with the + icon on the top. If you don’t want to wait for a timeout, you can create a new terminal session with the + icon on the top. If you don’t want to wait for a timeout, you can create a new terminal session with the + icon on the top. #9 Identity Aware Policy In order to grant access to a pod to the /version endpoint, we have to label it appropriately with app=version_reader . This will enable the identity-aware policy, where instead of targeting pods via their IP:PORT combo, we can use K8s labels too. Leveraging labels instead of IPs works well with the ephemeral nature of Kubernetes infrastructure. /version app=version_reader kubectl run -it --rm debug2 \ --image=radial/busyboxplus:curl \ --labels app=version_reader \ --restart=Never \ -- curl -w "\n" http://go-api-svc/version kubectl run -it --rm debug2 \ --image=radial/busyboxplus:curl \ --labels app=version_reader \ --restart=Never \ -- curl -w "\n" http://go-api-svc/version This should print out VERSION Page . Let’s try to access the /about endpoint from the same pod. Will it work? VERSION Page /about kubectl run -it --rm debug2 \ --image=radial/busyboxplus:curl \ --labels app=version_reader \ --timeout=15s \ --restart=Never \ -- curl -w "\n" http://go-api-svc/about kubectl run -it --rm debug2 \ --image=radial/busyboxplus:curl \ --labels app=version_reader \ --timeout=15s \ --restart=Never \ -- curl -w "\n" http://go-api-svc/about Conclusion We have just scratched the surface of what Cilium is capable of, but I believe that focusing on a practical use case helps us learn actual skills with Cilium, rather than learning about Cilium. The current trend with the cloud-native ecosystem is to embrace eBPF in more and more scenarios. This technology is currently used by Google, Facebook, SUSE, AWS and many others as a powerful and flexible low-level solution that addresses a much better set of existing challenges. Cilium bridges the abstraction gap between low-level eBFP primitives and end-users and IMO is one of the most promising cloud-native projects. Also Published Here Here