A Kubernetes Special: Even More OpenTelemetry to Chew On

I have presented my OpenTelemetry demo many times, and I still do. Each time, the audience is different. To make no two presentations the same, I always ask attendees what stack they are more interested in. I also regularly add new features for the same reason. OpenTelemetry demo regularly add new features I was a victim of the IT crisis last summer, and my company fired me, so I no longer work on Apache APISIX. They say that the Chinese ideogram for crisis contains the ideogram for opportunity. I used this opportunity to join LoftLabs. LoftLabs offers a cluster virtualization solution called vCluster. LoftLabs vCluster Hence, this batch of changes to the demo has a lot of Kubernetes-related features. From Docker Compose to Helm Docker Compose is great for demos: docker compose up, and you're good to go, but I know of no organization that uses it in production. Deploying workloads to Kubernetes is much more involved than that. I've used Kubernetes for demos in the past; typing kubectl apply -f is dull fast. In addition to GitOps, which isn't feasible for demos, the two main competitors are Helm and Kustomize. I chose the former for its ability to add dependencies. docker compose up kubectl apply -f Helm Kustomize Helm Subcharts for the Win! I started to migrate by creating my own Helm templates for infrastructure. While it worked, it was hugely inefficient. In the end, I used the following Helm Charts as sub-charts in my infra chart: Valkey Traefik OpenTelemetry Collector OpenTelemetry Operator Jaeger PostgreSQL Valkey Valkey Traefik Traefik OpenTelemetry Collector OpenTelemetry Collector OpenTelemetry Operator OpenTelemetry Operator Jaeger Jaeger PostgreSQL PostgreSQL At the start, I kept a custom Jaeger manifest because the official Jaeger Helm Chart doesn't provide a way to expose the UI outside the cluster. Then, I realized that I was using Traefik and that I could use it. It's precisely the architecture I used on Docker Compose with Apache APISIX. I did keep a custom-made manifest for Mosquitto, but that's because the official one doesn't add benefit, IMHO: you need a config file. Kubernetes Topology Besides OpenTelemetry, I want to highlight the capabilities of vCluster. For this reason, I separate the different Docker Compose services into two groups: an infrastructure group (Traefik, Jaeger, Valkey, Mosquitto, and an OpenTelemetry Collector) and an app group (PostgreSQL, Mosquitto, the OTEL Operator, an OTEL collector, and all apps). I deploy infrastructure-related pods on the host cluster and the app pods in a dedicated virtual cluster's default namespace. infrastructure app default helm upgrade --install vcluster vcluster/vcluster --namespace otel --values helm/vcluster.yaml #1 helm upgrade --install otel-infra infra --values helm/infra/values.yaml --namespace otel #2 vcluster connect vcluster #3 helm upgrade --install otel-apps apps --values helm/apps/values.yaml #4 helm upgrade --install vcluster vcluster/vcluster --namespace otel --values helm/vcluster.yaml #1 helm upgrade --install otel-infra infra --values helm/infra/values.yaml --namespace otel #2 vcluster connect vcluster #3 helm upgrade --install otel-apps apps --values helm/apps/values.yaml #4 Create the virtual cluster in the otel namespace. Deploy infrastructure pods in the otel namespace on the host cluster. Connect to the virtual cluster. I'm now in a completely isolated Kubernetes cluster. Only we know it's a virtual one. Schedule infrastructure pods in the default namespace of the vcluster virtual cluster Create the virtual cluster in the otel namespace. Create the virtual cluster in the otel namespace. otel Deploy infrastructure pods in the otel namespace on the host cluster. Deploy infrastructure pods in the otel namespace on the host cluster. otel Connect to the virtual cluster. I'm now in a completely isolated Kubernetes cluster. Only we know it's a virtual one. Connect to the virtual cluster. I'm now in a completely isolated Kubernetes cluster. Only we know it's a virtual one. Schedule infrastructure pods in the default namespace of the vcluster virtual cluster Schedule infrastructure pods in the default namespace of the vcluster virtual cluster default vcluster Traefik as an Ingress Controller When I stopped working on APISIX, I decided to see how API Gateways worked and migrated to KrakenD on Docker Compose. For basic usage, it's simple enough. KrakenD I wanted to try another one on Kubernetes. I chose Traefik because my searches mentioned it was the easiest to use in Kubernetes. As I mentioned, Traefik provides a Helm Chart, which makes it easy to install. Additionally, it integrates with OpenTelemetry. Traefik traefik: fullnameOverride: traefik deployment: replicas: 1 gateway: enabled: false rbac: enabled: true ports: web: nodePort: 30080 tracing: otlp: enabled: true #1 grpc: enabled: true #2 endpoint: collector:4317 #3 insecure: true #4 traefik: fullnameOverride: traefik deployment: replicas: 1 gateway: enabled: false rbac: enabled: true ports: web: nodePort: 30080 tracing: otlp: enabled: true #1 grpc: enabled: true #2 endpoint: collector:4317 #3 insecure: true #4 Enable OpenTelemetry Prefer gRPC over HTTP Set the collector endpoint Security is not the point of the demo Enable OpenTelemetry Prefer gRPC over HTTP Set the collector endpoint Security is not the point of the demo Exposing Services and the Ingress Class The idea of scheduling infrastructure pods on the host cluster and apps pods in the virtual cluster is excellent from an architectural point of view, but how do the latter access the former? You probably use Service to access pods inside the same cluster. Services provide a stable IP over pods that come and go. Across regular clusters, you'll need an Ingress Controller or a Gateway API implementation. Service vCluster offers a straightforward alternative. In the demo, I must: Access services on the host cluster from pods scheduled on the virtual cluster. Configure Ingresses on the virtual cluster with an Ingress class whose Ingress Controller is deployed on the host cluster. Access services on the host cluster from pods scheduled on the virtual cluster. Access services on the host cluster from pods scheduled on the virtual cluster. Configure Ingresses on the virtual cluster with an Ingress class whose Ingress Controller is deployed on the host cluster. Configure Ingresses on the virtual cluster with an Ingress class whose Ingress Controller is deployed on the host cluster. As seen above, we can configure both when we create the virtual cluster. networking: replicateServices: fromHost: - from: otel/valkey-primary #1 to: default/cache - from: otel/jaeger #1 to: default/tracing - from: otel/traefik #1 to: default/ingress - from: otel/mosquitto #1 to: default/messages sync: toHost: ingresses: enabled: true #2 networking: replicateServices: fromHost: - from: otel/valkey-primary #1 to: default/cache - from: otel/jaeger #1 to: default/tracing - from: otel/traefik #1 to: default/ingress - from: otel/mosquitto #1 to: default/messages sync: toHost: ingresses: enabled: true #2 Synchronize a Service from the host cluster to the virtual cluster. The pattern is . Ingresses deployed on the virtual cluster are collected on the Ingress Controller configured on the host cluster. Synchronize a Service from the host cluster to the virtual cluster. The pattern is . Synchronize a Service from the host cluster to the virtual cluster. The pattern is . Service from to . Ingresses deployed on the virtual cluster are collected on the Ingress Controller configured on the host cluster. Ingresses deployed on the virtual cluster are collected on the Ingress Controller configured on the host cluster. A Touch of the OpenTelemetry Collector I chose not to use an OpenTelemetry Collector in the Docker Compose setup because it didn't add anything. With the new two-cluster architecture, it was beneficial to introduce one per cluster. At first, I configured each Collector to add a custom tag regarding the cluster type, host or virtual. Then, I discovered that the Collector Helm Chart provided a Kubernetes preset. host virtual opentelemetry-collector: image: repository: "otel/opentelemetry-collector-k8s" fullnameOverride: collector enabled: true mode: deployment presets: kubernetesAttributes: enabled: true opentelemetry-collector: image: repository: "otel/opentelemetry-collector-k8s" fullnameOverride: collector enabled: true mode: deployment presets: kubernetesAttributes: enabled: true It automatically adds Kubernetes-related data. Here's the result in one of the app pods: Key Value k8s.deployment.name warehouse-uk k8s.namespace.name default k8s.node.name orbstack k8s.pod.ip 192.168.194.44 k8s.pod.name warehouse-uk-5965658cfd-sbkgm k8s.pod.start_time 2024-11-29T16:36:30Z k8s.pod.uid b6ba838e-621c-46b1-ac32-c9225c80e142 Key Value k8s.deployment.name warehouse-uk k8s.namespace.name default k8s.node.name orbstack k8s.pod.ip 192.168.194.44 k8s.pod.name warehouse-uk-5965658cfd-sbkgm k8s.pod.start_time 2024-11-29T16:36:30Z k8s.pod.uid b6ba838e-621c-46b1-ac32-c9225c80e142 Key Value Key Key Value Value k8s.deployment.name warehouse-uk k8s.deployment.name k8s.deployment.name k8s.deployment.name warehouse-uk warehouse-uk warehouse-uk k8s.namespace.name default k8s.namespace.name k8s.namespace.name k8s.namespace.name default default default k8s.node.name orbstack k8s.node.name k8s.node.name k8s.node.name orbstack orbstack orbstack k8s.pod.ip 192.168.194.44 k8s.pod.ip k8s.pod.ip k8s.pod.ip 192.168.194.44 192.168.194.44 192.168.194.44 k8s.pod.name warehouse-uk-5965658cfd-sbkgm k8s.pod.name k8s.pod.name k8s.pod.name warehouse-uk-5965658cfd-sbkgm warehouse-uk-5965658cfd-sbkgm warehouse-uk-5965658cfd-sbkgm k8s.pod.start_time 2024-11-29T16:36:30Z k8s.pod.start_time k8s.pod.start_time k8s.pod.start_time 2024-11-29T16:36:30Z 2024-11-29T16:36:30Z 2024-11-29T16:36:30Z k8s.pod.uid b6ba838e-621c-46b1-ac32-c9225c80e142 k8s.pod.uid k8s.pod.uid k8s.pod.uid b6ba838e-621c-46b1-ac32-c9225c80e142 b6ba838e-621c-46b1-ac32-c9225c80e142 b6ba838e-621c-46b1-ac32-c9225c80e142 Initializing PostgreSQL With Data The "official" PostgreSQL Docker image allows initialization of the database via SQL scripts. You only need to mount the scripts folder to the container's /docker-entrypoint-initdb.d folder. When the container starts, it will run these scripts in alphabetical order. On Kubernetes, I started to write all the scripts' content in a static ConfigMap, but I didn't find this approach maintainable. PostgreSQL Docker image /docker-entrypoint-initdb.d ConfigMap Given that I'm using a Helm Chart, I thought I could find a better way, and I did. I put all the scripts in a dedicated files/sql folder in the Helm Chart. Then, I wrote the following template to concatenate them in a ConfigMap: files/sql ConfigMap apiVersion: v1 kind: ConfigMap metadata: name: postgres-init-scripts namespace: {{ .Release.Namespace }} data: {{- $files := .Files.Glob "files/sql/*.sql" }} #1 {{- range $path, $content := $files }} #2 {{ base $path }}: > #3 {{ $content | toString | indent 4 }} #4 {{- end }} apiVersion: v1 kind: ConfigMap metadata: name: postgres-init-scripts namespace: {{ .Release.Namespace }} data: {{- $files := .Files.Glob "files/sql/*.sql" }} #1 {{- range $path, $content := $files }} #2 {{ base $path }}: > #3 {{ $content | toString | indent 4 }} #4 {{- end }} List all files under the files/sql/ folder Iterate through all files Write the file name Write the file's content List all files under the files/sql/ folder files/sql/ Iterate through all files Write the file name Write the file's content At this point, we can use the ConfigMap in the global Helm file under the PostgreSQL section: ConfigMap postgresql: primary: persistence: enabled: false initdb: scriptsConfigMap: postgres-init-scripts postgresql: primary: persistence: enabled: false initdb: scriptsConfigMap: postgres-init-scripts Kubernetes Instrumentation of Pods I recently learned that Kubernetes could instrument pods running certain technologies with OpenTelemetry. I wanted to try it out, so I added a new OpenTelemetry-free Python-based image to the overall architecture. It's a two-step process: OpenTelemetry Install the OpenTelemetry Operator Apply an Instrumentor: apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: demo-instrumentation spec: exporter: endpoint: http://collector:4318 #1 propagators: #2 - tracecontext - baggage sampler: type: always_on #3 Configure the OpenTelemetry collector endpoint. Set the propagators. I want all the traces for my demo; in production environments, you'd probably want to sample them. Install the OpenTelemetry Operator Install the OpenTelemetry Operator OpenTelemetry Operator Apply an Instrumentor: apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: demo-instrumentation spec: exporter: endpoint: http://collector:4318 #1 propagators: #2 - tracecontext - baggage sampler: type: always_on #3 Configure the OpenTelemetry collector endpoint. Set the propagators. I want all the traces for my demo; in production environments, you'd probably want to sample them. Apply an Instrumentor: Instrumentor apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: demo-instrumentation spec: exporter: endpoint: http://collector:4318 #1 propagators: #2 - tracecontext - baggage sampler: type: always_on #3 apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: demo-instrumentation spec: exporter: endpoint: http://collector:4318 #1 propagators: #2 - tracecontext - baggage sampler: type: always_on #3 Configure the OpenTelemetry collector endpoint. Set the propagators. I want all the traces for my demo; in production environments, you'd probably want to sample them. Configure the OpenTelemetry collector endpoint. Set the propagators. I want all the traces for my demo; in production environments, you'd probably want to sample them. At this point, when we schedule pods, Kubernetes will pass them through a new mutating webhook. If the pod has annotations that the instrumentation recognizes, the latter will add a sidecar that instruments the main pod with OpenTelemetry. For the Python app I mentioned above looks like the following: apiVersion: apps/v1 kind: Deployment metadata: name: recommendations labels: app: recommendations spec: replicas: 1 selector: matchLabels: app: recommendations template: metadata: annotations: instrumentation.opentelemetry.io/inject-python: "true" #1 labels: app: recommendations spec: containers: - name: recommendations image: recommendations:1.0 apiVersion: apps/v1 kind: Deployment metadata: name: recommendations labels: app: recommendations spec: replicas: 1 selector: matchLabels: app: recommendations template: metadata: annotations: instrumentation.opentelemetry.io/inject-python: "true" #1 labels: app: recommendations spec: containers: - name: recommendations image: recommendations:1.0 The mutating webhook instruments the pod with the Python-specific image. The mutating webhook instruments the pod with the Python-specific image. Here's a screenshot of the resulting traces: What's impressive is that it's all based on Kubernetes. Neither the app developer nor the container build had to do anything related to OpenTelemetry. A New Quarkus Component This last section is unrelated to Kubernetes. I'm already using Spring in two components for the demo, one instrumented with the OpenTelemetry Java agent and the other with Micrometer Tracing and compiled with GraalVM native. I wanted to demo the OpenTelemetry integration capabilities of Quarkus. You only need two dependencies for auto-instrumentation: io.quarkus quarkus-opentelemetry io.opentelemetry.instrumentation opentelemetry-jdbc io.quarkus quarkus-opentelemetry io.opentelemetry.instrumentation opentelemetry-jdbc You can customize the traces with annotations: @Path("/stocks") @Produces(MediaType.APPLICATION_JSON) public class StockLevelResource { private final StockLevelRepository repository; @Inject public StockLevelResource(StockLevelRepository repository) { this.repository = repository; } @GET @Path("/{id}") @WithSpan //1 public List stockLevels(@PathParam("id") @SpanAttribute("id") Long id) { //2 return repository.findByProductId(id); } } @Path("/stocks") @Produces(MediaType.APPLICATION_JSON) public class StockLevelResource { private final StockLevelRepository repository; @Inject public StockLevelResource(StockLevelRepository repository) { this.repository = repository; } @GET @Path("/{id}") @WithSpan //1 public List stockLevels(@PathParam("id") @SpanAttribute("id") Long id) { //2 return repository.findByProductId(id); } } Add a new span whose value is the method's name. Add a new id attribute whose value is the id's value at runtime. Add a new span whose value is the method's name. Add a new span whose value is the method's name. Add a new id attribute whose value is the id's value at runtime. Add a new id attribute whose value is the id's value at runtime. id id Summary I added a couple of new features to my existing OpenTelemetry demo: Migrating from Docker Compose to Kubernetes Installing via Helm Charts Using vCluster to isolate the app pods from the infrastructure pods Using Traefik as an Ingress Controller Installing an OpenTelemetry collector instance in each cluster Instrumenting an OpenTelemetry-free app with the Kubernetes Instrumentation Adding a new Quarkus application Migrating from Docker Compose to Kubernetes Installing via Helm Charts Using vCluster to isolate the app pods from the infrastructure pods Using Traefik as an Ingress Controller Installing an OpenTelemetry collector instance in each cluster Instrumenting an OpenTelemetry-free app with the Kubernetes Instrumentation Adding a new Quarkus application I'm always happy to add new components that feature a new stack. At the moment, I'm thinking .Net is missing from the landscape. Suggestions and PRs are welcome! Suggestions and PRs To go further: To go further: Helm OpenTelemetry Collector Chart OpenTelemetry Operator Chart Injecting Auto-instrumentation on Kubernetes vCluster What are virtual clusters? Traefik Traefik & Kubernetes "Quarkus: Using OpenTelemetry" Helm Helm OpenTelemetry Collector Chart OpenTelemetry Collector Chart OpenTelemetry Operator Chart OpenTelemetry Operator Chart Injecting Auto-instrumentation on Kubernetes Injecting Auto-instrumentation on Kubernetes vCluster vCluster What are virtual clusters? What are virtual clusters? Traefik Traefik Traefik & Kubernetes Traefik & Kubernetes "Quarkus: Using OpenTelemetry" "Quarkus: Using OpenTelemetry" Originally published at A Java Geek on April 6th, 2025 Originally published at A Java Geek on April 6th, 2025 A Java Geek