Key Management Service (KMS) is a way to manage your secrets in a more secure manner. But before diving into KMS, let’s do a quick primer on Kubernetes Secrets. What Are Kubernetes Secrets? A Secret is any sensitive information—such as a database password, an API token, or cloud credentials. In most applications, you separate such configuration data from the actual application logic. Secret That’s where Kubernetes Secrets come in. You can store confidential configuration as a separate Kubernetes resource called a Secret. Kubernetes Secrets Secret Where Are Secrets Stored? Kubernetes stores secrets in etcd, which is the key-value store used by the Kubernetes control plane. Unless encryption at rest is enabled, the secrets are stored in plaintext in etcd—more on this later. etcd plaintext How to Create Kubernetes Secrets? Let’s walk through three ways to create secrets in Kubernetes. For reference, I’m using an OpenShift cluster with the oc CLI, but everything here applies to kubectl as well. three ways oc kubectl Create a Namespace % oc new-project kms % oc new-project kms a. Creating a Secret from a File % echo -n 'somepassword' > password.txt % oc create secret generic kms-file-secret \ --from-file=password=password.txt % echo -n 'somepassword' > password.txt % oc create secret generic kms-file-secret \ --from-file=password=password.txt In the first line, we create a file password.txt with the contents somepassword. The second command creates a secret named kms-file-secret by reading the contents of that file. password.txt somepassword kms-file-secret To inspect the created secret: % oc get secret kms-file-secret -o yaml % oc get secret kms-file-secret -o yaml You'll see: % oc get secret kms-file-secret -o yaml apiVersion: v1 data: password: c29tZXBhc3N3b3Jk kind: Secret metadata: creationTimestamp: "2025-05-16T13:35:30Z" name: kms-file-secret namespace: kms resourceVersion: "807159" uid: 66691c51-4a6c-4a15-a6b1-1d2de6d8fff7 type: Opaque % oc get secret kms-file-secret -o yaml apiVersion: v1 data: password: c29tZXBhc3N3b3Jk kind: Secret metadata: creationTimestamp: "2025-05-16T13:35:30Z" name: kms-file-secret namespace: kms resourceVersion: "807159" uid: 66691c51-4a6c-4a15-a6b1-1d2de6d8fff7 type: Opaque That string c29tZXBhc3N3b3Jk is just a base64 encoded version of somepassword. c29tZXBhc3N3b3Jk base64 somepassword b. Creating a Secret from a Literal Value % oc create secret generic kms-literal-secret \ --from-literal=password=somepassword % oc create secret generic kms-literal-secret \ --from-literal=password=somepassword This creates a secret by passing the value directly as a literal string. The result is the same base64 encoded password in the data field: base64 % oc get secret kms-literal-secret -o yaml apiVersion: v1 data: password: c29tZXBhc3N3b3Jk kind: Secret metadata: creationTimestamp: "2025-05-16T14:05:34Z" name: kms-literal-secret namespace: kms resourceVersion: "819400" uid: d4ecd109-cd95-4afe-b2e8-77b2da3bcfca type: Opaque % oc get secret kms-literal-secret -o yaml apiVersion: v1 data: password: c29tZXBhc3N3b3Jk kind: Secret metadata: creationTimestamp: "2025-05-16T14:05:34Z" name: kms-literal-secret namespace: kms resourceVersion: "819400" uid: d4ecd109-cd95-4afe-b2e8-77b2da3bcfca type: Opaque c. Creating a Secret Using a Manifest File You can also define secrets in YAML files using either the data or stringData fields. data stringData stringData: human-readable strings (Kubernetes will encode them to base64) data: requires values to already be base64 encoded stringData base64 data base64 Let’s go with the data field here. data % echo -n 'somepassword' | base64 c29tZXBhc3N3b3Jk % echo -n 'somepassword' | base64 c29tZXBhc3N3b3Jk Now use that in a manifest: % oc apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: kms-manifest-secret namespace: kms type: Opaque data: password: c29tZXBhc3N3b3Jk EOF % oc apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: kms-manifest-secret namespace: kms type: Opaque data: password: c29tZXBhc3N3b3Jk EOF Verifying: % oc get secret kms-manifest-secret -o yaml apiVersion: v1 data: password: c29tZXBhc3N3b3Jk kind: Secret metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","data":{"password":"c29tZXBhc3N3b3Jk"},"kind":"Secret","metadata":{"annotations":{},"name":"kms-manifest-secret","namespace":"kms"},"type":"Opaque"} creationTimestamp: "2025-05-16T14:24:58Z" name: kms-manifest-secret namespace: kms resourceVersion: "827156" uid: ebb157dd-c493-47af-94c4-ba5d96ff9a29 type: Opaque % oc get secret kms-manifest-secret -o yaml apiVersion: v1 data: password: c29tZXBhc3N3b3Jk kind: Secret metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","data":{"password":"c29tZXBhc3N3b3Jk"},"kind":"Secret","metadata":{"annotations":{},"name":"kms-manifest-secret","namespace":"kms"},"type":"Opaque"} creationTimestamp: "2025-05-16T14:24:58Z" name: kms-manifest-secret namespace: kms resourceVersion: "827156" uid: ebb157dd-c493-47af-94c4-ba5d96ff9a29 type: Opaque Why Use Kubernetes Secrets? Security: Better than storing sensitive info in ConfigMaps (which are plaintext). Isolation: Keeps secrets separate from your application code. Portability: Easily consumed by multiple Pods using environment variables or mounted volumes. Security: Better than storing sensitive info in ConfigMaps (which are plaintext). Security ConfigMaps Isolation: Keeps secrets separate from your application code. Isolation Portability: Easily consumed by multiple Pods using environment variables or mounted volumes. Portability But… What’s the Problem? At first glance, the secret looks like it’s encrypted due to its unreadable format: data: password: c29tZXBhc3N3b3Jk data: password: c29tZXBhc3N3b3Jk But that’s just base64 encoding—it’s easily reversible: % oc get secret kms-file-secret -o jsonpath='{.data.password}' | base64 --decode somepassword % oc get secret kms-file-secret -o jsonpath='{.data.password}' | base64 --decode somepassword So, if RBAC policies are misconfigured or someone has unauthorized read access to secrets, they can easily extract sensitive data. Not good. So, What’s the Solution? To make secrets more secure, Kubernetes offers multiple enhancements: Encryption at rest: Secrets are encrypted before being stored in etcd. KMS providers: Use cloud-based or external Key Management Service to encrypt secrets (e.g., AWS KMS, Azure Key Vault, HashiCorp Vault). Secrets Store CSI Driver: Mount secrets from external providers directly into Pods. Encryption at rest: Secrets are encrypted before being stored in etcd. Encryption at rest KMS providers: Use cloud-based or external Key Management Service to encrypt secrets (e.g., AWS KMS, Azure Key Vault, HashiCorp Vault). KMS providers Secrets Store CSI Driver: Mount secrets from external providers directly into Pods. Secrets Store CSI Driver We’ll dive into these solutions in Part 2 of the series.
