Secrets Are No Fun—Unless You Compare Them in Kubernetes

Kubernetes secrets are frequently used to securely store sensitive data, including passwords, API credentials, and certificates. In some cases, it may be necessary to compare secrets across several namespaces, clusters, or even two sets deployed by different systems. In this article, we will look at easy and effective methods for analyzing two sets of Kubernetes secrets. Problem in the wild I can't claim that comparing Kubernetes secrets is part of the usual routine for DevOps or cloud engineers; however, it may be necessary for the following reasons: Maintaining consistency across environments Teams may need to make a replica of the current environment for testing, including all configurations and secrets. In these cases, it's important to ensure that the two sets of secrets are equivalent in terms of structure and values. To avoid any structural difference during the deployment of an application to different environments (such as development, staging, and production), it's important to maintain consistency in underlying secret structures (not values). Otherwise, any differences could lead to security vulnerabilities or application failures (e.g., due to missing attributes). Maintaining consistency across provisioners This use case frequently happens when two secret delivery systems are tested concurrently or when you migrate from one to the other (for example, from SealedSecrets to ExternalSecrets). To verify the secret provisioning mechanism, it is necessary to compare two sets of secrets that were deployed by different systems and represent the same K8S objects. Versioning and updates During upgrades, you may need to ensure that the updated versions of secrets are properly propagated across the application. Comparing secrets helps to verify that no obsolete or stale secrets remain, reducing the risk of using invalid or expired credentials in the system. Audit and security Secret comparison can also play a role in security audits, checking that sensitive information, such as passwords or API keys, hasn't been mistakenly exposed, updated, or leaked across environments. For example, production environments must always have an individual set of secrets. To deal with any of these scenarios, a structural (keys and formats) or combination (structure and values) comparison of Kubernetes secrets may be required. Let's see how to do it. How to compare Kubernetes secrets To compare Kubernetes secrets, we need to perform the steps listed below, which we will implement as a script further: Retrieve a list of Kubernetes secrets with the first prefix (e.g.: kvendingoldo1) using the Kubernetes API (kubectl). Retrieve a list of Kubernetes secrets with the second prefix (e.g.: alextest) using the Kubernetes API (kubectl). Iterate through the first list, checking each secret: Attempt to find a matching secret in the second list. If found, compare both structure and values using tools like jq. Repeat the process for the second list to ensure completeness. Generate a report in the terminal, highlighting: Missing secrets Missing fields within secrets Differences in values The basic implementation of the algorithm is something like this: #!/bin/bash # # inputs # SECRET1_PREFIX="${1}" SECRET1_NS="${2}" SECRET2_PREFIX="${3}" SECRET2_NS="${4}" # # ANSI color codes # COLOR_GREEN="\033[0;32m" COLOR_YELLOW="\033[1;33m" COLOR_RED="\033[0;31m" COLOR_RESET="\033[0m" function get_secrets_with_prefix() { local prefix="${1}" kubectl get secrets -n="${SECRET1_NS}" -o json | jq -r ".items[] | select(.metadata.name | startswith(\"${prefix}\")) | .metadata.name" } function mask_value() { local value=${1} local length=${#value} local mask_length=$((length / 2)) local visible_part=${value:2:mask_length} echo "${visible_part}******" } function compare_secrets() { local secret1=${1} local secret2=${2} # Get keys (structure) for both secrets keys1=$(kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r '.data | keys[]') keys2=$(kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r '.data | keys[]') # Find common keys between the two secrets common_keys=$(echo -e "${keys1}\n${keys2}" | sort | uniq -d) if [ -z "${common_keys}" ]; then echo -e "No matching keys between ${secret1} and ${secret2}" return fi # Compare values of each common key for key in ${common_keys}; do value1=$(kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode) value2=$(kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode) if [ "${value1}" != "${value2}" ]; then echo -e "${COLOR_YELLOW}Difference found in key '${key}':${COLOR_RESET}" echo -e " - ${secret1}: $(mask_value ${value1})" echo -e " - ${secret2}: $(mask_value ${value2})" else echo -e "${COLOR_GREEN}Key '${key}' is identical in both secrets.${COLOR_RESET}" fi done # Check for keys that are missing in one of the secrets for key in ${keys1}; do if ! echo -e "${keys2}" | grep -q "${key}"; then echo -e "${COLOR_RED}Key '${key}' is missing in ${secret2}.${COLOR_RESET}" fi done for key in ${keys2}; do if ! echo -e "${keys1}" | grep -q "${key}"; then echo -e "${COLOR_RED}Key '${key}' is missing in ${secret1}.${COLOR_RESET}" fi done } function main() { secrets1=$(get_secrets_with_prefix "${SECRET1_PREFIX}") secrets2=$(get_secrets_with_prefix "${SECRET2_PREFIX}") for s1 in ${secrets1}; do suffix=${s1#$SECRET1_PREFIX} # Get the suffix by removing the prefix s2="${SECRET2_PREFIX}${suffix}" # Construct the corresponding secret in the second namespace # Check if the corresponding secret exists if echo -e "${secrets2}" | grep -q "${s2}"; then echo -e "Comparing ${s1} and ${s2}:" compare_secrets "${s1}" "${s2}" echo -e else echo -e "No corresponding secret for ${s1} in ${SECRET2_PREFIX}." fi done } main "${@}" As you can see, the basic script supports different Kubernetes clusters, but it’s not difficult to implement.Let's do it with easy modifications: Add KUBECONFIG variables for each secret set KUBECONFIG1="${1}" SECRET1_PREFIX="${2}" SECRET1_NS="${3}" KUBECONFIG2="${4}" SECRET2_PREFIX="${5}" SECRET2_NS="${6}" Add kubeconfig variable into get_secrets_with_prefix function function get_secrets_with_prefix() { local kubeconfig=${1} local prefix=${2} local namespace=${3} KUBECONFIG=${kubeconfig} kubectl get secrets -n=${namespace} -o json | jq -r ".items[] | select(.metadata.name | startswith(\"${prefix}\")) | .metadata.name" } Add kubeconfig1 and kubeconfig2 variables to compare_secrets As a result, you will get the following script that can work with different clusters: #!/bin/bash # # inputs # KUBECONFIG1="${1}" SECRET1_PREFIX="${2}" SECRET1_NS="${3}" KUBECONFIG2="${4}" SECRET2_PREFIX="${5}" SECRET2_NS="${6}" # # ANSI color codes # COLOR_GREEN="\033[0;32m" COLOR_YELLOW="\033[1;33m" COLOR_RED="\033[0;31m" COLOR_RESET="\033[0m" function mask_value() { local value=${1} local length=${#value} local mask_length=$((length / 2)) local visible_part=${value:2:mask_length} echo "${visible_part}******" } function get_secrets_with_prefix() { local kubeconfig="${1}" local prefix="${2}" local namespace="${3}" KUBECONFIG=${kubeconfig} kubectl get secrets -n=${namespace} -o json | jq -r ".items[] | select(.metadata.name | startswith(\"${prefix}\")) | .metadata.name" } function compare_secrets() { local kubeconfig1="${1}" local kubeconfig2="${2}" local secret1=""${3}" local secret2="${4}" # Get keys (structure) for both secrets keys1=$(KUBECONFIG=${kubeconfig1} kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r '.data | keys[]') keys2=$(KUBECONFIG=${kubeconfig2} kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r '.data | keys[]') # Find common keys between the two secrets common_keys=$(echo -e "${keys1}\n${keys2}" | sort | uniq -d) if [ -z "${common_keys}" ]; then echo -e "No matching keys between ${secret1} and ${secret2}" return fi # Compare values of each common key for key in ${common_keys}; do value1=$(KUBECONFIG=${kubeconfig1} kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode) value2=$(KUBECONFIG=${kubeconfig2} kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode) if [ "${value1}" != "${value2}" ]; then echo -e "${COLOR_YELLOW}Difference found in key '${key}':${COLOR_RESET}" echo -e " - ${secret1}: $(mask_value ${value1})" echo -e " - ${secret2}: $(mask_value ${value2})" else echo -e "${COLOR_GREEN}Key '${key}' is identical in both secrets.${COLOR_RESET}" fi done # Check for keys that are missing in one of the secrets for key in ${keys1}; do if ! echo -e "${keys2}" | grep -q "${key}"; then echo -e "${COLOR_RED}Key '${key}' is missing in ${secret2}.${COLOR_RESET}" fi done for key in ${keys2}; do if ! echo -e "${keys1}" | grep -q "${key}"; then echo -e "${COLOR_RED}Key '${key}' is missing in ${secret1}.${COLOR_RESET}" fi done } function main() { secrets1=$(get_secrets_with_prefix "${KUBECONFIG1}" "${SECRET1_PREFIX}" "${SECRET1_NS}") secrets2=$(get_secrets_with_prefix "${KUBECONFIG2}" "${SECRET2_PREFIX}" "${SECRET2_NS}") for s1 in ${secrets1}; do suffix=${s1#$SECRET1_PREFIX} # Get the suffix by removing the prefix s2="${SECRET2_PREFIX}${suffix}" # Construct the corresponding secret in the second namespace # Check if the corresponding secret exists if echo -e "${secrets2}" | grep -q "${s2}"; then echo -e "Comparing ${s1} in Cluster1 and ${s2} in Cluster2:" compare_secrets "${KUBECONFIG1}" "${KUBECONFIG2}" "${s1}" "${s2}" echo -e else echo -e "No corresponding secret for ${s1} in Cluster2 (${SECRET2_PREFIX})." fi done } main "${@}" Testing the script To test the script, let's generate test data using another bash script. The script source code: #!/bin/bash # # inputs # SECRET1_PREFIX="${1}" SECRET1_NS="${2}" SECRET2_PREFIX="${3}" SECRET2_NS="${4}" NUM_SECRETS=5 # Number of secrets to generate NUM_KEYS=4 # Number of keys per secret function generate_random_value() { head -c 8 /dev/urandom | base64 } function create_test_secrets() { local prefix=${1} local namespace=${2} for i in $(seq 1 $NUM_SECRETS); do secret_name="${prefix}test-secret-${i}" data="" if (( i % 3 == 1 )); then # Generate identical values data=" key1: $(echo -n 'fixedValue' | base64)\n key2: $(echo -n 'fixedValue' | base64)\n key3: $(echo -n 'fixedValue' | base64)" elif (( i % 3 == 2 )); then # Generate different values data=" key1: $(generate_random_value)\n key2: $(generate_random_value)\n key3: $(generate_random_value)" else # Generate different keys data=" keyA: $(generate_random_value)\n keyB: $(generate_random_value)" fi echo "Creating secret ${secret_name} in namespace ${namespace}" echo -e "apiVersion: v1\nkind: Secret\nmetadata:\n name: ${secret_name}\n namespace: ${namespace}\ntype: Opaque\ndata:\n${data}" | kubectl apply -f - done } function main() { create_test_secrets "${SECRET1_PREFIX}" "${SECRET1_NS}" create_test_secrets "${SECRET2_PREFIX}" "${SECRET2_NS}" } main "${@}" To run it, do the following commands: $ bash generate_test_data.sh kvendingoldo test-secrets-demo alextest test-secrets-demo $ bash compare_secrets.sh kvendingoldo test-secrets-demo alextest test-secrets-demo Following that, you will see the output log, which shows that some secrets are identical while others have different values: Conclusion Comparing Kubernetes secrets is essential to maintaining consistency, security, and versioning in your application secrets. The approach described in this post is an excellent solution for rapid manual activities, which CloudOps/DevOps teams may need. You may need to modify the script a little bit for your task, but this should be easy thanks to the Bash language. Feel free to experiment, and keep your secrets safeguarded! Kubernetes secrets are frequently used to securely store sensitive data, including passwords, API credentials, and certificates. In some cases, it may be necessary to compare secrets across several namespaces, clusters, or even two sets deployed by different systems. In this article, we will look at easy and effective methods for analyzing two sets of Kubernetes secrets. Problem in the wild Problem in the wild I can't claim that comparing Kubernetes secrets is part of the usual routine for DevOps or cloud engineers; however, it may be necessary for the following reasons: Maintaining consistency across environments Maintaining consistency across environments Teams may need to make a replica of the current environment for testing, including all configurations and secrets. In these cases, it's important to ensure that the two sets of secrets are equivalent in terms of structure and values. To avoid any structural difference during the deployment of an application to different environments (such as development, staging, and production), it's important to maintain consistency in underlying secret structures (not values). Otherwise, any differences could lead to security vulnerabilities or application failures (e.g., due to missing attributes). Teams may need to make a replica of the current environment for testing, including all configurations and secrets. In these cases, it's important to ensure that the two sets of secrets are equivalent in terms of structure and values. To avoid any structural difference during the deployment of an application to different environments (such as development, staging, and production), it's important to maintain consistency in underlying secret structures (not values). Otherwise, any differences could lead to security vulnerabilities or application failures (e.g., due to missing attributes). Maintaining consistency across provisioners Maintaining consistency across provisioners This use case frequently happens when two secret delivery systems are tested concurrently or when you migrate from one to the other (for example, from SealedSecrets to ExternalSecrets). To verify the secret provisioning mechanism, it is necessary to compare two sets of secrets that were deployed by different systems and represent the same K8S objects. This use case frequently happens when two secret delivery systems are tested concurrently or when you migrate from one to the other (for example, from SealedSecrets to ExternalSecrets ). To verify the secret provisioning mechanism, it is necessary to compare two sets of secrets that were deployed by different systems and represent the same K8S objects. SealedSecrets ExternalSecrets Versioning and updates Versioning and updates During upgrades, you may need to ensure that the updated versions of secrets are properly propagated across the application. Comparing secrets helps to verify that no obsolete or stale secrets remain, reducing the risk of using invalid or expired credentials in the system. During upgrades, you may need to ensure that the updated versions of secrets are properly propagated across the application. Comparing secrets helps to verify that no obsolete or stale secrets remain, reducing the risk of using invalid or expired credentials in the system. Audit and security Audit and security Secret comparison can also play a role in security audits, checking that sensitive information, such as passwords or API keys, hasn't been mistakenly exposed, updated, or leaked across environments. For example, production environments must always have an individual set of secrets. Secret comparison can also play a role in security audits, checking that sensitive information, such as passwords or API keys, hasn't been mistakenly exposed, updated, or leaked across environments. For example, production environments must always have an individual set of secrets. To deal with any of these scenarios, a structural (keys and formats) or combination (structure and values) comparison of Kubernetes secrets may be required. Let's see how to do it. How to compare Kubernetes secrets How to compare Kubernetes secrets To compare Kubernetes secrets, we need to perform the steps listed below, which we will implement as a script further: Retrieve a list of Kubernetes secrets with the first prefix (e.g.: kvendingoldo1) using the Kubernetes API (kubectl). Retrieve a list of Kubernetes secrets with the second prefix (e.g.: alextest) using the Kubernetes API (kubectl). Iterate through the first list, checking each secret: Attempt to find a matching secret in the second list. If found, compare both structure and values using tools like jq. Repeat the process for the second list to ensure completeness. Generate a report in the terminal, highlighting: Missing secrets Missing fields within secrets Differences in values Retrieve a list of Kubernetes secrets with the first prefix (e.g.: kvendingoldo1 ) using the Kubernetes API (kubectl). kvendingoldo1 Retrieve a list of Kubernetes secrets with the second prefix (e.g.: alextest ) using the Kubernetes API (kubectl). alextest Iterate through the first list, checking each secret: Attempt to find a matching secret in the second list. If found, compare both structure and values using tools like jq. Attempt to find a matching secret in the second list. If found, compare both structure and values using tools like jq. Attempt to find a matching secret in the second list. If found, compare both structure and values using tools like jq. Repeat the process for the second list to ensure completeness. Generate a report in the terminal, highlighting: Missing secrets Missing fields within secrets Differences in values Missing secrets Missing fields within secrets Differences in values Missing secrets Missing fields within secrets Differences in values The basic implementation of the algorithm is something like this: The basic implementation of the algorithm is something like this: #!/bin/bash # # inputs # SECRET1_PREFIX="${1}" SECRET1_NS="${2}" SECRET2_PREFIX="${3}" SECRET2_NS="${4}" # # ANSI color codes # COLOR_GREEN="\033[0;32m" COLOR_YELLOW="\033[1;33m" COLOR_RED="\033[0;31m" COLOR_RESET="\033[0m" function get_secrets_with_prefix() { local prefix="${1}" kubectl get secrets -n="${SECRET1_NS}" -o json | jq -r ".items[] | select(.metadata.name | startswith(\"${prefix}\")) | .metadata.name" } function mask_value() { local value=${1} local length=${#value} local mask_length=$((length / 2)) local visible_part=${value:2:mask_length} echo "${visible_part}******" } function compare_secrets() { local secret1=${1} local secret2=${2} # Get keys (structure) for both secrets keys1=$(kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r '.data | keys[]') keys2=$(kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r '.data | keys[]') # Find common keys between the two secrets common_keys=$(echo -e "${keys1}\n${keys2}" | sort | uniq -d) if [ -z "${common_keys}" ]; then echo -e "No matching keys between ${secret1} and ${secret2}" return fi # Compare values of each common key for key in ${common_keys}; do value1=$(kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode) value2=$(kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode) if [ "${value1}" != "${value2}" ]; then echo -e "${COLOR_YELLOW}Difference found in key '${key}':${COLOR_RESET}" echo -e " - ${secret1}: $(mask_value ${value1})" echo -e " - ${secret2}: $(mask_value ${value2})" else echo -e "${COLOR_GREEN}Key '${key}' is identical in both secrets.${COLOR_RESET}" fi done # Check for keys that are missing in one of the secrets for key in ${keys1}; do if ! echo -e "${keys2}" | grep -q "${key}"; then echo -e "${COLOR_RED}Key '${key}' is missing in ${secret2}.${COLOR_RESET}" fi done for key in ${keys2}; do if ! echo -e "${keys1}" | grep -q "${key}"; then echo -e "${COLOR_RED}Key '${key}' is missing in ${secret1}.${COLOR_RESET}" fi done } function main() { secrets1=$(get_secrets_with_prefix "${SECRET1_PREFIX}") secrets2=$(get_secrets_with_prefix "${SECRET2_PREFIX}") for s1 in ${secrets1}; do suffix=${s1#$SECRET1_PREFIX} # Get the suffix by removing the prefix s2="${SECRET2_PREFIX}${suffix}" # Construct the corresponding secret in the second namespace # Check if the corresponding secret exists if echo -e "${secrets2}" | grep -q "${s2}"; then echo -e "Comparing ${s1} and ${s2}:" compare_secrets "${s1}" "${s2}" echo -e else echo -e "No corresponding secret for ${s1} in ${SECRET2_PREFIX}." fi done } main "${@}" #!/bin/bash # # inputs # SECRET1_PREFIX="${1}" SECRET1_NS="${2}" SECRET2_PREFIX="${3}" SECRET2_NS="${4}" # # ANSI color codes # COLOR_GREEN="\033[0;32m" COLOR_YELLOW="\033[1;33m" COLOR_RED="\033[0;31m" COLOR_RESET="\033[0m" function get_secrets_with_prefix() { local prefix="${1}" kubectl get secrets -n="${SECRET1_NS}" -o json | jq -r ".items[] | select(.metadata.name | startswith(\"${prefix}\")) | .metadata.name" } function mask_value() { local value=${1} local length=${#value} local mask_length=$((length / 2)) local visible_part=${value:2:mask_length} echo "${visible_part}******" } function compare_secrets() { local secret1=${1} local secret2=${2} # Get keys (structure) for both secrets keys1=$(kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r '.data | keys[]') keys2=$(kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r '.data | keys[]') # Find common keys between the two secrets common_keys=$(echo -e "${keys1}\n${keys2}" | sort | uniq -d) if [ -z "${common_keys}" ]; then echo -e "No matching keys between ${secret1} and ${secret2}" return fi # Compare values of each common key for key in ${common_keys}; do value1=$(kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode) value2=$(kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode) if [ "${value1}" != "${value2}" ]; then echo -e "${COLOR_YELLOW}Difference found in key '${key}':${COLOR_RESET}" echo -e " - ${secret1}: $(mask_value ${value1})" echo -e " - ${secret2}: $(mask_value ${value2})" else echo -e "${COLOR_GREEN}Key '${key}' is identical in both secrets.${COLOR_RESET}" fi done # Check for keys that are missing in one of the secrets for key in ${keys1}; do if ! echo -e "${keys2}" | grep -q "${key}"; then echo -e "${COLOR_RED}Key '${key}' is missing in ${secret2}.${COLOR_RESET}" fi done for key in ${keys2}; do if ! echo -e "${keys1}" | grep -q "${key}"; then echo -e "${COLOR_RED}Key '${key}' is missing in ${secret1}.${COLOR_RESET}" fi done } function main() { secrets1=$(get_secrets_with_prefix "${SECRET1_PREFIX}") secrets2=$(get_secrets_with_prefix "${SECRET2_PREFIX}") for s1 in ${secrets1}; do suffix=${s1#$SECRET1_PREFIX} # Get the suffix by removing the prefix s2="${SECRET2_PREFIX}${suffix}" # Construct the corresponding secret in the second namespace # Check if the corresponding secret exists if echo -e "${secrets2}" | grep -q "${s2}"; then echo -e "Comparing ${s1} and ${s2}:" compare_secrets "${s1}" "${s2}" echo -e else echo -e "No corresponding secret for ${s1} in ${SECRET2_PREFIX}." fi done } main "${@}" As you can see, the basic script supports different Kubernetes clusters, but it’s not difficult to implement.Let's do it with easy modifications: Add KUBECONFIG variables for each secret set KUBECONFIG1="${1}" SECRET1_PREFIX="${2}" SECRET1_NS="${3}" KUBECONFIG2="${4}" SECRET2_PREFIX="${5}" SECRET2_NS="${6}" Add kubeconfig variable into get_secrets_with_prefix function function get_secrets_with_prefix() { local kubeconfig=${1} local prefix=${2} local namespace=${3} KUBECONFIG=${kubeconfig} kubectl get secrets -n=${namespace} -o json | jq -r ".items[] | select(.metadata.name | startswith(\"${prefix}\")) | .metadata.name" } Add kubeconfig1 and kubeconfig2 variables to compare_secrets Add KUBECONFIG variables for each secret set KUBECONFIG1="${1}" SECRET1_PREFIX="${2}" SECRET1_NS="${3}" KUBECONFIG2="${4}" SECRET2_PREFIX="${5}" SECRET2_NS="${6}" Add KUBECONFIG variables for each secret set KUBECONFIG1="${1}" SECRET1_PREFIX="${2}" SECRET1_NS="${3}" KUBECONFIG2="${4}" SECRET2_PREFIX="${5}" SECRET2_NS="${6}" KUBECONFIG1="${1}" SECRET1_PREFIX="${2}" SECRET1_NS="${3}" KUBECONFIG2="${4}" SECRET2_PREFIX="${5}" SECRET2_NS="${6}" Add kubeconfig variable into get_secrets_with_prefix function function get_secrets_with_prefix() { local kubeconfig=${1} local prefix=${2} local namespace=${3} KUBECONFIG=${kubeconfig} kubectl get secrets -n=${namespace} -o json | jq -r ".items[] | select(.metadata.name | startswith(\"${prefix}\")) | .metadata.name" } Add kubeconfig variable into get_secrets_with_prefix function get_secrets_with_prefix function get_secrets_with_prefix() { local kubeconfig=${1} local prefix=${2} local namespace=${3} KUBECONFIG=${kubeconfig} kubectl get secrets -n=${namespace} -o json | jq -r ".items[] | select(.metadata.name | startswith(\"${prefix}\")) | .metadata.name" } function get_secrets_with_prefix() { local kubeconfig=${1} local prefix=${2} local namespace=${3} KUBECONFIG=${kubeconfig} kubectl get secrets -n=${namespace} -o json | jq -r ".items[] | select(.metadata.name | startswith(\"${prefix}\")) | .metadata.name" } Add kubeconfig1 and kubeconfig2 variables to compare_secrets Add kubeconfig1 and kubeconfig2 variables to compare_secrets kubeconfig1 kubeconfig2 compare_secrets As a result, you will get the following script that can work with different clusters: #!/bin/bash # # inputs # KUBECONFIG1="${1}" SECRET1_PREFIX="${2}" SECRET1_NS="${3}" KUBECONFIG2="${4}" SECRET2_PREFIX="${5}" SECRET2_NS="${6}" # # ANSI color codes # COLOR_GREEN="\033[0;32m" COLOR_YELLOW="\033[1;33m" COLOR_RED="\033[0;31m" COLOR_RESET="\033[0m" function mask_value() { local value=${1} local length=${#value} local mask_length=$((length / 2)) local visible_part=${value:2:mask_length} echo "${visible_part}******" } function get_secrets_with_prefix() { local kubeconfig="${1}" local prefix="${2}" local namespace="${3}" KUBECONFIG=${kubeconfig} kubectl get secrets -n=${namespace} -o json | jq -r ".items[] | select(.metadata.name | startswith(\"${prefix}\")) | .metadata.name" } function compare_secrets() { local kubeconfig1="${1}" local kubeconfig2="${2}" local secret1=""${3}" local secret2="${4}" # Get keys (structure) for both secrets keys1=$(KUBECONFIG=${kubeconfig1} kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r '.data | keys[]') keys2=$(KUBECONFIG=${kubeconfig2} kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r '.data | keys[]') # Find common keys between the two secrets common_keys=$(echo -e "${keys1}\n${keys2}" | sort | uniq -d) if [ -z "${common_keys}" ]; then echo -e "No matching keys between ${secret1} and ${secret2}" return fi # Compare values of each common key for key in ${common_keys}; do value1=$(KUBECONFIG=${kubeconfig1} kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode) value2=$(KUBECONFIG=${kubeconfig2} kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode) if [ "${value1}" != "${value2}" ]; then echo -e "${COLOR_YELLOW}Difference found in key '${key}':${COLOR_RESET}" echo -e " - ${secret1}: $(mask_value ${value1})" echo -e " - ${secret2}: $(mask_value ${value2})" else echo -e "${COLOR_GREEN}Key '${key}' is identical in both secrets.${COLOR_RESET}" fi done # Check for keys that are missing in one of the secrets for key in ${keys1}; do if ! echo -e "${keys2}" | grep -q "${key}"; then echo -e "${COLOR_RED}Key '${key}' is missing in ${secret2}.${COLOR_RESET}" fi done for key in ${keys2}; do if ! echo -e "${keys1}" | grep -q "${key}"; then echo -e "${COLOR_RED}Key '${key}' is missing in ${secret1}.${COLOR_RESET}" fi done } function main() { secrets1=$(get_secrets_with_prefix "${KUBECONFIG1}" "${SECRET1_PREFIX}" "${SECRET1_NS}") secrets2=$(get_secrets_with_prefix "${KUBECONFIG2}" "${SECRET2_PREFIX}" "${SECRET2_NS}") for s1 in ${secrets1}; do suffix=${s1#$SECRET1_PREFIX} # Get the suffix by removing the prefix s2="${SECRET2_PREFIX}${suffix}" # Construct the corresponding secret in the second namespace # Check if the corresponding secret exists if echo -e "${secrets2}" | grep -q "${s2}"; then echo -e "Comparing ${s1} in Cluster1 and ${s2} in Cluster2:" compare_secrets "${KUBECONFIG1}" "${KUBECONFIG2}" "${s1}" "${s2}" echo -e else echo -e "No corresponding secret for ${s1} in Cluster2 (${SECRET2_PREFIX})." fi done } main "${@}" #!/bin/bash # # inputs # KUBECONFIG1="${1}" SECRET1_PREFIX="${2}" SECRET1_NS="${3}" KUBECONFIG2="${4}" SECRET2_PREFIX="${5}" SECRET2_NS="${6}" # # ANSI color codes # COLOR_GREEN="\033[0;32m" COLOR_YELLOW="\033[1;33m" COLOR_RED="\033[0;31m" COLOR_RESET="\033[0m" function mask_value() { local value=${1} local length=${#value} local mask_length=$((length / 2)) local visible_part=${value:2:mask_length} echo "${visible_part}******" } function get_secrets_with_prefix() { local kubeconfig="${1}" local prefix="${2}" local namespace="${3}" KUBECONFIG=${kubeconfig} kubectl get secrets -n=${namespace} -o json | jq -r ".items[] | select(.metadata.name | startswith(\"${prefix}\")) | .metadata.name" } function compare_secrets() { local kubeconfig1="${1}" local kubeconfig2="${2}" local secret1=""${3}" local secret2="${4}" # Get keys (structure) for both secrets keys1=$(KUBECONFIG=${kubeconfig1} kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r '.data | keys[]') keys2=$(KUBECONFIG=${kubeconfig2} kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r '.data | keys[]') # Find common keys between the two secrets common_keys=$(echo -e "${keys1}\n${keys2}" | sort | uniq -d) if [ -z "${common_keys}" ]; then echo -e "No matching keys between ${secret1} and ${secret2}" return fi # Compare values of each common key for key in ${common_keys}; do value1=$(KUBECONFIG=${kubeconfig1} kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode) value2=$(KUBECONFIG=${kubeconfig2} kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode) if [ "${value1}" != "${value2}" ]; then echo -e "${COLOR_YELLOW}Difference found in key '${key}':${COLOR_RESET}" echo -e " - ${secret1}: $(mask_value ${value1})" echo -e " - ${secret2}: $(mask_value ${value2})" else echo -e "${COLOR_GREEN}Key '${key}' is identical in both secrets.${COLOR_RESET}" fi done # Check for keys that are missing in one of the secrets for key in ${keys1}; do if ! echo -e "${keys2}" | grep -q "${key}"; then echo -e "${COLOR_RED}Key '${key}' is missing in ${secret2}.${COLOR_RESET}" fi done for key in ${keys2}; do if ! echo -e "${keys1}" | grep -q "${key}"; then echo -e "${COLOR_RED}Key '${key}' is missing in ${secret1}.${COLOR_RESET}" fi done } function main() { secrets1=$(get_secrets_with_prefix "${KUBECONFIG1}" "${SECRET1_PREFIX}" "${SECRET1_NS}") secrets2=$(get_secrets_with_prefix "${KUBECONFIG2}" "${SECRET2_PREFIX}" "${SECRET2_NS}") for s1 in ${secrets1}; do suffix=${s1#$SECRET1_PREFIX} # Get the suffix by removing the prefix s2="${SECRET2_PREFIX}${suffix}" # Construct the corresponding secret in the second namespace # Check if the corresponding secret exists if echo -e "${secrets2}" | grep -q "${s2}"; then echo -e "Comparing ${s1} in Cluster1 and ${s2} in Cluster2:" compare_secrets "${KUBECONFIG1}" "${KUBECONFIG2}" "${s1}" "${s2}" echo -e else echo -e "No corresponding secret for ${s1} in Cluster2 (${SECRET2_PREFIX})." fi done } main "${@}" Testing the script Testing the script To test the script, let's generate test data using another bash script. The script source code: #!/bin/bash # # inputs # SECRET1_PREFIX="${1}" SECRET1_NS="${2}" SECRET2_PREFIX="${3}" SECRET2_NS="${4}" NUM_SECRETS=5 # Number of secrets to generate NUM_KEYS=4 # Number of keys per secret function generate_random_value() { head -c 8 /dev/urandom | base64 } function create_test_secrets() { local prefix=${1} local namespace=${2} for i in $(seq 1 $NUM_SECRETS); do secret_name="${prefix}test-secret-${i}" data="" if (( i % 3 == 1 )); then # Generate identical values data=" key1: $(echo -n 'fixedValue' | base64)\n key2: $(echo -n 'fixedValue' | base64)\n key3: $(echo -n 'fixedValue' | base64)" elif (( i % 3 == 2 )); then # Generate different values data=" key1: $(generate_random_value)\n key2: $(generate_random_value)\n key3: $(generate_random_value)" else # Generate different keys data=" keyA: $(generate_random_value)\n keyB: $(generate_random_value)" fi echo "Creating secret ${secret_name} in namespace ${namespace}" echo -e "apiVersion: v1\nkind: Secret\nmetadata:\n name: ${secret_name}\n namespace: ${namespace}\ntype: Opaque\ndata:\n${data}" | kubectl apply -f - done } function main() { create_test_secrets "${SECRET1_PREFIX}" "${SECRET1_NS}" create_test_secrets "${SECRET2_PREFIX}" "${SECRET2_NS}" } main "${@}" #!/bin/bash # # inputs # SECRET1_PREFIX="${1}" SECRET1_NS="${2}" SECRET2_PREFIX="${3}" SECRET2_NS="${4}" NUM_SECRETS=5 # Number of secrets to generate NUM_KEYS=4 # Number of keys per secret function generate_random_value() { head -c 8 /dev/urandom | base64 } function create_test_secrets() { local prefix=${1} local namespace=${2} for i in $(seq 1 $NUM_SECRETS); do secret_name="${prefix}test-secret-${i}" data="" if (( i % 3 == 1 )); then # Generate identical values data=" key1: $(echo -n 'fixedValue' | base64)\n key2: $(echo -n 'fixedValue' | base64)\n key3: $(echo -n 'fixedValue' | base64)" elif (( i % 3 == 2 )); then # Generate different values data=" key1: $(generate_random_value)\n key2: $(generate_random_value)\n key3: $(generate_random_value)" else # Generate different keys data=" keyA: $(generate_random_value)\n keyB: $(generate_random_value)" fi echo "Creating secret ${secret_name} in namespace ${namespace}" echo -e "apiVersion: v1\nkind: Secret\nmetadata:\n name: ${secret_name}\n namespace: ${namespace}\ntype: Opaque\ndata:\n${data}" | kubectl apply -f - done } function main() { create_test_secrets "${SECRET1_PREFIX}" "${SECRET1_NS}" create_test_secrets "${SECRET2_PREFIX}" "${SECRET2_NS}" } main "${@}" To run it, do the following commands: $ bash generate_test_data.sh kvendingoldo test-secrets-demo alextest test-secrets-demo $ bash compare_secrets.sh kvendingoldo test-secrets-demo alextest test-secrets-demo $ bash generate_test_data.sh kvendingoldo test-secrets-demo alextest test-secrets-demo $ bash generate_test_data.sh kvendingoldo test-secrets-demo alextest test-secrets-demo $ bash compare_secrets.sh kvendingoldo test-secrets-demo alextest test-secrets-demo $ bash compare_secrets.sh kvendingoldo test-secrets-demo alextest test-secrets-demo Following that, you will see the output log, which shows that some secrets are identical while others have different values: Conclusion Conclusion Comparing Kubernetes secrets is essential to maintaining consistency, security, and versioning in your application secrets. The approach described in this post is an excellent solution for rapid manual activities, which CloudOps/DevOps teams may need. You may need to modify the script a little bit for your task, but this should be easy thanks to the Bash language. Feel free to experiment, and keep your secrets safeguarded!