Secrets Management and Encryption

A widespread misconception among Kubernetes newcomers is that a Secret object is an encrypted vault. In reality, native Kubernetes Secrets are only base64 encoded, not encrypted. Anyone with kubectl get secret -o yaml access or access to the underlying etcd disk can decode database passwords and API tokens in a fraction of a second.

Securing credentials in Kubernetes requires three essential defenses:

1. Encryption at rest in etcd.
2. External secret lifecycle management (e.g., HashiCorp Vault, AWS Secrets Manager).
3. Safe GitOps pipelines without plaintext commits.

1. etcd Encryption at Rest

By default, data written to etcd is stored as plaintext. If someone obtains a snapshot of your etcd database or physical access to master node storage, all secrets are compromised.

You must configure an EncryptionConfiguration file on the API Server using providers like kms (envelope encryption using AWS KMS, GCP Cloud KMS, or Azure Key Vault) or secretbox:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: aws-kms-provider
          endpoint: unix:///var/run/kmsplugin/socket.sock
      - identity: {} # Fallback

---

2. The External Secrets Operator (ESO)

Storing static secrets in Kubernetes manifests leads to credential leaks in Git and outdated keys. The industry standard pattern is to maintain secrets in a dedicated external provider (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager) and synchronize them into Kubernetes using the External Secrets Operator.

How ESO Works:

1. You define a SecretStore that authenticates to your cloud KMS or Vault via IAM.
2. You declare an ExternalSecret resource specifying which key to pull.
3. The operator pulls the value from the provider and creates a standard Kubernetes Secret in-cluster, refreshing it on a schedule.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: db-secret # K8s secret generated in namespace
  data:
    - secretKey: password
      remoteRef:
        key: prod/db/mysql
        property: db_password

---

3. Secrets in GitOps: Sealed Secrets

If your team embraces GitOps, manifests must live in Git. But you cannot commit plaintext K8s Secrets.

Bitnami Sealed Secrets solves this using asymmetric public-key cryptography:

• A developer encrypts a secret on their workstation using the cluster's public certificate: kubeseal < secret.yaml > sealedsecret.yaml.
• The SealedSecret is safe to commit to public or private Git repositories.
• The Sealed Secrets controller inside your cluster holds the private key, decrypts the object, and generates a live Kubernetes Secret.

---

Summary Checklist for Secrets

• Never commit raw Kubernetes Secrets to version control.
• Enable KMS-based envelope encryption for etcd.
• Rotate credentials automatically via External Secrets or Vault.
• Mount secrets as memory-backed volumes (emptyDir: { medium: Memory }) rather than environment variables to prevent accidental leakages in process tables and crash logs.