[EN] Best Practices for Managing Secrets in Kubernetes

Managing secrets in Kubernetes requires careful attention. The default implementation has significant security limitations. Encoding secrets in base64 offers no real protection, it’s a strategy that exposes sensitive data to serious risks.

Essential Practices for Cluster Administrators

1. Encrypt Secrets at Rest

It is critical to encrypt secret data stored in etcd, Kubernetes’ internal database.

· Immediate action: Configure an encryption provider on the Kubernetes API server. You can do this using an encryption configuration file. For managed clusters (EKS, AKS, GKE), simply enable the managed encryption option.
· Advanced approach (KMS): For stronger security and control, use a Key Management Service provider (AWS KMS, Google Cloud KMS, Azure Key Vault). This enables envelope encryption, where keys are managed externally, reducing the risk of direct etcd access.

2. Control Access with Least Privilege (RBAC)

RBAC is the primary tool for controlling who or what can access secrets.

· Avoid unnecessary privileges: Never grant list, watch, or get permissions on secrets to users or service accounts that don’t explicitly need them. Remember that list permission can be used to view secret content.
· Use namespaces for isolation: Isolate secrets by namespace and use RoleBindings instead of ClusterRoleBindings to restrict access scope.

3. Protect etcd and Cluster Components

Beyond encryption, other measures help secure etcd, the cluster’s core datastore.

· Secure communication: Configure TLS between etcd nodes to protect data in transit.
· Sanitize storage: When decommissioning etcd persistent storage, use secure wiping methods (wipe or shred) to prevent data recovery.

4. Monitor and Audit Access

Auditing can help to detecting unauthorized access.

· Configure audit logs in Kubernetes to generate alerts for suspicious events: example, a single user reading many secrets in a short time.
· Integrate with a SIEM (Security Information and Event Management) tool for deeper analysis.

Essential Practices for Developers

1. Never Hardcode or Share Manifests with Secrets

Avoid storing literal values in code or sharing YAML manifests with plain‑text secrets, even in private repositories. The risk of exposure is too high.

2. Adopt Advanced Secret Management Tools

Instead of relying solely on native Secret resources, use tools that raise the security bar:

· External Secrets Operator (ESO): Synchronizes secrets from an external vault to Kubernetes. The cluster becomes a consumer, not the primary source of sensitive data.
· Secrets Store CSI Driver: Mounts secrets as a volume directly into a pod, avoiding storage in etcd or the Kubernetes API. The application reads the secret from the filesystem.
· HashiCorp Vault: A complete solution for high‑compliance environments, managing the full lifecycle of secrets with advanced access and auditing features.

3. Adopt Secure GitOps
If you store manifests in Git (standard GitOps practice), never commit secrets in plain text. Use encryption tools:

· Sealed Secrets : Encrypts the secret inside the Git repository. Only the controller in the cluster can decrypt it.
· Mozilla SOPS (Secrets OPerationS): Encrypts secret files, which can be versioned in Git, using keys from services such as AWS KMS, GCP KMS, Azure Key Vault, or PGP.

4. Automate Secret Rotation

· Automate with cloud provider : Use automated rotation from your cloud secrets manager (AWS Secrets Manager) combined with ESO. When a secret changes, ESO updates it in the cluster, and the pod can be configured to refresh the environment variable without restarting.
· Handle application restarts : For changes that require pod restarts, use tools like rolloutRestartTargets (from the Vault Secrets Operator) to orchestrate the process automatically.

Myths vs. Reality

It is important to distinguish between encoding and encryption.

Myth - What Kubernetes does not do by default

1. "A Secret in Kubernetes is encrypted by default."
   
   Reality: No. A Secret is simply a base64‑encoded string, which offers zero security. Anyone with access to the cluster can decode it.

2. "If my RBAC is configured, my data is safe."
   
   Reality: RBAC alone is not enough. Direct access to etcd (or its backups) or to the Kubernetes API (kubectl get secret) exposes all data in plain text, regardless of RBAC.

Summary and Recommendations

Build a robust secret management strategy by starting with the fundamentals and increasing complexity as needed.

Step 1 - Foundation (for everyone)

· Enable encryption at rest for etcd on your managed cluster.
· Implement RBAC with least privilege.
· Educate your team to never store secrets as plain text in Git.

Step 2 - Evolution (for most teams)

· Adopt an external secret manager. Start with the External Secrets Operator (ESO), which is simpler and offers an excellent balance between security and usability.

Step 3 - Advanced (for high compliance)

· If you operate under strict compliance regimes (SOC2, HIPAA) or in multi‑cloud environments, invest in HashiCorp Vault.
· For teams deeply invested in GitOps, use Sealed Secrets or SOPS to encrypt manifests before versioning.

References

• https://kubernetes.io/docs/concepts/security/secrets-good-practices/ -https://kubernetes.io/docs/concepts/security/rbac-good-practices/
• https://external-secrets.io/
• https://secrets-store-csi-driver.sigs.k8s.io/ -https://developer.hashicorp.com/vault/docs/secrets/kubernetes
• https://github.com/bitnami-labs/sealed-secrets
• https://github.com/mozilla/sops
• https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/