Secrets Management

Committing a secret to source control is dangerous because Git history is permanent: even if you delete the secret in a later commit, it remains recoverable in the history, and anyone with repository access (or anyone if the repo is or becomes public) can extract it. Leaked credentials are routinely scraped from public repos within minutes. The correct response to an accidentally committed secret is to rotate it immediately (invalidate and reissue), not just delete the line. Secrets belong in dedicated secret managers or injected environment variables, never in code.

A secrets manager (Vault, AWS Secrets Manager, Azure Key Vault) centralizes secret storage with capabilities a config file can never offer: fine-grained access control (who/what can read each secret), audit logging (every access recorded), encryption at rest and in transit, automatic rotation, and — powerfully — dynamic secrets: it can generate short-lived, on-demand credentials (e.g. a database username/password valid for one hour) rather than handing out long-lived static ones. This dramatically shrinks the window of exposure if a credential leaks.

Rotation means regularly replacing a secret with a new value — and crucially, doing so the moment exposure is suspected. Its purpose is to bound the lifetime of any leaked credential: even if an attacker obtains a key, it becomes useless after the next rotation. Manual rotation is error-prone, so mature setups automate it (the secrets manager rotates database passwords on a schedule and updates consumers). Dynamic, short-lived credentials take this to the limit — they effectively rotate on every use — minimizing the value of any single stolen secret.

A KMS manages cryptographic keys with hardware-backed security and access control, and performs encrypt/decrypt operations without exposing the master key. Envelope encryption is the standard pattern: instead of asking the KMS to encrypt large data directly, you generate a data encryption key (DEK), encrypt your data locally with it, then ask the KMS to encrypt (wrap) the DEK with its master key encryption key (KEK). You store the encrypted data alongside the wrapped DEK. The master key never leaves the KMS, while you still get fast local bulk encryption.

A Kubernetes Secret is only base64-encoded by default — that is encoding, not encryption — and is stored in etcd readable by anyone with sufficient API/etcd access. Improvements: enable encryption at rest for etcd (often KMS-backed), apply strict RBAC so only the right service accounts can read specific secrets, avoid mounting secrets where not needed, and integrate an external secret store — e.g. the External Secrets Operator syncing from Vault/AWS, or Sealed Secrets that lets you safely commit an encrypted secret to Git. These layers turn a lightly-protected object into a properly secured one.