Dynamic credentials and just-in-time access — Victor Agnew

The problem

Credentials were rotated every six months. Fixing that meant answering four questions at once: where do secrets live, how do they rotate on their own, how do we stop credential files from lying around, and how does all of it work in Kubernetes without a person in the loop?

Options

Sealed Secrets keeps encrypted secrets next to the infrastructure code and is easy to set up. But encrypted values are hard to read, the encryption key is created and stored by hand (lose it and everything is lost), and every change means building a new SealedSecret by hand.

Other common answers: Mozilla SOPS, AWS and GCP Secret Manager, Infisical, Bitwarden, 1Password.

HashiCorp Vault is secure by default, supports many auth methods (OIDC, JWT, LDAP, GitHub, tokens), is production-ready, and has a simple UI and a large ecosystem. Its cost is that it's complicated to set up and run, so most of the work below goes into automating that away.

What I built

Vault, set up by a job

Vault runs in-cluster in high-availability mode with Raft storage, auto-unseals through the cloud KMS, and uses workload identity to reach cloud services. Nobody runs the setup by hand: a Kubernetes Job, part of the infrastructure rollout, waits for Vault to come up, initialises it, stores the recovery keys in Secret Manager, enables the secret engines, policies and roles, and finally revokes the root token.

Secrets into Kubernetes

The Vault Secrets Operator (VSO) syncs secrets from Vault into native Kubernetes Secrets, detects and reconciles drift, and restarts the deployments that use a secret when its value changes. It authenticates with Kubernetes service-account tokens, rotates its own tokens, and each role is scoped to specific namespaces and policies.

Dynamic database credentials

Static secrets were only the first step. Vault's database engine creates a database user per lease from a creation statement and removes it with a revocation statement. Services receive credentials that rotate every 24 hours with zero downtime: VSO renews the lease, writes the new credentials and restarts the deployment.

People, through Boundary

Developers reach databases and clusters through HashiCorp Boundary: a controller with Google sign-in, and a worker inside each environment's network. Vault brokers short-lived credentials into each session, whether that's a database user or a Kubernetes service account for exec-ing into pods.

Just-in-time approval

Production is protected. You give a reason, the request lands in Slack, and an approver clicks approve. The approval creates a temporary Boundary role that lets that one person authorize a session on that one target, then deletes the role seconds later. The session keeps running; the permission doesn't.

One command

A small CLI wraps all of it: Google sign-in, pick the environment and the target, wait for approval when it's production, open the tunnel, and launch the database client with the brokered credentials already filled in.

Results

• Credentials rotate every 24 hours, with zero downtime, instead of every six months.
• Production sessions are requested, approved in Slack and audited.
• Every session gets short-lived credentials brokered by Vault.
• Vault and Boundary set themselves up as part of the infrastructure rollout, with the root token revoked at the end.