Secrets Management in Production: Vault, AWS Secrets Manager, and SOPS
Hardcoded credentials in configuration files and environment variables are the number one cause of security breaches in cloud-native applications. Secrets management tools like HashiCorp Vault, AWS Secrets Manager, and Mozilla SOPS solve this by centralizing, encrypting, and automating access to sensitive data. Therefore, this guide covers practical implementation patterns for each tool and helps you choose the right one for your infrastructure. Moreover, it digs into the operational details — rotation, leasing, audit logging, and failure modes — that determine whether a deployment actually improves your security posture or merely moves the problem.
Why Secrets Management Matters
The traditional approach — storing database passwords in .env files, API keys in CI/CD variables, and TLS certificates on disk — creates several problems. Secrets sprawl across dozens of systems with no audit trail. Rotation requires manual updates across every service. A single leaked .env file exposes everything. Moreover, compliance frameworks like SOC 2, PCI DSS, and HIPAA require centralized storage with rotation and audit logging.
A proper system provides four capabilities: centralized storage with encryption at rest, access control with audit logging, automatic rotation, and dynamic credentials that are created on-demand and expire automatically. Additionally, it should integrate with your deployment pipeline so applications never see secrets in plaintext configuration files. The goal is not just to hide secrets, but to make a leaked secret useless within a short, bounded window.
Consider the blast radius of a single static database password shared across forty service instances. If one instance is compromised, the attacker holds a credential that works everywhere and never expires. By contrast, a dynamic credential with a one-hour TTL limits exposure to that instance and that hour. This shift — from “secrets you protect” to “secrets that self-destruct” — is the conceptual heart of the modern discipline.
HashiCorp Vault: The Full-Featured Solution
Vault is the most flexible tool here, supporting static secrets, dynamic credentials, PKI certificates, and encryption as a service. It runs as a standalone server (or cluster) and provides HTTP API, CLI, and UI access. However, this flexibility comes with operational complexity — running Vault in production requires careful attention to storage backends, unsealing, and high availability.
# Initialize Vault with auto-unseal (AWS KMS)
vault operator init -recovery-shares=5 -recovery-threshold=3
# Enable secrets engines
vault secrets enable -path=database database
vault secrets enable -path=kv-v2 kv-v2
# Configure dynamic database credentials
vault write database/config/myapp-db \
plugin_name=postgresql-database-plugin \
allowed_roles="myapp-readonly,myapp-readwrite" \
connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/myapp" \
username="vault_admin" \
password="vault_admin_password"
# Create a role that generates time-limited credentials
vault write database/roles/myapp-readonly \
db_name=myapp-db \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
default_ttl="1h" \
max_ttl="24h"
# Application requests credentials — gets unique, time-limited creds
vault read database/creds/myapp-readonly
# Key Value
# lease_id database/creds/myapp-readonly/abc123
# lease_duration 1h
# username v-token-myapp-r-xyz789
# password A1b2C3d4E5-random-generatedDynamic secrets are Vault’s killer feature. Instead of sharing a single database password across all application instances, each instance gets its own credentials that expire automatically. If an instance is compromised, only its credentials are affected, and they expire within the hour. Consequently, credential rotation happens automatically without any application changes.
Unsealing, Leasing, and the Operational Reality of Vault
Vault’s flexibility hides genuine operational weight, and teams that skip this section tend to learn it during an outage. When a Vault node starts, it boots in a sealed state: it knows nothing, because its data is encrypted by a master key that Vault itself does not hold in plaintext. Until it is unsealed, every request fails. In production you almost never want manual unsealing with Shamir key shares — instead you configure auto-unseal against a cloud KMS, as shown above with -recovery-shares, so a restarted pod rejoins the cluster without a human distributing key fragments.
Leasing is the second concept that surprises newcomers. Every dynamic credential carries a lease, and Vault tracks every outstanding lease. If your application requests a one-hour credential but never renews it, Vault revokes it at expiry by running the role’s revocation SQL. This is powerful, but it means Vault’s storage backend must scale with your credential churn. A fleet generating thousands of short-lived leases per minute puts real load on the backend; teams typically use Raft integrated storage or Consul and monitor lease counts closely.
# Renew a lease before it expires (clients should renew at ~2/3 of TTL)
vault lease renew database/creds/myapp-readonly/abc123
# Revoke immediately on suspected compromise — runs the revocation SQL now
vault lease revoke database/creds/myapp-readonly/abc123
# Revoke every lease under a path (incident response: kill all app DB creds)
vault lease revoke -prefix database/creds/myapp-readonlyThat last command is the reason teams adopt Vault despite the overhead. During an incident, revoking every database credential issued under a path is a single command, and the database connections drop within seconds. Achieving the same with static passwords means a coordinated rotation across every service — exactly the manual scramble that a centralized credential store exists to eliminate.
AWS Secrets Manager: Cloud-Native Simplicity
If your infrastructure is primarily AWS, Secrets Manager is the path of least resistance. It integrates natively with RDS, Lambda, ECS, and EKS. Automatic rotation for RDS credentials works with a single configuration. Furthermore, IAM policies control access, so you use the same permission model as the rest of your AWS infrastructure.
import boto3
import json
from botocore.exceptions import ClientError
# Retrieve a secret — with caching for performance
class SecretCache:
def __init__(self, region="us-east-1"):
self.client = boto3.client("secretsmanager", region_name=region)
self._cache = {}
def get_secret(self, secret_name):
if secret_name in self._cache:
return self._cache[secret_name]
try:
response = self.client.get_secret_value(SecretId=secret_name)
secret = json.loads(response["SecretString"])
self._cache[secret_name] = secret
return secret
except ClientError as e:
if e.response["Error"]["Code"] == "ResourceNotFoundException":
raise ValueError(f"Secret {secret_name} not found")
raise
def invalidate(self, secret_name):
self._cache.pop(secret_name, None)
# Usage in application
secrets = SecretCache()
db_creds = secrets.get_secret("prod/myapp/database")
connection_string = f"postgresql://{db_creds['username']}:{db_creds['password']}@{db_creds['host']}/myapp"
# Enable automatic rotation for RDS secrets
# AWS provides Lambda rotation functions out of the box
# aws secretsmanager rotate-secret --secret-id prod/myapp/database \
# --rotation-lambda-arn arn:aws:lambda:us-east-1:123456:function:SecretsManagerRotation \
# --rotation-rules AutomaticallyAfterDays=30The naive caching above hides a subtle edge case: when Secrets Manager rotates a secret, your in-process cache goes stale. AWS solves this with a multi-stage versioning model that every production integration should understand. Each secret value has staging labels — AWSCURRENT is the active version, AWSPENDING is the new value mid-rotation, and AWSPREVIOUS is the last good value. The rotation Lambda creates a pending version, tests it, then atomically promotes it to current. Because both old and new values stay valid during the overlap window, clients that briefly cache AWSPREVIOUS continue to authenticate while they pick up the new value.
For long-lived processes, the AWS-provided client-side caching libraries (in Python, Java, .NET, and Go) handle TTL-based refresh automatically, so you don’t hand-roll the invalidation logic shown above. A common production mistake is caching forever and never refreshing — then a rotation silently breaks every instance an hour later. Set a cache TTL (five minutes is typical) and let the library re-fetch.
Mozilla SOPS: Encrypted Files in Git
SOPS (Secrets OPerationS) takes a different approach — it encrypts secret values within configuration files while leaving keys visible. This means you can store encrypted secrets directly in Git, review diffs, and use your existing deployment pipeline. SOPS supports AWS KMS, GCP KMS, Azure Key Vault, and age/PGP for encryption.
# secrets.enc.yaml — encrypted with SOPS
# Keys are visible (for diffs), values are encrypted
database:
host: ENC[AES256_GCM,data:abc123...==,tag:xyz...]
port: ENC[AES256_GCM,data:def456...==,tag:uvw...]
username: ENC[AES256_GCM,data:ghi789...==,tag:rst...]
password: ENC[AES256_GCM,data:jkl012...==,tag:opq...]
api_keys:
stripe: ENC[AES256_GCM,data:mno345...==,tag:lmn...]
sendgrid: ENC[AES256_GCM,data:pqr678...==,tag:ijk...]
sops:
kms:
- arn: arn:aws:kms:us-east-1:123456:key/abc-def-ghi
version: 3.8.1# Encrypt a file
sops --encrypt --kms arn:aws:kms:us-east-1:123456:key/abc secrets.yaml > secrets.enc.yaml
# Edit encrypted file (decrypts in temp editor, re-encrypts on save)
sops secrets.enc.yaml
# Decrypt for deployment
sops --decrypt secrets.enc.yaml > /tmp/secrets.yaml
# Use with Kubernetes — decrypt and apply
sops --decrypt k8s-secrets.enc.yaml | kubectl apply -f -
# Use with Terraform
# terraform plan -var-file=<(sops --decrypt terraform.enc.tfvars)SOPS works exceptionally well for GitOps workflows where all configuration lives in Git. You get version history, pull request reviews, and audit trails through Git itself. However, SOPS doesn't handle dynamic secrets or automatic rotation — it's purely a storage and encryption solution.
One detail that pays off in practice is the .sops.yaml creation-rules file. Rather than passing the KMS ARN on every command, you declare path-based rules so the right key is selected automatically. This also lets you encrypt only specific fields — for example, leaving non-sensitive config in cleartext while encrypting anything matching a password or token regex, which keeps diffs readable.
# .sops.yaml — automatic key selection and field-level encryption
creation_rules:
- path_regex: secrets/prod/.*\.yaml$
kms: arn:aws:kms:us-east-1:123456:key/prod-key
encrypted_regex: '^(password|token|secret|.*_key)