Kubernetes Secrets Management with External Secrets Operator: Production Guide

External Secrets Operator for Kubernetes Secrets Management

External Secrets Operator Kubernetes integration solves one of the most persistent challenges in container orchestration: managing sensitive configuration data securely. Native Kubernetes Secrets are base64-encoded (not encrypted), stored in etcd, and difficult to rotate. External Secrets Operator (ESO) bridges Kubernetes with enterprise secret management platforms, automatically syncing secrets from AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, and GCP Secret Manager into your clusters.

This guide covers installing and configuring ESO for production environments, including multi-cluster synchronization, automatic rotation, and disaster recovery patterns. Moreover, you will learn how to implement zero-trust secret access policies that satisfy SOC 2 and PCI-DSS compliance requirements. Because secrets touch every service in a cluster, getting this layer right early prevents a painful retrofit later.

Why Native Kubernetes Secrets Fall Short

Kubernetes Secrets have fundamental limitations that make them unsuitable for production secret management. They are only base64-encoded — anyone with RBAC access to read secrets can decode them trivially. Furthermore, secrets stored in etcd may not be encrypted at rest depending on your cluster configuration. Even when EncryptionConfiguration is enabled, the decryption key often lives next to etcd, which limits how much real protection it adds against a host compromise.

Secret rotation requires redeploying pods. There is no native audit trail for who accessed which secret and when. Additionally, sharing secrets across multiple clusters requires manual synchronization or custom tooling that often becomes a maintenance burden. In practice, teams accumulate copies of the same credential across namespaces and clusters, and rotating that credential becomes a frightening exercise because nobody is sure where every copy lives.

How the Operator Actually Works Under the Hood

It helps to understand the control loop before wiring anything up. ESO runs a controller that reconciles three core custom resources. A SecretStore (or its cluster-scoped sibling ClusterSecretStore) describes where secrets live and how to authenticate to that provider. An ExternalSecret describes which remote keys to fetch and how to shape them into a native Kubernetes Secret. The controller then polls the provider on the refreshInterval you set, compares the fetched values against the materialized secret, and writes an update only when something changed.

Consequently, your application never talks to AWS or Vault directly. It mounts an ordinary Kubernetes Secret, exactly as it always has, while ESO does the privileged fetching on a service account that your workloads cannot read. That separation is the whole point: the blast radius of an application compromise no longer includes your provider credentials, because the workload only ever sees the materialized, namespace-scoped result.

Installing External Secrets Operator

# Install ESO via Helm
helm repo add external-secrets https://charts.external-secrets.io
helm repo update

helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --set installCRDs=true \
  --set webhook.port=9443 \
  --set certController.requeueInterval=5m

External Secrets Operator Kubernetes: AWS Provider Setup

# cluster-secret-store.yaml — Cluster-wide secret store
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets
---
# IRSA (IAM Role for Service Account) configuration
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets-sa
  namespace: external-secrets
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/external-secrets-role

# IAM policy for the ESO role (Terraform)
resource "aws_iam_policy" "external_secrets" {
  name = "external-secrets-policy"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "secretsmanager:GetSecretValue",
          "secretsmanager:DescribeSecret",
          "secretsmanager:ListSecretVersionIds"
        ]
        Resource = "arn:aws:secretsmanager:us-east-1:123456789:secret:myapp/*"
      },
      {
        Effect = "Allow"
        Action = [
          "kms:Decrypt",
          "kms:DescribeKey"
        ]
        Resource = "arn:aws:kms:us-east-1:123456789:key/mrk-*"
      }
    ]
  })
}

Notice how tightly the IAM resource is scoped: the policy grants GetSecretValue only on myapp/*, not on every secret in the account. This least-privilege boundary matters because the ESO service account effectively becomes a high-value credential. The docs recommend scoping each ClusterSecretStore to its own role so a misconfigured ExternalSecret in one team’s namespace cannot reach another team’s secrets. For Vault or Azure Key Vault, the same principle applies through Vault policies or Azure RBAC role assignments respectively.

Syncing Secrets from External Providers

Therefore, ExternalSecret resources define which secrets to sync and how to map them into Kubernetes Secrets. ESO watches these resources and automatically creates or updates the corresponding Kubernetes Secret whenever the source changes. The template block is where ESO earns its keep — instead of dumping raw values, you can assemble derived fields like a full connection string from individual properties.

# external-secret.yaml — Sync database credentials
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: production
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
    deletionPolicy: Retain
    template:
      type: Opaque
      data:
        DB_HOST: "{{ .host }}"
        DB_PORT: "{{ .port }}"
        DB_USERNAME: "{{ .username }}"
        DB_PASSWORD: "{{ .password }}"
        DB_CONNECTION_STRING: "postgresql://{{ .username }}:{{ .password }}@{{ .host }}:{{ .port }}/{{ .database }}"
  data:
    - secretKey: host
      remoteRef:
        key: myapp/production/database
        property: host
    - secretKey: port
      remoteRef:
        key: myapp/production/database
        property: port
    - secretKey: username
      remoteRef:
        key: myapp/production/database
        property: username
    - secretKey: password
      remoteRef:
        key: myapp/production/database
        property: password
    - secretKey: database
      remoteRef:
        key: myapp/production/database
        property: database

Two fields deserve emphasis. The creationPolicy: Owner setting means ESO owns the resulting Secret and will overwrite manual edits — which is exactly what you want for drift prevention. Meanwhile deletionPolicy: Retain ensures that if someone accidentally deletes the ExternalSecret, the materialized Secret survives, so running pods do not lose their credentials mid-flight. For high-cardinality secrets you can also skip enumerating every property and use dataFrom with extract, which pulls an entire JSON secret and maps every key automatically.

Automatic Secret Rotation

Consequently, ESO can trigger pod restarts when secrets are rotated in the source provider. Combined with AWS Secrets Manager’s automatic rotation for RDS credentials, this creates a fully automated rotation pipeline. The missing link is that a refreshed Kubernetes Secret does not, by itself, restart pods that consumed it through envFrom. A common pattern is to pair ESO with Stakater Reloader, which watches referenced secrets and performs a rolling restart when their contents change.

# Deployment with secret rotation awareness
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  namespace: production
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  template:
    spec:
      containers:
        - name: api
          image: myapp/api:latest
          envFrom:
            - secretRef:
                name: db-credentials
          volumeMounts:
            - name: tls-certs
              mountPath: /etc/tls
              readOnly: true
      volumes:
        - name: tls-certs
          secret:
            secretName: api-tls-cert

There is an important edge case here. Environment variables are read once at process start, so an env-injected secret only takes effect after a restart — that is why Reloader is needed. Secrets mounted as volumes, by contrast, are eventually updated in place by the kubelet (typically within a minute), so applications that re-read a file on a timer can pick up rotations without a restart. For long-lived database connections, prefer the restart approach so stale connection pools are torn down cleanly rather than failing on the next reconnect.

Multi-Cluster Secret Synchronization

Additionally, for organizations running multiple Kubernetes clusters, ESO enables consistent secret distribution from a single source of truth. Each cluster runs its own ESO instance pointing to the same secret store, ensuring all environments have synchronized credentials. The ClusterExternalSecret resource extends this idea inside a single cluster by fanning a secret out to every namespace that matches a label selector.

# ClusterExternalSecret — sync across all namespaces
apiVersion: external-secrets.io/v1beta1
kind: ClusterExternalSecret
metadata:
  name: shared-tls-certificates
spec:
  namespaceSelector:
    matchLabels:
      requires-tls: "true"
  externalSecretSpec:
    refreshInterval: 15m
    secretStoreRef:
      name: aws-secrets-manager
      kind: ClusterSecretStore
    target:
      name: shared-tls
      creationPolicy: Owner
    data:
      - secretKey: tls.crt
        remoteRef:
          key: shared/tls/wildcard-cert
          property: certificate
      - secretKey: tls.key
        remoteRef:
          key: shared/tls/wildcard-cert
          property: private_key

Observability and Failure Modes Worth Watching

Once ESO is in the critical path, you need to know when sync breaks. ESO emits Prometheus metrics — most usefully externalsecret_sync_calls_error and externalsecret_status_condition — which let you alert on secrets that have stopped reconciling. Every ExternalSecret also carries a SecretSynced status condition, so a quick kubectl get externalsecret -A surfaces anything stuck in SecretSyncedError. Set an alert on sync errors lasting longer than a couple of refresh intervals, because a silently failing sync means your secret is now frozen at its last-known value.

Watch out for provider-side throttling too. If hundreds of ExternalSecret objects all use an aggressive refreshInterval of 1m, you can hit AWS Secrets Manager API rate limits and cause cascading sync failures. Benchmarks and the project’s own guidance suggest that most secrets change rarely, so a refreshInterval of 1h is plenty for the majority of credentials; reserve tight intervals for things that genuinely rotate often, such as short-lived dynamic database credentials from Vault.

When NOT to Use External Secrets Operator

For simple applications with a handful of non-sensitive configuration values, ESO adds operational overhead that may not be justified. If your secrets rarely change and your cluster already encrypts etcd at rest, native Kubernetes Secrets with RBAC restrictions may suffice. As a result, evaluate the actual threat model before committing to ESO. A two-service hobby cluster does not need a controller, three custom resources, and an IAM trust relationship to hold one API key.

ESO introduces a dependency on external secret providers — if AWS Secrets Manager experiences an outage, new pods cannot start because they cannot fetch their secrets. Plan for this with appropriate caching and fallback strategies. In particular, remember that ESO is a sync tool, not a runtime proxy: once a Secret is materialized it survives a provider outage, but autoscaling events that schedule fresh pods during a provider incident can still stall if the cached Secret was never created. Compared with a sidecar like the Vault Agent Injector, which fetches secrets at pod startup, ESO trades some freshness for far simpler application code and a smaller failure surface. Choose based on whether your priority is real-time revocation or operational simplicity.

Key Takeaways

External Secrets Operator Kubernetes integration transforms secret management from a manual, error-prone process into an automated, auditable pipeline. By syncing secrets from enterprise-grade providers, you get encryption at rest, access auditing, and automatic rotation without changing your application code. Furthermore, the ClusterExternalSecret pattern enables consistent secret distribution across multi-cluster environments, while metrics and status conditions give you the observability to trust that distribution.

Start with a single non-critical application to validate the ESO workflow before rolling it out organization-wide. For more information, see the External Secrets Operator documentation and AWS Secrets Manager best practices. Our guides on Kubernetes network policies, zero trust security architecture, and mTLS in Kubernetes complement this security-focused approach.

In conclusion, Kubernetes Secrets External Secrets is an essential topic for modern software development. By applying the patterns and practices covered in this guide, you can build more robust, scalable, and maintainable systems. Start with the fundamentals, iterate on your implementation, and continuously measure results to ensure you are getting the most value from these approaches.