Secrets Management Done Right: HashiCorp Vault for Infrastructure and Applications

The Problem With How You Manage Secrets Today

Let us be direct about common practices we find during security assessments:

• Database passwords in .env files committed to Git (even if the repo is private).
• Kubernetes Secrets that are base64-encoded (not encrypted) and stored in plain YAML in a GitOps repo.
• Shared service account keys copied between team members via Slack.
• API tokens that were created two years ago by someone who no longer works at the company.

None of these are security. Base64 is an encoding, not encryption. A private Git repo is one misconfiguration away from being public. And shared, long-lived credentials violate every principle of least privilege.

HashiCorp Vault solves these problems by providing centralized secrets management with access control, audit logging, encryption, and — most importantly — dynamic, short-lived credentials that are generated on demand and automatically revoked.

graph TD
    K8S[Kubernetes Pods] -->|K8s Auth| V[HashiCorp Vault]
    TF[Terraform] -->|Vault Provider| V
    ARGO[ArgoCD / ESO] -->|API| V
    CI[GitLab CI] -->|JWT Auth| V
    V -->|Dynamic Creds| DB[(Databases)]
    V -->|PKI Certs| TLS[TLS Certificates]
    V -->|Static Secrets| KV[KV Store]

Vault Architecture for Production

A production Vault deployment requires high availability. We deploy Vault on Kubernetes using the official Helm chart with integrated Raft storage, which eliminates the need for a separate Consul cluster:

# values-vault.yaml
server:
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      config: |
        ui = true

        listener "tcp" {
          tls_disable = 0
          address = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_cert_file = "/vault/tls/tls.crt"
          tls_key_file = "/vault/tls/tls.key"
        }

        storage "raft" {
          path = "/vault/data"

          retry_join {
            leader_api_addr = "https://vault-0.vault-internal:8200"
            leader_tls_servername = "vault.vault.svc"
          }
          retry_join {
            leader_api_addr = "https://vault-1.vault-internal:8200"
            leader_tls_servername = "vault.vault.svc"
          }
          retry_join {
            leader_api_addr = "https://vault-2.vault-internal:8200"
            leader_tls_servername = "vault.vault.svc"
          }
        }

        service_registration "kubernetes" {}

  dataStorage:
    enabled: true
    size: 10Gi
    storageClass: fast-ssd

  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      memory: 512Mi

  # TLS from cert-manager
  volumes:
    - name: tls
      secret:
        secretName: vault-tls
  volumeMounts:
    - name: tls
      mountPath: /vault/tls
      readOnly: true

ui:
  enabled: true

injector:
  enabled: true
  resources:
    requests:
      memory: 64Mi
      cpu: 50m

Deploy with:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm upgrade --install vault hashicorp/vault \
  --namespace vault --create-namespace \
  --values values-vault.yaml

After deployment, you need to initialize and unseal Vault. For production, use auto-unseal with a cloud KMS (AWS KMS, GCP Cloud KMS, or Azure Key Vault) so that pods can restart without manual intervention:

# Add to the raft config above for auto-unseal
seal "awskms" {
  region     = "eu-central-1"
  kms_key_id = "alias/vault-unseal"
}

Kubernetes Authentication

The most natural way for pods to authenticate to Vault is the Kubernetes auth method. It uses the pod’s service account token — which Kubernetes already provides — as proof of identity:

# Enable Kubernetes auth
vault auth enable kubernetes

# Configure it to talk to the Kubernetes API
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443"

# Create a policy
vault policy write my-api-read - <<EOF
path "secret/data/my-api/*" {
  capabilities = ["read"]
}
path "database/creds/my-api-readonly" {
  capabilities = ["read"]
}
EOF

# Bind the policy to a Kubernetes service account
vault write auth/kubernetes/role/my-api \
  bound_service_account_names=my-api \
  bound_service_account_namespaces=production \
  policies=my-api-read \
  ttl=1h

This configuration says: if a pod in the production namespace running as the my-api service account authenticates to Vault, grant it the my-api-read policy for one hour. The policy allows reading static secrets under secret/data/my-api/ and generating dynamic database credentials.

Dynamic Secrets: The Real Power

Static secrets in Vault are already better than .env files because access is controlled, audited, and centralized. But dynamic secrets are where Vault fundamentally changes your security posture.

Instead of a single, long-lived database password shared by every instance of your application, Vault generates a unique, short-lived credential for each pod on demand:

# Enable the database secrets engine
vault secrets enable database

# Configure a PostgreSQL connection
vault write database/config/my-postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="my-api-readonly,my-api-readwrite" \
  connection_url="postgresql://{{username}}:{{password}}@postgres.database.svc:5432/mydb?sslmode=require" \
  username="vault_admin" \
  password="initial-password"

# Rotate the root password so even admins don't know it
vault write -force database/rotate-root/my-postgres

# Create a read-only role
vault write database/roles/my-api-readonly \
  db_name=my-postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
    GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  revocation_statements="DROP ROLE IF EXISTS \"{{name}}\";" \
  default_ttl=1h \
  max_ttl=24h

Now when a pod requests credentials via vault read database/creds/my-api-readonly, it gets a unique username and password that expire in one hour. If that pod is compromised, the blast radius is limited to one hour and read-only access. Compare that to a shared password in an environment variable that grants full access forever.

Injecting Secrets into Pods

The Vault Agent Injector (deployed by the Helm chart) uses annotations to inject secrets into pods as files:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-api
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "my-api"
        vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/my-api-readonly"
        vault.hashicorp.com/agent-inject-template-db-creds: |
          {{- with secret "database/creds/my-api-readonly" -}}
          DB_HOST=postgres.database.svc
          DB_PORT=5432
          DB_NAME=mydb
          DB_USER={{ .Data.username }}
          DB_PASS={{ .Data.password }}
          {{- end }}
    spec:
      serviceAccountName: my-api
      containers:
        - name: my-api
          image: my-registry/my-api:v1.2.3
          command: ["sh", "-c", "source /vault/secrets/db-creds && exec python main.py"]

The injector adds a sidecar container that authenticates to Vault, retrieves the secrets, writes them to a shared volume at /vault/secrets/, and renews them before they expire. Your application reads a file instead of an environment variable — and that file is updated automatically when credentials rotate.

For applications that support it, the External Secrets Operator is an alternative that syncs Vault secrets into Kubernetes Secrets, which some frameworks find easier to consume.

PKI: Automated Certificate Management

Vault’s PKI secrets engine can act as your internal certificate authority, issuing short-lived TLS certificates automatically:

# Enable PKI
vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki

# Generate root CA
vault write -field=certificate pki/root/generate/internal \
  common_name="Company Internal CA" \
  ttl=87600h > ca_cert.pem

# Enable intermediate CA
vault secrets enable -path=pki_int pki
vault write -field=csr pki_int/intermediate/generate/internal \
  common_name="Company Intermediate CA" > intermediate.csr

# Sign and set the intermediate
vault write -field=certificate pki/root/sign-intermediate \
  csr=@intermediate.csr ttl=43800h > intermediate_cert.pem
vault write pki_int/intermediate/set-signed certificate=@intermediate_cert.pem

# Create a role for issuing certificates
vault write pki_int/roles/internal-tls \
  allowed_domains="svc.cluster.local,internal.company.com" \
  allow_subdomains=true \
  max_ttl=72h

Pair this with cert-manager’s Vault issuer, and every Ingress and service mesh certificate is automatically provisioned and rotated.

Integration with Terraform

In our GitOps pipeline, Terraform provisions infrastructure. It also needs secrets — cloud credentials, API tokens, database passwords for initial setup. The Vault provider for Terraform lets you read secrets at plan/apply time without storing them in state:

provider "vault" {
  address = "https://vault.internal.company.com"
  # Auth via OIDC or token from CI/CD
}

data "vault_generic_secret" "db_admin" {
  path = "secret/data/infrastructure/db-admin"
}

resource "postgresql_role" "app_user" {
  name     = "app_user"
  password = data.vault_generic_secret.db_admin.data["password"]
}

Better yet, use Vault’s dynamic AWS credentials so Terraform itself uses short-lived IAM credentials:

data "vault_aws_access_credentials" "deploy" {
  backend = "aws"
  role    = "terraform-deploy"
  type    = "sts"
}

provider "aws" {
  access_key = data.vault_aws_access_credentials.deploy.access_key
  secret_key = data.vault_aws_access_credentials.deploy.secret_key
  token      = data.vault_aws_access_credentials.deploy.security_token
  region     = "eu-central-1"
}

Integration with ArgoCD

ArgoCD needs to deploy applications that reference Vault secrets. There are two clean patterns:

Pattern 1: ArgoCD Vault Plugin. The AVP replaces <placeholder> tokens in manifests with Vault values during rendering:

# In the GitOps repo
apiVersion: v1
kind: Secret
metadata:
  name: my-api-config
  annotations:
    avp.kubernetes.io/path: "secret/data/my-api/config"
type: Opaque
stringData:
  API_KEY: <API_KEY>
  WEBHOOK_SECRET: <WEBHOOK_SECRET>

Pattern 2: External Secrets Operator. ESO runs as a controller that syncs Vault secrets into Kubernetes Secrets independently of ArgoCD:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-api-config
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: my-api-config
  data:
    - secretKey: API_KEY
      remoteRef:
        key: secret/data/my-api/config
        property: API_KEY

We prefer Pattern 2 because it separates concerns: ArgoCD manages deployments, ESO manages secrets, and neither needs special plugins.

Audit and Compliance

Every Vault operation is logged. Enable the audit log to capture who accessed what and when:

vault audit enable file file_path=/vault/audit/audit.log

In a Kubernetes deployment, ship these logs to your observability stack via Loki. You can then build Grafana dashboards showing secret access patterns, failed authentication attempts, and policy violations. This is the kind of evidence that compliance auditors ask for — and with Vault, it is generated automatically.

Migration Path

If you are currently using Kubernetes Secrets or environment variables, here is a pragmatic migration path:

1. Deploy Vault with the Helm chart. Enable Kubernetes auth.
2. Migrate one non-critical application’s secrets to Vault. Use the agent injector.
3. Validate that credential rotation works as expected.
4. Roll out to remaining applications, starting with staging environments.
5. Enable dynamic secrets for databases once you are comfortable with the operational model.
6. Remove plaintext secrets from your Git repositories. Audit Git history with tools like trufflehog to ensure no secrets remain in old commits.

Wrapping Up

Secrets management is not a nice-to-have — it is a fundamental security control. Vault provides centralized access control, comprehensive audit logging, and dynamic credentials that limit blast radius. Combined with the GitOps pipeline and observability stack we have covered in this series, it completes the foundation for operating infrastructure that is both agile and secure.

The initial investment in Vault pays for itself the first time you need to rotate credentials across your entire platform and can do it in minutes instead of days. Start small, prove the value with one application, and expand from there.