HashiCorp Vault on Kubernetes: Production HA Setup Guide

Deploy HashiCorp Vault on Kubernetes for production: Raft HA Helm values, TLS, KMS auto-unseal, Kubernetes auth, injector vs ESO, and snapshot backups.

This guide deploys HashiCorp Vault on Kubernetes the way you would run it in production: a three-node Raft cluster installed with the official Helm chart, TLS on every listener, auto-unseal through a cloud KMS, Kubernetes authentication for workloads, and scheduled Raft snapshots you can actually restore.

It assumes you already know what Vault is and how its CLI works. If not, start with How to install and set up HashiCorp Vault in Linux, which covers the basics (secrets engines, policies, tokens) on a single server.

Everything below was run on a four-node kind cluster (Kubernetes 1.37) with Vault Helm chart 0.34.1 (Vault 2.0.4), cert-manager 1.21 and External Secrets Operator 2.14. The test cluster had no cloud KMS, so it auto-unsealed with Vault’s transit seal (described below) instead of awskms; everything else is identical.

What you need

  • A Kubernetes cluster with at least three worker nodes. The chart’s default anti-affinity places each Vault pod on a different node, so with fewer nodes some pods stay Pending.
  • A default StorageClass backed by SSDs. Raft writes every change to disk and is sensitive to slow fsyncs.
  • kubectl and helm.
  • cert-manager for the TLS certificate (or any other way to get a certificate into a Secret).
  • A key for auto-unseal: AWS KMS, Google Cloud KMS, Azure Key Vault, or another Vault with the transit engine.

Add the chart repository:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm search repo hashicorp/vault
NAME                            	CHART VERSION	APP VERSION	DESCRIPTION
hashicorp/vault                 	0.34.1       	2.0.4      	Official HashiCorp Vault Chart
hashicorp/vault-secrets-operator	1.6.2        	1.6.2      	Official Vault Secrets Operator Chart

Create the TLS certificate

Run Vault with TLS inside the cluster too, not only at the ingress. Clients (External Secrets, the Vault Agent sidecar, your apps) send tokens on every request, and the Raft retry_join below uses the API port to find the leader.

The certificate needs the in-cluster service names and the per-pod names behind the vault-internal headless service. If you already have an internal CA issuer in cert-manager, use it. Otherwise this bootstraps a private CA for Vault (certs.yaml):

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned
  namespace: vault
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: vault-ca
  namespace: vault
spec:
  isCA: true
  commonName: vault-ca
  secretName: vault-ca
  duration: 87600h # 10 years
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned
    kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-ca
  namespace: vault
spec:
  ca:
    secretName: vault-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: vault-tls
  namespace: vault
spec:
  secretName: vault-tls
  duration: 8760h # 1 year
  renewBefore: 720h # 30 days
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - vault
    - vault.vault.svc
    - vault.vault.svc.cluster.local
    - vault-active.vault.svc
    - "*.vault-internal"
    - "*.vault-internal.vault.svc"
    - "*.vault-internal.vault.svc.cluster.local"
  ipAddresses:
    - 127.0.0.1
  issuerRef:
    name: vault-ca
    kind: Issuer
kubectl create namespace vault
kubectl apply -f certs.yaml
kubectl -n vault wait --for=condition=Ready certificate/vault-tls --timeout=60s

The vault-tls Secret now holds tls.crt, tls.key and ca.crt, and cert-manager renews it 30 days before expiry. Vault re-reads its certificate on SIGHUP; the simplest way to pick up a renewed one is a rolling restart (see Upgrades and restarts).

Helm values for a Raft HA cluster

Save as values.yaml:

global:
  tlsDisable: false

injector:
  enabled: false # using External Secrets Operator; see below

server:
  image:
    repository: hashicorp/vault
    tag: "2.0.4"

  resources:
    requests:
      cpu: 250m
      memory: 256Mi
    limits:
      memory: 1Gi

  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-tls/ca.crt
    VAULT_TLSCERT: /vault/userconfig/vault-tls/tls.crt
    VAULT_TLSKEY: /vault/userconfig/vault-tls/tls.key

  volumes:
    - name: userconfig-vault-tls
      secret:
        secretName: vault-tls
        defaultMode: 0440
  volumeMounts:
    - name: userconfig-vault-tls
      mountPath: /vault/userconfig/vault-tls
      readOnly: true

  dataStorage:
    enabled: true
    size: 10Gi
    storageClass: null # cluster default; use an SSD-backed class

  auditStorage:
    enabled: true
    size: 10Gi

  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = true

        listener "tcp" {
          address         = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_cert_file   = "/vault/userconfig/vault-tls/tls.crt"
          tls_key_file    = "/vault/userconfig/vault-tls/tls.key"
        }

        storage "raft" {
          path = "/vault/data"

          retry_join {
            leader_api_addr     = "https://vault-0.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
          }
          retry_join {
            leader_api_addr     = "https://vault-1.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
          }
          retry_join {
            leader_api_addr     = "https://vault-2.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
          }
        }

        seal "awskms" {
          region     = "eu-west-1"
          kms_key_id = "1234abcd-12ab-34cd-56ef-1234567890ab"
        }

        service_registration "kubernetes" {}

  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/vault-unseal

ui:
  enabled: true

What matters in it:

  • ha.raft with retry_join: Vault’s integrated storage, so there is no Consul or external database to run. Every pod lists all three peers, so a new or replaced pod finds the leader and joins on its own; there is no manual vault operator raft join step.
  • setNodeId: true: the Raft node ID is the pod name (vault-0, …), which keeps list-peers readable and stable across restarts.
  • TLS: the listener uses the cert-manager Secret mounted under /vault/userconfig/, and VAULT_CACERT lets the vault CLI inside the pods trust it.
  • auditStorage: a separate volume for the audit log, so audit growth cannot fill the Raft data volume.
  • What the chart already does for you: a PodDisruptionBudget with maxUnavailable: 1 (server.ha.disruptionBudget), required pod anti-affinity across nodes, a non-root security context, and a readiness probe that only passes on an unsealed node. You do not need to add these.
  • ui.enabled creates a vault-ui ClusterIP service. Expose it through your ingress controller only if you need the UI outside the cluster, and configure the ingress to talk HTTPS to port 8200.

Render the chart and validate it against the cluster before installing:

helm template vault hashicorp/vault --version 0.34.1 -n vault -f values.yaml \
  | kubectl apply --dry-run=server -f -

Auto-unseal

A Vault node starts sealed and cannot serve requests until its root key is decrypted. With the default Shamir seal, that needs a quorum of people entering unseal keys every time any pod restarts: a node drain or cluster upgrade at 3 a.m. leaves Vault down until someone shows up. Auto-unseal hands that job to a KMS key, so pods unseal themselves when they start.

The seal "awskms" block above uses an AWS KMS key. The pod gets AWS credentials through IRSA (the eks.amazonaws.com/role-arn annotation on the service account); the IAM role needs kms:Encrypt, kms:Decrypt and kms:DescribeKey on that key. EKS Pod Identity works too, without the annotation. Other clouds use their own seal block in the same place:

# Google Cloud KMS (credentials via Workload Identity)
seal "gcpckms" {
  project    = "my-project"
  region     = "global"
  key_ring   = "vault"
  crypto_key = "vault-unseal"
}

# Azure Key Vault (credentials via Workload Identity)
seal "azurekeyvault" {
  tenant_id  = "00000000-0000-0000-0000-000000000000"
  vault_name = "my-key-vault"
  key_name   = "vault-unseal"
}

On-premises, without a cloud KMS, use the transit seal: a second, small Vault whose transit engine holds the key.

seal "transit" {
  address    = "https://vault-unsealer.example.com:8200"
  key_name   = "autounseal"
  mount_path = "transit/"
}

Its token goes in the VAULT_TOKEN environment variable, from a Secret:

server:
  extraSecretEnvironmentVars:
    - envName: VAULT_TOKEN
      secretName: vault-unseal-token
      secretKey: token

Make it an orphan, periodic token whose policy allows only update on transit/encrypt/autounseal and transit/decrypt/autounseal. The vault CLI inside the pods also picks that variable up, so pass your own VAULT_TOKEN explicitly when you kubectl exec into them.

Whichever seal you use, that key now protects all of Vault’s data. Deleting it, or losing access to it, makes the Raft data and every snapshot unreadable. Protect it from deletion and use it for nothing else.

Install and initialize

helm install vault hashicorp/vault --version 0.34.1 -n vault -f values.yaml
kubectl -n vault get pods -o wide

The three pods run on three different nodes but are not ready, because Vault is not initialized yet:

NAME      READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
vault-0   0/1     Running   0          30s   10.244.3.4   rw-c-worker    <none>           <none>
vault-1   0/1     Running   0          30s   10.244.1.7   rw-c-worker2   <none>           <none>
vault-2   0/1     Running   0          30s   10.244.2.6   rw-c-worker3   <none>           <none>

Initialize once, on vault-0. With auto-unseal you get recovery keys instead of unseal keys:

kubectl -n vault exec vault-0 -- vault operator init \
  -recovery-shares=5 -recovery-threshold=3 -format=json > vault-init.json

vault-init.json holds the five recovery keys and the initial root token. Move it into offline secret storage (split the recovery keys between people) and delete the file. Recovery keys do not unseal Vault, but you need them to generate a new root token, so do not lose them.

The other two pods join through retry_join and unseal through the KMS within seconds:

kubectl -n vault get pods
kubectl -n vault exec vault-0 -- vault status
NAME      READY   STATUS    RESTARTS   AGE
vault-0   1/1     Running   0          73s
vault-1   1/1     Running   0          73s
vault-2   1/1     Running   0          73s
Key                      Value
---                      -----
Seal Type                transit
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  2.0.4
Build Date               2026-08-03T16:14:36Z
Storage Type             raft
Cluster Name             vault-cluster-355cff28
Cluster ID               968f1305-1b40-5704-c775-18ddf702a857
Removed From Cluster     false
HA Enabled               true
HA Cluster               https://vault-0.vault-internal:8201
HA Mode                  active
Active Since             2026-11-10T08:35:10.601002336Z
Raft Committed Index     45
Raft Applied Index       45

With a cloud KMS, Seal Type reads awskms, gcpckms and so on. Log in with the root token and check the Raft peers:

kubectl -n vault exec -it vault-0 -- vault login
kubectl -n vault exec vault-0 -- vault operator raft list-peers
Node       Address                        State       Voter
----       -------                        -----       -----
vault-0    vault-0.vault-internal:8201    leader      true
vault-1    vault-1.vault-internal:8201    follower    true
vault-2    vault-2.vault-internal:8201    follower    true

Prove that auto-unseal works before you rely on it. Delete a pod and check it comes back unsealed without anyone touching it:

kubectl -n vault delete pod vault-1
kubectl -n vault wait --for=condition=Ready pod/vault-1 --timeout=120s
kubectl -n vault exec vault-1 -- vault status | grep -E 'Sealed|HA Mode'
pod/vault-1 condition met
Sealed                   false
HA Mode                  standby

Enable an audit device straight away, so every request is logged from the start:

kubectl -n vault exec vault-0 -- vault audit enable file file_path=/vault/audit/vault_audit.log

When every enabled audit device fails to write, Vault refuses requests rather than serving them unaudited. Ship /vault/audit/vault_audit.log to your log system and watch the auditStorage volume’s free space.

Finally, set up admin access through an auth method (OIDC, or userpass for a small team) with an admin policy, then revoke the root token:

kubectl -n vault exec vault-0 -- vault token revoke -self

Kubernetes authentication

Workloads should log in to Vault with their Kubernetes service account, not with long-lived Vault tokens stored in Secrets. Enable the Kubernetes auth method from a shell in vault-0:

kubectl -n vault exec -it vault-0 -- sh
vault auth enable kubernetes
vault write auth/kubernetes/config kubernetes_host=https://kubernetes.default.svc:443

Because Vault runs in the same cluster, that is all the configuration it needs: it uses its own service account token and CA to review the tokens it receives. The chart grants it the system:auth-delegator role for that (server.authDelegator, on by default).

Store a secret, write a policy for it, and bind a role to the application’s service account and namespace:

vault secrets enable -path=secret kv-v2
vault kv put secret/myapp/config username=app password=example-not-a-real-password

vault policy write myapp-read - <<EOF
path "secret/data/myapp/*" {
  capabilities = ["read"]
}
EOF

vault write auth/kubernetes/role/myapp \
  bound_service_account_names=myapp \
  bound_service_account_namespaces=myapp \
  policies=myapp-read \
  audience=vault \
  ttl=1h

audience=vault makes Vault accept only tokens issued for the audience vault, so a token meant for the API server or another service cannot be replayed against Vault. Without it, Vault accepts the role but warns:

WARNING! The following warnings were returned from Vault:

  * Role myapp-injector does not have an audience configured. While audiences
  are not required, consider specifying one if your use case would benefit
  from additional JWT claim verification.

Getting secrets to pods: injector or External Secrets

There are two common ways to deliver Vault secrets to an application. Pick one per cluster.

Vault Agent InjectorExternal Secrets Operator (ESO)
HowA webhook adds a Vault Agent sidecar that writes secrets to a file in the podA controller copies Vault secrets into ordinary Kubernetes Secrets
Secret stored in etcdNo, only in the pod’s in-memory volumeYes, as a Kubernetes Secret
App changesApp reads a file (/vault/secrets/...)None: envFrom / secretKeyRef as usual
Dynamic secrets with leasesYes, the agent renews themFetched and refreshed on an interval
OverheadOne sidecar per podOne controller per cluster
Other backendsVault onlyAWS, GCP, Azure, Vault and many more

Use the injector when secrets must not land in etcd, or when the app uses dynamic credentials (database, cloud) with leases. Use ESO when apps expect ordinary Kubernetes Secrets, or when you also pull secrets from a cloud provider, as in External Secrets with AWS Secrets Manager.

External Secrets Operator

Install ESO, create the app’s namespace and service account, and make the Vault CA available there:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace --wait

kubectl create namespace myapp
kubectl -n myapp create serviceaccount myapp
kubectl -n vault get secret vault-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
kubectl -n myapp create configmap vault-ca --from-file=ca.crt=ca.crt

A SecretStore that logs in as the myapp service account (ESO requests a token for it with the vault audience), and an ExternalSecret that copies one field (eso.yaml):

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault
  namespace: myapp
spec:
  provider:
    vault:
      server: https://vault.vault.svc:8200
      path: secret
      version: v2
      caProvider:
        type: ConfigMap
        name: vault-ca
        key: ca.crt
      auth:
        kubernetes:
          mountPath: kubernetes
          role: myapp
          serviceAccountRef:
            name: myapp
            audiences:
              - vault
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: myapp-config
  namespace: myapp
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault
    kind: SecretStore
  target:
    name: myapp-config
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: myapp/config
        property: password
kubectl apply -f eso.yaml
kubectl -n myapp get secretstore,externalsecret
NAME                                    AGE   STATUS   CAPABILITIES   READY
secretstore.external-secrets.io/vault   10s   Valid    ReadWrite      True

NAME                                              STORETYPE     STORE   REFRESH INTERVAL   STATUS         READY   LAST SYNC
externalsecret.external-secrets.io/myapp-config   SecretStore   vault   1h                 SecretSynced   True    9s

The myapp-config Secret now exists in the myapp namespace with a DB_PASSWORD key. Current ESO releases serve the external-secrets.io/v1 API; older examples written for v1beta1 need their apiVersion updated.

Vault Agent Injector

To use the injector instead, set injector.enabled: true in values.yaml and run helm upgrade. Copy the CA into the app namespace as a Secret and annotate the pod template:

kubectl -n myapp create secret generic vault-ca --from-file=ca.crt=ca.crt
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "myapp-injector"
        vault.hashicorp.com/tls-secret: "vault-ca"
        vault.hashicorp.com/ca-cert: "/vault/tls/ca.crt"
        vault.hashicorp.com/agent-inject-secret-config: "secret/data/myapp/config"
        vault.hashicorp.com/agent-inject-template-config: |
          {{- with secret "secret/data/myapp/config" -}}
          DB_USER={{ .Data.data.username }}
          DB_PASSWORD={{ .Data.data.password }}
          {{- end }}
    spec:
      serviceAccountName: myapp
      containers:
        - name: app
          image: busybox:1.37
          command: ["sh", "-c", "sleep 3600"]

The agent logs in with the pod’s default service account token, whose audience is the API server’s rather than vault, so it needs a role without audience (this is where the warning above came from):

vault write auth/kubernetes/role/myapp-injector \
  bound_service_account_names=myapp \
  bound_service_account_namespaces=myapp \
  policies=myapp-read \
  ttl=1h

The pod starts with two containers (2/2), and the rendered file is in the app container:

kubectl -n myapp exec deploy/myapp -c app -- cat /vault/secrets/config
DB_USER=app
DB_PASSWORD=example-not-a-real-password

Backups: Raft snapshots

A Raft snapshot is a consistent, point-in-time copy of all of Vault’s data. Vault Enterprise has an automated snapshot agent; on the community edition you schedule vault operator raft snapshot save yourself. A CronJob that logs in with Kubernetes auth keeps it in the cluster with no stored token.

A policy and role for the CronJob’s service account:

vault policy write raft-snapshot - <<EOF
path "sys/storage/raft/snapshot" {
  capabilities = ["read"]
}
EOF

vault write auth/kubernetes/role/raft-snapshot \
  bound_service_account_names=vault-snapshot \
  bound_service_account_namespaces=vault \
  policies=raft-snapshot \
  audience=vault \
  ttl=10m

The CronJob, snapshot-cronjob.yaml (if CronJobs are new to you, see Kubernetes Jobs and CronJobs):

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-snapshot
  namespace: vault
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vault-snapshots
  namespace: vault
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 10Gi
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: vault-snapshot
  namespace: vault
spec:
  schedule: "0 */6 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          serviceAccountName: vault-snapshot
          restartPolicy: OnFailure
          containers:
            - name: snapshot
              image: hashicorp/vault:2.0.4
              env:
                - name: VAULT_ADDR
                  value: https://vault-active.vault.svc:8200
                - name: VAULT_CACERT
                  value: /vault/tls/ca.crt
              command:
                - /bin/sh
                - -ec
                - |
                  export VAULT_TOKEN=$(vault write -field=token auth/kubernetes/login \
                    role=raft-snapshot jwt=@/var/run/secrets/vault/token)
                  file=/snapshots/vault-$(date -u +%Y%m%dT%H%M%SZ).snap
                  vault operator raft snapshot save "$file"
                  ls -l "$file"
                  # keep the 28 newest snapshots (7 days at 4 a day)
                  ls -1t /snapshots/vault-*.snap | tail -n +29 | xargs -r rm -f
              volumeMounts:
                - name: snapshots
                  mountPath: /snapshots
                - name: vault-ca
                  mountPath: /vault/tls
                  readOnly: true
                - name: vault-token
                  mountPath: /var/run/secrets/vault
                  readOnly: true
          volumes:
            - name: snapshots
              persistentVolumeClaim:
                claimName: vault-snapshots
            - name: vault-ca
              secret:
                secretName: vault-tls
                items:
                  - key: ca.crt
                    path: ca.crt
            - name: vault-token
              projected:
                sources:
                  - serviceAccountToken:
                      audience: vault
                      expirationSeconds: 600
                      path: token

The projected serviceAccountToken volume gives the Job a short-lived token with the vault audience that the role expects. Run it once by hand to test it:

kubectl apply -f snapshot-cronjob.yaml
kubectl -n vault create job --from=cronjob/vault-snapshot snapshot-test
kubectl -n vault wait --for=condition=complete job/snapshot-test --timeout=120s
kubectl -n vault logs job/snapshot-test
-rw-------    1 vault    vault        31714 Nov 10 08:39 /snapshots/vault-20261110T083920Z.snap

A snapshot on a PVC in the same cluster is not a backup. Add a step that copies the file off the cluster (an S3 bucket with object lock, another region) and alert when the newest copy is older than the schedule.

Restore a snapshot

Copy the snapshot into a Vault pod and restore it with an admin token:

kubectl -n vault cp ./vault-20261110T083920Z.snap vault-0:/tmp/restore.snap
kubectl -n vault exec -it vault-0 -- vault operator raft snapshot restore /tmp/restore.snap

The restore replaces the data on every node while the cluster keeps running. In the test cluster, a secret written after the snapshot was taken was gone after the restore, and all three peers stayed in the cluster. Use snapshot restore -force only when restoring into a different cluster.

With auto-unseal, a snapshot can only be restored into a Vault that has access to the same seal key. Restore into a scratch cluster regularly: a backup you have never restored is a hope, not a backup.

Upgrades and restarts

The chart sets the StatefulSet’s update strategy to OnDelete, so helm upgrade changes the StatefulSet but restarts no pods. Roll the change out yourself, standbys first. Check which pod is active (HA Mode: active in vault status); it is not always vault-0. With vault-0 active:

helm upgrade vault hashicorp/vault --version 0.34.1 -n vault -f values.yaml

kubectl -n vault delete pod vault-2
kubectl -n vault wait --for=condition=Ready pod/vault-2 --timeout=180s
kubectl -n vault delete pod vault-1
kubectl -n vault wait --for=condition=Ready pod/vault-1 --timeout=180s

kubectl -n vault exec vault-0 -- vault operator step-down
kubectl -n vault delete pod vault-0

step-down hands leadership to an already-restarted node before the last pod goes:

Success! Stepped down: https://127.0.0.1:8200

Take a snapshot before upgrading, and read the Vault upgrade guides before a major version jump.

Monitoring

At minimum, alert on:

  • A sealed or unhealthy node. GET /v1/sys/health returns 200 on the active node, 429 on a standby, 503 when sealed and 501 when not initialized. The readiness probe uses it, so a sealed pod also shows as not ready.

  • Raft health. vault operator raft autopilot state shows whether the cluster can lose a node:

    Healthy:                         true
    Failure Tolerance:               1
    Leader:                          vault-0
    
  • Free space on the audit volume, because a failing audit device blocks requests.

  • Snapshot age in off-cluster storage.

Vault serves Prometheus metrics at /v1/sys/metrics?format=prometheus once you add a telemetry block to the server config; scrape them with Prometheus on Kubernetes.

Troubleshooting

  • Pods Pending: fewer schedulable nodes than replicas, because of the required anti-affinity. Add nodes.
  • Pods stay 0/1 after init: they have not unsealed. kubectl -n vault logs vault-1 shows KMS errors (missing IAM permission, wrong key ID, no workload identity), and vault status shows Sealed true.
  • retry_join fails with a TLS error: the certificate lacks the *.vault-internal names, or leader_ca_cert_file points at the wrong file.
  • permission denied on Kubernetes login: the service account name or namespace does not match the role, or the token’s audience does not match the role’s audience (a default pod token versus a projected vault token).
  • ESO SecretStore not Valid: kubectl -n myapp describe secretstore vault shows the error; usually the CA ConfigMap is missing or the role does not match.
  • A peer lost its PVC: remove it with vault operator raft remove-peer vault-2, delete the pod and its PVC, and let retry_join add it back as a fresh node.

A note on licensing and OpenBao

Since version 1.15 (2023), Vault is released under the Business Source License (BSL 1.1) rather than an open-source license. Running it for your own infrastructure, as in this guide, is allowed; offering it as a competing hosted product is not. HashiCorp is now part of IBM.

If you need an open-source license, OpenBao is the Linux Foundation fork of the last MPL-licensed Vault. It has its own Helm chart (openbao/openbao, shipping OpenBao 2.x), the same Raft storage and seal options and a largely compatible API, so the approach in this guide carries over, and ESO’s vault provider works against it. It does not have every feature added to Vault since the fork, so check the ones you rely on before switching.

comments powered by Disqus
Citizix Ltd
Built with Hugo
Theme Stack designed by Jimmy