💫 Secrets in Kubernetes

Introduction

Kubernetes makes it very easy to create a Secret, but that does not automatically mean your secret management is good.

A Secret object is still just data stored in the cluster API. By default, it is base64-encoded, not encrypted.

The real protection comes from:

• RBAC
• etcd encryption at rest
• namespace isolation
• avoiding leaks in Git, CI logs, or Helm output

So the question is not only:

How do I create a secret?

but rather:

Where should the source of truth live?

In practice, I use three levels:

1. Let Helm generate a value when the application only needs an internal password.
2. Reuse the existing value on upgrades so the chart remains stable.
3. Move to an external tool when the secret must be shared, audited, rotated, or stored outside the cluster.

The naive Helm pattern

The most direct approach is to generate a random value inside the chart.

In Helm, this kind of reusable snippet is usually written as a helper inside:

templates/_helpers.tpl

For example:

1{{- define "mychart.password" -}}
2{{- randAlphaNum 32 | quote -}}
3{{- end -}}

Then another template in the same chart can call it with include:

1password: {{ include "mychart.password" . }}

mychart.password is simply the helper name.

The . passes the current chart context into the helper.

The _helpers.tpl filename is only a convention, but it is the usual one. Helm loads templates from templates/, and files starting with _ are commonly used for partials and helpers rather than rendered Kubernetes objects.

This works, but it has one major problem:

it generates a new random value every time Helm renders the template.

That means a simple helm upgrade can rotate credentials even when you did not want any rotation at all.

For a database password, an admin token, or any value reused by another workload, that is usually the wrong behavior.

This pattern is only acceptable for short-lived or disposable environments, for example:

• a demo namespace
• a preview deployment
• a local development installation that is regularly reset

For real credentials such as a database password, admin password, or API key, this is usually not the right pattern.

A better Helm pattern with lookup

Usually, what we want is:

• generate the secret on the first installation
• keep it during upgrades
• regenerate it only if the secret is deliberately deleted

Helm’s lookup function is a simple way to do that:

 1{{- $secret := lookup "v1" "Secret" .Release.Namespace "my-secret" -}}
 2{{- $password := "" -}}
 3
 4{{- if $secret -}}
 5  {{- $password = index $secret.data "password" | b64dec -}}
 6{{- else -}}
 7  {{- $password = randAlphaNum 32 -}}
 8{{- end -}}
 9
10apiVersion: v1
11kind: Secret
12metadata:
13  name: my-secret
14type: Opaque
15stringData:
16  password: {{ $password | quote }}

This gives us a much safer lifecycle:

1. First installation: generate a random 32-character password.
2. Upgrade: read the existing password from the cluster.
3. Normal upgrades do not change the credentials.
4. Delete the Secret and reinstall: a new password is generated.

For many small applications, this is enough.

It is lazy in the good sense:

• no new component
• no operator
• no additional dependency

There are still a few caveats:

• lookup needs access to the Kubernetes API during rendering.
• helm template and offline rendering do not behave in exactly the same way as helm install or helm upgrade.
• The Helm client or controller must have permission to read the existing Secret.
• The source of truth remains inside the cluster, which is fine for application-local credentials but less suitable for shared or audited secrets.

When the in-cluster pattern is enough

The lookup pattern is a good fit when:

• the secret belongs to one application
• the chart owns its full lifecycle
• you do not need manual review in Git
• rotation is rare and can be handled deliberately

Examples:

• an internal database password
• a first-start admin password stored only inside the cluster
• a Redis password used by one release

Once the same value must be reused across teams, environments, or delivery pipelines, I would stop generating it inside Helm and move the source of truth somewhere else.

Encrypting secrets in Git with sops

If you want Git to remain the source of truth, sops is one of the cleanest options.

Instead of committing plaintext secrets, you commit an encrypted file and decrypt it only for authorized users or automation.

sops works well with:

• age
• GPG
• cloud KMS backends

A common small-scale setup is sops + age:

simple keys, simple files, and no additional service.

A typical encrypted file still looks like normal YAML:

 1apiVersion: v1
 2kind: Secret
 3metadata:
 4  name: my-secret
 5type: Opaque
 6stringData:
 7  password: ENC[AES256_GCM,data:...,type:str]
 8sops:
 9  age:
10    - recipient: age1...
11  lastmodified: "2026-01-15T00:00:00Z"
12  version: 3.9.0

Why this approach is nice:

• the manifest stays in Git
• reviews still happen in Git
• access to the raw secret is controlled by keys, not only by repository access
• it works well in GitOps workflows

The trade-off is simple:

you now need key management.

That is still much better than storing plaintext credentials in the repository.

A minimal sops workflow

Start by creating an age key pair on the machine that is allowed to decrypt secrets:

1mkdir -p ~/.config/sops/age
2age-keygen -o ~/.config/sops/age/keys.txt
3age-keygen -y ~/.config/sops/age/keys.txt

The first command creates the private key file.

The second prints the matching public key, which starts with age1....

Then add a minimal .sops.yaml at the root of the repository:

1creation_rules:
2  - path_regex: .*secret(\.sops)?\.ya?ml$
3    encrypted_regex: ^(data|stringData)$
4    key_groups:
5      - age:
6          - age1replace-with-your-public-key

This means:

• any file named like db-secret.yaml, db-secret.sops.yaml, or secret.yml matches the rule
• only the data and stringData fields are encrypted
• the matching private key can decrypt the file later

Now create a normal Kubernetes manifest:

1apiVersion: v1
2kind: Secret
3metadata:
4  name: db-secret
5type: Opaque
6stringData:
7  password: supersecret

Save it, for example, as:

1db-secret.sops.yaml

Then encrypt it in place:

1sops -e -i db-secret.sops.yaml

From there, the day-to-day workflow is simple:

1# View decrypted content locally
2sops -d db-secret.sops.yaml
3
4# Edit the secret safely
5sops db-secret.sops.yaml
6
7# Apply it to the cluster
8sops -d db-secret.sops.yaml | kubectl apply -f -

That last command is the key idea:

Git stores only the encrypted file, while Kubernetes still receives a normal Secret manifest.

If you deploy with Helm rather than plain kubectl, the same pattern works with an encrypted values file:

1# values.secret.sops.yaml
2db:
3  password: supersecret

1sops -e -i values.secret.sops.yaml
2
3helm upgrade --install myapp ./chart \
4  -f values.yaml \
5  -f <(sops -d values.secret.sops.yaml)

That process-substitution example works in Bash or Zsh.

In CI, you can achieve the same thing with a temporary file if needed.

How to manage the private keys

The age1... values in .sops.yaml are public keys, so committing them is fine.

The sensitive part is the matching private key, usually stored in:

1~/.config/sops/age/keys.txt

For a small team, a practical setup is:

• one private key per person
• one separate private key for CI
• one recovery key kept offline or in a tightly controlled vault

Then .sops.yaml includes all the matching public keys as recipients.

Some simple rules are:

• keep your working private key locally on your machine
• back it up in a password manager or secure vault
• never commit it to Git
• do not share one human key between multiple people

This model is also manageable for a small team.

For example, this is a healthy pattern:

• Alice has her own age key pair
• Bob has his own age key pair
• CI has its own age key pair stored in CI secrets
• a break-glass recovery key is stored in a secure vault

If someone leaves the team, rotate the recipients by:

1. adding the new public key to .sops.yaml
2. re-encrypting the files
3. removing the old public key

This is another reason age is nice:

the model is boring and easy to reason about.

About the larger .sops.yaml examples you often see

You will often find a more advanced configuration like this:

 1---
 2# This example uses YAML anchors, which allow us to reuse multiple keys
 3# without repeating ourselves.
 4
 5keys:
 6  age:
 7    - &ci_age_key age1wdapsm2nw9pr0nmak892gqwat44uhay7d7z5fqtwgsmm6ecatggsfql3yx
 8    - &toto_age_key age19zycawkart4t4l7a238w0n2w0rmw6anjadwv9yr3ce3js4dz3dcqqgkvsw
 9    - &tata_age_key age16krrdc5j8t9pf5qyldhgq59ayjktrkzs6kq730dpu4lpy6gkt9qqd8keur
10
11creation_rules:
12  - path_regex: .+secret(\.sops)?\.ya?ml
13    input_type: yaml
14    encrypted_regex: ^(data|stringData)$
15    key_groups:
16      - age: &key_groups
17          - *ci_age_key
18          - *toto_age_key
19          - *tata_age_key
20
21  - path_regex: .+secret(\.sops)?\.env
22    input_type: env
23    key_groups:
24      - age: *key_groups
25
26stores:
27  yaml:
28    indent: 2

Here:

• keys: defines reusable recipient lists
• &name and *name are YAML anchors and aliases, simply a DRY way to reuse the same values
• creation_rules: defines which files match which policy
• input_type: yaml or env tells sops how to parse the file
• encrypted_regex: ^(data|stringData)$ means only those Kubernetes fields are encrypted
• stores.yaml.indent: 2 controls output formatting

So the longer version is not a different mechanism.

It is simply the same policy written in a more reusable way for teams with multiple keys and file types.

ksops: sops for Kustomize workflows

If your deployment workflow is based on Kustomize, ksops is the natural extension.

It lets Kustomize consume sops-encrypted files and render regular Kubernetes manifests from them.

A minimal example looks like this:

1# kustomization.yaml
2apiVersion: kustomize.config.k8s.io/v1beta1
3kind: Kustomization
4resources:
5  - deployment.yaml
6generators:
7  - ksops-secret-generator.yaml

1# ksops-secret-generator.yaml
2apiVersion: viaduct.ai/v1
3kind: ksops
4metadata:
5  name: my-secret
6files:
7  - secret.enc.yaml

This is useful when:

• you already use Kustomize overlays
• you want encrypted manifests in Git
• you do not want to switch the whole deployment workflow to Helm just for secrets

A minimal ksops workflow

First, keep using the same sops setup as above.

The encrypted file can remain a normal Kubernetes Secret, for example:

1apiVersion: v1
2kind: Secret
3metadata:
4  name: my-secret
5type: Opaque
6stringData:
7  password: supersecret

Encrypt it:

1sops -e -i secret.sops.yaml

Then wire it into Kustomize:

1# kustomization.yaml
2apiVersion: kustomize.config.k8s.io/v1beta1
3kind: Kustomization
4resources:
5  - deployment.yaml
6generators:
7  - ksops-secret-generator.yaml

1# ksops-secret-generator.yaml
2apiVersion: viaduct.ai/v1
3kind: ksops
4metadata:
5  name: my-secret
6files:
7  - secret.sops.yaml

Build and apply it with:

1kustomize build --enable-alpha-plugins --enable-exec . | kubectl apply -f -

The important caveat is that ksops is a Kustomize plugin.

So you usually render with kustomize build rather than plain:

1kubectl apply -k

In a GitOps setup, the same idea applies, but the controller needs a plugin or sidecar capable of running ksops during render time.

In short:

sops solves encrypted files, while ksops plugs that idea into a Kustomize-based deployment process.

Using Vault when the source of truth is external

At this point, sops and ksops still keep Git as the source of truth.

That works well when secrets are relatively static and belong to the deployment process.

But some secrets should not live in Git at all.

This is especially true when:

• credentials must be rotated frequently
• the same secret is consumed by several workloads
• access needs to be audited centrally
• credentials should be generated dynamically
• the secret lifecycle should be independent from the application repository

This is where a dedicated secret manager such as Vault becomes more interesting.

With Vault, Kubernetes workloads can fetch secrets at runtime, and in some cases they can fetch dynamic secrets with a lease and automatic expiration.

This is much better suited to environments where you do not want one long-lived password copied into every deployment.

Common patterns include:

• Vault Agent Injector writes secrets into the pod
• the CSI driver mounts secrets as files
• an operator syncs selected values into Kubernetes Secret objects
• Bank-Vaults injects values into application environment variables

The difference is mainly where the final secret appears:

1Vault
2  │
3  ├── Vault Agent Injector ───► file inside the Pod
4  │
5  ├── CSI Driver ─────────────► mounted volume
6  │
7  ├── Vault operator ─────────► Kubernetes Secret
8  │
9  └── Bank-Vaults webhook ────► environment variable

So choosing Vault does not automatically define how applications consume secrets.

That is a separate architectural decision.

Vault is worth the extra moving parts when you need:

• centralized secret management across many clusters
• short-lived credentials
• strong audit requirements
• database or cloud credentials generated on demand

A minimal Vault workflow

In practice, the platform team configures Vault and Kubernetes authentication once, then applications consume it.

From an application point of view, the workflow is roughly:

1. Store the secret in Vault.
2. Bind a Vault role to the Kubernetes service account.
3. Annotate the pod so Vault injects the secret at runtime.

For example, write a secret to Vault:

1vault kv put kv/myapp password="supersecret"

If you want Vault to generate the password for you instead of typing one yourself, use Vault’s random endpoint and store the result:

1vault kv put kv/myapp \
2  password="$(vault write -field=random_bytes sys/tools/random/24 format=base64)"

That is a nice middle ground:

• Vault generates the random value
• Vault stores it in KV
• the pod still consumes it through Vault Agent injection

If the backend supports dynamic credentials, go one step further and let Vault generate them on demand through a secrets engine instead of storing one long-lived password at all.

Then create a policy and role.

Assuming Kubernetes authentication is already configured in Vault, the role can be bound to one service account:

1# myapp-policy.hcl
2path "kv/data/myapp" {
3  capabilities = ["read"]
4}

1vault policy write myapp myapp-policy.hcl
2
3vault write auth/kubernetes/role/myapp \
4  bound_service_account_names=myapp \
5  bound_service_account_namespaces=default \
6  policies=myapp \
7  ttl=24h

Then annotate the workload:

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  name: myapp
 5spec:
 6  replicas: 1
 7  selector:
 8    matchLabels:
 9      app: myapp
10  template:
11    metadata:
12      labels:
13        app: myapp
14      annotations:
15        vault.hashicorp.com/agent-inject: "true"
16        vault.hashicorp.com/role: "myapp"
17        vault.hashicorp.com/agent-inject-secret-config.txt: "kv/data/myapp"
18        vault.hashicorp.com/agent-inject-template-config.txt: |
19          {{`{{- with secret "kv/data/myapp" -}}`}}
20          export PASSWORD="{{`{{ .Data.data.password }}`}}"
21          {{`{{- end -}}`}}
22    spec:
23      serviceAccountName: myapp
24      containers:
25        - name: myapp
26          image: nginx:stable

When the pod starts, Vault Agent injects a file containing the rendered secret.

The application reads that file instead of reading from a Kubernetes Secret object.

This is the big difference from sops:

with Vault, the secret can stay outside Git and even outside Kubernetes until the pod starts.

For one small application, Vault is probably overkill.

For a platform shared by many teams, however, it can become a very clean model.

Once Vault is the source of truth, the next question is:

how should the secret reach the application?

The Vault Agent Injector is one answer, but it is not the only one.

Bank-Vaults Mutating Webhook

Another approach is the Bank-Vaults mutating webhook.

At first sight, it looks quite similar to the HashiCorp Vault Agent Injector because both rely on a Kubernetes Mutating Admission Webhook.

The difference is mostly in how the secret reaches the application.

With the HashiCorp Vault Agent Injector, the webhook adds Vault Agent containers to the pod.

The agent authenticates to Vault, retrieves the secret, and usually renders it into a file shared with the application container.

The flow looks roughly like this:

 1Pod
 2│
 3├── Vault Agent
 4│      │
 5│      └──► Vault
 6│             │
 7│             ▼
 8│      /vault/secrets/config
 9│
10└── Application
11       │
12       └── reads the file

Bank-Vaults takes another approach.

Its webhook can detect environment variables containing a vault: reference and inject vault-env into the container startup process.

For example:

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  name: myapp
 5spec:
 6  replicas: 1
 7  selector:
 8    matchLabels:
 9      app: myapp
10  template:
11    metadata:
12      labels:
13        app: myapp
14      annotations:
15        vault.security.banzaicloud.io/vault-addr: "https://vault:8200"
16        vault.security.banzaicloud.io/vault-role: "myapp"
17        vault.security.banzaicloud.io/vault-path: "kubernetes"
18    spec:
19      serviceAccountName: myapp
20      containers:
21        - name: myapp
22          image: nginx:stable
23          env:
24            - name: PASSWORD
25              value: vault:kv/data/myapp#password

At startup, vault-env authenticates to Vault and resolves:

1vault:kv/data/myapp#password

It then starts the application with the resolved value exposed as:

1PASSWORD=supersecret

The architecture is therefore closer to:

 1Pod
 2│
 3└── vault-env
 4       │
 5       ├──► Vault
 6       │      │
 7       │      ▼
 8       │    secret
 9       │
10       ▼
11   Environment
12       │
13       ▼
14   Application

This can be very convenient for applications that already expect their configuration through environment variables.

Inject a complete Vault path

Bank-Vaults can also expose all the keys from a Vault path as environment variables using the vault-env-from-path annotation.

For example:

1metadata:
2  annotations:
3    vault.security.banzaicloud.io/vault-role: "myapp"
4    vault.security.banzaicloud.io/vault-env-from-path: "kv/data/myapp"

If the Vault secret contains:

1{
2  "username": "admin",
3  "password": "supersecret"
4}

the application can receive those values as environment variables instead of declaring each key individually.

Vault Agent Injector vs Bank-Vaults

The two solutions therefore solve a similar problem, but use different runtime models.

The Vault Agent model is particularly interesting when the application can read secrets from files and you want Vault Agent to keep handling secret rendering and lease management while the pod is running.

Bank-Vaults is attractive when the application already expects configuration through environment variables and you want to avoid a permanent sidecar.

There is one important consequence, however.

An environment variable belongs to the process environment.

If the value changes inside Vault, the already-running application does not automatically receive a new environment variable.

With Vault Agent templates, a secret file can be rendered again when Vault Agent detects a new value or renews a leased secret, although the application still needs a way to reload that file if required.

Same Vault policy model

The interesting part is that the Vault-side configuration does not fundamentally change.

We can still use the same policy:

1path "kv/data/myapp" {
2  capabilities = ["read"]
3}

and bind it to the Kubernetes service account:

1vault policy write myapp myapp-policy.hcl
2
3vault write auth/kubernetes/role/myapp \
4  bound_service_account_names=myapp \
5  bound_service_account_namespaces=default \
6  policies=myapp \
7  ttl=24h

What changes is mostly the last mile between Vault and the application.

With Vault Agent:

 1Vault
 2  │
 3  ▼
 4Vault Agent
 5  │
 6  ▼
 7File
 8  │
 9  ▼
10Application

With Bank-Vaults:

 1Vault
 2  │
 3  ▼
 4vault-env
 5  │
 6  ▼
 7Environment variable
 8  │
 9  ▼
10Application

So the choice is not really about whether to use Vault or not.

It is about how the workload consumes the secret.

If the application already supports file-based configuration, the Vault Agent Injector is usually a very natural fit.

If the application is strongly environment-variable oriented, Bank-Vaults can provide a lighter and very convenient integration.

Both approaches also avoid storing the final secret value directly in a normal Kubernetes Secret object.

Which option should you pick?

My practical rule is simple:

Conclusion

Kubernetes secrets are easy to create, but the important design choice is where the source of truth lives.

If the value is local to one release, the lookup pattern is simple and effective.

If the value must live in Git, encrypt it with sops.

If your world is Kustomize, use ksops.

If secrets need to be shared, rotated, audited, or generated dynamically, move them to Vault and let Kubernetes consume them instead of owning them.

The simple answer is often the best one:

keep secrets as close as possible to the workload, but move the source of truth outward only when sharing, rotation, auditing, or lifecycle requirements justify it.

Sources

• SOPS
• KSOPS
• HashiCorp Vault Agent Injector annotations
• HashiCorp Vault Agent Injector examples
• Bank-Vaults mutating webhook annotations
• Bank-Vaults webhook configuration examples