Dev SOPS and age

info

This runbook explains how SOPS and age are used to encrypt Kubernetes Secrets in both the Blaster app repo and the Flux infra repo, so secrets stay in Git but remain unreadable at rest. It covers key creation, local tooling, .sops.yaml policies, Flux decryption and common checks.

Blaster GitOps series

1. High-level model

There are three main pieces:

age keypair on your workstation
- Private key on disk, never committed.
- Public key in .sops.yaml in each repo and in the sops-age Secret in the cluster.
SOPS policies (.sops.yaml) per repo
- App repo: encrypts Secrets under k8s/**.enc.yaml.
- Infra repo: encrypts Secrets under clusters/my-cluster/**/secrets/*.yaml.
Flux SOPS decryption
- Flux uses the sops-age Secret to decrypt files at apply time.
- Enabled on the main flux-system Kustomization and on the Blaster app Kustomization.

Once this is in place:

Secrets can be edited locally with sops and committed safely.
Flux applies decrypted manifests to the cluster at runtime.
Raw passwords and tokens never appear in Git in plain text.

Related runbooks:

2. age keypair on your Mac

2.1 Generate an age keypair

On your workstation (once only):

mkdir -p "$HOME/.sops"
cd "$HOME/.sops"

age-keygen -o age.key

Show the key:

cat "$HOME/.sops/age.key"

Example (do not reuse):

# created: 2025-11-16T12:34:56+08:00
# public key: age1examplepublickeystringgoeshere
AGE-SECRET-KEY-1exampleverylongsecretkeymaterial

The line starting with AGE-SECRET-KEY- is the private key.
The comment line with public key: contains the public key used in .sops.yaml and in the sops-age Secret.

2.2 Shell environment

Tell SOPS where to find the age key:

# ~/.zshrc
export SOPS_AGE_KEY_FILE="$HOME/.sops/age.key"

Reload your shell:

source ~/.zshrc

Check SOPS can see the key:

sops --version
echo "$SOPS_AGE_KEY_FILE"

3. App repo SOPS policy (`games/blaster`)

The Blaster app repo uses SOPS for in-cluster app secrets under k8s/dev and k8s/prod.

3.1 `.sops.yaml` in the app repo

File: games/blaster/.sops.yaml

# .sops.yaml
creation_rules:
  - path_regex: k8s/.*\.enc\.ya?ml$
    encrypted_regex: '^(data|stringData)$'
    age: ['AGE_PUBLIC_KEY_HERE']

Replace AGE_PUBLIC_KEY_HERE with the public key from age.key, for example:

age1examplepublickeystringgoeshere

Meaning:

Any file under k8s/ whose name ends in .enc.yaml or .enc.yml is subject to SOPS.
Only the data and stringData sections are encrypted.
Everything else (metadata, apiVersion, kind) stays in clear text so kubectl and Flux can parse it.

3.2 App secrets pattern

Example Secrets before encryption:

---
# k8s/dev/10-secret-db.enc.yaml
apiVersion: v1
kind: Secret
metadata:
  name: blaster-db-secret
  namespace: blaster-dev
type: Opaque
stringData:
  POSTGRES_DB: blaster_game
  POSTGRES_USER: blaster_user
  POSTGRES_PASSWORD: "REPLACE_WITH_STRONG_PASSWORD"
---
# k8s/dev/30-secret-app.enc.yaml
apiVersion: v1
kind: Secret
metadata:
  name: blaster-app-secret
  namespace: blaster-dev
type: Opaque
stringData:
  CLERK_SECRET_KEY: "REPLACE_WITH_REAL_CLERK_SECRET_KEY"

Encrypt them:

sops -e -i k8s/dev/10-secret-db.enc.yaml
sops -e -i k8s/dev/30-secret-app.enc.yaml

After encryption you should see:

stringData replaced with data containing ENC[...] blobs.
A sops: block at the bottom with metadata.

Verification:

grep -n 'sops:' k8s/dev/*.enc.yaml
head -n 20 k8s/dev/10-secret-db.enc.yaml

3.3 Editing app secrets

To make changes later:

sops k8s/dev/10-secret-db.enc.yaml
sops k8s/dev/30-secret-app.enc.yaml

SOPS flow:

Decrypts into a temp buffer.
Opens in your $EDITOR.
Re-encrypts when you save and exit.

Never edit encrypted files directly with a plain text editor.

4. Infra repo SOPS policy (`fluxgitops/flux-config`)

The infra repo uses SOPS for cluster-level secrets, typically under clusters/my-cluster/**/secrets/.

4.1 `.sops.yaml` in the infra repo

File: flux-config/.sops.yaml

# .sops.yaml
creation_rules:
  - path_regex: 'clusters/my-cluster/.*/secrets/.*\.yaml$'
    encrypted_regex: '^(data|stringData)$'
    age: ['AGE_PUBLIC_KEY_HERE']

Same public key as the app repo.

This covers, for example:

clusters/my-cluster/flux-system/secrets/blaster-dev-registry.yaml
Any future secrets you store under clusters/my-cluster/.../secrets/.

4.2 Example: registry Secret manifest

To create a Secret for Flux to read registry credentials:

cd ~/Projects/flux-config

kubectl -n flux-system create secret docker-registry blaster-dev-registry   --docker-server=registry.reids.net.au   --docker-username='blaster-dev'   --docker-password='REPLACE-ME'   --docker-email='andrew@reids.net.au'   --dry-run=client -o yaml   > clusters/my-cluster/flux-system/secrets/blaster-dev-registry.yaml

Encrypt:

sops -e -i clusters/my-cluster/flux-system/secrets/blaster-dev-registry.yaml

Check:

head -n 20 clusters/my-cluster/flux-system/secrets/blaster-dev-registry.yaml
grep -n 'sops:' clusters/my-cluster/flux-system/secrets/blaster-dev-registry.yaml

The file should now include an encrypted .dockerconfigjson and a sops block.

5. Flux SOPS decryption

Flux needs to know:

Which provider to use (sops).
Which Secret contains the age private key (sops-age).

5.1 `sops-age` Secret in the cluster

Create a Kubernetes Secret from age.key:

kubectl -n flux-system create secret generic sops-age   --from-file=age.key="$HOME/.sops/age.key"

You should see:

kubectl -n flux-system get secret sops-age

Output (example):

NAME       TYPE     DATA   AGE
sops-age   Opaque   1      35d

Flux will use this Secret to decrypt any SOPS-managed files in the repos it syncs.

5.2 Flux-system Kustomization decryption

The Flux bootstrap generates clusters/my-cluster/flux-system/kustomization.yaml and a Kustomization resource named flux-system in the flux-system namespace.

You do not edit gotk-sync.yaml or the in-cluster YAML directly. Instead, patch the Kustomization in place:

kubectl -n flux-system patch kustomization flux-system   --type='merge'   -p '{
    "spec": {
      "decryption": {
        "provider": "sops",
        "secretRef": {
          "name": "sops-age"
        }
      }
    }
  }'

Verify:

kubectl -n flux-system get kustomization flux-system -o yaml | sed -n '1,80p'

Look for:

spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-age
  path: ./clusters/my-cluster
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

This enables SOPS for all encrypted secrets under clusters/my-cluster/** that are part of the root tree.

5.3 Blaster app Kustomization decryption

The Blaster dev Kustomization also needs decryption enabled so it can read k8s/dev/*.enc.yaml from the app repo.

File: clusters/my-cluster/blaster/dev/kustomization.yaml

# clusters/my-cluster/blaster/dev/kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: blaster-dev
  namespace: flux-system
spec:
  interval: 1m
  path: ./k8s/dev
  prune: true
  sourceRef:
    kind: GitRepository
    name: blaster-dev
  wait: true
  timeout: 5m
  decryption:
    provider: sops
    secretRef:
      name: sops-age

Now the dev app Kustomization will:

Pull the games/blaster repo using the blaster-dev GitRepository.
Decrypt Secrets in k8s/dev/*.enc.yaml using the same sops-age Secret.
Apply them to the blaster-dev namespace.

6. Typical workflows

6.1 Add a new secret to the app repo

Create a new *.enc.yaml under k8s/dev or k8s/prod:

cat > k8s/dev/70-some-secret.enc.yaml << 'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: blaster-some-secret
  namespace: blaster-dev
type: Opaque
stringData:
  SOME_KEY: "REPLACE_ME"
EOF

Encrypt with SOPS:

sops -e -i k8s/dev/70-some-secret.enc.yaml

Commit and push:

git add k8s/dev/70-some-secret.enc.yaml
git commit -m "Add some-secret for blaster dev"
git push

Let Flux pick up the change, or force reconcile:

flux reconcile kustomization blaster-dev -n flux-system --with-source

6.2 Rotate a secret value

Edit via SOPS:
```
sops k8s/dev/10-secret-db.enc.yaml
```
Change the password value, save and exit.
Commit and push.
Flux will apply the updated secret; the app will roll on next restart or according to your Deployment rollout.

6.3 Debugging decryption issues

Commands:

# Flux-system events
kubectl -n flux-system get kustomizations
kubectl -n flux-system describe kustomization flux-system

# Blaster-dev Kustomization
kubectl -n flux-system describe kustomization blaster-dev

# Check that sops-age exists
kubectl -n flux-system get secret sops-age

Common problems:

sops-age Secret missing or in the wrong namespace.
Public key mismatch between .sops.yaml and the key inside sops-age.
SOPS formatting broken (for example editing encrypted YAML without SOPS).

If decryption fails, Flux will log an error in the Kustomization status and set READY=False.

7. Security notes

The age private key stays in:
- Your local age.key file.
- The sops-age Secret in the cluster.
Treat both as sensitive secrets. Rotate if you suspect compromise.
The age public key is safe to commit in .sops.yaml.
Avoid copying decrypted secrets into terminals or logs; edit directly with sops.
Use different age keypairs for different environments if you want stronger separation. If you do this, update:
- .sops.yaml in each repo.
- The sops-age Secret in any clusters that need to decrypt those secrets.

8. Verification checklist

Once all of these are true, SOPS and age are correctly configured for both the Blaster app and the Flux infra configuration.

Blaster GitOps series​

1. High-level model​

2. age keypair on your Mac​

2.1 Generate an age keypair​

2.2 Shell environment​

3. App repo SOPS policy (games/blaster)​

3.1 .sops.yaml in the app repo​

3.2 App secrets pattern​

3.3 Editing app secrets​

4. Infra repo SOPS policy (fluxgitops/flux-config)​

4.1 .sops.yaml in the infra repo​

4.2 Example: registry Secret manifest​

5. Flux SOPS decryption​

5.1 sops-age Secret in the cluster​

5.2 Flux-system Kustomization decryption​

5.3 Blaster app Kustomization decryption​

6. Typical workflows​

6.1 Add a new secret to the app repo​

6.2 Rotate a secret value​

6.3 Debugging decryption issues​

7. Security notes​

8. Verification checklist​