Skip to main content

Dev Kubernetes manifests

info

This runbook explains what lives under k8s/dev and k8s/prod in the app repo and how the manifests work together: Secrets, ConfigMaps, database, app Deployment, Service, Ingress and SOPS encryption.

Blaster GitOps series​

  1. Blaster GitOps summary
  2. Blaster repo and branches
  3. Dockerfile & GitLab CI
  4. Clerk authentication & user setup
  5. Google OAuth for Clerk
  6. Blaster prep for automation
  7. Dev app k8s manifests - you are here
  8. Dev flux sources & Kustomizations
  9. Dev image automation
  10. Dev SOPS & age
  11. Dev verification & troubleshooting
  12. Dev full runbook
  13. Prod overview
  14. Prod app k8s manifests and deployment
  15. Prod Flux GitOps and image automation
  16. Prod Cloudflare, Origin CA and tunnel routing
  17. Prod full runbook
  18. Post development branches

1. Context​

Dev repo layout in games/blaster:

games/blaster/
├── .dockerignore
├── .gitlab-ci.yml
├── .sops.yaml
├── app
├── database
├── db
├── Dockerfile
├── k8s
│   └── dev
│       ├── 10-secret-db.enc.yaml
│       ├── 20-db-statefulset.yaml
│       ├── 30-secret-app.enc.yaml
│       ├── 40-app-config.yaml
│       ├── 50-app-deployment.yaml
│       ├── 60-ingress.yaml
│       └── kustomization.yaml
├── lib
└── package.json

Environments:

  • Dev
    • Namespace: blaster-dev
    • Host: blaster.reids.net.au
    • Branch: develop
  • Prod
    • Namespace: blaster
    • Host: blaster.muppit.au
    • Branch: main

Flux Kustomizations in the infra repo point at these directories. For repo and branches see Blaster repo and branches.

2. SOPS policy in the app repo​

All Secrets under k8s/ are committed encrypted using SOPS with age.

File: games/blaster/.sops.yaml

# .sops.yaml
creation_rules:
  - path_regex: k8s/.*\.enc\.ya?ml$
    encrypted_regex: '^(data|stringData)$'
    age: ['AGE_PUBLIC_KEY_HERE']
  • Any file matching k8s/**.enc.yaml or k8s/**.enc.yml will have data or stringData encrypted.
  • Replace AGE_PUBLIC_KEY_HERE with your age public key (starts with age1).

Secrets are edited with:

sops -e -i k8s/dev/10-secret-db.enc.yaml
sops -e -i k8s/dev/30-secret-app.enc.yaml
warning

Do not edit encrypted files without sops. You will end up with broken YAML or plain text secrets in Git.

3. Dev manifests overview (k8s/dev)​

3.1 Kustomization​

File: k8s/dev/kustomization.yaml

# k8s/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: blaster-dev
resources:
  - 10-secret-db.enc.yaml
  - 20-db-statefulset.yaml
  - 30-secret-app.enc.yaml
  - 40-app-config.yaml
  - 50-app-deployment.yaml
  - 60-ingress.yaml

This:

  • Sets the default namespace to blaster-dev.
  • Applies Secrets, ConfigMap, StatefulSet, Deployment, Service and Ingress as a unit.

3.2 Database Secret​

File: k8s/dev/10-secret-db.enc.yaml

---
# k8s/dev/10-secret-db.enc.yaml
apiVersion: v1
kind: Secret
metadata:
  name: blaster-db-secret
  namespace: blaster-dev
type: Opaque
stringData:
  POSTGRES_DB: blaster_game
  POSTGRES_USER: blaster_user
  POSTGRES_PASSWORD: "REPLACE_WITH_STRONG_PASSWORD"

Use sops to encrypt this file. The same Secret is consumed by both the database and the app.

3.3 PostgreSQL StatefulSet and Service​

File: k8s/dev/20-db-statefulset.yaml

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: blaster-db
  namespace: blaster-dev
spec:
  serviceName: blaster-db
  replicas: 1
  selector:
    matchLabels:
      app: blaster-db
  template:
    metadata:
      labels:
        app: blaster-db
    spec:
      containers:
        - name: postgres
          image: postgres:15.8
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5432
              name: postgres
          envFrom:
            - secretRef:
                name: blaster-db-secret
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
          readinessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
            initialDelaySeconds: 10
            periodSeconds: 10
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
            initialDelaySeconds: 30
            periodSeconds: 30
          resources:
            requests:
              cpu: "100m"
              memory: "256Mi"
            limits:
              cpu: "500m"
              memory: "512Mi"
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: blaster-db
  namespace: blaster-dev
spec:
  type: ClusterIP
  selector:
    app: blaster-db
  ports:
    - name: postgres
      port: 5432
      targetPort: 5432

Notes:

  • Uses blaster-db-secret as single source of truth.
  • Small but non-trivial resource requests and limits for dev.
  • Readiness and liveness both call pg_isready.

3.4 App Secrets and Config​

Files before encryption:

---
# k8s/dev/30-secret-app.enc.yaml
apiVersion: v1
kind: Secret
metadata:
  name: blaster-app-secret
  namespace: blaster-dev
type: Opaque
stringData:
  CLERK_SECRET_KEY: "REPLACE_WITH_REAL_CLERK_SECRET_KEY"
---
# k8s/dev/40-app-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: blaster-app-config
  namespace: blaster-dev
data:
  NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY: "REPLACE_WITH_PUBLIC_KEY"
  • 30-secret-app.enc.yaml is encrypted with SOPS.
  • 40-app-config.yaml is plain text because it only contains public configuration.

3.5 App Deployment and Service​

File: k8s/dev/50-app-deployment.yaml

---
# k8s/dev/50-app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: blaster-app
  namespace: blaster-dev
spec:
  replicas: 2
  selector:
    matchLabels:
      app: blaster-app
  template:
    metadata:
      labels:
        app: blaster-app
    spec:
      imagePullSecrets:
        - name: blaster-dev-registry
      containers:
        - name: blaster
          image: registry.reids.net.au/games/blaster:dev-20251115.42 # {"$imagepolicy": "flux-system:blaster-dev-policy"}
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http
          envFrom:
            - secretRef:
                name: blaster-db-secret      # DB name/user/password
            - secretRef:
                name: blaster-app-secret     # Clerk secret
            - configMapRef:
                name: blaster-app-config     # public config
          env:
            - name: POSTGRES_HOST
              value: "blaster-db"
            - name: POSTGRES_PORT
              value: "5432"
          readinessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 15
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 60
            periodSeconds: 20
          resources:
            requests:
              cpu: "200m"
              memory: "512Mi"
            limits:
              cpu: "1"
              memory: "1Gi"
---
apiVersion: v1
kind: Service
metadata:
  name: blaster-app
  namespace: blaster-dev
spec:
  type: ClusterIP
  selector:
    app: blaster-app
  ports:
    - name: http
      port: 80
      targetPort: 3000

Important pieces:

  • imagePullSecrets references a registry Secret created in the infra repo namespace flux-system and mirrored into blaster-dev (see Image automation runbook).
  • envFrom pulls in:
    • Database credentials.
    • Clerk secret.
    • Public config from ConfigMap.
  • Explicit probes and resource requests/limits help Flux and the scheduler see health correctly.

3.6 Ingress for dev​

File: k8s/dev/60-ingress.yaml

---
# k8s/dev/60-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blaster-ingress
  namespace: blaster-dev
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - blaster.reids.net.au
  rules:
    - host: "blaster.reids.net.au"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: blaster-app
                port:
                  number: 80
  • Uses the nginx ingress class.
  • Exposes the app as https://blaster.reids.net.au via Ingress NGINX.
  • TLS is handled by the cluster side (cert-manager wildcard certificate) rather than this page.

4. Prod manifests overview (k8s/prod)​

Prod manifests follow the same pattern as dev with these differences:

  • Namespace: blaster instead of blaster-dev.
  • Image tag: prod-YYYYMMDD.N tags referenced in the Deployment.
  • Ingress host: blaster.muppit.au.
  • Resources: can be higher (more replicas, more CPU and memory).

Skeleton layout:

k8s/prod/
  kustomization.yaml
  10-secret-db.enc.yaml
  20-db-statefulset.yaml
  30-secret-app.enc.yaml
  40-app-config.yaml
  50-app-deployment.yaml
  60-ingress.yaml

Example k8s/prod/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: blaster
resources:
  - 10-secret-db.enc.yaml
  - 20-db-statefulset.yaml
  - 30-secret-app.enc.yaml
  - 40-app-config.yaml
  - 50-app-deployment.yaml
  - 60-ingress.yaml

Ingress for prod would be the same as dev but with blaster.muppit.au as the host.

tip

Keep the file names and structure identical between dev and prod where possible. It makes diffing and promotion much easier.

5. SOPS workflow for app secrets​

5.1 Environment variables​

Set SOPS_AGE_KEY_FILE on your Mac, for example in ~/.zshrc:

export SOPS_AGE_KEY_FILE="$HOME/.sops/age.key"

5.2 Editing a secret​

To edit the dev DB Secret:

sops k8s/dev/10-secret-db.enc.yaml

SOPS will:

  • Decrypt the file to a temp buffer.
  • Open it in $EDITOR.
  • Re-encrypt on save.

You can run the same command for app secrets and for prod equivalents under k8s/prod.

6. How Flux applies these manifests​

Flux sees the manifests via a Kustomization in the infra repo:

# clusters/my-cluster/blaster/dev/kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: blaster-dev
  namespace: flux-system
spec:
  interval: 1m
  path: ./k8s/dev
  prune: true
  sourceRef:
    kind: GitRepository
    name: blaster-dev
  wait: true
  timeout: 5m
  decryption:
    provider: sops
    secretRef:
      name: sops-age

Key points:

  • path: ./k8s/dev tells Flux to use the app repo Kustomization.
  • decryption.provider: sops allows SOPS-encrypted files to be applied.
  • prune: true removes objects deleted from Git.

Prod will use a similar Kustomization pointing at ./k8s/prod and the main branch.

For full Flux configuration, see Flux sources & Kustomizations and Image automation.

7. Verification commands​

7.1 Check resources in blaster-dev​

kubectl -n blaster-dev get pods,svc,ingress
kubectl -n blaster-dev get deploy,sts
kubectl -n blaster-dev get cm,secret

You should see:

  • One StatefulSet blaster-db with one ready replica.
  • One Deployment blaster-app with the expected replicas.
  • Services blaster-app and blaster-db.
  • Ingress blaster-ingress pointing at blaster-app.

7.2 Check that images match the expected tag​

kubectl -n blaster-dev get deploy blaster-app -o jsonpath='{.spec.template.spec.containers[0].image}'
echo

For dev, expect something like:

registry.reids.net.au/games/blaster:dev-20251115.51

7.3 Check SOPS-encrypted files​

grep -n 'sops:' k8s/dev/*.enc.yaml
head -n 20 k8s/dev/10-secret-db.enc.yaml

You should see an sops: section and no raw passwords.

8. Verification checklist​

  • games/blaster/k8s/dev exists with the files listed above.
  • games/blaster/k8s/prod exists and mirrors the dev structure.
  • .sops.yaml at repo root matches the SOPS rule for *.enc.yaml.
  • All Secrets in Git are encrypted and edited via sops.
  • blaster-dev namespace has:
    • Running blaster-db StatefulSet.
    • Running blaster-app Deployment.
    • Services and Ingress as expected.
  • https://blaster.reids.net.au serves the dev game.
  • https://blaster.muppit.au serves the prod game when prod manifests are configured.

Once this checklist is true, the app side of the Blaster Kubernetes manifests is in good shape, and the Flux runbooks can take over for GitOps reconciliation and image automation.