Skip to main content

Prod K8s manifests and deployment

info

This runbook starts from the games/blaster repo and shows how the blaster namespace is built for production: DB, app, Ingress and Secrets, all wrapped by a Kustomization and protected with SOPS.

Blaster GitOps series​

  1. Blaster GitOps summary
  2. Blaster repo and branches
  3. Dockerfile & GitLab CI
  4. Clerk authentication & user setup
  5. Google OAuth for Clerk
  6. Blaster prep for automation
  7. Dev app k8s manifests
  8. Dev flux sources & Kustomizations
  9. Dev image automation
  10. Dev SOPS & age
  11. Dev verification & troubleshooting
  12. Dev full runbook
  13. Prod overview
  14. Prod app k8s manifests and deployment - you are here
  15. Prod Flux GitOps and image automation
  16. Prod Cloudflare, Origin CA and tunnel routing
  17. Prod full runbook
  18. Post development branches

1. Repo state and layout​

1.1 Ensure repo is clean and up to date​

cd ~/Projects/blaster
git pull
Already up to date.
git status
On branch develop
Your branch is up to date with 'origin/develop'.

nothing to commit, working tree clean

1.2 Inspect tree and K8s layout​

tree -a -L 6 -I '.git|.DS_Store|node_modules|.next|dist'
.
├── .dockerignore
├── .gitlab-ci.yml
├── .sops.yaml
├── app
├── database
├── db
├── Dockerfile
├── k8s
│   ├── dev
│   │   ├── 10-secret-db.enc.yaml
│   │   ├── 20-db-statefulset.yaml
│   │   ├── 30-secret-app.enc.yaml
│   │   ├── 40-app-config.yaml
│   │   ├── 50-app-deployment.yaml
│   │   ├── 60-ingress.yaml
│   │   └── kustomization.yaml
│   └── prod
│       ├── 10-secret-db.enc.yaml
│       ├── 20-db-statefulset.yaml
│       ├── 30-secret-app.enc.yaml
│       ├── 40-app-config.yaml
│       ├── 50-app-deployment.yaml
│       ├── 60-ingress.yaml
│       ├── 70-secret-cfapi-token.enc.yaml
│       ├── 80-originissuer.yaml
│       ├── 90-certificate-blaster.yaml
│       └── kustomization.yaml
├── lib
└── package.json

2. Production Kustomization​

2.1 Kustomization for k8s/prod​

# k8s/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: blaster
resources:
  - 10-secret-db.enc.yaml
  - 20-db-statefulset.yaml
  - 30-secret-app.enc.yaml
  - 40-app-config.yaml
  - 50-app-deployment.yaml
  - 60-ingress.yaml
  - 70-secret-cfapi-token.enc.yaml
  - 80-originissuer.yaml
  - 90-certificate-blaster.yaml

2.2 SOPS policy at repo root​

note

A SOPS policy already exists at repo root. It was added during the dev build and controls which keys are used to encrypt Secrets under k8s/.

2.3 Creating the k8s/prod tree​

Use this only when bootstrapping a new environment.

Create the files under k8s/prod/.

mkdir -p k8s/prod

3. Database – Secret, StatefulSet and Service​

3.1 DB Secret – single source of truth​

Before SOPS encryption

---
# k8s/prod/10-secret-db.enc.yaml
apiVersion: v1
kind: Secret
metadata:
  name: blaster-db-secret
  namespace: blaster
type: Opaque
stringData:
  POSTGRES_DB: blaster_game
  POSTGRES_USER: blaster_user
  POSTGRES_PASSWORD: "REPLACE_WITH_STRONG_PASSWORD"

This is the single source of truth for DB credentials. The DB and app both consume this Secret via envFrom.

note

Edit via sops so Secrets stay encrypted in Git.

3.2 PostgreSQL StatefulSet and Service​

---
# k8s/prod/20-db-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: blaster-db
  namespace: blaster
spec:
  serviceName: blaster-db
  replicas: 1
  selector:
    matchLabels:
      app: blaster-db
  template:
    metadata:
      labels:
        app: blaster-db
    spec:
      containers:
        - name: postgres
          image: postgres:15.8
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5432
              name: postgres
          envFrom:
            - secretRef:
                name: blaster-db-secret
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
          readinessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
            initialDelaySeconds: 10
            periodSeconds: 10
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
            initialDelaySeconds: 30
            periodSeconds: 30
          resources:
            requests:
              cpu: "200m"
              memory: "512Mi"
            limits:
              cpu: "1"
              memory: "2Gi"
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi
---
apiVersion: v1
kind: Service
metadata:
  name: blaster-db
  namespace: blaster
spec:
  type: ClusterIP
  selector:
    app: blaster-db
  ports:
    - name: postgres
      port: 5432
      targetPort: 5432

4. App secrets, config and Deployment​

4.1 App-level Secret​

Before SOPS encryption

---
# k8s/prod/30-secret-app.enc.yaml
apiVersion: v1
kind: Secret
metadata:
  name: blaster-app-secret
  namespace: blaster
type: Opaque
stringData:
  CLERK_SECRET_KEY: "REPLACE_WITH_REAL_CLERK_SECRET_KEY"

4.2 ConfigMap for public app config​

---
# k8s/prod/40-app-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: blaster-app-config
  namespace: blaster
data:
  NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY: "REPLACE_WITH_PUBLIC_KEY"

4.3 App Deployment and Service​

The image will automatically get updated

---
# k8s/prod/50-app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: blaster-app
  namespace: blaster
spec:
  replicas: 3
  selector:
    matchLabels:
      app: blaster-app
  template:
    metadata:
      labels:
        app: blaster-app
    spec:
      imagePullSecrets:
        - name: blaster-prod-registry
      # Run database migrations before starting the app
      initContainers:
        - name: migrate
          image: registry.reids.net.au/games/blaster:prod-20251114.14 # {"$imagepolicy": "flux-system:blaster-prod-policy"}
          imagePullPolicy: IfNotPresent
          command: ["npm", "run", "migrate"]
          envFrom:
            - secretRef:
                name: blaster-db-secret # DB name/user/password
          env:
            - name: POSTGRES_HOST
              value: "blaster-db"
            - name: POSTGRES_PORT
              value: "5432"
      containers:
        - name: blaster
          image: registry.reids.net.au/games/blaster:prod-20251114.14 # {"$imagepolicy": "flux-system:blaster-prod-policy"}
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http
          envFrom:
            - secretRef:
                name: blaster-db-secret
            - secretRef:
                name: blaster-app-secret
            - configMapRef:
                name: blaster-app-config
          env:
            - name: POSTGRES_HOST
              value: "blaster-db"
            - name: POSTGRES_PORT
              value: "5432"
          readinessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 15
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 60
            periodSeconds: 20
          resources:
            requests:
              cpu: "300m"
              memory: "768Mi"
            limits:
              cpu: "2"
              memory: "2Gi"
---
apiVersion: v1
kind: Service
metadata:
  name: blaster-app
  namespace: blaster
spec:
  type: ClusterIP
  selector:
    app: blaster-app
  ports:
    - name: http
      port: 80
      targetPort: 3000

5. Ingress and TLS​

5.1 Ingress for blaster.muppit.au​

---
# k8s/prod/60-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blaster-ingress
  namespace: blaster
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  tls:
    - hosts:
        - blaster.muppit.au
      secretName: blaster-muppit-au-tls
  rules:
    - host: "blaster.muppit.au"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: blaster-app
                port:
                  number: 80

6. SOPS encryption workflow​

6.1 Encrypt production Secrets​

note

The SOPS_AGE_KEY_FILE was set during dev.

Encrypt the Secrets:

sops -e -i k8s/prod/10-secret-db.enc.yaml 
sops -e -i k8s/prod/30-secret-app.enc.yaml
sops -e -i k8s/prod/70-secret-cfapi-token.enc.yaml

6.2 Verify Secrets are encrypted​

head -n 20 k8s/prod/10-secret-db.enc.yaml
head -n 20 k8s/prod/30-secret-app.enc.yaml
head -n 20 k8s/prod/70-secret-cfapi-token.enc.yaml
# k8s/prod/10-secret-db.enc.yaml
apiVersion: v1
kind: Secret
metadata:
    name: blaster-db-secret
    namespace: blaster
type: Opaque
stringData:
    POSTGRES_DB: ENC[AES256_GCM,data:type:str]
    POSTGRES_USER: ENC[AES256_GCM,data:type:str]
    POSTGRES_PASSWORD: ENC[AES256_GCM,data:type:str]
sops:
    age:
        - recipient: age...
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
# k8s/prod/30-secret-app.enc.yaml
apiVersion: v1
kind: Secret
metadata:
    name: blaster-app-secret
    namespace: blaster
type: Opaque
stringData:
    CLERK_SECRET_KEY: ENC[AES256_GCM,data:type:str]
sops:
    age:
        - recipient: age1...
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
# k8s/prod/70-secret-cfapi-token.enc.yaml
apiVersion: v1
kind: Secret
metadata:
    name: cfapi-token
    namespace: blaster
type: Opaque
stringData:
    key: ENC[AES256_GCM,data:type:str]
sops:
    age:
        - recipient: age1...
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----

7. Commit and cluster verification​

7.1 Commit and push​

git add .
git commit -m "Added prod config"
[develop 0753f72] Added prod config
git push
Enumerating objects: 13, done.
Counting objects: 100% (13/13), done.
Delta compression using up to 24 threads
Compressing objects: 100% (11/11), done.
Writing objects: 100% (11/11), 4.28 KiB | 4.28 MiB/s, done.
Total 11 (delta 1), reused 0 (delta 0), pack-reused 0 (from 0)
remote: 
remote: To create a merge request for develop, visit:
remote:   https://gitlab.reids.net.au/games/blaster/-/merge_requests/new?merge_request%5Bsource_branch%5D=develop
remote: 
To https://gitlab.reids.net.au/games/blaster.git
   80ee521..0753f72  develop -> develop
git pull
Already up to date.

7.2 Kubernetes health checks​

kubectl -n blaster get certificate blaster-muppit-au-origin
NAME                       READY   SECRET                  AGE
blaster-muppit-au-origin   True    blaster-muppit-au-tls   3m45s
kubectl -n blaster get secret blaster-muppit-au-tls
NAME                    TYPE                DATA   AGE
blaster-muppit-au-tls   kubernetes.io/tls   3      4m3s
kubectl get all -n blaster
NAME                               READY   STATUS    RESTARTS   AGE
pod/blaster-app-645d7dbbf5-4c7pk   1/1     Running   0          7m35s
pod/blaster-app-645d7dbbf5-bq98x   1/1     Running   0          7m56s
pod/blaster-app-645d7dbbf5-glqj9   1/1     Running   0          8m26s
pod/blaster-db-0                   1/1     Running   0          64m

NAME                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/blaster-app   ClusterIP   10.50.96.236   <none>        80/TCP     64m
service/blaster-db    ClusterIP   10.50.109.50   <none>        5432/TCP   64m

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/blaster-app   3/3     3            3           64m

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/blaster-app-645d7dbbf5   3         3         3       8m26s
replicaset.apps/blaster-app-796cd7db74   0         0         0       64m
replicaset.apps/blaster-app-86c777c59b   0         0         0       61m

NAME                          READY   AGE
statefulset.apps/blaster-db   1/1     64m
kubectl get all -n blaster -o wide
NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE       NOMINATED NODE   READINESS GATES
pod/blaster-app-645d7dbbf5-4c7pk   1/1     Running   0          7m44s   10.50.153.176   k8s-w-p1   <none>           <none>
pod/blaster-app-645d7dbbf5-bq98x   1/1     Running   0          8m5s    10.50.153.181   k8s-w-p1   <none>           <none>
pod/blaster-app-645d7dbbf5-glqj9   1/1     Running   0          8m35s   10.50.153.179   k8s-w-p1   <none>           <none>
pod/blaster-db-0                   1/1     Running   0          64m     10.50.153.161   k8s-w-p1   <none>           <none>

NAME                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE   SELECTOR
service/blaster-app   ClusterIP   10.50.96.236   <none>        80/TCP     64m   app=blaster-app
service/blaster-db    ClusterIP   10.50.109.50   <none>        5432/TCP   64m   app=blaster-db

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                                 SELECTOR
deployment.apps/blaster-app   3/3     3            3           64m   blaster      registry.reids.net.au/games/blaster:prod-20251116.62   app=blaster-app

NAME                                     DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES                                                 SELECTOR
replicaset.apps/blaster-app-645d7dbbf5   3         3         3       8m35s   blaster      registry.reids.net.au/games/blaster:prod-20251116.62   app=blaster-app,pod-template-hash=645d7dbbf5
replicaset.apps/blaster-app-796cd7db74   0         0         0       64m     blaster      registry.reids.net.au/games/blaster:prod-20251114.14   app=blaster-app,pod-template-hash=796cd7db74
replicaset.apps/blaster-app-86c777c59b   0         0         0       61m     blaster      registry.reids.net.au/games/blaster:prod-20251116.58   app=blaster-app,pod-template-hash=86c777c59b

NAME                          READY   AGE   CONTAINERS   IMAGES
statefulset.apps/blaster-db   1/1     64m   postgres     postgres:15.8

8. Verification checklist​

  • blaster-db StatefulSet is Ready=1/1.
  • blaster-app Deployment is Ready=3/3 and using the expected prod image tag.
  • blaster-ingress exists and points at blaster-app Service on port 80.
  • blaster-muppit-au-tls Secret present and referenced by the Ingress.
  • curl -k https://blaster.muppit.au/ -I returns HTTP/2 200 once Cloudflare is configured.