Dev GitOps via flux runbook

This runbook provides end-to-end instructions on how to deploy a game onto the cluster using GitOps automation via Flux.

• This runbook has been tested on:
  • Kubernetes version: v1.31.9
  • OS-Image: Ubuntu 24.04.2 LTS
  • Kernel version: 6.8.0-62-generic
  • Container runtime: containerd://2.0.5

info

Code moves from local dev to k8s dev to k8s prod using GitLab CI (with Kaniko), FluxCD and merge requests. Prod is only updated from tested commits on main. Images are automatically built and dynamic image tags used.

Blaster GitOps series

1. Blaster GitOps summary
2. Blaster repo and branches
3. Dockerfile & GitLab CI
4. Clerk authentication & user setup
5. Google OAuth for Clerk
6. Blaster prep for automation
7. Dev app k8s manifests
8. Dev flux sources & Kustomizations
9. Dev image automation
10. Dev SOPS & age
11. Dev verification & troubleshooting
12. Dev full runbook - you are here
13. Prod overview
14. Prod app k8s manifests and deployment
15. Prod Flux GitOps and image automation
16. Prod Cloudflare, Origin CA and tunnel routing
17. Prod full runbook
18. Post development branches

• Flux image automation and reflector use the flux-system secret to talk to the registry.
• The Blaster Deployment specifies imagePullSecrets: blaster-dev-registry in the blaster-dev namespace, and Kubelet only looks in that namespace for image pull secrets.
• Define a SOPS-encrypted blaster-dev-registry Secret in the blaster-dev and flux-system namespaces, added via Kustomizations:
  • Git is source of truth.
  • Flux decrypts and applies both secrets.
  • Controllers and Pods can all authenticate to the same registry.

Controllers in flux-system use their copy, the app namespace uses its copy, and the image automation loop keeps ticking without manual kubectl patching.

0.1 Prerequisites

• GitLab:
  • Project: games/blaster.
  • GitLab Container Registry enabled for the project.
  • GitLab Runner using the Kubernetes executor in namespace gitlab.
• Cluster & GitOps:
  • Kubernetes cluster reachable from the GitLab Runner.
  • FluxCD installed and managing the cluster (namespace flux-system).
  • An infra repo (for example clusters/my-cluster) where you define Flux GitRepository and Kustomization objects.
• Networking & DNS:
  • Internal DNS / ingress for blaster.reids.net.au (dev, cluster-only).
  • Public DNS / Cloudflare tunnel for blaster.muppit.au (prod).
• Slack (optional but configured here):
  • Slack workspace and app with Bot token (xoxb-…).
  • Target Slack channel ID for pipeline notifications.
• Blaster app:
  • Next.js app with package.json scripts:
    • "build": "next build"
    • "start": "next start"
    • "lint": "next lint"
    • "test": "npm run lint" (or similar)
  • Tailwind/PostCSS configured as devDependencies and a single Dockerfile at repo root (see below).

---

1.1 Prepare GitLab project

Blank project with no readme.

• Group: games
• Project: games/blaster
• Clone with HTTPS:https://gitlab.reids.net.au/games/blaster.git**

2.1 Initialise Git

cd ~/Projects/blaster 
git init
git add .
git commit -m "Initial commit"

Initialized empty Git repository in /Users/andy/Projects/blaster/.git/
[master (root-commit) 421441a] Initial commit

2.1.1 Push to the blank GitLab project

git remote add origin https://gitlab.reids.net.au/games/blaster.git
git branch -M main
git push -u origin main

Enumerating objects: 54, done.
Counting objects: 100% (54/54), done.
Delta compression using up to 24 threads
Compressing objects: 100% (50/50), done.
Writing objects: 100% (54/54), 39.37 MiB | 35.27 MiB/s, done.
Total 54 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
To https://gitlab.reids.net.au/games/blaster.git
 * [new branch]      main -> main
branch 'main' set up to track 'origin/main'.

---

2.2 Give Flux write access to the Git repo

Image automation needs to commit back to your GitLab repo. You only do this once per cluster.

At a high level:

1. Identify which SSH key Flux is using for Git.
2. Ensure the matching deploy key in GitLab has write access.
3. Optionally use a separate keypair for image automation if you want stricter separation.

2.2.1 Confirm which key Flux uses

Check the GitRepository that points to your infra repo:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: blaster
  namespace: flux-system
spec:
  url: ssh://git@gitlab.reids.net.au/muppit-apps/clusters.git
  secretRef:
    name: flux-ssh-auth

Then inspect the secret:

kubectl -n flux-system get secret flux-ssh-auth -o yaml

You should see something like:

data:
  identity: <base64-encoded-private-key>
  identity.pub: <base64-encoded-public-key>

Decode the public key if you want to double-check it against GitLab:

kubectl -n flux-system get secret flux-ssh-auth \
  -o jsonpath='{.data.identity\.pub}' | base64 -d

2.2.2 Enable write access on the GitLab deploy key

In GitLab, for the infra repo that Flux is updating:

1. Go to Settings → Repository → Deploy keys.
2. Find the key whose value matches the identity.pub you just decoded (for example, flux-ssh-key).
3. Tick “Write access allowed”.
4. Save.

This allows Flux to push commits (for example, when the image automation controller bumps the image tag fields in your manifests).

warning

Be deliberate about where you grant write access. Any process using this key can push to that repo.

2.2.3 Quick verification

• GitRepository points at the correct SSH URL for your infra repo.
• secretRef.name exists and contains a valid private key (identity) and public key (identity.pub).
• Matching deploy key in GitLab has write access enabled.
• If using ImageUpdateAutomation.spec.git.secretRef, it points to the secret you expect.

2.2.4 Create a GitLab deploy token for the registry

In GitLab → games/blaster project:

1. Go to Settings → Repository → Deploy Tokens.
2. Create a token:
   1. Name: blaster-dev-registry
   2. Scopes: read_registry and read_repository
   3. Save the username and token.

In that GitLab form:

• Name: blaster-dev-token
• Username: either:
  • leave blank → GitLab will generate gitlab+deploy-token or set something short like blaster-dev if you prefer readability
• Scopes:
  • read_repository
  • read_registry

This gives one token that can:

• be used by Flux GitRepository over HTTPS:
• and by Kubernetes as a docker-registry secret to pull registry.reids.net.au/games/blaster:*:

2.2.5 Create a docker-registry secret in blaster-dev

On your workstation:

kubectl create secret docker-registry blaster-dev-registry \
  -n blaster-dev \
  --docker-server=registry.reids.net.au \
  --docker-username='<DEPLOY_TOKEN_USERNAME>' \
  --docker-password='<DEPLOY_TOKEN_VALUE>' \
  --docker-email='blaster-dev-registry@example.local'

Replace the username/token with the values GitLab gave you.

secret/blaster-dev-registry created

You can confirm:

kubectl get secret blaster-dev-registry -n blaster-dev

NAME                   TYPE                             DATA   AGE
blaster-dev-registry   kubernetes.io/dockerconfigjson   1      21s

---

2.3 Single Blaster repo

Use a single games/blaster repo with:

• Branch-per-stage model:
  • main for prod.
  • develop for k8s dev.
  • feature/* branches off develop.
• Env-specific k8s overlays in the app repo:
  • k8s/dev and k8s/prod.
• Flux GitRepository per env:
  • One points to develop + k8s/dev.
  • One points to main + k8s/prod.
• GitLab CI:
  • Lint + test on all branches.
  • Image builds only on develop and main using Kaniko.
• Promotion by merge request:
  • feature/* → develop for k8s dev.
  • develop → main for k8s prod.

Everything runs through Git for full audit trail.

2.4 Git branches

Recommended model for games/blaster:

Branch	Purpose	Deployed where	Who can push
main	Production code only	k8s prod (blaster.muppit.au)	Merge requests only
develop	Integration / k8s dev	k8s dev (blaster.reids.net.au)	Merge requests only
feature/*	Local feature work	Local dev only (localhost:3000)	Direct by developer

Rules:

• Protect main and develop in GitLab.
• Require a green pipeline before merging into develop or main.
• Only tag or release from main.

---

3.1 High-level layout

Two repos, clear responsibilities:

• App repo: games/blaster

  • Next.js + Phaser game code
  • Dockerfile for the app
  • .gitlab-ci.yml that builds and pushes an image to the GitLab Container Registry
  • Kubernetes manifests under k8s/prod and k8s/dev

• Infra repo: fluxgitops/flux-config

  • Cluster-level config; Flux root at ./clusters/my-cluster
  • New folder: clusters/my-cluster/blaster
  • Numbered files:
    • 00-blaster-source.yaml (GitRepository pointing at games/blaster)
    • 10-blaster-kustomization.yaml (Kustomization pointing at ./k8s/prod)
    • kustomization.yaml (local aggregator)

Namespace strategy:

• Namespace for dev is blaster-dev.
• Namespace for prod is blaster.

---

3.1.1 Directory layout in the app repo

Top level of games/blaster:

games/blaster/
  .env.local          # Local dev only, never committed
  Dockerfile
  package.json
  app/, public/, etc.
  k8s/
    dev/
      kustomization.yaml
      deployment.yaml
      service.yaml
      ingress.yaml    # blaster.reids.net.au, internal only
    prod/
      kustomization.yaml
      deployment.yaml
      service.yaml
      ingress.yaml    # blaster.muppit.au via Cloudflare tunnel

Principle:

• App-specific manifests live with the app.
• Cluster-level Flux config stays in the infra repo (clusters/my-cluster/...).

3.1.2 Creating the initial develop branch

From inside games/blaster:

git checkout main
git pull origin main

git branch develop
git push -u origin develop

To protect main and develop and enforce merge requests, do this in GitLab:

1. In GitLab, open your games/blaster project.
2. Left sidebar: go to Settings → Repository.
3. Scroll down to the Protected branches section.

For each branch (main and develop):

4. In Branch dropdown, pick main and click Protect (or “Protect branch” after setting options).
   • Set:
     • Allowed to merge: Maintainers (or a small role set you are happy with).
     • Allowed to push: ideally No one or Maintainers (but rely on MRs).
   • Tick options like:
     • “Require approval” / “Require merge request approval” (if you use approvals).
5. Repeat for develop.

To require merge requests (no direct pushes):

• With the branch protected and Allowed to push not including regular developers, they have to open an MR.
• Also, under Settings → General → Merge request approvals and Settings → General → Merge requests, you can:
  • Require a pipeline to succeed before merge.
  • Disable fast-forward merges if you want a clean history.

---

4.1 Single Dockerfile for dev and prod

Keep one Dockerfile at the repo root and let GitLab CI + tags + env create different images for develop and main. The image recipe stays the same; the branch, tag and k8s environment variables make it “dev” or “prod”.

• develop builds registry.reids.net.au/games/blaster:dev-YYYYMMDD.N.
• main builds registry.reids.net.au/games/blaster:prod-YYYYMMDD.N.

Most differences between k8s dev and prod live in Kubernetes manifests and env vars, not in duplicate Dockerfiles.

Dockerfile:

# syntax=docker/dockerfile:1.7

FROM node:22-alpine3.20 AS build
WORKDIR /app

# Build args for compile-time config
ARG APP_ENV=prod
ENV APP_ENV=${APP_ENV}

ARG NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY
ENV NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=${NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY}

COPY package*.json ./

# Install deps INCLUDING devDependencies (tailwindcss, postcss, etc)
RUN npm ci --include=dev

COPY . .
RUN npm run build

FROM node:22-alpine3.20 AS runtime
WORKDIR /app

ENV NODE_ENV=production
COPY --from=build /app ./

# Explicit form - same as ["npm", "start"]
CMD ["npm", "run", "start"]

This:

• Installs devDependencies so Tailwind/PostCSS are available for next build in the build stage.
• Copies the built app into a clean runtime image.
• Starts the app using the start script from package.json.

Kubernetes Deployment objects supply the runtime env (for example Clerk keys, API URLs and any APP_ENV override) via env or envFrom on the container.

---

5.1 GitLab CI: lint, test and Kaniko builds (with Slack)

5.1.1 CI/CD variables for Slack notifications

Required only if you want pipeline notifications to Slack.

note

Replace xoxb-REDACTED and C0123ABCD with your own values.

In GitLab, under Settings → CI/CD → Variables, add:

1. SLACK_BOT_TOKEN
   • Value: xoxb-REDACTED
   • Expanded, masked, hidden, not protected.
2. SLACK_CHANNEL
   • Value: C0123ABCD
   • Expanded, not protected.
3. SLACK_NOTIFY
   • Value: true
   • Expanded, not protected.
4. NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY
   • Required for the build process.
   • Expanded, not protected.

5.1.2 .gitlab-ci.yml

Do not hardcode registry.reids.net.au in CI; GitLab fills in:

• $CI_REGISTRY (e.g. registry.reids.net.au)
• $CI_REGISTRY_IMAGE (e.g. registry.reids.net.au/games/blaster)

Working pipeline for Blaster (using Kaniko):

stages:
  - notify
  - lint
  - test
  - build
  - notify_end

variables:
  # GitLab project image, e.g. registry.reids.net.au/games/blaster
  DOCKER_IMAGE: "$CI_REGISTRY_IMAGE"

# --------------------------------------------------------------------
# Slack threaded notifications (bot token)
#
# Required CI/CD variables:
#   SLACK_NOTIFY=true
#   SLACK_BOT_TOKEN=xoxb-...
#   SLACK_CHANNEL=C0123ABC
# --------------------------------------------------------------------

notify:start:
  stage: notify
  image: alpine:3.20
  rules:
    - if: '$SLACK_NOTIFY == "true"'
  before_script:
    - apk add --no-cache curl jq
  script:
    - |
      MSG="*Pipeline* <${CI_PIPELINE_URL}|#${CI_PIPELINE_ID}> for *${CI_PROJECT_PATH}* started on ${CI_COMMIT_REF_NAME} by ${GITLAB_USER_LOGIN}."
      MORE="${CI_COMMIT_SHORT_SHA}: $(echo "$CI_COMMIT_TITLE" | head -c 120)"
      jq -n --arg ch "$SLACK_CHANNEL" --arg msg "$MSG" --arg more "$MORE" \
        '{channel:$ch, text:$msg,
          blocks:[
            { "type":"section","text":{"type":"mrkdwn","text":$msg}},
            { "type":"context","elements":[{"type":"mrkdwn","text":$more}]}
          ]}' > payload.json
      curl -sS -H "Authorization: Bearer $SLACK_BOT_TOKEN" \
               -H "Content-type: application/json" \
               -X POST https://slack.com/api/chat.postMessage \
               --data @payload.json | tee slack.resp.json
      jq -r '.ts // empty' slack.resp.json | tee slack.ts
  artifacts:
    paths:
      - slack.ts
      - slack.resp.json
    expire_in: 1 day

# --------------------------------------------------------------------
# Lint and test (no container build)
# --------------------------------------------------------------------

lint:
  stage: lint
  image: node:22-alpine3.20
  script:
    - npm ci
    - npm run lint

test:
  stage: test
  image: node:22-alpine3.20
  script:
    - npm ci
    - npm test   # your package.json now has a "test" script

# --------------------------------------------------------------------
# Kaniko builds - no docker daemon, good for Kubernetes runners
# Tag format for Flux:
#   dev:  registry.reids.net.au/games/blaster:dev-YYYYMMDD.IID
#   prod: registry.reids.net.au/games/blaster:prod-YYYYMMDD.IID
# --------------------------------------------------------------------

# Build and push for develop (k8s dev)
build:develop:
  stage: build
  rules:
    # 1. If commit message contains [skip ci], do not run this job
    - if: '$CI_COMMIT_MESSAGE =~ /\\[skip ci\\]/i'
      when: never
    # 2. Run on every commit to develop
    - if: '$CI_COMMIT_BRANCH == "develop"'
      when: on_success
    # 3. Fallback: never
    - when: never
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - export DATE_TAG="$(date +%Y%m%d).${CI_PIPELINE_IID}"
    - export TAG="dev-${DATE_TAG}"
    - export IMAGE="${DOCKER_IMAGE}:${TAG}"
    - echo "Building tag ${IMAGE} with Kaniko"
    - mkdir -p /kaniko/.docker
    - |
      cat <<EOF >/kaniko/.docker/config.json
      {
        "auths": {
          "${CI_REGISTRY}": {
            "username": "${CI_REGISTRY_USER}",
            "password": "${CI_REGISTRY_PASSWORD}"
          }
        }
      }
      EOF
    - |
      /kaniko/executor \
        --context "${CI_PROJECT_DIR}" \
        --dockerfile "${CI_PROJECT_DIR}/Dockerfile" \
        --destination "${IMAGE}" \
        --cache=false \
        --build-arg NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY="${NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY}"

# Build and push for main (k8s prod)
build:main:
  stage: build
  rules:
    - if: '$CI_COMMIT_MESSAGE =~ /\\[skip ci\\]/i'
      when: never
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: on_success
    - when: never
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - export DATE_TAG="$(date +%Y%m%d).${CI_PIPELINE_IID}"
    - export TAG="prod-${DATE_TAG}"
    - export IMAGE="${DOCKER_IMAGE}:${TAG}"
    - echo "Building tag ${IMAGE} with Kaniko"
    - mkdir -p /kaniko/.docker
    - |
      cat <<EOF >/kaniko/.docker/config.json
      {
        "auths": {
          "${CI_REGISTRY}": {
            "username": "${CI_REGISTRY_USER}",
            "password": "${CI_REGISTRY_PASSWORD}"
          }
        }
      }
      EOF
    - |
      /kaniko/executor \
        --context "${CI_PROJECT_DIR}" \
        --dockerfile "${CI_PROJECT_DIR}/Dockerfile" \
        --destination "${IMAGE}" \
        --cache=false \
        --build-arg NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY="${NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY}"

# --------------------------------------------------------------------
# Slack notifications on success/failure
# --------------------------------------------------------------------

notify:success:
  stage: notify_end
  image: alpine:3.20
  rules:
    - if: '$SLACK_NOTIFY == "true"'
      when: on_success
  before_script:
    - apk add --no-cache curl jq
  dependencies:
    - notify:start
  script:
    - |
      TS="$(cat slack.ts 2>/dev/null || true)"
      jq -n --arg ch "$SLACK_CHANNEL" --arg ts "$TS" --arg msg "Pipeline succeeded: <${CI_PIPELINE_URL}|#${CI_PIPELINE_ID}>." \
        '{channel:$ch, thread_ts:$ts, text:$msg}' > payload.json
      curl -sS -H "Authorization: Bearer $SLACK_BOT_TOKEN" \
               -H "Content-type: application/json" \
               -X POST https://slack.com/api/chat.postMessage \
               --data @payload.json >/dev/null

notify:failure:
  stage: notify_end
  image: alpine:3.20
  rules:
    - if: '$SLACK_NOTIFY == "true"'
      when: on_failure
  allow_failure: true
  before_script:
    - apk add --no-cache curl jq
  dependencies:
    - notify:start
  script:
    - |
      TS="$(cat slack.ts 2>/dev/null || true)"
      jq -n --arg ch "$SLACK_CHANNEL" --arg ts "$TS" --arg msg "Pipeline failed: <${CI_PIPELINE_URL}|#${CI_PIPELINE_ID}>. Stage ${CI_JOB_STAGE}, job ${CI_JOB_NAME}." \
        '{channel:$ch, thread_ts:$ts, text:$msg}' > payload.json
      curl -sS -H "Authorization: Bearer $SLACK_BOT_TOKEN" \
               -H "Content-type: application/json" \
               -X POST https://slack.com/api/chat.postMessage \
               --data @payload.json >/dev/null

Notes:

• Lint and test use a Node image only, no Docker.
• Kaniko jobs use CI_REGISTRY, CI_REGISTRY_USER and CI_REGISTRY_PASSWORD injected by GitLab.
• Tags are date-based: dev-YYYYMMDD.N / prod-YYYYMMDD.N, where N is the pipeline IID.

5.1.3 Example success output

When everything is configured correctly, a successful build:main job shows:

> galaga-emoji@0.1.0 build
> next build
  ▲ Next.js 14.2.33
   Creating an optimized production build ...
   ...
   Finalizing page optimization ...
   Collecting build traces ...

Route (app)                              Size     First Load JS
┌ ○ /                                    21.4 kB         145 kB
...

INFO[0041] CMD ["npm", "run", "start"]
INFO[0041] Pushing image to registry.reids.net.au/games/blaster:prod-20251114.11
INFO[0083] Pushed registry.reids.net.au/games/blaster@sha256:28b8ae84...
Job succeeded

You should also see the new tag (prod-20251114.11 in this example) in GitLab → Packages & Registries → Container Registry for the project.

info

The container registry will contain tags for each dev and prod image. Such as: dev-20251114.15; dev-20251114.12; prod-20251114.14; prod-20251114.11.

---

6.1 App repo: k8s/dev

games/blaster/k8s/dev

Switch to the develop branch.

git switch develop

Switched to branch 'develop'

git pull

Your branch is up to date with 'origin/develop'.

End state games/blaster repo structure:

├── .gitlab-ci.yml
├── .sops.yaml
├── app
├── Dockerfile
├── k8s
│   ├── dev
│   │   ├── 10-secret-db.enc.yaml
│   │   ├── 20-db-statefulset.yaml
│   │   ├── 30-secret-app.enc.yaml
│   │   ├── 40-app-config.yaml
│   │   ├── 50-app-deployment.yaml
│   │   ├── 60-ingress.yaml
│   │   └── kustomization.yaml
└── package.json

6.1.1 SOPS policy

Add a SOPS policy at repo root:

# .sops.yaml
creation_rules:
  - path_regex: k8s/.*\.enc\.ya?ml$
    encrypted_regex: '^(data|stringData)$'
    age: ['AGE_PUBLIC_KEY_HERE']

Replace AGE_PUBLIC_KEY_HERE with your public key (the one starting with age1...).

6.1.2 Create manifests before SOPS encryption

Create the files under k8s/dev/.

mkdir -p k8s/dev

6.1.2.2 Shared DB secret

Before SOPS encryption

---
# k8s/dev/10-secret-db.enc.yaml
apiVersion: v1
kind: Secret
metadata:
  name: blaster-db-secret
  namespace: blaster-dev
type: Opaque
stringData:
  POSTGRES_DB: blaster_game
  POSTGRES_USER: blaster_user
  POSTGRES_PASSWORD: "REPLACE_WITH_STRONG_PASSWORD"

This is the single source of truth for DB credentials. The DB and app both consume this Secret via envFrom.

note

Edit via sops so secrets stay encrypted in Git.

6.1.2.3 PostgreSQL StatefulSet and Service

---
# k8s/dev/20-db-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: blaster-db
  namespace: blaster-dev
spec:
  serviceName: blaster-db
  replicas: 1
  selector:
    matchLabels:
      app: blaster-db
  template:
    metadata:
      labels:
        app: blaster-db
    spec:
      containers:
        - name: postgres
          image: postgres:15.8
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5432
              name: postgres
          envFrom:
            - secretRef:
                name: blaster-db-secret
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
          readinessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
            initialDelaySeconds: 10
            periodSeconds: 10
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
            initialDelaySeconds: 30
            periodSeconds: 30
          resources:
            requests:
              cpu: "100m"
              memory: "256Mi"
            limits:
              cpu: "500m"
              memory: "512Mi"
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: blaster-db
  namespace: blaster-dev
spec:
  type: ClusterIP
  selector:
    app: blaster-db
  ports:
    - name: postgres
      port: 5432
      targetPort: 5432

6.1.2.4 App secrets and config

Before SOPS encryption

---
# k8s/dev/30-secret-app.enc.yaml
apiVersion: v1
kind: Secret
metadata:
  name: blaster-app-secret
  namespace: blaster-dev
type: Opaque
stringData:
  CLERK_SECRET_KEY: "REPLACE_WITH_REAL_CLERK_SECRET_KEY"

---
# k8s/dev/40-app-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: blaster-app-config
  namespace: blaster-dev
data:
  NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY: "REPLACE_WITH_PUBLIC_KEY"

6.1.2.5 App Deployment and Service

The image will automatically get updated

---
# k8s/dev/50-app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: blaster-app
  namespace: blaster-dev
spec:
  replicas: 2
  selector:
    matchLabels:
      app: blaster-app
  template:
    metadata:
      labels:
        app: blaster-app
    spec:
      imagePullSecrets:
        - name: blaster-dev-registry
      containers:
        - name: blaster
          image: registry.reids.net.au/games/blaster:dev-20251115.42 # {"$imagepolicy": "flux-system:blaster-dev-policy"}
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http
          envFrom:
            - secretRef:
                name: blaster-db-secret      # DB name/user/password
            - secretRef:
                name: blaster-app-secret     # Clerk secret
            - configMapRef:
                name: blaster-app-config     # public config
          env:
            - name: POSTGRES_HOST
              value: "blaster-db"
            - name: POSTGRES_PORT
              value: "5432"
          readinessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 15
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 60
            periodSeconds: 20
          resources:
            requests:
              cpu: "200m"
              memory: "512Mi"
            limits:
              cpu: "1"
              memory: "1Gi"
---
apiVersion: v1
kind: Service
metadata:
  name: blaster-app
  namespace: blaster-dev
spec:
  type: ClusterIP
  selector:
    app: blaster-app
  ports:
    - name: http
      port: 80
      targetPort: 3000

6.1.2.6 Ingress

blaster.reids.net.au

File: k8s/dev/60-ingress.yaml

---
# k8s/dev/60-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blaster-ingress
  namespace: blaster-dev

spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - blaster.reids.net.au
  rules:
    - host: "blaster.reids.net.au"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: blaster-app
                port:
                  number: 80

6.1.2.7 Kustomization

File: k8s/dev/kustomization.yaml

# k8s/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: blaster-dev
resources:
  - 10-secret-db.enc.yaml
  - 20-db-statefulset.yaml
  - 30-secret-app.enc.yaml
  - 40-app-config.yaml
  - 50-app-deployment.yaml
  - 60-ingress.yaml

---

6.1.3 SOPS encryption

note

Ensure that SOPS_AGE_KEY_FILE has been set.

Assuming that the age.key is stored in $HOME/.sops/

# ~/.zshrc
# SOPS
export SOPS_AGE_KEY_FILE="$HOME/.sops/age.key"

Encrypt the two Secrets:

sops -e -i k8s/dev/10-secret-db.enc.yaml
sops -e -i k8s/dev/30-secret-app.enc.yaml

Verify they’re encrypted:

head -n 20 k8s/dev/10-secret-db.enc.yaml
head -n 20 k8s/dev/30-secret-app.enc.yaml
grep -n 'sops:' k8s/dev/*.enc.yaml

---

6.1.4 Git commit and push games/blaster

git add .
git commit -m "Updated k8s/dev manifests"
git push

Running with gitlab-runner 18.1.0 (0731d300)
  on gitlab-gitlab-runner-654997b8fc-cxvqm tPrZ7hky2, system ID: r_q6JnKeFaNsFc
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab
$ echo "Building tag ${IMAGE} with Kaniko"
Building tag registry.reids.net.au/games/blaster:dev-20251115.49 with Kaniko
$ mkdir -p /kaniko/.docker
   Collecting page data ...
   Generating static pages (0/5) ...
   Generating static pages (1/5) 
   Generating static pages (2/5) 
   Generating static pages (3/5) 
 ✓ Generating static pages (5/5)
   Finalizing page optimization ...
   Collecting build traces ...
○  (Static)   prerendered as static content
ƒ  (Dynamic)  server-rendered on demand
INFO[0036] Pushing image to registry.reids.net.au/games/blaster:dev-20251115.49 
INFO[0071] Pushed registry.reids.net.au/games/blaster@sha256:0e24aa956ed3a142f87b1fbc997135535a0f1af0e03d69ed6b268adc7d63dad8 
Cleaning up project directory and file based variables
00:00
Job succeeded

6.1.5 Confirm repo layout

Final repo layout from repo root:

tree -a -L 3 -I '.git|.DS_Store|node_modules|.next|dist'

├── .dockerignore
├── .gitlab-ci.yml
├── .sops.yaml
├── app
├── database
├── db
├── Dockerfile
├── k8s
│   └── dev
│       ├── 10-secret-db.enc.yaml
│       ├── 20-db-statefulset.yaml
│       ├── 30-secret-app.enc.yaml
│       ├── 40-app-config.yaml
│       ├── 50-app-deployment.yaml
│       ├── 60-ingress.yaml
│       └── kustomization.yaml
├── lib
└── package.json

---

Database initialisation

warning

This was too manual when setting it up. Need to automate it so that if there is no database schema then a default one will be imported automatically.

---

7.1 Add games/blaster to infra repo: flux-config

In the blaster GitLab project → Settings → Repository → Deploy Keys → Privately accessible deploy keys

• Enable the same Flux deploy key you already use (flux-ssh-auth).
• When enabled it will appear under Enabled deploy keys.

On the workstation, navigate to the previously cloned flux-config project and pull any updates to ensure it is up to date:

cd ~/Projects/flux-config
git pull

7.1.1 Automate image updates to Git

kubectl get crd imagepolicies.image.toolkit.fluxcd.io

• If you get a line of output → CRD exists, and it’s just the wrong apiVersion in your YAML.
• If you get Error from server (NotFound) → you have no image CRDs at all.

If they are not found then you must install the image components by extending the flux bootstrap command to include the image-reflector-controller and image-automation-controller components.

Re-running flux bootstrap with the same repo/branch/path is idempotent:

• It updates the Flux system manifests under clusters/my-cluster/flux-system:
• gotk-components.yaml gets regenerated with the extra components.
• gotk-sync.yaml stays aligned with your repo URL / branch / path.
• It applies those updated manifests back to the cluster.
• It does not delete or reset:
  • Your existing clusters/my-cluster/apps, blaster, wordpress, etc.
  • Any of your GitRepository, Kustomization, ImagePolicy, etc that live elsewhere in the repo.

note

Regenerate the flux-system scaffolding in Git and make the cluster match it.

It will:

• Add image-reflector-controller and image-automation-controller to the components set.
• Create the image CRDs.
• Leave your existing Flux config (including the new blaster stuff) alone.

Refer to the Flux guide for more infomation.

cd ~/Projects/flux-config
git status

On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

7.1.2 Add flux components

warning

Use the same bootstrap command for original installation but with the extra components.

flux bootstrap gitlab \
  --hostname=gitlab.reids.net.au \
  --owner=fluxgitops \
  --repository=flux-config \
  --branch=main \
  --path=clusters/my-cluster \
  --deploy-token-auth \
  --insecure-skip-tls-verify \
  --read-write-key \
  --components-extra=image-reflector-controller,image-automation-controller

Please enter your GitLab personal access token (PAT): 
► connecting to https://gitlab.reids.net.au
► cloning branch "main" from Git repository "https://gitlab.reids.net.au/fluxgitops/flux-config.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ component manifests are up to date
✔ reconciled components
► checking to reconcile deploy token for source secret
✔ configured deploy token "flux-system-main-flux-system-./clusters/my-cluster" for "https://gitlab.reids.net.au/fluxgitops/flux-config"
► determining if source secret "flux-system/flux-system" exists
► generating source secret
► applying source secret "flux-system/flux-system"
✔ reconciled source secret
► generating sync manifests
✔ generated sync manifests
✔ sync manifests are up to date
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for GitRepository "flux-system/flux-system" to be reconciled
✔ GitRepository reconciled successfully
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✔ Kustomization reconciled successfully
► confirming components are healthy
✔ helm-controller: deployment ready
✔ image-automation-controller: deployment ready
✔ image-reflector-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy

7.1.3 Create manifests (plaintext)

Create the following files under clusters/my-cluster/blaster/.

mkdir -p clusters/my-cluster/blaster/dev

Create a dev aggregator for the blaster app in the infra repo.

fluxgitops/flux-config/
├── .sops.yaml
└── clusters
    └── my-cluster
        ├── blaster
        │   ├── 00-namespace.yaml
        │   ├── dev
        │   │   ├── 20-blaster-images-dev.yaml
        │   │   ├── 30-image-automation.yaml
        │   │   ├── kustomization.yaml
        │   │   ├── secrets
        │   │   │   └── blaster-dev-registry.yaml
        │   │   └── source.yaml
        │   ├── kustomization.yaml
        ├── flux-system
        │   ├── gotk-components.yaml
        │   ├── gotk-sync.yaml
        │   ├── kustomization.yaml
        │   └── secrets
        │       └── blaster-dev-registry.yaml
        └── kustomization.yaml

7.1.3.1 Kustomization

Add blaster to the existing Kustomization

# clusters/my-cluster/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ./flux-system
  - ./origin-ca-issuer
  - ./cloudflare
  - ./blaster

7.1.3.2 Blaster kustomization

# clusters/my-cluster/blaster/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
resources:
  - ./00-namespace.yaml
  - ./dev/source.yaml
  - ./dev/kustomization.yaml
  - ./dev/20-blaster-images-dev.yaml
  - ./dev/30-image-automation.yaml

7.1.3.3 Blaster namespace

# clusters/my-cluster/blaster/00-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: blaster-dev
  labels:
    name: blaster-dev

7.1.3.4 Blaster source

# clusters/my-cluster/blaster/dev/source.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: blaster-dev
  namespace: flux-system
spec:
  interval: 1m
  timeout: 60s
  url: ssh://git-ssh.reids.net.au/games/blaster.git
  ref:
    branch: develop
  secretRef:
    name: flux-ssh-auth

7.1.3.5 Dev kustomization

dev/kustomization.yaml connects the GitRepository to the app manifests:

• Namespace: flux-system
• Kustomization name: blaster-dev
• sourceRef.name: blaster-dev → points at the app repo
• path: ./k8s/dev → inside the app repo
• prune: true → delete resources that disappear from Git
• decryption → if any SOPS-encrypted files exist under k8s/dev, use sops-age to decrypt

# clusters/my-cluster/blaster/dev/kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: blaster-dev
  namespace: flux-system
spec:
  interval: 1m
  path: ./k8s/dev
  prune: true
  sourceRef:
    kind: GitRepository
    name: blaster-dev
  wait: true
  timeout: 5m
  decryption:
    provider: sops
    secretRef:
      name: sops-age

7.1.3.6 ImageRepository and ImagePolicy for dev

# clusters/my-cluster/blaster/dev/20-blaster-images-dev.yaml
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: blaster-dev-repo
  namespace: flux-system
spec:
  image: registry.reids.net.au/games/blaster
  interval: 1m
  secretRef:
    name: blaster-dev-registry
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: blaster-dev-policy
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: blaster-dev-repo
  filterTags:
    pattern: '^dev-(?P<date>[0-9]{8})\.(?P<build>[0-9]+)$'
    extract: '$date$build'
  policy:
    numerical:
      order: asc

What this does:

• Scans all tags for registry.reids.net.au/games/blaster.
• Filters tags that look like dev-20251114.1.
• Extracts both the date and build then picks the largest as the winner.
• Exposes the winning tag via the blaster-dev-policy resource.

7.1.3.7 Image update automation

warning

Ensure the [skip ci] is appended to the commit message otherwise a loop can occur.

# clusters/my-cluster/blaster/dev/30-image-automation.yaml
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata:
  name: blaster-dev-automation
  namespace: flux-system
spec:
  interval: 1m
  sourceRef:
    kind: GitRepository
    name: blaster-dev
  git:
    checkout:
      ref:
        branch: develop
    commit:
      author:
        name: FluxCD
        email: andrew@reids.net.au
      messageTemplate: '{{range .Changed.Changes}}{{print .OldValue}} -> {{println .NewValue}}{{end}} [skip ci]'
    push:
      branch: develop
  update:
    strategy: Setters
    path: ./k8s/dev

---

7.1.4 Create manifests

Create the following files under clusters/my-cluster/flux-system/.

mkdir -p clusters/my-cluster/flux-system/secrets

7.1.5 Registry secret

Before SOPS encryption.

info

The dry-run tag allows the command to complete without the namespace existing yet.

kubectl -n flux-system create secret docker-registry blaster-dev-registry \
  --docker-server=registry.reids.net.au \
  --docker-username='blaster-dev' \
  --docker-password='REPLACE-ME' \
  --docker-email='andrew@reids.net.au' \
  --dry-run=client -o yaml \
  > clusters/my-cluster/flux-system/secrets/blaster-dev-registry.yaml

7.1.6 SOPS policy

Add a SOPS policy at repo root:

# .sops.yaml
creation_rules:
  - path_regex: 'clusters/my-cluster/.*/secrets/.*\.yaml$'
    encrypted_regex: '^(data|stringData)$'
    age: ['AGE_PUBLIC_KEY_HERE']

Replace AGE_PUBLIC_KEY_HERE with your public key (the one starting with age1...).

7.1.7 SOPS encryption

note

Ensure that SOPS_AGE_KEY_FILE has been set.

Assuming that the age.key is stored in $HOME/.sops/

# ~/.zshrc
# SOPS
export SOPS_AGE_KEY_FILE="$HOME/.sops/age.key"

Encrypt the Secret:

sops -e -i clusters/my-cluster/flux-system/secrets/blaster-dev-registry.yaml

Verify it is encrypted:

head -n 20 clusters/my-cluster/flux-system/secrets/blaster-dev-registry.yaml

apiVersion: v1
data:
    .dockerconfigjson: ENC[AES256_GCM,data:...]
kind: Secret
metadata:
    name: blaster-dev-registry
    namespace: flux-system
type: kubernetes.io/dockerconfigjson
sops:
    age:
        - recipient: age...
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2025-11-15T11:07:08Z"

Check the sops-age secret on the cluster:

kubectl -n flux-system get secret sops-age

NAME       TYPE     DATA   AGE
sops-age   Opaque   1      35d

7.1.8 SOPS decryption

Flux needs to use the SOPS provider to decrypt the secret and clusters/my-cluster/flux-system/gotk-sync.yaml needs to be extended. However, the manifest was generated by flux. DO NOT EDIT manually.

Patch the existing Kustomization in-cluster for SOPS decryption:

kubectl -n flux-system patch kustomization flux-system \
  --type='merge' \
  -p '{
    "spec": {
      "decryption": {
        "provider": "sops",
        "secretRef": {
          "name": "sops-age"
        }
      }
    }
  }'

Verify:

kubectl -n flux-system get kustomization flux-system -o yaml | sed -n '1,80p'

...
spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-age
  force: false
  interval: 10m0s
  path: ./clusters/my-cluster
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
...

7.1.9 Flux-system kustomization

Add the registry secret file to the resources.

# clusters/my-cluster/flux-system/kustomization.yaml  
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
  - ./secrets/blaster-dev-registry.yaml

---

7.2 Create secret for registry in blaster

kubectl -n blaster create secret docker-registry blaster-dev-registry \
  --docker-server=registry.reids.net.au \
  --docker-username='blaster-dev' \
  --docker-password='REDACTED' \
  --docker-email='andrew@reids.net.au' \
  --dry-run=client -o yaml \
  > clusters/my-cluster/blaster/dev/secrets/blaster-prod-registry.yaml

sops -e -i clusters/my-cluster/blaster/dev/secrets/blaster-prod-registry.yaml

---

7.2.1 Add the secret into the blaster Kustomization

Your current clusters/my-cluster/blaster/kustomization.yaml is:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ./00-namespace.yaml
  - ./dev/source.yaml
  - ./dev/kustomization.yaml
  - ./dev/20-blaster-images-dev.yaml
  - ./dev/30-image-automation.yaml

Add the secret as another resource:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ./00-namespace.yaml
  - ./dev/source.yaml
  - ./dev/kustomization.yaml
  - ./dev/20-blaster-images-dev.yaml
  - ./dev/30-image-automation.yaml
  - ./dev/secrets/blaster-dev-registry.yaml

Then:

git status
git add .sops.yaml clusters/my-cluster/blaster/dev/secrets/blaster-dev-registry.yaml  clusters/my-cluster/blaster/kustomization.yaml
git commit -m "GitOps: add blaster-dev registry secret in app namespace"
git push

---

7.2.2 Reconcile Flux and verify

Kick Flux:

flux reconcile source git flux-system -n flux-system
flux reconcile kustomization flux-system -n flux-system

flux reconcile kustomization flux-system -n flux-system
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision main@sha1:0c88a8099077ae9cefe663344f45ce41b8fac0e3
► annotating Kustomization flux-system in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ applied revision main@sha1:0c88a8099077ae9cefe663344f45ce41b8fac0e3

Then check the Secret appears in the app namespace:

kubectl -n blaster-dev get secret blaster-dev-registry
kubectl -n blaster-dev describe secret blaster-dev-registry | sed -n '1,20p'

NAME                   TYPE                             DATA   AGE
blaster-dev-registry   kubernetes.io/dockerconfigjson   1      79s
Name:         blaster-dev-registry
Namespace:    blaster-dev
Labels:       kustomize.toolkit.fluxcd.io/name=flux-system
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  <none>

Type:  kubernetes.io/dockerconfigjson

Data
====
.dockerconfigjson:  193 bytes

Finally, check that your app Pods are running:

kubectl -n blaster-dev get pods

NAME                           READY   STATUS    RESTARTS   AGE
blaster-app-67786c4c99-d9xxx   1/1     Running   0          13m
blaster-app-67786c4c99-hmhg4   1/1     Running   0          12m
blaster-db-0                   1/1     Running   0          10h

kubectl -n blaster-dev describe pod blaster-app-67786c4c99-d9xxx | grep -i "image" -A3

    Image:          registry.reids.net.au/games/blaster:dev-20251115.53
    Image ID:       registry.reids.net.au/games/blaster@sha256:a3b6ecd1dfa4c221c3b9d7a10ffd79d40bd24139e41064c8f7f2a89e5c6196ca
    Port:           3000/TCP (http)
    Host Port:      0/TCP (http)
    State:          Running
--
  Normal  Pulling    14m   kubelet            Pulling image "registry.reids.net.au/games/blaster:dev-20251115.53"
  Normal  Pulled     14m   kubelet            Successfully pulled image "registry.reids.net.au/games/blaster:dev-20251115.53" in 8.189s (8.189s including waiting). Image size: 363322908 bytes.
  Normal  Created    14m   kubelet            Created container: blaster
  Normal  Started    14m   kubelet            Started container blaster

kubectl get imagerepository -A

NAMESPACE     NAME               LAST SCAN              TAGS
flux-system   blaster-dev-repo   2025-11-15T16:59:58Z   36

kubectl -n flux-system describe imagerepository blaster-dev-repo

Name:         blaster-dev-repo
Namespace:    flux-system
Labels:       kustomize.toolkit.fluxcd.io/name=flux-system
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  reconcile.fluxcd.io/requestedAt: 2025-11-15T20:30:20.617306+08:00
API Version:  image.toolkit.fluxcd.io/v1beta2
Kind:         ImageRepository
Metadata:
  Creation Timestamp:  2025-11-15T10:57:09Z
  Finalizers:
    finalizers.fluxcd.io
  Generation:        1
  Resource Version:  52221205
  UID:               c4492ae9-ee72-4cd1-9c67-b06c0684d472
Spec:
  Exclusion List:
    ^.*\.sig$
  Image:     registry.reids.net.au/games/blaster
  Interval:  1m
  Provider:  generic
  Secret Ref:
    Name:  blaster-dev-registry
Status:
  Canonical Image Name:  registry.reids.net.au/games/blaster
  Conditions:
    Last Transition Time:     2025-11-15T16:42:56Z
    Message:                  successful scan: found 36 tags
    Observed Generation:      1
    Reason:                   Succeeded
    Status:                   True
    Type:                     Ready
  Last Handled Reconcile At:  2025-11-15T20:30:20.617306+08:00
  Last Scan Result:
    Latest Tags:
      prod-20251114.14
      prod-20251114.11
      dev-20251115.53
      dev-20251115.51
      dev-20251115.49
      dev-20251115.47
      dev-20251115.43
      dev-20251115.42
      dev-20251115.41
      dev-20251115.40
    Scan Time:  2025-11-15T17:00:58Z
    Tag Count:  36
  Observed Exclusion List:
    ^.*\.sig$
  Observed Generation:  1
Events:
  Type    Reason     Age                 From                        Message
  ----    ------     ----                ----                        -------
  Normal  Succeeded  29s (x269 over 6h)  image-reflector-controller  no new tags found, next scan in 1m0s

These outputs confirm that the latest tag for dev dev-20251115.53 is the image running on the pod registry.reids.net.au/games/blaster:dev-20251115.53.

Watching the pods shows new containers being created with the new image and existing containers terminating after a git push to the develop branch.

kubectl -n blaster-dev get pods -w

NAME                        READY   STATUS    RESTARTS   AGE
blaster-app-8dc864f-662j6   1/1     Running   0          117m
blaster-app-8dc864f-tqz8c   1/1     Running   0          117m
blaster-db-0                1/1     Running   0          9h
blaster-app-67786c4c99-d9xxx   0/1     Pending   0          0s
blaster-app-67786c4c99-d9xxx   0/1     Pending   0          0s
blaster-app-67786c4c99-d9xxx   0/1     ContainerCreating   0          0s
blaster-app-67786c4c99-d9xxx   0/1     ContainerCreating   0          0s
blaster-app-67786c4c99-d9xxx   0/1     Running             0          9s
blaster-app-67786c4c99-d9xxx   1/1     Running             0          30s
blaster-app-8dc864f-tqz8c      1/1     Terminating         0          120m
blaster-app-67786c4c99-hmhg4   0/1     Pending             0          0s
blaster-app-67786c4c99-hmhg4   0/1     Pending             0          0s
blaster-app-67786c4c99-hmhg4   0/1     ContainerCreating   0          0s
blaster-app-8dc864f-tqz8c      1/1     Terminating         0          120m
blaster-app-8dc864f-tqz8c      0/1     Completed           0          120m
blaster-app-67786c4c99-hmhg4   0/1     ContainerCreating   0          1s
blaster-app-8dc864f-tqz8c      0/1     Completed           0          120m
blaster-app-8dc864f-tqz8c      0/1     Completed           0          120m
blaster-app-67786c4c99-hmhg4   0/1     Running             0          2s
blaster-app-67786c4c99-hmhg4   1/1     Running             0          21s
blaster-app-8dc864f-662j6      1/1     Terminating         0          121m
blaster-app-8dc864f-662j6      1/1     Terminating         0          121m
blaster-app-8dc864f-662j6      0/1     Completed           0          121m
blaster-app-8dc864f-662j6      0/1     Completed           0          121m
blaster-app-8dc864f-662j6      0/1     Completed           0          121m

---

7.2.3 Git commit and push fluxgitops/flux-config

Need to add all the changed files.

git add .
git commit -m "update flux config with blaster dev"
git push

7.2.4 Confirm repo layout

Final repo layout:

tree -a -L 6 -I '.git|.DS_Store|node_modules|.next|dist'

.
├── .sops.yaml
└── clusters
    └── my-cluster
        ├── blaster
        │   ├── 00-namespace.yaml
        │   ├── dev
        │   │   ├── 20-blaster-images-dev.yaml
        │   │   ├── 30-image-automation.yaml
        │   │   ├── kustomization.yaml
        │   │   ├── secrets
        │   │   │   └── blaster-dev-registry.yaml
        │   │   └── source.yaml
        │   └── kustomization.yaml
        ├── flux-system
        │   ├── gotk-components.yaml
        │   ├── gotk-sync.yaml
        │   ├── kustomization.yaml
        │   └── secrets
        │       └── blaster-dev-registry.yaml
        └── kustomization.yaml

7.2.5 Reconcile and check

• Force reconciliation instead of waiting for the timer:

  flux reconcile source git flux-system -n flux-system \
    && flux reconcile kustomization flux-system -n flux-system --with-source

  flux reconcile kustomization blaster-dev -n flux-system --with-source

• Check that all sources are applied and ready:

  flux get sources git -n flux-system

NAME                     	REVISION             	SUSPENDED	READY	MESSAGE                                              
app-manifests            	main@sha1:2c4e91b2   	False    	True 	stored artifact for revision 'main@sha1:2c4e91b2'   	
blaster-dev              	develop@sha1:880afbae	False    	True 	stored artifact for revision 'develop@sha1:880afbae'	
cloudflare-app           	main@sha1:ff25bd44   	False    	True 	stored artifact for revision 'main@sha1:ff25bd44'   	
coach-app-dev            	dev@sha1:edd43eb1    	False    	True 	stored artifact for revision 'dev@sha1:edd43eb1'    	
flux-system              	main@sha1:975f688e   	False    	True 	stored artifact for revision 'main@sha1:975f688e'   	
origin-ca-issuer-upstream	v0.12.1@sha1:86d908ed	False    	True 	stored artifact for revision 'v0.12.1@sha1:86d908ed'    

• Check that all Kustomizations are applied and ready

  flux get kustomizations -n flux-system

NAME                       	REVISION             	SUSPENDED	READY	MESSAGE                                 
app-source-kustomization   	main@sha1:2c4e91b2   	False    	True 	Applied revision: main@sha1:2c4e91b2   	
blaster-dev                	develop@sha1:880afbae	False    	True 	Applied revision: develop@sha1:880afbae	
cloudflare-app             	main@sha1:ff25bd44   	False    	True 	Applied revision: main@sha1:ff25bd44   	
cloudflare-ns              	main@sha1:975f688e   	False    	True 	Applied revision: main@sha1:975f688e   	
coach-app-dev              	dev@sha1:edd43eb1    	False    	True 	Applied revision: dev@sha1:edd43eb1    	
flux-system                	main@sha1:975f688e   	False    	True 	Applied revision: main@sha1:975f688e   	
origin-ca-issuer-controller	v0.12.1@sha1:86d908ed	False    	True 	Applied revision: v0.12.1@sha1:86d908ed	
origin-ca-issuer-crds      	v0.12.1@sha1:86d908ed	False    	True 	Applied revision: v0.12.1@sha1:86d908ed	
origin-ca-issuer-ns        	main@sha1:975f688e   	False    	True 	Applied revision: main@sha1:975f688e   	
origin-ca-issuer-rbac      	v0.12.1@sha1:86d908ed	False    	True 	Applied revision: v0.12.1@sha1:86d908ed	

7.2.6 Wait for deployment to be ready

• Watch app namespace:

kubectl -n blaster-dev get pods,svc,ingress

NAME                            READY   STATUS    RESTARTS   AGE
pod/blaster-app-8dc864f-662j6   1/1     Running   0          56m
pod/blaster-app-8dc864f-tqz8c   1/1     Running   0          56m
pod/blaster-db-0                1/1     Running   0          8h

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/blaster-app   ClusterIP   10.50.119.110   <none>        80/TCP     10h
service/blaster-db    ClusterIP   10.50.77.169    <none>        5432/TCP   10h

NAME                                        CLASS   HOSTS                  ADDRESS     PORTS     AGE
ingress.networking.k8s.io/blaster-ingress   nginx   blaster.reids.net.au   10.50.1.5   80, 443   10h

• Check the blaster-dev namespace:

kubectl get all -n blaster-dev

NAME                            READY   STATUS    RESTARTS   AGE
pod/blaster-app-8dc864f-662j6   1/1     Running   0          57m
pod/blaster-app-8dc864f-tqz8c   1/1     Running   0          56m
pod/blaster-db-0                1/1     Running   0          8h

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/blaster-app   ClusterIP   10.50.119.110   <none>        80/TCP     10h
service/blaster-db    ClusterIP   10.50.77.169    <none>        5432/TCP   10h

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/blaster-app   2/2     2            2           10h

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/blaster-app-587cf676d7   0         0         0       3h9m
replicaset.apps/blaster-app-5dcc9b9558   0         0         0       4h40m
replicaset.apps/blaster-app-679b956765   0         0         0       4h37m
replicaset.apps/blaster-app-6964ccd7cd   0         0         0       172m
replicaset.apps/blaster-app-77bc8bbdfc   0         0         0       3h3m
replicaset.apps/blaster-app-7d576d98d4   0         0         0       4h33m
replicaset.apps/blaster-app-84dc44c4cb   0         0         0       114m
replicaset.apps/blaster-app-8797c7d67    0         0         0       4h31m
replicaset.apps/blaster-app-8dc864f      2         2         2       57m
replicaset.apps/blaster-app-97887fc4d    0         0         0       3h6m
replicaset.apps/blaster-app-c946b67fd    0         0         0       156m

NAME                          READY   AGE
statefulset.apps/blaster-db   1/1     10h

• Check the config map:

kubectl -n blaster-dev get cm

NAME                 DATA   AGE
blaster-app-config   1      10h
kube-root-ca.crt     1      11h

7.2.7 Confirm everything is healthy

kubectl -n flux-system get secret blaster-dev-registry
flux get image repository -A
flux get image policy -A
flux get image update -A

NAME                   TYPE                             DATA   AGE
blaster-dev-registry   kubernetes.io/dockerconfigjson   1      3h10m
NAMESPACE  	NAME            	LAST SCAN                	SUSPENDED	READY	MESSAGE                        
flux-system	blaster-dev-repo	2025-11-15T23:38:48+08:00	False    	True 	successful scan: found 35 tags	
NAMESPACE  	NAME              	LATEST IMAGE                                       	READY	MESSAGE                                                                                                                                               
flux-system	blaster-dev-policy	registry.reids.net.au/games/blaster:dev-20251115.51	True 	Latest image tag for registry.reids.net.au/games/blaster resolved to dev-20251115.51 (previously registry.reids.net.au/games/blaster:dev-20251115.49)	
NAMESPACE  	NAME                  	LAST RUN                 	SUSPENDED	READY	MESSAGE               
flux-system	blaster-dev-automation	2025-11-15T23:38:28+08:00	False    	True 	repository up-to-date

---

8.1 End-to-end flow

8.1.1 Local dev on Mac

1. Create a feature branch from develop, for example feature/shooting-speed.
2. Run npm run dev locally with .env.local and a local database.
3. Commit code changes (keeping secrets out of Git).
4. Push the feature branch and open a merge request into develop.

8.2.2 Deploy to k8s dev

1. Merge the feature branch into develop once tests and review pass.
2. GitLab CI on develop:
   • Runs lint and test.
   • Builds registry.reids.net.au/games/blaster:dev-YYYYMMDD.N with Kaniko.
3. k8s/dev/deployment.yaml uses the new dev- automatically.
4. Flux blaster-dev Kustomization notices the change on develop and syncs k8s/dev.
5. Test Blaster at https://blaster.reids.net.au inside the network.

8.3.3 Promote to k8s prod

Once you are happy with dev:

1. Create a merge request from develop into main.
2. In that MR:
   • Ensure k8s/prod/deployment.yaml uses the prod- image tag you want to promote.
   • Optionally adjust prod-only settings (replicas, resources, ingress host blaster.muppit.au).
3. Merge into main once the pipeline passes.
4. GitLab CI on main:
   • Runs lint and test.
   • Builds registry.reids.net.au/games/blaster:prod-YYYYMMDD.N with Kaniko.
5. Flux blaster-prod Kustomization sees the new commit on main and syncs k8s/prod.
6. Blaster is updated on https://blaster.muppit.au via Cloudflare tunnel.

Production changes therefore require:

• A commit on main,
• That passed the pipeline,
• And went through a merge request.

---

9.1 Guardrails and habits

warning

Do not kubectl apply directly in the blaster namespace. Treat it as Git-only. If you must hotfix, commit to the right branch and let Flux sync.

Recommended guardrails:

• Protect the blaster namespaces with NetworkPolicies so they are only reachable via ingress.
• Add health probes and sensible resource requests/limits in both dev and prod.
• Configure GitLab so:
  • main and develop are protected.
  • Merge requests are required.
  • Pipelines must succeed before merge.
• Consider semantic versions (for example v0.1.0) for prod tags once the game stabilises.

---

10.1 Verification checklist

• games/blaster has k8s/dev and k8s/prod overlays.
• develop branch deploys only to blaster-dev via Flux.
• main branch deploys only to blaster-prod via Flux.
• GitLab CI builds images on develop and main and pushes them to the registry.
• Image tags in k8s/dev and k8s/prod are explicit, not latest.
• No-one runs kubectl apply by hand in the Blaster namespaces.
• Manual tests happen on blaster.reids.net.au before merging develop into main.