Skip to main content

Prod GitOps via Flux runbook

This runbook provides end-to-end instructions on how to deploy a game onto the cluster using GitOps automation via Flux from Dev to Prod.

  • This runbook has been tested on:
    • Kubernetes version: v1.31.9
    • OS-Image: Ubuntu 24.04.2 LTS
    • Kernel version: 6.8.0-62-generic
    • Container runtime: containerd://2.0.5
info

Code moves from local dev to k8s dev to k8s prod using GitLab CI (with Kaniko), FluxCD and merge requests. Prod is only updated from tested commits on main. Images are automatically built and dynamic image tags used.

Blaster GitOps series​

  1. Blaster GitOps summary
  2. Blaster repo and branches
  3. Dockerfile & GitLab CI
  4. Clerk authentication & user setup
  5. Google OAuth for Clerk
  6. Blaster prep for automation
  7. Dev app k8s manifests
  8. Dev flux sources & Kustomizations
  9. Dev image automation
  10. Dev SOPS & age
  11. Dev verification & troubleshooting
  12. Dev full runbook
  13. Prod overview
  14. Prod app k8s manifests and deployment
  15. Prod Flux GitOps and image automation
  16. Prod Cloudflare, Origin CA and tunnel routing
  17. Prod full runbook - you are here
  18. Post development branches

Blaster repo​

cd ~/Projects/blaster
git pull
Already up to date.
git status
On branch develop
Your branch is up to date with 'origin/develop'.

nothing to commit, working tree clean
tree -a -L 6 -I '.git|.DS_Store|node_modules|.next|dist'
.
├── .dockerignore
├── .gitlab-ci.yml
├── .sops.yaml
├── app
├── database
├── db
├── Dockerfile
├── k8s
│   ├── dev
│   │   ├── 10-secret-db.enc.yaml
│   │   ├── 20-db-statefulset.yaml
│   │   ├── 30-secret-app.enc.yaml
│   │   ├── 40-app-config.yaml
│   │   ├── 50-app-deployment.yaml
│   │   ├── 60-ingress.yaml
│   │   └── kustomization.yaml
│   └── prod
│       ├── 10-secret-db.enc.yaml
│       ├── 20-db-statefulset.yaml
│       ├── 30-secret-app.enc.yaml
│       ├── 40-app-config.yaml
│       ├── 50-app-deployment.yaml
│       ├── 60-ingress.yaml
│       ├── 70-secret-cfapi-token.enc.yaml
│       ├── 80-originissuer.yaml
│       ├── 90-certificate-blaster.yaml
│       └── kustomization.yaml
├── lib
└── package.json

Kustomization​

# k8s/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: blaster
resources:
  - 10-secret-db.enc.yaml
  - 20-db-statefulset.yaml
  - 30-secret-app.enc.yaml
  - 40-app-config.yaml
  - 50-app-deployment.yaml
  - 60-ingress.yaml
  - 70-secret-cfapi-token.enc.yaml
  - 80-originissuer.yaml
  - 90-certificate-blaster.yaml

SOPS policy​

note

A SOPS policy already exists at repo root. It was added during dev build.

Create manifests before SOPS encryption​

Create the files under k8s/prod/.

mkdir -p k8s/prod

Shared DB secret​

Before SOPS encryption

---
# k8s/prod/10-secret-db.enc.yaml
apiVersion: v1
kind: Secret
metadata:
  name: blaster-db-secret
  namespace: blaster
type: Opaque
stringData:
  POSTGRES_DB: blaster_game
  POSTGRES_USER: blaster_user
  POSTGRES_PASSWORD: "REPLACE_WITH_STRONG_PASSWORD"

This is the single source of truth for DB credentials. The DB and app both consume this Secret via envFrom.

note

Edit via sops so secrets stay encrypted in Git.

PostgreSQL StatefulSet and Service​

---
# k8s/prod/20-db-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: blaster-db
  namespace: blaster
spec:
  serviceName: blaster-db
  replicas: 1
  selector:
    matchLabels:
      app: blaster-db
  template:
    metadata:
      labels:
        app: blaster-db
    spec:
      containers:
        - name: postgres
          image: postgres:15.8
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5432
              name: postgres
          envFrom:
            - secretRef:
                name: blaster-db-secret
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
          readinessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
            initialDelaySeconds: 10
            periodSeconds: 10
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB"
            initialDelaySeconds: 30
            periodSeconds: 30
          resources:
            requests:
              cpu: "200m"
              memory: "512Mi"
            limits:
              cpu: "1"
              memory: "2Gi"
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi
---
apiVersion: v1
kind: Service
metadata:
  name: blaster-db
  namespace: blaster
spec:
  type: ClusterIP
  selector:
    app: blaster-db
  ports:
    - name: postgres
      port: 5432
      targetPort: 5432

App secrets and config​

Before SOPS encryption

---
# k8s/prod/30-secret-app.enc.yaml
apiVersion: v1
kind: Secret
metadata:
  name: blaster-app-secret
  namespace: blaster
type: Opaque
stringData:
  CLERK_SECRET_KEY: "REPLACE_WITH_REAL_CLERK_SECRET_KEY"
---
# k8s/prod/40-app-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: blaster-app-config
  namespace: blaster
data:
  NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY: "REPLACE_WITH_PUBLIC_KEY"

App Deployment and Service​

The image will automatically get updated

---
# k8s/prod/50-app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: blaster-app
  namespace: blaster
spec:
  replicas: 3
  selector:
    matchLabels:
      app: blaster-app
  template:
    metadata:
      labels:
        app: blaster-app
    spec:
      imagePullSecrets:
        - name: blaster-prod-registry
      # Run database migrations before starting the app
      initContainers:
        - name: migrate
          image: registry.reids.net.au/games/blaster:prod-20251114.14 # {"$imagepolicy": "flux-system:blaster-prod-policy"}
          imagePullPolicy: IfNotPresent
          command: ["npm", "run", "migrate"]
          envFrom:
            - secretRef:
                name: blaster-db-secret # DB name/user/password
          env:
            - name: POSTGRES_HOST
              value: "blaster-db"
            - name: POSTGRES_PORT
              value: "5432"
      containers:
        - name: blaster
          image: registry.reids.net.au/games/blaster:prod-20251114.14 # {"$imagepolicy": "flux-system:blaster-prod-policy"}
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http
          envFrom:
            - secretRef:
                name: blaster-db-secret
            - secretRef:
                name: blaster-app-secret
            - configMapRef:
                name: blaster-app-config
          env:
            - name: POSTGRES_HOST
              value: "blaster-db"
            - name: POSTGRES_PORT
              value: "5432"
          readinessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 15
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 60
            periodSeconds: 20
          resources:
            requests:
              cpu: "300m"
              memory: "768Mi"
            limits:
              cpu: "2"
              memory: "2Gi"
---
apiVersion: v1
kind: Service
metadata:
  name: blaster-app
  namespace: blaster
spec:
  type: ClusterIP
  selector:
    app: blaster-app
  ports:
    - name: http
      port: 80
      targetPort: 3000

Ingress​

blaster.muppit.au

---
# k8s/prod/60-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blaster-ingress
  namespace: blaster
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  tls:
    - hosts:
        - blaster.muppit.au
      secretName: blaster-muppit-au-tls
  rules:
    - host: "blaster.muppit.au"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: blaster-app
                port:
                  number: 80

Secret CFAPI token​

Reuse the same Cloudflare API token value as your cloudflare app repo; this copy must live in blaster so OriginIssuer can issue a cert in this namespace.

# k8s/prod/70-secret-cfapi-token.enc.yaml
apiVersion: v1
kind: Secret
metadata:
  name: cfapi-token
  namespace: blaster
type: Opaque
stringData:
  key: PASTE_YOUR_API_TOKEN

Origin issuer​

# k8s/prod/80-originissuer.yaml
apiVersion: cert-manager.k8s.cloudflare.com/v1
kind: OriginIssuer
metadata:
  name: cf-origin
  namespace: blaster
spec:
  requestType: OriginECC
  auth:
    tokenRef:
      name: cfapi-token
      key: key

Certificate issuer​

Issue an Origin CA certificate in the same namespace as your Ingress (blaster).

# k8s/prod/90-certificate-blaster.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: blaster-muppit-au-origin
  namespace: blaster
spec:
  secretName: blaster-muppit-au-tls
  dnsNames:
    - blaster.muppit.au
  issuerRef:
    group: cert-manager.k8s.cloudflare.com
    kind: OriginIssuer
    name: cf-origin

6.1.3 SOPS encryption​

note

The SOPS_AGE_KEY_FILE was set during dev.

Encrypt the Secrets:

sops -e -i k8s/prod/10-secret-db.enc.yaml 
sops -e -i k8s/prod/30-secret-app.enc.yaml
sops -e -i k8s/prod/70-secret-cfapi-token.enc.yaml

Verify they’re encrypted:

head -n 20 k8s/prod/10-secret-db.enc.yaml
head -n 20 k8s/prod/30-secret-app.enc.yaml
head -n 20 k8s/prod/70-secret-cfapi-token.enc.yaml
# k8s/prod/10-secret-db.enc.yaml
apiVersion: v1
kind: Secret
metadata:
    name: blaster-db-secret
    namespace: blaster
type: Opaque
stringData:
    POSTGRES_DB: ENC[AES256_GCM,data:type:str]
    POSTGRES_USER: ENC[AES256_GCM,data:type:str]
    POSTGRES_PASSWORD: ENC[AES256_GCM,data:type:str]
sops:
    age:
        - recipient: age...
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
# k8s/prod/30-secret-app.enc.yaml
apiVersion: v1
kind: Secret
metadata:
    name: blaster-app-secret
    namespace: blaster
type: Opaque
stringData:
    CLERK_SECRET_KEY: ENC[AES256_GCM,data:type:str]
sops:
    age:
        - recipient: age1...
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
# k8s/prod/70-secret-cfapi-token.enc.yaml
apiVersion: v1
kind: Secret
metadata:
    name: cfapi-token
    namespace: blaster
type: Opaque
stringData:
    key: ENC[AES256_GCM,data:type:str]
sops:
    age:
        - recipient: age1...
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
git add .
git commit -m "Added prod config"
[develop 0753f72] Added prod config
```bash
git push
Enumerating objects: 13, done.
Counting objects: 100% (13/13), done.
Delta compression using up to 24 threads
Compressing objects: 100% (11/11), done.
Writing objects: 100% (11/11), 4.28 KiB | 4.28 MiB/s, done.
Total 11 (delta 1), reused 0 (delta 0), pack-reused 0 (from 0)
remote: 
remote: To create a merge request for develop, visit:
remote:   https://gitlab.reids.net.au/games/blaster/-/merge_requests/new?merge_request%5Bsource_branch%5D=develop
remote: 
To https://gitlab.reids.net.au/games/blaster.git
   80ee521..0753f72  develop -> develop
git pull
Already up to date.
kubectl -n blaster get certificate blaster-muppit-au-origin
NAME                       READY   SECRET                  AGE
blaster-muppit-au-origin   True    blaster-muppit-au-tls   3m45s
kubectl -n blaster get secret blaster-muppit-au-tls
NAME                    TYPE                DATA   AGE
blaster-muppit-au-tls   kubernetes.io/tls   3      4m3s
kubectl get all -n blaster
NAME                               READY   STATUS    RESTARTS   AGE
pod/blaster-app-645d7dbbf5-4c7pk   1/1     Running   0          7m35s
pod/blaster-app-645d7dbbf5-bq98x   1/1     Running   0          7m56s
pod/blaster-app-645d7dbbf5-glqj9   1/1     Running   0          8m26s
pod/blaster-db-0                   1/1     Running   0          64m

NAME                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/blaster-app   ClusterIP   10.50.96.236   <none>        80/TCP     64m
service/blaster-db    ClusterIP   10.50.109.50   <none>        5432/TCP   64m

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/blaster-app   3/3     3            3           64m

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/blaster-app-645d7dbbf5   3         3         3       8m26s
replicaset.apps/blaster-app-796cd7db74   0         0         0       64m
replicaset.apps/blaster-app-86c777c59b   0         0         0       61m

NAME                          READY   AGE
statefulset.apps/blaster-db   1/1     64m
kubectl get all -n blaster -o wide
NAME                               READY   STATUS    RESTARTS   AGE     IP              NODE       NOMINATED NODE   READINESS GATES
pod/blaster-app-645d7dbbf5-4c7pk   1/1     Running   0          7m44s   10.50.153.176   k8s-w-p1   <none>           <none>
pod/blaster-app-645d7dbbf5-bq98x   1/1     Running   0          8m5s    10.50.153.181   k8s-w-p1   <none>           <none>
pod/blaster-app-645d7dbbf5-glqj9   1/1     Running   0          8m35s   10.50.153.179   k8s-w-p1   <none>           <none>
pod/blaster-db-0                   1/1     Running   0          64m     10.50.153.161   k8s-w-p1   <none>           <none>

NAME                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE   SELECTOR
service/blaster-app   ClusterIP   10.50.96.236   <none>        80/TCP     64m   app=blaster-app
service/blaster-db    ClusterIP   10.50.109.50   <none>        5432/TCP   64m   app=blaster-db

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                                 SELECTOR
deployment.apps/blaster-app   3/3     3            3           64m   blaster      registry.reids.net.au/games/blaster:prod-20251116.62   app=blaster-app

NAME                                     DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES                                                 SELECTOR
replicaset.apps/blaster-app-645d7dbbf5   3         3         3       8m35s   blaster      registry.reids.net.au/games/blaster:prod-20251116.62   app=blaster-app,pod-template-hash=645d7dbbf5
replicaset.apps/blaster-app-796cd7db74   0         0         0       64m     blaster      registry.reids.net.au/games/blaster:prod-20251114.14   app=blaster-app,pod-template-hash=796cd7db74
replicaset.apps/blaster-app-86c777c59b   0         0         0       61m     blaster      registry.reids.net.au/games/blaster:prod-20251116.58   app=blaster-app,pod-template-hash=86c777c59b

NAME                          READY   AGE   CONTAINERS   IMAGES
statefulset.apps/blaster-db   1/1     64m   postgres     postgres:15.8
kubectl -n blaster get ingress                     
NAME              CLASS    HOSTS               ADDRESS     PORTS     AGE
blaster-ingress   <none>   blaster.muppit.au   10.50.1.5   80, 443   4m48s
git pull --rebase
remote: Enumerating objects: 16, done.
remote: Counting objects: 100% (16/16), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 8 (delta 4), reused 0 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (8/8), 832 bytes | 166.00 KiB/s, done.
From https://gitlab.reids.net.au/games/blaster
   eb3c88c..e92d00a  main       -> origin/main
Already up to date.
git checkout -b security/hardening-auth
Switched to a new branch 'security/hardening-auth'
git push -u origin security/hardening-auth
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: 
remote: To create a merge request for security/hardening-auth, visit:
remote:   https://gitlab.reids.net.au/games/blaster/-/merge_requests/new?merge_request%5Bsource_branch%5D=security%2Fhardening-auth
remote: 
To https://gitlab.reids.net.au/games/blaster.git
 * [new branch]      security/hardening-auth -> security/hardening-auth
branch 'security/hardening-auth' set up to track 'origin/security/hardening-auth'.
git status
On branch security/hardening-auth
Your branch is up to date with 'origin/security/hardening-auth'.

nothing to commit, working tree clean

Cloudflare repo​

cd ~/Projects/cloudflare 
git pull
Already up to date.
---
# k8s/prod/40-configmap-cloudflared.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: cloudflared
  namespace: cloudflare
data:
  config.yaml: |
    tunnel: 3183eb59-d124-483f-90a1-3409e41b1d69
    credentials-file: /etc/cloudflared/creds/credentials.json
    metrics: 0.0.0.0:2000

    ingress:
      # (optional) Block WP admin to public
      - hostname: muppit.au
        path: ^/(wp-login\.php|wp-admin(?:/.*)?$)
        service: http_status:403

      # Send to ingress-nginx over TLS and verify with Origin CA
      - hostname: muppit.au
        service: https://ingress-nginx-controller.ingress-nginx.svc.cluster.local:443
        originRequest:
          originServerName: muppit.au
          caPool: /etc/cloudflared/certs/origin_ca.pem

      - hostname: blaster.muppit.au
        service: https://ingress-nginx-controller.ingress-nginx.svc.cluster.local:443
        originRequest:
          originServerName: blaster.muppit.au
          caPool: /etc/cloudflared/certs/origin_ca.pem

      # Catch-all
      - service: http_status:404
git add .
git commit -m "Added blaster to configmap"
[main 4d03596] Added blaster to configmap
 1 file changed, 6 insertions(+)
git push
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 24 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 459 bytes | 459.00 KiB/s, done.
Total 5 (delta 2), reused 0 (delta 0), pack-reused 0 (from 0)
To https://gitlab.reids.net.au/muppit-apps/cloudflare.git
   ff25bd4..4d03596  main -> main
```bash
flux reconcile source git origin-ca-issuer-upstream -n flux-system && \
flux reconcile kustomization origin-ca-issuer-crds -n flux-system && \
flux reconcile kustomization origin-ca-issuer-rbac -n flux-system && \
flux reconcile kustomization origin-ca-issuer-controller -n flux-system
► annotating GitRepository origin-ca-issuer-upstream in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision v0.12.1@sha1:86d908eda6c91815557c04b7f3e871ca4f0a0cce
► annotating Kustomization origin-ca-issuer-crds in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ applied revision v0.12.1@sha1:86d908eda6c91815557c04b7f3e871ca4f0a0cce
► annotating Kustomization origin-ca-issuer-rbac in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ applied revision v0.12.1@sha1:86d908eda6c91815557c04b7f3e871ca4f0a0cce
► annotating Kustomization origin-ca-issuer-controller in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ applied revision v0.12.1@sha1:86d908eda6c91815557c04b7f3e871ca4f0a0cce

Flux-config repo​

cd ~/Projects/flux-config 
git pull
Already up to date.
git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
tree -a -L 7 -I '.git|.DS_Store|node_modules|.next|dist'
├── .sops.yaml
└── clusters
    └── my-cluster
        ├── blaster
        │   ├── 00-namespace.yaml
        │   ├── 10-namespace-prod.yaml
        │   ├── dev
        │   │   ├── 20-blaster-images-dev.yaml
        │   │   ├── 30-image-automation.yaml
        │   │   ├── kustomization.yaml
        │   │   ├── secrets
        │   │   │   └── blaster-dev-registry.yaml
        │   │   └── source.yaml
        │   ├── kustomization.yaml
        │   └── prod
        │       ├── 20-blaster-images-prod.yaml
        │       ├── 30-image-automation.yaml
        │       ├── kustomization.yaml
        │       ├── secrets
        │       │   └── blaster-prod-registry.yaml
        │       └── source.yaml
        ├── flux-system
        │   ├── gotk-components.yaml
        │   ├── gotk-sync.yaml
        │   ├── kustomization.yaml
        │   └── secrets
        │       ├── blaster-dev-registry.yaml
        │       └── blaster-prod-registry.yaml
        └── kustomization.yaml
mkdir -p clusters/my-cluster/blaster/prod/secrets

Kustomization​

# clusters/my-cluster/blaster/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ./00-namespace.yaml
  - ./10-namespace-prod.yaml
  # Dev
  - ./dev/source.yaml
  - ./dev/kustomization.yaml
  - ./dev/20-blaster-images-dev.yaml
  - ./dev/30-image-automation.yaml
  - ./dev/secrets/blaster-dev-registry.yaml
  # Prod
  - ./prod/source.yaml
  - ./prod/kustomization.yaml
  - ./prod/20-blaster-images-prod.yaml
  - ./prod/30-image-automation.yaml
  - ./prod/secrets/blaster-prod-registry.yaml

Namespace​

# clusters/my-cluster/blaster/10-namespace-prod.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: blaster
  labels:
    name: blaster

Source​

# clusters/my-cluster/blaster/prod/source.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: blaster-prod
  namespace: flux-system
spec:
  interval: 1m
  timeout: 60s
  url: ssh://git-ssh.reids.net.au/games/blaster.git
  ref:
    branch: main
  secretRef:
    name: flux-ssh-auth

Kustomization​

# clusters/my-cluster/blaster/prod/kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: blaster-prod
  namespace: flux-system
spec:
  interval: 1m
  path: ./k8s/prod
  prune: true
  sourceRef:
    kind: GitRepository
    name: blaster-prod
  wait: true
  timeout: 5m
  decryption:
    provider: sops
    secretRef:
      name: sops-age

Image repository and policy​

# clusters/my-cluster/blaster/prod/20-blaster-images-prod.yaml
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: blaster-prod-repo
  namespace: flux-system
spec:
  image: registry.reids.net.au/games/blaster
  interval: 1m
  secretRef:
    name: blaster-prod-registry
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: blaster-prod-policy
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: blaster-prod-repo
  filterTags:
    pattern: '^prod-(?P<date>[0-9]{8})\.(?P<build>[0-9]+)$'
    extract: '$date$build'
  policy:
    numerical:
      order: asc

Image automation​

# clusters/my-cluster/blaster/prod/30-image-automation.yaml
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata:
  name: blaster-prod-automation
  namespace: flux-system
spec:
  interval: 1m
  sourceRef:
    kind: GitRepository
    name: blaster-prod
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        name: FluxCD
        email: andrew@reids.net.au
      messageTemplate: '{{range .Changed.Changes}}{{print .OldValue}} -> {{println .NewValue}}{{end}} [skip ci]'
    push:
      branch: main
  update:
    strategy: Setters
    path: ./k8s/prod

Secret for registry in blaster​

kubectl -n blaster create secret docker-registry blaster-prod-registry \
  --docker-server=registry.reids.net.au \
  --docker-username='blaster-prod' \
  --docker-password='REDACTED' \
  --docker-email='andrew@reids.net.au' \
  --dry-run=client -o yaml \
  > clusters/my-cluster/blaster/prod/secrets/blaster-prod-registry.yaml
sops -e -i clusters/my-cluster/blaster/prod/secrets/blaster-prod-registry.yaml

Flux-system kustomization​

clusters/my-cluster/flux-system/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
  - ./secrets/blaster-dev-registry.yaml
  - ./secrets/blaster-prod-registry.yaml

Secret for registry in flux-system​

flux-config % kubectl -n flux-system create secret docker-registry blaster-prod-registry \
  --docker-server=registry.reids.net.au \
  --docker-username='blaster-prod' \
  --docker-password='REDACTED' \
  --docker-email='andrew@reids.net.au' \
  --dry-run=client -o yaml \
  > clusters/my-cluster/flux-system/secrets/blaster-prod-registry.yaml
sops -e -i clusters/my-cluster/flux-system/secrets/blaster-prod-registry.yaml

Commit to Git​

git add .
git commit -m "Added blaster prod"
git push
flux reconcile source git flux-system -n flux-system
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision main@sha1:56e96573b106eef6460ec8f79fee666b5b04fe51
flux reconcile kustomization flux-system -n flux-system
► annotating Kustomization flux-system in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ applied revision main@sha1:56e96573b106eef6460ec8f79fee666b5b04fe51
kubectl -n blaster get secret blaster-prod-registry
NAME                    TYPE                             DATA   AGE
blaster-prod-registry   kubernetes.io/dockerconfigjson   1      2m5s

Cloudflare tunnel and dashboard​

Initial reachability test (failed)​

info

404 response as the blaster.muppit.au has not been added to the tunnel yet.

curl -k https://blaster.muppit.au/ -I
HTTP/2 404

Existing tunnel ID​

:::The existing tunnel ID is required for the target when creating a new DNS entry.

cloudflared tunnel list
You can obtain more detailed information for each tunnel with `cloudflared tunnel info <name/uuid>`
ID         NAME       CREATED              CONNECTIONS

Cloudflare dashboard​

  1. Login to https://https://dash.cloudflare.com
  2. Add a new DNS entry
    1. Type: cname
    2. Name: blaster
    3. Target: TUNNELID.cfargotunnel.com
    4. Proxy status: proxied

Restart clouldfared tunnel​

The issue:

  • The ConfigMap was changed in Git.
  • Flux reconciled and updated the ConfigMap object in the cluster.
  • The cloudflared Pod keeps running with the old config in memory.
  • Kubernetes does not automatically restart Pods when a ConfigMap changes.
  • cloudflared does not re-read its config unless it is restarted (or started with a special watch behaviour), so it keeps serving the old routing until a forced rollout.

Flux:

  • Synced Git → cluster (ConfigMap updated).
  • Does not restart the Deployment (because nothing in the Pod template changed).

From Kubernetes point of view, the Deployment spec (containers, env, volumes, etc.) are unchanged, so there is no reason to roll a new ReplicaSet. No new RS = no new Pods = old config still in memory.

Manual restart:

kubectl -n cloudflare rollout restart deploy/cloudflared
kubectl -n cloudflare rollout status deploy/cloudflared
deployment.apps/cloudflared restarted
Waiting for deployment "cloudflared" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "cloudflared" rollout to finish: 1 old replicas are pending termination...
deployment "cloudflared" successfully rolled out

Reachability test after restart (passed)​

curl -k https://blaster.muppit.au/ -I
HTTP/2 200