Skip to main content

Prod GitOps and image automation

info

This runbook lives in the flux-config repo. It wires FluxCD to pull manifests from games/blaster, decrypt SOPS secrets, and auto-update the prod Blaster image based on prod-YYYYMMDD.BUILD tags.

Blaster GitOps series​

  1. Blaster GitOps summary
  2. Blaster repo and branches
  3. Dockerfile & GitLab CI
  4. Clerk authentication & user setup
  5. Google OAuth for Clerk
  6. Blaster prep for automation
  7. Dev app k8s manifests
  8. Dev flux sources & Kustomizations
  9. Dev image automation
  10. Dev SOPS & age
  11. Dev verification & troubleshooting
  12. Dev full runbook
  13. Prod overview
  14. Prod app k8s manifests and deployment
  15. Prod Flux GitOps and image automation - you are here
  16. Prod Cloudflare, Origin CA and tunnel routing
  17. Prod full runbook
  18. Post development branches

1. Repo state and layout​

1.1 Confirm repo is clean​

cd ~/Projects/flux-config 
git pull
Already up to date.
git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

1.2 Inspect Flux config tree​

tree -a -L 7 -I '.git|.DS_Store|node_modules|.next|dist'
├── .sops.yaml
└── clusters
    └── my-cluster
        ├── blaster
        │   ├── 00-namespace.yaml
        │   ├── 10-namespace-prod.yaml
        │   ├── dev
        │   │   ├── 20-blaster-images-dev.yaml
        │   │   ├── 30-image-automation.yaml
        │   │   ├── kustomization.yaml
        │   │   ├── secrets
        │   │   │   └── blaster-dev-registry.yaml
        │   │   └── source.yaml
        │   ├── kustomization.yaml
        │   └── prod
        │       ├── 20-blaster-images-prod.yaml
        │       ├── 30-image-automation.yaml
        │       ├── kustomization.yaml
        │       ├── secrets
        │       │   └── blaster-prod-registry.yaml
        │       └── source.yaml
        ├── flux-system
        │   ├── gotk-components.yaml
        │   ├── gotk-sync.yaml
        │   ├── kustomization.yaml
        │   └── secrets
        │       ├── blaster-dev-registry.yaml
        │       └── blaster-prod-registry.yaml
        └── kustomization.yaml
mkdir -p clusters/my-cluster/blaster/prod/secrets

2. Blaster-wide Kustomization​

2.1 clusters/my-cluster/blaster/kustomization.yaml​

# clusters/my-cluster/blaster/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ./00-namespace.yaml
  - ./10-namespace-prod.yaml
  # Dev
  - ./dev/source.yaml
  - ./dev/kustomization.yaml
  - ./dev/20-blaster-images-dev.yaml
  - ./dev/30-image-automation.yaml
  - ./dev/secrets/blaster-dev-registry.yaml
  # Prod
  - ./prod/source.yaml
  - ./prod/kustomization.yaml
  - ./prod/20-blaster-images-prod.yaml
  - ./prod/30-image-automation.yaml
  - ./prod/secrets/blaster-prod-registry.yaml

3. Namespace for prod​

3.1 Prod namespace​

# clusters/my-cluster/blaster/10-namespace-prod.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: blaster
  labels:
    name: blaster

4. Source and Kustomization for prod​

4.1 GitRepository – games/blaster on main​

# clusters/my-cluster/blaster/prod/source.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: blaster-prod
  namespace: flux-system
spec:
  interval: 1m
  timeout: 60s
  url: ssh://git-ssh.reids.net.au/games/blaster.git
  ref:
    branch: main
  secretRef:
    name: flux-ssh-auth

4.2 Kustomization – apply k8s/prod with SOPS​

# clusters/my-cluster/blaster/prod/kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: blaster-prod
  namespace: flux-system
spec:
  interval: 1m
  path: ./k8s/prod
  prune: true
  sourceRef:
    kind: GitRepository
    name: blaster-prod
  wait: true
  timeout: 5m
  decryption:
    provider: sops
    secretRef:
      name: sops-age

5. Image automation for prod​

5.1 ImageRepository and ImagePolicy​

# clusters/my-cluster/blaster/prod/20-blaster-images-prod.yaml
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: blaster-prod-repo
  namespace: flux-system
spec:
  image: registry.reids.net.au/games/blaster
  interval: 1m
  secretRef:
    name: blaster-prod-registry
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: blaster-prod-policy
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: blaster-prod-repo
  filterTags:
    pattern: '^prod-(?P<date>[0-9]{8})\.(?P<build>[0-9]+)$'
    extract: '$date$build'
  policy:
    numerical:
      order: asc

5.2 ImageUpdateAutomation​

# clusters/my-cluster/blaster/prod/30-image-automation.yaml
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata:
  name: blaster-prod-automation
  namespace: flux-system
spec:
  interval: 1m
  sourceRef:
    kind: GitRepository
    name: blaster-prod
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        name: FluxCD
        email: andrew@reids.net.au
      messageTemplate: '{{range .Changed.Changes}}{{print .OldValue}} -> {{println .NewValue}}{{end}} [skip ci]'
    push:
      branch: main
  update:
    strategy: Setters
    path: ./k8s/prod

6. Registry Secrets​

6.1 Secret for registry in blaster namespace​

kubectl -n blaster create secret docker-registry blaster-prod-registry   --docker-server=registry.reids.net.au   --docker-username='blaster-prod'   --docker-password='REDACTED'   --docker-email='andrew@reids.net.au'   --dry-run=client -o yaml   > clusters/my-cluster/blaster/prod/secrets/blaster-prod-registry.yaml
sops -e -i clusters/my-cluster/blaster/prod/secrets/blaster-prod-registry.yaml

6.2 Flux-system Kustomization​

clusters/my-cluster/flux-system/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
  - ./secrets/blaster-dev-registry.yaml
  - ./secrets/blaster-prod-registry.yaml

6.3 Secret for registry in flux-system​

flux-config % kubectl -n flux-system create secret docker-registry blaster-prod-registry   --docker-server=registry.reids.net.au   --docker-username='blaster-prod'   --docker-password='REDACTED'   --docker-email='andrew@reids.net.au'   --dry-run=client -o yaml   > clusters/my-cluster/flux-system/secrets/blaster-prod-registry.yaml
sops -e -i clusters/my-cluster/flux-system/secrets/blaster-prod-registry.yaml

7. Commit and reconcile​

7.1 Commit and push Flux config​

git add .
git commit -m "Added blaster prod"
git push

7.2 Reconcile Flux​

flux reconcile source git flux-system -n flux-system
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision main@sha1:56e96573b106eef6460ec8f79fee666b5b04fe51
flux reconcile kustomization flux-system -n flux-system
► annotating Kustomization flux-system in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ applied revision main@sha1:56e96573b106eef6460ec8f79fee666b5b04fe51
kubectl -n blaster get secret blaster-prod-registry
NAME                    TYPE                             DATA   AGE
blaster-prod-registry   kubernetes.io/dockerconfigjson   1      2m5s

8. Verification checklist​

  • GitRepository/blaster-prod and Kustomization/blaster-prod are Ready=True.
  • ImageRepository/blaster-prod-repo and ImagePolicy/blaster-prod-policy are Ready=True.
  • ImageUpdateAutomation/blaster-prod-automation shows recent commits against games/blaster.
  • blaster-prod-registry Secret present and valid in both blaster and flux-system namespaces.
  • blaster-app Deployment uses the expected prod-YYYYMMDD.BUILD tag.