Skip to main content

Dev Flux sources and Kustomizations

info

This runbook explains how Flux sees the Blaster code and manifests via GitRepository and Kustomization resources in the infra repo, and how those tie back to the games/blaster app repo and k8s/dev / k8s/prod overlays.

Blaster GitOps series​

  1. Blaster GitOps summary
  2. Blaster repo and branches
  3. Dockerfile & GitLab CI
  4. Clerk authentication & user setup
  5. Google OAuth for Clerk
  6. Blaster prep for automation
  7. Dev app k8s manifests
  8. Dev flux sources & Kustomizations - you are here
  9. Dev image automation
  10. Dev SOPS & age
  11. Dev verification & troubleshooting
  12. Dev full runbook
  13. Prod overview
  14. Prod app k8s manifests and deployment
  15. Prod Flux GitOps and image automation
  16. Prod Cloudflare, Origin CA and tunnel routing
  17. Prod full runbook
  18. Post development branches

1. Context​

Repositories:

  • App repo: games/blaster

  • Infra repo: fluxgitops/flux-config

    • Contains all Flux configuration for the cluster under clusters/my-cluster.
    • Blaster-specific configuration lives under clusters/my-cluster/blaster.

Cluster:

  • Flux installed in namespace flux-system.
  • Flux bootstrapped against fluxgitops/flux-config on the main branch.
  • SOPS used for secrets with an age key stored in the sops-age Secret in flux-system.

2. Flux bootstrap and components​

Flux was installed using flux bootstrap gitlab pointed at the infra repo and cluster path:

flux bootstrap gitlab \
  --hostname=gitlab.reids.net.au \
  --owner=fluxgitops \
  --repository=flux-config \
  --branch=main \
  --path=clusters/my-cluster \
  --deploy-token-auth \
  --insecure-skip-tls-verify \
  --read-write-key

Later, the image automation components were added by re-running bootstrap with extra components:

flux bootstrap gitlab \
  --hostname=gitlab.reids.net.au \
  --owner=fluxgitops \
  --repository=flux-config \
  --branch=main \
  --path=clusters/my-cluster \
  --deploy-token-auth \
  --insecure-skip-tls-verify \
  --read-write-key \
  --components-extra=image-reflector-controller,image-automation-controller

Notes:

  • Re-running bootstrap with the same repo / branch / path is idempotent.
  • It regenerates gotk-components.yaml and keeps gotk-sync.yaml aligned with the repo.
  • It does not delete existing application configuration under clusters/my-cluster/*.

3. Flux-system layout in the infra repo​

At a high level:

flux-config/
  .sops.yaml
  clusters/
    my-cluster/
      flux-system/
        gotk-components.yaml
        gotk-sync.yaml
        kustomization.yaml
        secrets/
          blaster-dev-registry.yaml
      blaster/
        00-namespace.yaml
        dev/
          20-blaster-images-dev.yaml
          30-image-automation.yaml
          kustomization.yaml
          source.yaml
        kustomization.yaml
      apps/
      cloudflare/
      dev/
      origin-ca-issuer/
      prod/
      kustomization.yaml

Key points:

  • clusters/my-cluster/kustomization.yaml is the cluster root for Flux.
  • clusters/my-cluster/flux-system holds the Flux core manifests.
  • clusters/my-cluster/blaster holds Blaster-specific namespaces, sources and Kustomizations.

4. Top-level cluster Kustomization​

File: clusters/my-cluster/kustomization.yaml

# clusters/my-cluster/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ./flux-system
  - ./apps
  - ./dev
  - ./origin-ca-issuer
  - ./cloudflare
  - ./blaster

This:

  • Ensures the flux-system components are applied first.
  • Includes shared apps, dev, cloudflare and origin-ca-issuer trees.
  • Adds the blaster folder so that Blaster-specific objects are managed by Flux.

5. Flux-system Kustomization and SOPS​

File: clusters/my-cluster/flux-system/kustomization.yaml

# clusters/my-cluster/flux-system/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
  - ./secrets/blaster-dev-registry.yaml

Secrets in flux-system are encrypted with SOPS according to the infra repo policy:

# .sops.yaml (infra repo)
creation_rules:
  - path_regex: 'clusters/my-cluster/.*/secrets/.*\.yaml$'
    encrypted_regex: '^(data|stringData)$'
    age: ['AGE_PUBLIC_KEY_HERE']

Flux is told to use SOPS decryption via a patch to the flux-system Kustomization inside the cluster:

kubectl -n flux-system patch kustomization flux-system \
  --type='merge' \
  -p '{
    "spec": {
      "decryption": {
        "provider": "sops",
        "secretRef": {
          "name": "sops-age"
        }
      }
    }
  }'

Result (snippet):

spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-age
  interval: 10m0s
  path: ./clusters/my-cluster
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

This allows Flux to decrypt SOPS-encrypted Secrets under clusters/my-cluster/**.

6. Blaster namespace and aggregator Kustomization​

6.1 Namespace​

File: clusters/my-cluster/blaster/00-namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: blaster-dev
  labels:
    name: blaster-dev

Dev uses blaster-dev. A separate blaster namespace will be used for prod.

6.2 Blaster aggregator​

File: clusters/my-cluster/blaster/kustomization.yaml

# clusters/my-cluster/blaster/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ./00-namespace.yaml
  - ./dev/source.yaml
  - ./dev/kustomization.yaml
  - ./dev/20-blaster-images-dev.yaml
  - ./dev/30-image-automation.yaml

This file:

  • Creates the blaster-dev namespace.
  • Configures the Blaster dev GitRepository, Kustomization and image automation resources.
  • Keeps everything for the Blaster app grouped under one tree for clarity.

Prod will later add a sibling prod/ directory and extra resources.

7. GitRepository for Blaster dev​

File: clusters/my-cluster/blaster/dev/source.yaml

# clusters/my-cluster/blaster/dev/source.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: blaster-dev
  namespace: flux-system
spec:
  interval: 1m
  timeout: 60s
  url: ssh://git-ssh.reids.net.au/games/blaster.git
  ref:
    branch: develop
  secretRef:
    name: flux-ssh-auth

Key fields:

  • namespace: flux-system
    All Flux sources live in flux-system.

  • url
    Uses SSH to access the games/blaster repo.

  • ref.branch: develop
    The dev environment tracks the develop branch of the app repo.

  • secretRef.name: flux-ssh-auth
    This Secret contains the private key for Git over SSH and must match a deploy key in GitLab with write access if image automation commits back. See Image automation.

Verification commands:

flux get sources git -n flux-system
kubectl -n flux-system get gitrepository blaster-dev -o yaml | sed -n '1,80p'

You should see READY=True and a recent artifact revision such as develop@sha1:....

8. Kustomization for Blaster dev​

File: clusters/my-cluster/blaster/dev/kustomization.yaml

# clusters/my-cluster/blaster/dev/kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: blaster-dev
  namespace: flux-system
spec:
  interval: 1m
  path: ./k8s/dev
  prune: true
  sourceRef:
    kind: GitRepository
    name: blaster-dev
  wait: true
  timeout: 5m
  decryption:
    provider: sops
    secretRef:
      name: sops-age

What each field does:

  • path: ./k8s/dev
    Tells Flux to look under k8s/dev in the app repo for the app Kustomization and manifests.

  • sourceRef.name: blaster-dev
    Connects this Kustomization to the GitRepository defined in source.yaml.

  • prune: true
    Resources removed from Git are garbage-collected from the cluster.

  • decryption.provider: sops
    Allows SOPS-encrypted Secrets in k8s/dev (for example 10-secret-db.enc.yaml) to be decrypted using the sops-age Secret in flux-system.

This Kustomization is what actually applies the blaster-dev application stack. For the contents of k8s/dev, see App k8s manifests.

9. Registry Secret for Flux image scanning​

Flux image automation needs credentials to scan the private registry tags for Blaster.

A docker-registry Secret is created and stored in Git, then encrypted with SOPS:

kubectl -n flux-system create secret docker-registry blaster-dev-registry \
  --docker-server=registry.reids.net.au \
  --docker-username='blaster-dev' \
  --docker-password='REPLACE-ME' \
  --docker-email='andrew@reids.net.au' \
  --dry-run=client -o yaml \
  > clusters/my-cluster/flux-system/secrets/blaster-dev-registry.yaml

The file is then encrypted:

sops -e -i clusters/my-cluster/flux-system/secrets/blaster-dev-registry.yaml

And referenced in clusters/my-cluster/flux-system/kustomization.yaml as shown earlier.

Image automation resources (ImageRepository, ImagePolicy, ImageUpdateAutomation) for dev are defined under:

  • clusters/my-cluster/blaster/dev/20-blaster-images-dev.yaml
  • clusters/my-cluster/blaster/dev/30-image-automation.yaml

See Image automation for full details.

10. Reconciling and checking status​

10.1 Force a reconcile​

To avoid waiting for intervals, you can manually reconcile both the core and Blaster dev Kustomizations:

flux reconcile source git flux-system -n flux-system \
  && flux reconcile kustomization flux-system -n flux-system --with-source

flux reconcile kustomization blaster-dev -n flux-system --with-source

10.2 Check sources​

flux get sources git -n flux-system

Expected output (example):

NAME          REVISION               SUSPENDED  READY  MESSAGE
blaster-dev   develop@sha1:880afbae  False      True   stored artifact for revision 'develop@sha1:880afbae'
flux-system   main@sha1:975f688e     False      True   stored artifact for revision 'main@sha1:975f688e'
...

10.3 Check Kustomizations​

flux get kustomizations -n flux-system

Example:

NAME          REVISION               SUSPENDED  READY  MESSAGE
blaster-dev   develop@sha1:880afbae  False      True   Applied revision: develop@sha1:880afbae
flux-system   main@sha1:975f688e     False      True   Applied revision: main@sha1:975f688e
...

When both the source and Kustomization are READY=True, Blaster dev manifests have been applied successfully.

11. Behaviour across environments​

Once prod is configured, you will have:

  • GitRepository blaster-dev tracking develop, path ./k8s/dev.
  • GitRepository blaster-prod tracking main, path ./k8s/prod.
  • Separate Kustomizations for each environment, sharing the same app repo but different branches and paths.

High-level behaviour:

  • Changes merged into develop:

    • Trigger GitLab CI to build a dev-YYYYMMDD.N image.
    • Flux image automation updates the dev Deployment image tag in Git.
    • blaster-dev Kustomization syncs the cluster.

  • Changes merged into main:

    • Trigger GitLab CI to build a prod-YYYYMMDD.N image.
    • Prod Kustomization syncs k8s/prod to update the Blaster prod namespace.

For Dockerfile and CI details see Dockerfile & GitLab CI.

12. Verification checklist​

  • fluxgitops/flux-config has a clusters/my-cluster/blaster directory with:
    • 00-namespace.yaml
    • dev/source.yaml
    • dev/kustomization.yaml
    • image automation YAMLs for dev.
  • clusters/my-cluster/kustomization.yaml includes ./blaster.
  • GitRepository blaster-dev in flux-system points to games/blaster on the develop branch.
  • Kustomization blaster-dev in flux-system points to ./k8s/dev and uses SOPS decryption.
  • blaster-dev-registry Secret exists in flux-system and is SOPS-encrypted in Git.
  • flux get sources git -n flux-system shows READY=True for blaster-dev.
  • flux get kustomizations -n flux-system shows READY=True for blaster-dev.
  • kubectl -n blaster-dev get pods,svc,ingress shows a healthy app and database.

Once these are all true, the Flux sources and Kustomizations for Blaster dev are correctly configured and ready to support GitOps and image automation.