Skip to main content

Dev verification

info

Use this page to help troubleshoot pipeline issues or to confirm if everything is configured correctly. It goes through a standard set of checks from Git and CI, through Flux and image automation, down to Kubernetes resources and DNS.

Blaster GitOps series​

  1. Blaster GitOps summary
  2. Blaster repo and branches
  3. Dockerfile & GitLab CI
  4. Clerk authentication & user setup
  5. Google OAuth for Clerk
  6. Blaster prep for automation
  7. Dev app k8s manifests
  8. Dev flux sources & Kustomizations
  9. Dev image automation
  10. Dev SOPS & age
  11. Dev verification & troubleshooting - you are here
  12. Dev full runbook
  13. Prod overview
  14. Prod app k8s manifests and deployment
  15. Prod Flux GitOps and image automation
  16. Prod Cloudflare, Origin CA and tunnel routing
  17. Prod full runbook
  18. Post development branches

1. Fast end to end smoke test​

If you just want a quick answer to the question "is Blaster healthy right now", run these in order:

# 1. App namespace: pods, services, ingress
kubectl -n blaster-dev get pods,svc,ingress

# 2. App Deployment rollout status
kubectl -n blaster-dev rollout status deploy/blaster-app --timeout=120s

# 3. Database
kubectl -n blaster-dev get statefulset,pod,svc | grep blaster-db || true

# 4. Flux status
flux get kustomizations -n flux-system
flux get image repository,policy,update -n flux-system

# 5. GitLab latest pipelines
# Run in browser: GitLab -> games/blaster -> CI/CD -> Pipelines

If all of these look healthy and https://blaster.reids.net.au responds inside the cluster, the system is broadly fine.

2. Git and branches​

2.1 Check local branches​

From games/blaster:

git status
git branch -vv
git remote -v

You should see:

  • origin pointing at https://gitlab.reids.net.au/games/blaster.git
  • main and develop both tracking origin/main and origin/develop
  • A clean working tree before changing manifests or CI

2.2 Confirm branch protections​

In GitLab for games/blaster:

  • Settings -> Repository -> Protected branches
    • main protected; only maintainers can push; merge requests required
    • develop protected; same pattern

Check that:

  • You do not use direct pushes to main or develop
  • Pipelines must succeed before merge

If you see direct pushes to main, investigate before trusting prod.

3. GitLab CI pipeline checks​

3.1 Latest pipeline per branch​

In GitLab:

  • Go to CI/CD -> Pipelines

  • Filter by branch:

    • develop: should show recent pipelines with stages notify, lint, test, build, notify_end
    • main: similar, but only when you promote changes

Make sure:

  • Pipelines are green for the commit you expect Flux to deploy
  • build:develop and build:main show successful Kaniko pushes to the registry

3.2 Confirm image tags​

From the Kaniko logs on a successful job you should see lines similar to:

Pushing image to registry.reids.net.au/games/blaster:dev-YYYYMMDD.N
Pushed registry.reids.net.au/games/blaster@sha256:...

Also check GitLab:

  • Packages & Registries -> Container Registry -> games/blaster
  • Confirm tags like:
    • dev-20251115.49
    • prod-20251115.11

If the image tag you expect does not exist in the registry, Flux cannot deploy it.

4. SOPS and age sanity checks​

If encrypted secrets are not being applied, check the basics.

4.1 SOPS environment on your workstation​

echo "$SOPS_AGE_KEY_FILE"
ls -l "$SOPS_AGE_KEY_FILE"
sops --version

You should see:

  • SOPS_AGE_KEY_FILE set and pointing at age.key
  • The file readable by your user

4.2 File level checks​

In the app repo:

cd ~/Projects/blaster
grep -n 'sops:' k8s/dev/*.enc.yaml

In the infra repo:

cd ~/Projects/flux-config
grep -n 'sops:' clusters/my-cluster/flux-system/secrets/*.yaml

You should see:

  • Each managed secret file contains a sops: block
  • No plain text passwords or tokens in Git

4.3 Cluster side check​

kubectl -n flux-system get secret sops-age
kubectl -n flux-system get kustomization flux-system -o yaml | sed -n '1,80p'
kubectl -n flux-system get kustomization blaster-dev -o yaml | sed -n '1,80p'

Both Kustomizations should include:

decryption:
  provider: sops
  secretRef:
    name: sops-age

If not, follow the steps in SOPS & age.

5. Flux sources and Kustomizations​

5.1 Git sources​

flux get sources git -n flux-system

Look for:

  • flux-system pointing at fluxgitops/flux-config on main
  • blaster-dev pointing at games/blaster on develop

Both should be:

  • READY=True
  • With a recent REVISION such as develop@sha1:...

If blaster-dev is not ready:

  • Run flux reconcile source git blaster-dev -n flux-system --with-source
  • Check logs using kubectl -n flux-system logs deployment/source-controller and filter for blaster-dev

5.2 Kustomizations​

flux get kustomizations -n flux-system

Look for:

  • flux-system with READY=True
  • blaster-dev with READY=True and correct REVISION

If blaster-dev is not ready:

flux reconcile kustomization blaster-dev -n flux-system --with-source
kubectl -n flux-system describe kustomization blaster-dev

Common problems:

  • Path mismatch (for example path: ./k8s/dev missing)
  • Decryption errors due to SOPS misconfiguration
  • Invalid YAML in k8s/dev manifests

6. Image automation checks​

If the app is running but not picking up new images, check the Flux image stack.

6.1 ImageRepository and ImagePolicy​

flux get image repository,policy -n flux-system
kubectl -n flux-system get imagerepository blaster-dev-repo -o yaml | sed -n '1,80p'
kubectl -n flux-system get imagepolicy blaster-dev-policy -o yaml | sed -n '1,80p'

You should see:

  • blaster-dev-repo with READY=True and a recent lastScanResult
  • blaster-dev-policy with READY=True and a latestImage pointing at a dev-YYYYMMDD.N tag

If ImageRepository is not ready:

  • Check the blaster-dev-registry Secret in flux-system
  • Check connectivity to registry.reids.net.au

If ImagePolicy is not ready:

  • Confirm that tags in the registry actually match the filter pattern
  • For example dev-20251115.49 matches pattern ^dev-(?P<ts>[0-9]{8}\.[0-9]+)$

6.2 ImageUpdateAutomation​

flux get image update -n flux-system
kubectl -n flux-system get imageupdateautomation blaster-dev-automation -o yaml | sed -n '1,80p'

Check:

  • READY=True
  • lastRunTime is recent
  • status.conditions does not show permission errors

If you see errors like permission denied or could not push, confirm:

  • The SSH key referenced by GitRepository blaster-dev has write access in GitLab
  • Commit messages include [skip ci] to avoid CI loops

For configuration details see Image automation.

6.3 Deployment annotation​

In games/blaster:

grep -n 'imagepolicy' k8s/dev/50-app-deployment.yaml

You should see something like:

image: registry.reids.net.au/games/blaster:dev-20251115.42 # {"$imagepolicy": "flux-system:blaster-dev-policy"}

If the comment is missing, image automation will not update the tag.

7. Kubernetes resources and ingress​

7.1 Namespace inventory​

kubectl -n blaster-dev get all

Expect at least:

  • statefulset/blaster-db
  • deployment/blaster-app
  • service/blaster-db
  • service/blaster-app
  • ingress/blaster-ingress

If resources are missing:

  • Confirm k8s/dev manifests in the app repo
  • Check blaster-dev Kustomization logs for apply errors

7.2 Ingress and DNS​

kubectl -n blaster-dev get ingress blaster-ingress -o yaml | sed -n '1,40p'

You should see host blaster.reids.net.au and ingressClassName: nginx.

On a node with access to your DNS:

dig +short blaster.reids.net.au
curl -vk https://blaster.reids.net.au/ -H "Host: blaster.reids.net.au"

If DNS does not resolve or the IP does not match the ingress controller, fix DNS or MetalLB / VIP configuration.

8. Common failure scenarios and fixes​

8.1 Pipeline green, but cluster still old image​

Likely causes:

  • Image automation not enabled or failing
  • Flux Kustomization not reconciling

Checks:

  1. flux get image policy -n flux-system
  2. flux get image update -n flux-system
  3. git log develop -n 5 in games/blaster to look for Flux generated commits with [skip ci]
  4. kubectl -n blaster-dev get deploy blaster-app -o jsonpath='{.spec.template.spec.containers[0].image}'

If policies and updates look good but the image in the Deployment is old, the Kustomization might not be tracking the latest commit; reconcile:

flux reconcile kustomization blaster-dev -n flux-system --with-source

8.2 Secrets not applied or CrashLoopBackOff​

Symptoms:

  • Pods crashing with env var errors
  • Missing secret error events

Checks:

kubectl -n blaster-dev describe pod <pod-name>
kubectl -n blaster-dev get secret

If the expected Secrets are missing:

  • Confirm k8s/dev/*.enc.yaml exists and is valid
  • Confirm blaster-dev Kustomization decryption is configured
  • Check Flux errors via kubectl -n flux-system describe kustomization blaster-dev

8.3 Flux complaining about SOPS​

Typical messages:

  • failed to decrypt sops secret
  • no matching private key

Checklist:

  • kubectl -n flux-system get secret sops-age
  • Public key in .sops.yaml matches comment line in age.key
  • age.key in sops-age is the same as your local file

If you have accidentally generated a second age key, either:

  • Replace sops-age with the original one
  • Or rotate your repos to the new key and update .sops.yaml and sops-age together

9. Safe recovery patterns​

9.1 When in doubt, do not hotfix in-cluster​

warning

Avoid running kubectl apply directly on app manifests in Blaster namespaces. Always prefer fixing Git and letting Flux converge.

Safe sequence:

  1. Fix the manifest or secret in the appropriate repo and branch.

  2. Commit and push.

  3. Reconcile Flux:

    flux reconcile source git blaster-dev -n flux-system --with-source
    flux reconcile kustomization blaster-dev -n flux-system --with-source

  4. Watch the namespace:

    kubectl -n blaster-dev get pods -w

9.2 Roll back a bad deploy​

If a bad image or manifest has just gone out:

  1. In games/blaster, develop branch:

    • Use git revert on the offending commit
    • Or reset to a known good commit and force push only if you are sure

  2. Force a reconcile:

    flux reconcile kustomization blaster-dev -n flux-system --with-source

  3. Confirm the Deployment image has reverted and pods are healthy.

10. Verification checklist​

  • games/blaster main and develop branches exist and are protected in GitLab.
  • GitLab CI pipelines are green for the latest develop and main commits.
  • The expected dev-* and prod-* tags exist in the GitLab Container Registry.
  • SOPS is configured in both app and infra repos and secrets show sops metadata, not clear text.
  • The sops-age Secret exists in flux-system and Flux Kustomizations have decryption.provider: sops.
  • flux get sources git -n flux-system shows READY=True for flux-system and blaster-dev.
  • flux get kustomizations -n flux-system shows READY=True for flux-system and blaster-dev.
  • Flux image repository, policy and update objects are READY=True and using the expected tags.
  • kubectl -n blaster-dev get pods,svc,ingress shows healthy pods and a single blaster-ingress.
  • https://blaster.reids.net.au works from an internal client and loads the expected build.

If all of these are true, the Blaster GitOps pipeline is healthy end to end.