Skip to main content

GitOps

info

GitOps means driving Kubernetes changes from Git, using automation to reconcile the cluster to whatever is described in the repo. You review and approve changes in Git; controllers do the rest.

1. What is GitOps​

GitOps is an operating model where:

  • Git is the source of truth for Kubernetes manifests, config and policies.
  • Automation watches Git and reconciles the cluster to match the desired state.
  • Changes go via merge requests, not kubectl apply from a laptop.

In practice:

  • You treat manifests, Helm charts, Kustomize overlays and policies as code.
  • You use a pull-based controller in the cluster (for example Flux) that:
    • Pulls from Git.
    • Applies changes.
    • Heals drift when something in the cluster diverges from Git.

This gives you:

  • Audit trail – every change is a commit with an author and review history.
  • Repeatability – you can recreate the same state in another cluster from Git.
  • Safety – you can roll back by reverting a commit and letting GitOps reconcile.

2. Core GitOps principles​

  1. Declarative configuration

    • Everything that matters (apps, networking, ingress, security, policies) is defined declaratively.
    • The cluster is expected to converge to this declared state.

  2. Versioned and immutable

    • All config is stored in Git with history and tags.
    • You store only the desired state, not imperative scripts.

  3. Automated reconciliation

    • An operator (Flux) continuously compares the cluster with Git.
    • If the cluster drifts, Flux attempts to bring it back to the Git state.

  4. Operations via pull requests

    • All changes go through GitLab merge requests.
    • Reviews, approvals and CI pipelines happen before anything reaches the cluster.

  5. Git is the control plane for config

    • You do not treat kubectl as the source of truth.
    • If it is not in Git, it is either temporary or a bug.

3. How I implement GitOps​

This section describes the approach used for Muppit clusters and the Blaster game.

3.1 Repositories​

  • App repo – for example games/blaster:
    • Application code (Next.js, Phaser).
    • Dockerfile.
    • Kubernetes manifests under k8s/ (dev and prod overlays).
  • Flux config repo – for example flux-config:
    • Cluster definitions under clusters/my-cluster/....
    • GitRepository, Kustomization, ImageRepository, ImagePolicy and ImageUpdateAutomation objects.
    • Shared infrastructure components (Cloudflare tunnel, Ingress-NGINX, cert-manager, etc).

This separation lets you:

  • Evolve the app independently of the cluster plumbing.
  • Reuse the Flux config across multiple applications.

3.2 CI and image build flow​

GitLab CI builds and publishes container images; Flux deploys them.

  • GitLab Runner using the Kubernetes executor runs pipelines in the gitlab namespace.
  • Kaniko builds images and pushes them to registry.reids.net.au.

For Blaster:

  • Commits to develop produce images tagged like
    registry.reids.net.au/games/blaster:dev-YYYYMMDD.IID.
  • Commits to main produce images tagged like
    registry.reids.net.au/games/blaster:prod-YYYYMMDD.IID.
  • Pipelines are marked [skip ci] when Flux image automation commits back to Git.

GitLab CI does not talk to the cluster directly. It only:

  • Builds images.
  • Updates image references in the app repo (via Flux image automation).
  • Lets Flux reconcile the running workloads.

3.3 Flux and reconciliation​

Flux runs inside the cluster and owns the reconciliation loop.

Key objects:

  • GitRepository – tells Flux which repo and branch to watch.
  • Kustomization – points at a path inside the repo and controls:
    • Sync interval.
    • Pruning.
    • Health checks.
  • ImageRepository and ImagePolicy – tell Flux how to discover and select image tags.
  • ImageUpdateAutomation – commits new image tags back to the app repo.

Flow:

  1. GitLab CI pushes a new dev-YYYYMMDD.IID or prod-YYYYMMDD.IID image tag.
  2. Flux ImageRepository sees the new tag in the registry.
  3. ImagePolicy selects the latest acceptable tag (for example by timestamp).
  4. ImageUpdateAutomation updates the image field in the deployment manifest and commits to Git.
  5. The relevant Kustomization sees the commit, applies the change and waits for the rollout to become healthy.

At every point, the cluster state is driven from Git.

3.4 Environments and overlays​

Environments are split by directory and Flux Kustomization objects.

For example, in the app repo:

  • k8s/dev/... – manifests for the internal dev environment:
    • Namespace blaster.
    • Internal DNS such as blaster.reids.net.au.
    • Less aggressive resource limits.
  • k8s/prod/... – manifests for the public production environment:
    • Namespace blaster.
    • Public DNS such as blaster.muppit.au.
    • Production resource requests, limits, probes and NetworkPolicy.

In the Flux config repo:

  • clusters/my-cluster/blaster/10-blaster-kustomization.yaml points at the dev path.
  • clusters/my-cluster/blaster-prod/10-blaster-kustomization.yaml points at the prod path.

This gives:

  • A single app codebase.
  • Clear separation of dev and prod operational concerns.
  • Git as the place where you can diff and review environment changes.

3.5 Secrets and SOPS​

Secrets are stored in Git in encrypted form using SOPS with age keys.

Pattern:

  • Plaintext Secret manifests never live in Git.
  • Instead you keep secret.enc.yaml encrypted with SOPS.
  • Flux integrates with SOPS so it can decrypt at apply time using an age key stored in the cluster.

Benefits:

  • Secrets are versioned and reviewed like everything else.
  • You keep compliance and auditability without leaking credentials.

3.6 Branching and GitOps​

Branching interacts with GitOps as follows:

  • main – reflects what is in production. Only updated via MR from develop.
  • develop – integration branch used by the dev environment.
  • Feature branches – short-lived branches for specific changes that merge into develop.

Flux is wired to:

  • Track develop for dev workloads.
  • Track main for prod workloads.

When you merge a feature branch into develop:

  • CI builds a new dev image.
  • Flux updates the dev deployment automatically.

When you later merge develop into main:

  • CI builds a new prod image.
  • Flux updates the prod deployment automatically.

The Kubernetes cluster never depends on someone manually applying YAML.

4. When GitOps is and is not used​

GitOps is used for:

  • Long-lived Kubernetes workloads (apps, databases, ingress, tunnels).
  • Cluster add-ons and infrastructure.
  • Policies and security objects.

GitOps is not used for:

  • One-off debug commands (kubectl exec, kubectl logs).
  • Temporary exploratory resources created in a live debugging session (these should either be codified later or removed).
  • Local-only development using next dev or similar.

Anything that needs to be recreated, audited or promoted between environments should be brought under GitOps.

5. Verification checklist​

  • All Kubernetes manifests for long-lived workloads live in Git.
  • Flux (or another GitOps controller) is installed in the cluster and healthy.
  • App repo and Flux config repo boundaries are clear and documented.
  • GitLab CI builds images and pushes to registry.reids.net.au.
  • Flux image automation updates manifests with new image tags.
  • Environments (dev, prod) are separated by directories and Flux Kustomization objects.
  • Secrets are stored in Git only in SOPS-encrypted form.
  • main reflects production; develop reflects integration; feature branches are short-lived and merge into develop.