Flux integration

info

This page covers the Flux config structure, dependency control to avoid race conditions, and deployment commands.

Observability stack series

1. Observability stack
2. Architecture
3. Manifests
4. Flux integration - You are here
5. Operations

Two-repo GitOps pattern

The monitoring stack follows a two-repo pattern where Flux config and app manifests are separated:

Flux config structure

The Flux configuration lives in your-flux-org/flux-config under clusters/my-cluster/monitoring/.

clusters/my-cluster/monitoring/
├── kustomization.yaml              # Kustomize wrapper
├── source.yaml                     # GitRepository for app repo
├── 00-kustomization-ns.yaml        # Flux Kustomization for namespace + HelmRepos
├── 10-kustomization-app.yaml       # Flux Kustomization for app (dependsOn: monitoring-ns)
└── ns/
    ├── kustomization.yaml
    ├── namespace.yaml              # monitoring namespace
    ├── 10-helm-repo-prometheus.yaml
    └── 11-helm-repo-grafana.yaml

Dependency control

The deployment order is critical. HelmReleases cannot deploy until:

1. The monitoring namespace exists
2. The HelmRepositories exist in that namespace

warning

Without proper dependencies, Flux would try to create HelmReleases before the namespace and HelmRepositories exist, causing failures that can be confusing to debug.

Why the ns/ folder contains HelmRepositories

HelmRepositories are placed in the ns/ folder alongside the namespace so they are applied together by the same Flux Kustomization. This ensures:

1. Namespace is created first (Kustomize orders Namespace resources before others)
2. HelmRepositories are created in the same reconciliation
3. The app Kustomization waits for monitoring-ns to be ready before applying HelmReleases

Namespace manifest

# clusters/my-cluster/monitoring/ns/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
  labels:
    name: monitoring

HelmRepositories

# clusters/my-cluster/monitoring/ns/10-helm-repo-prometheus.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: prometheus-community
  namespace: monitoring
spec:
  interval: 1h
  url: https://prometheus-community.github.io/helm-charts

# clusters/my-cluster/monitoring/ns/11-helm-repo-grafana.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: grafana
  namespace: monitoring
spec:
  interval: 1h
  url: https://grafana.github.io/helm-charts

Namespace Flux Kustomization

This applies the namespace and HelmRepositories together:

# clusters/my-cluster/monitoring/00-kustomization-ns.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: monitoring-ns
  namespace: flux-system
spec:
  interval: 10m
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  path: ./clusters/my-cluster/monitoring/ns
  timeout: 2m

Key points:

• Sources from flux-system GitRepository (the flux-config repo itself)
• Applies the ns/ folder which contains namespace and HelmRepositories

GitRepository source

# clusters/my-cluster/monitoring/source.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: observability-monitoring
  namespace: flux-system
spec:
  interval: 5m
  timeout: 60s
  url: ssh://git-ssh.example.local/observability/monitoring.git
  ref:
    branch: main
  secretRef:
    name: flux-ssh-auth
  ignore: |
    /*
    !/k8s/

The ignore pattern tells Flux to only watch the k8s/ directory, reducing unnecessary reconciliations when other files change.

App Flux Kustomization

This applies the app repo manifests after the namespace is ready:

# clusters/my-cluster/monitoring/10-kustomization-app.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: observability-monitoring
  namespace: flux-system
spec:
  dependsOn:
    - name: monitoring-ns
  interval: 10m
  timeout: 15m
  prune: true
  wait: true
  sourceRef:
    kind: GitRepository
    name: observability-monitoring
    namespace: flux-system
  path: ./k8s/prod
  targetNamespace: monitoring
  decryption:
    provider: sops
    secretRef:
      name: sops-age

Key points:

Setting	Value	Purpose
dependsOn	monitoring-ns	Wait for namespace and HelmRepositories
sourceRef	observability-monitoring	The app repo GitRepository
path	./k8s/prod	Apply manifests from this path
targetNamespace	monitoring	Deploy all resources to monitoring namespace
decryption.provider	sops	Decrypt SOPS-encrypted secrets
timeout	15m	Allow time for HelmReleases to become ready

Kustomize wrapper

The folder's kustomization.yaml ties everything together:

# clusters/my-cluster/monitoring/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - source.yaml
  - 00-kustomization-ns.yaml
  - 10-kustomization-app.yaml

Root aggregator

Add the monitoring folder to the root aggregator:

# clusters/my-cluster/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cert-manager.yaml
  - ./flux-system
  - ./apps
  - ./dev
  - ./origin-ca-issuer
  - ./cloudflare
  - ./blaster
  - ./identity-internal
  - ./infra-trust
  - ./cert-manager
  - ./wordpress
  - ./monitoring    # Add this line

Deployment commands

After committing both repos:

# 1. Commit and push the app repo
cd /path/to/observability/monitoring
git add .
git commit -m "Add monitoring stack manifests"
git push

# 2. Commit and push flux-config
cd /path/to/flux-config
git add clusters/my-cluster/{kustomization.yaml,monitoring}
git commit -m "Add monitoring stack via Flux"
git push

# 3. Reconcile Flux
flux reconcile source git flux-system -n flux-system
flux reconcile kustomization flux-system -n flux-system --with-source

# 4. Watch deployment progress
flux get kustomizations -n flux-system | grep monitoring
kubectl get pods -n monitoring -w

Verification

# Check Flux Kustomizations
flux get kustomizations -n flux-system | grep monitoring
# Expected:
# monitoring-ns              main@sha1:...  False  True   Applied revision: ...
# observability-monitoring   main@sha1:...  False  True   Applied revision: ...

# Check HelmReleases
flux get helmreleases -n monitoring
# Expected:
# kube-prometheus-stack  monitoring  80.6.0  True  Release reconciliation succeeded
# loki                   monitoring  6.46.0  True  Release reconciliation succeeded
# alloy                  monitoring  1.5.1   True  Release reconciliation succeeded

# Check HelmRepositories
kubectl get helmrepositories -n monitoring
# Expected:
# prometheus-community  https://prometheus-community.github.io/helm-charts
# grafana               https://grafana.github.io/helm-charts

Troubleshooting

HelmRelease stuck on "HelmRepository not found"

This means the HelmRepositories were not created or the app Kustomization did not wait for them.

# Check if HelmRepositories exist
kubectl get helmrepositories -n monitoring

# If missing, check the namespace Kustomization
flux get kustomizations monitoring-ns -n flux-system

# Force reconcile
flux reconcile kustomization monitoring-ns -n flux-system

Namespace does not exist error

The app Kustomization tried to deploy before the namespace was ready.

# Check dependency status
flux get kustomizations -n flux-system | grep monitoring

# The monitoring-ns should be Ready before observability-monitoring starts

SOPS decryption failed

The sops-age secret is missing or does not contain the correct key.

# Check if the secret exists
kubectl get secret sops-age -n flux-system

# The secret should contain the age private key