Skip to main content

Cloudflare GitOps via Flux

This runbook provides end-to-end instructions on how to deploy and manage Cloudflare and the Cloudflare CA issuer onto the cluster using GitOps automation via Flux.

  • The Origin CA Issuer controller (from Cloudflare’s upstream repo) via Flux
  • A Cloudflare namespace with a local‑managed cloudflared tunnel, Origin CA certs, Ingress, and NetworkPolicies
  • GitOps flow from GitLab (Flux continuously reconciles everything)
  • SOPS to keep Kubernetes Secrets encrypted in Git
  • This playbook has been tested on:
    • Kubernetes version: v1.31.9
    • OS-Image: Ubuntu 24.04.2 LTS
    • Kernel version: 6.8.0-62-generic
    • Container runtime: containerd://2.0.5

1.0 - Environment assumptions​

Environment assumptions

  • Flux v2 already bootstrapped (flux-system namespace exists).
  • You use an “infra” repo (Flux root) called flux-config (GitRepository name: flux-system in flux-system ns).
  • You will create a new “app” repo for Cloudflare stack: muppit-apps/cloudflare.
  • Ingress is ingress-nginx. cert-manager is installed and healthy.
  • Cloudflare account and a local-managed named tunnel created on your Mac (cloudflared tunnel create ...) so you have a credentials.json file.

1.0.1 Prerequisites​

  • Cluster add‑ons present and Ready
    • cert-manager (CRDs + controller + webhook).
    • ingress-nginx (LB IP/ClusterIP reachable from pods).
  • Flux already installed and syncing your infra repo (called here flux-config). Flux root points to: ./clusters/my-cluster.
  • GitLab (self-hosted): repos available under your group, e.g. muppit-apps and fluxgitops.
  • Domains: muppit.au managed in Cloudflare; you have permission to manage DNS, Tunnels, and Page Rules/Redirect Rules.

1.0.2 Create the Cloudflare app repo (GitLab)​

Create a blank project (web UI) before you start (no README):

  • App repo for Cloudflare stack: https://gitlab.reids.net.au/muppit-apps/cloudflare.git
    • Group: muppit-apps
    • Project name: cloudflare
    • We’ll push K8s manifests (namespace, Secrets, ConfigMap, Deployment, etc.) under k8s/prod/ in this repo.
  • Infra repo (already present): https://gitlab.reids.net.au/fluxgitops/flux-config.git

SSH vs HTTPS note
Your GitLab uses git-ssh.reids.net.au for SSH on a custom port. From your Mac you prefer HTTPS clones. Flux can continue using SSH or HTTPS; adjust URLs accordingly.

1.1 - Flux root aggregator (infra repo)​

Confirm that the flux root points to ./clusters/my-cluster in flux-config. If not adjust all configuration to match your flux root.

kubectl -n flux-system get kustomization flux-system -o jsonpath='{.spec.path}'; echo

Clone the flux-config project (Mac) into Projects directory and change directory to the flux-config root (all git commands will be run from here):

git clone https://gitlab.reids.net.au/fluxgitops/flux-config.git
cd flux-config

Create the aggregator clusters/my-cluster/kustomization.yaml if it does not exist and comment out any resources you do not want to manage, or will be adding later. The origin-ca-issuer is commented out intially until all the configuration has been created and pushed to the repo.

# clusters/my-cluster/kustomization.yaml (in flux-config repo)
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ./flux-system
  - ./apps
  - ./dev
#  - ./prod
#  - ./origin-ca-issuer
#  - ./cloudflare

Create placeholder kustomizations in each child folder if they don’t exist (Flux requires a kustomization file in any referenced directory). Example:

# clusters/my-cluster/origin-ca-issuer/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - source.yaml
  - 00-kustomization-ns.yaml
  - 10-kustomization-crds.yaml
  - 20-kustomization-rbac.yaml
  - 30-kustomization-controller.yaml

Commit & push; reconcile root:

git add .
git commit -m "Add aggregator"
git push
flux reconcile kustomization flux-system -n flux-system --with-source

1.2 - Repo and image pinning strategy​

Pin a repo and image for deterministic builds with audit changes via Git diffs. Use a scheduled workflow (or manual) to change tags with a PR.

1.2.1 Pin controller repo to a release tag​

Recommended approach.

How to pin Origin CA Issuer to a specific release such as v0.12.1:

  1. Find tags: 
    git ls-remote --tags --refs https://github.com/cloudflare/origin-ca-issuer.git
  2. Edit clusters/my-cluster/origin-ca-issuer/source.yaml to the new ref.tag:; commit & push.
  3. Reconcile: 
    flux reconcile source git origin-ca-issuer-upstream -n flux-system && \
    flux reconcile kustomization origin-ca-issuer-crds -n flux-system && \
    flux reconcile kustomization origin-ca-issuer-rbac -n flux-system && \
    flux reconcile kustomization origin-ca-issuer-controller -n flux-system

1.2.2 Pin container images​

Cloudflared, your apps.

Find “latest” semantic calendar tag with crane (installed via brew install crane):

crane ls cloudflare/cloudflared \
  | grep -E '^[0-9]{4}\.[0-9]+(\.[0-9]+)?$' \
  | sort -V | tail -n1
# e.g. 2025.9.1

Update the Deployment image: to that exact tag and commit.
Repeat this pattern for all app images to ensure reproducible rollouts.

1.2.3 Intervals​

Keep spec.interval in Flux sources/kustomizations (e.g., 30m) — Flux uses it to refresh artifacts & status, and to re-apply drifted resources. With a pinned tag, Flux won’t pull newer commits automatically, but the interval still governs reconciliation/health.

1.2.4 GitLab CE limitations​

In GitLab CE (free tier) the mirror direction selector is hard-coded to Push and the Pull option is greyed out. This means it does not support mirroring from https://github.com/cloudflare/origin-ca-issuer.git or pinning a brach from the public repo.

1.3 - Origin CA Issuer (controller)​

Via Cloudflare upstream (PINNED).

We’ll deploy Cloudflare’s controller origin-ca-issuer from their official repo and pin to a released tag so it’s stable.

The Namespace was put in its own sub-directory to make the Namespace its own reconciliation and prune boundary and provide:

  1. Reliable ordering so the namespace exists before CRDs/RBAC/controller apply.
  2. Safer teardown so you can remove or rebuild the controller without accidentally deleting the namespace.
  3. A tidy place to keep namespace-scoped defaults like labels, quotas, imagePullSecrets and NetworkPolicies.
  4. A predictable repo pattern that is easy to scan.

1.3.1 Clone the flux-config repo​

Create origin-ca-issuer structure.

git clone https://gitlab.reids.net.au/fluxgitops/flux-config.git
cd flux-config
mkdir -p clusters/my-cluster/origin-ca-issuer/ns

Note that the -p flag will create any missing parent directories along the path.

1.3.2 Create manifests (plaintext)​

Create the following files under clusters/my-cluster/origin-ca-issuer/. Note that I remain in the root directory

1.3.2.1 Create namespace​

# clusters/my-cluster/origin-ca-issuer/ns/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: origin-ca-issuer

1.3.2.2 Create Kustomization​

# clusters/my-cluster/origin-ca-issuer/00-kustomization-ns.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: origin-ca-issuer-ns
  namespace: flux-system
spec:
  interval: 10m
  prune: true
  path: ./clusters/my-cluster/origin-ca-issuer/ns
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  timeout: 2m

1.3.2.3 Create upstream Git source​

Pinned to release v0.12.1.

# clusters/my-cluster/origin-ca-issuer/source.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: origin-ca-issuer-upstream
  namespace: flux-system
spec:
  interval: 30m
  url: https://github.com/cloudflare/origin-ca-issuer
  ref:
    tag: v0.12.1
  timeout: 2m

1.3.2.4 Create CRDs → RBAC → controller​

# clusters/my-cluster/origin-ca-issuer/10-kustomization-crds.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: origin-ca-issuer-crds
  namespace: flux-system
spec:
  dependsOn:
    - name: origin-ca-issuer-ns
  interval: 30m
  prune: true
  path: ./deploy/crds
  sourceRef:
    kind: GitRepository
    name: origin-ca-issuer-upstream
    namespace: flux-system
  timeout: 2m

1.3.2.4 Create RBAC​

# clusters/my-cluster/origin-ca-issuer/20-kustomization-rbac.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: origin-ca-issuer-rbac
  namespace: flux-system
spec:
  dependsOn:
    - name: origin-ca-issuer-crds
  interval: 30m
  prune: true
  path: ./deploy/rbac
  sourceRef:
    kind: GitRepository
    name: origin-ca-issuer-upstream
    namespace: flux-system
  timeout: 2m

1.3.2.5 Create controller​

# clusters/my-cluster/origin-ca-issuer/30-kustomization-controller.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: origin-ca-issuer-controller
  namespace: flux-system
spec:
  dependsOn:
    - name: origin-ca-issuer-rbac
  interval: 30m
  prune: true
  path: ./deploy/manifests
  sourceRef:
    kind: GitRepository
    name: origin-ca-issuer-upstream
    namespace: flux-system
  timeout: 2m
  targetNamespace: origin-ca-issuer

1.3.3 Commit and push​

git add .
git commit -m "init origin-ca-controller"
git push

1.3.4 Confirm flux-config repo layout​

Final repo layout:

tree -a -I '.git|.DS_Store'
.
└── clusters
    └── my-cluster
        ├── apps
        │   ├── app-manifests-kustomization.yaml
        │   ├── app-manifests-source.yaml
        │   └── kustomization.yaml
        ├── cloudflare
        │   ├── .gitkeep
        │   └── ns
        ├── dev
        │   ├── coach-app-dev-kustomization.yaml
        │   ├── coach-app-dev-source.yaml
        │   └── kustomization.yaml
        ├── flux-system
        │   ├── gotk-components.yaml
        │   ├── gotk-sync.yaml
        │   └── kustomization.yaml
        ├── kustomization.yaml
        ├── origin-ca-issuer
        │   ├── .gitkeep
        │   ├── 00-kustomization-ns.yaml
        │   ├── 10-kustomization-crds.yaml
        │   ├── 20-kustomization-rbac.yaml
        │   ├── 30-kustomization-controller.yaml
        │   ├── kustomization.yaml
        │   ├── ns
        │   │   └── namespace.yaml
        │   └── source.yaml
        └── prod
            └── .gitkeep

11 directories, 20 files

1.3.5 Reconcile and verify​

  • Interval on a Kustomization controls periodic drift checks and applies. Sources and Kustomizations both reconcile on their own intervals.

    kubectl -n flux-system get gitrepositories  -o custom-columns=NAME:.metadata.name,INTERVAL:.spec.interval
    NAME                        INTERVAL
    app-manifests               1m0s
    coach-app-dev               5m0s
    flux-system                 5m0s
    origin-ca-issuer-upstream   30m
    kubectl -n flux-system get kustomizations   -o custom-columns=NAME:.metadata.name,INTERVAL:.spec.interval
    NAME                          INTERVAL
    app-source-kustomization      2m0s
    coach-app-dev                 10m0s
    flux-system                   10m0s
    origin-ca-issuer-controller   30m
    origin-ca-issuer-crds         30m
    origin-ca-issuer-ns           30m
    origin-ca-issuer-rbac         30m

  • Even though origin-ca-issuer-*- Kustomizations reconcile on their own timers, new upstream changes will only be noticed after the Git source refreshes, which is every 30m:

    • GitRepository origin-ca-issuer-upstream: every 30m.
    • Kustomization origin-ca-issuer-ns (creates the Namespace): every 30m.
    • Kustomizations origin-ca-issuer-crds, origin-ca-issuer-rbac, origin-ca-issuer-controller: every 30m.

  • Force reconciliation immediately (instead of waiting for configured interval):

    flux reconcile source git origin-ca-issuer-upstream -n flux-system && \
    flux reconcile kustomization origin-ca-issuer-crds -n flux-system && \
    flux reconcile kustomization origin-ca-issuer-rbac -n flux-system && \
    flux reconcile kustomization origin-ca-issuer-controller -n flux-system
    ► annotating GitRepository origin-ca-issuer-upstream in flux-system namespace
    ✔ GitRepository annotated
    ◎ waiting for GitRepository reconciliation
    ✔ fetched revision trunk@sha1:debc90bd0e97881293fc8631fb56acdccfea9806
    ► annotating Kustomization origin-ca-issuer-crds in flux-system namespace
    ✔ Kustomization annotated
    ◎ waiting for Kustomization reconciliation
    ✔ applied revision trunk@sha1:debc90bd0e97881293fc8631fb56acdccfea9806
    ► annotating Kustomization origin-ca-issuer-rbac in flux-system namespace
    ✔ Kustomization annotated
    ◎ waiting for Kustomization reconciliation
    ✔ applied revision trunk@sha1:debc90bd0e97881293fc8631fb56acdccfea9806
    ► annotating Kustomization origin-ca-issuer-controller in flux-system namespace
    ✔ Kustomization annotated
    ◎ waiting for Kustomization reconciliation
    ✔ applied revision trunk@sha1:debc90bd0e97881293fc8631fb56acdccfea9806

  • Or force reconciliation with one command:

    flux reconcile kustomization flux-system -n flux-system --with-source
    ► annotating GitRepository flux-system in flux-system namespace
    ✔ GitRepository annotated
    ◎ waiting for GitRepository reconciliation
    ✔ fetched revision main@sha1:53f1413ba96cbf6154d9a4d8766adb64cd7be448
    ► annotating Kustomization flux-system in flux-system namespace
    ✔ Kustomization annotated
    ◎ waiting for Kustomization reconciliation
    ✔ applied revision main@sha1:53f1413ba96cbf6154d9a4d8766adb64cd7be448

  • Verify:

    • All should be in the ready state:
    flux get sources git -n flux-system
    NAME                     	REVISION             	SUSPENDED	READY	MESSAGE                                              
    app-manifests            	main@sha1:2c4e91b2   	False    	True 	stored artifact for revision 'main@sha1:2c4e91b2'   	
    coach-app-dev            	dev@sha1:9c3ba21c    	False    	True 	stored artifact for revision 'dev@sha1:9c3ba21c'    	
    flux-system              	main@sha1:96b9b2e1   	False    	True 	stored artifact for revision 'main@sha1:96b9b2e1'   	
    origin-ca-issuer-upstream	v0.12.1@sha1:86d908ed	False    	True 	stored artifact for revision 'v0.12.1@sha1:86d908ed'
    flux get kustomizations -n flux-system
    NAME                       	REVISION           	SUSPENDED	READY	MESSAGE                               
    app-source-kustomization   	main@sha1:2c4e91b2 	False    	True 	Applied revision: main@sha1:2c4e91b2 	
    coach-app-dev              	dev@sha1:9c3ba21c  	False    	True 	Applied revision: dev@sha1:9c3ba21c  	
    flux-system                	main@sha1:b4948359 	False    	True 	Applied revision: main@sha1:b4948359 	
    origin-ca-issuer-controller	trunk@sha1:debc90bd	False    	True 	Applied revision: trunk@sha1:debc90bd	
    origin-ca-issuer-crds      	trunk@sha1:debc90bd	False    	True 	Applied revision: trunk@sha1:debc90bd	
    origin-ca-issuer-ns        	main@sha1:b4948359 	False    	True 	Applied revision: main@sha1:b4948359 	
    origin-ca-issuer-rbac      	trunk@sha1:debc90bd	False    	True 	Applied revision: trunk@sha1:debc90bd
    • Confirm deployment:
    kubectl -n origin-ca-issuer get deploy,pods
    NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/origin-ca-issuer   1/1     1            1           4h27m

    NAME                                    READY   STATUS    RESTARTS   AGE
    pod/origin-ca-issuer-7c8b955544-6trhh   1/1     Running   0          4h27m
    • Confirm version:
    kubectl -n flux-system get gitrepository origin-ca-issuer-upstream -o jsonpath='{.status.artifact.revision}'; echo
    # expect: v0.12.1@sha1:...
    v0.12.1@sha1:86d908eda6c91815557c04b7f3e871ca4f0a0cce

1.4 - SOPS & Age (secrets encryption)​

SOPS is a Git-friendly encryption tool for Kubernetes manifests that encrypts only secret fields using Age; it isn’t built into Kubernetes itself, but Flux supports it natively so secrets are safely stored in Git and decrypted inside the cluster.

Create the cluster Secret sops-age in flux-system from your Mac.

Generate key (Mac):

mkdir -p ~/.sops
age-keygen -o ~/.sops/age.key
# Save the shown "public key: age1..." for .sops.yaml

Store the private key in the cluster so Flux can decrypt:

kubectl -n flux-system create secret generic sops-age \
  --from-file=age.agekey=$HOME/.sops/age.key \
  -o yaml --dry-run=client | kubectl apply -f -

Note: age keys don’t expire automatically. Rotate when you choose (see Operations).

1.5 - Cloudflare stack cloudflare​

App repo: muppit-apps/cloudflare.git.

1.5.1 Prepare the App Repo in GitLab​

On your Mac.

1.5.1.1 Create the GitLab project​

  • Project: muppit-apps/cloudflare
  • SSH URL (example): ssh://git-ssh.reids.net.au/muppit-apps/cloudflare.git
    (scp style also works: git@git-ssh.reids.net.au:muppit-apps/cloudflare.git)

1.5.1.2 Give Flux read access​

Deploy Key.

  • In the new GitLab project → Settings → Repository → Deploy Keys
  • Enable the same Flux deploy key you already use (flux-ssh-auth), or add its public key again.
  • If you must recover the public key from the cluster (prefer the original copy instead): 
    kubectl -n flux-system get secret flux-ssh-auth \
      -o jsonpath='{.data.identity}' | base64 -d > /tmp/flux_id
    ssh-keygen -y -f /tmp/flux_id > /tmp/flux_id.pub
    # paste /tmp/flux_id.pub into Deploy Keys (Read-only)

1.5.2 Clone the app repo​

Create repo structure.

git clone https://gitlab.reids.net.au/muppit-apps/cloudflare.git
cd cloudflare
mkdir -p k8s/prod

Repo layout on your Mac:

tree -a -I '.git|.DS_Store'   # Initial structure

Add a SOPS policy at repo root:

# .sops.yaml
creation_rules:
  - path_regex: k8s/.*\.enc\.ya?ml$
    encrypted_regex: '^(data|stringData)$'
    age: ['AGE_PUBLIC_KEY_HERE']

Replace AGE_PUBLIC_KEY_HERE with your public key (the one starting with age1...).

1.5.3 Create manifests (plaintext)​

Create the following files under k8s/prod/.

1.5.3.1 Minimal RBAC for Origin CA Issuer​

Namespace-scoped.

# k8s/prod/10-rbac-origin-issuer-minimal.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: origin-ca-issuer-writer
  namespace: cloudflare
rules:
  - apiGroups: ["cert-manager.k8s.cloudflare.com"]
    resources: ["originissuers"]
    verbs: ["get","list","watch"]
  - apiGroups: ["cert-manager.k8s.cloudflare.com"]
    resources: ["originissuers/status"]
    verbs: ["get","update","patch"]
  - apiGroups: ["cert-manager.io"]
    resources: ["certificaterequests"]
    verbs: ["get","list","watch"]
  - apiGroups: ["cert-manager.io"]
    resources: ["certificaterequests/status"]
    verbs: ["get","update","patch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get","list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: origin-ca-issuer-writer
  namespace: cloudflare
subjects:
  - kind: ServiceAccount
    name: originissuer-control
    namespace: origin-ca-issuer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: origin-ca-issuer-writer

1.5.3.2 OriginIssuer (ECC)​

# k8s/prod/21-originissuer.yaml
apiVersion: cert-manager.k8s.cloudflare.com/v1
kind: OriginIssuer
metadata:
  name: cf-origin
  namespace: cloudflare
spec:
  requestType: OriginECC    # or OriginRSA
  auth:
    tokenRef:
      name: cfapi-token
      key: key

1.5.3.3 Origin CA root pool (ECC)​

# k8s/prod/30-configmap-origin-ca-pool.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: cf-origin-capool
  namespace: cloudflare
data:
  origin_ca.pem: |
    # Paste the ECC Origin CA root PEM here (from Cloudflare docs)
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

1.5.3.4 cloudflared config​

Tunnel → ingress-nginx over TLS.

# k8s/prod/40-configmap-cloudflared.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: cloudflared
  namespace: cloudflare
data:
  config.yaml: |
    tunnel: <TUNNEL-UUID>            # from `cloudflared tunnel list`
    credentials-file: /etc/cloudflared/creds/credentials.json
    metrics: 0.0.0.0:2000

    ingress:
      # (optional) Block WP admin to public
      - hostname: muppit.au
        path: ^/(wp-login\.php|wp-admin(?:/.*)?$)
        service: http_status:403

      # Send to ingress-nginx over TLS and verify with Origin CA
      - hostname: muppit.au
        service: https://ingress-nginx-controller.ingress-nginx.svc.cluster.local:443
        originRequest:
          originServerName: muppit.au
          caPool: /etc/cloudflared/certs/origin_ca.pem

      # Catch-all
      - service: http_status:404

1.5.3.5 cloudflared Deployment​

PIN CONTAINER TAG.

Use a pinned image tag (using crane to find a suitable version).

# k8s/prod/42-deploy-cloudflared.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudflared
  namespace: cloudflare
spec:
  replicas: 1
  selector:
    matchLabels: { app: cloudflared }
  template:
    metadata:
      labels: { app: cloudflared }
    spec:
      serviceAccountName: default
      containers:
        - name: cloudflared
          image: cloudflare/cloudflared:2025.9.1   # <— PINNED example
          args: ["tunnel","--config","/etc/cloudflared/config/config.yaml","run"]
          ports:
            - containerPort: 2000
              name: metrics
          volumeMounts:
            - name: config
              mountPath: /etc/cloudflared/config
            - name: creds
              mountPath: /etc/cloudflared/creds
            - name: capool
              mountPath: /etc/cloudflared/certs
      volumes:
        - name: config
          configMap: { name: cloudflared, items: [{ key: config.yaml, path: config.yaml }] }
        - name: creds
          secret:   { secretName: cloudflared-credentials }
        - name: capool
          configMap: { name: cf-origin-capool, items: [{ key: origin_ca.pem, path: origin_ca.pem }] }

1.5.3.6 Certificate (Origin CA) for muppit.au​

# k8s/prod/50-certificate-muppit.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: muppit-au-origin
  namespace: cloudflare
spec:
  secretName: muppit-au-origin-tls
  dnsNames: ["muppit.au"]
  issuerRef:
    group: cert-manager.k8s.cloudflare.com
    kind: OriginIssuer
    name: cf-origin

1.5.3.7 Hello test app + Ingress​

# k8s/prod/60-hello-app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-app
  namespace: cloudflare
spec:
  replicas: 1
  selector:
    matchLabels: { app: hello }
  template:
    metadata:
      labels: { app: hello }
    spec:
      containers:
        - name: hello
          image: gcr.io/google-samples/hello-app:2.0
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: hello-service
  namespace: cloudflare
spec:
  type: ClusterIP
  selector: { app: hello }
  ports:
    - name: http
      port: 80
      targetPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-muppit-au
  namespace: cloudflare
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "16m"
spec:
  ingressClassName: nginx
  tls:
    - hosts: ["muppit.au"]
      secretName: muppit-au-origin-tls
  rules:
    - host: muppit.au
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: hello-service
                port:
                  number: 80

1.5.3.8 Network Policies​

# k8s/prod/70-networkpolicies.yaml

# Allow only ingress-nginx to reach hello pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-nginx-to-hello
  namespace: cloudflare
spec:
  podSelector:
    matchLabels: { app: hello }
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
      ports:
        - protocol: TCP
          port: 8080

---
# (Optional) Default deny for other workloads in cloudflare ns
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: cloudflare
spec:
  podSelector: {}
  policyTypes: ["Ingress","Egress"]

Label your ingress controller namespace (once):

kubectl label ns ingress-nginx ingress=nginx --overwrite
kubectl get ns ingress-nginx --show-labels

1.5.4 Create the two Secrets (plaintext files)​

Then encrypt with SOPS.

Cloudflare API token (replace the value):

cat > k8s/prod/20-secret-cfapi-token.enc.yaml << 'YAML'
apiVersion: v1
kind: Secret
metadata:
  name: cfapi-token
  namespace: cloudflare
type: Opaque
stringData:
  key: PASTE_YOUR_API_TOKEN
YAML

Tunnel credentials Secret:

  • Open your tunnel credentials.json from cloudflared tunnel create (it may be named as <UUID>.json).
  • Ensure it is valid JSON (trim any trailing % etc.). Quick check: 
    jq -e . < /path/to/credentials.json >/dev/null && echo "OK JSON" || echo "Invalid JSON"
  • Put the entire JSON under key credentials.json (the key name must be exactly credentials.json):
cat > k8s/prod/41-secret-cloudflared-credentials.enc.yaml << 'YAML'
apiVersion: v1
kind: Secret
metadata:
  name: cloudflared-credentials
  namespace: cloudflare
type: Opaque
stringData:
  credentials.json: |
    { PASTE_THE_ENTIRE_JSON_ONE_INDENTED_BLOCK_HERE }
YAML

Encrypt the Secrets (on your Mac):

sops --encrypt --in-place k8s/prod/20-secret-cfapi-token.enc.yaml
sops --encrypt --in-place k8s/prod/41-secret-cloudflared-credentials.enc.yaml

Verify they’re encrypted:

head -n 20 k8s/prod/20-secret-cfapi-token.enc.yaml
head -n 20 k8s/prod/41-secret-cloudflared-credentials.enc.yaml
grep -n 'sops:' k8s/prod/*.enc.yaml

1.5.5 Commit and push​

git add .
git commit -m "init cloudflare stack (SOPS-encrypted secrets)"
git push

1.5.6 Confirm repo layout​

Final repo layout:

tree -a -I '.git|.DS_Store'
.
├── .sops.yaml
└── k8s
    └── prod
        ├── 00-namespace.yaml
        ├── 10-rbac-origin-issuer-minimal.yaml
        ├── 20-secret-cfapi-token.enc.yaml
        ├── 21-originissuer.yaml
        ├── 30-configmap-origin-ca-pool.yaml
        ├── 40-configmap-cloudflared.yaml
        ├── 41-secret-cloudflared-credentials.enc.yaml
        ├── 42-deploy-cloudflared.yaml
        ├── 50-certificate-muppit.yaml
        ├── 60-hello-app.yaml
        └── 70-networkpolicies.yaml

3 directories, 12 files

1.5.7 Add cloudflare to flux-config​

In the cloudflare GitLab project → Settings → Repository → Deploy Keys → Privately accessible deploy keys

  • Enable the same Flux deploy key you already use (flux-ssh-auth).
  • When enabled it will appear under Enabled deploy keys.

On the Mac, navigate to the cloned flux-config project and pull any updates to ensure it is up to date:

cd ~/Projects/flux-config
git pull

1.5.8 Create manifests (plaintext)​

Create the following files under clusters/my-cluster/cloudflare/.

1.5.8.1 Namespace (via Flux)​

# clusters/my-cluster/cloudflare/ns/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: cloudflare

1.5.8.2 Kustomization​

# clusters/my-cluster/cloudflare/00-kustomization-ns.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: cloudflare-ns
  namespace: flux-system
spec:
  interval: 10m
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  path: ./clusters/my-cluster/cloudflare/ns
  timeout: 2m

1.5.8.3 Point Flux to your app repo​

muppit-apps/cloudflare. Uses Deploy Key (flux-ssh-auth) and the SSH URL via git-ssh.reids.net.au.

# clusters/my-cluster/cloudflare/source.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: cloudflare-app
  namespace: flux-system
spec:
  interval: 5m
  timeout: 60s
  url: ssh://git-ssh.reids.net.au/muppit-apps/cloudflare.git
  ref:
    branch: main
  secretRef:
    name: flux-ssh-auth

1.5.8.4 Kustomization to apply the app repo​

Decrypt SOPS.

Applies ./k8s/prod from the app repo, targets the cloudflare namespace, decrypts SOPS secrets using the sops-age secret in flux-system, and waits for the NS + origin-ca-issuer controller (so certificates can issue).

# clusters/my-cluster/cloudflare/10-kustomization-app.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: cloudflare-app
  namespace: flux-system
spec:
  dependsOn:
    - name: cloudflare-ns
    - name: origin-ca-issuer-controller
  interval: 10m
  timeout: 4m
  prune: true
  wait: true
  sourceRef:
    kind: GitRepository
    name: cloudflare-app
    namespace: flux-system
  path: ./k8s/prod
  targetNamespace: cloudflare
  decryption:
    provider: sops
    secretRef:
      name: sops-age

1.5.8.5 Add the default kustomizations​

# clusters/my-cluster/cloudflare/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - source.yaml
  - 00-kustomization-ns.yaml
  - 10-kustomization-app.yaml
# clusters/my-cluster/cloudflare/ns/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml

1.5.8.6 Ensure the root aggregator includes cloudflare/​

Edit the aggregator clusters/my-cluster/kustomization.yaml and add cloudflare to the resources list.

# clusters/my-cluster/kustomization.yaml (in flux-config repo)
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ./flux-system
  - ./apps
  - ./dev
#  - ./prod
  - ./origin-ca-issuer
  - ./cloudflare

1.5.8.7 Check the cloudflare flux-config file structure​

tree -a -I '.git|.DS_Store' clusters/my-cluster/cloudflare
clusters/my-cluster/cloudflare
├── .gitkeep
├── 00-kustomization-ns.yaml
├── 10-kustomization-app.yaml
├── kustomization.yaml
├── ns
│   ├── kustomization.yaml
│   └── namespace.yaml
└── source.yaml

2 directories, 7 files

1.5.9 Commit and push​

From the flux-config repo on workstation:

git add clusters/my-cluster/{kustomization.yaml,cloudflare}
git commit -m "Add Flux source+kustomization for cloudflare app + NS"
git push

1.5.10 Reconcile and watch​

  • Force reconciliation instead of waiting for the timer:

    flux reconcile source git flux-system -n flux-system \
      && flux reconcile kustomization flux-system -n flux-system --with-source

  • Check that all Kustomizations are applied and ready

    flux get kustomizations -n flux-system
    NAME                       	REVISION             	SUSPENDED	READY	MESSAGE                                 
    app-source-kustomization   	main@sha1:2c4e91b2   	False    	True 	Applied revision: main@sha1:2c4e91b2   	
    cloudflare-app             	main@sha1:cc75d871   	False    	True 	Applied revision: main@sha1:cc75d871   	
    cloudflare-ns              	main@sha1:59e6cffb   	False    	True 	Applied revision: main@sha1:59e6cffb   	
    coach-app-dev              	dev@sha1:9c3ba21c    	False    	True 	Applied revision: dev@sha1:9c3ba21c    	
    flux-system                	main@sha1:59e6cffb   	False    	True 	Applied revision: main@sha1:59e6cffb   	
    origin-ca-issuer-controller	v0.12.1@sha1:86d908ed	False    	True 	Applied revision: v0.12.1@sha1:86d908ed	
    origin-ca-issuer-crds      	v0.12.1@sha1:86d908ed	False    	True 	Applied revision: v0.12.1@sha1:86d908ed	
    origin-ca-issuer-ns        	main@sha1:59e6cffb   	False    	True 	Applied revision: main@sha1:59e6cffb   	
    origin-ca-issuer-rbac      	v0.12.1@sha1:86d908ed	False    	True 	Applied revision: v0.12.1@sha1:86d908e

  • Check the cloudflare namespace:

    kubectl get all -n cloudflare
    NAME                              READY   STATUS    RESTARTS   AGE
    pod/cloudflared-d869ccdc9-64zzn   1/1     Running   0          9m4s
    pod/hello-app-859ddb5df-g74mj     1/1     Running   0          9m4s

    NAME                    TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
    service/hello-service   ClusterIP   10.50.70.51   <none>        80/TCP    9m4s

    NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/cloudflared   1/1     1            1           9m4s
    deployment.apps/hello-app     1/1     1            1           9m4s

    NAME                                    DESIRED   CURRENT   READY   AGE
    replicaset.apps/cloudflared-d869ccdc9   1         1         1       9m4s
    replicaset.apps/hello-app-859ddb5df     1         1         1       9m4s

  • Check the config maps and secret

    kubectl -n cloudflare get cm,secret
    NAME                         DATA   AGE
    configmap/cf-origin-capool   1      31m
    configmap/cloudflared        1      31m
    configmap/kube-root-ca.crt   1      31m

    NAME                             TYPE                DATA   AGE
    secret/cfapi-token               Opaque              1      31m
    secret/cloudflared-credentials   Opaque              1      31m
    secret/muppit-au-origin-tls      kubernetes.io/tls   3      31m

  • Check the certificates

    kubectl -n cloudflare get certificaterequests,certificate
    AME                                                    APPROVED   DENIED   READY   ISSUER      REQUESTER                                         AGE
    certificaterequest.cert-manager.io/muppit-au-origin-1   True                True    cf-origin   system:serviceaccount:cert-manager:cert-manager   30m

    NAME                                           READY   SECRET                 AGE
    certificate.cert-manager.io/muppit-au-origin   True    muppit-au-origin-tls   30m

  • Check from a workstation connected to the Internet, not LAN:

    curl -I https://muppit.au
    HTTP/2 200 
    date: Sat, 11 Oct 2025 14:44:15 GMT
    content-type: text/plain; charset=utf-8
    content-length: 65
    strict-transport-security: max-age=31536000; includeSubDomains
    cf-cache-status: DYNAMIC
    report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=9iVUuR%2BDx9gSDC8F6yPUk%2BdkmY2c4uRdkUotkA7n2NibSWB%2F%2FrazzX73lWUY7d%2FYn1HugFnBWiCGDSYMGVqPDuig513qB7M%3D"}]}
    nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
    server: cloudflare
    cf-ray: 98cf226b19858673-PER
    alt-svc: h3=":443"; ma=86400
    curl https://muppit.au
    Hello, world!
    Version: 2.0.0
    Hostname: hello-app-859ddb5df-g74mj

1.6 - Cloudflare DNS & Redirects​

  • Ensure muppit.au and www records exist and are proxied (orange cloud).

  • If you want www → root redirect, Cloudflare has a ready template (“Redirect from WWW to root”).
    If it warns that www isn’t proxied, remove duplicate DNS entries and let the template create/manage the www CNAME + rule together.

  • For the tunnel DNS route (optional CLI):

    cloudflared tunnel route dns <TUNNEL_NAME> muppit.au
    cloudflared tunnel route dns <TUNNEL_NAME> www.muppit.au

1.7 - Validate end‑to‑end​

1.7.1 Config drift checks​

kubectl diff -f k8s/prod/30-configmap-cloudflared.yaml     # expect no diff if applied

1.7.2 Origin CA certificate health​

kubectl -n cloudflare wait certificate/muppit-au-origin --for=condition=Ready --timeout=3m
kubectl -n cloudflare get secret muppit-au-origin-tls

Inspect cert:

kubectl -n cloudflare get secret muppit-au-origin-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/muppit-origin.crt
openssl x509 -in /tmp/muppit-origin.crt -noout -subject -issuer -startdate -enddate -serial

1.7.3 TLS to ingress‑nginx with Origin CA​

kubectl -n cloudflare get cm cf-origin-capool -o jsonpath='{.data.origin_ca\.pem}' > /tmp/origin_ca.pem
SVC_IP=$(kubectl -n ingress-nginx get svc ingress-nginx-controller -o jsonpath='{.spec.clusterIP}')
curl -sv --resolve muppit.au:443:$SVC_IP https://muppit.au/ --cacert /tmp/origin_ca.pem -o /dev/null

1.7.4 Cloudflared connectivity​

kubectl -n cloudflare logs deploy/cloudflared --tail=100
kubectl -n cloudflare port-forward deploy/cloudflared 2000:2000 &
curl -s localhost:2000/metrics | head

1.7.5 Public check & redirect​

curl -I https://muppit.au/
curl -I 'https://www.muppit.au/test?x=1'   # expect 301 → https://muppit.au/test?x=1

1.8 - Troubleshooting snippets​

  • No cert issued but OriginIssuer Ready → check RBAC (controller must update originissuers/status and certificaterequests/status) and the token Secret name/keys.
  • Ingress 404 → confirm ingress class, host, TLS secret name, and that cloudflared routes to the right Service (ClusterIP + port 443) with correct originServerName.
  • NetworkPolicy gotchas → introducing a restrictive policy without allowing traffic from ingress-nginx to app will break routing. Validate with a test pod in the nginx namespace hitting the service DNS name.

1.9 - Cleanup (full remove) — Flux‑first​

warning

Only follow this section if you want to completely remove origin-ca-issuer. Prefer declarative cleanup: delete Git definitions and let Flux prune.

1.9.1 Remove Cloudflare stack​

warning

This will remove the Cloudflare stack.

  • In infra repo: delete clusters/my-cluster/cloudflare/{source.yaml,kustomization.yaml} or comment them out.
  • Commit & push, then: 
    flux reconcile kustomization flux-system -n flux-system --with-source
  • Verify resources in cloudflare namespace are pruned: 
    kubectl get all -n cloudflare

1.9.2 Remove Origin CA Issuer​

warning

This will remove the Cloudflare Origin CA Issuer.

  • In infra repo: delete files under clusters/my-cluster/origin-ca-issuer/ in this exact order (or remove the top‑level kustomization.yaml resources):
    • 30-kustomization-controller.yaml
    • 20-kustomization-rbac.yaml
    • 10-kustomization-crds.yaml
    • 00-kustomization-ns.yaml
    • source.yaml
  • Commit & push; reconcile Flux root: 
    flux reconcile kustomization flux-system -n flux-system --with-source
  • Confirm the controller namespace empties: 
    kubectl -n origin-ca-issuer get all || true

1.9.3 Manual emergency removal (only if needed)​

warning

Manual removal process.

# Remove Kustomizations and source
kubectl -n flux-system delete kustomization muppit-cloudflare origin-ca-issuer-{controller,rbac,crds,ns} --ignore-not-found
kubectl -n flux-system delete gitrepository muppit-cloudflare origin-ca-issuer-upstream --ignore-not-found

# Remove namespaces (and all contained objects)
kubectl delete ns cloudflare origin-ca-issuer --ignore-not-found

# RBAC leftovers
kubectl get clusterrole,clusterrolebinding | egrep -i 'origin(-|)ca(-|)issuer|cloudflare|cert-manager' || true
# Delete any stray items you don’t want:
# kubectl delete clusterrole <NAME>
# kubectl delete clusterrolebinding <NAME>

1.10 - Operations (rotations & routine tasks)​

Rotate Origin CA certificate (on demand)

kubectl -n cloudflare delete secret muppit-au-origin-tls
kubectl -n cloudflare wait certificate/muppit-au-origin --for=condition=Ready --timeout=3m

Update tunnel config

  • Edit k8s/prod/40-configmap-cloudflared.yaml (or credentials Secret), commit → Flux rolls the Deployment.

Add another site

  • Add DNS/route in Cloudflare (proxied or cloudflared tunnel route dns <tunnel> <host>).
  • Add new Certificate + Ingress (and optional NetPol) in k8s/prod/.
  • Commit/push; verify with SNI curl against ingress ClusterIP.

age key rotation (no downtime)

  1. Generate a new keypair:
age-keygen -o ~/.sops/age-new.key
age-keygen -y ~/.sops/age-new.key   # copy the new public key
  1. Add new public key to .sops.yaml alongside the old one; re‑encrypt all:
find k8s/prod -name '*.enc.yaml' -print0 | xargs -0 -n1 sops --encrypt --in-place
git commit -am "SOPS: add new recipient" && git push
  1. Update Flux secret to include both private keys:
cat ~/.sops/age.key ~/.sops/age-new.key > /tmp/age-rotating.keys
kubectl -n flux-system create secret generic sops-age \
  --from-file=age.agekey=/tmp/age-rotating.keys \
  -o yaml --dry-run=client | kubectl apply -f -
  1. Remove old recipient from .sops.yaml, re‑encrypt all again, and update Flux secret to contain only the new key.

Rotate Cloudflare API token

  • Create a new token in Cloudflare (same scope).
  • sops k8s/prod/20-secret-cfapi-token.enc.yaml → replace value → save → commit/push.
  • Check: 
    kubectl -n cloudflare get secret cfapi-token -o yaml | grep -E 'name:|namespace:' -n
    kubectl -n cloudflare get originissuer cf-origin -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'

Rotate cloudflared tunnel credentials

  • Replace JSON in k8s/prod/41-secret-cloudflared-credentials.enc.yaml via sops.
  • Commit/push; check kubectl -n cloudflare logs deploy/cloudflared --since=3m.

Hygiene

  • kubectl diff -f <file-or-dir> before manual applies
  • flux get kustomizations -A and flux events -A --since=1h
  • kubectl -n cloudflare get secrets to audit what exists

1.11 - Useful commands​

Check Flux trees

flux get kustomizations -A
flux tree kustomization flux-system -n flux-system

Diff any manifest against live

kubectl diff -f path/to/file.yaml

Find latest cloudflared tag (calendar‑versioned)

crane ls cloudflare/cloudflared | grep -E '^[0-9]{4}\.[0-9]+(\.[0-9]+)?$' | sort -V | tail -n1

Pin Origin CA Issuer to a new tag

vi clusters/my-cluster/origin-ca-issuer/source.yaml   # change spec.ref.tag:
git add . && git commit -m "bump origin-ca-issuer to vX.Y.Z" && git push
flux reconcile source git origin-ca-issuer-upstream -n flux-system && \
flux reconcile kustomization origin-ca-issuer-crds -n flux-system && \
flux reconcile kustomization origin-ca-issuer-rbac -n flux-system && \
flux reconcile kustomization origin-ca-issuer-controller -n flux-system

Repo layout preview

tree -a -I '.git|.DS_Store'

Cloudflare redirect test

curl -I 'https://www.muppit.au/test?x=1'
# Expect: HTTP/2 301 with Location: https://muppit.au/test?x=1