Skip to main content

IdP internal SMTP

info

This runbook documents how secrets, wildcard TLS trust, and SMTP are handled for the identity-internal ZITADEL instance (auth.reids.net.au). It complements the deployment and backup/restore guides.

Identity provider series​

  1. IdP dual overview
  2. IdP dual architecture
  3. IdP internal deployment
  4. IdP internal console
  5. IdP internal SMTP - you are here
  6. IdP internal LDAP
  7. IdP internal OIDC
  8. IdP internal OAUTH2 proxy
  9. IdP internal backup and restore

Overview​

For the internal identity provider you:

  • Keep database, masterkey and LDAP credentials in Git via SOPS.
  • Treat SMTP as an application-level concern stored inside ZITADEL itself, not Kubernetes.
  • Rely on the cluster-wide wildcard TLS trust mechanism (trust-manager + Gatekeeper + SSL_CERT_FILE) instead of namespace-specific TLS plumbing.
  • Avoid deleting the identity-internal namespace in normal operations so the PostgreSQL PVC, secrets and backups remain intact.

This separation keeps GitOps focused on infrastructure and ZITADEL focused on identity and notification behaviour.

Secret layout​

Stored in Git (SOPS)​

All infrastructure-critical secrets for identity-internal live as SOPS-encrypted YAML in the identity-internal repo:

  • k8s/prod/20-secrets-db.enc.yaml
    • POSTGRES_USER
    • POSTGRES_PASSWORD
    • POSTGRES_DB
  • k8s/prod/21-secrets-zitadel.enc.yaml
    • masterkey (32-byte ZITADEL master key)
  • Any LDAP bind credentials required for the QNAP LDAP provider.

Characteristics:

  • Encrypted at rest in Git using the shared age key.
  • Decrypted at apply time by Flux using the sops-age secret.
  • Versioned like any other manifest, so you can see when credentials change.
warning

Never commit decrypted secrets. Only the .enc.yaml files live in Git. The age private key is stored outside the cluster (password manager / NAS) and injected as sops-age when bootstrapping Flux.

Not stored in Git​

These are intentionally not represented as Kubernetes Secrets or SOPS files:

  • SMTP username and password.
  • Any future API keys configured directly in the ZITADEL console (for example external IdPs, if you add them later).

Reason: these are treated as application-level configuration and are persisted in the ZITADEL database, which is already protected and backed up.

Wildcard TLS trust​

TLS trust for internal services (NAS LDAP, SMTP, etc.) is handled once at cluster level:

  • A wildcard certificate for *.reids.net.au is installed on the NAS and presented on TLS endpoints (for example nas.reids.net.au:636 for LDAPS).
  • trust-manager and Gatekeeper inject a CA bundle into nodes/pods so that the wildcard certificate is trusted.
  • ZITADEL receives trust via the SSL_CERT_FILE=/etc/ssl/certs/wildcard-reids.crt environment variable in the HelmRelease values.

Effects:

  • ZITADEL pods automatically trust nas.reids.net.au and other servers in the local domain without namespace-local CA secrets.
  • LDAP and SMTP providers only need correct hostnames and ports; no extra TLS configuration is required in identity-internal.

If TLS trust is broken, you fix it in the trust runbook (bundle + Gatekeeper Assign resources), not here.

SMTP configuration approach​

Why SMTP is not a Kubernetes Secret​

For the internal IdP you only have a single, self-hosted ZITADEL instance. Keeping SMTP inside ZITADEL has benefits:

  • One source of truth: SMTP is configured where it is used.
  • No duplication between console, secrets and Helm values.
  • Fewer high-value credentials in Git, even when encrypted.
  • Restores are simpler because SMTP is included automatically when you restore the database.

GitOps still manages everything needed to run ZITADEL, but not SMTP credentials themselves.

SMTP configured in the console​

Configure SMTP once via the ZITADEL UI at https://auth.reids.net.au:

  1. Open the console and log in as an admin.
  2. Go to the global settings for email (SMTP provider).
  3. Configure something like:
    • Host: EMAIL_SERVER_URL
    • Port: EMAIL_SERVER_PORT
    • TLS: enabled
    • User: SMTP account user (for example AUTHORISED_EMAIL_ACCOUNT)
    • Password: EMAIL_ACCOUNT_PASSWORD
    • From address: REPLY_EMAIL_ADDRESS
    • From name: "Internal Auth" (or similar)
  4. Send a test email and verify delivery.

Then:

  • Store the SMTP username and password in your password manager.
  • Do not copy them into Kubernetes Secrets or Git.

If the SMTP provider changes (for example new hostname or password), update it in the console and your password manager, then let normal database backups capture the new configuration.

Interaction with backup and restore​

The identity-internal backup/restore runbook uses PostgreSQL logical dumps written to NAS. Because SMTP is stored in the ZITADEL database:

  • A database backup already includes SMTP configuration.
  • A database restore recreates SMTP provider settings as they were at the time of the dump.

Implications:

  • After restoring ZITADEL from a backup, SMTP will work again as long as:
    • The SMTP server is reachable and still has the same credentials.
    • Wildcard TLS trust is restored at cluster level so ZITADEL trusts the SMTP server certificate.
  • There is no separate YAML or secret restore process for SMTP.

If you rotate SMTP credentials after a restore, you only need to:

  1. Update the SMTP provider in the ZITADEL console.
  2. Update the credentials in your password manager.
  3. Let the next database backup pick up the new values.

When to update this runbook​

Update this runbook when any of the following change:

  • SMTP provider details (hostname, port, TLS mode, from address pattern).
  • The way wildcard TLS trust is provided (for example if you move away from trust-manager or change the bundle path).
  • The secret layout for identity-internal (new SOPS files, different naming, additional LDAP providers).
  • You decide to manage SMTP via declarative configuration instead of the console (not currently planned).

Verification checklist​

Use this checklist when reviewing the identity-internal identity stack or after making changes.

Secrets layout​

  • k8s/prod/20-secrets-db.enc.yaml exists and decrypts correctly with SOPS.
  • k8s/prod/21-secrets-zitadel.enc.yaml exists and contains a stable masterkey.
  • LDAP bind credentials (if any) live only in SOPS-encrypted files and a password manager.
  • No SMTP credentials are present in Kubernetes Secrets or plain YAML in Git.
  • The age private key for SOPS is stored outside the cluster and recoverable.

Wildcard TLS trust​

  • Wildcard certificate for *.reids.net.au is valid and installed on NAS/SMTP endpoints.
  • trust-manager and Gatekeeper Assigns are applied and healthy.
  • ZITADEL pods have SSL_CERT_FILE=/etc/ssl/certs/wildcard-reids.crt set.
  • LDAP over LDAPS (ldaps://nas.reids.net.au:636) works from inside the cluster.
  • SMTP over TLS works via the ZITADEL SMTP test.

SMTP configuration​

  • SMTP provider is configured and marked healthy in the ZITADEL console.
  • Test email succeeds to an internal address.
  • SMTP credentials are stored only in the password manager and ZITADEL database.
  • Recent database backups exist that include the current SMTP provider settings.