Operations

info

This page covers Azure App Registration setup, smtp2graph configuration, Prometheus metrics, and troubleshooting.

Email relay series

1. Email relay
2. Architecture
3. Manifests
4. Flux integration
5. Operations - You are here

Azure App Registration

The smtp2graph component requires an Azure App Registration with Mail.Send permission to send emails via Microsoft Graph API.

Create the App Registration

Portal: https://entra.microsoft.com/#view/Microsoft_AAD_RegisteredApps/ApplicationsListBlade

1. New registration

   • Name: Email Relay
   • Supported account types: "Accounts in this organizational directory only"
   • Click Register

2. Copy IDs (from Overview page):

   • Application (client) ID
   • Directory (tenant) ID

3. Create Client Secret:

   • Go to Certificates & secrets → New client secret
   • Description: smtp2graph
   • Expires: 730 days (24 months)
   • Copy the Value immediately (shown only once)

4. Add API Permission:

   • Go to API permissions → Add a permission
   • Select Microsoft Graph → Application permissions
   • Search for Mail.Send → Select it → Add permissions
   • Click Grant admin consent for [tenant]

smtp2graph configuration

Create the config secret with your Azure credentials:

# k8s/prod/15-secret-smtp2graph-config.enc.yaml (before encryption)
apiVersion: v1
kind: Secret
metadata:
  name: smtp2graph-config
  namespace: email-relay
type: Opaque
stringData:
  config.yml: |
    mode: full
    send:
      appReg:
        tenant: <your-tenant-id>
        id: <your-client-id>
        secret: <your-client-secret>
      retryLimit: 3
      retryInterval: 5
    receive:
      port: 2525
      banner: "Email Relay"
      maxSize: 25m
      ipWhitelist:
        - 127.0.0.1
      allowedFrom:
        - you@yourdomain.com

Encrypt before committing:

sops -e -i k8s/prod/15-secret-smtp2graph-config.enc.yaml

Prometheus metrics

MX Validator metrics

Metric	Type	Description
mx_validator_accepted_total	Counter	Emails with valid MX records
mx_validator_rejected_total	Counter	Emails rejected (no MX)
mx_validator_connections_total	Counter	Total SMTP connections
mx_validator_active_connections	Gauge	Current active connections
mx_validator_ics_stripped_total	Counter	Emails with ICS calendar attachments stripped

Mailpit metrics

Metric	Type	Description
mailpit_messages	Gauge	Messages in database
mailpit_smtp_accepted_total	Counter	SMTP messages accepted
mailpit_smtp_rejected_total	Counter	SMTP messages rejected

Example Grafana queries

# Rejection rate per minute
rate(mx_validator_rejected_total[5m]) * 60

# Acceptance rate per minute
rate(mx_validator_accepted_total[5m]) * 60

# Total emails processed
mx_validator_accepted_total + mx_validator_rejected_total

# Rejection percentage
mx_validator_rejected_total / (mx_validator_accepted_total + mx_validator_rejected_total) * 100

# Messages waiting in Mailpit
mailpit_messages

# ICS stripping rate
rate(mx_validator_ics_stripped_total[5m]) * 60

Alerting rules

The PrometheusRule in Manifests defines:

Alert	Condition	Severity
MXValidatorHighRejectionRate	>1 rejection/min for 5m	Warning
EmailRelayDown	Pod not responding	Critical
MailpitHighMessageCount	>1000 messages	Warning

Accessing Mailpit UI

Mailpit provides a web UI for viewing all emails that pass through the relay.

Via port-forward

kubectl port-forward -n email-relay svc/email-relay 8025:8025
# Open http://localhost:8025

Via ingress (optional)

Create an ingress for internal network access:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mailpit-ingress
  namespace: email-relay
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - mailpit.example.local
  rules:
    - host: "mailpit.example.local"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: email-relay
                port:
                  number: 8025

Verification commands

Check pods

kubectl get pods -n email-relay
# Expected:
# email-relay-xxxxx   3/3   Running   0   ...

Check logs

# MX validator logs (shows accepted/rejected)
kubectl logs -n email-relay -l app=email-relay -c mx-validator -f

# Mailpit logs (shows relayed messages)
kubectl logs -n email-relay -l app=email-relay -c mailpit -f

# smtp2graph logs (shows Graph API delivery)
kubectl logs -n email-relay -l app=email-relay -c smtp2graph -f

Check metrics

# MX validator metrics
kubectl exec -n email-relay deployment/email-relay -c mx-validator -- \
  wget -q -O- http://localhost:9090/metrics | grep mx_validator

# Mailpit metrics
kubectl exec -n email-relay deployment/email-relay -c mx-validator -- \
  wget -q -O- http://localhost:9091/metrics | grep mailpit

Test MX validation

# Check if domain has MX records
dig +short MX example.com

# Test SMTP connection
kubectl run test-smtp --rm -it --restart=Never --image=busybox -- \
  sh -c "echo -e 'EHLO test\nQUIT' | nc email-relay.email-relay.svc.cluster.local 25"

Verify ICS stripping

# Check if ICS stripping is enabled
kubectl logs -n email-relay -l app=email-relay -c mx-validator --tail=20 | grep "ICS stripping"

# Check ICS stripped count
kubectl exec -n email-relay deployment/email-relay -c mx-validator -- \
  wget -q -O- http://localhost:9090/metrics | grep ics_stripped

# Compare attachments in Mailpit
# Emails to STRIP_ICS_RECIPIENTS should have 0 attachments
# Emails to other recipients should retain ICS attachments

Troubleshooting

Emails not sending

1. Check MX validator logs for rejections:

   kubectl logs -n email-relay -l app=email-relay -c mx-validator | grep -i rejected

2. Check smtp2graph logs for Graph API errors:

   kubectl logs -n email-relay -l app=email-relay -c smtp2graph | tail -50

3. Verify Azure App Registration:

   • Mail.Send permission granted
   • Admin consent given
   • Client secret not expired
   • Correct tenant ID

MX validation rejecting valid domains

The MX cache might be stale. Restart the pod to clear:

kubectl delete pod -n email-relay -l app=email-relay

Or check DNS resolution from the pod:

kubectl exec -n email-relay deployment/email-relay -c mx-validator -- \
  nslookup -type=MX gmail.com

Mailpit not showing emails

Check if mx-validator is forwarding to Mailpit:

kubectl logs -n email-relay -l app=email-relay -c mx-validator | grep -i accepted

Check Mailpit SMTP listener:

kubectl exec -n email-relay deployment/email-relay -c mailpit -- \
  wget -q -O- http://localhost:8025/api/v1/info

smtp2graph authentication errors

Check logs for OAuth errors:

kubectl logs -n email-relay -l app=email-relay -c smtp2graph | grep -i error

Common issues:

• "AADSTS700016": Client ID not found - verify the Application ID
• "AADSTS7000215": Invalid client secret - regenerate in Azure portal
• "AADSTS65001": No admin consent - grant consent in Azure portal

Pod stuck in CrashLoopBackOff

Check which container is failing:

kubectl describe pod -n email-relay -l app=email-relay

Check individual container logs:

kubectl logs -n email-relay -l app=email-relay -c mx-validator --previous
kubectl logs -n email-relay -l app=email-relay -c mailpit --previous
kubectl logs -n email-relay -l app=email-relay -c smtp2graph --previous

Force reconciliation

# Reconcile the source
flux reconcile source git email-relay -n flux-system

# Reconcile the app Kustomization
flux reconcile kustomization email-relay -n flux-system --with-source

Client secret rotation

When the Azure client secret expires (after 24 months):

1. Create new secret in Azure portal
2. Update the SOPS-encrypted secret:

   sops k8s/prod/15-secret-smtp2graph-config.enc.yaml
   # Update the secret value

3. Commit and push
4. Flux will reconcile and restart the pod

Using the relay from applications

Configure your application to use the relay:

env:
  - name: EMAIL_SERVER_HOST
    value: "email-relay.email-relay.svc.cluster.local"
  - name: EMAIL_SERVER_PORT
    value: "25"

See Cal.com scheduling for a complete integration example including email verification for spam protection.