MetalLB helm with FRR sidecar

Process to deploy MetalLB via helm. I will be using BGP with my upstream router and enabling FRR for routing.

tip

I prefer this option over installing the operator with FRR as it consumes far less resources and I can still edit the route-maps once deployed.

When you run eBGP between your worker nodes (via MetalLB FRR) and a FortiGate, the FortiGate will, by default, re-advertise routes learned from one eBGP neighbour (e.g. Worker A) to other eBGP neighbours in the same AS (e.g. Worker B)
Cisco and Check Point, on the other hand, do not do this by default. They follow a stricter interpretation of RFC 4271 section 9.1.3, which says an eBGP route should only be re-advertised to other eBGP peers unless filters or policies explicitly allow it, and even then, not when the peers are in the same remote AS

note

Fortigate's implementation of BGP is slightly different when compared with Cisco or Checkpoint.

Router BGP

On the upstream router enable BGP and configure as follows:

BGP global settings

Setting	Value
Local AS	64500
Router ID	192.168.1.254

AS path lists

Name	Action	Expression
from-AS64502	permit	`_64502$`

Route maps

Name	Statement	Action	Match condition
block-readvertised-64502	1	deny	AS path list: `from-AS64502`
block-readvertised-64502	2	permit	-

Neighbors summary

Neighbor IP	Remote AS	Interface	Update source	IPv4 active	Route map out	Soft reconfig	Capabilities
192.168.1.207	64502	LAN	LAN	Yes	block-readvertised-64502	Enabled	graceful restart, route refresh

Neighbor 192.168.1.207 details

Field	Value
IP	192.168.1.207
Remote AS	64502
Interface	LAN
Update source	LAN
Activate IPv4	Yes
Route map out	block-readvertised-64502
Soft reconfiguration	Enabled
Capabilities	graceful restart, route refresh

Helm install with values

Create a values file specifying the path and server:

vi ~/manifests/metallb-system-metallb-values.yaml

crds:
  enabled: true

speaker:
  frr:
    enabled: true
  nodeSelector:
    metallb: enabled

Install metallb using helm, referencing the values file:

helm upgrade --install metallb metallb/metallb \
--namespace metallb-system --create-namespace \
--values ~/manifests/metallb-system-metallb-values.yaml

Release "metallb" does not exist. Installing it now.
NAME: metallb
LAST DEPLOYED: Sun Oct  5 19:49:40 2025
NAMESPACE: metallb-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
MetalLB is now running in the cluster.

Now you can configure it via its CRs. Please refer to the metallb official docs
on how to use the CRs.

BGP peer configuration

Prepare the BGP peer

vi metallb-bgp-peer.yaml

apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
  name: edge-router
  namespace: metallb-system
spec:
  myASN: 64502
  peerASN: 64500
  peerAddress: 192.168.1.254

Apply the BGP peer configuration:

kubectl apply -f metallb-bgp-peer.yaml

bgppeer.metallb.io/edge-router created

Prepare the IP address pool

vi metallb-ip-pool.yaml

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: pool-10-70-1
  namespace: metallb-system
spec:
  addresses:
    - 10.70.1.0/24
  avoidBuggyIPs: true

Apply the IP pool configuration

kubectl apply -f metallb-ip-pool.yaml

ipaddresspool.metallb.io/pool-10-70-1 created

Prepare the BGP advertisment

vi metallb-bgp-adv.yaml

apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
  name: adv-10-70-1
  namespace: metallb-system
spec:
  ipAddressPools:
    - pool-10-70-1
  aggregationLength: 24

Apply the BGP advertisment configuration:

kubectl apply -f metallb-bgp-adv.yaml

bgpadvertisement.metallb.io/adv-10-70-1 created

Label the nodes that speak BGP. In this instance only worker-gpu-1:
```
kubectl label node worker-gpu-1 metallb=enabled
```

Test BGP

Create an app based on Google's hello app and configure it to use the load balancer service:

vi hello-app-lb.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-app
  namespace: dev
spec:
  selector:
    matchLabels:
      app: hello
  replicas: 1
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
        - name: hello
          image: 'gcr.io/google-samples/hello-app:2.0'
---
apiVersion: v1
kind: Service
metadata:
  name: hello-service
  namespace: dev
  labels:
    app: hello
spec:
  type: LoadBalancer
  selector:
    app: hello
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP

Apply the app:

kubectl apply -f hello-app-lb.yaml

deployment.apps/hello-app created
service/hello-service created

Confirm an external IP has been allocated

kubectl get svc -n dev

NAME            TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
hello-service   LoadBalancer   10.70.168.159   10.70.1.1     80:32497/TCP   7m57s

Check using curl from both the master node and your workstation:

curl http://10.70.1.1

Hello, world!
Version: 2.0.0
Hostname: hello-app-c98b499bb-t5w4p

Open browser and navigate to: http://10.70.1.1

BGP verification (from k8s side)

See which prefixes we advertise to the peer and what we receive

POD=$(kubectl -n metallb-system get pods -o name | grep speaker | head -1 | sed 's#pod/##')
kubectl -n metallb-system exec -it "$POD" -c frr -- vtysh -c "show bgp summary"

IPv4 Unicast Summary (VRF default):
BGP router identifier 192.168.1.207, local AS number 64502 vrf-id 0
BGP table version 1
RIB entries 1, using 96 bytes of memory
Peers 1, using 13 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
192.168.1.254  4      64500        10        10        1    0    0 00:05:46            0        1 N/A

Total number of neighbors 1

kubectl -n metallb-system exec -it "$POD" -c frr -- vtysh -c "show bgp ipv4 unicast neighbors 192.168.1.254 advertised-routes"

BGP table version is 1, local router ID is 192.168.1.207, vrf id 0
Default local pref 100, local AS 64502
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *> 10.70.1.0/24     0.0.0.0                  0         32768 i

kubectl -n metallb-system exec -it "$POD" -c frr -- vtysh -c "show bgp ipv4 unicast neighbors 192.168.1.254 routes"

kubectl -n metallb-system exec -it "$POD" -c frr -- vtysh -c "show running-config"

Building configuration...

Current configuration:
!
frr version 9.1_git
frr defaults traditional
hostname worker-gpu-1
log file /etc/frr/frr.log informational
log timestamp precision 3
service integrated-vtysh-config
!
router bgp 64502
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 bgp graceful-restart preserve-fw-state
 no bgp network import-check
 neighbor 192.168.1.254 remote-as 64500
 !
 address-family ipv4 unicast
  network 10.70.1.0/24
  neighbor 192.168.1.254 activate
  neighbor 192.168.1.254 route-map 192.168.1.254-in in
  neighbor 192.168.1.254 route-map 192.168.1.254-out out
 exit-address-family
exit
!
ip prefix-list 192.168.1.254-allowed-ipv4 seq 1 permit 10.70.1.0/24
!
ipv6 prefix-list 192.168.1.254-allowed-ipv6 seq 1 deny any
!
route-map 192.168.1.254-in deny 20
exit
!
route-map 192.168.1.254-out permit 1
 match ip address prefix-list 192.168.1.254-allowed-ipv4
exit
!
route-map 192.168.1.254-out permit 2
 match ipv6 address prefix-list 192.168.1.254-allowed-ipv6
exit
!
end

Inbound routes are denied by default

info

Sidecar (speaker+FRR) mode: MetalLB’s CRDs don’t let you filter inbound routes.

MetalLB isn’t a transit router; it doesn’t use received prefixes for data-plane decisions. FRR will learn whatever the peer sends, but MetalLB ignores them for forwarding. You can tune session knobs (ASN, holdtime, BFD, srcAddress, node selectors), but not import policy.

Persist via MetalLB CRDs
- Express everything in MetalLB’s own CRs (these are the metallb.io ones, not frrk8s.metallb.io):
  - IPAddressPool: the LB ranges (incl. avoidBuggyIPs, autoAssign, etc.)
  - BGPAdvertisement: what to advertise (communities, aggregation length, localpref, etc.)
  - BGPPeer: who to peer with (peer address/ASN, my ASN, holdtime, srcAddress, BFD, node selectors)
- The speaker converts those CRs into FRR config on startup and on any change. That’s the supported persistence mechanism in sidecar mode
What you can’t persist in sidecar mode
- Arbitrary FRR statements (custom route-maps/prefix-lists beyond what MetalLB models)
- vtysh commands run by hand or “write” commands inside the container

With the FRR sidecar you can’t make manual vtysh edits persistent. The speaker regenerates FRR config from Kubernetes objects and overwrites whatever you hand-type; a pod restart wipes it too.

However, although they will not be persistent, it is possible to configure the inbound BGP route-map while operating in sidecar mode.

Get the pod name

POD=$(kubectl -n metallb-system get pods -o name | grep speaker | head -1 | sed 's#pod/##')

Then confirm by running a show command

kubectl -n metallb-system exec -it "$POD" -c frr -- vtysh -c "show bgp summary"

IPv4 Unicast Summary (VRF default):
BGP router identifier 192.168.1.207, local AS number 64502 vrf-id 0
BGP table version 1
RIB entries 1, using 96 bytes of memory
Peers 1, using 13 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
192.168.1.254  4      64500        10        10        1    0    0 00:05:46            0        1 N/A

Connect to the command line of the frr process within the pod

kubectl -n metallb-system exec -it "$POD" -c frr -- vtysh

Hello, this is FRRouting (version 9.1_git).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

Now you can execute commands, such as show running-config, very similar to a Cisco router:

show running-config

Building configuration...

Current configuration:
!
frr version 9.1_git
frr defaults traditional
hostname worker-gpu-1
log file /etc/frr/frr.log informational
log timestamp precision 3
service integrated-vtysh-config
!
router bgp 64502
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 bgp graceful-restart preserve-fw-state
 no bgp network import-check
 neighbor 192.168.1.254 remote-as 64500
 !
 address-family ipv4 unicast
  network 10.70.1.0/24
  neighbor 192.168.1.254 activate
  neighbor 192.168.1.254 route-map 192.168.1.254-in in
  neighbor 192.168.1.254 route-map 192.168.1.254-out out
 exit-address-family
exit
!
ip prefix-list 192.168.1.254-allowed-ipv4 seq 1 permit 10.70.1.0/24
!
ipv6 prefix-list 192.168.1.254-allowed-ipv6 seq 1 deny any
!
route-map 192.168.1.254-in deny 20
exit
!
route-map 192.168.1.254-out permit 1
 match ip address prefix-list 192.168.1.254-allowed-ipv4
exit
!
route-map 192.168.1.254-out permit 2
 match ipv6 address prefix-list 192.168.1.254-allowed-ipv6
exit
!
end

Enter configuration mode:
```
configure terminal
```

Change the BGP configuration, by adding a prefix-list and a route-map entry. Also enable soft reconfiguration on the BGP peer and save the config:

worker-gpu-1(config)# ip prefix-list 192.168.1.254-allowed-ipv4-in seq 1 permit any
worker-gpu-1(config)# route-map 192.168.1.254-in permit 10
worker-gpu-1(config-route-map)# match ip address prefix-list 192.168.1.254-allowed-ipv4-in
worker-gpu-1(config-route-map)# exit
worker-gpu-1(config)# router bgp 64502
worker-gpu-1(config-router)#  neighbor 192.168.1.254 soft-reconfiguration inbound
worker-gpu-1(config-router)# end
worker-gpu-1# write
Note: this version of vtysh never writes vtysh.conf
Building Configuration...
Integrated configuration saved to /etc/frr/frr.conf
[OK]

Use a soft clear of the BGP session to make the changes take effect without resetting the peer:
```
worker-gpu-1# clear bgp ipv4 unicast 192.168.1.254 soft in
```

Confirm that the peer is now receiving 11 routes:

show ip bgp sum

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
192.168.1.254  4      64500       226       200       12    0    0 03:11:35           11        1 N/A

Check the routing table:

show ip route

Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B   0.0.0.0/0 [20/0] via 192.168.1.254, eno1, weight 1, 00:36:50
K>* 0.0.0.0/0 [0/0] via 192.168.1.254, eno1, 03:46:39
...

List all resources in the metallb-system namespace

Get all resources

kubectl get all -n metallb-system

NAME                                      READY   STATUS    RESTARTS   AGE
pod/metallb-controller-5754956df6-mdh6r   1/1     Running   0          5m10s
pod/metallb-speaker-zb997                 4/4     Running   0          5m10s

NAME                              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/metallb-webhook-service   ClusterIP   10.70.156.55   <none>        443/TCP   5m10s

NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                            AGE
daemonset.apps/metallb-speaker   1         1         1       1            1           kubernetes.io/os=linux,metallb=enabled   5m10s

NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/metallb-controller   1/1     1            1           5m10s

NAME                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/metallb-controller-5754956df6   1         1         1       5m10s

Router BGP​

BGP global settings​

AS path lists​

Route maps​

Neighbors summary​

Neighbor 192.168.1.207 details​

Helm install with values​

BGP peer configuration​

Test BGP​

BGP verification (from k8s side)​

Inbound routes are denied by default​

List all resources in the metallb-system namespace​