Kubespray deployment
Running the playbooks and validating success.
Kubespray - Deploy the k8s cluster
- Before running the playbook check the haproxy log for activity from each master node, in this dev environment we only have one master node:
tail -f /var/log/haproxy.log - We are now ready to deploy the k8s cluster using the ansible playbook
- First ensure that all playbook commands are run from the kubespray directory:
cd ~/kubespray-devcluster/kubespray/ - This command will create the cluster
ansible-playbook -i inventory/devcluster/inventory.ini --become --become-user=root cluster.yml -u dev -K - Enter the
become password. This is the dev user's password - Wait for the playbook to run all tasks. This may take a while as it is dependant upon a number of factors, such as resources available and the number of nodes being deployed in the cluster
- Play recap:
PLAY RECAP ********************************************************************************************************************************
dev-m-v1 : ok=573 changed=137 unreachable=0 failed=0 skipped=883 rescued=0 ignored=5
dev-w-v1 : ok=434 changed=81 unreachable=0 failed=0 skipped=676 rescued=0 ignored=0
Saturday 04 October 2025 01:53:10 +0000 (0:00:00.026) 0:06:57.242 ******
===============================================================================
download : Download_container | Download image if required ------------------------------------------------------------------------ 39.33s
download : Download_file | Download item ------------------------------------------------------------------------------------------ 31.05s
download : Download_file | Download item ------------------------------------------------------------------------------------------ 30.99s
download : Download_container | Download image if required ------------------------------------------------------------------------ 22.86s
download : Download_file | Download item ------------------------------------------------------------------------------------------ 21.74s
kubernetes/kubeadm : Join to cluster if needed ------------------------------------------------------------------------------------ 15.98s
download : Download_file | Download item ------------------------------------------------------------------------------------------ 13.96s
kubernetes/control-plane : Kubeadm | Initialize first control plane node (1st try) ------------------------------------------------- 9.41s
download : Download_container | Download image if required ------------------------------------------------------------------------- 8.76s
kubernetes/control-plane : Control plane | wait for kube-scheduler ----------------------------------------------------------------- 8.66s
etcd : Restart etcd ---------------------------------------------------------------------------------------------------------------- 8.65s
download : Download_container | Download image if required ------------------------------------------------------------------------- 8.55s
download : Download_container | Download image if required ------------------------------------------------------------------------- 7.78s
system_packages : Manage packages -------------------------------------------------------------------------------------------------- 7.47s
download : Download_container | Download image if required ------------------------------------------------------------------------- 7.17s
download : Download_container | Download image if required ------------------------------------------------------------------------- 6.74s
download : Download_container | Download image if required ------------------------------------------------------------------------- 6.34s
container-engine/runc : Download_file | Download item ------------------------------------------------------------------------------ 5.22s
etcd : Configure | Check if etcd cluster is healthy -------------------------------------------------------------------------------- 5.17s
download : Download_file | Download item ------------------------------------------------------------------------------------------- 5.10s - Check the load balancer haproxy logs to confirm master node is up:
2025-10-04T01:52:18.838954+00:00 dev-lb-v1 haproxy[846]: backend apiserver has no server available!
2025-10-04T01:52:18.839019+00:00 dev-lb-v1 haproxy[846]: backend apiserver has no server available!
2025-10-04T01:52:18.839200+00:00 dev-lb-v1 haproxy[846]: 192.168.30.203:46530 [04/Oct/2025:01:52:17.424] apiserver apiserver/dev-m-v1 1/1003/1414 2762 -- 15/15/14/14/1 0/0
2025-10-04T01:52:18.840634+00:00 dev-lb-v1 haproxy[846]: 192.168.30.203:46514 [04/Oct/2025:01:52:15.306] apiserver apiserver/dev-m-v1 1/3015/3533 2762 -- 14/14/13/13/3 0/0
2025-10-04T01:52:18.840765+00:00 dev-lb-v1 haproxy[846]: 192.168.30.203:46502 [04/Oct/2025:01:52:15.156] apiserver apiserver/dev-m-v1 1/3018/3684 2762 -- 13/13/12/12/3 0/0
2025-10-04T01:52:18.840979+00:00 dev-lb-v1 haproxy[846]: 192.168.30.203:46500 [04/Oct/2025:01:52:15.102] apiserver apiserver/dev-m-v1 1/3013/3738 2762 -- 12/12/11/11/3 0/0
2025-10-04T01:52:18.857161+00:00 dev-lb-v1 haproxy[846]: 192.168.30.203:33906 [04/Oct/2025:01:52:18.698] apiserver apiserver/dev-m-v1 1/0/158 2762 -- 11/11/10/10/0 0/0
2025-10-04T01:52:18.857635+00:00 dev-lb-v1 haproxy[846]: 192.168.30.203:33896 [04/Oct/2025:01:52:18.697] apiserver apiserver/dev-m-v1 1/0/160 2762 -- 10/10/9/9/0 0/0
2025-10-04T01:52:18.858426+00:00 dev-lb-v1 haproxy[846]: 192.168.30.203:33888 [04/Oct/2025:01:52:18.696] apiserver apiserver/dev-m-v1 1/0/162 2762 -- 9/9/8/8/0 0/0
2025-10-04T01:52:18.858862+00:00 dev-lb-v1 haproxy[846]: 192.168.30.203:33860 [04/Oct/2025:01:52:18.661] apiserver apiserver/dev-m-v1 1/0/197 2762 -- 8/8/7/7/0 0/0
2025-10-04T01:52:18.861159+00:00 dev-lb-v1 haproxy[846]: 127.0.0.1:41102 [04/Oct/2025:01:52:16.460] apiserver apiserver/dev-m-v1 1/2007/2400 2419 CD 7/7/6/6/2 0/0
2025-10-04T01:52:18.862531+00:00 dev-lb-v1 haproxy[846]: 192.168.30.203:33942 [04/Oct/2025:01:52:18.826] apiserver apiserver/dev-m-v1 1/0/35 2762 -- 6/6/5/5/0 0/0
2025-10-04T01:52:18.864475+00:00 dev-lb-v1 haproxy[846]: 192.168.30.203:33934 [04/Oct/2025:01:52:18.823] apiserver apiserver/dev-m-v1 1/0/40 2762 -- 5/5/4/4/0 0/0
2025-10-04T01:52:22.864751+00:00 dev-lb-v1 haproxy[846]: [WARNING] (846) : Server apiserver/dev-m-v1 is UP, reason: Layer7 check passed, code: 200, check duration: 9ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
2025-10-04T01:52:22.864991+00:00 dev-lb-v1 haproxy[846]: Server apiserver/dev-m-v1 is UP, reason: Layer7 check passed, code: 200, check duration: 9ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
2025-10-04T01:52:22.865107+00:00 dev-lb-v1 haproxy[846]: Server apiserver/dev-m-v1 is UP, reason: Layer7 check passed, code: 200, check duration: 9ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
2025-10-04T01:52:23.309619+00:00 dev-lb-v1 haproxy[846]: 192.168.30.203:34002 [04/Oct/2025:01:52:23.304] apiserver apiserver/dev-m-v1 1/0/5 2226 CD 6/6/5/5/0 0/0
2025-10-04T01:52:23.488005+00:00 dev-lb-v1 haproxy[846]: 127.0.0.1:59618 [04/Oct/2025:01:52:23.484] apiserver apiserver/dev-m-v1 1/0/3 2251 -- 6/6/5/5/0 0/0 - Check the load balancer keepalived logs to confirm master state and priority:
journalctl -u keepalived -fOct 04 00:55:43 dev-lb-v1 Keepalived_vrrp[703]: (VI_1) Changing effective priority from 254 to 104
Oct 04 00:55:43 dev-lb-v1 keepalived-notify[726]: 2025-10-04 00:55:43: The load balancer instance on dev-lb-v1 is currently marked BACKUP
Oct 04 00:55:43 dev-lb-v1 systemd[1]: Started keepalived.service - Keepalive Daemon (LVS and VRRP).
Oct 04 00:55:47 dev-lb-v1 Keepalived_vrrp[703]: (VI_1) Entering MASTER STATE
Oct 04 01:52:06 dev-lb-v1 Keepalived_vrrp[703]: Script `check_apiserver` now returning 0
Oct 04 01:52:07 dev-lb-v1 Keepalived_vrrp[703]: VRRP_Script(check_apiserver) succeeded
Oct 04 01:52:07 dev-lb-v1 Keepalived_vrrp[703]: (VI_1) Changing effective priority from 104 to 254
Oct 04 01:52:19 dev-lb-v1 Keepalived_vrrp[703]: Track script check_apiserver is already running, expect idle - skipping run
Oct 04 01:52:21 dev-lb-v1 Keepalived_vrrp[703]: Script `check_apiserver` now returning 1
Oct 04 01:52:23 dev-lb-v1 Keepalived_vrrp[703]: Script `check_apiserver` now returning 0 - Check haproxy status to confirm that all master nodes are active:
sudo service haproxy status● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; preset: enabled)
Active: active (running) since Sat 2025-10-04 00:55:43 UTC; 1h 9min ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Main PID: 770 (haproxy)
Status: "Ready."
Tasks: 3 (limit: 2213)
Memory: 45.1M (peak: 46.3M)
CPU: 6.133s
CGroup: /system.slice/haproxy.service
├─770 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
└─846 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
Oct 04 02:04:56 dev-lb-v1 haproxy[846]: 127.0.0.1:56370 [04/Oct/2025:02:04:56.045] apiserver apiserver/dev-m-v1 1/0/7 2251 -- 9/9/8/8/0 0/0
Oct 04 02:04:56 dev-lb-v1 haproxy[846]: 192.168.30.201:54714 [04/Oct/2025:02:04:56.058] apiserver apiserver/dev-m-v1 1/0/5 2252 -- 9/9/8/8>
Oct 04 02:04:59 dev-lb-v1 haproxy[846]: 127.0.0.1:56376 [04/Oct/2025:02:04:59.055] apiserver apiserver/dev-m-v1 1/0/8 2251 -- 9/9/8/8/0 0/0
Oct 04 02:04:59 dev-lb-v1 haproxy[846]: 192.168.30.201:54718 [04/Oct/2025:02:04:59.070] apiserver apiserver/dev-m-v1 1/0/7 2252 -- 9/9/8/8>
Oct 04 02:05:02 dev-lb-v1 haproxy[846]: 127.0.0.1:32832 [04/Oct/2025:02:05:02.062] apiserver apiserver/dev-m-v1 1/4/15 2251 -- 9/9/8/8/0 0>
Oct 04 02:05:02 dev-lb-v1 haproxy[846]: 192.168.30.201:48434 [04/Oct/2025:02:05:02.085] apiserver apiserver/dev-m-v1 1/0/6 2251 -- 9/9/8/8>
Oct 04 02:05:05 dev-lb-v1 haproxy[846]: 127.0.0.1:32834 [04/Oct/2025:02:05:05.061] apiserver apiserver/dev-m-v1 1/0/9 2251 -- 9/9/8/8/0 0/0
Oct 04 02:05:05 dev-lb-v1 haproxy[846]: 192.168.30.201:48448 [04/Oct/2025:02:05:05.079] apiserver apiserver/dev-m-v1 1/0/7 2251 -- 9/9/8/8>
Oct 04 02:05:08 dev-lb-v1 haproxy[846]: 127.0.0.1:32846 [04/Oct/2025:02:05:08.052] apiserver apiserver/dev-m-v1 1/0/4 2273 -- 9/9/8/8/0 0/0
Kubespray - k8s deployment success
- Connect to any master node:
ssh dev-m-v1.reids.net.au- Confirm all nodes are ready:
kubectl get nodes -o wideNAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
dev-m-v1 Ready control-plane 17m v1.32.9 192.168.30.203 <none> Ubuntu 24.04.3 LTS 6.8.0-84-generic containerd://2.1.4
dev-w-v1 Ready <none> 17m v1.32.9 192.168.30.206 <none> Ubuntu 24.04.3 LTS 6.8.0-84-generic containerd://2.1.4 - Confirm all pods are running, all daemonsets and deployments are ready and available, and all replicasets are ready. Also confirm that pod restarts are not incrementing, it is usual for a couple of pods to have several restarts when the cluster is first created, but if they are increasing in number and the age for the running pod is low then further investigation will be required
kubectl get all --all-namespacesNAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/calico-kube-controllers-85b459fb9c-9hbhd 1/1 Running 0 17m
kube-system pod/calico-node-lpg9q 1/1 Running 0 17m
kube-system pod/calico-node-xd6th 1/1 Running 0 17m
kube-system pod/coredns-56f9f7f8b5-cvt6s 1/1 Running 0 17m
kube-system pod/coredns-56f9f7f8b5-m24bn 1/1 Running 0 17m
kube-system pod/dns-autoscaler-56cb45595c-jhgsx 1/1 Running 0 17m
kube-system pod/kube-apiserver-dev-m-v1 1/1 Running 1 18m
kube-system pod/kube-controller-manager-dev-m-v1 1/1 Running 1 18m
kube-system pod/kube-proxy-965tm 1/1 Running 0 18m
kube-system pod/kube-proxy-n4kgt 1/1 Running 0 17m
kube-system pod/kube-scheduler-dev-m-v1 1/1 Running 1 18m
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.70.128.1 <none> 443/TCP 18m
kube-system service/coredns ClusterIP 10.70.128.3 <none> 53/UDP,53/TCP,9153/TCP 17m
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/calico-node 2 2 2 2 2 kubernetes.io/os=linux 17m
kube-system daemonset.apps/kube-proxy 2 2 2 2 2 kubernetes.io/os=linux 18m
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/calico-kube-controllers 1/1 1 1 17m
kube-system deployment.apps/coredns 2/2 2 2 17m
kube-system deployment.apps/dns-autoscaler 1/1 1 1 17m
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/calico-kube-controllers-85b459fb9c 1 1 1 17m
kube-system replicaset.apps/coredns-56f9f7f8b5 2 2 2 17m
kube-system replicaset.apps/dns-autoscaler-56cb45595c 1 1 1 17m
- Confirm all nodes are ready: