Skip to main content

Kubespray deployment

Running the playbooks and validating success.

Kubespray - Deploy the k8s cluster​

  • Before running the playbook check the haproxy log for activity from each master node, in this dev environment we only have one master node: tail -f /var/log/haproxy.log
  • We are now ready to deploy the k8s cluster using the ansible playbook
  • First ensure that all playbook commands are run from the kubespray directory: 
    cd ~/kubespray-devcluster/kubespray/
  • This command will create the cluster 
    ansible-playbook -i inventory/devcluster/inventory.ini --become --become-user=root cluster.yml -u dev -K
  • Enter the become password. This is the dev user's password
  • Wait for the playbook to run all tasks. This may take a while as it is dependant upon a number of factors, such as resources available and the number of nodes being deployed in the cluster
  • Play recap: 
    PLAY RECAP ********************************************************************************************************************************
    master-1                   : ok=573  changed=137  unreachable=0    failed=0    skipped=883  rescued=0    ignored=5
    worker-1                   : ok=434  changed=81   unreachable=0    failed=0    skipped=676  rescued=0    ignored=0

    Saturday 04 October 2025  01:53:10 +0000 (0:00:00.026)       0:06:57.242 ******
    ===============================================================================
    download : Download_container | Download image if required ------------------------------------------------------------------------ 39.33s
    download : Download_file | Download item ------------------------------------------------------------------------------------------ 31.05s
    download : Download_file | Download item ------------------------------------------------------------------------------------------ 30.99s
    download : Download_container | Download image if required ------------------------------------------------------------------------ 22.86s
    download : Download_file | Download item ------------------------------------------------------------------------------------------ 21.74s
    kubernetes/kubeadm : Join to cluster if needed ------------------------------------------------------------------------------------ 15.98s
    download : Download_file | Download item ------------------------------------------------------------------------------------------ 13.96s
    kubernetes/control-plane : Kubeadm | Initialize first control plane node (1st try) ------------------------------------------------- 9.41s
    download : Download_container | Download image if required ------------------------------------------------------------------------- 8.76s
    kubernetes/control-plane : Control plane | wait for kube-scheduler ----------------------------------------------------------------- 8.66s
    etcd : Restart etcd ---------------------------------------------------------------------------------------------------------------- 8.65s
    download : Download_container | Download image if required ------------------------------------------------------------------------- 8.55s
    download : Download_container | Download image if required ------------------------------------------------------------------------- 7.78s
    system_packages : Manage packages -------------------------------------------------------------------------------------------------- 7.47s
    download : Download_container | Download image if required ------------------------------------------------------------------------- 7.17s
    download : Download_container | Download image if required ------------------------------------------------------------------------- 6.74s
    download : Download_container | Download image if required ------------------------------------------------------------------------- 6.34s
    container-engine/runc : Download_file | Download item ------------------------------------------------------------------------------ 5.22s
    etcd : Configure | Check if etcd cluster is healthy -------------------------------------------------------------------------------- 5.17s
    download : Download_file | Download item ------------------------------------------------------------------------------------------- 5.10s
  • Check the load balancer haproxy logs to confirm master node is up: 
    2025-10-04T01:52:18.838954+00:00 lb-1 haproxy[846]: backend apiserver has no server available!
    2025-10-04T01:52:18.839019+00:00 lb-1 haproxy[846]: backend apiserver has no server available!
    2025-10-04T01:52:18.839200+00:00 lb-1 haproxy[846]: 192.168.1.203:46530 [04/Oct/2025:01:52:17.424] apiserver apiserver/master-1 1/1003/1414 2762 -- 15/15/14/14/1 0/0
    2025-10-04T01:52:18.840634+00:00 lb-1 haproxy[846]: 192.168.1.203:46514 [04/Oct/2025:01:52:15.306] apiserver apiserver/master-1 1/3015/3533 2762 -- 14/14/13/13/3 0/0
    2025-10-04T01:52:18.840765+00:00 lb-1 haproxy[846]: 192.168.1.203:46502 [04/Oct/2025:01:52:15.156] apiserver apiserver/master-1 1/3018/3684 2762 -- 13/13/12/12/3 0/0
    2025-10-04T01:52:18.840979+00:00 lb-1 haproxy[846]: 192.168.1.203:46500 [04/Oct/2025:01:52:15.102] apiserver apiserver/master-1 1/3013/3738 2762 -- 12/12/11/11/3 0/0
    2025-10-04T01:52:18.857161+00:00 lb-1 haproxy[846]: 192.168.1.203:33906 [04/Oct/2025:01:52:18.698] apiserver apiserver/master-1 1/0/158 2762 -- 11/11/10/10/0 0/0
    2025-10-04T01:52:18.857635+00:00 lb-1 haproxy[846]: 192.168.1.203:33896 [04/Oct/2025:01:52:18.697] apiserver apiserver/master-1 1/0/160 2762 -- 10/10/9/9/0 0/0
    2025-10-04T01:52:18.858426+00:00 lb-1 haproxy[846]: 192.168.1.203:33888 [04/Oct/2025:01:52:18.696] apiserver apiserver/master-1 1/0/162 2762 -- 9/9/8/8/0 0/0
    2025-10-04T01:52:18.858862+00:00 lb-1 haproxy[846]: 192.168.1.203:33860 [04/Oct/2025:01:52:18.661] apiserver apiserver/master-1 1/0/197 2762 -- 8/8/7/7/0 0/0
    2025-10-04T01:52:18.861159+00:00 lb-1 haproxy[846]: 127.0.0.1:41102 [04/Oct/2025:01:52:16.460] apiserver apiserver/master-1 1/2007/2400 2419 CD 7/7/6/6/2 0/0
    2025-10-04T01:52:18.862531+00:00 lb-1 haproxy[846]: 192.168.1.203:33942 [04/Oct/2025:01:52:18.826] apiserver apiserver/master-1 1/0/35 2762 -- 6/6/5/5/0 0/0
    2025-10-04T01:52:18.864475+00:00 lb-1 haproxy[846]: 192.168.1.203:33934 [04/Oct/2025:01:52:18.823] apiserver apiserver/master-1 1/0/40 2762 -- 5/5/4/4/0 0/0
    2025-10-04T01:52:22.864751+00:00 lb-1 haproxy[846]: [WARNING]  (846) : Server apiserver/master-1 is UP, reason: Layer7 check passed, code: 200, check duration: 9ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
    2025-10-04T01:52:22.864991+00:00 lb-1 haproxy[846]: Server apiserver/master-1 is UP, reason: Layer7 check passed, code: 200, check duration: 9ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
    2025-10-04T01:52:22.865107+00:00 lb-1 haproxy[846]: Server apiserver/master-1 is UP, reason: Layer7 check passed, code: 200, check duration: 9ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
    2025-10-04T01:52:23.309619+00:00 lb-1 haproxy[846]: 192.168.1.203:34002 [04/Oct/2025:01:52:23.304] apiserver apiserver/master-1 1/0/5 2226 CD 6/6/5/5/0 0/0
    2025-10-04T01:52:23.488005+00:00 lb-1 haproxy[846]: 127.0.0.1:59618 [04/Oct/2025:01:52:23.484] apiserver apiserver/master-1 1/0/3 2251 -- 6/6/5/5/0 0/0
  • Check the load balancer keepalived logs to confirm master state and priority: journalctl -u keepalived -f 
    Oct 04 00:55:43 lb-1 Keepalived_vrrp[703]: (VI_1) Changing effective priority from 254 to 104
    Oct 04 00:55:43 lb-1 keepalived-notify[726]: 2025-10-04 00:55:43: The load balancer instance on lb-1 is currently marked BACKUP
    Oct 04 00:55:43 lb-1 systemd[1]: Started keepalived.service - Keepalive Daemon (LVS and VRRP).
    Oct 04 00:55:47 lb-1 Keepalived_vrrp[703]: (VI_1) Entering MASTER STATE
    Oct 04 01:52:06 lb-1 Keepalived_vrrp[703]: Script `check_apiserver` now returning 0
    Oct 04 01:52:07 lb-1 Keepalived_vrrp[703]: VRRP_Script(check_apiserver) succeeded
    Oct 04 01:52:07 lb-1 Keepalived_vrrp[703]: (VI_1) Changing effective priority from 104 to 254
    Oct 04 01:52:19 lb-1 Keepalived_vrrp[703]: Track script check_apiserver is already running, expect idle - skipping run
    Oct 04 01:52:21 lb-1 Keepalived_vrrp[703]: Script `check_apiserver` now returning 1
    Oct 04 01:52:23 lb-1 Keepalived_vrrp[703]: Script `check_apiserver` now returning 0
  • Check haproxy status to confirm that all master nodes are active: sudo service haproxy status 
    ● haproxy.service - HAProxy Load Balancer
         Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; preset: enabled)
         Active: active (running) since Sat 2025-10-04 00:55:43 UTC; 1h 9min ago
           Docs: man:haproxy(1)
                 file:/usr/share/doc/haproxy/configuration.txt.gz
       Main PID: 770 (haproxy)
         Status: "Ready."
          Tasks: 3 (limit: 2213)
         Memory: 45.1M (peak: 46.3M)
            CPU: 6.133s
         CGroup: /system.slice/haproxy.service
                 ├─770 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
                 └─846 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock

    Oct 04 02:04:56 lb-1 haproxy[846]: 127.0.0.1:56370 [04/Oct/2025:02:04:56.045] apiserver apiserver/master-1 1/0/7 2251 -- 9/9/8/8/0 0/0
    Oct 04 02:04:56 lb-1 haproxy[846]: 192.168.1.201:54714 [04/Oct/2025:02:04:56.058] apiserver apiserver/master-1 1/0/5 2252 -- 9/9/8/8>
    Oct 04 02:04:59 lb-1 haproxy[846]: 127.0.0.1:56376 [04/Oct/2025:02:04:59.055] apiserver apiserver/master-1 1/0/8 2251 -- 9/9/8/8/0 0/0
    Oct 04 02:04:59 lb-1 haproxy[846]: 192.168.1.201:54718 [04/Oct/2025:02:04:59.070] apiserver apiserver/master-1 1/0/7 2252 -- 9/9/8/8>
    Oct 04 02:05:02 lb-1 haproxy[846]: 127.0.0.1:32832 [04/Oct/2025:02:05:02.062] apiserver apiserver/master-1 1/4/15 2251 -- 9/9/8/8/0 0>
    Oct 04 02:05:02 lb-1 haproxy[846]: 192.168.1.201:48434 [04/Oct/2025:02:05:02.085] apiserver apiserver/master-1 1/0/6 2251 -- 9/9/8/8>
    Oct 04 02:05:05 lb-1 haproxy[846]: 127.0.0.1:32834 [04/Oct/2025:02:05:05.061] apiserver apiserver/master-1 1/0/9 2251 -- 9/9/8/8/0 0/0
    Oct 04 02:05:05 lb-1 haproxy[846]: 192.168.1.201:48448 [04/Oct/2025:02:05:05.079] apiserver apiserver/master-1 1/0/7 2251 -- 9/9/8/8>
    Oct 04 02:05:08 lb-1 haproxy[846]: 127.0.0.1:32846 [04/Oct/2025:02:05:08.052] apiserver apiserver/master-1 1/0/4 2273 -- 9/9/8/8/0 0/0

Kubespray - k8s deployment success​

  • Connect to any master node: ssh master-1.internal.example.com
    • Confirm all nodes are ready: 
      kubectl get nodes -o wide
      NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
      master-1   Ready    control-plane   17m   v1.32.9   192.168.1.203   <none>        Ubuntu 24.04.3 LTS   6.8.0-84-generic   containerd://2.1.4
      worker-1   Ready    <none>          17m   v1.32.9   192.168.1.206   <none>        Ubuntu 24.04.3 LTS   6.8.0-84-generic   containerd://2.1.4
    • Confirm all pods are running, all daemonsets and deployments are ready and available, and all replicasets are ready. Also confirm that pod restarts are not incrementing, it is usual for a couple of pods to have several restarts when the cluster is first created, but if they are increasing in number and the age for the running pod is low then further investigation will be required 
      kubectl get all --all-namespaces
      NAMESPACE     NAME                                           READY   STATUS    RESTARTS   AGE
      kube-system   pod/calico-kube-controllers-85b459fb9c-9hbhd   1/1     Running   0          17m
      kube-system   pod/calico-node-lpg9q                          1/1     Running   0          17m
      kube-system   pod/calico-node-xd6th                          1/1     Running   0          17m
      kube-system   pod/coredns-56f9f7f8b5-cvt6s                   1/1     Running   0          17m
      kube-system   pod/coredns-56f9f7f8b5-m24bn                   1/1     Running   0          17m
      kube-system   pod/dns-autoscaler-56cb45595c-jhgsx            1/1     Running   0          17m
      kube-system   pod/kube-apiserver-master-1                    1/1     Running   1          18m
      kube-system   pod/kube-controller-manager-master-1           1/1     Running   1          18m
      kube-system   pod/kube-proxy-965tm                           1/1     Running   0          18m
      kube-system   pod/kube-proxy-n4kgt                           1/1     Running   0          17m
      kube-system   pod/kube-scheduler-master-1                    1/1     Running   1          18m

      NAMESPACE     NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                  AGE
      default       service/kubernetes   ClusterIP   10.70.128.1   <none>        443/TCP                  18m
      kube-system   service/coredns      ClusterIP   10.70.128.3   <none>        53/UDP,53/TCP,9153/TCP   17m

      NAMESPACE     NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
      kube-system   daemonset.apps/calico-node   2         2         2       2            2           kubernetes.io/os=linux   17m
      kube-system   daemonset.apps/kube-proxy    2         2         2       2            2           kubernetes.io/os=linux   18m

      NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
      kube-system   deployment.apps/calico-kube-controllers   1/1     1            1           17m
      kube-system   deployment.apps/coredns                   2/2     2            2           17m
      kube-system   deployment.apps/dns-autoscaler            1/1     1            1           17m

      NAMESPACE     NAME                                                 DESIRED   CURRENT   READY   AGE
      kube-system   replicaset.apps/calico-kube-controllers-85b459fb9c   1         1         1       17m
      kube-system   replicaset.apps/coredns-56f9f7f8b5                   2         2         2       17m
      kube-system   replicaset.apps/dns-autoscaler-56cb45595c            1         1         1       17m