Skip to main content

Kubespray configuration

Key variables in all.yml, k8s-cluster.yml, and network settings.

Kubespray cluster settings​

  • Kubespray organises cluster settings into different YAML files so you don’t mix up responsibilities
    • all.yml: baseline cluster setup (paths, load balancer VIP/hostnames, NTP, global flags)
      • Scope: Entire cluster, all hosts.
      • Purpose: Global environment setup.
      • Examples:
        • Binary install paths (bin_dir)
        • API load balancer details (loadbalancer_apiserver,apiserver_loadbalancer_domain_name)
        • Time sync (ntp_servers)
        • General Ansible behaviour flags
    • k8s-cluster.yml: Kubernetes-wide settings (version, pod/service CIDRs, DNS domain, kube-proxy, security defaults)
      • Scope: Kubernetes cluster behaviour.
      • Purpose: Core Kubernetes settings.
      • Examples:
        • Kubernetes version (kube_version)
        • Pod & service CIDRs (kube_pods_subnet, kube_service_addresses)
        • Cluster DNS domain (dns_domain, cluster_name)
        • API server flags (kube_api_anonymous_auth, remove_anonymous_access)
        • DNS settings (CoreDNS, NodeLocalDNS)
        • kube-proxy mode (IPVS vs iptables)
    • k8s-net-calico.yml: Calico-specific details (encapsulation type, MTU, block sizes, node autodetection)
      • Scope: Calico-specific networking.
      • Purpose: Defines how Calico provides pod networking.
      • Examples:
        • Encapsulation mode (calico_ipip_mode, calico_vxlan_mode)
        • Routing backend (calico_network_backend)
        • Per-node blocksize (calico_pool_blocksize)
        • Pod MTU (calico_veth_mtu)
        • Node IP detection (calico_ip_auto_method)

Kubespray - all.yml​

  • all.yml defines the environment for all nodes
  • The all.yml file in Kubespray inventories is the top-level global config file for cluster deployment. It sets universal defaults (bin paths, API load-balancer details, NTP servers, etc.) that all nodes inherit, while more specialized YAML files (like k8s-cluster.yml) refine component-level settings
    • Location of all.yml: vi ~/kubespray-devcluster/kubespray/inventory/devcluster/group_vars/all/all.yml
  • Sample all.yml (not including comments): 
    bin_dir: /usr/local/bin
    loadbalancer_apiserver_port: 6443
    loadbalancer_apiserver_healthcheck_port: 8081
    no_proxy_exclude_workers: false
    kube_webhook_token_auth: false
    kube_webhook_token_auth_url_skip_tls_verify: false
    ntp_enabled: false
    ntp_manage_config: false
    ntp_servers:
      - '0.pool.ntp.org iburst'
      - '1.pool.ntp.org iburst'
      - '2.pool.ntp.org iburst'
      - '3.pool.ntp.org iburst'
    unsafe_show_logs: false
    allow_unsupported_distribution_setup: false
  • Add the following: 
    apiserver_loadbalancer_domain_name: 'dev.internal.example.com'
    loadbalancer_apiserver:
      address: '192.168.1.200'
      port: '8443'
  • Change the following: 
    ntp_enabled: true # was false
    ntp_manage_config: true # was false
    ntp_servers:
      - '192.168.1.254 iburst' # replaced the pool.ntp.org list
  • Remove the following: 
    ntp_servers:
      - '0.pool.ntp.org iburst'
      - '1.pool.ntp.org iburst'
      - '2.pool.ntp.org iburst'
      - '3.pool.ntp.org iburst'
  • Combined changes: 
    apiserver_loadbalancer_domain_name: 'dev.internal.example.com'
    loadbalancer_apiserver:
      address: '192.168.1.200'
      port: '8443'
    ntp_enabled: true
    ntp_manage_config: true
    ntp_servers:
      - '192.168.1.254 iburst'
  • You can either manually edit the file by hand or utilise yq (installed in an earlier step) 
    yq -i -y '
    .apiserver_loadbalancer_domain_name = "dev.internal.example.com" |
    .loadbalancer_apiserver.address = "192.168.1.200" |
    .loadbalancer_apiserver.port = "8443" |
    .ntp_enabled = true |
    .ntp_manage_config = true |
    .ntp_servers = ["192.168.1.254 iburst"]
    ' ~/kubespray-devcluster/kubespray/inventory/devcluster/group_vars/all/all.yml

Kubespray - k8s-cluster.yml​

  • k8s-cluster.yml defines the shape and scope of the Kubernetes cluster
  • Kubernetes versions can be found here Kubernetes Versions
    • Kubernetes versions are expressed as x.y.z, where x is the major version, y is the minor version, and z is the patch version
    • Unlike previous versions of Kubespray, the current version will install the latest Kubernetes version. However, I prefer to control this and specify the version (not the latest one) kube_version: 1.32.9 released 9th September 2025
  • During the preparation stage we defined a number of variables:
    • k8s dev internal network for services: 10.70.128.0/18
    • k8s dev pods subnet: 10.70.192.0/18
    • cluster name: dev.internal.example.com
    • cluster IP address: 192.168.1.200
  • Location of k8s-cluster.yml: vi ~/kubespray-devcluster/kubespray/inventory/devcluster/group_vars/k8s_cluster/k8s-cluster.yml
  • The sample k8s-cluster.yml looks like this (not included commented out sections) 
    kube_config_dir: /etc/kubernetes
    kube_script_dir: '{{ bin_dir }}/kubernetes-scripts'
    kube_manifest_dir: '{{ kube_config_dir }}/manifests'
    kube_cert_dir: '{{ kube_config_dir }}/ssl'
    kube_token_dir: '{{ kube_config_dir }}/tokens'
    kube_api_anonymous_auth: true
    local_release_dir: '/tmp/releases'
    retry_stagger: 5
    kube_owner: kube
    kube_cert_group: kube-cert
    kube_log_level: 2
    credentials_dir: '{{ inventory_dir }}/credentials'
    kube_network_plugin: calico
    kube_network_plugin_multus: false
    kube_service_addresses: 10.233.0.0/18
    kube_pods_subnet: 10.233.64.0/18
    kube_network_node_prefix: 24
    kube_service_addresses_ipv6: fd85:ee78:d8a6:8607::1000/116
    kube_pods_subnet_ipv6: fd85:ee78:d8a6:8607::1:0000/112
    kube_network_node_prefix_ipv6: 120
    kube_apiserver_ip: "{{ kube_service_subnets.split(',') | first | ansible.utils.ipaddr('net') | ansible.utils.ipaddr(1) | ansible.utils.ipaddr('address') }}"
    kube_apiserver_port: 6443 # (https)
    kube_proxy_mode: ipvs
    kube_proxy_strict_arp: false
    kube_proxy_nodeport_addresses: >-
      {%- if kube_proxy_nodeport_addresses_cidr is defined -%}
      [{{ kube_proxy_nodeport_addresses_cidr }}]
      {%- else -%}
      []
      {%- endif -%}
    kube_encrypt_secret_data: false
    cluster_name: cluster.local
    ndots: 2
    dns_mode: coredns
    enable_nodelocaldns: true
    enable_nodelocaldns_secondary: false
    nodelocaldns_ip: 169.254.25.10
    nodelocaldns_health_port: 9254
    nodelocaldns_second_health_port: 9256
    nodelocaldns_bind_metrics_host_ip: false
    nodelocaldns_secondary_skew_seconds: 5
    enable_coredns_k8s_external: false
    coredns_k8s_external_zone: k8s_external.local
    enable_coredns_k8s_endpoint_pod_names: false
    resolvconf_mode: host_resolvconf
    deploy_netchecker: false
    skydns_server: "{{ kube_service_subnets.split(',') | first | ansible.utils.ipaddr('net') | ansible.utils.ipaddr(3) | ansible.utils.ipaddr('address') }}"
    skydns_server_secondary: "{{ kube_service_subnets.split(',') | first | ansible.utils.ipaddr('net') | ansible.utils.ipaddr(4) | ansible.utils.ipaddr('address') }}"
    dns_domain: '{{ cluster_name }}'
    container_manager: containerd
    kata_containers_enabled: false
    kubeadm_certificate_key: "{{ lookup('password', credentials_dir + '/kubeadm_certificate_key.creds length=64 chars=hexdigits') | lower }}"
    k8s_image_pull_policy: IfNotPresent
    kubernetes_audit: false
    default_kubelet_config_dir: '{{ kube_config_dir }}/dynamic_kubelet_dir'
    volume_cross_zone_attachment: false
    persistent_volumes_enabled: false
    event_ttl_duration: '1h0m0s'
    auto_renew_certificates: false
    kubeadm_patches_dir: '{{ kube_config_dir }}/patches'
    kubeadm_patches: []
    remove_anonymous_access: false
  • These are the changes I make to the cluster configuration 
    kube_version: 1.32.9
    kube_service_addresses: 10.70.128.0/18
    kube_pods_subnet: 10.70.192.0/18
    cluster_name: dev.internal.example.com
    coredns_kubernetes_extra_domains: 'cluster.local'
    enable_nodelocaldns: false
    kubernetes_audit: true
    kubeconfig_localhost: true
    kubectl_localhost: true
    supplementary_addresses_in_ssl_keys: [192.168.1.200]
  • Noting that the following configuration is set by default: dns_domain: "{{ cluster_name }}"
  • You can either manually edit the file by hand or utilise yq (installed in an earlier step) 
    yq -i -y '
    .kube_version = "1.32.9" |
    .kube_service_addresses = "10.70.128.0/18" |
    .kube_pods_subnet = "10.70.192.0/18" |
    .cluster_name = "dev.internal.example.com" |
    .coredns_kubernetes_extra_domains = "cluster.local" |
    .enable_nodelocaldns = false |
    .kubernetes_audit = true |
    .kubeconfig_localhost = true |
    .kubectl_localhost = true |
    .supplementary_addresses_in_ssl_keys = ["192.168.1.200"]
    ' ~/kubespray-devcluster/kubespray/inventory/devcluster/group_vars/k8s_cluster/k8s-cluster.yml
  • Note. I add the extra cluster.local domain to CoreDNS as I have found some applications are hardwired to use it. This means CoreDNS is serving both dev.internal.example.com and the default cluster.local zone

Kubespray - k8s-net-calico.yml​

  • k8s-net-calico.yml defines how Calico implements the networking inside that scope. As calico will be deployed across virtual and physical environments where the interface name will be different I have chosen to determine the IP based on the CIDR, in my dev network this is 192.168.1.0/24. If you configure the interface name instead then calico will not start on nodes with different interface names
  • Location of k8s-net-calico.yml: ~/kubespray-devcluster/kubespray/inventory/devcluster/group_vars/k8s_cluster/k8s-net-calico.yml
  • The sample k8s-cluster.yml looks like this (not included commented out sections) 
    calico_cni_name: k8s-pod-network
    calico_pool_blocksize: 26
  • Change the following: 
    calico_pool_blocksize: 24
  • Add the following: 
    calico_veth_mtu: 1480
    calico_network_backend: bird
    calico_ipip_mode: 'Always'
    calico_vxlan_mode: 'Never'
    calico_ip_auto_method: 'cidr=192.168.1.0/24'
  • You can either manually edit the file by hand or utilise yq (installed in an earlier step): 
    yq -i -y '
    .calico_pool_blocksize = 24 |
    .calico_veth_mtu = 1480 |
    .calico_network_backend = "bird" |
    .calico_ipip_mode = "Always" |
    .calico_vxlan_mode = "Never" |
    .calico_ip_auto_method = "cidr=192.168.1.0/24"
    ' ~/kubespray-devcluster/kubespray/inventory/devcluster/group_vars/k8s_cluster/k8s-net-calico.yml

SSH keys​

Prepare ssh passwordless connections to each host so that ansible can connect.

  • Create the key pair using the command ssh-keygen
    • this creates a private identification file: id_ed25519 and a public key: id_ed25519.pub saved in /home/dev/.ssh/
  • Copy the ssh key to the Kubernetes nodes (no need to copy to load balancers)
    • If there are only a small number of nodes, manual will be quicker 
      cd ~/.ssh/
      ssh-copy-id  lb-1
      ssh-copy-id  master-1
      ssh-copy-id  worker-1
    • If there are a lot of nodes then list them in a nodes file: vi ~/nodes 
      lb-1
      master-1
      worker-1
    • Create a changenodes script: vi ~/changenodes.sh 
      #!/usr/bin/env bash
      set -euo pipefail

      NODES_FILE="${1:-$HOME/nodes}"                 # pass a file path or default to ~/nodes
      SSH_USER="${SSH_USER:-$USER}"                  # override with: SSH_USER=dev ./changenodes.sh
      KEY="${SSH_KEY:-$HOME/.ssh/id_ed25519.pub}"    # override with: SSH_KEY=~/.ssh/id_rsa.pub
      TIMEOUT="${SSH_TIMEOUT:-7}"

      # Ensure key exists; create one if missing
      if [[ ! -f "$KEY" ]]; then
        echo "Key $KEY not found. Creating ed25519 key..."
        ssh-keygen -t ed25519 -f "${KEY%.pub}" -N "" -C "${USER}@$(hostname)" >/dev/null
      fi

      # Read nodes file line-by-line; ignore blanks and comments
      while IFS= read -r host; do
        [[ -z "$host" || "$host" =~ ^# ]] && continue
        echo ">>> Installing key on ${SSH_USER}@${host}"
        ssh-copy-id \
          -i "$KEY" \
          -o ConnectTimeout="$TIMEOUT" \
          -o StrictHostKeyChecking=accept-new \
          "${SSH_USER}@${host}"
      done < "$NODES_FILE"

      echo "All done. Quick test on the first host in the list:"
      first_host="$(grep -vE '^\s*($|#)' "$NODES_FILE" | head -n1)"
      [[ -n "$first_host" ]] && ssh -o BatchMode=yes -o ConnectTimeout="$TIMEOUT" "${SSH_USER}@${first_host}" 'echo "SSH OK on $(hostname)"'
    • Allow script to be executable: chmod +x changenodes.sh
    • Run script: ./changenodes.sh 
      >>> Installing key on dev@lb-1
      /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/dev/.ssh/id_ed25519.pub"
      /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
      /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
      dev@lb-1's password:

      Number of key(s) added: 1

      Now try logging into the machine, with:   "ssh -o 'ConnectTimeout=7' -o 'StrictHostKeyChecking=accept-new' 'dev@lb-1'"
      and check to make sure that only the key(s) you wanted were added.

      >>> Installing key on dev@master-1
      /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/dev/.ssh/id_ed25519.pub"
      /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
      /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
      dev@master-1's password:

      Number of key(s) added: 1

      Now try logging into the machine, with:   "ssh -o 'ConnectTimeout=7' -o 'StrictHostKeyChecking=accept-new' 'dev@master-1'"
      and check to make sure that only the key(s) you wanted were added.

      >>> Installing key on dev@worker-1
      /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/dev/.ssh/id_ed25519.pub"
      /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
      /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
      dev@worker-1's password:

      Number of key(s) added: 1

      Now try logging into the machine, with:   "ssh -o 'ConnectTimeout=7' -o 'StrictHostKeyChecking=accept-new' 'dev@worker-1'"
      and check to make sure that only the key(s) you wanted were added.

      All done. Quick test on the first host in the list:
      SSH OK on lb-1
    • The ansible server will now be able to ssh to the three nodes without a password. It is good practice to check them all
      • ssh lb-1
      • ssh master-1
      • ssh worker-1