Kubespray configuration
Key variables in all.yml, k8s-cluster.yml, and network settings.
Kubespray cluster settings
- Kubespray organises cluster settings into different YAML files so you don’t mix up responsibilities
- all.yml: baseline cluster setup (paths, load balancer VIP/hostnames, NTP, global flags)
- Scope: Entire cluster, all hosts.
- Purpose: Global environment setup.
- Examples:
- Binary install paths (bin_dir)
- API load balancer details (loadbalancer_apiserver,apiserver_loadbalancer_domain_name)
- Time sync (ntp_servers)
- General Ansible behaviour flags
- k8s-cluster.yml: Kubernetes-wide settings (version, pod/service CIDRs, DNS domain, kube-proxy, security defaults)
- Scope: Kubernetes cluster behaviour.
- Purpose: Core Kubernetes settings.
- Examples:
- Kubernetes version (kube_version)
- Pod & service CIDRs (kube_pods_subnet, kube_service_addresses)
- Cluster DNS domain (dns_domain, cluster_name)
- API server flags (kube_api_anonymous_auth, remove_anonymous_access)
- DNS settings (CoreDNS, NodeLocalDNS)
- kube-proxy mode (IPVS vs iptables)
- k8s-net-calico.yml: Calico-specific details (encapsulation type, MTU, block sizes, node autodetection)
- Scope: Calico-specific networking.
- Purpose: Defines how Calico provides pod networking.
- Examples:
- Encapsulation mode (calico_ipip_mode, calico_vxlan_mode)
- Routing backend (calico_network_backend)
- Per-node blocksize (calico_pool_blocksize)
- Pod MTU (calico_veth_mtu)
- Node IP detection (calico_ip_auto_method)
- all.yml: baseline cluster setup (paths, load balancer VIP/hostnames, NTP, global flags)
Kubespray - all.yml
- all.yml defines the environment for all nodes
- The all.yml file in Kubespray inventories is the top-level global config file for cluster deployment. It sets universal defaults (bin paths, API load-balancer details, NTP servers, etc.) that all nodes inherit, while more specialized YAML files (like k8s-cluster.yml) refine component-level settings
-
- Location of all.yml:
vi ~/kubespray-devcluster/kubespray/inventory/devcluster/group_vars/all/all.yml
- Location of all.yml:
- Sample all.yml (not including comments):
bin_dir: /usr/local/bin
loadbalancer_apiserver_port: 6443
loadbalancer_apiserver_healthcheck_port: 8081
no_proxy_exclude_workers: false
kube_webhook_token_auth: false
kube_webhook_token_auth_url_skip_tls_verify: false
ntp_enabled: false
ntp_manage_config: false
ntp_servers:
- '0.pool.ntp.org iburst'
- '1.pool.ntp.org iburst'
- '2.pool.ntp.org iburst'
- '3.pool.ntp.org iburst'
unsafe_show_logs: false
allow_unsupported_distribution_setup: false - Add the following:
apiserver_loadbalancer_domain_name: 'dev.reids.net.au'
loadbalancer_apiserver:
address: '192.168.30.200'
port: '8443' - Change the following:
ntp_enabled: true # was false
ntp_manage_config: true # was false
ntp_servers:
- '192.168.30.254 iburst' # replaced the pool.ntp.org list - Remove the following:
ntp_servers:
- '0.pool.ntp.org iburst'
- '1.pool.ntp.org iburst'
- '2.pool.ntp.org iburst'
- '3.pool.ntp.org iburst' - Combined changes:
apiserver_loadbalancer_domain_name: 'dev.reids.net.au'
loadbalancer_apiserver:
address: '192.168.30.200'
port: '8443'
ntp_enabled: true
ntp_manage_config: true
ntp_servers:
- '192.168.30.254 iburst' - You can either manually edit the file by hand or utilise yq (installed in an earlier step)
yq -i -y '
.apiserver_loadbalancer_domain_name = "dev.reids.net.au" |
.loadbalancer_apiserver.address = "192.168.30.200" |
.loadbalancer_apiserver.port = "8443" |
.ntp_enabled = true |
.ntp_manage_config = true |
.ntp_servers = ["192.168.30.254 iburst"]
' ~/kubespray-devcluster/kubespray/inventory/devcluster/group_vars/all/all.yml
Kubespray - k8s-cluster.yml
- k8s-cluster.yml defines the shape and scope of the Kubernetes cluster
- Kubernetes versions can be found here Kubernetes Versions
- Kubernetes versions are expressed as x.y.z, where x is the major version, y is the minor version, and z is the patch version
- Unlike previous versions of Kubespray, the current version will install the latest Kubernetes version. However, I prefer to control this and specify the version (not the latest one)
kube_version: 1.32.9released 9th September 2025
- During the preparation stage we defined a number of variables:
- k8s dev internal network for services: 10.70.128.0/18
- k8s dev pods subnet: 10.70.192.0/18
- cluster name: dev.reids.net.au
- cluster IP address: 192.168.30.200
- Location of k8s-cluster.yml:
vi ~/kubespray-devcluster/kubespray/inventory/devcluster/group_vars/k8s_cluster/k8s-cluster.yml - The sample k8s-cluster.yml looks like this (not included commented out sections)
kube_config_dir: /etc/kubernetes
kube_script_dir: '{{ bin_dir }}/kubernetes-scripts'
kube_manifest_dir: '{{ kube_config_dir }}/manifests'
kube_cert_dir: '{{ kube_config_dir }}/ssl'
kube_token_dir: '{{ kube_config_dir }}/tokens'
kube_api_anonymous_auth: true
local_release_dir: '/tmp/releases'
retry_stagger: 5
kube_owner: kube
kube_cert_group: kube-cert
kube_log_level: 2
credentials_dir: '{{ inventory_dir }}/credentials'
kube_network_plugin: calico
kube_network_plugin_multus: false
kube_service_addresses: 10.233.0.0/18
kube_pods_subnet: 10.233.64.0/18
kube_network_node_prefix: 24
kube_service_addresses_ipv6: fd85:ee78:d8a6:8607::1000/116
kube_pods_subnet_ipv6: fd85:ee78:d8a6:8607::1:0000/112
kube_network_node_prefix_ipv6: 120
kube_apiserver_ip: "{{ kube_service_subnets.split(',') | first | ansible.utils.ipaddr('net') | ansible.utils.ipaddr(1) | ansible.utils.ipaddr('address') }}"
kube_apiserver_port: 6443 # (https)
kube_proxy_mode: ipvs
kube_proxy_strict_arp: false
kube_proxy_nodeport_addresses: >-
{%- if kube_proxy_nodeport_addresses_cidr is defined -%}
[{{ kube_proxy_nodeport_addresses_cidr }}]
{%- else -%}
[]
{%- endif -%}
kube_encrypt_secret_data: false
cluster_name: cluster.local
ndots: 2
dns_mode: coredns
enable_nodelocaldns: true
enable_nodelocaldns_secondary: false
nodelocaldns_ip: 169.254.25.10
nodelocaldns_health_port: 9254
nodelocaldns_second_health_port: 9256
nodelocaldns_bind_metrics_host_ip: false
nodelocaldns_secondary_skew_seconds: 5
enable_coredns_k8s_external: false
coredns_k8s_external_zone: k8s_external.local
enable_coredns_k8s_endpoint_pod_names: false
resolvconf_mode: host_resolvconf
deploy_netchecker: false
skydns_server: "{{ kube_service_subnets.split(',') | first | ansible.utils.ipaddr('net') | ansible.utils.ipaddr(3) | ansible.utils.ipaddr('address') }}"
skydns_server_secondary: "{{ kube_service_subnets.split(',') | first | ansible.utils.ipaddr('net') | ansible.utils.ipaddr(4) | ansible.utils.ipaddr('address') }}"
dns_domain: '{{ cluster_name }}'
container_manager: containerd
kata_containers_enabled: false
kubeadm_certificate_key: "{{ lookup('password', credentials_dir + '/kubeadm_certificate_key.creds length=64 chars=hexdigits') | lower }}"
k8s_image_pull_policy: IfNotPresent
kubernetes_audit: false
default_kubelet_config_dir: '{{ kube_config_dir }}/dynamic_kubelet_dir'
volume_cross_zone_attachment: false
persistent_volumes_enabled: false
event_ttl_duration: '1h0m0s'
auto_renew_certificates: false
kubeadm_patches_dir: '{{ kube_config_dir }}/patches'
kubeadm_patches: []
remove_anonymous_access: false - These are the changes I make to the cluster configuration
kube_version: 1.32.9
kube_service_addresses: 10.70.128.0/18
kube_pods_subnet: 10.70.192.0/18
cluster_name: dev.reids.net.au
coredns_kubernetes_extra_domains: 'cluster.local'
enable_nodelocaldns: false
kubernetes_audit: true
kubeconfig_localhost: true
kubectl_localhost: true
supplementary_addresses_in_ssl_keys: [192.168.30.200] - Noting that the following configuration is set by default:
dns_domain: "{{ cluster_name }}" - You can either manually edit the file by hand or utilise yq (installed in an earlier step)
yq -i -y '
.kube_version = "1.32.9" |
.kube_service_addresses = "10.70.128.0/18" |
.kube_pods_subnet = "10.70.192.0/18" |
.cluster_name = "dev.reids.net.au" |
.coredns_kubernetes_extra_domains = "cluster.local" |
.enable_nodelocaldns = false |
.kubernetes_audit = true |
.kubeconfig_localhost = true |
.kubectl_localhost = true |
.supplementary_addresses_in_ssl_keys = ["192.168.30.200"]
' ~/kubespray-devcluster/kubespray/inventory/devcluster/group_vars/k8s_cluster/k8s-cluster.yml - Note. I add the extra
cluster.localdomain to CoreDNS as I have found some applications are hardwired to use it. This means CoreDNS is serving both dev.reids.net.au and the default cluster.local zone
Kubespray - k8s-net-calico.yml
- k8s-net-calico.yml defines how Calico implements the networking inside that scope. As calico will be deployed across virtual and physical environments where the interface name will be different I have chosen to determine the IP based on the CIDR, in my dev network this is 192.168.30.0/24. If you configure the interface name instead then calico will not start on nodes with different interface names
- Location of k8s-net-calico.yml:
~/kubespray-devcluster/kubespray/inventory/devcluster/group_vars/k8s_cluster/k8s-net-calico.yml - The sample k8s-cluster.yml looks like this (not included commented out sections)
calico_cni_name: k8s-pod-network
calico_pool_blocksize: 26 - Change the following:
calico_pool_blocksize: 24 - Add the following:
calico_veth_mtu: 1480
calico_network_backend: bird
calico_ipip_mode: 'Always'
calico_vxlan_mode: 'Never'
calico_ip_auto_method: 'cidr=192.168.30.0/24' - You can either manually edit the file by hand or utilise yq (installed in an earlier step):
yq -i -y '
.calico_pool_blocksize = 24 |
.calico_veth_mtu = 1480 |
.calico_network_backend = "bird" |
.calico_ipip_mode = "Always" |
.calico_vxlan_mode = "Never" |
.calico_ip_auto_method = "cidr=192.168.30.0/24"
' ~/kubespray-devcluster/kubespray/inventory/devcluster/group_vars/k8s_cluster/k8s-net-calico.yml
SSH keys
Prepare ssh passwordless connections to each host so that ansible can connect.
- Create the key pair using the command
ssh-keygen- this creates a private identification file:
id_ed25519and a public key:id_ed25519.pubsaved in/home/dev/.ssh/
- this creates a private identification file:
- Copy the ssh key to the Kubernetes nodes (no need to copy to load balancers)
- If there are only a small number of nodes, manual will be quicker
cd ~/.ssh/
ssh-copy-id dev-lb-v1
ssh-copy-id dev-m-v1
ssh-copy-id dev-w-v1 - If there are a lot of nodes then list them in a nodes file:
vi ~/nodesdev-lb-v1
dev-m-v1
dev-w-v1 - Create a changenodes script:
vi ~/changenodes.sh#!/usr/bin/env bash
set -euo pipefail
NODES_FILE="${1:-$HOME/nodes}" # pass a file path or default to ~/nodes
SSH_USER="${SSH_USER:-$USER}" # override with: SSH_USER=dev ./changenodes.sh
KEY="${SSH_KEY:-$HOME/.ssh/id_ed25519.pub}" # override with: SSH_KEY=~/.ssh/id_rsa.pub
TIMEOUT="${SSH_TIMEOUT:-7}"
# Ensure key exists; create one if missing
if [[ ! -f "$KEY" ]]; then
echo "Key $KEY not found. Creating ed25519 key..."
ssh-keygen -t ed25519 -f "${KEY%.pub}" -N "" -C "${USER}@$(hostname)" >/dev/null
fi
# Read nodes file line-by-line; ignore blanks and comments
while IFS= read -r host; do
[[ -z "$host" || "$host" =~ ^# ]] && continue
echo ">>> Installing key on ${SSH_USER}@${host}"
ssh-copy-id \
-i "$KEY" \
-o ConnectTimeout="$TIMEOUT" \
-o StrictHostKeyChecking=accept-new \
"${SSH_USER}@${host}"
done < "$NODES_FILE"
echo "All done. Quick test on the first host in the list:"
first_host="$(grep -vE '^\s*($|#)' "$NODES_FILE" | head -n1)"
[[ -n "$first_host" ]] && ssh -o BatchMode=yes -o ConnectTimeout="$TIMEOUT" "${SSH_USER}@${first_host}" 'echo "SSH OK on $(hostname)"' - Allow script to be executable:
chmod +x changenodes.sh - Run script:
./changenodes.sh>>> Installing key on dev@dev-lb-v1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/dev/.ssh/id_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
dev@dev-lb-v1's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -o 'ConnectTimeout=7' -o 'StrictHostKeyChecking=accept-new' 'dev@dev-lb-v1'"
and check to make sure that only the key(s) you wanted were added.
>>> Installing key on dev@dev-m-v1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/dev/.ssh/id_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
dev@dev-m-v1's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -o 'ConnectTimeout=7' -o 'StrictHostKeyChecking=accept-new' 'dev@dev-m-v1'"
and check to make sure that only the key(s) you wanted were added.
>>> Installing key on dev@dev-w-v1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/dev/.ssh/id_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
dev@dev-w-v1's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -o 'ConnectTimeout=7' -o 'StrictHostKeyChecking=accept-new' 'dev@dev-w-v1'"
and check to make sure that only the key(s) you wanted were added.
All done. Quick test on the first host in the list:
SSH OK on dev-lb-v1 - The ansible server will now be able to ssh to the three nodes without a password. It is good practice to check them all
- ssh dev-lb-v1
- ssh dev-m-v1
- ssh dev-w-v1
- If there are only a small number of nodes, manual will be quicker