Repository navigation
kcli create cluster k3s fails when provisioning an IPv6 Only cluster. #822
Description
Description
As a user, I want to provision a Kubernetes cluster on nodes that do not have an IPv4 interface using k3s and kcli using the configuration below. On the host, net br0 is configured on a vlan that only has DHCPv6 using globally unique addresses.
api_ip: 2607:aaaa:bbb:830::50
apps: []
autoscale: False
cloud_api_internal: False
cloud_dns: False
cloud_lb: False
cloud_native: False
cloud_storage: False
cluster: myk3s
ctlplane_memory: 4096
ctlplane_numcpus: 2
ctlplanes: 1
ctlplanes_threaded: False
disk_size: 100
domain: example.local
extra_args:
- --cluster-cidr=2607:aaaa:bbb:1000::/64
- --service-cidr=2607:aaaa:bbb:1000:ff00::/112
- --cluster-dns=2607:aaaa:bbb:1000:ff00::10
- --tls-san=myk3s.example.local
- --flannel-backend=none
- --disable-network-policy=true
- --disable-traefik=true
extra_ctlplane_args: []
extra_ctlplane_disks: []
extra_ctlplane_networks: []
network: br0
extra_disks: []
extra_networks: []
extra_scripts: []
extra_worker_args: []
extra_worker_disks: []
extra_worker_networks: []
image: ubuntu2204
keys: []
kubevirt_disk_size: None
notify: False
notifycmd: kubectl get pod -A
numcpus: 2
pool: kcli
sdn: None
storedparameters: True
tempkey: False
threaded: False
token: supersecret
virtual_router_id: None
worker_memory: 8192
worker_numcpus: 4
workers: 0
workers_threaded: FalseExpected Result
k3s cluster is successfully provisioned with a single node.
Actual Result
automation-user@mf-ms-a2-01:~$ kcli create cluster k3s --pf k3s-cluster.yaml myk3s
Deploying Vms...
Waiting for vm myk3s-ctlplane-0 to finish customisation
Waiting for vm to be accessible...
Waiting for vm to be accessible...
Waiting for vm to be accessible...
Waiting for vm to be accessible...
Waiting for vm to be accessible...
Waiting for vm to be accessible...
Waiting for vm to be accessible...
Waiting for vm to be accessible...
Waiting for vm to be accessible...
myk3s-ctlplane-0 deployed on local
K3s cluster myk3s deployed!!!
INFO export KUBECONFIG=$HOME/.kcli/clusters/myk3s/auth/kubeconfig
INFO export PATH=$PWD:$PATH
automation-user@mf-ms-a2-01:~$ kcli ssh myk3s-ctlplane-0
Welcome to Ubuntu 22.04.5 LTS (GNU/Linux 5.15.0-143-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro
System information as of Mon Jul 28 04:00:20 UTC 2025
System load: 0.18
Usage of /: 1.8% of 96.73GB
Memory usage: 6%
Swap usage: 0%
Processes: 121
Users logged in: 0
IPv6 address for ens3: 2607:aaaa:bbb:830:5054:ff:fe5f:bba9
Expanded Security Maintenance for Applications is not enabled.
31 updates can be applied immediately.
28 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
New release '24.04.2 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
ubuntu@myk3s-ctlplane-0:~$ cat /var/log/cloud-init-output.log
Cloud-init v. 25.1.2-0ubuntu0~22.04.2 running 'init-local' at Mon, 28 Jul 2025 03:57:37 +0000. Up 6.59 seconds.
Cloud-init v. 25.1.2-0ubuntu0~22.04.2 running 'init' at Mon, 28 Jul 2025 03:57:43 +0000. Up 12.27 seconds.
ci-info: ++++++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++++++
ci-info: +--------+------+-----------------------------------------+-----------+--------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+------+-----------------------------------------+-----------+--------+-------------------+
ci-info: | ens3 | True | 2607:aaaa:bbb:830:5054:ff:fe5f:bba9/64 | . | global | 52:54:00:5f:bb:a9 |
ci-info: | ens3 | True | fe80::5054:ff:fe5f:bba9/64 | . | link | 52:54:00:5f:bb:a9 |
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
ci-info: | lo | True | ::1/128 | . | host | . |
ci-info: +--------+------+-----------------------------------------+-----------+--------+-------------------+
ci-info: ++++++++++++++++++++++++++++++++++Route IPv6 info++++++++++++++++++++++++++++++++++
ci-info: +-------+-------------------------+---------------------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Interface | Flags |
ci-info: +-------+-------------------------+---------------------------+-----------+-------+
ci-info: | 1 | 2607:aaaa:bbb:830::/64 | :: | ens3 | Ue |
ci-info: | 2 | fe80::/64 | :: | ens3 | U |
ci-info: | 3 | ::/0 | fe80::ae8b:a9ff:fe6e:13de | ens3 | UGe |
ci-info: | 5 | local | :: | ens3 | U |
ci-info: | 6 | local | :: | ens3 | U |
ci-info: | 7 | multicast | :: | ens3 | U |
ci-info: +-------+-------------------------+---------------------------+-----------+-------+
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub
The key fingerprint is:
SHA256:kURDC+v6xIJjfu+vhlepBd+WPbCB3rgzDP7RNw3+ypI root@myk3s-ctlplane-0
The key's randomart image is:
+---[RSA 3072]----+
| .o= |
| + + |
| . +. |
| .. ..o |
| .+S= *. |
| . o. B.*.oo |
| + ooo*.o..+.. |
| o .o+= =.Eo o |
| .. ==+oo .o.. |
+----[SHA256]-----+
Generating public/private ecdsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key
Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub
The key fingerprint is:
SHA256:ltjPxh8elEBtc11bwLa5S17a+sOvIeiO3Bga9ARH+d0 root@myk3s-ctlplane-0
The key's randomart image is:
+---[ECDSA 256]---+
| .... ..o+|
| ... + + +|
| . ...o = + |
| = .....E |
| o S o . |
| . + + o o .|
| . o * +oo= |
| + B o +++.|
| . +.+ o o+=|
+----[SHA256]-----+
Generating public/private ed25519 key pair.
Your identification has been saved in /etc/ssh/ssh_host_ed25519_key
Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub
The key fingerprint is:
SHA256:ed8YxA24fb1oL38JOSlbwWn89GnnXOcxo8N+Cc6oI6I root@myk3s-ctlplane-0
The key's randomart image is:
+--[ED25519 256]--+
| .. |
| .. o |
| o= o. |
| ....*...|
| S . o.B +|
| . o & O=|
| % O+X|
| . . . o B ==|
| E. . ..o ..=..|
+----[SHA256]-----+
Cloud-init v. 25.1.2-0ubuntu0~22.04.2 running 'modules:config' at Mon, 28 Jul 2025 03:57:55 +0000. Up 24.41 seconds.
Cloud-init v. 25.1.2-0ubuntu0~22.04.2 running 'modules:final' at Mon, 28 Jul 2025 03:57:55 +0000. Up 25.15 seconds.
Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB]
Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [14.1 MB]
Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2519 kB]
Get:7 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [375 kB]
Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [3912 kB]
Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [708 kB]
Get:10 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [992 kB]
Get:11 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [216 kB]
Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [21.7 kB]
Get:13 http://archive.ubuntu.com/ubuntu jammy/universe Translation-en [5652 kB]
Get:14 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [40.3 kB]
Get:15 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [8908 B]
Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [368 B]
Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 c-n-f Metadata [286 kB]
Get:18 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [217 kB]
Get:19 http://archive.ubuntu.com/ubuntu jammy/multiverse Translation-en [112 kB]
Get:20 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 c-n-f Metadata [8372 B]
Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2764 kB]
Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [439 kB]
Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [4058 kB]
Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [734 kB]
Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1225 kB]
Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [303 kB]
Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [28.7 kB]
Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [59.5 kB]
Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [14.2 kB]
Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [592 B]
Get:31 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [68.8 kB]
Get:32 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.4 kB]
Get:33 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [392 B]
Get:34 http://archive.ubuntu.com/ubuntu jammy-backports/restricted amd64 c-n-f Metadata [116 B]
Get:35 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [30.0 kB]
Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB]
Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B]
Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/multiverse amd64 c-n-f Metadata [116 B]
Fetched 39.3 MB in 3s (11.5 MB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
liburing2
The following NEW packages will be installed:
liburing2 qemu-guest-agent
0 upgraded, 2 newly installed, 0 to remove and 27 not upgraded.
Need to get 261 kB of archives.
After this operation, 906 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liburing2 amd64 2.1-2build1 [10.3 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 qemu-guest-agent amd64 1:6.2+dfsg-2ubuntu6.26 [251 kB]
dpkg-preconfigure: unable to re-open stdin: No such file or directory
Fetched 261 kB in 0s (1206 kB/s)
Selecting previously unselected package liburing2:amd64.
(Reading database ... 64592 files and directories currently installed.)
Preparing to unpack .../liburing2_2.1-2build1_amd64.deb ...
Unpacking liburing2:amd64 (2.1-2build1) ...
Selecting previously unselected package qemu-guest-agent.
Preparing to unpack .../qemu-guest-agent_1%3a6.2+dfsg-2ubuntu6.26_amd64.deb ...
Unpacking qemu-guest-agent (1:6.2+dfsg-2ubuntu6.26) ...
Setting up liburing2:amd64 (2.1-2build1) ...
Setting up qemu-guest-agent (1:6.2+dfsg-2ubuntu6.26) ...
qemu-guest-agent.service is a disabled or a static unit, not starting it.
Processing triggers for libc-bin (2.35-0ubuntu3.10) ...
Processing triggers for man-db (2.10.2-1) ...
Running kernel seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
Starting qemu-guest-agent (via systemctl): qemu-guest-agent.service.
Running script bootstrap.sh
Reading package lists...
Building dependency tree...
Reading state information...
curl is already the newest version (7.81.0-1ubuntu1.20).
curl set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 27 not upgraded.
[INFO] Finding release for channel stable
curl: (7) Couldn't connect to server
[INFO] Using v1.32.6+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.32.6+k3s1/sha256sum-amd64.txt
[ERROR] Download failed
cat: /var/lib/rancher/k3s/server/node-token: No such file or directory
sed: can't read /etc/rancher/k3s/k3s.yaml: No such file or directory
Running script keepalived.sh
Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
ipvsadm libsensors-config libsensors5 libsnmp-base libsnmp40
Suggested packages:
heartbeat ldirectord lm-sensors snmp-mibs-downloader
The following NEW packages will be installed:
ipvsadm keepalived libsensors-config libsensors5 libsnmp-base libsnmp40
0 upgraded, 6 newly installed, 0 to remove and 27 not upgraded.
Need to get 1797 kB of archives.
After this operation, 6185 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 libsensors-config all 1:3.6.0-7ubuntu1 [5274 B]
Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 libsensors5 amd64 1:3.6.0-7ubuntu1 [26.3 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libsnmp-base all 5.9.1+dfsg-1ubuntu2.8 [201 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libsnmp40 amd64 5.9.1+dfsg-1ubuntu2.8 [1070 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 keepalived amd64 1:2.2.4-0.2build1 [453 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 ipvsadm amd64 1:1.31-1build2 [42.2 kB]
Fetched 1797 kB in 0s (6177 kB/s)
Selecting previously unselected package libsensors-config.
(Reading database ... 64610 files and directories currently installed.)
Preparing to unpack .../0-libsensors-config_1%3a3.6.0-7ubuntu1_all.deb ...
Unpacking libsensors-config (1:3.6.0-7ubuntu1) ...
Selecting previously unselected package libsensors5:amd64.
Preparing to unpack .../1-libsensors5_1%3a3.6.0-7ubuntu1_amd64.deb ...
Unpacking libsensors5:amd64 (1:3.6.0-7ubuntu1) ...
Selecting previously unselected package libsnmp-base.
Preparing to unpack .../2-libsnmp-base_5.9.1+dfsg-1ubuntu2.8_all.deb ...
Unpacking libsnmp-base (5.9.1+dfsg-1ubuntu2.8) ...
Selecting previously unselected package libsnmp40:amd64.
Preparing to unpack .../3-libsnmp40_5.9.1+dfsg-1ubuntu2.8_amd64.deb ...
Unpacking libsnmp40:amd64 (5.9.1+dfsg-1ubuntu2.8) ...
Selecting previously unselected package keepalived.
Preparing to unpack .../4-keepalived_1%3a2.2.4-0.2build1_amd64.deb ...
Unpacking keepalived (1:2.2.4-0.2build1) ...
Selecting previously unselected package ipvsadm.
Preparing to unpack .../5-ipvsadm_1%3a1.31-1build2_amd64.deb ...
Unpacking ipvsadm (1:1.31-1build2) ...
Setting up ipvsadm (1:1.31-1build2) ...
Setting up libsnmp-base (5.9.1+dfsg-1ubuntu2.8) ...
Setting up libsensors-config (1:3.6.0-7ubuntu1) ...
Setting up libsensors5:amd64 (1:3.6.0-7ubuntu1) ...
Setting up libsnmp40:amd64 (5.9.1+dfsg-1ubuntu2.8) ...
Setting up keepalived (1:2.2.4-0.2build1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/keepalived.service → /lib/systemd/system/keepalived.service.
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for dbus (1.12.20-2ubuntu4.1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.10) ...
NEEDRESTART-VER: 3.5
NEEDRESTART-KCUR: 5.15.0-143-generic
NEEDRESTART-KEXP: 5.15.0-143-generic
NEEDRESTART-KSTA: 1
Synchronizing state of keepalived.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable keepalived
kcli boot finished, up 45.42 seconds
ubuntu@myk3s-ctlplane-0:~$
Key Issue
Github Releases are unreachable over IPv6
Reading package lists...
Building dependency tree...
Reading state information...
curl is already the newest version (7.81.0-1ubuntu1.20).
curl set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 27 not upgraded.
[INFO] Finding release for channel stable
curl: (7) Couldn't connect to server
[INFO] Using v1.32.6+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.32.6+k3s1/sha256sum-amd64.txt
[ERROR] Download failed
cat: /var/lib/rancher/k3s/server/node-token: No such file or directory
sed: can't read /etc/rancher/k3s/k3s.yaml: No such file or directory
Activity
karmab commentedon Jul 28, 2025
just to clarify, you're using the following parameters
Based on this link, you should set cluster and service cirds to other networks (I think you're using a subset as service cidr)
That said, are you able to route traffic using ipv6 only to github? supposedly, k3s.io redirects to github releases and github lacks ipv6 support, so that shouldnt work but as per the link, it seems possible
karmab commentedon Jul 28, 2025
Probably the article omits having to pick k3s manually, let's see if we can add support for downloading the binary from a reachable (private address)
estenrye commentedon Jul 28, 2025
@karmab I anonymized my public prefixes. I was looking at both
Muhammad Iqbal's article and this one by Shane Renshaw. I found Shane's more helpful, and found the part of his article where he points out that you need to download the binary. He also had an interesting point about configuring the docker registry for the IPv6 endpoint.
estenrye commentedon Jul 28, 2025
Another approach I was thinking about was to configure some sort of NAT64 setup with VyOS or something similar. But I like the idea of being able to externally load the binary.
estenrye commentedon Jul 29, 2025
It looks like the k3s install script has an environment variable
INSTALL_K3S_SKIP_DOWNLOADthat will disable the download of k3s hash and binary when set totruethe script will then expect the k3s binary to be located in
${BIN_DIR}/k3s. The script expectsBIN_DIRto be:BIN_DIR=/usr/local/binif root can write to it.BIN_DIR=/opt/binif root can't write to/usr/local/binand/opt/binis a directoryBIN_DIR=${INSTALL_K3S_BIN_DIR}ifINSTALL_K3S_BIN_DIRis not empty.It also looks like the script already provides an environment variable
GITHUB_URLto override the location from which it downloads assets from github.estenrye commentedon Jul 29, 2025
Workaround In Progress
On the host machine:
Additional cluster config
Still haven't had a successful install yet, but trying to figure it out.
One challenge to this approach is that the controlplanes support pushing files but the workers do not.
kcli/kvirt/cluster/k3s/workers.yml
Lines 20 to 21 in e66146c
karmab commentedon Jul 29, 2025
my bad, I should have clarified that I tested that yesterday, but saw that the k3s script (which I downloaded manually to the vm along with the k3s binary) still tries to reach https://update.k3s.io/v1-release/channels, despite setting the VERSION
estenrye commentedon Jul 29, 2025
It looks like that URL can be overridden by
INSTALL_K3S_CHANNEL_URLbut there doesn't seem to be a mechanism for telling it to use a local location in the script.estenrye commentedon Jul 29, 2025
I think we can utilize the airgap installation method to side-load the images from https://update.k3s.io/v1-release/channels
either the manually deploy images or possibly the embeded registry approaches might work I think.
estenrye commentedon Jul 29, 2025
Workaround in Progress
On the host machine:
Additional Cluster Config: