Making Sysbox work with k8s 1.33

v0.6.7 previously worked with AL2023 EKS 1.32.

With AL2023 EKS 1.33.5 I can't get Sysbox to work.

Tried both v0.6.7 and master.

Installing using a custom script derived from the Daemonset.

When using CRIO I have been building the latest 1.33 branch of CRIO (1.33.8) + Sysbox patch.

Without any changes on the node I get, ""reading ID mappings from "/proc/0/uid_map": open /proc/0/uid_map: no such file or directory: CreateContainerError"

Disabling UsernameSpaceSupport on the node fixes the, "reading ID mappings from "/proc/0/uid_map": open /proc/0/uid_map: no such file or directory: CreateContainerError". However, PVCs get permission denied errors on writes making this unusable for us.

Tried master + UsernameSpaceSupport enabled + containerd and get an "aws cni can't get IP address" error (I don't get this error with flannel in a lima VM with AL2023.)

Going to try master + UsernamespaceSupport enabled + CRIO (without the auto userns annotation on the pod) to see if it somehow behaves differently.

Has anyone cracked the Contra code for getting Sysbox to work on EKS 1.33?

Ideally I'd love to have Sysbox work with containerd, but I'll settle for CRIO as long as it works.

If you got it to work with EKS and v1.33, please share what works for you. Thanks!

Also hit the same error on GKE (k8s 1.33/1.34, kernel 6.8.0-1041-gke, Ubuntu 24.04 nodes, sysbox 0.7.0). Sysbox worked on 1.32 with older node images but fails on 1.33+. This is using CRI-O.

Dropping the io.kubernetes.cri-o.userns-mode annotation gets past the /proc/0/uid_map error a bit. The next errors on pod creation are things like mount through procfd: operation not permitted for sysfs/procfs/mqueue, and setns: operation not permitted for IPC amongst other errors.

I spent a day with Claude and it came up with patches that work for my workload (I can start a pod with sysbox runc, exec in via another pod attached to std devices), but I am almost certain that it has broken the host/sysbox/pod namespace isolation and mounting so don't want to share anything here.

I managed to have sysbox from the master branch work on GKE (k8s 1.33 / kernel 6.8.0-1041-gke). The only patch I had to make is to hardcode the volatile option that is somehow not read from config.

--- a/pkg/config/config.go
+++ b/pkg/config/config.go
@@ -195,6 +195,9 @@
 		GraphDriverName:    c.Storage,
 		GraphDriverOptions: c.StorageOptions,
 		PullOptions:        c.pullOptions,
+		// Disable volatile overlay mounts (incompatible with sysbox user namespaces ?).
+		// See https://github.com/nestybox/sysbox/issues/975
+		DisableVolatile: true,
 	})
 }

For running the pods with sysbox runtime class, the key is to use the user namespace support that is available by default in k8s 1.33, hostUsers: false, instead of the annotations.

@inoc603

I managed to have sysbox from the master branch work on GKE (k8s 1.33 / kernel 6.8.0-1041-gke). The only patch I had to make is to hardcode the volatile option that is somehow not read from config.
--- a/pkg/config/config.go
+++ b/pkg/config/config.go

Interesting @inoc603, where are you making that change? I don't see those files in sysbox repos?

@jamonation

I managed to have sysbox from the master branch work on GKE (k8s 1.33 / kernel 6.8.0-1041-gke). The only patch I had to make is to hardcode the volatile option that is somehow not read from config.
--- a/pkg/config/config.go
+++ b/pkg/config/config.go
Interesting @inoc603, where are you making that change? I don't see those files in sysbox repos?

@jamonation It's for github.com/nestybox/cri-o. It's one of the binaries that need to be built for the sysbox image. I only kept version cri-o 1.33 for the environment I'm working on.

Have folks tried the 0.7.0 release without any patches?

feat(k8s): Docker-in-Docker support via rootless DinD on dedicated Ub…

@inoc603

For running the pods with sysbox runtime class, the key is to use the user namespace support that is available by default in k8s 1.33, hostUsers: false, instead of the annotations.

@inoc603 Thanks for suggesting to run withhostUsers: false instead of the io.kubernetes.cri-o.userns-mode annotation. I've tried sysbox v0.7.0 without any patches and it worked on GKE 1.33 with Ubuntu 24.04 node pool.

@EddieX64

Hi @EddieX64 ,

For me it didnt work for GKE 1.33.12 and 0.7.0

and 0.6.7 didnt work because 1.33.12 has ubuntu 24.

i can see daemonset said successfull installation but n restarting kubelet it get stuck

$ k logs -n sysbox-system -l sysbox-install=yes --all-containers=true -f
Running CRI-O installer agent on the host (may take several seconds) ...
Removing CRI-O installer agent from the host ...
Configuring CRI-O ...
Configuring CRI-O for GKE
Adding K8s label "sysbox-runtime=installing" to node ...
node/gke-ops-platform-dev-sysbox-pool-6d59-5222ef95-34t0 labeled
Installing Sysbox dependencies on host ...
Skipping shiftfs installation (kernel version 6.8 is above the max required for shiftfs (6.2)).
Deploying Sysbox installer helper on the host ...
Running Sysbox installer helper on the host (may take several seconds) ...
Stopping the Sysbox installer helper on the host ...
Removing Sysbox installer helper from the host ...
Failed to get unit file state for sysbox.service: No such file or directory
Installing Sysbox on host ...
Configuring host sysctls ...
kernel.unprivileged_userns_clone = 1
fs.inotify.max_queued_events = 1048576
fs.inotify.max_user_watches = 1048576
fs.inotify.max_user_instances = 1048576
kernel.keys.maxkeys = 20000
kernel.keys.maxbytes = 1400000
kernel.pid_max = 4194304
vm.max_map_count = 262144
Created symlink from /etc/systemd/system/multi-user.target.wants/sysbox.service to /usr/lib/systemd/system/sysbox.service.
Created symlink from /etc/systemd/system/sysbox.service.wants/sysbox-mgr.service to /usr/lib/systemd/system/sysbox-mgr.service.
Created symlink from /etc/systemd/system/sysbox.service.wants/sysbox-fs.service to /usr/lib/systemd/system/sysbox-fs.service.
Starting Sysbox ...
Adding Sysbox to CRI-O config ...
Detected containerd version 2.0.4.
containerd version 2.0.4 does not support user-namespaces with alternative runtimes (buggy versions 2.0.1 -> 2.0.4).
Restarting CRI-O ...
Deploying Kubelet config agent on the host ...
Running Kubelet config agent on the host (will restart Kubelet and temporary bring down all pods on this node for ~1 min)

after this i just check and daemonset pod is stuck since long

...
NAME                      READY   STATUS              RESTARTS   AGE
sysbox-deploy-k8s-98dlh   0/1     ContainerCreating   1          105s

if i describe it it showing below error in event


$ k describe pods sysbox-deploy-k8s-98dlh -n sysbox-system

Name:             sysbox-deploy-k8s-98dlh
Namespace:        sysbox-system
Priority:         0
Service Account:  sysbox-label-node
Node:             gke-ops-platform-dev-sysbox-pool-6d59-5222ef95-34t0/10.226.212.17
Start Time:       Sat, 23 May 2026 22:51:01 +0200
Labels:           controller-revision-hash=7546467d6c
                  pod-template-generation=2
                  sysbox-install=yes
Annotations:      cluster-autoscaler.kubernetes.io/safe-to-evict: true
Status:           Running
IP:
IPs:              <none>
Controlled By:    DaemonSet/sysbox-deploy-k8s
Containers:
  sysbox-deploy-k8s:
    Container ID:
    Image:         /remote-ghcr/nestybox/sysbox-deploy-k8s:v0.7.0-0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      bash
      -c
      /opt/sysbox/scripts/sysbox-deploy-k8s.sh ce install
    State:          Waiting
      Reason:       ContainerCreating
    Last State:     Terminated
      Reason:       ContainerStatusUnknown
      Message:      The container could not be located when the pod was deleted.  The container used to be Running
      Exit Code:    137
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Ready:          False
    Restart Count:  1
    Environment:
      NODE_NAME:           (v1:spec.nodeName)
      SYSBOX_MGR_CONFIG:  <set to the key 'SYSBOX_MGR_CONFIG' of config map 'sysbox-operational-attributes'>  Optional: false
      SYSBOX_FS_CONFIG:   <set to the key 'SYSBOX_FS_CONFIG' of config map 'sysbox-operational-attributes'>   Optional: false
      SYSBOX_TAINT:       startup-taint.cluster-autoscaler.kubernetes.io/sysbox-runtime=not-running:NoSchedule
    Mounts:
      /mnt/host/etc from host-etc (rw)
      /mnt/host/etc/systemd/system from host-etc-systemd (rw)
      /mnt/host/lib/sysctl.d from host-lib-sysctl (rw)
      /mnt/host/lib/systemd/system from host-lib-systemd (rw)
      /mnt/host/opt/bin from host-opt-bin (rw)
      /mnt/host/opt/lib/modules-load.d from host-opt-lib-mod-load (rw)
      /mnt/host/opt/lib/sysctl.d from host-opt-lib-sysctl (rw)
      /mnt/host/opt/local/bin from host-opt-local-bin (rw)
      /mnt/host/os-release from host-osrelease (rw)
      /mnt/host/run from host-run (rw)
      /mnt/host/usr/bin from host-usr-bin (rw)
      /mnt/host/usr/lib/modules-load.d from host-usr-lib-mod-load (rw)
      /mnt/host/usr/local/bin from host-usr-local-bin (rw)
      /mnt/host/var/lib from host-var-lib (rw)
      /run/systemd from host-run-systemd (rw)
      /var/run/dbus from host-dbus (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pgz8h (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   False
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  host-etc:
    Type:          HostPath (bare host directory volume)
    Path:          /etc
    HostPathType:
  host-osrelease:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/os-release
    HostPathType:
  host-dbus:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/dbus
    HostPathType:
  host-run-systemd:
    Type:          HostPath (bare host directory volume)
    Path:          /run/systemd
    HostPathType:
  host-lib-systemd:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/systemd/system
    HostPathType:
  host-etc-systemd:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/systemd/system
    HostPathType:
  host-lib-sysctl:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/sysctl.d
    HostPathType:
  host-opt-lib-sysctl:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/lib/sysctl.d
    HostPathType:
  host-usr-bin:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/bin/
    HostPathType:
  host-opt-bin:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/bin/
    HostPathType:
  host-usr-local-bin:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/bin/
    HostPathType:
  host-opt-local-bin:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/local/bin/
    HostPathType:
  host-usr-lib-mod-load:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/lib/modules-load.d
    HostPathType:
  host-opt-lib-mod-load:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/lib/modules-load.d
    HostPathType:
  host-run:
    Type:          HostPath (bare host directory volume)
    Path:          /run
    HostPathType:
  host-var-lib:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib
    HostPathType:
  kube-api-access-pgz8h:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
                             startup-taint.cluster-autoscaler.kubernetes.io/sysbox-runtime=not-running:NoSchedule
Events:
  Type     Reason                  Age                From               Message
  ----     ------                  ----               ----               -------
  Normal   Scheduled               2m17s              default-scheduler  Successfully assigned sysbox-system/sysbox-deploy-k8s-98dlh to gke-ops-platform-dev-sysbox-pool-6d59-5222ef95-34t0
  Normal   Pulling                 2m15s              kubelet            Pulling image "emote-ghcr/nestybox/sysbox-deploy-k8s:v0.7.0-0"
  Normal   Pulled                  2m2s               kubelet            Successfully pulled image "/nestybox/sysbox-deploy-k8s:v0.7.0-0" in 12.313s (12.313s including waiting). Image size: 774868586 bytes.
  Normal   Created                 2m2s               kubelet            Created container: sysbox-deploy-k8s
  Normal   Started                 2m2s               kubelet            Started container sysbox-deploy-k8s
  Warning  FailedMount             89s (x4 over 92s)  kubelet            MountVolume.SetUp failed for volume "kube-api-access-pgz8h" : object "sysbox-system"/"kube-root-ca.crt" not registered
  Warning  NetworkNotReady         88s (x3 over 92s)  kubelet            network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: plugin status uninitialized
  Warning  FailedCreatePodSandBox  10s (x2 over 55s)  kubelet            Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = creating pod sandbox with name "k8s_sysbox-deploy-k8s-98dlh_sysbox-system_b467def7-3f96-406c-bb5c-a65020864eb9_0": unable to try pulling possible OCI artifact: get manifest: build image source: pinging container registry registry.k8s.io: Get "https://registry.k8s.io/v2/": dial tcp 34.96.108.209:443: i/o timeout

It look like its not able to remove containerd, as both containerd and crio are still on gke node.

Although kubelet-config-helper.service completed its task but kubelet has two --container-runtime-endpoint arguments

$ sudo systemctl status kubelet-config-helper.service
● kubelet-config-helper.service - Kubelet config service
     Loaded: loaded (/usr/lib/systemd/system/kubelet-config-helper.service; static)
     Active: active (exited) since Sun 2026-05-24 00:05:17 UTC; 2s ago
    Process: 14112 ExecStart=/bin/sh -c /usr/local/bin/kubelet-config-helper.sh (code=exited, status=0/SUCCESS)
   Main PID: 14112 (code=exited, status=0/SUCCESS)
        CPU: 135ms

May 24 00:05:17 gke-ops-platform-dev-sysbox-pool-6d59-5222ef95-k9p9 sh[14113]: unix:///var/run/crio/crio.sock'
May 24 00:05:17 gke-ops-platform-dev-sysbox-pool-6d59-5222ef95-k9p9 sh[14113]: + set -e
May 24 00:05:17 gke-ops-platform-dev-sysbox-pool-6d59-5222ef95-k9p9 sh[14113]: + [[ unix:///run/containerd/containerd.sock
May 24 00:05:17 gke-ops-platform-dev-sysbox-pool-6d59-5222ef95-k9p9 sh[14113]: unix:///var/run/crio/crio.sock == '' ]]
May 24 00:05:17 gke-ops-platform-dev-sysbox-pool-6d59-5222ef95-k9p9 sh[14113]: + [[ unix:///run/containerd/containerd.sock
May 24 00:05:17 gke-ops-platform-dev-sysbox-pool-6d59-5222ef95-k9p9 sh[14113]: unix:///var/run/crio/crio.sock =~ crio ]]
May 24 00:05:17 gke-ops-platform-dev-sysbox-pool-6d59-5222ef95-k9p9 sh[14113]: + echo 'Kubelet is already using CRI-O; no action will be taken.'
May 24 00:05:17 gke-ops-platform-dev-sysbox-pool-6d59-5222ef95-k9p9 sh[14113]: Kubelet is already using CRI-O; no action will be taken.
May 24 00:05:17 gke-ops-platform-dev-sysbox-pool-6d59-5222ef95-k9p9 sh[14113]: + return
May 24 00:05:17 gke-ops-platform-dev-sysbox-pool-6d59-5222ef95-k9p9 systemd[1]: Finished kubelet-config-helper.service - Kubelet config service.

kubelet status

$ ps aux | grep kubelet | grep container-runtime-endpoint
root       13965  1.9  0.3 2420552 99592 ?       Ssl  00:03   0:02 /home/kubernetes/bin/the-kubelet --v=2 --cloud-provider=external --experimental-mounter-path=/home/kubernetes/containerized_mounter/mounter --cert-dir=/var/lib/kubelet/pki/ --kubeconfig=/var/lib/kubelet/kubeconfig --image-credential-provider-config=/etc/srv/kubernetes/cri_auth_config.yaml --image-credential-provider-bin-dir=/home/kubernetes/bin --max-pods=110 --register-with-taints=[startup-taint.cluster-autoscaler.kubernetes.io/cilium-agent-not-ready=true:NoExecute,startup-taint.cluster-autoscaler.kubernetes.io/sysbox-runtime=not-running:NoSchedule](http://startup-taint.cluster-autoscaler.kubernetes.io/cilium-agent-not-ready=true:NoExecute,startup-taint.cluster-autoscaler.kubernetes.io/sysbox-runtime=not-running:NoSchedule) --node-labels=[addon.gke.io/node-local-dns-ds-ready=true,cloud.google.com/gke-boot-disk=pd-ssd,cloud.google.com/gke-container-runtime=containerd,cloud.google.com/gke-cpu-scaling-level=8,cloud.google.com/gke-logging-variant=DEFAULT,cloud.google.com/gke-max-pods-per-node=110,cloud.google.com/gke-memory-gb-scaling-level=32,cloud.google.com/gke-netd-ready=true,cloud.google.com/gke-nodepool=sysbox-pool-6d59,cloud.google.com/gke-os-distribution=ubuntu,cloud.google.com/gke-provisioning=standard,cloud.google.com/gke-stack-type=IPV4,cloud.google.com/machine-family=n2d,cloud.google.com/private-node=true,iam.gke.io/gke-metadata-server-enabled=true,node.kubernetes.io/masq-agent-ds-ready=true](http://addon.gke.io/node-local-dns-ds-ready=true,cloud.google.com/gke-boot-disk=pd-ssd,cloud.google.com/gke-container-runtime=containerd,cloud.google.com/gke-cpu-scaling-level=8,cloud.google.com/gke-logging-variant=DEFAULT,cloud.google.com/gke-max-pods-per-node=110,cloud.google.com/gke-memory-gb-scaling-level=32,cloud.google.com/gke-netd-ready=true,cloud.google.com/gke-nodepool=sysbox-pool-6d59,cloud.google.com/gke-os-distribution=ubuntu,cloud.google.com/gke-provisioning=standard,cloud.google.com/gke-stack-type=IPV4,cloud.google.com/machine-family=n2d,cloud.google.com/private-node=true,iam.gke.io/gke-metadata-server-enabled=true,node.kubernetes.io/masq-agent-ds-ready=true) --volume-plugin-dir=/home/kubernetes/flexvolume --node-status-max-images=25 --container-runtime-endpoint=unix:///run/containerd/containerd.sock --runtime-cgroups=/system.slice/containerd.service --registry-qps=10 --registry-burst=20 --config /home/kubernetes/kubelet-config.yaml --pod-sysctls=net.core.optmem_max=20480,net.core.somaxconn=1024,net.ipv4.conf.all.accept_redirects=0,net.ipv4.conf.all.forwarding=1,net.ipv4.conf.all.route_localnet=1,net.ipv4.conf.default.forwarding=1,net.ipv4.ip_forward=1,net.ipv4.tcp_fin_timeout=60,net.ipv4.tcp_keepalive_intvl=60,net.ipv4.tcp_keepalive_probes=5,net.ipv4.tcp_keepalive_time=300,net.ipv4.tcp_rmem=4096 [87380 6291456](tel:873806291456),net.ipv4.tcp_syn_retries=6,net.ipv4.tcp_tw_reuse=0,net.ipv4.tcp_wmem=4096 [16384 4194304](tel:163844194304),net.ipv4.udp_rmem_min=4096,net.ipv4.udp_wmem_min=4096,net.ipv6.conf.all.disable_ipv6=1,net.ipv6.conf.default.accept_ra=0,net.ipv6.conf.default.disable_ipv6=1,[net.netfilter.nf](http://net.netfilter.nf/)_conntrack_generic_timeout=600,[net.netfilter.nf](http://net.netfilter.nf/)_conntrack_tcp_be_liberal=1,[net.netfilter.nf](http://net.netfilter.nf/)_conntrack_tcp_timeout_close_wait=3600,[net.netfilter.nf](http://net.netfilter.nf/)_conntrack_tcp_timeout_established=86400 --version=v1.33.12-gke.1000000 --container-runtime-endpoint=unix:///var/run/crio/crio.sock --image-service-endpoint=unix:///var/run/crio/crio.sock --runtime-cgroups=/system.slice/crio.service --runtime-request-timeout=6m

It was failing for me on sysbox nodepool version 1.33.

This is what got me working on AKS with below specs:

sysbox nodepool version : v1.33.1
OS : Ubuntu 22.04.5 LTS
Container Runtime : cri-o://1.33.3
sysbox image : v0.7.0-0

Removed : spec .hostUsers: false from POD specifications . Redeployed and it worked

Navigation Menu

Repository navigation

Making Sysbox work with k8s 1.33 #987

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Making Sysbox work with k8s 1.33 #987

Description

Activity

jamonation commented on Mar 11, 2026

inoc603 commented on Mar 11, 2026

jamonation commented on Mar 11, 2026

inoc603 commented on Mar 12, 2026

AkshayPall commented on Mar 13, 2026

EddieX64 commented on Apr 6, 2026

imranrazakhan commented on May 23, 2026

tonycesta2628 commented on Aug 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions