K3D equal k3s in a container. a tools to create single- and multi-node k3s clusters.
Our favorite use case, is with podman and rootless. So there is some customization upstream to do.
One downside Iโve found with k3d is that the Kubernetes version it uses is behind the current k3s release.
1curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
2
3k3d completion zsh > "$ZSH/completions/_k3d"
1k3d cluster create test
2
3ERRO[0000] Failed to get nodes for cluster 'test': docker failed to get containers with labels 'map[k3d.cluster:test]': failed to list containers: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.46/containers/json?all=1&filters=%7B%22label%22%3A%7B%22app%3Dk3d%22%3Atrue%2C%22k3d.cluster%3Dtest%22%3Atrue%7D%7D": dial unix /var/run/docker.sock: connect: permission denied
1sudo mkdir -p /etc/containers/containers.conf.d
2sudo sh -c "echo 'service_timeout=0' > /etc/containers/containers.conf.d/timeout.conf"
3
4sudo ln -s /run/podman/podman.sock /var/run/docker.sock
5
6XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)}
7export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock
8export DOCKER_SOCK=$XDG_RUNTIME_DIR/podman/podman.sock
9
10systemctl --user enable --now podman.socket
If /sys/fs/cgroup/cgroup.controllers is present on your system, you are using v2, otherwise you are using v1.
in rootless, to run properly we need to enable CPU, CPUSET, and I/O delegation
1sudo mkdir -p /etc/systemd/system/user@.service.d
2cat > /etc/systemd/system/user@.service.d/delegate.conf <<EOF
3[Service]
4Delegate=cpu cpuset io memory pids
5EOF
6systemctl daemon-reload
1podman network create k3d
2podman network inspect k3d -f '{{ .DNSEnabled }}'
3true
1k3d registry create --default-network podman mycluster-registry
2
3k3d cluster create --registry-use mycluster-registry mycluster
Important
Pay attention to always export those variables if you use k3d with podman and rootless
XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)}export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sockexport DOCKER_SOCK=$XDG_RUNTIME_DIR/podman/podman.sock
1k3d cluster list
2k3d node list
3k3d registry list
config.yaml 1apiVersion: k3d.io/v1alpha5
2kind: Simple
3image: rancher/k3s:v1.29.3+k3s1
4
5metadata:
6 name: mycluster
7
8servers: 1
9agents: 1
10
11options:
12 k3s:
13 extraArgs:
14 - arg: "--disable=traefik"
15 nodeFilters:
16 - server:*
17 - arg: "--disable=servicelb"
18 nodeFilters:
19 - server:*
20
21registries:
22 use:
23 - mycluster-registry
1k3d cluster create --config config.yaml