General Checks Nice gist to troubleshoot etcd link 1journalctl -u rke2-server.service -f 2 3tail -f /var/lib/rancher/rke2/agent/containerd/containerd.log 4 5tail -f /var/lib/rancher/rke2/agent/logs/kubelet.log 6 7# crictl 8export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml 9/var/lib/rancher/rke2/bin/crictl ps 10 11/var/lib/rancher/rke2/bin/crictl --config /var/lib/rancher/rke2/agent/etc/crictl.yaml ps 12 13/var/lib/rancher/rke2/bin/crictl --runtime-endpoint unix:///run/k3s/containerd/containerd.sock ps -a 14 15/var/lib/rancher/rke2/bin/ctr --address /run/k3s/containerd/containerd.sock --namespace k8s.io container ls 16 17# Kubectl 18export KUBECONFIG=/etc/rancher/rke2/rke2.yaml 19export PATH=$PATH:/usr/local/bin/:/var/lib/rancher/rke2/bin/ 20kubectl get addon -A Check etcd endpoint status 1export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml 2etcdcontainer=$(/var/lib/rancher/rke2/bin/crictl ps --label io.kubernetes.container.name=etcd --quiet) 3/var/lib/rancher/rke2/bin/crictl exec $etcdcontainer sh -c "ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/rke2/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/rke2/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status --cluster --write-out=table" Check etcd health status 1export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml 2etcdcontainer=$(/var/lib/rancher/rke2/bin/crictl ps --label io.kubernetes.container.name=etcd --quiet) 3/var/lib/rancher/rke2/bin/crictl exec $etcdcontainer sh -c "ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/rke2/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/rke2/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint health --cluster --write-out=table"

See also documentation about Podman and Docker How to use a docker regsitry 1# list index catalog 2curl https://registry.k3s.example.com/v2/_catalog | jq 3 4# List tags available regarding an image 5curl https://registry.k3s.example.com/v2/myhaproxy/tags/list 6 7# list index catalog - with user/password 8curl https://registry-admin:<PWD>@registry.k3s.example.com/v2/_catalog | jq 9 10# list index catalog - when you need to specify the CA 11curl -u user:password https://<url>:<port>/v2/_catalog --cacert ca.crt | jq 12 13# list index catalog - for OCP 14curl -u user:password https://<url>:<port>/v2/ocp4/openshift4/tags/list | jq 15 16# Login to registry with podman 17podman login -u registry-admin -p <PWD> registry.k3s.example.com 18 19# Push images in the registry 20skopeo copy "--dest-creds=registry-admin:<PWD>" docker://docker.io/goharbor/harbor-core:v2.6.1 docker://registry.k3s.example.com/goharbor/harbor-core:v2.6.1 Install a Local private docker registry Change Docker Daemon config to allow insecure connection with your ip 1ip a 2sudo vi /etc/docker/daemon.json 1{ 2"insecure-registries": ["192.168.1.11:5000"] 3} 1sudo systemctl restart docker 2docker info Check docker config

Connection to k8s cluster Kubeconfig Define KUBECONFIG in your profile 1# Default one 2KUBECONFIG=~/.kube/config 3 4# Several context - to keep splited 5KUBECONFIG=~/.kube/k3sup-lab:~/.kube/k3s-dev 6 7# Or can be specified in command 8kubectl get pods --kubeconfig=admin-kube-config View and Set 1kubectl config view 2kubectl config current-context 3 4kubectl config set-context \ 5dev-context \ 6--namespace=dev-namespace \ 7--cluster=docker-desktop \ 8--user=dev-user 9 10kubectl config use-context lab Switch context 1#set Namespace 2kubectl config set-context --current --namespace=nexus3 3kubectl config get-contexts Kubecm The problem with the kubeconfig is that it get nexted in one kubeconfig and difficult to manage on long term. The best way to install it, is with Arkade arkade get kubecm - see arkade.

Trust a CA on Linux host 1# [RHEL] RootCA from DC need to be installed on host: 2cp my-domain-issuing.crt /etc/pki/ca-trust/source/anchors/my_domain_issuing.crt 3cp my-domain-rootca.crt /etc/pki/ca-trust/source/anchors/my_domain_rootca.crt 4update-ca-trust extract 5 6# [Ubuntu] 7sudo apt-get install -y ca-certificates 8sudo cp local-ca.crt /usr/local/share/ca-certificates 9sudo update-ca-certificates

Administration See what is currently installed 1helm list -A 2NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION 3nesux3 default 1 2022-08-12 20:01:16.0982324 +0200 CEST deployed nexus3-1.0.6 3.37.3 Install/Uninstall 1helm status nesux3 2helm uninstall nesux3 3helm install nexus3 <chart> # chart URL or path 4helm history nexus3 5 6# work even if already installed 7helm upgrade --install ingress-nginx ${DIR}/helm/ingress-nginx \ 8 --namespace=ingress-nginx \ 9 --create-namespace \ 10 -f ${DIR}/helm/ingress-values.yml 11 12#Make helm unsee an apps (it does not delete the apps) 13kubectl delete secret -l owner=helm,name=argo-cd Handle Helm Repo and Charts 1#Handle repo 2helm repo list 3helm repo add gitlab https://charts.gitlab.io/ 4helm repo update 5 6#Pretty usefull to configure 7helm show values elastic/eck-operator 8helm show values grafana/grafana --version 8.5.1 9 10#See different version available 11helm search repo hashicorp/vault 12helm search repo hashicorp/vault -l 13 14# download a chart 15helm fetch ingress/ingress-nginx --untar

Troubleshoot DNS vi dns.yml 1apiVersion: v1 2kind: Pod 3metadata: 4 name: dnsutils 5 namespace: default 6spec: 7 containers: 8 - name: dnsutils 9 image: registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 10 command: 11 - sleep 12 - "infinity" 13 imagePullPolicy: IfNotPresent 14 restartPolicy: Always deploy dnsutils 1k apply -f dns.yml 2pod/dnsutils created 3 4kubectl get pods dnsutils 5NAME READY STATUS RESTARTS AGE 6dnsutils 1/1 Running 0 36s Troubleshoot with dnsutils 1kubectl exec -i -t dnsutils -- nslookup kubernetes.default 2;; connection timed out; no servers could be reached 3command terminated with exit code 1 4 5kubectl exec -ti dnsutils -- cat /etc/resolv.conf 6search default.svc.cluster.local svc.cluster.local cluster.local example.local 7nameserver 10.43.0.10 8options ndots:5 9 10kubectl get endpoints kube-dns --namespace=kube-system 11NAME ENDPOINTS AGE 12kube-dns 10.42.0.6:53,10.42.0.6:53,10.42.0.6:9153 5d1h 13 14kubectl get svc kube-dns --namespace=kube-system 15NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 16kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 5d1h CURL 1cat << EOF > curl.yml 2apiVersion: v1 3kind: Pod 4metadata: 5 name: curl 6 namespace: default 7spec: 8 containers: 9 - name: curl 10 image: curlimages/curl 11 command: 12 - sleep 13 - "infinity" 14 imagePullPolicy: IfNotPresent 15 restartPolicy: Always 16EOF 17 18k apply -f curl.yml 19 20#Test du DNS 21kubectl exec -i -t curl -- curl -v telnet://10.43.0.10:53 22kubectl exec -i -t curl -- curl -v telnet://kube-dns.kube-system.svc.cluster.local:53 23kubectl exec -i -t curl -- nslookup kube-dns.kube-system.svc.cluster.local 24 25curl -k -I --resolve subdomain.domain.com:52.165.230.62 https:/subdomain.domain.com/

Install 1# curl method 2curl -LsSf https://astral.sh/uv/install.sh | sh 3 4# Pip method 5pip install uv Quick example 1pyenv install 3.12 2pyenv local 3.12 3python -m venv .venv 4source .venv/bin/activate 5pip install pandas 6python 7 8# equivalent in uv 9uv run --python 3.12 --with pandas python Usefull 1uv python list --only-installed 2uv python install 3.12 3uv venv /path/to/environment --python 3.12 4uv pip install django 5uv pip compile requirements.in -o requirements.txt 6 7uv init myproject 8uv sync 9uv run manage.py runserver Run as script Put before the import statements: 1#!/usr/bin/env -S uv run --script 2# /// script 3# requires-python = ">=3.12" 4# dependencies = [ 5# "ffmpeg-normalize", 6# ] 7# /// Then can be run with uv run sync-flickr-dates.py. uv will create a Python 3.12 venv for us. For me this is in ~/.cache/uv (which you can find via uv cache dir).

Tutorials https://vimvalley.com/ https://vim-adventures.com/ https://www.vimgolf.com/ Plugins 1# HCL 2mkdir -p ~/.vim/pack/jvirtanen/start 3cd ~/.vim/pack/jvirtanen/start 4git clone https://github.com/jvirtanen/vim-hcl.git 5 6# Justfile 7mkdir -p ~/.vim/pack/vendor/start 8cd ~/.vim/pack/vendor/start 9git clone https://github.com/NoahTheDuke/vim-just.git Fun Facts trigger a vim tutorial vimtutor the most powerful commands: . : Repeat the last modification. * : Where the cursor is located, keeps the word in memory and goes to the next occurrence. .* : together, repeat an action on the next word.

Troubleshooting cloud-init status --wait usefull for scripting, waiting cloud-init to finish before going to next step. cloud-init status --long 1status: done 2extended_status: done 3boot_status_code: enabled-by-generator 4last_update: Thu, 01 Jan 1970 00:00:55 +0000 5detail: DataSourceNoCloud [seed=/dev/sr0] 6errors: [] 7recoverable_errors: {} sudo cloud-init analyze show 1-- Boot Record 01 -- 2The total time elapsed since completing an event is printed after the "@" character. 3The time the event takes is printed after the "+" character. 4 5Starting stage: init-local 6|`->no cache found @00.00600s +00.00000s 7|`->found local data from DataSourceNoCloud @00.01500s +00.12600s 8Finished stage: (init-local) 00.75400 seconds 9 10Starting stage: init-network 11|`->restored from cache with run check: DataSourceNoCloud [seed=/dev/sr0] @04.21100s +00.00200s 12|`->setting up datasource @04.22800s +00.00000s 13|`->reading and applying user-data @04.23400s +00.00500s 14|`->reading and applying vendor-data @04.23900s +00.00000s 15|`->reading and applying vendor-data2 @04.23900s +00.00000s 16|`->activating datasource @04.27100s +00.00100s 17|`->config-seed_random ran successfully and took 0.000 seconds @04.29500s +00.00100s 18|`->config-write_files ran successfully and took 0.001 seconds @04.29600s +00.00100s 19|`->config-growpart ran successfully and took 0.562 seconds @04.29700s +00.56200s 20|`->config-resizefs ran successfully and took 0.193 seconds @04.86000s +00.19200s 21|`->config-mounts ran successfully and took 0.001 seconds @05.05200s +00.00100s 22|`->config-set_hostname ran successfully and took 0.004 seconds @05.05300s +00.00500s 23|`->config-update_hostname ran successfully and took 0.001 seconds @05.05800s +00.00100s 24|`->config-update_etc_hosts ran successfully and took 0.005 seconds @05.05900s +00.00500s 25|`->config-users_groups ran successfully and took 0.216 seconds @05.06400s +00.21600s 26|`->config-ssh ran successfully and took 0.404 seconds @05.28100s +00.40400s 27|`->config-set_passwords ran successfully and took 0.001 seconds @05.68500s +00.00200s 28Finished stage: (init-network) 01.50000 seconds 29 30Starting stage: modules-config 31|`->config-ssh_import_id ran successfully and took 0.001 seconds @07.43300s +00.00100s 32|`->config-locale ran successfully and took 0.003 seconds @07.43400s +00.00300s 33|`->config-grub_dpkg ran successfully and took 0.352 seconds @07.43700s +00.35200s 34|`->config-apt_configure ran successfully and took 0.049 seconds @07.79000s +00.04800s 35|`->config-timezone ran successfully and took 0.007 seconds @07.83900s +00.00700s 36|`->config-runcmd ran successfully and took 0.001 seconds @07.84600s +00.00100s 37|`->config-byobu ran successfully and took 0.000 seconds @07.84700s +00.00100s 38Finished stage: (modules-config) 00.45400 seconds 39 40Starting stage: modules-final 41|`->config-package_update_upgrade_install ran successfully and took 26.632 seconds @20.56700s +26.63300s 42|`->config-write_files_deferred ran successfully and took 0.001 seconds @47.20000s +00.00200s 43|`->config-reset_rmc ran successfully and took 0.000 seconds @47.20200s +00.00100s 44|`->config-scripts_vendor ran successfully and took 0.001 seconds @47.20300s +00.00000s 45|`->config-scripts_per_once ran successfully and took 0.000 seconds @47.20300s +00.00100s 46|`->config-scripts_per_boot ran successfully and took 0.000 seconds @47.20400s +00.00000s 47|`->config-scripts_per_instance ran successfully and took 0.000 seconds @47.20400s +00.00100s 48|`->config-scripts_user ran successfully and took 0.558 seconds @47.20500s +00.55800s 49|`->config-ssh_authkey_fingerprints ran successfully and took 0.005 seconds @47.76400s +00.00500s 50|`->config-keys_to_console ran successfully and took 0.054 seconds @47.76900s +00.05500s 51|`->config-install_hotplug ran successfully and took 0.001 seconds @47.82400s +00.00100s 52|`->config-final_message ran successfully and took 0.001 seconds @47.82500s +00.00100s 53Finished stage: (modules-final) 27.29600 seconds Check the logs: sudo tail -n 50 /var/log/cloud-init-output.log

Most usefull shortcuts Ctrl + r : Reverse search. (ctrl+r to go back through the history). Ctrl + l : Clear the screen (instead of using the “clear” command). Ctrl + p : Repeat the last command. Ctrl + x + Ctrl + e : Edit the current command in an external editor (need to define export EDITOR=vim). Ctrl + shift + v : Copy / paste in Linux. Ctrl + a : Move to the beginning of the line. Ctrl + e : Move to the end of the line. Ctrl + xx : Move to the opposite end of the line. Ctrl + left : Move left one word. Ctrl + right : Move right one word.