Connection to k8s cluster Kubeconfig Define KUBECONFIG in your profile 1# Default one 2KUBECONFIG=~/.kube/config 3 4# Several context - to keep splited 5KUBECONFIG=~/.kube/k3sup-lab:~/.kube/k3s-dev 6 7# Or can be specified in command 8kubectl get pods --kubeconfig=admin-kube-config View and Set 1kubectl config view 2kubectl config current-context 3 4kubectl config set-context \ 5dev-context \ 6--namespace=dev-namespace \ 7--cluster=docker-desktop \ 8--user=dev-user 9 10kubectl config use-context lab Switch context 1#set Namespace 2kubectl config set-context --current --namespace=nexus3 3kubectl config get-contexts Kubecm The problem with the kubeconfig is that it get nexted in one kubeconfig and difficult to manage on long term. The best way to install it, is with Arkade arkade get kubecm - see arkade.
See also documentation about Podman and Docker
How to use a docker regsitry 1# list index catalog 2curl https://registry.k3s.example.com/v2/_catalog | jq 3 4# List tags available regarding an image 5curl https://registry.k3s.example.com/v2/myhaproxy/tags/list 6 7# list index catalog - with user/password 8curl https://registry-admin:<PWD>@registry.k3s.example.com/v2/_catalog | jq 9 10# list index catalog - when you need to specify the CA 11curl -u user:password https://<url>:<port>/v2/_catalog --cacert ca.crt | jq 12 13# list index catalog - for OCP 14curl -u user:password https://<url>:<port>/v2/ocp4/openshift4/tags/list | jq 15 16# Login to registry with podman 17podman login -u registry-admin -p <PWD> registry.k3s.example.com 18 19# Push images in the registry 20skopeo copy "--dest-creds=registry-admin:<PWD>" docker://docker.io/goharbor/harbor-core:v2.6.1 docker://registry.k3s.example.com/goharbor/harbor-core:v2.6.1 Install a Local private docker registry Change Docker Daemon config to allow insecure connection with your ip 1ip a 2sudo vi /etc/docker/daemon.json 1{ 2"insecure-registries": ["192.168.1.11:5000"] 3} 1sudo systemctl restart docker 2docker info Check docker config
General Checks Nice gist to troubleshoot etcd link
1journalctl -u rke2-server.service -f 2 3tail -f /var/lib/rancher/rke2/agent/containerd/containerd.log 4 5tail -f /var/lib/rancher/rke2/agent/logs/kubelet.log 6 7# crictl 8export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml 9/var/lib/rancher/rke2/bin/crictl ps 10 11/var/lib/rancher/rke2/bin/crictl --config /var/lib/rancher/rke2/agent/etc/crictl.yaml ps 12 13/var/lib/rancher/rke2/bin/crictl --runtime-endpoint unix:///run/k3s/containerd/containerd.sock ps -a 14 15/var/lib/rancher/rke2/bin/ctr --address /run/k3s/containerd/containerd.sock --namespace k8s.io container ls 16 17# Kubectl 18export KUBECONFIG=/etc/rancher/rke2/rke2.yaml 19export PATH=$PATH:/usr/local/bin/:/var/lib/rancher/rke2/bin/ 20kubectl get addon -A Check etcd endpoint status 1export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml 2etcdcontainer=$(/var/lib/rancher/rke2/bin/crictl ps --label io.kubernetes.container.name=etcd --quiet) 3/var/lib/rancher/rke2/bin/crictl exec $etcdcontainer sh -c "ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/rke2/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/rke2/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status --cluster --write-out=table" Check etcd health status 1export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml 2etcdcontainer=$(/var/lib/rancher/rke2/bin/crictl ps --label io.kubernetes.container.name=etcd --quiet) 3/var/lib/rancher/rke2/bin/crictl exec $etcdcontainer sh -c "ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/rke2/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/rke2/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint health --cluster --write-out=table"
Install Client 1# most simple 2arkade get doctl 3 4# normal way 5curl -OL https://github.com/digitalocean/doctl/releases/download/v1.104.0/doctl-1.104.0-linux-amd64.tar.gz 6tar xf doctl-1.104.0-linux-amd64.tar.gz 7mv doctl /usr/local/bin 8 9# Auto-Completion ZSH 10 doctl completion zsh > $ZSH/completions/_doctl Basics find possible droplet 1doctl compute region list 2doctl compute size list 3doctl compute image list-distribution 4doctl compute image list --public Auth 1doctl auth init --context test 2doctl auth list 3doctl auth switch --context test2 Create Project 1doctl projects create --name rkub --environment staging --purpose "stage rkub with github workflows" Create VM 1doctl compute ssh-key list 2doctl compute droplet create test --region fra1 --image rockylinux-9-x64 --size s-1vcpu-1gb --ssh-keys <fingerprint> 3doctl compute droplet delete test -f with Terraform 1export DO_PAT="dop_v1_xxxxxxxxxxxxxxxx" 2doctl auth init --context rkub 3 4# inside a dir with a tf file 5terraform init 6terraform validate 7terraform plan -var "do_token=${DO_PAT}" 8terraform apply -var "do_token=${DO_PAT}" -auto-approve 9 10# clean apply 11terraform plan -out=infra.tfplan -var "do_token=${DO_PAT}" 12terraform apply infra.tfplan 13 14# Control 15terraform show terraform.tfstate 16 17# Destroy 18terraform plan -destroy -out=terraform.tfplan -var "do_token=${DO_PAT}" 19terraform apply terraform.tfplan Connect to Droplet with private ssh key ssh root@$(terraform output -json ip_address_workers | jq -r ‘.[0]’) -i .key
A Command Builder for Go Cobra is a Go library for building command-line applications.
It is used by many well-known tools from the Go ecosystem because it gives us a convenient structure for:
commands subcommands arguments flags validation help shell completion error handling A Cobra application usually reads naturally:
1app command argument --flag value For example:
1git clone repository --bare 2kubectl get pods --namespace production A useful introduction is also available here:
K3D equal k3s in a container. a tools to create single- and multi-node k3s clusters. Our favorite use case, is with podman and rootless. So there is some customization upstream to do.
One downside I’ve found with k3d is that the Kubernetes version it uses is behind the current k3s release.
Note for ARM PC:
1sudo apt install qemu-user-static 2podman run --rm --privileged multiarch/qemu-user-static --reset -p yes Install 1# Manual way 2curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash 3 4# or with arkade: 5arkade get k3d 6 7# Auto-completion 8k3d completion zsh > "$ZSH/completions/_k3d" Tweaks for podman and rootless The issue: 1k3d cluster create test 2 3ERRO[0000] Failed to get nodes for cluster 'test': docker failed to get containers with labels 'map[k3d.cluster:test]': failed to list containers: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.46/containers/json?all=1&filters=%7B%22label%22%3A%7B%22app%3Dk3d%22%3Atrue%2C%22k3d.cluster%3Dtest%22%3Atrue%7D%7D": dial unix /var/run/docker.sock: connect: permission denied The solution: 1# TODO 2loginctl enable-linger $(whoami) 3 4# Either reload terminal or do below: 5export XDG_RUNTIME_DIR=/tmp/run-$(id -u) 6mkdir -p $XDG_RUNTIME_DIR 7chmod 700 $XDG_RUNTIME_DIR 8 9sudo mkdir -p /etc/containers/containers.conf.d 10sudo sh -c "echo 'service_timeout=0' > /etc/containers/containers.conf.d/timeout.conf" 11 12sudo ln -s /run/podman/podman.sock /var/run/docker.sock 13 14XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)} 15export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock 16export DOCKER_SOCK=$XDG_RUNTIME_DIR/podman/podman.sock 17 18systemctl --user enable --now podman.socket If /sys/fs/cgroup/cgroup.controllers is present on your system, you are using v2, otherwise you are using v1.
What is ArgoCD ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes.
Your Git repository is the single source of truth; ArgoCD watches it and continuously syncs the cluster to match the desired state.
1Git repo ──► ArgoCD ──► Kubernetes cluster Key concepts Application — a group of Kubernetes resources described in Git. Project — groups applications and scopes their permissions. Source — the repo to render from (git, helm, kustomize, …). Sync — reconcile the live cluster with the desired state. Install 1kubectl create namespace argocd 2kubectl apply -n argocd \ 3 -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml 4 5# expose the UI 6kubectl port-forward svc/argocd-server -n argocd 8080:443 CLI 1argocd login localhost:8080 2 3argocd app create myapp \ 4 --repo https://github.com/org/repo.git \ 5 --path deploy \ 6 --dest-server https://kubernetes.default.svc \ 7 --dest-namespace default 8 9argocd app list 10argocd app sync myapp 11argocd app get myapp Ops notes With syncPolicy.automated set, ArgoCD re-applies any drift on its own. Don’t commit plaintext secrets to Git — pair ArgoCD with an external secret manager (Vault, sealed-secrets, SOPS/ksops).
Create a small infra for kubernetes 1 #On your Azure CLI 2 az --version # Version expected 2.1.0 or higher 3 4 az group delete --name kubernetes -y 5 6 az group create -n kubernetes -l westeurope 7 8 az network vnet create -g kubernetes \ 9 -n kubernetes-vnet \ 10 --address-prefix 10.240.0.0/24 \ 11 --subnet-name kubernetes-subnet 12 13 az network nsg create -g kubernetes -n kubernetes-nsg 14 15 az network vnet subnet update -g kubernetes \ 16 -n kubernetes-subnet \ 17 --vnet-name kubernetes-vnet \ 18 --network-security-group kubernetes-nsg 19 20 az network nsg rule create -g kubernetes \ 21 -n kubernetes-allow-ssh \ 22 --access allow \ 23 --destination-address-prefix '*' \ 24 --destination-port-range 22 \ 25 --direction inbound \ 26 --nsg-name kubernetes-nsg \ 27 --protocol tcp \ 28 --source-address-prefix '*' \ 29 --source-port-range '*' \ 30 --priority 1000 31 32 az network nsg rule create -g kubernetes \ 33 -n kubernetes-allow-api-server \ 34 --access allow \ 35 --destination-address-prefix '*' \ 36 --destination-port-range 6443 \ 37 --direction inbound \ 38 --nsg-name kubernetes-nsg \ 39 --protocol tcp \ 40 --source-address-prefix '*' \ 41 --source-port-range '*' \ 42 --priority 1001 43 44 az network nsg rule list -g kubernetes --nsg-name kubernetes-nsg --query "[].{Name:name, Direction:direction, Priority:priority, Port:destinationPortRange}" -o table 45 46 az network lb create -g kubernetes --sku Standard \ 47 -n kubernetes-lb \ 48 --backend-pool-name kubernetes-lb-pool \ 49 --public-ip-address kubernetes-pip \ 50 --public-ip-address-allocation static 51 52 az network public-ip list --query="[?name=='kubernetes-pip'].{ResourceGroup:resourceGroup, Region:location,Allocation:publicIpAllocationMethod,IP:ipAddress}" -o table 53 #For Ubuntu 54 # az vm image list --location westeurope --publisher Canonical --offer UbuntuServer --sku 18.04-LTS --all -o table 55 # For Redhat 56 # az vm image list --location westeurope --publisher RedHat --offer RHEL --sku 8 --all -o table 57 # => choosen one : 8-lvm-gen2 58 WHICHOS="RedHat:RHEL:8-lvm-gen2:8.5.2022032206" 59 60 # K8s Controller 61 az vm availability-set create -g kubernetes -n controller-as 62 63 for i in 0 1 2; do 64 echo "[Controller ${i}] Creating public IP..." 65 az network public-ip create -n controller-${i}-pip -g kubernetes --sku Standard > /dev/null 66 echo "[Controller ${i}] Creating NIC..." 67 az network nic create -g kubernetes \ 68 -n controller-${i}-nic \ 69 --private-ip-address 10.240.0.1${i} \ 70 --public-ip-address controller-${i}-pip \ 71 --vnet kubernetes-vnet \ 72 --subnet kubernetes-subnet \ 73 --ip-forwarding \ 74 --lb-name kubernetes-lb \ 75 --lb-address-pools kubernetes-lb-pool >/dev/null 76 77 echo "[Controller ${i}] Creating VM..." 78 az vm create -g kubernetes \ 79 -n controller-${i} \ 80 --image ${WHICHOS} \ 81 --nics controller-${i}-nic \ 82 --availability-set controller-as \ 83 --nsg '' \ 84 --admin-username 'kuberoot' \ 85 --admin-password 'Changeme!' \ 86 --size Standard_B2s \ 87 --storage-sku StandardSSD_LRS 88 #--generate-ssh-keys > /dev/null 89 done 90 91 #K8s Worker 92 az vm availability-set create -g kubernetes -n worker-as 93 for i in 0 1; do 94 echo "[Worker ${i}] Creating public IP..." 95 az network public-ip create -n worker-${i}-pip -g kubernetes --sku Standard > /dev/null 96 echo "[Worker ${i}] Creating NIC..." 97 az network nic create -g kubernetes \ 98 -n worker-${i}-nic \ 99 --private-ip-address 10.240.0.2${i} \ 100 --public-ip-address worker-${i}-pip \ 101 --vnet kubernetes-vnet \ 102 --subnet kubernetes-subnet \ 103 --ip-forwarding > /dev/null 104 echo "[Worker ${i}] Creating VM..." 105 az vm create -g kubernetes \ 106 -n worker-${i} \ 107 --image ${WHICHOS} \ 108 --nics worker-${i}-nic \ 109 --tags pod-cidr=10.200.${i}.0/24 \ 110 --availability-set worker-as \ 111 --nsg '' \ 112 --generate-ssh-keys \ 113 --size Standard_B2s \ 114 --storage-sku StandardSSD_LRS \ 115 --admin-username 'kuberoot'> /dev/null \ 116 --admin-password 'Changeme!' \ 117 done 118 119 #Summarize 120 az vm list -d -g kubernetes -o table
OKD vs OpenShift They are essentially the same Kubernetes distribution.
OKD — short for “The Community Distribution of Kubernetes that powers Red Hat OpenShift” — is the free, upstream/community edition: the same codebase, community support, and images pulled from quay.io.
OpenShift is Red Hat’s enterprise product: OKD plus commercial support, certifications, a longer support lifecycle, Red Hat registries (registry.redhat.io) and OperatorHub access.
Day-to-day they are nearly interchangeable — both use the same oc, openshift-install and oc-mirror tooling, and the same install-config.yaml layout. Unless stated otherwise, the notes below apply to both.
A list of nice findings for Kubernetes List all images in Helm chart 1images=$(helm template $helm |yq -N '..|.image? | select(. == "*" and . != null)'|sort|uniq|grep ":"|egrep -v '*:[[:blank:]]' || echo "") upload images listed in an Helm chart 1load_helm_images(){ 2 # look in helm charts 3 for helm in $(ls ../../roles/*/files/helm/*.tgz); do 4 printf "\e[1;34m[INFO]\e[m Look for images in ${helm}...\n" 5 6 images=$(helm template $helm |yq -N '..|.image? | select(. == "*" and . != null)'|sort|uniq|grep ":"|egrep -v '*:[[:blank:]]' || echo "") 7 8 dir=$( dirname $helm | xargs dirname ) 9 10 echo "####" 11 12 if [ "$images" != "" ]; then 13 printf "\e[1;34m[INFO]\e[m Images found in the helm charts: ${images}\n" 14 printf "\e[1;34m[INFO]\e[m Create ${dir}/images images...\n" 15 16 mkdir -p ${dir}/images 17 18 while i= read -r image_name; do 19 archive_name=$(basename -a $(awk -F : '{print $1}'<<<${image_name})); 20 printf "\e[1;34m[INFO]\e[m Pull images...\n" 21 podman pull ${image_name}; 22 printf "\e[1;34m[INFO]\e[m Push ${image_name} in ${dir}/images/${archive_name}\n" 23 podman save ${image_name} --format oci-archive -o ${dir}/images/${archive_name}; 24 done <<< ${images} 25 else 26 printf "\e[1;34m[INFO]\e[m No Images found in the helm charts: $helm\n" 27 fi 28 done 29} Check components version 1function checkComponentsInstall() { 2 componentsArray=("kubectl" "helm") 3 for i in "${componentsArray[@]}"; do 4 command -v "${i}" >/dev/null 2>&1 || 5 { echo "[ERROR] ${i} is required, but it's not installed. Aborting." >&2; exit 1; } 6 done 7} Version comparator 1function checkK8sVersion() { 2 currentK8sVersion=$(kubectl version --short | grep "Server Version" | awk '{gsub(/v/,$5)}1 {print $3}') 3 testVersionComparator 1.20 "$currentK8sVersion" '<' 4 if [[ $k8sVersion == "ok" ]]; then 5 echo "current kubernetes version is ok" 6 else 7 minikube start --kubernetes-version=v1.22.4; 8 fi 9} 10 11 12# the comparator based on https://stackoverflow.com/a/4025065 13versionComparator () { 14 if [[ $1 == $2 ]] 15 then 16 return 0 17 fi 18 local IFS=. 19 local i ver1=($1) ver2=($2) 20 # fill empty fields in ver1 with zeros 21 for ((i=${#ver1[@]}; i<${#ver2[@]}; i++)) 22 do 23 ver1[i]=0 24 done 25 for ((i=0; i<${#ver1[@]}; i++)) 26 do 27 if [[ -z ${ver2[i]} ]] 28 then 29 # fill empty fields in ver2 with zeros 30 ver2[i]=0 31 fi 32 if ((10#${ver1[i]} > 10#${ver2[i]})) 33 then 34 return 1 35 fi 36 if ((10#${ver1[i]} < 10#${ver2[i]})) 37 then 38 return 2 39 fi 40 done 41 return 0 42} 43 44testVersionComparator () { 45 versionComparator $1 $2 46 case $? in 47 0) op='=';; 48 1) op='>';; 49 2) op='<';; 50 esac 51 if [[ $op != "$3" ]] 52 then 53 echo "Kubernetes test fail: Expected '$3', Actual '$op', Arg1 '$1', Arg2 '$2'" 54 k8sVersion="not ok" 55 else 56 echo "Kubernetes test pass: '$1 $op $2'" 57 k8sVersion="ok" 58 fi 59}
The Basics awk is treat each line as a table, by default space are separators of columns.
General syntax is awk 'search {action}' file_to_parse.
1# Give the value higher than 75000 in column $4 2df | awk '$4 > 75000' 3 4# Print the all line when column $4 is higher than 75000 5df | awk '$4 > 75000 {print $0}' But if you look for a string, the search need to be included in /search/ or ;search;. When you print $0 represent the all line, $1 first column, $2 second column etc.
Description Buildah: is used to build Open Container Initiative (OCI) format or Docker format container images without the need for a daemon.
Podman: provides the ability to directly run container images without a daemon. Podman can pull container images from a container registry, if they are not available locally.
Skopeo: offers features for pulling and pushing containers to registries. Moving containers between registries is supported. Container image inspection is also offered and some introspective capabilities can be performed, without first downloading the container itself.
