Devops

Specific to RHEL 1# Create a trust zone for the two interconnect 2sudo firewall-cmd --permanent --zone=trusted --add-source=10.42.0.0/16 #pods 3sudo firewall-cmd --permanent --zone=trusted --add-source=10.43.0.0/16 #services 4sudo firewall-cmd --reload 5sudo firewall-cmd --list-all-zones 6 7# on Master 8sudo rm -f /var/lib/cni/networks/cbr0/lock 9sudo /usr/local/bin/k3s-killall.sh 10sudo systemctl restart k3s 11sudo systemctl status k3s 12 13# on Worker 14sudo rm -f /var/lib/cni/networks/cbr0/lock 15sudo /usr/local/bin/k3s-killall.sh 16sudo systemctl restart k3s-agent 17sudo systemctl status k3s-agent Check Certificates 1# Get CA from K3s master 2openssl s_client -connect localhost:6443 -showcerts < /dev/null 2>&1 | openssl x509 -noout -enddate 3openssl s_client -showcerts -connect 193.168.51.103:6443 < /dev/null 2>/dev/null|openssl x509 -outform PEM 4openssl s_client -showcerts -connect 193.168.51.103:6443 < /dev/null 2>/dev/null|openssl x509 -outform PEM | base64 | tr -d '\n' 5 6# Check end date: 7for i in `ls /var/lib/rancher/k3s/server/tls/*.crt`; do echo $i; openssl x509 -enddate -noout -in $i; done 8 9# More efficient: 10cd /var/lib/rancher/k3s/server/tls/ 11for crt in *.crt; do printf '%s: %s\n' "$(date --date="$(openssl x509 -enddate -noout -in "$crt"|cut -d= -f 2)" --iso-8601)" "$crt"; done | sort 12 13# Check CA issuer 14for i in $(find . -maxdepth 1 -type f -name "*.crt"); do openssl x509 -in ${i} -noout -issuer; done General Checks RKE2/K3S Nice gist to troubleshoot etcd link

Operators have 3 kinds : go, ansible, helm. 1## Init an Ansible project 2operator-sdk init --plugins=ansible --domain example.org --owner "Your name" 3 4## Command above will create a structure like: 5netbox-operator 6├── Dockerfile 7├── Makefile 8├── PROJECT 9├── config 10│ ├── crd 11│ ├── default 12│ ├── manager 13│ ├── manifests 14│ ├── prometheus 15│ ├── rbac 16│ ├── samples 17│ ├── scorecard 18│ └── testing 19├── molecule 20│ ├── default 21│ └── kind 22├── playbooks 23│ └── install.yml 24├── requirements.yml 25├── roles 26│ └── deployment 27└── watches.yaml 1## Create first role 2operator-sdk create api --group app --version v1alpha1 --kind Deployment --generate-role

Deploy a Quay.io / Mirror-registry on container Nothing original, it just the documentation of redhat, but can be usefull to kickstart a registry. Prerequisites: 10G /home 15G /var 300G /srv or /opt (regarding QuayRoot) min 2 or more vCPUs. min 8 GB of RAM. 1# packages 2sudo yum install -y podman 3sudo yum install -y rsync 4sudo yum install -y jq 5 6# Get tar 7mirror="https://mirror.openshift.com/pub/openshift-v4/clients" 8wget ${mirror}/mirror-registry/latest/mirror-registry.tar.gz 9tar zxvf mirror-registry.tar.gz 10 11# Get oc-mirror 12curl https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/latest/oc-mirror.rhel9.tar.gz -O 13 14# Basic install 15sudo ./mirror-registry install \ 16 --quayHostname quay01.example.local \ 17 --quayRoot /opt 18 19# More detailed install 20sudo ./mirror-registry install \ 21 --quayHostname quay01.example.local \ 22 --quayRoot /srv \ 23 --quayStorage /srv/quay-pg \ 24 --pgStorage /srv/quay-storage \ 25 --sslCert tls.crt \ 26 --sslKey tls.key 27 28podman login -u init \ 29 -p 7u2Dm68a1s3bQvz9twrh4Nel0i5EMXUB \ 30 quay01.example.local:8443 \ 31 --tls-verify=false 32 33# By default login go in: 34cat $XDG_RUNTIME_DIR/containers/auth.json 35 36# Get IP 37sudo podman inspect --format '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' quay-app 38 39#unistall 40sudo ./mirror-registry uninstall -v \ 41 --quayRoot <example_directory_name> 42 43# Info 44curl -u init:password https://quay01.example.local:8443/v2/_catalog | jq 45curl -u root:password https://<url>:<port>/v2/ocp4/openshift4/tags/list | jq 46 47# Get an example of imageset 48oc-mirror init --registry quay.example.com:8443/mirror/oc-mirror-metadata 49 50# Get list of Operators, channels, packages 51oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.14 52oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.14 --package=kubevirt-hyperconverged 53oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.14 --package=kubevirt-hyperconverged --channel=stable unlock user init/admin 1QUAY_POSTGRES=`podman ps | grep quay-postgres | awk '{print $1}'` 2 3podman exec -it $QUAY_POSTGRES psql -d quay -c "UPDATE "public.user" SET invalid_login_attempts = 0 WHERE username = 'init'" Source Mirror-registry

Presentation Gita is opensource project in python to handle a bit number of projects available: Here 1# Install 2pip3 install -U gita 3 4# add repo in gita 5gita add dcc/ssg/toolset 6gita add -r dcc/ssg # recursively add 7gita add -a dcc # resursively add and auto-group based on folder structure 8 9# create a group 10gita group add docs -n ccn 11 12# Checks 13gita ls 14gita ll -g 15gita group ls 16gita group ll 17gita st dcc 18 19# Use 20gita pull ccn 21gita push ccn 22 23gita freeze

List 1ansible-galaxy collection list Install an Ansible Collection 1# From Ansible Galaxy official repo 2ansible-galaxy collection install community.general 3 4# From a tarball locally 5ansible-galaxy collection install ./community-general-6.0.0.tar.gz 6 7# From custom Repo 8ansible-galaxy collection install git+https://git.example.com/projects/namespace.collectionName.git 9ansible-galaxy collection install git+https://git.example.com/projects/namespace.collectionName,v1.0.2 10ansible-galaxy collection install git+https://git.example.com/namespace/collectionName.git 11 12# From a requirement.yml file 13ansible-galaxy collection install -r ./requirement.yaml Requirement file to install Ansible Collection 1collections: 2- name: kubernetes.core 3 4- source: https://gitlab.example.com/super-group/collector.git 5 type: git 6 version: "v1.0.6" 7 8- source: https://gitlab.ipolicedev.int/another-projects/plates.git 9 type: git

GIT is a distributed version control system that was created by Linus Torvalds, the mastermind of Linux itself. It was designed to be a superior version control system to those that were readily available, the two most common of these being CVS and Subversion (SVN). Whereas CVS and SVN use the Client/Server model for their systems, GIT operates a little differently. Instead of downloading a project, making changes, and uploading it back to the server, GIT makes the local machine act as a server. Tecmint

Prerequis - Firewalld activated, important otherwise the routing to the app is not working - Podman, jq installed Import image 1podman pull docker.io/gitea/gitea:1-rootless 2podman save docker.io/gitea/gitea:1-rootless -o gitea-rootless.tar 3podman load < gitea-rootless.tar Install cat /etc/systemd/system/container-gitea-app.service 1# container-gitea-app.service 2[Unit] 3Description=Podman container-gitea-app.service 4 5Wants=network.target 6After=network-online.target 7RequiresMountsFor=/var/lib/containers/storage /var/run/containers/storage 8 9[Service] 10Environment=PODMAN_SYSTEMD_UNIT=%n 11Restart=on-failure 12TimeoutStopSec=70 13PIDFile=%t/container-gitea-app.pid 14Type=forking 15 16ExecStartPre=/bin/rm -f %t/container-gitea-app.pid %t/container-gitea-app.ctr-id 17ExecStart=/usr/bin/podman container run \ 18 --conmon-pidfile %t/container-gitea-app.pid \ 19 --cidfile %t/container-gitea-app.ctr-id \ 20 --cgroups=no-conmon \ 21 --replace \ 22 --detach \ 23 --tty \ 24 --env DB_TYPE=sqlite3 \ 25 --env DB_HOST=gitea-db:3306 \ 26 --env DB_NAME=gitea \ 27 --env DB_USER=gitea \ 28 --env DB_PASSWD=9Oq6P9Tsm6j8J7c18Jxc \ 29 --volume gitea-data-volume:/var/lib/gitea:Z \ 30 --volume gitea-config-volume:/etc/gitea:Z \ 31 --network gitea-net \ 32 --publish 2222:2222 \ 33 --publish 3000:3000 \ 34 --label "io.containers.autoupdate=registry" \ 35 --name gitea-app \ 36 docker.io/gitea/gitea:1-rootless 37 38ExecStop=/usr/bin/podman container stop \ 39 --ignore \ 40 --cidfile %t/container-gitea-app.ctr-id \ 41 -t 10 42 43ExecStopPost=/usr/bin/podman container rm \ 44 --ignore \ 45 -f \ 46 --cidfile %t/container-gitea-app.ctr-id 47 48[Install] 49WantedBy=multi-user.target default.target Configuration inside /var/lib/containers/storage/volumes/gitea-config-volume/_data/app.ini

Get tag_name from latest 1export RKE_VERSION=$(curl -s https://update.rke2.io/v1-release/channels | jq -r '.data[] | select(.id=="stable") | .latest' | awk -F"+" '{print $1}'| sed 's/v//') 2export CERT_VERSION=$(curl -s https://api.github.com/repos/cert-manager/cert-manager/releases/latest | jq -r .tag_name) 3export RANCHER_VERSION=$(curl -s https://api.github.com/repos/rancher/rancher/releases/latest | jq -r .tag_name) 4export LONGHORN_VERSION=$(curl -s https://api.github.com/repos/longhorn/longhorn/releases/latest | jq -r .tag_name) 5export NEU_VERSION=$(curl -s https://api.github.com/repos/neuvector/neuvector-helm/releases/latest | jq -r .tag_name) Install gh 1# ubuntu 2type -p curl >/dev/null || (sudo apt update && sudo apt install curl -y) 3curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg \ 4&& sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg \ 5&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null \ 6&& sudo apt update \ 7&& sudo apt install gh -y 8 9# Redhat 10sudo dnf install 'dnf-command(config-manager)' 11sudo dnf config-manager --add-repo https://cli.github.com/packages/rpm/gh-cli.repo 12sudo dnf install gh Autocompletions 1gh completion zsh > $ZSH/completions/_gh Create an ssh key ed Login 1gh auth login -p ssh -h GitHub.com -s read:project,delete:repo,repo,workflow -w 2 3gh auth status 4github.com 5 ✓ Logged in to github.com as MorzeBaltyk ($HOME/.config/gh/hosts.yml) 6 ✓ Git operations for github.com configured to use ssh protocol. 7 ✓ Token: gho_************************************ 8 ✓ Token scopes: delete_repo, gist, read:org, read:project, repo To use your key One way:

Glab CLI https://glab.readthedocs.io/en/latest/intro.html 1# add token 2glab auth login --hostname mygitlab.example.com 3# view fork of dep installer 4glab repo view mygitlab.example.com/copain/project 5# clone fork of dep installer 6glab repo clone mygitlab.example.com/copain/project Install 1Optimization 2puma['worker_processes'] = 16 3puma['worker_timeout'] = 60 4puma['min_threads'] = 1 5puma['max_threads'] = 4 6puma['per_worker_max_memory_mb'] = 2048 Certificats Generate CSR in /data/gitlab/csr/server_cert.cnf 1[req] 2default_bits = 2048 3distinguished_name = req_distinguished_name 4req_extensions = req_ext 5prompt = no 6 7[req_distinguished_name] 8C = PL 9ST = Poland 10L = Warsaw 11O = myOrg 12OU = DEV 13CN = gitlab.example.com 14 15[req_ext] 16subjectAltName = @alt_names 17 18[alt_names] 19DNS = gitlab.example.com 20IP = 192.168.01.01 1# Create CSR 2openssl req -new -newkey rsa:2048 -nodes -keyout gitlab.example.com.key -config /data/gitlab/csr/server_cert.cnf -out gitlab.example.com.csr 3 4openssl req -noout -text -in gitlab.example.com.csr 5 6# Sign your CSR with your PKI. If you PKI is a windows one, you should get back a .CER file. 7 8# check info: 9openssl x509 -text -in gitlab.example.com.cer -noout 1### push it in crt/key in Gitlab 2cp /tmp/gitlab.example.com.cer cert/gitlab.example.com.crt 3cp /tmp/gitlab.example.com.key cert/gitlab.example.com.key 4cp /tmp/gitlab.example.com.cer cert/192.168.01.01.crt 5cp /tmp/gitlab.example.com.key cert/192.168.01.01.key 6 7### push rootCA in gitlab 8cp /etc/pki/ca-trust/source/anchors/domain-issuing.crt /data/gitlab/config/trusted-certs/domain-issuing.crt 9cp /etc/pki/ca-trust/source/anchors/domain-rootca.crt /data/gitlab/config/trusted-certs/domain-rootca.crt 10 11### Reconfigure 12vi /data/gitlab/config/gitlab.rb 13docker exec gitlab bash -c 'update-ca-certificates' 14docker exec gitlab bash -c 'gitlab-ctl reconfigure' 15 16### Stop / Start 17docker stop gitlab 18docker rm gitlab 19docker run -d -p 5050:5050 -p 2289:22 -p 443:443 --restart=always \ 20-v /data/gitlab/config:/etc/gitlab \ 21-v /data/gitlab/logs:/var/log/gitlab \ 22-v /data/gitlab/data:/var/opt/gitlab \ 23-v /data/gitlab/cert:/etc/gitlab/ssl \ 24-v /data/gitlab/config/trusted-certs:/usr/local/share/ca-certificates \ 25--name gitlab gitlab/gitlab-ce:15.0.5-ce.0 Health-Checks 1docker exec gitlab bash -c 'gitlab-ctl status' 2docker exec -it gitlab gitlab-rake gitlab:check SANITIZE=true 3docker exec -it gitlab gitlab-rake gitlab:env:info Backup 1docker exec -it gitlab gitlab-rake gitlab:backup:create --trace 2 3#Alternate way to do it 4docker exec gitlab bash -c 'gitlab-backup create' 5docker exec gitlab bash -c 'gitlab-backup create SKIP=repositories' 6docker exec gitlab bash -c 'gitlab-backup create SKIP=registry' Restore from a Backup 1Restore 2gitlab-ctl reconfigure 3gitlab-ctl start 4gitlab-ctl stop unicorn 5gitlab-ctl stop sidekiq 6gitlab-ctl status 7ls -lart /var/opt/gitlab/backups 8 9docker exec -it gitlab gitlab-rake gitlab:backup:restore --trace 10docker exec -it gitlab gitlab-rake gitlab:backup:restore BACKUP=1537738690_2018_09_23_10.8.3 --trace 11 12Restart 13docker exec gitlab bash -c 'gitlab-ctl restart' Update Pre-checks before update sudo docker exec -it gitlab gitlab-rake gitlab:check sudo docker exec -it gitlab gitlab-rake gitlab:doctor:secrets

1ansible-inventory --list | jq -r 'map_values(select(.hosts != null and (.hosts | contains(["myhost"])))) | keys[]' 1kafka_host: "[{{ groups['KAFKA'] | map('extract', hostvars, 'inventory_hostname') | map('regex_replace', '^', '\"') | map('regex_replace', '\\\"', '\"') | map('regex_replace', '$', ':'+ kafka_port +'\"') | join(', ') }}]" 2 3elasticsearch_host: "{{ groups['ELASTICSEARCH'] | map('extract', hostvars, 'inventory_hostname') | map('regex_replace', '^', '\"') | map('regex_replace', '\\\"', '\"') | map('regex_replace', '$', ':'+ elasticsearch_port +'\"') | join(', ') }}"

Test locally a playbook 1ansible-pull -U https://github.com/MozeBaltyk/Okub.git ./playbooks/tasks/provision.yml Inside a cloud-init 1#cloud-config 2timezone: ${timezone} 3 4packages: 5 - qemu-guest-agent 6 - git 7 8package_update: true 9package_upgrade: true 10 11 12## Test 1 13ansible: 14 install_method: pip 15 package_name: ansible-core 16 run_user: ansible 17 galaxy: 18 actions: 19 - ["ansible-galaxy", "collection", "install", "community.general"] 20 - ["ansible-galaxy", "collection", "install", "ansible.posix"] 21 - ["ansible-galaxy", "collection", "install", "ansible.utils"] 22 pull: 23 playbook_name: ./playbooks/tasks/provision.yml 24 url: "https://github.com/MozeBaltyk/Okub.git" 25 26## Test 2 27ansible: 28 install_method: pip 29 package_name: ansible 30 #run_user only with install_method: pip 31 run_user: ansible 32 setup_controller: 33 repositories: 34 - path: /home/ansible/Okub 35 source: https://github.com/MozeBaltyk/Okub.git 36 run_ansible: 37 - playbook_dir: /home/ansible/Okub 38 playbook_name: ./playbooks/tasks/provision.yml 39######## Troubleshooting 1systemctl --failed 2systemctl list-jobs --after 3journalctl -e Checks user-data and config: