Devops sections in docs
Documentation regarding devops technologies as IaC, Config-Manager, and other devops tools.
1ansible-galaxy collection list
1# From Ansible Galaxy official repo
2ansible-galaxy collection install community.general
3
4# From a tarball locally
5ansible-galaxy collection install ./community-general-6.0.0.tar.gz
6
7# From custom Repo
8ansible-galaxy collection install git+https://git.example.com/projects/namespace.collectionName.git
9ansible-galaxy collection install git+https://git.example.com/projects/namespace.collectionName,v1.0.2
10ansible-galaxy collection install git+https://git.example.com/namespace/collectionName.git
11
12# From a requirement.yml file
13ansible-galaxy collection install -r ./requirement.yaml
1collections:
2- name: kubernetes.core
3
4- source: https://gitlab.example.com/super-group/collector.git
5 type: git
6 version: "v1.0.6"
7
8- source: https://gitlab.ipolicedev.int/another-projects/plates.git
9 type: git
1ansible-inventory --list | jq -r 'map_values(select(.hosts != null and (.hosts | contains(["myhost"])))) | keys[]'
1kafka_host: "[{{ groups['KAFKA'] | map('extract', hostvars, 'inventory_hostname') | map('regex_replace', '^', '\"') | map('regex_replace', '\\\"', '\"') | map('regex_replace', '$', ':'+ kafka_port +'\"') | join(', ') }}]"
2
3elasticsearch_host: "{{ groups['ELASTICSEARCH'] | map('extract', hostvars, 'inventory_hostname') | map('regex_replace', '^', '\"') | map('regex_replace', '\\\"', '\"') | map('regex_replace', '$', ':'+ elasticsearch_port +'\"') | join(', ') }}"
1ansible-pull -U https://github.com/MozeBaltyk/Okub.git ./playbooks/tasks/provision.yml
1#cloud-config
2timezone: ${timezone}
3
4packages:
5 - qemu-guest-agent
6 - git
7
8package_update: true
9package_upgrade: true
10
11
12## Test 1
13ansible:
14 install_method: pip
15 package_name: ansible-core
16 run_user: ansible
17 galaxy:
18 actions:
19 - ["ansible-galaxy", "collection", "install", "community.general"]
20 - ["ansible-galaxy", "collection", "install", "ansible.posix"]
21 - ["ansible-galaxy", "collection", "install", "ansible.utils"]
22 pull:
23 playbook_name: ./playbooks/tasks/provision.yml
24 url: "https://github.com/MozeBaltyk/Okub.git"
25
26## Test 2
27ansible:
28 install_method: pip
29 package_name: ansible
30 #run_user only with install_method: pip
31 run_user: ansible
32 setup_controller:
33 repositories:
34 - path: /home/ansible/Okub
35 source: https://github.com/MozeBaltyk/Okub.git
36 run_ansible:
37 - playbook_dir: /home/ansible/Okub
38 playbook_name: ./playbooks/tasks/provision.yml
39########
1systemctl --failed
2systemctl list-jobs --after
3journalctl -e
Checks user-data and config:
Buildah: is used to build Open Container Initiative (OCI) format or Docker format container images without the need for a daemon.
Podman: provides the ability to directly run container images without a daemon. Podman can pull container images from a container registry, if they are not available locally.
Skopeo: offers features for pulling and pushing containers to registries. Moving containers between registries is supported. Container image inspection is also offered and some introspective capabilities can be performed, without first downloading the container itself.
1# see images available on your hosts
2docker image list
3
4# equal to above
5docker images
6REPOSITORY TAG IMAGE ID CREATED SIZE
7httpd latest 6fa26f20557b 45 hours ago 164MB
8hello-world latest 75280d40a50b 4 months ago 1.69kB
9
10# give sha
11docker images --no-trunc=true
12
13# delete unused images
14docker rmi $(docker images -q)
15# delete images without tags
16docker rmi $(docker images | grep "^<none>" | awk '{print $3}')
1dirs -c
2for DIR in $(find ./examples -type d); do
3 pushd $DIR
4 terraform init
5 terraform fmt -check
6 terraform validate
7 popd
8 done
1export DO_PAT="dop_v1_xxxxxxxxxxxxxxxx"
2doctl auth init --context rkub
3
4# inside a dir with a tf file
5terraform init
6terraform validate
7terraform plan -var "do_token=${DO_PAT}"
8terraform apply -var "do_token=${DO_PAT}" -auto-approve
9
10# clean apply
11terraform plan -out=infra.tfplan -var "do_token=${DO_PAT}"
12terraform apply infra.tfplan
13
14# Control
15terraform show terraform.tfstate
16
17# Destroy
18terraform plan -destroy -out=terraform.tfplan -var "do_token=${DO_PAT}"
19terraform apply terraform.tfplan
1ssh root@$(terraform output -json ip_address_workers | jq -r '.[0]') -i .key
Two possibilities:
See also documentation about Podman and Docker
1# list index catalog
2curl https://registry.k3s.example.com/v2/_catalog | jq
3
4# List tags available regarding an image
5curl https://registry.k3s.example.com/v2/myhaproxy/tags/list
6
7# list index catalog - with user/password
8curl https://registry-admin:<PWD>@registry.k3s.example.com/v2/_catalog | jq
9
10# list index catalog - when you need to specify the CA
11curl -u user:password https://<url>:<port>/v2/_catalog --cacert ca.crt | jq
12
13# list index catalog - for OCP
14curl -u user:password https://<url>:<port>/v2/ocp4/openshift4/tags/list | jq
15
16# Login to registry with podman
17podman login -u registry-admin -p <PWD> registry.k3s.example.com
18
19# Push images in the registry
20skopeo copy "--dest-creds=registry-admin:<PWD>" docker://docker.io/goharbor/harbor-core:v2.6.1 docker://registry.k3s.example.com/goharbor/harbor-core:v2.6.1
1ip a
2sudo vi /etc/docker/daemon.json
1{
2"insecure-registries": ["192.168.1.11:5000"]
3}
1sudo systemctl restart docker
2docker info
Check docker config
Load the image
1podman pull sonatype/nexus3:3.59.0
2podman save sonatype/nexus3:3.59.0 -o nexus3.tar
3podman load < nexus3.tar
Create a service inside /etc/systemd/system/container-nexus3.service with content below:
1[Unit]
2Description=Nexus Podman container
3Wants=syslog.service
4
5[Service]
6User=nexus-system
7Group=nexus-system
8Restart=always
9ExecStart=/usr/bin/podman run \
10 --log-level=debug \
11 --rm \
12 -ti \
13 --publish 8081:8081 \
14 --name nexus \
15 sonatype/nexus3:3.59.0
16
17ExecStop=/usr/bin/podman stop -t 10 nexus
18
19[Install]
20WantedBy=multi-user.target
Nothing original, it just the documentation of redhat, but can be usefull to kickstart a registry.
Prerequisites:
1# packages
2sudo yum install -y podman
3sudo yum install -y rsync
4sudo yum install -y jq
5
6# Get tar
7mirror="https://mirror.openshift.com/pub/openshift-v4/clients"
8wget ${mirror}/mirror-registry/latest/mirror-registry.tar.gz
9tar zxvf mirror-registry.tar.gz
10
11# Get oc-mirror
12curl https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/latest/oc-mirror.rhel9.tar.gz -O
13
14# Basic install
15sudo ./mirror-registry install \
16 --quayHostname quay01.example.local \
17 --quayRoot /opt
18
19# More detailed install
20sudo ./mirror-registry install \
21 --quayHostname quay01.example.local \
22 --quayRoot /srv \
23 --quayStorage /srv/quay-pg \
24 --pgStorage /srv/quay-storage \
25 --sslCert tls.crt \
26 --sslKey tls.key
27
28podman login -u init \
29 -p 7u2Dm68a1s3bQvz9twrh4Nel0i5EMXUB \
30 quay01.example.local:8443 \
31 --tls-verify=false
32
33# By default login go in:
34cat $XDG_RUNTIME_DIR/containers/auth.json
35
36# Get IP
37sudo podman inspect --format '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' quay-app
38
39#unistall
40sudo ./mirror-registry uninstall -v \
41 --quayRoot <example_directory_name>
42
43# Info
44curl -u init:password https://quay01.example.local:8443/v2/_catalog | jq
45curl -u root:password https://<url>:<port>/v2/ocp4/openshift4/tags/list | jq
46
47# Get an example of imageset
48oc-mirror init --registry quay.example.com:8443/mirror/oc-mirror-metadata
49
50# Get list of Operators, channels, packages
51oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.14
52oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.14 --package=kubevirt-hyperconverged
53oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.14 --package=kubevirt-hyperconverged --channel=stable
1QUAY_POSTGRES=`podman ps | grep quay-postgres | awk '{print $1}'`
2
3podman exec -it $QUAY_POSTGRES psql -d quay -c "UPDATE "public.user" SET invalid_login_attempts = 0 WHERE username = 'init'"
- Firewalld activated, important otherwise the routing to the app is not working
- Podman, jq installed
1podman pull docker.io/gitea/gitea:1-rootless
2podman save docker.io/gitea/gitea:1-rootless -o gitea-rootless.tar
3podman load < gitea-rootless.tar
cat /etc/systemd/system/container-gitea-app.service
1# container-gitea-app.service
2[Unit]
3Description=Podman container-gitea-app.service
4
5Wants=network.target
6After=network-online.target
7RequiresMountsFor=/var/lib/containers/storage /var/run/containers/storage
8
9[Service]
10Environment=PODMAN_SYSTEMD_UNIT=%n
11Restart=on-failure
12TimeoutStopSec=70
13PIDFile=%t/container-gitea-app.pid
14Type=forking
15
16ExecStartPre=/bin/rm -f %t/container-gitea-app.pid %t/container-gitea-app.ctr-id
17ExecStart=/usr/bin/podman container run \
18 --conmon-pidfile %t/container-gitea-app.pid \
19 --cidfile %t/container-gitea-app.ctr-id \
20 --cgroups=no-conmon \
21 --replace \
22 --detach \
23 --tty \
24 --env DB_TYPE=sqlite3 \
25 --env DB_HOST=gitea-db:3306 \
26 --env DB_NAME=gitea \
27 --env DB_USER=gitea \
28 --env DB_PASSWD=9Oq6P9Tsm6j8J7c18Jxc \
29 --volume gitea-data-volume:/var/lib/gitea:Z \
30 --volume gitea-config-volume:/etc/gitea:Z \
31 --network gitea-net \
32 --publish 2222:2222 \
33 --publish 3000:3000 \
34 --label "io.containers.autoupdate=registry" \
35 --name gitea-app \
36 docker.io/gitea/gitea:1-rootless
37
38ExecStop=/usr/bin/podman container stop \
39 --ignore \
40 --cidfile %t/container-gitea-app.ctr-id \
41 -t 10
42
43ExecStopPost=/usr/bin/podman container rm \
44 --ignore \
45 -f \
46 --cidfile %t/container-gitea-app.ctr-id
47
48[Install]
49WantedBy=multi-user.target default.target
Configuration inside /var/lib/containers/storage/volumes/gitea-config-volume/_data/app.ini
1export RKE_VERSION=$(curl -s https://update.rke2.io/v1-release/channels | jq -r '.data[] | select(.id=="stable") | .latest' | awk -F"+" '{print $1}'| sed 's/v//')
2export CERT_VERSION=$(curl -s https://api.github.com/repos/cert-manager/cert-manager/releases/latest | jq -r .tag_name)
3export RANCHER_VERSION=$(curl -s https://api.github.com/repos/rancher/rancher/releases/latest | jq -r .tag_name)
4export LONGHORN_VERSION=$(curl -s https://api.github.com/repos/longhorn/longhorn/releases/latest | jq -r .tag_name)
5export NEU_VERSION=$(curl -s https://api.github.com/repos/neuvector/neuvector-helm/releases/latest | jq -r .tag_name)
1# ubuntu
2type -p curl >/dev/null || (sudo apt update && sudo apt install curl -y)
3curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg \
4&& sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg \
5&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
6&& sudo apt update \
7&& sudo apt install gh -y
8
9# Redhat
10sudo dnf install 'dnf-command(config-manager)'
11sudo dnf config-manager --add-repo https://cli.github.com/packages/rpm/gh-cli.repo
12sudo dnf install gh
1gh completion zsh > $ZSH/completions/_gh
1gh auth login -p ssh -h GitHub.com -s read:project,delete:repo,repo,workflow -w
2
3gh auth status
4github.com
5 โ Logged in to github.com as MorzeBaltyk ($HOME/.config/gh/hosts.yml)
6 โ Git operations for github.com configured to use ssh protocol.
7 โ Token: gho_************************************
8 โ Token scopes: delete_repo, gist, read:org, read:project, repo
One way:
https://glab.readthedocs.io/en/latest/intro.html
1# add token
2glab auth login --hostname mygitlab.example.com
3# view fork of dep installer
4glab repo view mygitlab.example.com/copain/project
5# clone fork of dep installer
6glab repo clone mygitlab.example.com/copain/project
1Optimization
2puma['worker_processes'] = 16
3puma['worker_timeout'] = 60
4puma['min_threads'] = 1
5puma['max_threads'] = 4
6puma['per_worker_max_memory_mb'] = 2048
Generate CSR in /data/gitlab/csr/server_cert.cnf
1[req]
2default_bits = 2048
3distinguished_name = req_distinguished_name
4req_extensions = req_ext
5prompt = no
6
7[req_distinguished_name]
8C = PL
9ST = Poland
10L = Warsaw
11O = myOrg
12OU = DEV
13CN = gitlab.example.com
14
15[req_ext]
16subjectAltName = @alt_names
17
18[alt_names]
19DNS = gitlab.example.com
20IP = 192.168.01.01
1# Create CSR
2openssl req -new -newkey rsa:2048 -nodes -keyout gitlab.example.com.key -config /data/gitlab/csr/server_cert.cnf -out gitlab.example.com.csr
3
4openssl req -noout -text -in gitlab.example.com.csr
5
6# Sign your CSR with your PKI. If you PKI is a windows one, you should get back a .CER file.
7
8# check info:
9openssl x509 -text -in gitlab.example.com.cer -noout
1### push it in crt/key in Gitlab
2cp /tmp/gitlab.example.com.cer cert/gitlab.example.com.crt
3cp /tmp/gitlab.example.com.key cert/gitlab.example.com.key
4cp /tmp/gitlab.example.com.cer cert/192.168.01.01.crt
5cp /tmp/gitlab.example.com.key cert/192.168.01.01.key
6
7### push rootCA in gitlab
8cp /etc/pki/ca-trust/source/anchors/domain-issuing.crt /data/gitlab/config/trusted-certs/domain-issuing.crt
9cp /etc/pki/ca-trust/source/anchors/domain-rootca.crt /data/gitlab/config/trusted-certs/domain-rootca.crt
10
11### Reconfigure
12vi /data/gitlab/config/gitlab.rb
13docker exec gitlab bash -c 'update-ca-certificates'
14docker exec gitlab bash -c 'gitlab-ctl reconfigure'
15
16### Stop / Start
17docker stop gitlab
18docker rm gitlab
19docker run -d -p 5050:5050 -p 2289:22 -p 443:443 --restart=always \
20-v /data/gitlab/config:/etc/gitlab \
21-v /data/gitlab/logs:/var/log/gitlab \
22-v /data/gitlab/data:/var/opt/gitlab \
23-v /data/gitlab/cert:/etc/gitlab/ssl \
24-v /data/gitlab/config/trusted-certs:/usr/local/share/ca-certificates \
25--name gitlab gitlab/gitlab-ce:15.0.5-ce.0
1docker exec gitlab bash -c 'gitlab-ctl status'
2docker exec -it gitlab gitlab-rake gitlab:check SANITIZE=true
3docker exec -it gitlab gitlab-rake gitlab:env:info
1docker exec -it gitlab gitlab-rake gitlab:backup:create --trace
2
3#Alternate way to do it
4docker exec gitlab bash -c 'gitlab-backup create'
5docker exec gitlab bash -c 'gitlab-backup create SKIP=repositories'
6docker exec gitlab bash -c 'gitlab-backup create SKIP=registry'
1Restore
2gitlab-ctl reconfigure
3gitlab-ctl start
4gitlab-ctl stop unicorn
5gitlab-ctl stop sidekiq
6gitlab-ctl status
7ls -lart /var/opt/gitlab/backups
8
9docker exec -it gitlab gitlab-rake gitlab:backup:restore --trace
10docker exec -it gitlab gitlab-rake gitlab:backup:restore BACKUP=1537738690_2018_09_23_10.8.3 --trace
11
12Restart
13docker exec gitlab bash -c 'gitlab-ctl restart'
sudo docker exec -it gitlab gitlab-rake gitlab:check sudo docker exec -it gitlab gitlab-rake gitlab:doctor:secrets
Gita is opensource project in python to handle a bit number of projects available: Here
1# Install
2pip3 install -U gita
3
4# add repo in gita
5gita add dcc/ssg/toolset
6gita add -r dcc/ssg # recursively add
7gita add -a dcc # resursively add and auto-group based on folder structure
8
9# create a group
10gita group add docs -n ccn
11
12# Checks
13gita ls
14gita ll -g
15gita group ls
16gita group ll
17gita st dcc
18
19# Use
20gita pull ccn
21gita push ccn
22
23gita freeze
GIT is a distributed version control system that was created by Linus Torvalds, the mastermind of Linux itself. It was designed to be a superior version control system to those that were readily available, the two most common of these being CVS and Subversion (SVN). Whereas CVS and SVN use the Client/Server model for their systems, GIT operates a little differently. Instead of downloading a project, making changes, and uploading it back to the server, GIT makes the local machine act as a server. Tecmint