SysAdmin

🔍️ Investigate
🔍️ Investigate
Ressources 1# in crontab or tmux session - take every hour a track of the memory usage 2for i in {1..24} ; do echo -n "===================== " ; date ; free -m ; top -b -n1 | head -n 15 ; sleep 3600; done >> /var/log/SYSADM/memory.log & Hardware Logs Health Checks
🚩 Compare
🚩 Compare
Compare staffs Compare two jar files: 1diff -W200 -y <(unzip -vqq file1.jar | awk '{ if ($1 > 0) {printf("%s\t%s\n", $1, $8)}}' | sort -k2) <(unzip -vqq file2.jar | awk '{ if ($1 > 0) {printf("%s\t%s\n", $1, $8)}}' | sort -k2)
🚩 Files
🚩 Files
Find a process blocking a file with fuser: 1fuser -m </dir or /files> # Find process blocking/using this directory or files. 2fuser -cu </dir or /files> # Same as above but add the user 3fuser -kcu </dir or /files> # Kill process 4fuser -v -k -HUP -i ./ # Send HUP signal to process 5 6# Output will send you <PID + letter>, here is the meaning: 7# c current directory. 8# e executable being run. 9# f open file. (omitted in default display mode). 10# F open file for writing. (omitted in default display mode). 11# r root directory. 12# m mmap'ed file or shared library. with lsof ( = list open file): 1lsof +D /var/log # Find all files blocked with the process and user. 2lsof -a +L1 <mountpoint> # Process blocking a FS. 3lsof -c ssh -c init # Find files open by thoses processes. 4lsof -p 1753 # Find files open by PID process. 5lsof -u root # Find files open by user. 6lsof -u ^user # Find files open by user except this one. 7kill -9 `lsof -t -u toto` # kill user's processes. (option -t output only PID). MacGyver method: 1#When you have no fuser or lsof: 2find /proc/*/fd -type f -links 0 -exec ls -lrt {} \;
🚩 Network Manager
🚩 Network Manager
Basic Troubleshooting Checks interfaces 1nmcli con show 2NAME UUID TYPE DEVICE 3ens192 4d0087a0-740a-4356-8d9e-f58b63fd180c ethernet ens192 4ens224 3dcb022b-62a2-4632-8b69-ab68e1901e3b ethernet ens224 5 6nmcli dev status 7DEVICE TYPE STATE CONNECTION 8ens192 ethernet connected ens192 9ens224 ethernet connected ens224 10ens256 ethernet connected ens256 11lo loopback unmanaged -- 12 13# Get interfaces details : 14nmcli connection show ens192 15nmcli -p con show ens192 16 17# Get DNS settings in interface 18UUID=$(nmcli --get-values connection.uuid c show "cloud-init eth0") 19nmcli --get-values ipv4.dns c show $UUID Changing Interface name 1nmcli connection add type ethernet mac "00:50:56:80:11:ff" ifname "ens224" 2nmcli connection add type ethernet mac "00:50:56:80:8a:0b" ifname "ens256" Create a custom config 1nmcli con load /etc/sysconfig/network-scripts/ifcfg-ens224 2nmcli con up ens192 Adding a Virtual IP 1nmcli con mod enp1s0 +ipv4.addresses "192.168.122.11/24" 2ip addr del 10.163.148.36/24 dev ens160 3 4nmcli con reload # before to reapply 5nmcli device reapply ens224 6systemctl status network.service 7systemctl restart network.service Add a DNS entry 1UUID=$(nmcli --get-values connection.uuid c show "cloud-init eth0") 2DNS_LIST=$(nmcli --get-values ipv4.dns c show $UUID) 3nmcli conn modify "$UUID" ipv4.dns "${DNS_LIST} ${DNS_IP}" 4 5# /etc/resolved is managed by systemd-resolved 6sudo systemctl restart systemd-resolved
🎶 Samba / CIFS
🎶 Samba / CIFS
Server Side First Install samba and samba-client (for debug + test) /etc/samba/smb.conf 1[home] 2Workgroup=WORKGROUP (le grp par defaul sur windows) 3Hosts allow = ... 4[shared] 5browseable = yes 6path = /shared 7valid users = user01, @un_group_au_choix 8writable = yes 9passdb backend = tdbsam #passwords are stored in the /var/lib/samba/private/passdb.tdb file. Test samba config testparm /usr/bin/testparm -s /etc/samba/smb.conf smbclient -L \192.168.56.102 -U test : list all samba shares available smbclient //192.168.56.102/sharedrepo -U test : connect to the share pdbedit -L : list user smb (better than smbclient)
🍻 SSHFS
🍻 SSHFS
SSHFS SshFS sert à monter sur son FS, un autre système de fichier distant, à travers une connexion SSH, le tout avec des droits utilisateur. L’avantage est de manipuler les données distantes avec n’importe quel gestionnaire de fichier (Nautilus, Konqueror, ROX, ou même la ligne de commande). - Pre-requis : droits d'administration, connexion ethernet, installation de FUSE et du paquet SSHFS. - Les utilisateurs de sshfs doivent faire partie du groupe fuse. Rq : FUSE permet à un utilisateur de monter lui-même un système de fichier. Normalement, pour monter un système de fichier, il faut être administrateur ou que celui-ci l’ait prévu dans « /etc/fstab » avec des informations en dur.
🐋 KVM
🐋 KVM
install KVM on RHEL 1# pre-checks hardware for intel CPU 2grep -e 'vmx' /proc/cpuinfo 3lscpu | grep Virtualization 4lsmod | grep kvm 5 6# on RHEL9 Workstation 7sudo dnf install virt-install virt-viewer -y 8sudo dnf install -y libvirt 9sudo dnf install virt-manager -y 10sudo dnf install -y virt-top libguestfs-tools guestfs-tools 11sudo gpasswd -a $USER libvirt 12 13# Helper 14sudo dnf -y install bridge-utils 15 16# Start libvirt 17sudo systemctl start libvirtd 18sudo systemctl enable libvirtd 19sudo systemctl status libvirtd Basic Checks 1virsh nodeinfo Config a Bridge network Important note that network are created with root user but VM with current user.
🐦 Awk
🐦 Awk
The Basics awk is treat each line as a table, by default space are separators of columns. General syntax is awk 'search {action}' file_to_parse. 1# Give the value higher than 75000 in column $4 2df | awk '$4 > 75000' 3 4# Print the all line when column $4 is higher than 75000 5df | awk '$4 > 75000 {print $0}' But if you look for a string, the search need to be included in /search/ or ;search;. When you print $0 represent the all line, $1 first column, $2 second column etc.
🐴 Sed
🐴 Sed
The Basics 1sed -e '…' -e '…' # Several execution 2sed -i # Replace in place 3sed -r # Play with REGEX 4 5# The most usefull 6sed -e '/^[ ]*#/d' -e '/^$/d' <fich.> # openfile without empty or commented lines 7sed 's/ -/\n -/g' # replace all "-" with new lines 8sed 's/my_match.*/ /g' # remove from the match till end of line 9sed -i '4048d;3375d' ~/.ssh/known_hosts # delete lines Number 10 11# Buffer 12s/.*@(.*)/$1/; # keep what is after @ put it in buffer ( ) and reuse it with $1. 13sed -e '/^;/! s/.*-reserv.*/; Reserved: &/' file.txt # resuse search with & 14 15# Search a line 16sed -e '/192.168.130/ s/^/#/g' -i /etc/hosts # Comment a line 17sed -re 's/^;(r|R)eserved:/; Reserved:/g' file.txt # Search several string 18 19# Insert - add two lines below a match pattern 20sed -i '/.*\"description\".*/s/$/ \n \"after\" : \"network.target\"\,\n \"requires\" : \"network.target\"\,/g' my_File 21 22# Append 23sed '/WORD/ a Add this line after every line with WORD' 24 25# if no occurence, then add it after "use_authtok" 26sed -e '/remember=10/!s/use_authtok/& remember=10/' -i /etc/pam.d/system-auth-permanent
👮 Justfile
👮 Justfile
Interesting example from justfile documentation: where it create mktemp and set it in variable then by concatenation you get a full path to the tar.gz. Then the Recipe “publish” create the artifact again and push it to a server. 1tmpdir := `mktemp` # Create a tmp file 2version := "0.2.7" 3tardir := tmpdir / "awesomesauce-" + version 4tarball := tardir + ".tar.gz" # use tmpfile path to create a tarball 5 6publish: 7 rm -f {{tarball}} 8 mkdir {{tardir}} 9 cp README.md *.c {{tardir}} 10 tar zcvf {{tarball}} {{tardir}} 11 scp {{tarball}} me@server.com:release/ 12 rm -rf {{tarball}} {{tardir}} This one can be really usefull to define a default value which can be redefine with env variable:
👷 Makefile
👷 Makefile
Shell Variable $$var $$( python -c ‘import sys; print(sys.implementation.name)’ ) Make Variable T ?= foo # give a default value T := $(shell whoami) # execute shell immediately to put in the var PHONY to execute several makefile Example 1 1SUBDIRS = foo bar baz 2 3## dir is a Shell variables 4## SUBDIR and MAKE are Internal make variables 5subdirs: 6 for dir in $(SUBDIRS); do \ 7 $(MAKE) -C $$dir; \ 8 done Example 2 1SUBDIRS = foo bar baz 2 3.PHONY: subdirs $(SUBDIRS) 4subdirs: $(SUBDIRS) 5$(SUBDIRS): 6 $(MAKE) -C $@ 7foo: baz Idea for a testing tools 1git clone xxx /tmp/xxx&& make -C !$/Makefile 2make download le conteneur 3make build le binaire 4make met le dans /use/local/bin 5make clean 6make help Sources: Tutorials
📐 Storage
📐 Storage
General concern If you want to move VMs to an another Storage Domain, you need to copy the template from it as well! Remove a disk: 1# IF RHV does not use anymore disk those should appear empty in lsblk: 2lsblk -a 3sdf 8:80 0 4T 0 disk 4└─36001405893b456536be4d67a7f6716e3 253:38 0 4T 0 mpath 5sdg 8:96 0 4T 0 disk 6└─36001405893b456536be4d67a7f6716e3 253:38 0 4T 0 mpath 7sdh 8:112 0 4T 0 disk 8└─36001405893b456536be4d67a7f6716e3 253:38 0 4T 0 mpath 9sdi 8:128 0 0 disk 10└─360014052ab23b1cee074fe38059d7c94 253:39 0 100G 0 mpath 11sdj 8:144 0 0 disk 12└─360014052ab23b1cee074fe38059d7c94 253:39 0 100G 0 mpath 13sdk 8:160 0 0 disk 14└─360014052ab23b1cee074fe38059d7c94 253:39 0 100G 0 mpath 15 16# find all disks from LUN ID 17LUN_ID="360014054ce7e566a01d44c1a4758b092" 18list_disk=$(dmsetup deps -o devname ${LUN_ID}| cut -f 2 |cut -c 3- |tr -d "()" | tr " " "\n") 19echo ${list_disk} 20 21# Remove from multipath 22multipath -f "${LUN_ID}" 23 24# remove disk 25for i in ${list_disk}; do echo ${i}; blockdev --flushbufs /dev/${i}; echo 1 > /sys/block/${i}/device/delete; done 26 27# You can which disk link with which LUN on CEPH side 28ls -l /dev/disk/by-* NFS for OLVM/oVirt Since oVirt need a shared stockage, we can create a local NFS to bypass this point if no Storage bay.