Memo

Basic Troubleshooting Checks interfaces 1nmcli con show 2NAME UUID TYPE DEVICE 3ens192 4d0087a0-740a-4356-8d9e-f58b63fd180c ethernet ens192 4ens224 3dcb022b-62a2-4632-8b69-ab68e1901e3b ethernet ens224 5 6nmcli dev status 7DEVICE TYPE STATE CONNECTION 8ens192 ethernet connected ens192 9ens224 ethernet connected ens224 10ens256 ethernet connected ens256 11lo loopback unmanaged -- 12 13# Get interfaces details : 14nmcli connection show ens192 15nmcli -p con show ens192 16 17# Get DNS settings in interface 18UUID=$(nmcli --get-values connection.uuid c show "cloud-init eth0") 19nmcli --get-values ipv4.dns c show $UUID Changing Interface name 1nmcli connection add type ethernet mac "00:50:56:80:11:ff" ifname "ens224" 2nmcli connection add type ethernet mac "00:50:56:80:8a:0b" ifname "ens256" Create a custom config 1nmcli con load /etc/sysconfig/network-scripts/ifcfg-ens224 2nmcli con up ens192 Adding a Virtual IP 1nmcli con mod enp1s0 +ipv4.addresses "192.168.122.11/24" 2ip addr del 10.10.10.36/24 dev ens160 3 4nmcli con reload # before to reapply 5nmcli device reapply ens224 6systemctl status network.service 7systemctl restart network.service Add a DNS entry 1UUID=$(nmcli --get-values connection.uuid c show "cloud-init eth0") 2DNS_LIST=$(nmcli --get-values ipv4.dns c show $UUID) 3nmcli conn modify "$UUID" ipv4.dns "${DNS_LIST} ${DNS_IP}" 4 5# /etc/resolved is managed by systemd-resolved 6sudo systemctl restart systemd-resolved

Server Side First Install samba and samba-client (for debug + test) /etc/samba/smb.conf 1[home] 2Workgroup=WORKGROUP (le grp par defaul sur windows) 3Hosts allow = ... 4[shared] 5browseable = yes 6path = /shared 7valid users = user01, @un_group_au_choix 8writable = yes 9passdb backend = tdbsam #passwords are stored in the /var/lib/samba/private/passdb.tdb file. Test samba config testparm /usr/bin/testparm -s /etc/samba/smb.conf smbclient -L \192.168.56.102 -U test : list all samba shares available smbclient //192.168.56.102/sharedrepo -U test : connect to the share pdbedit -L : list user smb (better than smbclient)

SSHFS SSHFS mounts a remote filesystem on your local filesystem through an SSH connection, all with user rights. The advantage is being able to manipulate remote data with any file manager (Nautilus, Konqueror, ROX, or even the command line). - Prerequisites: administrator rights, ethernet connection, installation of FUSE and the SSHFS package. - SSHFS users must belong to the `fuse` group. Note: FUSE allows a user to mount a filesystem themselves. Normally, mounting a filesystem requires being an administrator, or having it pre-approved in /etc/fstab with hard-coded information.

Investigate a user 1last # the last user connections to a server (based on /var/log/wtmp or btmp). 2ac -d # statistics of my connection time per day. 3ac -p <user> # the connection time of all users (or of a specific user). 4finger # who is connected (-l to also see mails and plans of all users). 5w # who is connected, doing what, and how much CPU they use. 6who # who is connected (-u for more info: PID, etc.). 7who am i # with which login I am connected. 8id -a # all info about the user I'm connected as (more precise than "who am i"). 9logname # the login name of the current account. Reboots & uptime 1last reboot # see all the reboots that took place. 2uptime # see how long the server has been up + the load average. 3lslogins -L # also shows whether a user shutdown/rebooted the machine.

Where the system logs live On a Unix machine, the system logs are in /var/log/messages (or /var/adm/messages on Solaris). This is where you find the errors, with log rotation. Default syslog output Linux Solaris HP-UX AIX BSD location /var/log/messages, /var/log/secure, /var/log/boot.log /var/adm/messages /var/adm/syslog/mail.log, /var/adm/syslog/syslog.log /tmp or none /var/log/syslog System accounting (login & process) Type Linux Solaris HP-UX AIX current logins /var/run/utmp /var/adm/utmpx /var/adm/utmp /etc/utmp login history /var/log/wtmp /var/adm/wtmpx /var/adm/wtmp /var/adm/wtmp process accounting /var/log/pacct /var/adm/pacct /var/adm/pacct /var/adm/pacct Login errors Linux Solaris HP-UX AIX failed logins /var/log/btmp, /var/log/messages /var/adm/loginlog, /var/adm/sulog /var/adm/sulog /etc/security/failedlogin Investigate the logs 1# today's logs 2grep "$(date '+%b %d')" /var/log/messages 3 4# disk errors (nawk: print the last field of the "Error Block" lines) 5nawk '/Error Block/{print $NF}' /var/adm/messages* | sort | uniq 6 7# find the IPs in a log, sort them and remove the duplicates 8cat /var/log/maillog | grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4 | uniq Network investigation 1# ping a list of servers 2for ip in $(awk '/192.168.45/ {print $1}' /etc/hosts); do ping -c 1 $ip; done Investigate on several servers 1for vm in vm{1..27}; do ssh -q $vm "hostname; free; sar -r 3 3"; done SSH authentication logs /var/log/auth.log — SSH connection logs. Check that there are not too many failed connections (a sign of an intrusion attempt).

Find files quickly 1locate <pattern> # find a directory or file quickly (uses an index); a brand-new file won't be found. 2updatedb # update the locate index. Open a file 1view <file> # opens a read-only vi view (preferred if you just want to search/view). 2gzcat / zcat <file.gz> # read a gzipped file. Info on a file or directory 1stat </my/file> # all info about a file (inode, creation, modification, access dates, etc.). 2stat -f <FS> # info about a filesystem. 3stat -c%s $LOGFILE # [scripting] get a precise value (size, modification date, etc.). grep 1grep -w 'xyz' # match the whole word. 2grep -x 'Hello, world!' # the whole line must match. 3grep -c <pattern> # count the matching lines. 4grep -l "ERROR:" *.log # search all .log files, list the files that match. 5grep -L <pattern> # inverse: list the files that do NOT match. 6grep -f <patternfile> <file> # apply the patterns read from patternfile. 7grep -i <pattern> # ignore case. 8grep -v <pattern> # return the lines that do NOT match. 9grep -m x <pattern> # stop after x matching lines. 10grep -n <pattern> # show the line number. 11grep -q <pattern> # quiet: exit 0 if found, 1 (or 2) otherwise (for scripting). 12grep -s <pattern> # suppress permission/inexistent-file error messages. 13grep -H <pattern> # show the filename next to each matching line. 14grep -h <pattern> # do not show the filename (default behaviour). 15grep -A x <pattern> # also show x lines After. 16grep -B x <pattern> # also show x lines Before. 17grep -C x <pattern> # show x lines of context (A + B). 18grep -a <pattern> <binary> # search a binary file as if it were text. 1egrep = grep -E # for complex regular expressions. 1# extract the 3rd field, then cut: 2cat file | grep /u01/grid/19c | awk '{print $3}' | cut -f2 -d'"' 3# is equivalent to: 4cat file | grep -o /u01/grid/19c

Configurations /etc/crontab : the daemon’s configuration file, defining the default behaviour of crond (SHELL, MAILTO, etc.). /etc/cron.d/... : system crontab (used by admins). /var/spool/cron/root : crontab per user. “Day of month” and “Day of week” are combined with a logical OR, so if both are set, the job runs on that day of the month and on that day of the week. 1MAILTO="admin@example.com" 2* * * * * root /usr/local/sbin/mycommand.sh > /dev/null 2>&1 Anacron /etc/anacrontab : file that runs, via the run-parts command, /etc/cron.daily, /etc/cron.weekly, /etc/cron.monthly. /etc/cron.d/0hourly : exception, runs /etc/cron.hourly via run-parts, checking the last run of the task in /var/spool/anacron/.... Special cases Exactly the last day of each month:

1logrotate -d /etc/logrotate.d/app # test a new configuration. 2reading config info for /data/log/app 3Handling 1 logs 4rotating pattern: /data/log/app after 1 days (10 rotations) 5empty log files are rotated, old logs are removed Options Usage: logrotate [OPTION...] <configfile> -d, --debug Don't do anything, just test (implies -v) -f, --force Force file rotation -m, --mail=command Command to send mail (instead of `/bin/mail') -s, --state=statefile Path of state file -v, --verbose Display messages during rotation

Change the root directory of a command or a process, and its children. In no case should `chroot` be relied upon as a security boundary — a process running as root can escape the jail. Example: creating a chroot 1# create the "jail" directory 2J=$HOME/jail 3mkdir -p $J 4mkdir -p $J/{bin,lib64,lib} 5cd $J 6 7# copy the binaries and their libraries into the jail 8cp -v /bin/{bash,ls} $J/bin 9 10list="$(ldd /bin/bash | egrep -o '/lib.*\.[0-9]')" 11for i in $list; do cp -v "$i" "${J}${i}"; done 12 13list="$(ldd /bin/ls | egrep -o '/lib.*\.[0-9]')" 14for i in $list; do cp -v "$i" "${J}${i}"; done 15 16# enter the jail 17sudo chroot $J /bin/bash

Shutdown / reboot Solaris Red Hat Ubuntu / Debian HP-UX AIX Power down shutdown -i5 -g0 -y shutdown -h shutdown -h shutdown -h now shutdown -F Reboot shutdown -i6 -g0 -y shutdown -r shutdown -r shutdown -r now shutdown -Fr OK prompt shutdown -i0 -g0 -y — — — — Fast reboot -- -r (reconfigure) shutdown -f (no fsck) shutdown -P (power off) shutdown -F (force fsck) — Force fsck touch /reconfigure touch /forcefsck edit /etc/default/rcS → FSCKFIX=yes — — Change runlevel Tool Solaris Red Hat Ubuntu / Debian HP-UX AIX halt ✅ ✅ ✅ ✅ ✅ init ✅ ✅ ✅ ✅ ✅ poweroff ✅ ✅ ✅ ✅ ✅ reboot ✅ ✅ ✅ ✅ ✅ shutdown ✅ ✅ ✅ ✅ ✅ telinit ✅ ✅ ✅ — ✅ uadmin ✅ — — — — Runlevels Level Solaris Red Hat Ubuntu / Debian HP-UX AIX 0 shutdown halt halt halt reserved 1 single user single user single user single user reserved 2 n/a multiuser (no networking) multiuser (default) multiuser (networking) multiuser + NFS 3 multi-user multiuser (networking) same as 2 multiuser + NFS + CDE GUI (default) user defined 4 n/a unused same as 2 multiuser + NFS + VUE GUI user defined 5 power off GUI same as 2 n/a user defined 6 reboot reboot reboot n/a user defined 7-9 — — — — user defined Change the default runlevel Solaris / Red Hat / HP-UX / AIX : edit the initdefault line in vi /etc/inittab. Ubuntu / Debian : edit vi /etc/event.d/rc-default. On systemd systems (RHEL 7+, Ubuntu 15+), the SysV runlevels are replaced by targets — e.g. systemctl isolate multi-user.target (runlevel 3), systemctl isolate graphical.target (runlevel 5), systemctl set-default multi-user.target. See the Systemd page.

Client verification (ntpd / chrony) 1ntpstat # see which NTP server we synchronise with, and whether the sync is good. 1synchronised to NTP server (192.168.1.12) at stratum 4 2 time correct to within 68 ms 3 polling server every 1024 s 1ntpq -p # see the state of the peers. 2ntpq -c peers remote refid st t when poll reach delay offset jitter ============================================================================== +192.168.1.11 192.168.2.4 4 u 259 1024 373 0.731 -0.980 0.557 *192.168.1.12 192.168.3.21 3 u 385 1024 377 0.773 0.146 0.365 192.168.4.255 .BCST. 16 u - 64 0 0.000 0.000 0.000 The server preceded by an asterisk (*) is the one being used. Those preceded by a - are currently discarded by the server-selection algorithm. Those whose name is preceded by a + are possible synchronisation candidates. A server preceded by a space is either unreachable or too distant. Column meaning remote — the server name. refid — the parent server’s identifier. st — the server’s stratum. t — the server type. when — seconds elapsed since the last contact. poll — seconds between each contact. reach — bitmask of successful contacts (octal): the server considers itself synchronised when reach reaches 177; a quality, stable connection shows 377. delay — estimated round-trip time (ms) of the UDP packet. offset — estimated difference between the peer’s clock and the internal clock. jitter — dispersion of the reference values obtained from this peer. Restart the NTP daemon 1service ntpd restart # or: systemctl restart ntpd Configuration & logs 1cat /etc/ntp.conf # "server example.com" + restart ntpd + enable 2/var/log/ntpstats ntpdate (legacy) Old service that synchronises NTP at boot (install the package first).

Configuration files File Check command Purpose /etc/passwd pwck user accounts /etc/group grpck groups /etc/shadow — password hashes and aging /etc/gshadow — group passwords /etc/skel — files installed by default when a user is created Basic commands 1useradd -g <GID> -G <GID2> <user> # create a user in primary group GID (and supplementary group GID2). 2usermod <options> <user> # modify a user. 3userdel -r <user> # delete a user (and its home directory). 4groupadd / groupmod / groupdel # manage groups. 5 6id -a # show all info about the current user (UID, GUID, groups, etc.) - more precise than "who am i". 7sg <group> -c '<command>' # execute a command as a different group ID (to run scripts or write to a file with group rights). Password management 1passwd -u <user> # unlock a user account. 2echo "password" | passwd --stdin <user> # scripted password change. Account aging 1chage -l <user> # see the expiration dates. 1# list the expiry of every account 2for account in $(cut -f1 -d: /etc/passwd); do 3 echo "ACCOUNT: $account , EXPIRES: $(chage -l $account | grep 'Account expires' | awk '{print $4, $5, $6}'), CHANGED: $(chage -l $account | grep 'Last password change' | awk '{print $5, $6, $7}')"; 4done 1# change the aging info interactively 2chage <user> To unlock an account, set “Last Password Change” to -1 in chage (or use passwd -u <user>).