Browse Docs

Systems

OS sections in docs

Documentation about Operating Systems and their administration.

In this section

  • Unix-Like

    Documentation about commands that should work on all Unix-like systems.

    • OS
      • ๐Ÿ‘ข Boot

        The Boot - starting process

        - The BIOS is started automatically and detects the peripherals.
- Loads the boot routine from the MBR (Master Boot Record) - it is the boot disk, located on the first sector of the hard disk.
- The MBR contains a loader that loads the "second stage loader": this is the "boot loader" specific to the system being loaded.
	-> Linux uses LILO (Linux Loader) or GRUB (Grand Unified Bootloader).
- LILO loads the kernel into memory, decompresses it, and passes it the parameters.
- The kernel mounts the `/` filesystem (from there, the commands in `/sbin` and `/bin` are available).
- The kernel runs its first process: `init`.

        LILO configuration

        LILO can offer several kernels as choices. The default choice: “Linux”. /etc/lilo.conf : configuration of the kernel parameters. /sbin/lilo : to write the new parameters to disk. -> creates the /boot/map file, which contains the physical blocks where the boot program is located.

      • โš™๏ธ Systemd

        systemd replaces the SysV init system: services are managed with systemctl, and runlevels map to targets.

        1systemctl status <unit>                   # status of a service.
2systemctl start|stop|restart <unit>      # run / stop / restart.
3systemctl enable|disable <unit>          # start at boot (or not).
4systemctl isolate multi-user.target      # equivalent of runlevel 3.
5systemctl set-default multi-user.target  # change the default target.
6systemctl get-default

        See the Runlevels & Shutdown page for the classic runlevel table.

      • ๐Ÿ” Runlevels & Shutdown

        Shutdown / reboot

        SolarisRed HatUbuntu / DebianHP-UXAIX
        Power downshutdown -i5 -g0 -yshutdown -hshutdown -hshutdown -h nowshutdown -F
        Rebootshutdown -i6 -g0 -yshutdown -rshutdown -rshutdown -r nowshutdown -Fr
        OK promptshutdown -i0 -g0 -yโ€”โ€”โ€”โ€”
        Fastreboot -- -r (reconfigure)shutdown -f (no fsck)shutdown -P (power off)shutdown -F (force fsck)โ€”
        Force fscktouch /reconfiguretouch /forcefsckedit /etc/default/rcS โ†’ FSCKFIX=yesโ€”โ€”

        Change runlevel

        ToolSolarisRed HatUbuntu / DebianHP-UXAIX
        haltโœ…โœ…โœ…โœ…โœ…
        initโœ…โœ…โœ…โœ…โœ…
        poweroffโœ…โœ…โœ…โœ…โœ…
        rebootโœ…โœ…โœ…โœ…โœ…
        shutdownโœ…โœ…โœ…โœ…โœ…
        telinitโœ…โœ…โœ…โ€”โœ…
        uadminโœ…โ€”โ€”โ€”โ€”

        Runlevels

        LevelSolarisRed HatUbuntu / DebianHP-UXAIX
        0shutdownhalthalthaltreserved
        1single usersingle usersingle usersingle userreserved
        2n/amultiuser (no networking)multiuser (default)multiuser (networking)multiuser + NFS
        3multi-usermultiuser (networking)same as 2multiuser + NFS + CDE GUI (default)user defined
        4n/aunusedsame as 2multiuser + NFS + VUE GUIuser defined
        5power offGUIsame as 2n/auser defined
        6rebootrebootrebootn/auser defined
        7-9โ€”โ€”โ€”โ€”user defined

        Change the default runlevel

        • Solaris / Red Hat / HP-UX / AIX : edit the initdefault line in vi /etc/inittab.
        • Ubuntu / Debian : edit vi /etc/event.d/rc-default.
      • โ˜๏ธ Cloud-Init

        Troubleshooting

        • cloud-init status --wait usefull for scripting, waiting cloud-init to finish before going to next step.

        • cloud-init status --long

        1status: done
2extended_status: done
3boot_status_code: enabled-by-generator
4last_update: Thu, 01 Jan 1970 00:00:55 +0000
5detail: DataSourceNoCloud [seed=/dev/sr0]
6errors: []
7recoverable_errors: {}
        • sudo cloud-init analyze show
         1-- Boot Record 01 --
 2The total time elapsed since completing an event is printed after the "@" character.
 3The time the event takes is printed after the "+" character.
 4
 5Starting stage: init-local
 6|`->no cache found @00.00600s +00.00000s
 7|`->found local data from DataSourceNoCloud @00.01500s +00.12600s
 8Finished stage: (init-local) 00.75400 seconds
 9
10Starting stage: init-network
11|`->restored from cache with run check: DataSourceNoCloud [seed=/dev/sr0] @04.21100s +00.00200s
12|`->setting up datasource @04.22800s +00.00000s
13|`->reading and applying user-data @04.23400s +00.00500s
14|`->reading and applying vendor-data @04.23900s +00.00000s
15|`->reading and applying vendor-data2 @04.23900s +00.00000s
16|`->activating datasource @04.27100s +00.00100s
17|`->config-seed_random ran successfully and took 0.000 seconds @04.29500s +00.00100s
18|`->config-write_files ran successfully and took 0.001 seconds @04.29600s +00.00100s
19|`->config-growpart ran successfully and took 0.562 seconds @04.29700s +00.56200s
20|`->config-resizefs ran successfully and took 0.193 seconds @04.86000s +00.19200s
21|`->config-mounts ran successfully and took 0.001 seconds @05.05200s +00.00100s
22|`->config-set_hostname ran successfully and took 0.004 seconds @05.05300s +00.00500s
23|`->config-update_hostname ran successfully and took 0.001 seconds @05.05800s +00.00100s
24|`->config-update_etc_hosts ran successfully and took 0.005 seconds @05.05900s +00.00500s
25|`->config-users_groups ran successfully and took 0.216 seconds @05.06400s +00.21600s
26|`->config-ssh ran successfully and took 0.404 seconds @05.28100s +00.40400s
27|`->config-set_passwords ran successfully and took 0.001 seconds @05.68500s +00.00200s
28Finished stage: (init-network) 01.50000 seconds
29
30Starting stage: modules-config
31|`->config-ssh_import_id ran successfully and took 0.001 seconds @07.43300s +00.00100s
32|`->config-locale ran successfully and took 0.003 seconds @07.43400s +00.00300s
33|`->config-grub_dpkg ran successfully and took 0.352 seconds @07.43700s +00.35200s
34|`->config-apt_configure ran successfully and took 0.049 seconds @07.79000s +00.04800s
35|`->config-timezone ran successfully and took 0.007 seconds @07.83900s +00.00700s
36|`->config-runcmd ran successfully and took 0.001 seconds @07.84600s +00.00100s
37|`->config-byobu ran successfully and took 0.000 seconds @07.84700s +00.00100s
38Finished stage: (modules-config) 00.45400 seconds
39
40Starting stage: modules-final
41|`->config-package_update_upgrade_install ran successfully and took 26.632 seconds @20.56700s +26.63300s
42|`->config-write_files_deferred ran successfully and took 0.001 seconds @47.20000s +00.00200s
43|`->config-reset_rmc ran successfully and took 0.000 seconds @47.20200s +00.00100s
44|`->config-scripts_vendor ran successfully and took 0.001 seconds @47.20300s +00.00000s
45|`->config-scripts_per_once ran successfully and took 0.000 seconds @47.20300s +00.00100s
46|`->config-scripts_per_boot ran successfully and took 0.000 seconds @47.20400s +00.00000s
47|`->config-scripts_per_instance ran successfully and took 0.000 seconds @47.20400s +00.00100s
48|`->config-scripts_user ran successfully and took 0.558 seconds @47.20500s +00.55800s
49|`->config-ssh_authkey_fingerprints ran successfully and took 0.005 seconds @47.76400s +00.00500s
50|`->config-keys_to_console ran successfully and took 0.054 seconds @47.76900s +00.05500s
51|`->config-install_hotplug ran successfully and took 0.001 seconds @47.82400s +00.00100s
52|`->config-final_message ran successfully and took 0.001 seconds @47.82500s +00.00100s
53Finished stage: (modules-final) 27.29600 seconds

        • Check the logs: sudo tail -n 50 /var/log/cloud-init-output.log

      • ๐ŸŽซ Certificates Authority

        Trust a CA on Linux host

        1# [RHEL] RootCA from DC need to be installed on host: 
2cp my-domain-issuing.crt /etc/pki/ca-trust/source/anchors/my_domain_issuing.crt
3cp my-domain-rootca.crt /etc/pki/ca-trust/source/anchors/my_domain_rootca.crt
4update-ca-trust extract
5
6# [Ubuntu] 
7sudo apt-get install -y ca-certificates
8sudo cp local-ca.crt /usr/local/share/ca-certificates
9sudo update-ca-certificates
      • ๐Ÿ–ฅ Out-of-Band Management

        Out-of-band management refers to accessing and controlling a server (console, power on/off, BIOS, remote media) through a dedicated management channel that is separate from the normal data path. “In-band” means going through the OS and its network interface; “out-of-band” uses an independent controller (a BMC โ€” Baseboard Management Controller) with its own network port, so it still works even when the OS is down, the machine is hung, or the network stack is broken. RSA is one vendor/technology family of out-of-band access.

      • ๐Ÿ” ISO Checksum

        Verify an ISO image

        To verify that an ISO image is good: compute its checksum with a sha1 or sha256 key, then compare it with the key published on the official website.

        1sha256sum image.iso
2sha1sum image.iso
      • ๐Ÿ“œ Logrotate
        1logrotate -d /etc/logrotate.d/app   # test a new configuration.
2reading config info for /data/log/app
3Handling 1 logs
4rotating pattern: /data/log/app  after 1 days (10 rotations)
5empty log files are rotated, old logs are removed

        Options

        Usage: logrotate [OPTION...] <configfile>
  -d, --debug               Don't do anything, just test (implies -v)
  -f, --force               Force file rotation
  -m, --mail=command        Command to send mail (instead of `/bin/mail')
  -s, --state=statefile     Path of state file
  -v, --verbose             Display messages during rotation
      • ๐Ÿ• NTP & Time Synchronisation

        Client verification (ntpd / chrony)

        1ntpstat    # see which NTP server we synchronise with, and whether the sync is good.

        1synchronised to NTP server (192.168.1.12) at stratum 4
2   time correct to within 68 ms
3   polling server every 1024 s

        1ntpq -p     # see the state of the peers.
2ntpq -c peers

             remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+192.168.1.11     192.168.2.4    4 u  259 1024  373    0.731   -0.980   0.557
*192.168.1.12     192.168.3.21   3 u  385 1024  377    0.773    0.146   0.365
 192.168.4.255    .BCST.         16 u    -   64    0    0.000    0.000   0.000
        • The server preceded by an asterisk (*) is the one being used.
        • Those preceded by a - are currently discarded by the server-selection algorithm.
        • Those whose name is preceded by a + are possible synchronisation candidates.
        • A server preceded by a space is either unreachable or too distant.

        Column meaning

        • remote โ€” the server name.
        • refid โ€” the parent server’s identifier.
        • st โ€” the server’s stratum.
        • t โ€” the server type.
        • when โ€” seconds elapsed since the last contact.
        • poll โ€” seconds between each contact.
        • reach โ€” bitmask of successful contacts (octal): the server considers itself synchronised when reach reaches 177; a quality, stable connection shows 377.
        • delay โ€” estimated round-trip time (ms) of the UDP packet.
        • offset โ€” estimated difference between the peer’s clock and the internal clock.
        • jitter โ€” dispersion of the reference values obtained from this peer.

        Restart the NTP daemon

        1service ntpd restart      # or: systemctl restart ntpd

        Configuration & logs

        1cat /etc/ntp.conf    # "server example.com" + restart ntpd + enable
2/var/log/ntpstats

        ntpdate (legacy)

        Old service that synchronises NTP at boot (install the package first).

    • Performance
      • ๐Ÿ“ˆ Performance Monitoring & Tuning

        How to approach performance monitoring and tuning in Linux, and the various subsystems (and performance metrics) that need to be monitored.

        On a very high level, the following four subsystems need to be monitored:

        • CPU
        • Memory
        • I/O
        • Network

        1. CPU

        Four critical performance metrics for the CPU: context switch, run queue, CPU utilization, and load average.

        Context Switch

        • When the CPU switches from one process (or thread) to another, it is called a context switch.
        • When a process switch happens, the kernel stores the current state of the CPU (of a process or thread) in memory.
        • The kernel also retrieves the previously stored state (of a process or thread) from memory and puts it in the CPU.
        • Context switching is essential for multitasking of the CPU.
        • However, a higher level of context switching can cause performance issues.

        Run Queue

        • The run queue indicates the total number of active processes in the current queue for the CPU.
        • When the CPU is ready to execute a process, it picks it up from the run queue based on the priority of the process.
        • Note that processes that are in a sleep state, or I/O wait state, are not in the run queue.
        • A higher number of processes in the run queue can therefore cause performance issues.

        CPU Utilization

        • Indicates how much of the CPU is currently being used.
        • 100% CPU utilization means the system is fully loaded.

        Load Average

        • Indicates the average CPU load over a specific time period.
        • On Linux, load average is displayed for the last 1 minute, 5 minutes, and 15 minutes. This helps to see whether the overall load on the system is going up or down. For example, a load average of 0.75 1.70 2.10 indicates that the load is coming down (0.75 = last 1 minute, 1.70 = last 5 minutes, 2.10 = last 15 minutes).
        • Note that this load average is calculated by combining both the total number of processes in the queue, and the total number of processes in the uninterruptible task status.

        2. Network

        • A good understanding of TCP/IP concepts is helpful when analyzing any network issue.
        • For network interfaces, monitor the total number of packets (and bytes) received/sent through the interface, the number of packets dropped, etc.

        3. I/O

        • I/O wait is the amount of time the CPU is waiting for I/O. Consistent high I/O wait on the system indicates a problem in the disk subsystem.
        • Monitor reads/second and writes/second. These are measured in blocks, i.e. the number of blocks read/written per second. They are also referred to as bi and bo (block in and block out).
        • tps indicates total transactions per second, which is the sum of rtps (read transactions per second) and wtps (write transactions per second).

        4. Memory

        • RAM is the physical memory. If you have 4 GB of RAM installed, you have 4 GB of physical memory.
        • Virtual memory = swap space available on disk + physical memory.
        • The virtual memory contains both user space and kernel space.
        • Using a 32-bit or a 64-bit system makes a big difference in determining how much memory a process can use:
          • On a 32-bit system a process can only access a maximum of 4 GB of virtual memory.
          • On a 64-bit system there is no such limitation.
        • Unused RAM is used by the kernel as filesystem cache.
        • Linux swaps when it needs more memory than the physical memory. When it swaps, it writes the least-used memory pages from the physical memory to the swap space on the disk.
        • Lots of swapping can cause performance issues: the disk is much slower than the physical memory, and it takes time to swap the memory pages from RAM to disk.

        The subsystems are interrelated

        All four subsystems are interrelated. Just because you see a high reads/second, writes/second, or I/O wait, it does not mean the issue is with the I/O subsystem. It also depends on what the application is doing. In most cases, the performance issue is caused by the application running on the Linux system.

    • Files
      • ๐Ÿ—œ Compression

        Zip / Unzip

        1zip <archive.zip> <file1> <file2>     # compress files.
2zip -r <archive.zip> <directory>      # compress a directory.
3unzip archive_name.zip [-d directory] # decompress an archive.

        Gzip / Gunzip

        gzip is based on the Deflate algorithm (a combination of the LZ77 and Huffman algorithms).

        1gzip -l                                          # show the size of the uncompressed file.
2gzip <file>                                      # compress.
3gunzip <file.gz> | gzip -d <file.gz>             # decompress.
4gzip -9 <my_file>                                # compress a file optimally.
5gzip -c <file1> <file2> > compressed_file.gz     # compress several files into a single one.

        Bzip2 / Bunzip2

        bzip2 is an alternative to gzip, more efficient but slower.

      • ๐Ÿ“ฆ Archive

        Tar - “tape archiver”

        • Preserve files permissions and ownership.

        • The Basics

         1# Archive
 2tar cvf my_archive.tar <file1> <file2> </dir/folder/>
 3
 4## Archive and compress with zstd everything in the current dir and push to /target/dir
 5tar -I zstd -vcf archive.tar.zstd -C /target/dir . 
 6
 7# Extract
 8tar xvf my_archive.tar
 9
10# Extract and push to target dir 
11tar -zxvf new.tar.gz -C /target/dir

        • Other useful options โ€ข t : list archive’s content. โ€ข T : archive list given by a file. โ€ข P : preserve absolute path (useful for backing up /etc). โ€ข X : exclude. โ€ข z : gunzip compression. โ€ข j : bzip2 compression. โ€ข J : lzma compression.

      • ๐Ÿ’พ Backup & Sync

        Rsync

        The classic formula

        1rsync -arv --info=progress2 photo backup_photo
        • a = archive โ€” preserves permissions (owner, group), times, symbolic links and devices.
        • r = recursive โ€” copies directories and sub-directories.
        • v = verbose โ€” prints what is being copied.

        Examples

        1rsync -apvz --stats --update --exclude gsast/olap_cubes --exclude gsast/param   user@server-src:/export/ user@server-dest:/home/
2rsync -av -e ssh root@192.168.1.10:/backup/DUMP/* .
3rsync -azp --stats root@oracle-src:/ec/sw/oracle/client/product/12.2.0.1/network/mesg/ ~/mesg/
4rsync -azp /home/user/mesg/ root@oracle-dest.example.com:/ec/sw/oracle/client/product/12.2.0.1/network/mesg/
5
6ssh root@oracle-dest.example.com "ls -lrt /ec/sw/oracle/client/product/12.2.0.1/network/mesg/"
7ssh root@oracle-dest.example.com "chown oracle:dc_dba /ec/sw/oracle/client/product/12.2.0.1/network/mesg/*"
8
9rsync -aS --delete --rsh /export/home backup-host:/export/save

        Propagate deletions to the backup

        If you delete files in the source directory, rsync does not propagate the deletion to the backup directory unless you add the --delete option.

    • Processes
      • ๐Ÿ” Find & Inspect Processes

        Find a process

        1ps -fp <pid>        # find a process by its PID.
2pidof httpd         # find the PIDs of httpd.
3pidstat -lp <pid>   # process name with all its complete arguments.
4                    # for a tomcat or weblogic process, you can split the arguments with: sed 's/ -/\n -/g'
5pidstat -C "mysql"  # find a process by its name (gives the PID and CPU load).

        The process tree

        1pstree -pu   # the process tree with PID and user (if pstree is not installed, use the alternatives below).

        1ps -ejH
2  PID  PGID   SID TTY          TIME CMD
3    1     1     1 ?        00:00:37 init
411016 11009 11009 ?        00:00:00       sshd
511017 11017 11017 pts/12   00:00:00         bash
611125 11125 11017 pts/12   00:00:00           telnet

        1ps axjf
2 PPID   PID  PGID   SID TTY      TPGID STAT   UID   TIME COMMAND
3    0     1     1     1 ?           -1 Ss       0   0:37 init [5]
4 8617 10610 10610 10610 ?           -1 Ss       0   0:00  \_ sshd: support [priv]
510610 10710 10610 10610 ?           -1 S     5027   0:00      \_ sshd: support@notty
610710 10711 10711 10711 ?           -1 Ss    5027   0:00          \_ sshd: support@internal-sftp-server

        1ps faux
2USER       PID  %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
3root         1   0.0  0.0  10372   696 ?        Ss   Aug09   0:37 init [5]
4user1      4168  0.0  0.0   8728   968 ?        Ss   Aug24   0:00  |   \_ /bin/bash -c perl /data/supports/scripts/SRAM_asr5k.pl &>/dev/null
5user1      4174  0.0  0.0  34068  5044 ?        S    Aug24   0:00  |       \_ perl /data/supports/SRAM_asr5k.pl
6user1      4188  0.0  0.0   8728   984 ?        S    Aug24   0:00  |           \_ sh -c grep -c SRAM /data/syslogCOLLECT/LTE_*/*/*.20160824.log

        /proc

        The /proc filesystem exposes per-process information:

      • ๐Ÿ“Š Process Monitoring (pidstat)

        pidstat reports the CPU, memory, I/O and context-switch activity of processes.

        Report the process context-switching activity

         1# pidstat -w -p 3446 2 5
 2Linux 3.10.0-123.13.2.el7.x86_64 (localhost.localdomain) 12/26/2014
 3_x86_64_ (1 CPU)
 407:23:38 AM UID PID cswch/s nvcswch/s Command
 507:23:40 AM 0 3446 0.50 0.00 sshd
 607:23:42 AM 0 3446 0.50 0.00 sshd
 707:23:44 AM 0 3446 0.50 0.00 sshd
 807:23:46 AM 0 3446 0.50 0.00 sshd
 907:23:48 AM 0 3446 0.50 0.00 sshd
10Average: 0 3446 0.50 0.00 sshd
        • cswch/s : number of voluntary context switches the task made per second. (A voluntary context switch occurs when a task blocks because it requires a resource that is unavailable.)
        • nvcswch/s : number of non-voluntary context switches the task made per second. (An involuntary context switch takes place when a task executes for the duration of its time slice and is then forced to relinquish the processor.)

        Page faults and memory

        1pidstat -r -p <PID> 3600 72    # every hour, 72 times - practical for long-term monitoring.
2
3pidstat -r -p <PID> 50 12
407:26:44 PM       PID   minflt/s  majflt/s     VSZ    RSS   %MEM  Command
507:27:34 PM     13775      1.64      0.00 34957320 18183312  55.30  java
        • minflt/s : number of minor faults the task has made per second โ€” those which did not require loading a memory page from disk.
        • majflt/s : number of major faults the task has made per second โ€” those which required loading a memory page from disk.
        • VSZ : Virtual Size โ€” the virtual memory usage of the entire task in kilobytes.
        • RSS : Resident Set Size โ€” the non-swapped physical memory used by the task in kilobytes.

        Disk I/O

        1pidstat -d -p <PID> 50 12

        pidstat -d reports I/O statistics (kernels 2.6.20 and later only). The following values are displayed:

      • ๐Ÿ› Tracing (strace / ltrace / gstack)

        Strace - trace system calls

        1strace -tt -p 24503
2
3strace -o strace01.out -e open -f bash --login -i   # see the files opened during a bash connection.
4    # -o redirects the output / -e filters the system calls / -f follows forks (child processes)
5
6strace -f <binary_script> 2> trace.log :  stdout -> the binary command result, stderr -> the binary's system calls.

        Ltrace - trace library calls

        ltrace traces shared-library calls (like strace, but at the library-call level).

      • ๐ŸŽฏ CPU Affinity

        Source : http://www.glennklockwood.com/hpc-howtos/process-affinity.html

        Taskset

        1sudo apt-get install util-linux   # or: yum install util-linux
2taskset -c 1 script.sh          # run script.sh on CPU number 1  (-c: CPU, -p: PID)
3taskset -c 1,2,3 script.sh      # give it several CPUs.

        Numactl

        1numactl --cpunodebind=0 simulation.x
2numactl --cpunodebind=0 --membind=0 simulation.x
3numactl -C 0 -N 0 simulation.x
4numactl -C +0,1,2,3 simulation.x    # similar to taskset
5numactl -H                          # see which memory corresponds to a CPU

        Note: with numactl, unlike taskset, you cannot change the CPU affinity of a process on the fly.

      • โฐ Jobs & Background

        Schedule a task (at / batch)

        1at      :  schedule a task to run at a later time (/!\ it executes what you give it on stdin).
2           ex :  at 18:22 < "date; ps -ef | wc -l"
3           or : at now + 5 hours  then type your commands then ctrl + d
4at -q a 16:05 tomorrow   :  -q defines the queue (a-z), i.e. the priority.
5at -c  <job_number>      :  see the context and the commands of the task.
6atq      :  list the pending jobs (= at -l).
7atrm     :  delete a job.
8
9batch   :  schedule a task when the load average is below a threshold.

        Run jobs in the background

        1jobs -l   : list the running tasks.

        1# Nohup in series
2for i in {1..6}; do echo "nohup sh -c \"shred -vfz -n 3 /dev/cciss/c0d${i} > nohup${i}.out 2>&1 \" &" ; done | bash

        Three points to remember:

      • โฒ๏ธ Cron & Anacron

        Configurations

        • /etc/crontab : the daemon’s configuration file, defining the default behaviour of crond (SHELL, MAILTO, etc.).
        • /etc/cron.d/... : system crontab (used by admins).
        • /var/spool/cron/root : crontab per user.
        1MAILTO="admin@example.com"
2* * * * *  root  /usr/local/sbin/mycommand.sh > /dev/null 2>&1

        Anacron

        • /etc/anacrontab : file that runs, via the run-parts command, /etc/cron.daily, /etc/cron.weekly, /etc/cron.monthly.
        • /etc/cron.d/0hourly : exception, runs /etc/cron.hourly via run-parts, checking the last run of the task in /var/spool/anacron/....

        Special cases

        Exactly the last day of each month:

      • ๐Ÿ“ฆ Chroot Jail

        Change the root directory of a command or a process, and its children.

        Example: creating a chroot

         1# create the "jail" directory
 2J=$HOME/jail
 3mkdir -p $J
 4mkdir -p $J/{bin,lib64,lib}
 5cd $J
 6
 7# copy the binaries and their libraries into the jail
 8cp -v /bin/{bash,ls} $J/bin
 9
10list="$(ldd /bin/bash | egrep -o '/lib.*\.[0-9]')"
11for i in $list; do cp -v "$i" "${J}${i}"; done
12
13list="$(ldd /bin/ls | egrep -o '/lib.*\.[0-9]')"
14for i in $list; do cp -v "$i" "${J}${i}"; done
15
16# enter the jail
17sudo chroot $J /bin/bash
    • Investigate
      • ๐Ÿ”Ž Search, Find & Compare

        Find files quickly

        1locate <pattern>    # find a directory or file quickly (uses an index); a brand-new file won't be found.
2updatedb            # update the locate index.

        Open a file

        1view <file>             # opens a read-only vi view (preferred if you just want to search/view).
2gzcat / zcat <file.gz>  # read a gzipped file.

        Info on a file or directory

        1stat </my/file>      # all info about a file (inode, creation, modification, access dates, etc.).
2stat -f <FS>         # info about a filesystem.
3stat -c%s $LOGFILE   # [scripting] get a precise value (size, modification date, etc.).

        grep

         1grep -w 'xyz'                # match the whole word.
 2grep -x 'Hello, world!'      # the whole line must match.
 3grep -c <pattern>            # count the matching lines.
 4grep -l "ERROR:" *.log       # search all .log files, list the files that match.
 5grep -L <pattern>            # inverse: list the files that do NOT match.
 6grep -f <patternfile> <file> # apply the patterns read from patternfile.
 7grep -i <pattern>            # ignore case.
 8grep -v <pattern>            # return the lines that do NOT match.
 9grep -m x <pattern>          # stop after x matching lines.
10grep -n <pattern>            # show the line number.
11grep -q <pattern>            # quiet: exit 0 if found, 1 (or 2) otherwise (for scripting).
12grep -s <pattern>            # suppress permission/inexistent-file error messages.
13grep -H <pattern>            # show the filename next to each matching line.
14grep -h <pattern>            # do not show the filename (default behaviour).
15grep -A x <pattern>          # also show x lines After.
16grep -B x <pattern>          # also show x lines Before.
17grep -C x <pattern>          # show x lines of context (A + B).
18grep -a <pattern> <binary>   # search a binary file as if it were text.

        1egrep = grep -E   # for complex regular expressions.

        1# extract the 3rd field, then cut:
2cat file | grep /u01/grid/19c | awk '{print $3}' | cut -f2 -d'"'
3# is equivalent to:
4cat file | grep -o /u01/grid/19c
      • ๐Ÿ“œ Logs

        Where the system logs live

        On a Unix machine, the system logs are in /var/log/messages (or /var/adm/messages on Solaris). This is where you find the errors, with log rotation.

        Default syslog output

        LinuxSolarisHP-UXAIXBSD
        location/var/log/messages, /var/log/secure, /var/log/boot.log/var/adm/messages/var/adm/syslog/mail.log, /var/adm/syslog/syslog.log/tmp or none/var/log/syslog

        System accounting (login & process)

        TypeLinuxSolarisHP-UXAIX
        current logins/var/run/utmp/var/adm/utmpx/var/adm/utmp/etc/utmp
        login history/var/log/wtmp/var/adm/wtmpx/var/adm/wtmp/var/adm/wtmp
        process accounting/var/log/pacct/var/adm/pacct/var/adm/pacct/var/adm/pacct

        Login errors

        LinuxSolarisHP-UXAIX
        failed logins/var/log/btmp, /var/log/messages/var/adm/loginlog, /var/adm/sulog/var/adm/sulog/etc/security/failedlogin

        Investigate the logs

        1# today's logs
2grep "$(date '+%b %d')" /var/log/messages
3
4# disk errors (nawk: print the last field of the "Error Block" lines)
5nawk '/Error Block/{print $NF}' /var/adm/messages* | sort | uniq
6
7# find the IPs in a log, sort them and remove the duplicates
8cat /var/log/maillog | grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4 | uniq

        Network investigation

        1# ping a list of servers
2for ip in $(awk '/192.168.45/ {print $1}' /etc/hosts); do ping -c 1 $ip; done

        Investigate on several servers

        1for vm in vm{1..27}; do ssh -q $vm "hostname; free; sar -r 3 3"; done

        SSH authentication logs

        /var/log/auth.log โ€” SSH connection logs. Check that there are not too many failed connections (a sign of an intrusion attempt).

      • ๐Ÿšฉ Files

        Find a process blocking a file

        • with fuser:
         1fuser  -m  </dir or /files>  # Find process blocking/using this directory or files. 
 2fuser -cu  </dir or /files>  # Same as above but add the user  
 3fuser -kcu </dir or /files>  # Kill process      
 4fuser -v  -k -HUP -i ./      # Send HUP signal to process
 5			
 6# Output will send you <PID + letter>, here is the meaning:
 7#   c  current directory.
 8#   e  executable being run.
 9#   f  open file.  (omitted in default display mode).
10#   F  open file for writing. (omitted in default display mode).
11#   r  root directory.
12#   m  mmap'ed file or shared library.
        • with lsof ( = list open file):
        1lsof +D /var/log          # Find all files blocked with the process and user.
2lsof -a +L1 <mountpoint>  # Process blocking a FS.
3lsof -c ssh -c init       # Find files open by thoses processes.
4lsof -p 1753              # Find files open by PID process.
5lsof -u root              # Find files open by user.
6lsof -u ^user             # Find files open by user except this one.
7kill -9 `lsof -t -u toto` # kill user's processes.  (option -t output only PID).
        • MacGyver method:
        1#When you have no fuser or lsof: 
2find /proc/*/fd -type f -links 0 -exec ls -lrt {} \;

        AIX specifics (fuser)

        1fuser -d /tmp                 # see the processes using the /tmp directory (AIX)
2fuser -c /your_FS             # all processes with an open file in the filesystem (AIX)
3fuser -cu /dev/vg01/lvol5     # also search with a filesystem or an LV
        • -c == -m ; -u also shows the process user.
        • to kill the processes: fuser -kcu.

        File deleted but space still held

        For detecting deleted-but-still-open files (lsof +L1) and freeing the held space, see the Disk Cleanup page.

      • ๐Ÿ‘ค Users & Connections

        Investigate a user

        1last              # the last user connections to a server (based on /var/log/wtmp or btmp).
2ac -d             # statistics of my connection time per day.
3ac -p <user>      # the connection time of all users (or of a specific user).
4finger            # who is connected (-l to also see mails and plans of all users).
5w                 # who is connected, doing what, and how much CPU they use.
6who               # who is connected (-u for more info: PID, etc.).
7who am i          # with which login I am connected.
8id -a             # all info about the user I'm connected as (more precise than "who am i").
9logname           # the login name of the current account.

        Reboots & uptime

        1last reboot   # see all the reboots that took place.
2uptime        # see how long the server has been up + the load average.
3lslogins -L   # also shows whether a user shutdown/rebooted the machine.
      • ๐Ÿšฉ Compare

        Compare files

        1diff <file1> <file2>       # -w to ignore whitespace.
2colordiff <file1> <file2>  # colourised diff.
3wdiff <file1> <file2>      # word diff: [โˆ’ โˆ’] replaced word, {+ +} added word.
4vimdiff <file1> <file2>    # open both files in vim (blue = entirely different lines, red = partially different).
5fgrep -f <list> <file>     # compare two lists (e.g. of hosts).

        Compare jar files

        1diff -W200 -y  <(unzip -vqq file1.jar | awk '{ if ($1 > 0) {printf("%s\t%s\n", $1, $8)}}' | sort -k2) <(unzip -vqq  file2.jar | awk '{ if ($1 > 0) {printf("%s\t%s\n", $1, $8)}}' | sort -k2)
      • ๐Ÿ”๏ธ Investigate

        Ressources

        1# in crontab or tmux session - take every hour a track of the memory usage
2for i in {1..24} ; do echo -n "===================== " ; date ; free -m ; top -b -n1 | head -n 15 ; sleep 3600; done >> /var/log/SYSADM/memory.log &
    • Disks
      • ๐Ÿงน Disk Cleanup

        Find old files

        1find . -type f -mtime +150 -exec ls -lrt {} \; | more
2find . -maxdepth 1 -name "*.log" -mtime +10 -exec ls -lrt {} \;
3find . -mtime +150 -exec rm -f {} \;

        The find loop is cheaper than a shell loop (for ...). You can make it even more efficient by batching the rm calls:

        1find . -type f -print -exec rm -- "{}" +   # note the "+" instead of the usual "\;"

        See which directories use the most space

        1du -max .                    # list all the FS sub-directories (-x avoids filesystems other than the requested one, "." = search from where you are)
2du -sh *                     # show the total without listing the sub-directories (h = human readable)
3du -max . | sort -n | tail -30   # the 30 largest files/directories
4du -ks * | sort -n           # size in kilobytes of all files and directories, where you are
5du -hsc * | sort -h          # from smallest to largest
6ls -lrS                      # list files by size (in bytes) - note: ls -l does not give the true value contained in a directory
7du -ch /dir/                 # size of the directories contained in /dir/ (with suffix) then the total

        Reduce / Truncate a file

        1perl -e 'truncate "wanted_file", 100000'
2truncate -s 0 /ftpusers/ftp.upload.log

        File deleted but space still held by a process

        1lsof +aL1                                # "+L1" selects open files that have been "unlinked" (deleted but still open)
2lsof -nP | grep '(deleted)'
3find /proc/*/fd -type f -links 0 -exec ls -lrt {} \;   # [SunOS]

        There are two solutions:

      • ๐Ÿ“‚ Filesystem

        FS Types

        ext4 : the most widespread on GNU/Linux (derived from ext2 and ext3). It is journaled, meaning it records write operations to guarantee data integrity in case of an abrupt disk stop. It can also handle volumes up to 1 EiB (1024 PiB), and allows pre-allocating a contiguous area for a file to minimize fragmentation. Use this filesystem if you want to be able to read data back from macOS or Windows.

      • ๐Ÿงฑ ISCSI

        Install

         1yum install iscsi-initiator-utils
 2
 3#Checks
 4iscsiadm -m session -P 0  #   get the target name
 5iscsiadm -m session -P 3 | grep "Target: iqn\|Attached scsi disk\|Current Portal"
 6
 7# Discover and mount ISCSI disk 
 8iscsiadm -m discovery -t st -p 192.168.1.112
 9iscsiadm --mode discovery --type sendtargets --portal 192.168.1.112
10
11# Login
12iscsiadm -m node -T iqn.1992-04.com.emc:cx.ckm00192201413.b0 -l
13iscsiadm -m node -T iqn.1992-04.com.emc:cx.ckm00192201413.b1 -l
14iscsiadm -m node -T iqn.1992-04.com.emc:cx.ckm00192201413.a1 -l
15iscsiadm -m node -T iqn.1992-04.com.emc:cx.ckm00192201413.a0 -l
16
17# Enable/Start service 
18systemctl enable iscsid iscsi && systemctl stop iscsid iscsi && systemctl start iscsid iscsi

        Rescan BUS

        1for BUS in /sys/class/scsi_host/host*/scan; do  echo "- - -" >  ${BUS} ; done
2
3sudo sh -c 'for BUS in /sys/class/scsi_host/host*/scan; do  echo "- - -" >  ${BUS} ; done '

        • Partition your FS

      • ๐Ÿง LVM

        The Basics

        list of component:

        • PV (Physical Volume)
        • VG (Volume Group)
        • LV (Logical Volume)
        • PE (Physical Extend)
        • LE (Logical Extend)
        • FS (File Sytem)

        LVM2 use a new driver, the device-mapper allow the us of diskยดs sectors in different targets: - linear (most used in LVM). - stripped (stripped on several disks) - error (all I/O are consider in errors) - snapshot (allow snapshot async)

        • mirror (integrate elements useful for the pvmove command)
        • below example show you a striped volume and linear volume
        1lvs --all --segments -o +devices
2server_xplore_col1   vgdata -wi-ao----   21 striped   1.07t /dev/md2(40229),/dev/md3(40229),/dev/md4(40229),/dev/md5(40229),โ€ฆ
3server_xplore_col2   vgdata -wi-ao----    1 linear  219.87g /dev/md48(0)

        Basic checks

         1# Summary 
 2pvs
 3vgs
 4lvs
 5
 6# Scanner
 7pvscan
 8vgscan
 9lvscan
10
11# Details info
12pvdisplay   [sda]
13pvdisplay   -m /dev/emcpowerd1 
14vgdisplay   [vg_root]
15lvdisplay   [/dev/vg_root/lv_usr]
16
17# Summary details
18lvmdiskscan
19  /dev/sda1 [     600.00 MiB]
20  /dev/sda2 [       1.00 GiB]
21  /dev/sda3 [      38.30 GiB] LVM physical volume
22  /dev/sdb1 [    <100.00 GiB] LVM physical volume
23  /dev/sdc1 [     <50.00 GiB] LVM physical volume
24  /dev/sdj  [      20.00 GiB]
25  1 disk
26  2 partitions
27  0 LVM physical volume whole disks
28  3 LVM physical volumes

        Usual Scenario in LVM

        • Extend an existing LVM filesystem:
         1parted /dev/sda resizepart 3 100%
 2udevadm settle
 3pvresize /dev/sda3
 4
 5# Extend a XFS to a fixe size 
 6lvextend -L 30G /dev/vg00/var
 7xfs_growfs /dev/vg00/var  
 8
 9# Add some space to a ext4 FS
10lvextend -L +10G /dev/vg00/var
11resize2fs /dev/vg00/var
12
13# Extend to a pourcentage and resize automaticly whatever is the FS type.
14lvextend -l +100%FREE /dev/vg00/var -r
        • Create a new LVM filesystem:
         1parted /dev/sdb mklabel gpt mkpart primary 1 100% set 1 lvm on
 2udevadm settle
 3pvcreate /dev/sdb1
 4vgcreate vg01 /dev/sdb1
 5lvcreate -n lv_data -l 100%FREE  vg01
 6
 7# Create a XFS
 8mkfs.xfs /dev/vg01/lv_data
 9mkdir /data
10echo "/dev/mapper/vg01-lv_data   /data                  xfs     defaults        0 0" >>  /etc/fstab 
11mount -a 
12
13# Create an ext4
14mkfs.ext4 /dev/vg01/lv_data
15mkdir /data
16echo "/dev/mapper/vg01-lv_data   /data                  ext4     defaults        0 0" >>  /etc/fstab 
17mount -a
        • Remove SWAP:
         1swapoff -v /dev/dm-1
 2lvremove /dev/vg00/swap
 3vi /etc/fstab
 4vi /etc/default/grub
 5grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
 6grubby --remove-args "rd.lvm.lv=vg00/swap" --update-kernel /boot/vmlinuz-3.10.0-1160.71.1.el7.x86_64
 7grubby --remove-args "rd.lvm.lv=vg00swap" --update-kernel /boot/vmlinuz-3.10.0-1160.el7.x86_64
 8grubby --remove-args "rd.lvm.lv=vg00/swap" --update-kernel /boot/vmlinuz-0-rescue-cd2525c8417d4f798a7e6c371121ef34
 9echo "vm.swappiness = 0" >> /etc/sysctl.conf
10sysctl -p
        • Move data form disk to another:
         1# #n case of crash, just relaunch pvmove without arguments
 2pvmove /dev/emcpowerd1 /dev/emcpowerc1
 3
 4# Remove PV from a VG
 5vgreduce /dev/emcpowerd1 vg01
 6
 7# Remove all unused PV from VG01
 8vgreduce -a vg01
 9
10# remove all PV
11pvremove /dev/emcpowerd1
        • mount /var even if doesn’t want:
        1lvchange -ay --ignorelockingfailure --sysinit vgroot/var
        • Renaming:
        1# VG rename
2vgrename 
3
4# LV rename
5lvrename
6
7# PV does not need to be rename

        LVM on partition VS on Raw Disk

        Even if in the past I was using partition MS-DOS disklabel or GPT disklabel for PV, I prefer now to use directly LVM on the main block device. There is no reason to use 2 disklabels, unless you have a very specific use case (like disk with boot sector and boot partition).

      • ๐ŸŒฑ MDadm

        The Basics

        mdadm (multiple devices admin) is software solution to manage RAID.

        It allow:

        • create, manage, monitor your disks in an RAID array.
        • you can the full disks (/dev/sdb, /dev/sdc) or (/dev/sdb1, /dev/sdc1)
        • replace or complete raidtools

        Checks

        • Basic checks
        1# View real-time information about your md devices
2cat /proc/mdstat 
3
4# Monitor for failed disks (indicated by "(F)" next to the disk)
5watch cat /proc/mdstat
        • Checks RAID
        1# Display details about the RAID array (replace /dev/md0 with your array)
2mdadm --detail /dev/md0 
3
4# Examine RAID disks for information (not volume) similar to --detail
5mdadm --examine /dev/sd*

        Settings

        The conf file /etc/mdadm.conf does not exist by default and need to be created once you finish your install. This file is required for the autobuild at boot.

      • ๐Ÿฉบ multipath

        Install and Set Multipath

        1yum install device-mapper-multipath
        • Check settings in vim /etc/multipath.conf:
        1defaults {
2user_friendly_names yes
3path_grouping_policy multibus
4}
        • add disk in blacklisted and a block
        1multipaths {
2        multipath {
3                wwid "36000d310004142000000000000000f23"
4                alias oralog1
5        }
        • Special config for some providers. For example, recommended settings for all Clariion/VNX/Unity class arrays that support ALUA:
         1    devices {
 2      device {
 3        vendor "DGC"
 4        product ".*"
 5        product_blacklist "LUNZ"
 6        :
 7        path_checker emc_clariion   ### Rev 47 alua
 8        hardware_handler "1 alua"   ### modified for alua
 9        prio alua                   ### modified for alua
10        :
11      }
12    }

        • Checks config with: multipathd show config |more

      • ๐Ÿ› NFS

        The Basics

        NFS vs iscsi

        • NFS can handle simultaniously writing from several clients.
        • NFS is a filesystem , iscsi is a block storage.
        • iscsi performance are same with NFS.
        • iscsi will appear as disk to the OS, not the case for NFS.

        Concurrent access to a block device like iSCSI is not possible with standard file systems. You’ll need a shared disk filesystem (like GFS or OCSFS) to allow this, but in most cases the easiest solution would be to just use a network share (via SMB/CIFS or NFS) if this is sufficient for your application.

      • ๐Ÿ—ฟ Partition

        Checks your disks

         1# check partion 
 2parted -l /dev/sda
 3fdisk -l 
 4
 5# check partition - visible before the mkfs
 6ls /sys/sda/sda*    
 7ls /dev/sd* 
 8
 9# give partition after the mkfs or pvcreate
10blkid
11blkid -o list
12
13# summary about the disks, partitions, FS and LVM 
14lsblk   
15lsblk -f

        Create Partition 1 on disk sdb

        in script mode

        1# with fdisk 
2printf "n\np\n1\n\n\nt\n8e\nw\n" | sudo fdisk "/dev/sdb"
3
4# with parted
5sudo parted /dev/sdb mklabel gpt mkpart primary 1 100% set 1 lvm on

        Gparted : interface graphique (ce base sur parted un utilitaire GNU - Table GPT)

      • ๐ŸŽถ Samba / CIFS

        Server Side

        First Install samba and samba-client (for debug + test)

        • /etc/samba/smb.conf
        1[home]
2Workgroup=WORKGROUP (le grp par defaul sur windows)
3Hosts allow = ...
4[shared]
5browseable = yes
6path = /shared
7valid users = user01, @un_group_au_choix
8writable = yes
9passdb backend = tdbsam #passwords are stored in the /var/lib/samba/private/passdb.tdb file.

        Test samba config

        testparm

        /usr/bin/testparm -s /etc/samba/smb.conf

        smbclient -L \192.168.56.102 -U test : list all samba shares available

        smbclient //192.168.56.102/sharedrepo -U test : connect to the share

        pdbedit -L : list user smb (better than smbclient)

      • ๐Ÿงช SMART

        S.M.A.R.T. is a technology that allows you to monitor and analyze the health and performance of your hard drives. It provides valuable information about the status of your storage devices. Here are some useful commands and tips for using S.M.A.R.T. with smartctl:

        Display S.M.A.R.T. Information

        To display S.M.A.R.T. information for a specific drive, you can use the following command:

        1smartctl -a /dev/sda

        This command will show all available S.M.A.R.T. data for the /dev/sda drive.

      • ๐Ÿป SSHFS

        SSHFS

        SSHFS mounts a remote filesystem on your local filesystem through an SSH connection, all with user rights. The advantage is being able to manipulate remote data with any file manager (Nautilus, Konqueror, ROX, or even the command line).

        - Prerequisites: administrator rights, ethernet connection, installation of FUSE and the SSHFS package.
- SSHFS users must belong to the `fuse` group.

        Note: FUSE allows a user to mount a filesystem themselves. Normally, mounting a filesystem requires being an administrator, or having it pre-approved in /etc/fstab with hard-coded information.

    • Networks
      • ๐Ÿšฉ Firewalld

        Basic Troubleshooting

         1# Get the state
 2firewall-cmd --state
 3systemctl status firewalld
 4
 5# Get infos
 6firewall-cmd --get-default-zone
 7firewall-cmd --get-active-zones
 8firewall-cmd --get-zones
 9firewall-cmd --set-default-zone=home
10
11firewall-cmd --permanent --zone=FedoraWorkstation --add-source=00:FF:B0:CB:30:0A
12firewall-cmd --permanent --zone=FedoraWorkstation --add-service=ssh
13
14firewall-cmd --get-log-denied
15firewall-cmd --set-log-denied=<all, unicast, broadcast, multicast, or off>

        Add/Remove/List Services

         1#Remove
 2firewall-cmd --zone=public --add-service=ftp --permanent
 3firewall-cmd --zone=public --remove-service=ftp --permanent
 4firewall-cmd --zone=public --remove-port=53/tcp --permanent
 5firewall-cmd --zone=public --list-services
 6
 7# Add
 8firewall-cmd --zone=public --new-service=portal --permanent
 9firewall-cmd --zone=public --service=portal --add-port=8080/tcp --permanent
10firewall-cmd --zone=public --service=portal --add-port=8443/tcp --permanent
11firewall-cmd --zone=public --add-service=portal --permanent
12firewall-cmd --reload
13
14firewall-cmd --zone=public --new-service=k3s-server --permanent
15firewall-cmd --zone=public --service=k3s-server --add-port=443/tcp --permanent
16firewall-cmd --zone=public --service=k3s-server --add-port=6443/tcp --permanent
17firewall-cmd --zone=public --service=k3s-server --add-port=8472/udp --permanent
18firewall-cmd --zone=public --service=k3s-server --add-port=10250/tcp --permanent
19firewall-cmd --zone=public --add-service=k3s-server --permanent
20firewall-cmd --reload
21
22firewall-cmd --zone=public --new-service=quay --permanent
23firewall-cmd --zone=public --service=quay --add-port=8443/tcp --permanent
24firewall-cmd --zone=public --add-service=quay --permanent
25firewall-cmd --reload
26
27firewall-cmd --get-services  # It's also possible to add a service from list
28firewall-cmd --runtime-to-permanent

        Checks and Get infos

        • list open port by services
        1for s in `firewall-cmd --list-services`; do echo $s; firewall-cmd --permanent --service "$s" --get-ports; done;
2
3sudo sh -c 'for s in `firewall-cmd --list-services`; do echo $s; firewall-cmd --permanent --service "$s" --get-ports; done;'
4ssh
522/tcp
6dhcpv6-client
7546/udp
        • Check one service
        1firewall-cmd --info-service cfrm-IC
2cfrm-IC
3  ports: 7780/tcp 8440/tcp 8443/tcp
4  protocols:
5  source-ports:
6  modules:
7  destination:
        • List zones and services associated
         1firewall-cmd --list-all
 2public (active)
 3  target: default
 4  icmp-block-inversion: no
 5  interfaces: ens192
 6  sources:
 7  services: ssh dhcpv6-client https Oracle nimsoft
 8  ports: 10050/tcp 1521/tcp
 9  protocols:
10  masquerade: no
11  forward-ports:
12  source-ports:
13  icmp-blocks:
14  rich rules:

        1firewall-cmd --zone=backup --list-all
        • Get active zones
        1firewall-cmd --get-active-zones
2backup
3  interfaces: ens224
4public
5  interfaces: ens192
        • Tree folder
        1ls /etc/firewalld/
2firewalld.conf    helpers/   icmptypes/  ipsets/    lockdown-whitelist.xml  services/   zones/

        IPSET

        1firewall-cmd --get-ipset-types
2firewall-cmd --permanent --get-ipsets
3firewall-cmd --permanent --info-ipset=integration
4firewall-cmd --ipset=integration --get-entries
5
6firewall-cmd --permanent --new-ipset=test --type=hash:net
7firewall-cmd --ipset=local-blocklist --add-entry=103.133.104.0/23
      • ๐Ÿšฉ Network Manager

        Basic Troubleshooting

        • Checks interfaces
         1nmcli con show
 2NAME    UUID                                  TYPE      DEVICE
 3ens192  4d0087a0-740a-4356-8d9e-f58b63fd180c  ethernet  ens192
 4ens224  3dcb022b-62a2-4632-8b69-ab68e1901e3b  ethernet  ens224
 5
 6nmcli dev status
 7DEVICE  TYPE      STATE      CONNECTION
 8ens192  ethernet  connected  ens192
 9ens224  ethernet  connected  ens224
10ens256  ethernet  connected  ens256
11lo      loopback  unmanaged  --
12
13# Get interfaces details :
14nmcli connection show ens192 
15nmcli -p con show ens192
16
17# Get DNS settings in interface
18UUID=$(nmcli --get-values connection.uuid c show "cloud-init eth0")
19nmcli --get-values ipv4.dns c show $UUID
        • Changing Interface name
        1nmcli connection add type ethernet mac "00:50:56:80:11:ff" ifname "ens224"
2nmcli connection add type ethernet mac "00:50:56:80:8a:0b" ifname "ens256"
        • Create a custom config
        1nmcli con load /etc/sysconfig/network-scripts/ifcfg-ens224
2nmcli con up ens192
        • Adding a Virtual IP
        1nmcli con mod enp1s0 +ipv4.addresses "192.168.122.11/24"
2ip addr del 10.10.10.36/24 dev ens160
3
4nmcli con reload                     # before to reapply
5nmcli device reapply ens224
6systemctl status network.service
7systemctl restart network.service
        • Add a DNS entry
        1UUID=$(nmcli --get-values connection.uuid c show "cloud-init eth0")
2DNS_LIST=$(nmcli --get-values ipv4.dns c show $UUID)
3nmcli conn modify "$UUID" ipv4.dns  "${DNS_LIST} ${DNS_IP}"
4
5# /etc/resolved is managed by systemd-resolved
6sudo systemctl restart systemd-resolved
    • Rights
      • ๐Ÿ‘ฅ Users & Groups

        Configuration files

        FileCheck commandPurpose
        /etc/passwdpwckuser accounts
        /etc/groupgrpckgroups
        /etc/shadowโ€”password hashes and aging
        /etc/gshadowโ€”group passwords
        /etc/skelโ€”files installed by default when a user is created

        Basic commands

        1useradd -g <GID> -G <GID2> <user>   # create a user in primary group GID (and supplementary group GID2).
2usermod <options> <user>            # modify a user.
3userdel -r <user>                   # delete a user (and its home directory).
4groupadd / groupmod / groupdel      # manage groups.
5
6id -a          # show all info about the current user (UID, GUID, groups, etc.) - more precise than "who am i".
7sg <group> -c '<command>'   # execute a command as a different group ID (to run scripts or write to a file with group rights).

        Password management

        1passwd -u <user>                       # unlock a user account.
2echo "password" | passwd --stdin <user> # scripted password change.

        Account aging

        1chage -l <user>   # see the expiration dates.

        1# list the expiry of every account
2for account in $(cut -f1 -d: /etc/passwd); do
3  echo "ACCOUNT: $account , EXPIRES: $(chage -l $account | grep 'Account expires' | awk '{print $4, $5, $6}'), CHANGED: $(chage -l $account | grep 'Last password change' | awk '{print $5, $6, $7}')";
4done

        1# change the aging info interactively
2chage <user>

        To unlock an account, set “Last Password Change” to -1 in chage (or use passwd -u <user>).

      • ๐Ÿ›ก๏ธ sudo

        /etc/sudoers

        The /etc/sudoers file contains the set of UNIX operating-system privileges that the local administrator has granted to UNIX users.

        In no case should this file be edited directly with vi; it must be edited with visudo.

        1sudo          :  execute a command as root.
2sudo su       :  become root and stay root.
3sudoers       :  file listing the commands allowed for certain users as the superuser (or another user).
4visudo -cs    :  strict syntax check of the sudoers file.
        • sudo -i is equivalent to su - in terms of rights.
          • with sudo -i, the user password is asked.
          • with su -, the root password is asked.

        Verification

        1sudo -l -U <user>

        Rules

        1# "user" runs "sudo -u target All_the_commands"
2user server=(target) NOPASSWD: ALL

        With aliases

        1Host_Alias LOAD_BALANCERS = server1,server2
2
3Cmnd_Alias SET_VIP = \
4/sbin/ip addr add 192.168.10.12/20 broadcast 192.168.15.255 dev eth0 label eth0\:0, \
5/sbin/ip addr del 192.168.10.12/20 dev eth0, \
6/sbin/arping -U -c 1 -I eth0 192.168.10.12
7
8loaduser  LOAD_BALANCERS=(root) NOPASSWD: SET_VIP
9syncuser  LOAD_BALANCERS=(root) NOPASSWD: SET_VIP
      • ๐Ÿ” PAM

        /etc/pam.d

        In /etc/pam.d, there is one PAM file per service.

        Syntax: module_type control_flag path_to_module_agent

        Module types

        • auth โ€” authentication.
        • account โ€” account-based restrictions (validity, time of day, etc.).
        • session โ€” things that run at login/logout.
        • password โ€” password updates.

        Control flags

        • required โ€” success needed; a failure is reported but only after the rest of the stack has run.
        • requisite โ€” like required, but a failure returns immediately without running the rest of the stack.
        • sufficient โ€” if this module succeeds, it is the last module tested in the stack.
        • optional โ€” its result is only taken into account if no other module succeeded or failed.
        • [value=action value=action2 ...] โ€” advanced control: map a module result to a specific action.

        Sample:

      • ๐Ÿ“– LDAP & Kerberos

        Kerberos

        1kinit <user>   # obtain a ticket.
2klist          # list the tickets in the cache.

        Services

        1systemctl status slapd   # OpenLDAP server.
2systemctl status sssd    # System Security Services Daemon.

        LDAP - search

        1ldapsearch -x -h <ldap-host> -b "ou=People,dc=example,dc=com" uid=<user>

        DN components:

        • cn : common name
        • ou : organizational unit
        • o : organization
        • c : country
        • dc : domain component

        LDAP - add / modify

         1# LDIF = the commands between EOF
 2# -W prompts for the LDAP admin password
 3# -w passes the password (put it in a variable)
 4# bind_dn : the DN that acts as the LDAP bind user
 5
 6export bind_dn="CN=directory manager,DC=example,DC=org"
 7
 8# Modify an entry
 9ldapadd -W -D "$bind_dn" -h $ldap_server -p 389 <<EOT
10dn: cn=user,ou=wiki,dc=example,dc=com
11changetype: modify
12add: memberUid
13memberUid: $login
14EOT
15
16# Create a new entry
17ldapadd -w $LDAPpwd -D "$bind_dn" -h $ldap_server -p 389 <<EOT
18dn: uid=${login},ou=People,dc=example,dc=org
19uid: ${login}
20loginShell: /bin/bash
21uidNumber: ${uid}
22gidNumber: 47110
23homeDirectory: /home/${login}
24shadowLastChange: 0
25shadowMax: -1
26objectClass: account
27objectClass: posixaccount
28objectClass: shadowaccount
29objectClass: top
30gecos: ${gecos}
31cn: ${gecos}
32userPassword: {CRYPT}`perl -e 'print crypt("${login}", "${login}")'`
33EOT
    • Families
      • ๐Ÿง Unix Families

        The Unix variants

        Unix proprietaryGNU / LinuxBSD / open sourceMainframeVirtualisation
        AIXDebianFreeBSDMVS (IBM)VMware / VirtualBox
        HP-UXSlackwareNetBSDSCOS (Bull)Cloud (IaaS)
        SunOS (BSD fork) โ†’ SolarisSUSEOpenBSDOpenStack (IaaS/SaaS)
        IRIX (SGI)Red HatFreeBSD โ†’ macOS
        Fedora
        openSUSE
        CentOS
        Ubuntu (Debian)
        Mint

        The Unix families

        Historically, Unix split into two main branches:

  • Terminal

    Documentation about how to be productive with a terminal.

    • โŒจ๏ธ Bash Shortcut

      Most usefull shortcuts

      Ctrl + r : Reverse search. (ctrl+r to go back through the history).
      Ctrl + l : Clear the screen (instead of using the “clear” command).
      Ctrl + p : Repeat the last command.
      Ctrl + x + Ctrl + e : Edit the current command in an external editor (need to define export EDITOR=vim).
      Ctrl + shift + v : Copy / paste in Linux.
      Ctrl + a : Move to the beginning of the line.
      Ctrl + e : Move to the end of the line.
      Ctrl + xx : Move to the opposite end of the line.
      Ctrl + left : Move left one word.
      Ctrl + right : Move right one word.

    • ๐Ÿ–ฅ๏ธ GUI
    • ๐Ÿ“– Manual

      Manuals for commands

      man <cmd> : Open man page of command.

      • space : go ahead page by page.
      • b : go back page by page.
      • q : quit.
      • Enter : go line by line.
      • /<word> : search a word in man.
      • n : go to the next expression that you search.
      • N : go back to search expression.

      man -k <key word> : look for in all man for your key words.
      man -k <word1>.*<word2> : “.*” allow to search several words.
      whatis <cmd> : give short explaination about the command.

    • ๐Ÿ—’๏ธ Sessions

      Register your session

      Useful to keep a trace, or to document and share what has been done.

      script : save all commands and results in a “typescript” file.
      script -a : append to an existing “typescript” file (otherwise erase the previous one).
      exit : to stop the session.

      asciinema : save the terminal session as a video.

      For RHEL - something like Tlog exists and can be configured and centralised with Rsyslog.

    • ๐ŸชŸ Tmux

      Tmux

      git clone https://github.com/tmux-plugins/tmux-logging.git

      Command line

      tmux new -s my_session : Create a new session.
      tmux attach : Attach to the last used session.
      tmux attach -t X : Attach to the tmux session with number X.
      tmux ls : List active tmux sessions.
      tmux split-window -dh "!!" : Run a command in a separate pane.
      tmux source-file ~/.tmux.conf : Reload config.

      Basic Commands with key-bindings

      C-b w : List sessions/panes.
      C-b x : Close pane or session.

    • ๐Ÿ”ฃ Unicode

      Unicode with With echo

      echo $’\xae’ = “ยฎ”

      Digraphs in VIM

      Vim has a special shorthand for entering characters with diacritical marks. If you need some familiar variant of a Latin alphabet character youโ€™ll be able to input it with the digraph system.

      Digraph input is started in insert or command mode (but not normal mode) by pressing Ctrl-k, then two printable characters in succession.
      The first is often the โ€œbaseโ€ form of the letter, and the second denotes the appropriate embellishment.

    • โœ๏ธ Vim

      Tutorials

      https://vimvalley.com/ https://vim-adventures.com/ https://www.vimgolf.com/

      Plugins

      1# HCL
2mkdir -p ~/.vim/pack/jvirtanen/start
3cd ~/.vim/pack/jvirtanen/start
4git clone https://github.com/jvirtanen/vim-hcl.git
5
6# Justfile
7mkdir -p ~/.vim/pack/vendor/start
8cd ~/.vim/pack/vendor/start
9git clone https://github.com/NoahTheDuke/vim-just.git

      Fun Facts

      • trigger a vim tutorial vimtutor

      • the most powerful commands:
        . : Repeat the last modification.
        * : Where the cursor is located, keeps the word in memory and goes to the next occurrence.
        .* : together, repeat an action on the next word.

  • Red Hat

    Documentation regarding Red Hat-like specific systems.

    • ๐Ÿ†” IDM

      Server Idm - Identity Manager

       1yum install -y ipa-server ipa-server-dns
 2
 3ipa-server-install \
 4    --domain=example.com \
 5    --realm=EXAMPLE.COM \
 6    --ds-password=password \
 7    --admin-password=password \
 8    --hostname=classroom.example.com \
 9    --ip-address=172.25.0.254 \
10    --reverse-zone=0.25.172.in-addr.arpa. \
11    --forwarder=208.67.222.222 \
12    --allow-zone-overlap \
13    --setup-dns \
14    --unattended

      1yum install -y ipa-client 
2
3ipa-client-install --mkhomedir --enable-dns-updates --force-ntpd -p admin@EXAMPLE.COM --password='password' --force-join -U
4
5# Test login
6echo -n 'password' | kinit admin

      Script if DNS config is right for a IDM server

       1sudo sh -c "cat <<EOF > ~/IdmZoneCheck.sh
 2#!/bin/bash
 3### IdM zone check ###
 4# Check if the zone name is provided as a parameter #
 5if [ -z "$1" ];
 6then
 7        echo -e "Provide the zone name to be checked as a parameter!\n(ex: IdmZoneCheck.sh domain.local)"
 8        exit
 9fi
10clear
11echo -e "### IDM / TCP ###\n\n"
12echo -e "TCP / kerberos-master (SRV)"
13dig +short _kerberos-master._tcp.$1. SRV
14echo -e "_TCP / kerberos (SRV)"
15dig +short _kerberos._tcp.$1. SRV
16echo -e "_TCP / kpasswd (SRV)"
17dig +short _kpasswd._tcp.$1. SRV
18echo -e "_TCP / ldap (SRV)"
19dig +short _ldap._tcp.$1. SRV
20echo -e "\n### IDM / UDP ###\n\n"
21echo -e "_UDP / kerberos-master (SRV)"
22dig +short _kerberos-master._udp.$1. SRV
23echo -e "_UDP / kerberos (SRV)"
24dig +short _kerberos._udp.$1. SRV
25echo -e "_UCP / kpasswd (SRV)"
26dig +short _kpasswd._udp.$1. SRV
27echo -e "\n### IDM / MSDCS DC TCP ###\n\n"
28echo -e "_MSDCS / TCP / kerberos (SRV)"
29dig +short _kerberos._tcp.dc._msdcs.$1. SRV
30echo -e "_MSDCS / TCP / ldap (SRV)"
31dig +short _ldap._tcp.dc._msdcs.$1. SRV
32echo -e "\n### IDM / MSDCS DC UDP ###\n\n"
33echo -e "_MSDCS / UDP / kerberos (SRV)"
34dig +short _kerberos._udp.dc._msdcs.$1. SRV
35echo -e "\n### IDM / REALM ###\n\n"
36echo -e "REALM (TXT)"
37dig +short _kerberos.$1. TXT
38echo -e "\n### IDM / CA ###\n\n"
39echo -e "A / ipa-ca"
40dig +short ipa-ca.$1. A
41echo -e "\n### IDM / A ###\n\n"
42echo -e "A / $HOSTNAME"
43dig +short $HOSTNAME. A
44EOF
      • Script usage :
      1./IdmZoneCheck.sh idm.example.com
    • ๐Ÿš€ KickStart

      KickStart

      Technology that allows deploying servers with a predefined configuration (Red Hat’s equivalent of Solaris JumpStart).

      • Default volume manager : LVM. See the LVM page for details.
    • ๐Ÿ›ฐ๏ธ Satellite

      Satellite - Repository

    • ๐Ÿ”‘ sssd

      Troubleshooting

      1sudo realm list
2authselect current
3sssctl domain-list
4sssctl config-check
5getent -s files passwd
6getent -s sss   passwd user
7getent          passwd
8dig -t SRV _ldap._tcp.example.com
9sssctl user-checks toto -s sshd -a auth

      Prerequisites :

      • Need port 369 and 3268

      for RHEL8 :

      1dnf -y install realmd adcli sssd oddjob oddjob-mkhomedir samba-common-tools krb5-workstation authselect-compat
2
3realm discover example.com
4realm join example.com -U svc-sssd --client-software=sssd --os-name=RedHat --os-version=8 
5
6sudo authselect select sssd with-mkhomedir
7sudo systemctl enable --now oddjobd.service
      • inside /etc/sssd/sssd.conf
       1[sssd]
 2services = nss, pam, ssh, sudo
 3domains = example.com
 4config_file_version = 2
 5default_domain_suffix = example.com
 6
 7[domain/example.com]
 8default_shell = /bin/bash
 9override_shell = /bin/bash
10
11ad_domain = example.com
12krb5_realm = example.com
13realmd_tags = manages-system joined-with-adcli
14cache_credentials = True
15id_provider = ad
16krb5_store_password_if_offline = True
17ldap_id_mapping = True
18ldap_user_objectsid = objectSid
19ldap_group_objectsid = objectSid
20ldap_user_primary_group = primaryGroupID
21
22use_fully_qualified_names = True
23fallback_homedir = /home/%u
24
25access_provider = ad
26ldap_access_order=filter,expire
27ldap_account_expire_policy = ad
28ad_access_filter =  (memberOf=CN=INTERNAL Team,OU=team-platform,OU=test-groups,DC=example,DC=com)
29
30
31[nss]
32homedir_substring = /home
33
34[pam]
35pam_pwd_expiration_warning = 7
36pam_account_expired_message = Account expired, please contact AD administrator.
37pam_account_locked_message = Account locked, please contact AD administrator.
38pam_verbosity = 3
39
40[ssh]
41
42[sudo]
      • Reload config:
      1sss_cache -E; systemctl restart sssd ; sss_cache -E
2systemctl status sssd
      • define sudoers rights /etc/sudoers.d/admin :
      1%EXAMPLE.COM\\internal\ team ALL=(ALL) ALL
      • reload sudoers rights:
      1realm permit -g 'internal team@example.com'
  • Windows
    Sunday, October 4, 2026 Monday, January 1, 1
    Networks