πŸ—Ώ Partition
πŸ—Ώ Partition
Checks your disks 1# check partion 2parted -l /dev/sda 3fdisk -l 4 5# check partition - visible before the mkfs 6ls /sys/sda/sda* 7ls /dev/sd* 8 9# give partition after the mkfs or pvcreate 10blkid 11blkid -o list 12 13# summary about the disks, partitions, FS and LVM 14lsblk 15lsblk -f Create Partition 1 on disk sdb in script mode 1# with fdisk 2printf "n\np\n1\n\n\nt\n8e\nw\n" | sudo fdisk "/dev/sdb" 3 4# with parted 5sudo parted /dev/sdb mklabel gpt mkpart primary 1 100% set 1 lvm on Gparted : interface graphique (ce base sur parted un utilitaire GNU - Table GPT)
🩺 multipath
🩺 multipath
Install and Set Multipath 1yum install device-mapper-multipath Check settings in vim /etc/multipath.conf: 1defaults { 2user_friendly_names yes 3path_grouping_policy multibus 4} add disk in blacklisted and a block 1multipaths { 2 multipath { 3 wwid "36000d310004142000000000000000f23" 4 alias oralog1 5 } Special config for some providers. For example, recommended settings for all Clariion/VNX/Unity class arrays that support ALUA: 1 devices { 2 device { 3 vendor "DGC" 4 product ".*" 5 product_blacklist "LUNZ" 6 : 7 path_checker emc_clariion ### Rev 47 alua 8 hardware_handler "1 alua" ### modified for alua 9 prio alua ### modified for alua 10 : 11 } 12 } Checks config with: multipathd show config |more
🧱 ISCSI
🧱 ISCSI
Install 1yum install iscsi-initiator-utils 2 3#Checks 4iscsiadm -m session -P 0 # get the target name 5iscsiadm -m session -P 3 | grep "Target: iqn\|Attached scsi disk\|Current Portal" 6 7# Discover and mount ISCSI disk 8iscsiadm -m discovery -t st -p 192.168.1.112 9iscsiadm --mode discovery --type sendtargets --portal 192.168.1.112 10 11# Login 12iscsiadm -m node -T iqn.1992-04.com.emc:cx.ckm00192201413.b0 -l 13iscsiadm -m node -T iqn.1992-04.com.emc:cx.ckm00192201413.b1 -l 14iscsiadm -m node -T iqn.1992-04.com.emc:cx.ckm00192201413.a1 -l 15iscsiadm -m node -T iqn.1992-04.com.emc:cx.ckm00192201413.a0 -l 16 17# Enable/Start service 18systemctl enable iscsid iscsi && systemctl stop iscsid iscsi && systemctl start iscsid iscsi Rescan BUS 1for BUS in /sys/class/scsi_host/host*/scan; do echo "- - -" > ${BUS} ; done 2 3sudo sh -c 'for BUS in /sys/class/scsi_host/host*/scan; do echo "- - -" > ${BUS} ; done ' Partition your FS
πŸ§ͺ SMART
πŸ§ͺ SMART
S.M.A.R.T. is a technology that allows you to monitor and analyze the health and performance of your hard drives. It provides valuable information about the status of your storage devices. Here are some useful commands and tips for using S.M.A.R.T. with smartctl: Display S.M.A.R.T. Information To display S.M.A.R.T. information for a specific drive, you can use the following command: 1smartctl -a /dev/sda This command will show all available S.M.A.R.T. data for the /dev/sda drive.
πŸ“‚ Filesystem
πŸ“‚ Filesystem
FS Types ext4 : the most widespread on GNU/Linux (derived from ext2 and ext3). It is journaled, meaning it records write operations to guarantee data integrity in case of an abrupt disk stop. It can also handle volumes up to 1 EiB (1024 PiB), and allows pre-allocating a contiguous area for a file to minimize fragmentation. Use this filesystem if you want to be able to read data back from macOS or Windows.
🌱 MDadm
🌱 MDadm
The Basics mdadm (multiple devices admin) is software solution to manage RAID. It allow: create, manage, monitor your disks in an RAID array. you can the full disks (/dev/sdb, /dev/sdc) or (/dev/sdb1, /dev/sdc1) replace or complete raidtools Checks Basic checks 1# View real-time information about your md devices 2cat /proc/mdstat 3 4# Monitor for failed disks (indicated by "(F)" next to the disk) 5watch cat /proc/mdstat Checks RAID 1# Display details about the RAID array (replace /dev/md0 with your array) 2mdadm --detail /dev/md0 3 4# Examine RAID disks for information (not volume) similar to --detail 5mdadm --examine /dev/sd* Settings The conf file /etc/mdadm.conf does not exist by default and need to be created once you finish your install. This file is required for the autobuild at boot.
🧐 LVM
🧐 LVM
The Basics list of component: PV (Physical Volume) VG (Volume Group) LV (Logical Volume) PE (Physical Extend) LE (Logical Extend) FS (File Sytem) LVM2 use a new driver, the device-mapper allow the us of diskΒ΄s sectors in different targets: - linear (most used in LVM). - stripped (stripped on several disks) - error (all I/O are consider in errors) - snapshot (allow snapshot async) mirror (integrate elements useful for the pvmove command) below example show you a striped volume and linear volume 1lvs --all --segments -o +devices 2server_xplore_col1 vgdata -wi-ao---- 21 striped 1.07t /dev/md2(40229),/dev/md3(40229),/dev/md4(40229),/dev/md5(40229),… 3server_xplore_col2 vgdata -wi-ao---- 1 linear 219.87g /dev/md48(0) Basic checks 1# Summary 2pvs 3vgs 4lvs 5 6# Scanner 7pvscan 8vgscan 9lvscan 10 11# Details info 12pvdisplay [sda] 13pvdisplay -m /dev/emcpowerd1 14vgdisplay [vg_root] 15lvdisplay [/dev/vg_root/lv_usr] 16 17# Summary details 18lvmdiskscan 19 /dev/sda1 [ 600.00 MiB] 20 /dev/sda2 [ 1.00 GiB] 21 /dev/sda3 [ 38.30 GiB] LVM physical volume 22 /dev/sdb1 [ <100.00 GiB] LVM physical volume 23 /dev/sdc1 [ <50.00 GiB] LVM physical volume 24 /dev/sdj [ 20.00 GiB] 25 1 disk 26 2 partitions 27 0 LVM physical volume whole disks 28 3 LVM physical volumes Usual Scenario in LVM Extend an existing LVM filesystem: 1parted /dev/sda resizepart 3 100% 2udevadm settle 3pvresize /dev/sda3 4 5# Extend a XFS to a fixe size 6lvextend -L 30G /dev/vg00/var 7xfs_growfs /dev/vg00/var 8 9# Add some space to a ext4 FS 10lvextend -L +10G /dev/vg00/var 11resize2fs /dev/vg00/var 12 13# Extend to a pourcentage and resize automaticly whatever is the FS type. 14lvextend -l +100%FREE /dev/vg00/var -r Create a new LVM filesystem: 1parted /dev/sdb mklabel gpt mkpart primary 1 100% set 1 lvm on 2udevadm settle 3pvcreate /dev/sdb1 4vgcreate vg01 /dev/sdb1 5lvcreate -n lv_data -l 100%FREE vg01 6 7# Create a XFS 8mkfs.xfs /dev/vg01/lv_data 9mkdir /data 10echo "/dev/mapper/vg01-lv_data /data xfs defaults 0 0" >> /etc/fstab 11mount -a 12 13# Create an ext4 14mkfs.ext4 /dev/vg01/lv_data 15mkdir /data 16echo "/dev/mapper/vg01-lv_data /data ext4 defaults 0 0" >> /etc/fstab 17mount -a Remove SWAP: 1swapoff -v /dev/dm-1 2lvremove /dev/vg00/swap 3vi /etc/fstab 4vi /etc/default/grub 5grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg 6grubby --remove-args "rd.lvm.lv=vg00/swap" --update-kernel /boot/vmlinuz-3.10.0-1160.71.1.el7.x86_64 7grubby --remove-args "rd.lvm.lv=vg00swap" --update-kernel /boot/vmlinuz-3.10.0-1160.el7.x86_64 8grubby --remove-args "rd.lvm.lv=vg00/swap" --update-kernel /boot/vmlinuz-0-rescue-cd2525c8417d4f798a7e6c371121ef34 9echo "vm.swappiness = 0" >> /etc/sysctl.conf 10sysctl -p Move data form disk to another: 1# #n case of crash, just relaunch pvmove without arguments 2pvmove /dev/emcpowerd1 /dev/emcpowerc1 3 4# Remove PV from a VG 5vgreduce /dev/emcpowerd1 vg01 6 7# Remove all unused PV from VG01 8vgreduce -a vg01 9 10# remove all PV 11pvremove /dev/emcpowerd1 mount /var even if doesn’t want: 1lvchange -ay --ignorelockingfailure --sysinit vgroot/var Renaming: 1# VG rename 2vgrename 3 4# LV rename 5lvrename 6 7# PV does not need to be rename LVM on partition VS on Raw Disk Even if in the past I was using partition MS-DOS disklabel or GPT disklabel for PV, I prefer now to use directly LVM on the main block device. There is no reason to use 2 disklabels, unless you have a very specific use case (like disk with boot sector and boot partition).
🚩 Network Manager
🚩 Network Manager
Basic Troubleshooting Checks interfaces 1nmcli con show 2NAME UUID TYPE DEVICE 3ens192 4d0087a0-740a-4356-8d9e-f58b63fd180c ethernet ens192 4ens224 3dcb022b-62a2-4632-8b69-ab68e1901e3b ethernet ens224 5 6nmcli dev status 7DEVICE TYPE STATE CONNECTION 8ens192 ethernet connected ens192 9ens224 ethernet connected ens224 10ens256 ethernet connected ens256 11lo loopback unmanaged -- 12 13# Get interfaces details : 14nmcli connection show ens192 15nmcli -p con show ens192 16 17# Get DNS settings in interface 18UUID=$(nmcli --get-values connection.uuid c show "cloud-init eth0") 19nmcli --get-values ipv4.dns c show $UUID Changing Interface name 1nmcli connection add type ethernet mac "00:50:56:80:11:ff" ifname "ens224" 2nmcli connection add type ethernet mac "00:50:56:80:8a:0b" ifname "ens256" Create a custom config 1nmcli con load /etc/sysconfig/network-scripts/ifcfg-ens224 2nmcli con up ens192 Adding a Virtual IP 1nmcli con mod enp1s0 +ipv4.addresses "192.168.122.11/24" 2ip addr del 10.10.10.36/24 dev ens160 3 4nmcli con reload # before to reapply 5nmcli device reapply ens224 6systemctl status network.service 7systemctl restart network.service Add a DNS entry 1UUID=$(nmcli --get-values connection.uuid c show "cloud-init eth0") 2DNS_LIST=$(nmcli --get-values ipv4.dns c show $UUID) 3nmcli conn modify "$UUID" ipv4.dns "${DNS_LIST} ${DNS_IP}" 4 5# /etc/resolved is managed by systemd-resolved 6sudo systemctl restart systemd-resolved
🚩 Files
🚩 Files
Find a process blocking a file with fuser: 1fuser -m </dir or /files> # Find process blocking/using this directory or files. 2fuser -cu </dir or /files> # Same as above but add the user 3fuser -kcu </dir or /files> # Kill process 4fuser -v -k -HUP -i ./ # Send HUP signal to process 5 6# Output will send you <PID + letter>, here is the meaning: 7# c current directory. 8# e executable being run. 9# f open file. (omitted in default display mode). 10# F open file for writing. (omitted in default display mode). 11# r root directory. 12# m mmap'ed file or shared library. with lsof ( = list open file): 1lsof +D /var/log # Find all files blocked with the process and user. 2lsof -a +L1 <mountpoint> # Process blocking a FS. 3lsof -c ssh -c init # Find files open by thoses processes. 4lsof -p 1753 # Find files open by PID process. 5lsof -u root # Find files open by user. 6lsof -u ^user # Find files open by user except this one. 7kill -9 `lsof -t -u toto` # kill user's processes. (option -t output only PID). MacGyver method: 1#When you have no fuser or lsof: 2find /proc/*/fd -type f -links 0 -exec ls -lrt {} \; AIX specifics (fuser) 1fuser -d /tmp # see the processes using the /tmp directory (AIX) 2fuser -c /your_FS # all processes with an open file in the filesystem (AIX) 3fuser -cu /dev/vg01/lvol5 # also search with a filesystem or an LV -c == -m ; -u also shows the process user. to kill the processes: fuser -kcu. File deleted but space still held For detecting deleted-but-still-open files (lsof +L1) and freeing the held space, see the Disk Cleanup page.
🚩 Compare
🚩 Compare
Compare files 1diff <file1> <file2> # -w to ignore whitespace. 2colordiff <file1> <file2> # colourised diff. 3wdiff <file1> <file2> # word diff: [βˆ’ βˆ’] replaced word, {+ +} added word. 4vimdiff <file1> <file2> # open both files in vim (blue = entirely different lines, red = partially different). 5fgrep -f <list> <file> # compare two lists (e.g. of hosts). Compare jar files 1diff -W200 -y <(unzip -vqq file1.jar | awk '{ if ($1 > 0) {printf("%s\t%s\n", $1, $8)}}' | sort -k2) <(unzip -vqq file2.jar | awk '{ if ($1 > 0) {printf("%s\t%s\n", $1, $8)}}' | sort -k2)
πŸ”οΈ Investigate
πŸ”οΈ Investigate
Ressources 1# in crontab or tmux session - take every hour a track of the memory usage 2for i in {1..24} ; do echo -n "===================== " ; date ; free -m ; top -b -n1 | head -n 15 ; sleep 3600; done >> /var/log/SYSADM/memory.log &
πŸ› NFS
πŸ› NFS
The Basics NFS vs iscsi NFS can handle simultaniously writing from several clients. NFS is a filesystem , iscsi is a block storage. iscsi performance are same with NFS. iscsi will appear as disk to the OS, not the case for NFS. Concurrent access to a block device like iSCSI is not possible with standard file systems. You’ll need a shared disk filesystem (like GFS or OCSFS) to allow this, but in most cases the easiest solution would be to just use a network share (via SMB/CIFS or NFS) if this is sufficient for your application.
🎢 Samba / CIFS
🎢 Samba / CIFS
Server Side First Install samba and samba-client (for debug + test) /etc/samba/smb.conf 1[home] 2Workgroup=WORKGROUP (le grp par defaul sur windows) 3Hosts allow = ... 4[shared] 5browseable = yes 6path = /shared 7valid users = user01, @un_group_au_choix 8writable = yes 9passdb backend = tdbsam #passwords are stored in the /var/lib/samba/private/passdb.tdb file. Test samba config testparm /usr/bin/testparm -s /etc/samba/smb.conf smbclient -L \192.168.56.102 -U test : list all samba shares available smbclient //192.168.56.102/sharedrepo -U test : connect to the share pdbedit -L : list user smb (better than smbclient)
🍻 SSHFS
🍻 SSHFS
SSHFS SSHFS mounts a remote filesystem on your local filesystem through an SSH connection, all with user rights. The advantage is being able to manipulate remote data with any file manager (Nautilus, Konqueror, ROX, or even the command line). - Prerequisites: administrator rights, ethernet connection, installation of FUSE and the SSHFS package. - SSHFS users must belong to the `fuse` group. Note: FUSE allows a user to mount a filesystem themselves. Normally, mounting a filesystem requires being an administrator, or having it pre-approved in /etc/fstab with hard-coded information.
πŸ”Ž Search, Find & Compare
πŸ”Ž Search, Find & Compare
Find files quickly 1locate <pattern> # find a directory or file quickly (uses an index); a brand-new file won't be found. 2updatedb # update the locate index. Open a file 1view <file> # opens a read-only vi view (preferred if you just want to search/view). 2gzcat / zcat <file.gz> # read a gzipped file. Info on a file or directory 1stat </my/file> # all info about a file (inode, creation, modification, access dates, etc.). 2stat -f <FS> # info about a filesystem. 3stat -c%s $LOGFILE # [scripting] get a precise value (size, modification date, etc.). grep 1grep -w 'xyz' # match the whole word. 2grep -x 'Hello, world!' # the whole line must match. 3grep -c <pattern> # count the matching lines. 4grep -l "ERROR:" *.log # search all .log files, list the files that match. 5grep -L <pattern> # inverse: list the files that do NOT match. 6grep -f <patternfile> <file> # apply the patterns read from patternfile. 7grep -i <pattern> # ignore case. 8grep -v <pattern> # return the lines that do NOT match. 9grep -m x <pattern> # stop after x matching lines. 10grep -n <pattern> # show the line number. 11grep -q <pattern> # quiet: exit 0 if found, 1 (or 2) otherwise (for scripting). 12grep -s <pattern> # suppress permission/inexistent-file error messages. 13grep -H <pattern> # show the filename next to each matching line. 14grep -h <pattern> # do not show the filename (default behaviour). 15grep -A x <pattern> # also show x lines After. 16grep -B x <pattern> # also show x lines Before. 17grep -C x <pattern> # show x lines of context (A + B). 18grep -a <pattern> <binary> # search a binary file as if it were text. 1egrep = grep -E # for complex regular expressions. 1# extract the 3rd field, then cut: 2cat file | grep /u01/grid/19c | awk '{print $3}' | cut -f2 -d'"' 3# is equivalent to: 4cat file | grep -o /u01/grid/19c
πŸ“œ Logs
πŸ“œ Logs
Where the system logs live On a Unix machine, the system logs are in /var/log/messages (or /var/adm/messages on Solaris). This is where you find the errors, with log rotation. Default syslog output Linux Solaris HP-UX AIX BSD location /var/log/messages, /var/log/secure, /var/log/boot.log /var/adm/messages /var/adm/syslog/mail.log, /var/adm/syslog/syslog.log /tmp or none /var/log/syslog System accounting (login & process) Type Linux Solaris HP-UX AIX current logins /var/run/utmp /var/adm/utmpx /var/adm/utmp /etc/utmp login history /var/log/wtmp /var/adm/wtmpx /var/adm/wtmp /var/adm/wtmp process accounting /var/log/pacct /var/adm/pacct /var/adm/pacct /var/adm/pacct Login errors Linux Solaris HP-UX AIX failed logins /var/log/btmp, /var/log/messages /var/adm/loginlog, /var/adm/sulog /var/adm/sulog /etc/security/failedlogin Investigate the logs 1# today's logs 2grep "$(date '+%b %d')" /var/log/messages 3 4# disk errors (nawk: print the last field of the "Error Block" lines) 5nawk '/Error Block/{print $NF}' /var/adm/messages* | sort | uniq 6 7# find the IPs in a log, sort them and remove the duplicates 8cat /var/log/maillog | grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4 | uniq Network investigation 1# ping a list of servers 2for ip in $(awk '/192.168.45/ {print $1}' /etc/hosts); do ping -c 1 $ip; done Investigate on several servers 1for vm in vm{1..27}; do ssh -q $vm "hostname; free; sar -r 3 3"; done SSH authentication logs /var/log/auth.log β€” SSH connection logs. Check that there are not too many failed connections (a sign of an intrusion attempt).
πŸ‘€ Users & Connections
πŸ‘€ Users & Connections
Investigate a user 1last # the last user connections to a server (based on /var/log/wtmp or btmp). 2ac -d # statistics of my connection time per day. 3ac -p <user> # the connection time of all users (or of a specific user). 4finger # who is connected (-l to also see mails and plans of all users). 5w # who is connected, doing what, and how much CPU they use. 6who # who is connected (-u for more info: PID, etc.). 7who am i # with which login I am connected. 8id -a # all info about the user I'm connected as (more precise than "who am i"). 9logname # the login name of the current account. Reboots & uptime 1last reboot # see all the reboots that took place. 2uptime # see how long the server has been up + the load average. 3lslogins -L # also shows whether a user shutdown/rebooted the machine.
πŸ• NTP & Time Synchronisation
πŸ• NTP & Time Synchronisation
Client verification (ntpd / chrony) 1ntpstat # see which NTP server we synchronise with, and whether the sync is good. 1synchronised to NTP server (192.168.1.12) at stratum 4 2 time correct to within 68 ms 3 polling server every 1024 s 1ntpq -p # see the state of the peers. 2ntpq -c peers remote refid st t when poll reach delay offset jitter ============================================================================== +192.168.1.11 192.168.2.4 4 u 259 1024 373 0.731 -0.980 0.557 *192.168.1.12 192.168.3.21 3 u 385 1024 377 0.773 0.146 0.365 192.168.4.255 .BCST. 16 u - 64 0 0.000 0.000 0.000 The server preceded by an asterisk (*) is the one being used. Those preceded by a - are currently discarded by the server-selection algorithm. Those whose name is preceded by a + are possible synchronisation candidates. A server preceded by a space is either unreachable or too distant. Column meaning remote β€” the server name. refid β€” the parent server’s identifier. st β€” the server’s stratum. t β€” the server type. when β€” seconds elapsed since the last contact. poll β€” seconds between each contact. reach β€” bitmask of successful contacts (octal): the server considers itself synchronised when reach reaches 177; a quality, stable connection shows 377. delay β€” estimated round-trip time (ms) of the UDP packet. offset β€” estimated difference between the peer’s clock and the internal clock. jitter β€” dispersion of the reference values obtained from this peer. Restart the NTP daemon 1service ntpd restart # or: systemctl restart ntpd Configuration & logs 1cat /etc/ntp.conf # "server example.com" + restart ntpd + enable 2/var/log/ntpstats ntpdate (legacy) Old service that synchronises NTP at boot (install the package first).
πŸ” Runlevels & Shutdown
πŸ” Runlevels & Shutdown
Shutdown / reboot Solaris Red Hat Ubuntu / Debian HP-UX AIX Power down shutdown -i5 -g0 -y shutdown -h shutdown -h shutdown -h now shutdown -F Reboot shutdown -i6 -g0 -y shutdown -r shutdown -r shutdown -r now shutdown -Fr OK prompt shutdown -i0 -g0 -y β€” β€” β€” β€” Fast reboot -- -r (reconfigure) shutdown -f (no fsck) shutdown -P (power off) shutdown -F (force fsck) β€” Force fsck touch /reconfigure touch /forcefsck edit /etc/default/rcS β†’ FSCKFIX=yes β€” β€” Change runlevel Tool Solaris Red Hat Ubuntu / Debian HP-UX AIX halt βœ… βœ… βœ… βœ… βœ… init βœ… βœ… βœ… βœ… βœ… poweroff βœ… βœ… βœ… βœ… βœ… reboot βœ… βœ… βœ… βœ… βœ… shutdown βœ… βœ… βœ… βœ… βœ… telinit βœ… βœ… βœ… β€” βœ… uadmin βœ… β€” β€” β€” β€” Runlevels Level Solaris Red Hat Ubuntu / Debian HP-UX AIX 0 shutdown halt halt halt reserved 1 single user single user single user single user reserved 2 n/a multiuser (no networking) multiuser (default) multiuser (networking) multiuser + NFS 3 multi-user multiuser (networking) same as 2 multiuser + NFS + CDE GUI (default) user defined 4 n/a unused same as 2 multiuser + NFS + VUE GUI user defined 5 power off GUI same as 2 n/a user defined 6 reboot reboot reboot n/a user defined 7-9 β€” β€” β€” β€” user defined Change the default runlevel Solaris / Red Hat / HP-UX / AIX : edit the initdefault line in vi /etc/inittab. Ubuntu / Debian : edit vi /etc/event.d/rc-default. On systemd systems (RHEL 7+, Ubuntu 15+), the SysV runlevels are replaced by targets β€” e.g. systemctl isolate multi-user.target (runlevel 3), systemctl isolate graphical.target (runlevel 5), systemctl set-default multi-user.target. See the Systemd page.
πŸ“¦ Chroot Jail
πŸ“¦ Chroot Jail
Change the root directory of a command or a process, and its children. In no case should `chroot` be relied upon as a security boundary β€” a process running as root can escape the jail. Example: creating a chroot 1# create the "jail" directory 2J=$HOME/jail 3mkdir -p $J 4mkdir -p $J/{bin,lib64,lib} 5cd $J 6 7# copy the binaries and their libraries into the jail 8cp -v /bin/{bash,ls} $J/bin 9 10list="$(ldd /bin/bash | egrep -o '/lib.*\.[0-9]')" 11for i in $list; do cp -v "$i" "${J}${i}"; done 12 13list="$(ldd /bin/ls | egrep -o '/lib.*\.[0-9]')" 14for i in $list; do cp -v "$i" "${J}${i}"; done 15 16# enter the jail 17sudo chroot $J /bin/bash
πŸ“œ Logrotate
πŸ“œ Logrotate
1logrotate -d /etc/logrotate.d/app # test a new configuration. 2reading config info for /data/log/app 3Handling 1 logs 4rotating pattern: /data/log/app after 1 days (10 rotations) 5empty log files are rotated, old logs are removed Options Usage: logrotate [OPTION...] <configfile> -d, --debug Don't do anything, just test (implies -v) -f, --force Force file rotation -m, --mail=command Command to send mail (instead of `/bin/mail') -s, --state=statefile Path of state file -v, --verbose Display messages during rotation
⏲️ Cron & Anacron
⏲️ Cron & Anacron
Configurations /etc/crontab : the daemon’s configuration file, defining the default behaviour of crond (SHELL, MAILTO, etc.). /etc/cron.d/... : system crontab (used by admins). /var/spool/cron/root : crontab per user. “Day of month” and “Day of week” are combined with a logical OR, so if both are set, the job runs on that day of the month and on that day of the week. 1MAILTO="admin@example.com" 2* * * * * root /usr/local/sbin/mycommand.sh > /dev/null 2>&1 Anacron /etc/anacrontab : file that runs, via the run-parts command, /etc/cron.daily, /etc/cron.weekly, /etc/cron.monthly. /etc/cron.d/0hourly : exception, runs /etc/cron.hourly via run-parts, checking the last run of the task in /var/spool/anacron/.... Special cases Exactly the last day of each month:
πŸ›‘οΈ sudo
πŸ›‘οΈ sudo
/etc/sudoers The /etc/sudoers file contains the set of UNIX operating-system privileges that the local administrator has granted to UNIX users. In no case should this file be edited directly with vi; it must be edited with visudo. 1sudo : execute a command as root. 2sudo su : become root and stay root. 3sudoers : file listing the commands allowed for certain users as the superuser (or another user). 4visudo -cs : strict syntax check of the sudoers file. sudo -i is equivalent to su - in terms of rights. with sudo -i, the user password is asked. with su -, the root password is asked. Verification 1sudo -l -U <user> Rules 1# "user" runs "sudo -u target All_the_commands" 2user server=(target) NOPASSWD: ALL With aliases 1Host_Alias LOAD_BALANCERS = server1,server2 2 3Cmnd_Alias SET_VIP = \ 4/sbin/ip addr add 192.168.10.12/20 broadcast 192.168.15.255 dev eth0 label eth0\:0, \ 5/sbin/ip addr del 192.168.10.12/20 dev eth0, \ 6/sbin/arping -U -c 1 -I eth0 192.168.10.12 7 8loaduser LOAD_BALANCERS=(root) NOPASSWD: SET_VIP 9syncuser LOAD_BALANCERS=(root) NOPASSWD: SET_VIP
πŸ” PAM
πŸ” PAM
/etc/pam.d In /etc/pam.d, there is one PAM file per service. Syntax: module_type control_flag path_to_module_agent Module types auth β€” authentication. account β€” account-based restrictions (validity, time of day, etc.). session β€” things that run at login/logout. password β€” password updates. Control flags required β€” success needed; a failure is reported but only after the rest of the stack has run. requisite β€” like required, but a failure returns immediately without running the rest of the stack. sufficient β€” if this module succeeds, it is the last module tested in the stack. optional β€” its result is only taken into account if no other module succeeded or failed. [value=action value=action2 ...] β€” advanced control: map a module result to a specific action. Sample:
πŸ“– LDAP & Kerberos
πŸ“– LDAP & Kerberos
Kerberos 1kinit <user> # obtain a ticket. 2klist # list the tickets in the cache. Services 1systemctl status slapd # OpenLDAP server. 2systemctl status sssd # System Security Services Daemon. LDAP - search 1ldapsearch -x -h <ldap-host> -b "ou=People,dc=example,dc=com" uid=<user> DN components: cn : common name ou : organizational unit o : organization c : country dc : domain component LDAP - add / modify 1# LDIF = the commands between EOF 2# -W prompts for the LDAP admin password 3# -w passes the password (put it in a variable) 4# bind_dn : the DN that acts as the LDAP bind user 5 6export bind_dn="CN=directory manager,DC=example,DC=org" 7 8# Modify an entry 9ldapadd -W -D "$bind_dn" -h $ldap_server -p 389 <<EOT 10dn: cn=user,ou=wiki,dc=example,dc=com 11changetype: modify 12add: memberUid 13memberUid: $login 14EOT 15 16# Create a new entry 17ldapadd -w $LDAPpwd -D "$bind_dn" -h $ldap_server -p 389 <<EOT 18dn: uid=${login},ou=People,dc=example,dc=org 19uid: ${login} 20loginShell: /bin/bash 21uidNumber: ${uid} 22gidNumber: 47110 23homeDirectory: /home/${login} 24shadowLastChange: 0 25shadowMax: -1 26objectClass: account 27objectClass: posixaccount 28objectClass: shadowaccount 29objectClass: top 30gecos: ${gecos} 31cn: ${gecos} 32userPassword: {CRYPT}`perl -e 'print crypt("${login}", "${login}")'` 33EOT
πŸ‘₯ Users & Groups
πŸ‘₯ Users & Groups
Configuration files File Check command Purpose /etc/passwd pwck user accounts /etc/group grpck groups /etc/shadow β€” password hashes and aging /etc/gshadow β€” group passwords /etc/skel β€” files installed by default when a user is created Basic commands 1useradd -g <GID> -G <GID2> <user> # create a user in primary group GID (and supplementary group GID2). 2usermod <options> <user> # modify a user. 3userdel -r <user> # delete a user (and its home directory). 4groupadd / groupmod / groupdel # manage groups. 5 6id -a # show all info about the current user (UID, GUID, groups, etc.) - more precise than "who am i". 7sg <group> -c '<command>' # execute a command as a different group ID (to run scripts or write to a file with group rights). Password management 1passwd -u <user> # unlock a user account. 2echo "password" | passwd --stdin <user> # scripted password change. Account aging 1chage -l <user> # see the expiration dates. 1# list the expiry of every account 2for account in $(cut -f1 -d: /etc/passwd); do 3 echo "ACCOUNT: $account , EXPIRES: $(chage -l $account | grep 'Account expires' | awk '{print $4, $5, $6}'), CHANGED: $(chage -l $account | grep 'Last password change' | awk '{print $5, $6, $7}')"; 4done 1# change the aging info interactively 2chage <user> To unlock an account, set “Last Password Change” to -1 in chage (or use passwd -u <user>).
⏰ Jobs & Background
⏰ Jobs & Background
Schedule a task (at / batch) 1at : schedule a task to run at a later time (/!\ it executes what you give it on stdin). 2 ex : at 18:22 < "date; ps -ef | wc -l" 3 or : at now + 5 hours then type your commands then ctrl + d 4at -q a 16:05 tomorrow : -q defines the queue (a-z), i.e. the priority. 5at -c <job_number> : see the context and the commands of the task. 6atq : list the pending jobs (= at -l). 7atrm : delete a job. 8 9batch : schedule a task when the load average is below a threshold. Run jobs in the background 1jobs -l : list the running tasks. 1# Nohup in series 2for i in {1..6}; do echo "nohup sh -c \"shred -vfz -n 3 /dev/cciss/c0d${i} > nohup${i}.out 2>&1 \" &" ; done | bash Three points to remember:
🎯 CPU Affinity
🎯 CPU Affinity
Source : http://www.glennklockwood.com/hpc-howtos/process-affinity.html Taskset 1sudo apt-get install util-linux # or: yum install util-linux 2taskset -c 1 script.sh # run script.sh on CPU number 1 (-c: CPU, -p: PID) 3taskset -c 1,2,3 script.sh # give it several CPUs. Numactl 1numactl --cpunodebind=0 simulation.x 2numactl --cpunodebind=0 --membind=0 simulation.x 3numactl -C 0 -N 0 simulation.x 4numactl -C +0,1,2,3 simulation.x # similar to taskset 5numactl -H # see which memory corresponds to a CPU Note: with numactl, unlike taskset, you cannot change the CPU affinity of a process on the fly.
πŸ› Tracing (strace / ltrace / gstack)
πŸ› Tracing (strace / ltrace / gstack)
Strace - trace system calls 1strace -tt -p 24503 2 3strace -o strace01.out -e open -f bash --login -i # see the files opened during a bash connection. 4 # -o redirects the output / -e filters the system calls / -f follows forks (child processes) 5 6strace -f <binary_script> 2> trace.log : stdout -> the binary command result, stderr -> the binary's system calls. Ltrace - trace library calls ltrace traces shared-library calls (like strace, but at the library-call level).
πŸ“Š Process Monitoring (pidstat)
πŸ“Š Process Monitoring (pidstat)
pidstat reports the CPU, memory, I/O and context-switch activity of processes. Report the process context-switching activity 1# pidstat -w -p 3446 2 5 2Linux 3.10.0-123.13.2.el7.x86_64 (localhost.localdomain) 12/26/2014 3_x86_64_ (1 CPU) 407:23:38 AM UID PID cswch/s nvcswch/s Command 507:23:40 AM 0 3446 0.50 0.00 sshd 607:23:42 AM 0 3446 0.50 0.00 sshd 707:23:44 AM 0 3446 0.50 0.00 sshd 807:23:46 AM 0 3446 0.50 0.00 sshd 907:23:48 AM 0 3446 0.50 0.00 sshd 10Average: 0 3446 0.50 0.00 sshd cswch/s : number of voluntary context switches the task made per second. (A voluntary context switch occurs when a task blocks because it requires a resource that is unavailable.) nvcswch/s : number of non-voluntary context switches the task made per second. (An involuntary context switch takes place when a task executes for the duration of its time slice and is then forced to relinquish the processor.) Page faults and memory 1pidstat -r -p <PID> 3600 72 # every hour, 72 times - practical for long-term monitoring. 2 3pidstat -r -p <PID> 50 12 407:26:44 PM PID minflt/s majflt/s VSZ RSS %MEM Command 507:27:34 PM 13775 1.64 0.00 34957320 18183312 55.30 java minflt/s : number of minor faults the task has made per second β€” those which did not require loading a memory page from disk. majflt/s : number of major faults the task has made per second β€” those which required loading a memory page from disk. VSZ : Virtual Size β€” the virtual memory usage of the entire task in kilobytes. RSS : Resident Set Size β€” the non-swapped physical memory used by the task in kilobytes. Disk I/O 1pidstat -d -p <PID> 50 12 pidstat -d reports I/O statistics (kernels 2.6.20 and later only). The following values are displayed: