Memo

126 Posts
2 Sections
2 Types

August 5, 2023 5 min read Docs

How to approach performance monitoring and tuning in Linux, and the various subsystems (and performance metrics) that need to be monitored. On a very high level, the following four subsystems need to be monitored: CPU Memory I/O Network 1. CPU Four critical performance metrics for the CPU: context switch, run queue, CPU utilization, and load average. Context Switch When the CPU switches from one process (or thread) to another, it is called a context switch. When a process switch happens, the kernel stores the current state of the CPU (of a process or thread) in memory. The kernel also retrieves the previously stored state (of a process or thread) from memory and puts it in the CPU. Context switching is essential for multitasking of the CPU. However, a higher level of context switching can cause performance issues. Run Queue The run queue indicates the total number of active processes in the current queue for the CPU. When the CPU is ready to execute a process, it picks it up from the run queue based on the priority of the process. Note that processes that are in a sleep state, or I/O wait state, are not in the run queue. A higher number of processes in the run queue can therefore cause performance issues. CPU Utilization Indicates how much of the CPU is currently being used. 100% CPU utilization means the system is fully loaded. Load Average Indicates the average CPU load over a specific time period. On Linux, load average is displayed for the last 1 minute, 5 minutes, and 15 minutes. This helps to see whether the overall load on the system is going up or down. For example, a load average of 0.75 1.70 2.10 indicates that the load is coming down (0.75 = last 1 minute, 1.70 = last 5 minutes, 2.10 = last 15 minutes). Note that this load average is calculated by combining both the total number of processes in the queue, and the total number of processes in the uninterruptible task status. 2. Network A good understanding of TCP/IP concepts is helpful when analyzing any network issue. For network interfaces, monitor the total number of packets (and bytes) received/sent through the interface, the number of packets dropped, etc. 3. I/O I/O wait is the amount of time the CPU is waiting for I/O. Consistent high I/O wait on the system indicates a problem in the disk subsystem. Monitor reads/second and writes/second. These are measured in blocks, i.e. the number of blocks read/written per second. They are also referred to as bi and bo (block in and block out). tps indicates total transactions per second, which is the sum of rtps (read transactions per second) and wtps (write transactions per second). 4. Memory RAM is the physical memory. If you have 4 GB of RAM installed, you have 4 GB of physical memory. Virtual memory = swap space available on disk + physical memory. The virtual memory contains both user space and kernel space. Using a 32-bit or a 64-bit system makes a big difference in determining how much memory a process can use: On a 32-bit system a process can only access a maximum of 4 GB of virtual memory. On a 64-bit system there is no such limitation. Unused RAM is used by the kernel as filesystem cache. Linux swaps when it needs more memory than the physical memory. When it swaps, it writes the least-used memory pages from the physical memory to the swap space on the disk. Lots of swapping can cause performance issues: the disk is much slower than the physical memory, and it takes time to swap the memory pages from RAM to disk. The subsystems are interrelated All four subsystems are interrelated. Just because you see a high reads/second, writes/second, or I/O wait, it does not mean the issue is with the I/O subsystem. It also depends on what the application is doing. In most cases, the performance issue is caused by the application running on the Linux system.

August 4, 2023 2 min read Docs

🧹 Disk Cleanup

Memo Linux Storage

Find old files 1find . -type f -mtime +150 -exec ls -lrt {} \; | more 2find . -maxdepth 1 -name "*.log" -mtime +10 -exec ls -lrt {} \; 3find . -mtime +150 -exec rm -f {} \; The find loop is cheaper than a shell loop (for ...). You can make it even more efficient by batching the rm calls: 1find . -type f -print -exec rm -- "{}" + # note the "+" instead of the usual "\;" See which directories use the most space 1du -max . # list all the FS sub-directories (-x avoids filesystems other than the requested one, "." = search from where you are) 2du -sh * # show the total without listing the sub-directories (h = human readable) 3du -max . | sort -n | tail -30 # the 30 largest files/directories 4du -ks * | sort -n # size in kilobytes of all files and directories, where you are 5du -hsc * | sort -h # from smallest to largest 6ls -lrS # list files by size (in bytes) - note: ls -l does not give the true value contained in a directory 7du -ch /dir/ # size of the directories contained in /dir/ (with suffix) then the total Reduce / Truncate a file 1perl -e 'truncate "wanted_file", 100000' 2truncate -s 0 /ftpusers/ftp.upload.log File deleted but space still held by a process 1lsof +aL1 # "+L1" selects open files that have been "unlinked" (deleted but still open) 2lsof -nP | grep '(deleted)' 3find /proc/*/fd -type f -links 0 -exec ls -lrt {} \; # [SunOS] There are two solutions:

Read more about ⌨️ Bash Shortcut.

August 1, 2023 3 min read Docs

☁️ Cloud-Init

Memo Linux

Troubleshooting cloud-init status --wait usefull for scripting, waiting cloud-init to finish before going to next step. cloud-init status --long 1status: done 2extended_status: done 3boot_status_code: enabled-by-generator 4last_update: Thu, 01 Jan 1970 00:00:55 +0000 5detail: DataSourceNoCloud [seed=/dev/sr0] 6errors: [] 7recoverable_errors: {} sudo cloud-init analyze show 1-- Boot Record 01 -- 2The total time elapsed since completing an event is printed after the "@" character. 3The time the event takes is printed after the "+" character. 4 5Starting stage: init-local 6|`->no cache found @00.00600s +00.00000s 7|`->found local data from DataSourceNoCloud @00.01500s +00.12600s 8Finished stage: (init-local) 00.75400 seconds 9 10Starting stage: init-network 11|`->restored from cache with run check: DataSourceNoCloud [seed=/dev/sr0] @04.21100s +00.00200s 12|`->setting up datasource @04.22800s +00.00000s 13|`->reading and applying user-data @04.23400s +00.00500s 14|`->reading and applying vendor-data @04.23900s +00.00000s 15|`->reading and applying vendor-data2 @04.23900s +00.00000s 16|`->activating datasource @04.27100s +00.00100s 17|`->config-seed_random ran successfully and took 0.000 seconds @04.29500s +00.00100s 18|`->config-write_files ran successfully and took 0.001 seconds @04.29600s +00.00100s 19|`->config-growpart ran successfully and took 0.562 seconds @04.29700s +00.56200s 20|`->config-resizefs ran successfully and took 0.193 seconds @04.86000s +00.19200s 21|`->config-mounts ran successfully and took 0.001 seconds @05.05200s +00.00100s 22|`->config-set_hostname ran successfully and took 0.004 seconds @05.05300s +00.00500s 23|`->config-update_hostname ran successfully and took 0.001 seconds @05.05800s +00.00100s 24|`->config-update_etc_hosts ran successfully and took 0.005 seconds @05.05900s +00.00500s 25|`->config-users_groups ran successfully and took 0.216 seconds @05.06400s +00.21600s 26|`->config-ssh ran successfully and took 0.404 seconds @05.28100s +00.40400s 27|`->config-set_passwords ran successfully and took 0.001 seconds @05.68500s +00.00200s 28Finished stage: (init-network) 01.50000 seconds 29 30Starting stage: modules-config 31|`->config-ssh_import_id ran successfully and took 0.001 seconds @07.43300s +00.00100s 32|`->config-locale ran successfully and took 0.003 seconds @07.43400s +00.00300s 33|`->config-grub_dpkg ran successfully and took 0.352 seconds @07.43700s +00.35200s 34|`->config-apt_configure ran successfully and took 0.049 seconds @07.79000s +00.04800s 35|`->config-timezone ran successfully and took 0.007 seconds @07.83900s +00.00700s 36|`->config-runcmd ran successfully and took 0.001 seconds @07.84600s +00.00100s 37|`->config-byobu ran successfully and took 0.000 seconds @07.84700s +00.00100s 38Finished stage: (modules-config) 00.45400 seconds 39 40Starting stage: modules-final 41|`->config-package_update_upgrade_install ran successfully and took 26.632 seconds @20.56700s +26.63300s 42|`->config-write_files_deferred ran successfully and took 0.001 seconds @47.20000s +00.00200s 43|`->config-reset_rmc ran successfully and took 0.000 seconds @47.20200s +00.00100s 44|`->config-scripts_vendor ran successfully and took 0.001 seconds @47.20300s +00.00000s 45|`->config-scripts_per_once ran successfully and took 0.000 seconds @47.20300s +00.00100s 46|`->config-scripts_per_boot ran successfully and took 0.000 seconds @47.20400s +00.00000s 47|`->config-scripts_per_instance ran successfully and took 0.000 seconds @47.20400s +00.00100s 48|`->config-scripts_user ran successfully and took 0.558 seconds @47.20500s +00.55800s 49|`->config-ssh_authkey_fingerprints ran successfully and took 0.005 seconds @47.76400s +00.00500s 50|`->config-keys_to_console ran successfully and took 0.054 seconds @47.76900s +00.05500s 51|`->config-install_hotplug ran successfully and took 0.001 seconds @47.82400s +00.00100s 52|`->config-final_message ran successfully and took 0.001 seconds @47.82500s +00.00100s 53Finished stage: (modules-final) 27.29600 seconds Check the logs: sudo tail -n 50 /var/log/cloud-init-output.log