Browse Docs

Processes

In this section

  • ๐Ÿ” Find & Inspect Processes

    Find a process

    1ps -fp <pid>        # find a process by its PID.
    2pidof httpd         # find the PIDs of httpd.
    3pidstat -lp <pid>   # process name with all its complete arguments.
    4                    # for a tomcat or weblogic process, you can split the arguments with: sed 's/ -/\n -/g'
    5pidstat -C "mysql"  # find a process by its name (gives the PID and CPU load).
    

    The process tree

    1pstree -pu   # the process tree with PID and user (if pstree is not installed, use the alternatives below).
    
    1ps -ejH
    2  PID  PGID   SID TTY          TIME CMD
    3    1     1     1 ?        00:00:37 init
    411016 11009 11009 ?        00:00:00       sshd
    511017 11017 11017 pts/12   00:00:00         bash
    611125 11125 11017 pts/12   00:00:00           telnet
    
    1ps axjf
    2 PPID   PID  PGID   SID TTY      TPGID STAT   UID   TIME COMMAND
    3    0     1     1     1 ?           -1 Ss       0   0:37 init [5]
    4 8617 10610 10610 10610 ?           -1 Ss       0   0:00  \_ sshd: support [priv]
    510610 10710 10610 10610 ?           -1 S     5027   0:00      \_ sshd: support@notty
    610710 10711 10711 10711 ?           -1 Ss    5027   0:00          \_ sshd: support@internal-sftp-server
    
    1ps faux
    2USER       PID  %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    3root         1   0.0  0.0  10372   696 ?        Ss   Aug09   0:37 init [5]
    4user1      4168  0.0  0.0   8728   968 ?        Ss   Aug24   0:00  |   \_ /bin/bash -c perl /data/supports/scripts/SRAM_asr5k.pl &>/dev/null
    5user1      4174  0.0  0.0  34068  5044 ?        S    Aug24   0:00  |       \_ perl /data/supports/SRAM_asr5k.pl
    6user1      4188  0.0  0.0   8728   984 ?        S    Aug24   0:00  |           \_ sh -c grep -c SRAM /data/syslogCOLLECT/LTE_*/*/*.20160824.log
    

    /proc

    The /proc filesystem exposes per-process information:

  • ๐Ÿ“Š Process Monitoring (pidstat)

    pidstat reports the CPU, memory, I/O and context-switch activity of processes.

    Report the process context-switching activity

     1# pidstat -w -p 3446 2 5
     2Linux 3.10.0-123.13.2.el7.x86_64 (localhost.localdomain) 12/26/2014
     3_x86_64_ (1 CPU)
     407:23:38 AM UID PID cswch/s nvcswch/s Command
     507:23:40 AM 0 3446 0.50 0.00 sshd
     607:23:42 AM 0 3446 0.50 0.00 sshd
     707:23:44 AM 0 3446 0.50 0.00 sshd
     807:23:46 AM 0 3446 0.50 0.00 sshd
     907:23:48 AM 0 3446 0.50 0.00 sshd
    10Average: 0 3446 0.50 0.00 sshd
    
    • cswch/s : number of voluntary context switches the task made per second. (A voluntary context switch occurs when a task blocks because it requires a resource that is unavailable.)
    • nvcswch/s : number of non-voluntary context switches the task made per second. (An involuntary context switch takes place when a task executes for the duration of its time slice and is then forced to relinquish the processor.)

    Page faults and memory

    1pidstat -r -p <PID> 3600 72    # every hour, 72 times - practical for long-term monitoring.
    2
    3pidstat -r -p <PID> 50 12
    407:26:44 PM       PID   minflt/s  majflt/s     VSZ    RSS   %MEM  Command
    507:27:34 PM     13775      1.64      0.00 34957320 18183312  55.30  java
    
    • minflt/s : number of minor faults the task has made per second โ€” those which did not require loading a memory page from disk.
    • majflt/s : number of major faults the task has made per second โ€” those which required loading a memory page from disk.
    • VSZ : Virtual Size โ€” the virtual memory usage of the entire task in kilobytes.
    • RSS : Resident Set Size โ€” the non-swapped physical memory used by the task in kilobytes.

    Disk I/O

    1pidstat -d -p <PID> 50 12
    

    pidstat -d reports I/O statistics (kernels 2.6.20 and later only). The following values are displayed:

  • ๐Ÿ› Tracing (strace / ltrace / gstack)

    Strace - trace system calls

    1strace -tt -p 24503
    2
    3strace -o strace01.out -e open -f bash --login -i   # see the files opened during a bash connection.
    4    # -o redirects the output / -e filters the system calls / -f follows forks (child processes)
    5
    6strace -f <binary_script> 2> trace.log :  stdout -> the binary command result, stderr -> the binary's system calls.
    

    Ltrace - trace library calls

    ltrace traces shared-library calls (like strace, but at the library-call level).

  • ๐ŸŽฏ CPU Affinity

    Source : http://www.glennklockwood.com/hpc-howtos/process-affinity.html

    Taskset

    1sudo apt-get install util-linux   # or: yum install util-linux
    2taskset -c 1 script.sh          # run script.sh on CPU number 1  (-c: CPU, -p: PID)
    3taskset -c 1,2,3 script.sh      # give it several CPUs.
    

    Numactl

    1numactl --cpunodebind=0 simulation.x
    2numactl --cpunodebind=0 --membind=0 simulation.x
    3numactl -C 0 -N 0 simulation.x
    4numactl -C +0,1,2,3 simulation.x    # similar to taskset
    5numactl -H                          # see which memory corresponds to a CPU
    

    Note: with numactl, unlike taskset, you cannot change the CPU affinity of a process on the fly.

  • โฐ Jobs & Background

    Schedule a task (at / batch)

    1at      :  schedule a task to run at a later time (/!\ it executes what you give it on stdin).
    2           ex :  at 18:22 < "date; ps -ef | wc -l"
    3           or : at now + 5 hours  then type your commands then ctrl + d
    4at -q a 16:05 tomorrow   :  -q defines the queue (a-z), i.e. the priority.
    5at -c  <job_number>      :  see the context and the commands of the task.
    6atq      :  list the pending jobs (= at -l).
    7atrm     :  delete a job.
    8
    9batch   :  schedule a task when the load average is below a threshold.
    

    Run jobs in the background

    1jobs -l   : list the running tasks.
    
    1# Nohup in series
    2for i in {1..6}; do echo "nohup sh -c \"shred -vfz -n 3 /dev/cciss/c0d${i} > nohup${i}.out 2>&1 \" &" ; done | bash
    

    Three points to remember:

  • โฒ๏ธ Cron & Anacron

    Configurations

    • /etc/crontab : the daemon’s configuration file, defining the default behaviour of crond (SHELL, MAILTO, etc.).
    • /etc/cron.d/... : system crontab (used by admins).
    • /var/spool/cron/root : crontab per user.
    1MAILTO="admin@example.com"
    2* * * * *  root  /usr/local/sbin/mycommand.sh > /dev/null 2>&1
    

    Anacron

    • /etc/anacrontab : file that runs, via the run-parts command, /etc/cron.daily, /etc/cron.weekly, /etc/cron.monthly.
    • /etc/cron.d/0hourly : exception, runs /etc/cron.hourly via run-parts, checking the last run of the task in /var/spool/anacron/....

    Special cases

    Exactly the last day of each month:

  • ๐Ÿ“ฆ Chroot Jail

    Change the root directory of a command or a process, and its children.

    Example: creating a chroot

     1# create the "jail" directory
     2J=$HOME/jail
     3mkdir -p $J
     4mkdir -p $J/{bin,lib64,lib}
     5cd $J
     6
     7# copy the binaries and their libraries into the jail
     8cp -v /bin/{bash,ls} $J/bin
     9
    10list="$(ldd /bin/bash | egrep -o '/lib.*\.[0-9]')"
    11for i in $list; do cp -v "$i" "${J}${i}"; done
    12
    13list="$(ldd /bin/ls | egrep -o '/lib.*\.[0-9]')"
    14for i in $list; do cp -v "$i" "${J}${i}"; done
    15
    16# enter the jail
    17sudo chroot $J /bin/bash
    
Sunday, October 4, 2026 Monday, January 1, 1