Browse Docs

Investigate

In this section

  • ๐Ÿ”Ž Search, Find & Compare

    Find files quickly

    1locate <pattern>    # find a directory or file quickly (uses an index); a brand-new file won't be found.
    2updatedb            # update the locate index.
    

    Open a file

    1view <file>             # opens a read-only vi view (preferred if you just want to search/view).
    2gzcat / zcat <file.gz>  # read a gzipped file.
    

    Info on a file or directory

    1stat </my/file>      # all info about a file (inode, creation, modification, access dates, etc.).
    2stat -f <FS>         # info about a filesystem.
    3stat -c%s $LOGFILE   # [scripting] get a precise value (size, modification date, etc.).
    

    grep

     1grep -w 'xyz'                # match the whole word.
     2grep -x 'Hello, world!'      # the whole line must match.
     3grep -c <pattern>            # count the matching lines.
     4grep -l "ERROR:" *.log       # search all .log files, list the files that match.
     5grep -L <pattern>            # inverse: list the files that do NOT match.
     6grep -f <patternfile> <file> # apply the patterns read from patternfile.
     7grep -i <pattern>            # ignore case.
     8grep -v <pattern>            # return the lines that do NOT match.
     9grep -m x <pattern>          # stop after x matching lines.
    10grep -n <pattern>            # show the line number.
    11grep -q <pattern>            # quiet: exit 0 if found, 1 (or 2) otherwise (for scripting).
    12grep -s <pattern>            # suppress permission/inexistent-file error messages.
    13grep -H <pattern>            # show the filename next to each matching line.
    14grep -h <pattern>            # do not show the filename (default behaviour).
    15grep -A x <pattern>          # also show x lines After.
    16grep -B x <pattern>          # also show x lines Before.
    17grep -C x <pattern>          # show x lines of context (A + B).
    18grep -a <pattern> <binary>   # search a binary file as if it were text.
    
    1egrep = grep -E   # for complex regular expressions.
    
    1# extract the 3rd field, then cut:
    2cat file | grep /u01/grid/19c | awk '{print $3}' | cut -f2 -d'"'
    3# is equivalent to:
    4cat file | grep -o /u01/grid/19c
    
  • ๐Ÿ“œ Logs

    Where the system logs live

    On a Unix machine, the system logs are in /var/log/messages (or /var/adm/messages on Solaris). This is where you find the errors, with log rotation.

    Default syslog output

    LinuxSolarisHP-UXAIXBSD
    location/var/log/messages, /var/log/secure, /var/log/boot.log/var/adm/messages/var/adm/syslog/mail.log, /var/adm/syslog/syslog.log/tmp or none/var/log/syslog

    System accounting (login & process)

    TypeLinuxSolarisHP-UXAIX
    current logins/var/run/utmp/var/adm/utmpx/var/adm/utmp/etc/utmp
    login history/var/log/wtmp/var/adm/wtmpx/var/adm/wtmp/var/adm/wtmp
    process accounting/var/log/pacct/var/adm/pacct/var/adm/pacct/var/adm/pacct

    Login errors

    LinuxSolarisHP-UXAIX
    failed logins/var/log/btmp, /var/log/messages/var/adm/loginlog, /var/adm/sulog/var/adm/sulog/etc/security/failedlogin

    Investigate the logs

    1# today's logs
    2grep "$(date '+%b %d')" /var/log/messages
    3
    4# disk errors (nawk: print the last field of the "Error Block" lines)
    5nawk '/Error Block/{print $NF}' /var/adm/messages* | sort | uniq
    6
    7# find the IPs in a log, sort them and remove the duplicates
    8cat /var/log/maillog | grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4 | uniq
    

    Network investigation

    1# ping a list of servers
    2for ip in $(awk '/192.168.45/ {print $1}' /etc/hosts); do ping -c 1 $ip; done
    

    Investigate on several servers

    1for vm in vm{1..27}; do ssh -q $vm "hostname; free; sar -r 3 3"; done
    

    SSH authentication logs

    /var/log/auth.log โ€” SSH connection logs. Check that there are not too many failed connections (a sign of an intrusion attempt).

  • ๐Ÿšฉ Files

    Find a process blocking a file

    • with fuser:
     1fuser  -m  </dir or /files>  # Find process blocking/using this directory or files. 
     2fuser -cu  </dir or /files>  # Same as above but add the user  
     3fuser -kcu </dir or /files>  # Kill process      
     4fuser -v  -k -HUP -i ./      # Send HUP signal to process
     5			
     6# Output will send you <PID + letter>, here is the meaning:
     7#   c  current directory.
     8#   e  executable being run.
     9#   f  open file.  (omitted in default display mode).
    10#   F  open file for writing. (omitted in default display mode).
    11#   r  root directory.
    12#   m  mmap'ed file or shared library.
    
    • with lsof ( = list open file):
    1lsof +D /var/log          # Find all files blocked with the process and user.
    2lsof -a +L1 <mountpoint>  # Process blocking a FS.
    3lsof -c ssh -c init       # Find files open by thoses processes.
    4lsof -p 1753              # Find files open by PID process.
    5lsof -u root              # Find files open by user.
    6lsof -u ^user             # Find files open by user except this one.
    7kill -9 `lsof -t -u toto` # kill user's processes.  (option -t output only PID).
    
    • MacGyver method:
    1#When you have no fuser or lsof: 
    2find /proc/*/fd -type f -links 0 -exec ls -lrt {} \;
    

    AIX specifics (fuser)

    1fuser -d /tmp                 # see the processes using the /tmp directory (AIX)
    2fuser -c /your_FS             # all processes with an open file in the filesystem (AIX)
    3fuser -cu /dev/vg01/lvol5     # also search with a filesystem or an LV
    
    • -c == -m ; -u also shows the process user.
    • to kill the processes: fuser -kcu.

    File deleted but space still held

    For detecting deleted-but-still-open files (lsof +L1) and freeing the held space, see the Disk Cleanup page.

  • ๐Ÿ‘ค Users & Connections

    Investigate a user

    1last              # the last user connections to a server (based on /var/log/wtmp or btmp).
    2ac -d             # statistics of my connection time per day.
    3ac -p <user>      # the connection time of all users (or of a specific user).
    4finger            # who is connected (-l to also see mails and plans of all users).
    5w                 # who is connected, doing what, and how much CPU they use.
    6who               # who is connected (-u for more info: PID, etc.).
    7who am i          # with which login I am connected.
    8id -a             # all info about the user I'm connected as (more precise than "who am i").
    9logname           # the login name of the current account.
    

    Reboots & uptime

    1last reboot   # see all the reboots that took place.
    2uptime        # see how long the server has been up + the load average.
    3lslogins -L   # also shows whether a user shutdown/rebooted the machine.
    
  • ๐Ÿšฉ Compare

    Compare files

    1diff <file1> <file2>       # -w to ignore whitespace.
    2colordiff <file1> <file2>  # colourised diff.
    3wdiff <file1> <file2>      # word diff: [โˆ’ โˆ’] replaced word, {+ +} added word.
    4vimdiff <file1> <file2>    # open both files in vim (blue = entirely different lines, red = partially different).
    5fgrep -f <list> <file>     # compare two lists (e.g. of hosts).
    

    Compare jar files

    1diff -W200 -y  <(unzip -vqq file1.jar | awk '{ if ($1 > 0) {printf("%s\t%s\n", $1, $8)}}' | sort -k2) <(unzip -vqq  file2.jar | awk '{ if ($1 > 0) {printf("%s\t%s\n", $1, $8)}}' | sort -k2)
    
  • ๐Ÿ”๏ธ Investigate

    Ressources

    1# in crontab or tmux session - take every hour a track of the memory usage
    2for i in {1..24} ; do echo -n "===================== " ; date ; free -m ; top -b -n1 | head -n 15 ; sleep 3600; done >> /var/log/SYSADM/memory.log &
    
Sunday, October 4, 2026 Monday, January 1, 1