Where the system logs live
On a Unix machine, the system logs are in /var/log/messages (or /var/adm/messages on Solaris). This is where you find the errors, with log rotation.
Default syslog output
| Linux | Solaris | HP-UX | AIX | BSD |
|---|
| location | /var/log/messages, /var/log/secure, /var/log/boot.log | /var/adm/messages | /var/adm/syslog/mail.log, /var/adm/syslog/syslog.log | /tmp or none | /var/log/syslog |
System accounting (login & process)
| Type | Linux | Solaris | HP-UX | AIX |
|---|
| current logins | /var/run/utmp | /var/adm/utmpx | /var/adm/utmp | /etc/utmp |
| login history | /var/log/wtmp | /var/adm/wtmpx | /var/adm/wtmp | /var/adm/wtmp |
| process accounting | /var/log/pacct | /var/adm/pacct | /var/adm/pacct | /var/adm/pacct |
Login errors
| Linux | Solaris | HP-UX | AIX |
|---|
| failed logins | /var/log/btmp, /var/log/messages | /var/adm/loginlog, /var/adm/sulog | /var/adm/sulog | /etc/security/failedlogin |
Investigate the logs
1# today's logs
2grep "$(date '+%b %d')" /var/log/messages
3
4# disk errors (nawk: print the last field of the "Error Block" lines)
5nawk '/Error Block/{print $NF}' /var/adm/messages* | sort | uniq
6
7# find the IPs in a log, sort them and remove the duplicates
8cat /var/log/maillog | grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4 | uniq
Network investigation
1# ping a list of servers
2for ip in $(awk '/192.168.45/ {print $1}' /etc/hosts); do ping -c 1 $ip; done
Investigate on several servers
1for vm in vm{1..27}; do ssh -q $vm "hostname; free; sar -r 3 3"; done
SSH authentication logs
/var/log/auth.log โ SSH connection logs. Check that there are not too many failed connections (a sign of an intrusion attempt).
1grep "Invalid" /var/log/auth.log
Audit & accounting
/var/log/audit/audit.log โ if auditd is configured.- Process accounting (
acct / pacct):
1lastcomm root # the last commands run by a user (searchable by cmd, user, tty).