Linux Services Organization

Our goal, introduce Linux services to the enterprise world.
Contact us in contact@linuxsv.org

Linux Services Organization : Linux AUDIT Linux Server

Auditd

The Linux Auditing System based on auditd daemon on kernel 2.6 logs events, system calls and file access on file /var/log/audit/audit.log by default. These logs can be used to audit the system regularly performing forensics tasks looking for not regular events (for example security breaches) using 'aureport' and 'ausearch' tools..

Auditd is installed by audit RPM and is enabled by default on RHEL6 systems. To make sure that auditd will be up and running :

# chkconfig auditd on
# /etc/init.d/auditd start


It can be also activated booting the system with the kernel parameter 'audit=1'. In this case if the auditd is not started audit messages will be written on /var/log/messages file.

/etc/audit/auditd.conf

This file must be used to configure how auditd daemon runs on the system. The following is a typical example of this file :

# cat /etc/audit/auditd.conf
#
# This file controls the configuration of the audit daemon
#

log_file = /var/log/audit/audit.log
log_format = RAW
log_group = root
priority_boost = 4
flush = INCREMENTAL
freq = 20
num_logs = 4
disp_qos = lossy
dispatcher = /sbin/audispd
name_format = NONE
##name = mydomain
max_log_file = 5
# Use 'max_log_file_action = IGNORE' to NOT rotate audit.log
max_log_file_action = ROTATE
space_left = 75
space_left_action = SYSLOG
action_mail_acct = root
admin_space_left = 50
admin_space_left_action = SUSPEND
disk_full_action = SUSPEND
disk_error_action = SUSPEND
##tcp_listen_port =
tcp_listen_queue = 5
tcp_max_per_addr = 1
##tcp_client_ports = 1024-65535
tcp_client_max_idle = 0
enable_krb5 = no
krb5_principal = auditd
##krb5_key_file = /etc/audit/audit.key


In this file can be configured where to log audit traces (log_file), the trace format used (log_format), log_file rotation (max_log_file, max_log_file_action), etc ... For more info 'man auditd.conf'. Of course if this file is modified the auditd must be restarted to take effect the modification :

# /etc/init.d/auditd restart

/etc/audit/audit.rules

This file must be used to write rules for events such as file access or system calls. These rules will be applied on auditd daemon through 'auditctl' tool when auditd start, so if you want make a audit rule permanent (reboot persistent) the rule must be written on this file.

Audit rules can be classified in two major categories: system calls and file access (or watches). The following are valid examples of this categories (rules just begin after "AUDIT RULES BEGIN HERE" line) :

# cat /etc/audit/audit.rules

# This file contains the auditctl rules that are loaded
# whenever the audit daemon is started via the initscripts.
# The rules are simply the parameters that would be passed
# to auditctl.

# First rule - delete all
-D

# Increase the buffers to survive stress events.
# Make this bigger for busy systems
-b 320

# AUDIT RULES BEGIN HERE. See auditctl man page

###--> SYSTEM CALL RULES

# Next rule audits 'rm' system calls exit, so any execution of 'rm' command will be audited !!!
-a exit,always -S unlinkat

# Next rule audits all activity done by user uid=500 on the system. Be careful because lot traces will be written !!!
-a exit,always -S all -F uid=500

###--> FILE ACCESS (WATCHES)

# Next rule audits read,write,execute and append modifications on file /etc/shadow
-w /etc/shadow -k shadow-access -p rwxa

# Next rule audits any execution of ssh-client
-w /usr/bin/ssh -k ssh-access -p x



Once the file has been configured, auditd daemon will be restarted in order to apply the changes:

# /etc/init.d/auditd restart

Lets test all this rules ...

Test rule '-a exit,always -S unlinkat'

As user john (uid=500) create and remove /tmp/qqq file :

# su - john
john-$ touch /tmp/qqq
john-$ rm /tmp/qqq


The following trace is written on /var/log/audit/audit.log auditing the command 'rm /tmp/qqq' :

# tail -100 /var/log/audit/audit.log
...
type=SYSCALL msg=audit(1314068461.669:18608): arch=40000003 syscall=301 success=yes exit=0 a0=ffffff9c a1=853b8c0 a2=0 a3=853b860 items=2 ppid=5476 pid=5496 auid=0 uid=500 gid=500 euid=500 suid=500 fsuid=500 egid=500 sgid=500 fsgid=500 tty=pts0 ses=67 comm="rm" exe="/bin/rm" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
type=CWD msg=audit(1314068461.669:18608): cwd="/home/john"
type=PATH msg=audit(1314068461.669:18608): item=0 name="/tmp/" inode=19 dev=fd:00 mode=041777 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:tmp_t:s0
type=PATH msg=audit(1314068461.669:18608): item=1 name="/tmp/qqq" inode=16433 dev=fd:00 mode=0100664 ouid=500 ogid=500 rdev=00:00 obj=unconfined_u:object_r:user_tmp_t:s0

Test rule '-a exit,always -S all -F uid=500'

As user john execute 'ls /tmp' :
john-$ ls /tmp

The following trace is written on /var/log/audit/audit.log auditing any command executed by user john (uid=500) :

# tail -100 /var/log/audit/audit.log
...
type=SYSCALL msg=audit(1314068990.122:18927): arch=40000003 syscall=4 success=yes exit=16 a0=2 a1=b7773000 a2=10 a3=10 items=0 ppid=5526 pid=5535 auid=0 uid=500 gid=500 euid=500 suid=500 fsuid=500 egid=500 sgid=500 fsgid=500 tty=pts0 ses=67 comm="bash" exe="/bin/bash" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
type=SYSCALL msg=audit(1314068990.122:18928): arch=40000003 syscall=175 success=yes exit=0 a0=2 a1=8120560 a2=0 a3=8 items=0 ppid=5526 pid=5535 auid=0 uid=500 gid=500 euid=500 suid=500 fsuid=500 egid=500 sgid=500 fsgid=500 tty=pts0 ses=67 comm="bash" exe="/bin/bash" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
...much more lines ...

Test rule '-w /etc/shadow -k shadow-access -p rwxa'

As user john change the password :
john-$ passwd
...

The following trace is written on /var/log/audit/audit.log auditing the command 'passwd' executed by user john (uid=500) :

# tail -100 /var/log/audit/audit.log
...
type=SYSCALL msg=audit(1314069251.296:18942): arch=40000003 syscall=5 success=yes exit=3 a0=817a19 a1=80000 a2=1b6 a3=8179b5 items=1 ppid=5627 pid=5628 auid=0 uid=0 gid=500 euid=0 suid=0 fsuid=0 egid=500 sgid=500 fsgid=500 tty=pts0 ses=67 comm="unix_chkpwd" exe="/sbin/unix_chkpwd" subj=unconfined_u:unconfined_r:chkpwd_t:s0-s0:c0.c1023 key="shadow-access"
type=CWD msg=audit(1314069251.296:18942): cwd="/home/john"
type=PATH msg=audit(1314069251.296:18942): item=0 name="/etc/shadow" inode=21803 dev=fd:00 mode=0100000 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:shadow_t:s0

Test rule '-w /usr/bin/ssh -k ssh-access -p x'

As user john try to ssh to the remote server with IP 10.0.0.10
john-$ ssh 10.0.0.10
...

The following trace is written on /var/log/audit/audit.log auditing the command 'ssh 10.0.0.10' executed by user john (uid=500) :

# tail -100 /var/log/audit/audit.log
...
type=SYSCALL msg=audit(1314069685.221:18956): arch=40000003 syscall=11 success=yes exit=0 a0=8d8ecc0 a1=8d96990 a2=8d92e30 a3=8d96990 items=2 ppid=5535 pid=5680 auid=0 uid=500 gid=500 euid=500 suid=500 fsuid=500 egid=500 sgid=500 fsgid=500 tty=pts0 ses=67 comm="ssh" exe="/usr/bin/ssh" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="ssh-access"
type=EXECVE msg=audit(1314069685.221:18956): argc=2 a0="ssh" a1="10.0.0.10"
type=CWD msg=audit(1314069685.221:18956): cwd="/home/john"
type=PATH msg=audit(1314069685.221:18956): item=0 name="/usr/bin/ssh" inode=27447 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ssh_exec_t:s0
type=PATH msg=audit(1314069685.221:18956): item=1 name=(null) inode=27586 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ld_so_t:s0

Analyzing Audit logs

By default traces on /var/log/audit/audit.log are written with the format received by the Kernel, 'vi' or 'emacs' can be used in order to read the information. The following commands can be used in order to make easy audit log analysis :

aureport
Used to format audit logs and generate reports and summaries. Only root user can execute this command.

ausearch
Used to query audit records based on different search criteria. Only root user can execute this command.

autrace
Used to generate audit records from a specific process. Only root user can execute this command.

aureport

As said before aureport is a tool that produces summary reports of the audit system logs. By default it reads the information on /var/log/audit/audit.log that is only accessible to root user. It is recommended the use of '-i' flag to print data in human-readable format, also use '-if file.log' flag to make audit reports from audit traces on file file.log. The following are some examples :

# aureport -a -i

AVC Report
========================================================
# date time comm subj syscall class permission obj event
========================================================
1. 08/22/2011 18:37:48 httpd unconfined_u:system_r:httpd_t:s0 stat64 dir getattr unconfined_u:object_r:default_t:s0 denied 16026
2. 08/22/2011 18:45:58 httpd unconfined_u:system_r:httpd_t:s0 stat64 dir getattr unconfined_u:object_r:default_t:s0 denied 16039
3. 08/22/2011 18:51:54 httpd unconfined_u:system_r:httpd_t:s0 stat64 dir getattr unconfined_u:object_r:default_t:s0 denied 16046
...


Report audit traces about avc messages, SElinux violations.

# aureport -au -i

Authentication Report
============================================
# date time acct host term exe success event
============================================
1. 08/22/2011 17:21:56 root ? tty1 /bin/login yes 15957
2. 08/22/2011 18:36:36 root 10.0.0.10 ssh /usr/sbin/sshd yes 16017
3. 08/22/2011 19:14:07 john ? pts/0 /bin/su yes 16068
4. 08/22/2011 20:49:09 john ? tty3 /bin/login no 15713
...


Report audit traces about authentication attempts.

# aureport -au -i --failed

Authentication Report
============================================
# date time acct host term exe success event
============================================
1. 08/22/2011 20:49:09 john ? tty3 /bin/login no 15713
...


Report audit traces about FAILED authentication attempts.

# aureport -f -i

File Report
===============================================
# date time file syscall success exe auid event
===============================================
1. 08/22/2011 18:37:48 /web stat64 no /usr/sbin/httpd root 16026
2. 08/22/2011 18:45:58 /web stat64 no /usr/sbin/httpd root 16039
3. 08/22/2011 18:51:54 /web stat64 no /usr/sbin/httpd root 16046
4. 08/22/2011 19:15:53 /home/john lstat64 no /usr/sbin/httpd root 16072
...


Report audit traces about files. For more info 'man aureport'.

ausearch

As said before ausearch is a tool that can query the audit daemon logs based for events based on different search criteria. As aureport command only root user can execute this command because of only root can read audit log files. It is recommended the use of '-i' flag to print data in human-redeable format, also use '-if file.log' flag to make audit reports from audit traces on file file.log. The following are some examples :

# ausearch -i -f /etc/passwd
----
type=PATH msg=audit(08/23/2011 04:14:19.216:16459) : item=0 name=/etc/passwd inode=39852 dev=fd:00 mode=file,644 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:etc_t:s0
type=CWD msg=audit(08/23/2011 04:14:19.216:16459) : cwd=/home/john
type=SYSCALL msg=audit(08/23/2011 04:14:19.216:16459) : arch=i386 syscall=open success=yes exit=3 a0=4069f8 a1=80000 a2=1b6 a3=4069b5 items=1 ppid=4992 pid=4993 auid=root uid=john gid=john euid=john suid=john fsuid=john egid=john sgid=john fsgid=john tty=pts0 ses=5 comm=bash exe=/bin/bash subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
...


It searches audit traces related with /etc/passwd file.

ausearch -i -x su
----
type=USER_AUTH msg=audit(08/22/2011 19:14:07.759:16068) : user pid=22587 uid=root auid=root ses=11 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:authentication acct=john exe=/bin/su hostname=? addr=? terminal=pts/0 res=success'
----
type=USER_ACCT msg=audit(08/22/2011 19:14:07.760:16069) : user pid=22587 uid=root auid=root ses=11 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:accounting acct=john exe=/bin/su hostname=? addr=? terminal=pts/0 res=success'
...


It searches audit traces related with the execution of 'su' command.

# ausearch -m avc -ts yesterday
----
time->Mon Aug 22 18:37:48 2011
type=SYSCALL msg=audit(1314031068.667:16026): arch=40000003 syscall=195 success=no exit=-13 a0=b7818e80 a1=bfbb3010 a2=68fff4 a3=8000 items=0 ppid=22234 pid=22237 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=11 comm="httpd" exe="/usr/sbin/httpd" subj=unconfined_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1314031068.667:16026): avc: denied { getattr } for pid=22237 comm="httpd" path="/web" dev=dm-0 ino=9533 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=dir
----
time->Mon Aug 22 18:45:58 2011
type=SYSCALL msg=audit(1314031558.583:16039): arch=40000003 syscall=195 success=no exit=-13 a0=c38e80 a1=bf9bace0 a2=7abff4 a3=8000 items=0 ppid=22434 pid=22435 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=11 comm="httpd" exe="/usr/sbin/httpd" subj=unconfined_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1314031558.583:16039): avc: denied { getattr } for pid=22435 comm="httpd" path="/web" dev=dm-0 ino=9533 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=dir
...


It searches for yesterday SElinux audit traces. For more info 'man ausearch'

autrace

It is a program that will add the audit rules to trace a process similar to strace. It will then execute the program passing arguments to it. The resulting audit information will be in the audit logs if the audit daemon is running or syslog.

# autrace /bin/cat /etc/hosts

type=CONFIG_CHANGE msg=audit(1314088139.995:22322): auid=0 ses=61 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 op="add rule" key=(null) list=4 res=1
type=CONFIG_CHANGE msg=audit(1314088139.995:22323): auid=0 ses=61 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 op="add rule" key=(null) list=4 res=1
type=SYSCALL msg=audit(1314088139.996:22324): arch=40000003 syscall=6 success=yes exit=0 a0=4 a1=1 a2=bfd8366a a3=bfd83748 items=0 ppid=6813 pid=6815 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=61 comm="autrace" exe="/sbin/autrace" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
type=SYSCALL msg=audit(1314088139.998:22325): arch=40000003 syscall=197 success=yes exit=0 a0=1 a1=bfd82f80 a2=297ff4 a3=2984e0 items=0 ppid=6813 pid=6815 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=61 comm="autrace" exe="/sbin/autrace" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
...
...
Trace complete. You can locate the records with 'ausearch -i -p 6815'


As said in traces, with the command 'ausearch -i -p 6815' we can look for that process audit traces.

Process Accounting (PSACCT)

On RHEL6 systems the RPM psacct can be used in order to account all process executed on the system. By default this service is disabled so the first step will be enable the service on boot and then start the service:

# yum install psacct
# chkconfig psacct on
# /etc/init.d/psacct start


From this moment all commands executed by all users on your system will be recorded and ready to be queried. Lets see some examples of how to query this information :

ac

Prints statistics about users connect time. For more info 'man ac'

# ac -p
root 40.61
john 0.08
total 40.69

lastcomm

Prints out information about previously executed commands. For more info 'man lastcomm'.

# lastcomm john
bash S john pts/1 0.02 secs Tue Aug 23 11:05
ps john pts/1 0.06 secs Tue Aug 23 11:05
bash F john pts/1 0.00 secs Tue Aug 23 11:05
id john pts/1 0.00 secs Tue Aug 23 11:05
...

sa

It summarizes accounting information. For more info 'man sa'.

# sa -m
272 935.97re 0.06cp 929k
root 252 935.86re 0.06cp 934k
john 20 0.11re 0.00cp 864k
...

Questions

1.- Auditd is enabled by default on RHEL6 systems (true/false).

2.- Psacct is enabled by default on RHEL6 systems (true/false).

3.- Which configuration parameter must be configured on auditd.conf file in order to not rotate audit.log file?

4.- Which line must be added on audit.rules file in order to audit '/sbin/ifdown' command execution ?

5.- Which audit command can be used in order to report SElinux violation events ?

6.- Which audit command can be used in order to search all login attempts audit traces ?

7.- Which audit command can be used in order to trace all audit traces related with /sbin/reboot command ?

8.- Which psacct command can be used in order to display last commands executed by user john ?

9.- Which psacct command can be used in order to summarize account information per user ?

10.- In which file auditd writes the audit traces by default ?

A - /etc/securety/audit.log
B - /var/log/audit/audit.log
C - Both of them
D - None of them

11.- Which file is using command 'ac' as information source ?

A - /var/log/audit/audit.log
B - /var/log/wtmp
C - Both of them
D - None of them

Labs

1.- Configure audit to report any attempt of mounting a filesystem on rhel6 server. Verify and analyse collected traces.

2.- Configure audit to report any attempt of accessing to /etc/passwd file. Verify and analyse collected traces.

3.- Install and enable psacct on your RHEL6 system. Use it to report last commands executed by user root.

-- This page is part of Linux Server online tutorial --