Monitoring Processes in User Mode with the Linux Audit Daemon

Process monitoring in Linux can be useful for a security audit, performance analysis, software improvement, and many other development activities. The Linux ecosystem provides several key ways to monitor processes, but they differ in their efficiency, ease of use, and possible risks to your system. 

In this article, we overview several popular ways to monitor Linux processes in user mode, examine the pros and cons of doing so with the Linux audit daemon (auditd), and show you how to work with this tool. This article will be useful for Linux developers who are looking for a safe and efficient process monitoring method.

How to monitor processes in Linux

Process monitoring can help you solve numerous issues with Linux-based software. Whether you experience a resource drain, incorrect or unexpected software behavior, or security vulnerabilities, you can examine the process responsible for this issue and see what went wrong. Monitoring is also an essential part of process management in Linux.

Being an extremely flexible system for software development needs, the OS provides several Linux process monitoring tools. The choice between them depends on your needs, the type of process, and skills in working with Linux. Let’s take a look at three common Linux process monitoring methods:

Native Linux commands. Linux has a number of commands for working with active processes:

• /proc — pulls the list of active processes, except for short-lived ones
• ps — provides the same information on processes as /proc with the parameters you select: process identifier (PID), name of the command that launched the process, active CPU time, memory usage, etc.
• top and htop — both commands return a dynamic real-time list of active processes; htop also shows a dynamic graph of machine resource use
• lsof — shows the list of open files; you can review all currently opened files, files opened by a certain process, or files opened by a certain user

These commands aren’t enough to have complete control over processes and monitor all details about them. Also, native Linux commands don’t allow you to work with short-lived processes.

Kernel debugging solutions. Using Linux kernel probes like tracepoints and kprobes/kretprobes provides you with a more robust set of process monitoring tools compared to the commands mentioned above. In kernel mode, you can monitor short-lived processes and read information about processes straight from the kernel. However, such solutions may cause unstable performance or system crashes since they operate directly in the kernel.

You can mitigate the risk of unstable performance with eBPF, which creates a sandbox for programs that work in the Linux kernel.  

Linux audit daemon. This Linux user mode process monitoring tool is a component of the Linux auditing system that provides records about kernel events that recently happened without hooking the system kernel. It allows you to configure any system auditing parameters you want, view audit logs, and record them to the disk.

Compared to previous process monitoring methods, using audit daemon for Linux process monitoring provides you with information about all system processes while working safely in user mode. That’s why we prefer using the audit daemon in our Linux development projects. Let’s take a closer look at how to check process in Linux with this tool and the pros and cons of using it.

Pros and cons of using the Linux audit daemon

The audit daemon, or auditd, is a versatile and widely popular tool in the Linux development community. It’s part of the Linux auditing system that is supported by most Linux distributions. The audit daemon provides in-depth logs of processes. Here is an example of a log:

type=SYSCALL msg=audit(1623654476.149:5467642): arch=c000003e syscall=56 success=yes exit=115384 a0=1200011 a1=0 a2=0 a3=7f0f6a765b90 items=0 ppid=1 pid=1159 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="postmaster" exe="/usr/pgsql-11/bin/postgres" subj=system_u:system_r:unconfined_service_t:s0 key="MyMonitor"

This record contains quite a lot of useful information: 

• Syscall arguments for the process 
• The identifier of the user that launched the process 
• System time 
• A full path to the process image 

The easiest way to read such logs is by using the ausearch tool via commands like these:

auditctl -a always,exit -F arch=b32 -S exit,fork,execve,clone,vfork,exit_group -F key=MyMonitor

Where: 

• -k filters by a key specified in rules
• --checkpoint specifies the checkpoint file; if the file is specified, the audit daemon generates only records that are newer than the previous checkpoint time

Let’s examine the pros and cons of using the audit daemon for process monitoring.

Key benefits of working with auditd are:

• Robust set of features. The Linux audit daemon provides you with log rotation, numerous tools for filtering and searching, the ability to adjust kernel and user mode buffer sizes, etc. 
• No risks for the Linux kernel. Auditd works in user mode and doesn’t introduce any kernel mode hooks that may crash the target system. There is also no dependency on the constantly changing Linux kernel interface. 
• Supported by most Linux distros. Operating systems from CentOS 6.1 with the 2.x kernel to the latest Ubuntu 22.04 support the Linux audit daemon. 

When working with the audit daemon, be ready to face the following challenges:

• Kernel-related limitations. Since auditd works in user mode, it can’t track processes invoked in kernel mode. It also depends on CONFIG_AUDIT, CONFIG_AUDITSYSCALL, CONFIG_BSD_PROCESS_ACCT, and CONFIG_BSD_PROCESS_ACCT_V3 kernel options, which are enabled by default.
• Potentially inaccessible process image files. When auditd handles the process record, its image file can be inaccessible if a user or some program has deleted or moved the corresponding .exe file. 
• Support for a single auditor. The Linux auditing system supports only one auditor, which is auditd by default. If someone installs their own auditor, it breaks auditd functionality. 
• Possible rule conflicts. The Linux audit daemon installs new auditd rules, which can conflict with other solutions installed on the device or with administrator activity. 

You can monitor processes in auditd using rules and syscalls. Let’s take a closer look at how to use the Linux audit daemon.

Read also

Controlling and Monitoring a Network with User Mode and Driver Mode Techniques: Overview, Pros and Cons, WFP Implementation

Uncover expert insights on maximizing network management efficiency with Apriorit’s guide. Learn how to implement powerful user-mode and driver-mode techniques to optimize performance and enhance security.

Learn more

How to configure rules and syscalls for the Linux audit daemon

To get records about processes, you need to set up rules as shown below:

-a always,exit -F arch=b32 -S exit,fork,execve,clone,vfork,exit_group -F key=MyMonitor

You can add such a rule using auditctl commands similar to this one:

auditctl -a always,exit -F arch=b32 -S exit,fork,execve,clone,vfork,exit_group -F key=MyMonitor

Rules also can be added permanently to audit.rules at /etc/audit/audit.rules or in some systems at /etc/audit/rules.d/audit.rules.

Alternatively, you can use syscall numbers instead of syscall names. In this case, the kernel intercepts syscalls according to the rules, generates records, and inserts them into the kernel ring buffer. Then, the Linux audit daemon takes out the records and adds them to the logs. 
If the daemon can’t pull out the record before the ring buffer overflows, some entries will be lost and the daemon will add the log entry audit: backlog limit exceeded. The buffer size is called the backlog limit, and you can adjust its runtime value with this command:

> auditctl -b $NUM.

You can adjust it permanently in the audit.rules file mentioned above.

Read also

Hooking Linux Kernel Functions, Part 1: Looking for the Perfect Solution

Master Linux function hooking to enhance your product functionality and enable adaptability. Explore our guide for practical strategies and leverage them for your project!

Learn more

Syscalls to monitor processes

For process monitoring, you need to use the following syscalls:

• clone — create a child process with possible changes to the original process
• fork — create a child process by duplicating the original process
• vfork — create a child process and block the parent process
• execve — execute the process
• exit — terminate the process
• exit_group — terminate all threads within the process

Here are the syscall tables for these syscalls:

// arch\x86\syscalls\syscall_64.tbl 
enum class x64Syscalls 
{  
	eSys_clone = 56, 
	eSys_fork = 57, 
	eSys_vfork = 58, 
	eSys_execve = 59, 
	eSys_exit = 60, 
	eSys_exit_group = 231, 
}; 
  
// arch\x86\syscalls\syscall_32.tbl 
enum class x32Syscalls 
{ 
	eSys_exit = 1, 
	eSys_fork = 2, 
	eSys_execve = 11, 
	eSys_clone = 120, 
	eSys_vfork = 190, 
	eSys_exit_group = 252, 
};

Note: In this code example, we reference both 64-bit and 32-bit tables. That’s because we’re using a 64-bit Linux distribution, and such distributions have the CONFIG_IA32_EMULATION option enabled by default. That means they have 32-bit syscall tables too. 

These syscall tables are located at arch\x86\syscalls\syscall_64.tbl and arch\x86\syscalls\syscall_32.tbl respectively. 

For clone, fork, and vfork syscalls, the typical output looks like this:

type=SYSCALL msg=audit(1623870284.582:20569): arch=40000003 syscall=120 per=400000 success=yes exit=12878 a0=1200011 a1=0 a2=0 a3=0 items=0 ppid=5794 pid=6132 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=10 comm="postmaster" exe=/usr/pgsql-11/bin/postgres" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="MyMonitor"

Note that the exit parameter is the PID of the newly created process. The newborn process has the same .exe file as the parent process. 

The typical output for the execve syscall is:

type=SYSCALL msg=audit(1623870343.481:20900): arch=40000003 syscall=11 per=400000 success=yes exit=0 a0=b67cbd4c a1=9a7d9cd0 a2=bfc571bc a3=4 items=2 ppid=5812 pid=12988 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=10 comm="sh" exe="/bin/bash" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="MyMonitor"

Here we can get the new image and associate it with the existing process. 

A lot of processes get terminated by exit_group, which produces the following output:

type=SYSCALL msg=audit(1623870899.267:24250): arch=40000003 syscall=252 per=400000 a0=0 a1=0 a2=b77177c0 a3=b771425c items=0 ppid=14106 pid=14107 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=10 comm="ausearch" exe="/sbin/ausearch" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="MyMonitor"

The record obviously doesn’t have the exit parameter at all. 

For the exit syscall, there will be no record, since in this case, the process is not terminated via a syscall.

Creating rules and using syscalls in auditd is straightforward, but you may encounter some issues. Let’s take a look at them in the next section.

Related project

Developing a Custom Secrets Management Desktop Application for Secure Password Sharing and Storage

Learn how the Apriorit team engineered a custom secrets management solution that helped our client fortify their data protection, streamline access control, and enhance operational efficiency for their business!

Project details

Common process monitoring issues and how to solve them

Linux processes can be terminated without any exit calls at all, which is an issue for process monitoring. For example, if the process gets a SIGSEGV signal, that indicates a segmentation error. Thus, the kernel can’t force the process to call a syscall, and it kills the process forcibly. Another scenario is when a user presses CTRL+C, which makes the kernel send the SIGINT signal to the group of processes and terminate them without calling any syscalls. 

In these cases, you’ll get no information about process termination when reviewing syscalls. The Linux audit system has a handler for exiting a process, but it logs an exit event only if it’s executed in the context of a syscall:

You need a different approach for proper tracking of process exit events. The best possible user mode option is acct syscall, which is designed for collecting profile data about existing processes.

The syscall starts process accounting with acct_v3 records:

	struct acct_v3 { 
     	char  	ac_flag;  	/* Flags */ 
     	char  	ac_version;   /* Always set to ACCT_VERSION (3) */ 
     	u_int16_t ac_tty;   	/* Controlling terminal */ 
     	u_int32_t ac_exitcode;  /* Process termination status */ 
     	u_int32_t ac_uid;   	/* Real user ID */ 
     	u_int32_t ac_gid;   	/* Real group ID */ 
     	u_int32_t ac_pid;   	/* Process ID */ 
     	u_int32_t ac_ppid;  	/* Parent process ID */ 
     	u_int32_t ac_btime; 	/* Process creation time */ 
     	float 	ac_etime; 	/* Elapsed time */ 
     	comp_t	ac_utime; 	/* User CPU time */ 
     	comp_t	ac_stime; 	/* System time */ 
     	comp_t	ac_mem;   	/* Average memory usage (kB) */ 
     	comp_t	ac_io;    	/* Characters transferred (unused) */ 
     	comp_t	ac_rw;    	/* Blocks read or written 
                  (unused) */ 
     	comp_t	ac_minflt;	/* Minor page faults */ 
     	comp_t	ac_majflt;	/* Major page faults */ 
     	comp_t	ac_swaps; 	/* Number of swaps (unused) */ 
     	char  	ac_comm[ACCT_COMM]; /* Command name */ 
     };

This binary file contains all necessary information for the file exit monitoring event. 

Another issue you can encounter when working with auditd is the reordering of records. Here’s an example:

type=SYSCALL msg=audit(1623863313.002:43768): arch=40000003 syscall=11 success=yes exit=0 a0=8574d70 a1=857b048 a2=857f380 a3=857b048 items=2 ppid=1658 pid=22093 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=9 comm="ls" exe="/bin/ls" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="MyMonitor" 
type=SYSCALL msg=audit(1623863313.003:43769): arch=40000003 syscall=252 a0=0 a1=0 a2=0 a3=4ef25c items=0 ppid=1658 pid=22093 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=9 comm="ls" exe="/bin/ls" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="MyMonitor" 
type=SYSCALL msg=audit(1623863313.002:43767): arch=40000003 syscall=120 success=yes exit=22093 a0=1200011 a1=0 a2=0 a3=0 items=0 ppid=1654 pid=1658 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=9 comm="bash" exe="/bin/bash" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="MyMonitor"

Here, the clone syscall is recorded after the exit_group syscall, probably because ls is a very short-living process. When monitoring processes or building your own monitoring solution, you have to keep in mind the possibility of getting system records in an unusual order.

Conclusion 

Linux has built-in mechanisms to help you establish process monitoring from both user mode and kernel mode. Working with the kernel can damage your system, so you’ll want to focus on user mode commands and tools.

At Apriorit, we mostly use the Linux audit daemon (auditd) for our process monitoring needs. This tool is supported by many Linux distributions, is easy and safe to use, and provides you with lots of information about Linux processes. 

However, if you’re working on a dedicated process monitoring solution or need to track an unusual process behavior, you may need a more powerful tool. In this case, you can leverage the experience of our kernel and driver developers, who can perform miracles under Linux’s hood.