When programs running on Linux want to use the resources managed by the operating system (reading files, creating processes, etc.), they make system calls to the OS. System calls work at the kernel level and perform the necessary operations, leaving control back to the calling program. The strace tool provides the ability to trace these system calls on Linux.

Typical Use of the strace Command

To monitor system calls for an application, just invoke the command with strace in the following format:

        strace ls /tmp

However, there are often processes that start much earlier and continue to work in the background. Due to any problems, you might want to collect additional information associated with such processes. You can attach strace to any running application by giving the process ID of the process to the -p parameter:

        strace -p 2759
    

Output:

strace p command

Keep Track of Threads and Forks of an App

With strace, you can check all threads and other child processes that are a fork of the application using the -f flag.

        strace -f -p 2759
    

Output:

strace -f -p command

Check Certain System Calls With strace

The default strace output can be quite crowded to follow at times. If you only want to track certain system calls, you can do so with the -e parameter:

        strace -f -e trace=open,write,close,connect,select -p 19770
    

To trace only system calls related to file operations, use -e trace=file:

        strace -e trace=file -p 19770
    

To filter only network-related system calls, specify -e trace=network in the command:

        strace -e trace=network -p 19770
    

Get Time Information in Seconds

When outputting system calls, you can use the -t parameter to get time information with precision in seconds. Most of the time the precision will not be enough for your needs. In such situations, you can use the -tt parameter to get time information with microsecond precision:

        strace -tt ls /tmp
    
strace tt ls tmp command

Collect Statistics About System Calls

With the -c parameter, you can collect statistics about system calls for as long as you want:

        strace -f -c -p 19770
    
strace-f-c-p-command

Save Logs to a File

If you run strace for a long time and want to examine the resulting logs in more detail later, you'll need to save the logs. With the -o parameter you can specify the file in which strace should save the logs:

        strace -f -o /tmp/strace.log -e trace=file ls /tmp
    

ptrace Blocking Process

Using the prctl system call, any application under Linux can prevent itself from being controlled by non-root users using ptrace. If the application clears the PR_SET_DUMPABLE flag for itself via prctl, users other than root won't be able to control this application with ptrace, even if they have the right to signal the application.

One of the most typical uses of this feature is seen in the OpenSSH authentication agent software. Thus, the control of the application by another application with ptrace is prevented at user authentication.

ptrace and Security

Due to the ptrace facility set in the traditional Linux process model, any software you run on your system with your user has the authority to insert malicious code into it. From the simplest xterm tool to advanced web browser applications, such malware can take control of all your other running applications—thanks to the ptrace system call—and copy important information without you noticing.

In response to this situation, which many users are not aware of, a protection mechanism has been developed with the security module called Yama in the Linux kernel.

You can control the response to the ptrace system call via the /proc/sys/kernel/yama/ptrace_scope file. By default, this file writes a value of 0.

The following values are acceptable:

Value

Meaning

0

Conventional behavior: All applications that have the right to ptrace can be checked.

1

Restricted ptrace: Only the direct parent of the application or debug applications allowed by the application with the PR_SET_PTRACER option have the control. Thus, the uses of gdb program_name and strace program_name will continue to work, but you won't be able to attach a running application afterward.

2

Ptrace to the system administrator: Only applications with defined CAP_SYS_PTRACE property or child processes that define the PTRACE_TRACEME option with prctl can be controlled.

3

Completely disabled: No ptrace is allowed under any circumstances. If this property is defined once, you can't change it again at runtime.

Many developers do not know that applications can disable ptrace themselves via prctl, except for the root user. Although security-related software such as the OpenSSH agent performs these operations, it would not be right to expect the same behavior from all software running on the system.

Recently, some Linux distributions have started to set the default value of the ptrace_scope file, described above, to 1. Thus, with ptrace operations restricted, a safer working environment is provided throughout the system.

Using an Example strace

Register the sample application below with the name ministrace.c. Then you can compile it with the following command:

        gcc -o ministrace ministrace.c

Code:

        #include <sys/ptrace.h>
#include <sys/reg.h>
#include <sys/wait.h>
#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>

int wait_for_syscall (pid_t child)
{
int status;
while (1) {
ptrace(PTRACE_SYSCALL, child, 0, 0);
waitpid(child, &status, 0);
if (WIFSTOPPED(status) && WSTOPSIG(status) & 0x80)
return 0;
if (WIFEXITED(status))
return 1;
}
}

int do_child (int argc, char **argv)
{
char *args [argc+1];
memcpy(args, argv, argc * sizeof(char*));
args[argc] = NULL;
ptrace(PTRACE_TRACEME);
kill(getpid(), SIGSTOP);
return execvp(args[0], args);
}

int do_trace (pid_t child)
{
int status, syscall, retval;
waitpid(child, &status, 0);
ptrace(PTRACE_SETOPTIONS, child, 0, PTRACE_O_TRACESYSGOOD);
while(1) {
if (wait_for_syscall(child) != 0) break;

syscall = ptrace(PTRACE_PEEKUSER, child, sizeof(long)*ORIG_RAX);
fprintf(stderr, "syscall(%d) = ", syscall);

if (wait_for_syscall(child) != 0) break;

retval = ptrace(PTRACE_PEEKUSER, child, sizeof(long)*RAX);
fprintf(stderr, "%d\n", retval);
}
return 0;
}

int main (int argc, char **argv)
{
if (argc < 2) {
fprintf(stderr, "Usage: %s prog args\n", argv[0]);
exit(1);
}
pid_t child = fork();
if (child == 0) {
return do_child(argc-1, argv+1);
} else {
return do_trace(child);
}
}

After compiling the application, you can run any command with ministrace and examine the output:

ministrace-date-command

You Can Use strace for Many Purposes

strace can help find bugs in programs that unnecessarily use system resources. Likewise, the characteristic that a program exhibits while using operating system resources can also be revealed with strace.

Since strace directly listens to system calls, it can reveal runtime dynamics regardless of whether the code of the program being run is open/closed. It is possible to get an idea about why the programs throw an error when started using strace.

Similarly, strace helps you understand why a program terminates unexpectedly. Therefore, being familiar with strace is very important in Linux kernel development and system administration.