How to Utilize Ubuntu Logs for Troubleshooting

When troubleshooting system issues in Ubuntu Linux, the best way to get started is by perusing the system logs. Basically every system event is logged somewhere. This includes kernel messages, service status messages, and almost anything else you can imagine. It is not uncommon for system logs to quickly fill up with millions of entries over a short period of time. With all this information available, we just need to know how to find what we are looking for, and then can quickly pinpoint the culprit of pesky system issues in Ubuntu.

In this tutorial, you will learn about various methods to utilize system logs for troubleshooting issues on Ubuntu Linux. You will see different places that log entries can be stored, and how best to parse through the data and isolate the relevant information. This is the first step in a troubleshooting process, and will give you additional hints about how to trace the root cause of issues arising on your system. Let’s get started!

In this tutorial you will learn:

• How to use the journalctl command to view log entries
• How to view kernel related and service related logs
• How to utilize the log files within /var/log
• How to use AppArmor to generate logs for specific resources

Software Requirements and Linux Command Line Conventions

Category	Requirements, Conventions or Software Version Used
System	Ubuntu Linux
Software	journald, logrotate, AppArmor
Other	Privileged access to your Linux system as root or via the sudo command.
Conventions	# – requires given linux commands to be executed with root privileges either directly as a root user or by use of sudo command $ – requires given linux commands to be executed as a regular non-privileged user

Using systemd Journal for Viewing Logs

---

---

Ubuntu is one of the many operating systems built on top of systemd. Part of the systemd framework includes journald, which is a service that collects logs. Since almost all services are integrated into systemd, the journal daemon is able to collect tons of information about the processes running on Ubuntu. The kernel is also closely integrated with systemd, so kernel-specific messages will also be collected by journald.

The systemd journal service can be queried by using the journalctl command. This is how we view log entries and specify what type of logs we want to see. Let’s look at a few example commands to see how we can utilize this command for viewing logs in Ubuntu:

1. The most basic commad syntax for journalctl is to execute it with no extra options. The output will include everything your system has logged, which will probably prove too cumbersome to manually go through due to the sheer amount of data. Remember, it is pretty common for the log entries to number into the millions once a machine has been powered on for some time.

   $ journalctl
   

   Check the screenshot below to see the general format for the logs. The output will get piped to less automatically. Use the Enter key to scroll line by line, or Space to scroll page by page. To exit the log output, press q on your keyboard.

   

   NOTE
   Running journalctl by itself is not very helpful. Part of the trick to utilizing logs is understanding how to isolate the relevant information by using additional command options or piping to other tools such as grep.
2. One convenience of journald is that is that the log entries are stored separately for each system boot. This convention makes it easier to pinpoint errors that occurred, for example, on the most recent system boot. Use the --list-boots option to see a list of all the logs from previous boots.

   $ journalctl --list-boots
   

   The screenshot below shows that journalctl can access the logs from the past 11 boots of our system. Our current boot is indicated with number 0, and our previous boot is number 1.

   

3. Now let’s try viewing the logs that were recorded during the previous system boot. To do this, we will use the -b (boot) option and the number that corresponds to the most recent boot, or -1. You can specify a different number if you want to view the logs for an even older boot.

   $ journalctl -b -1
   

4. You can also use the boot ID to view the logs of a previous boot. This alphanumeric string will not change, unlike the sequential numbers which continue to increment after each boot.

   $ journalctl -b b4b4ea9fd1eb431699634b90e1c24d3f
   

5. Perhaps you need to find the log entries that occurred with a specific timeframe. For example, if you know that you encountered a system error within that last hour. In this case, the --since and --until options are helpful for isolating relevant log files from a certain timeframe. For example, to see all of the logs since yesterday:

   $ journalctl --since yesterday
   

   ---

   ---

6. You can also use the options in conjunction with each other. In this example we are checking all log entries from yesterday up until two hours ago:

   $ journalctl --since yesterday --until "2 hours ago"
   

7. Use the YYYY-MM-DD HH:MM:SS date format with these two options if you want to isolate log entries for a very particular timeframe:

   $ journalctl --since 2022-10-03 01:00:00 --until 2022-10-04 14:30:00
   

   This can be useful if you know for certain that an error or relevant event occurred somewhere during this time, and need to pinpoint when exactly it happened or see what data was logged when it did.

8. To see entries that have been logged for a particular system service, use the -u flag. For example, to see all entries logged by Nginx:

   $ journalctl -u nginx.service
   

   This would show all log entries that the Nginx service has recorded, including status messages, warnings, errors, etc.

9. To see some of the most recent entries, we can use the -n option. By default, this will show you the last 10 log entries.

   $ journalctl -n 10
   

   Note that this option would be most useful when combined with the -u option covered above. Otherwise, you will just be seeing the last allotted number of log entries for the entire system:

   $ journalctl -n 10 -u nginx.service
   

10. Viewing kernel logs with journalctl

    To see only kernel related messages logged in journald, we supply the -k option:

    $ journalctl -k
    

    But, since this option alone would return too many log entries, it can be helpful to only see those of a certain priority. The highest priority is level 0, and the lowest is level 7. The log levels are as follows:

    0: emergency
    1: alert
    2: critical
    3: error
    4: warning
    5: notice
    6: info
    7: debug
    

    More information here: Introduction to the Linux kernel log levels
    

    ---

    ---

11. Use the -p option to see logs of a certain level, plus any above it in priority. In this example, we will go with level 3, which is any entry marked as an error, critical, alert, or emergency.

    $ journalctl -p 3
    

12. Using additional commands with journalctl

    We can also make use of commands like grep to help find relevant data in the output of journalctl. For example, let’s look for entries that contain the text ‘GNOME’ (the name of our installed GUI). Note that the -i option just makes the search case insensitive:

    $ journalctl | grep -i gnome
    

13. Piping to the tail command is useful for checking the latest entries contained in the output:

    $ journalctl | grep -i gnome | tail
    

Viewing Log Files

Before the introduction of a universal logging system (journald), logs were typically stored inside files located in the /var/log directory. Perusing this directory does not prove as useful as it once did, since the kernel and most services will rely on journald to store log entries these days, and we already have the handy journalctl command for viewing those efficiently.

However, some services like the apt package manager, and system events like the boot itself will store entries in /var/log. Niche programs and custom scripts may also make use of this directory, so it can be worth checking into.

Simply use the cat or less command to view the log entries in a file. You may also need to make use of grep in order to isolate the exact information that you are searching for:

$ cat /var/log/service/file.log

Utilizing AppArmor Logs

AppArmor is a Mandatory Access Control permission system (MAC) implemented in various distributions, including Ubuntu Linux. AppArmor works by assigning “profiles” to processes, networking ports, files, and users. By utilizing AppArmor, users can audit and log what entities are attempting to access confined resources.

Here are some commands to interact with AppArmor and begin logging information about selected resources:

1. First, to check the current status of AppArmor:

   $ sudo apparmor_status
   

   Possible modes are enforced, complain, and unconfined.

2. To put a profile into complain mode (issues warnings but takes no action):
   

   ---

   ---

   $ sudo aa-complain /path/to/binary
   

3. To put a profile into enforce mode (generates logs but also enforces access control policies):

   $ sudo aa-enforce /path/to/binary
   

4. To view the log files generated by AppArmor, we can check the /var/log/dmesg log file:

   $ cat /var/log/dmesg | grep apparmor
   

   However, it would be more efficient to view related AppArmor log entries with the journalctl command as shown above.

Closing Thoughts

In this tutorial, we saw how to utilize logs for troubleshooting system issues on an Ubuntu Linux system. The go to method is usually going to be the journalctl command, which allows us to efficiently peruse kernel and service log messages. We learned how various options can help us to isolate the information we need in order to assist with troubleshooting. Other methods included the /var/log directory and the AppArmor service, which provide some additional ways to help with logging.