How to troubleshoot SELinux policy violations

There are numerous different approaches to securing a computer. You can:

• Limit user privileges.
• Prevent users from logging in as root.
• Use antivirus and similar software.
• Use current versions of your operating system (OS) and applications.
• Remove unneeded software.
• Disable unnecessary services.

For Linux systems, SELinux is another option. This article will focus on diagnosing and addressing SELinux policy violations.

Get started

To start troubleshooting SELinux issues, confirm that the issue is an actual SELinux issue. For example, what if you configure a web server on a non-standard port (port 1234) and place the index.html file along a non-standard path, and now you cannot see the expected content? There could be several explanations:

• A permissions issue
• A misconfiguration in the httpd config file
• An SELinux issue
• A firewall might be blocking access to the port
• Other issues, such as network connectivity

When looking into a suspected SELinux issue, the typical troubleshooting steps still apply: Was this a previously working implementation? If yes, what changed? Examples could include the day and time you're trying to access the given service since there could be security rules controlling when something is available or accessible. What do the log files show on your computer, the remote web server, and points in between?

This article goes through the steps to investigate a web server that isn't displaying expected content.

Check the service status

First off, was this working before? This example is a new implementation, so it's never run before. Is the httpd service running? Use the systemctl status httpd command to check (from the server):

$ sudo systemctl status httpd
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2022-03-18 15:36:57 EDT; 2min 30s ago
Docs: man:httpd.service(8)
Process: 1346461 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
[...]
Mar 18 15:36:57 rhel8prod.usersys.redhat.com systemd[1]: httpd.service: Succeeded.
Mar 18 15:36:57 rhel8prod.usersys.redhat.com systemd[1]: Stopped The Apache HTTP Server.
Mar 18 15:36:57 rhel8prod.usersys.redhat.com systemd[1]: Starting The Apache HTTP Server...
Mar 18 15:36:57 rhel8prod.usersys.redhat.com httpd[1374493]: (13)Permission denied: AH00072: make_sock: could not bind to address [::]:1234
[...]

The above output shows that the service is not running (Active: failed) and that there is an issue binding to port 1234 (could not bind to address [::]:1234). This is the nonstandard port you configured. If you try to restart the service, you could get some helpful information:

$ sudo systemctl restart httpd
Job for httpd.service failed because the control process exited with error code.
See "systemctl status httpd.service" and "journalctl -xe" for details.

The first suggestion is to look at the status of the service. You just did that, but rerun it in case something important changed:

$ sudo systemctl status httpd
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2022-03-18 16:05:31 EDT; 1min 31s ago
[...]
Mar 18 16:05:31 rhel8prod.usersys.redhat.com httpd[1375987]: (13)Permission denied: AH00072: make_sock: could not bind to address [::]:1234
[...]

There isn't anything new there, so next try the journalctl command. You can follow the logs in one terminal as you run an action in another terminal. To do so, run journalctl like this:

$ sudo journalctl -xef

Use a few hard returns to get a clear break between the old logs and where the new ones will appear. Next, go into another terminal window and restart the service. As expected, the service fails to restart, so view what journalctl reported. The key lines are:

Mar 18 16:10:07 rhel8prod.usersys.redhat.com setroubleshoot[1376120]: SELinux is preventing httpd from name_bind access on the tcp_socket port 1234. For complete SELinux messages run: sealert -l 29ddb7b9-dac1-4cf3-bb09-3531e3e5621f

Mar 18 16:10:07 rhel8prod.usersys.redhat.com setroubleshoot[1376120]: SELinux is preventing httpd from name_bind access on the tcp_socket port 1234.
***** Plugin catchall (100. confidence) suggests **************************
If you believe that httpd should be allowed name_bind access on the port 1234 tcp_socket by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do allow this access for now by executing:

# ausearch -c 'httpd' --raw | audit2allow -M my-httpd

# semodule -X 300 -i my-httpd.pp

The message tells you SELinux is blocking the service on port 1234, so that's the next part of the investigation.

Check SELinux

One way to diagnose SELinux issues is to run sealert to get the messages for that event, and you can run the suggested ausearch, audit2allow, and semodule commands to allow access. OK, but what are those commands, and what will they do? Here is an explanation of all three (semodule is more involved and is covered below).

The ausearch command parses audit daemon logs. You can view the man page for all of the details, but the -c 'httpd' argument will search for any event with that httpd name. The –raw argument omits formatting from the output. This is useful when you want to use the information in other audit tools but not very useful when parsing the output manually.

[ Improve your skills managing and using SELinux with this helpful guide. ]

The audit2allow command generates an SELinux policy based on logs returned by ausearch. This tells you that the first command parses the audit logs for anything with an event based on httpd and then generates an SELinux policy to allow it. I'll review those commands step by step.

First, here's an abbreviated look at what ausearch displays (without the –raw argument, for readability):

$ sudo ausearch -c 'httpd'

…..

----

time->Fri Mar 18 16:10:04 2022

type=PROCTITLE msg=audit(1647634204.753:2789): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44

type=SYSCALL msg=audit(1647634204.753:2789): arch=c000003e syscall=49 success=no exit=-13 a0=4 a1=5628aace8980 a2=1c a3=7ffd7fd67afc items=0 ppid=1 pid=1376115 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0

[...]

If you look through it carefully, the output has comm="httpd" (as expected, because it was your search query).

Here's the result of running the first line of commands from the journalctl output:

$ sudo ausearch -c 'httpd' --raw | audit2allow -M my-httpd
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i my-httpd.pp

$ ls
my-httpd.pp my-httpd.te

$ cat my-httpd.te

module my-httpd 1.0;

require {
type httpd_t;
type monopd_port_t;
type var_t;
class tcp_socket name_bind;
class file { getattr map open read };
}

#============= httpd_t ==============
allow httpd_t monopd_port_t:tcp_socket name_bind;

#!!!! This avc can be allowed using the boolean 'domain_can_mmap_files'
allow httpd_t var_t:file map;
allow httpd_t var_t:file { getattr open read };

Manage policy modules with semodule

Next, look at the semodule command. This manages the policy modules used by SELinux. You can reload a policy, install a new one, remove one, or take other actions. For this example, use these commands to install the my-httpd module:

$ sudo semodule -i my-httpd.pp

$ sudo systemctl restart httpd

$ sudo systemctl status httpd
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2022-03-18 16:35:49 EDT; 4s ago
Docs: man:httpd.service(8)
[...]
Mar 18 16:35:50 rhel8prod.usersys.redhat.com httpd[1376906]: Server configured, listening on: port 1234

The service now starts, but if you look at the my-httpd.te file, it states you could use an already existing module:

#============= httpd_t ==============
allow httpd_t monopd_port_t:tcp_socket name_bind;

#!!!! This avc can be allowed using the boolean 'domain_can_mmap_files'
allow httpd_t var_t:file map;
allow httpd_t var_t:file { getattr open read };

Note that it is not referring to everything in the policy, just the one AVC (Access Vector Cache). To demonstrate that, you can disable the custom policy, enable the boolean, and then restart the httpd service. When you do that, the journalctl logs show the same SELinux issues:

Mar 18 16:46:55 rhel8prod.usersys.redhat.com setroubleshoot[1377422]: SELinux is preventing httpd from name_bind access on the tcp_socket port 1234. For complete SELinux messages run: sealert -l 29ddb7b9-dac1-4cf3-bb09-3531e3e5621f

Mar 18 16:46:55 rhel8prod.usersys.redhat.com setroubleshoot[1377422]: SELinux is preventing httpd from name_bind access on the tcp_socket port 1234.

When you enable the custom policy, the httpd service starts correctly:

$ sudo semodule -i my-httpd.pp

$ sudo systemctl restart httpd

$ sudo systemctl status httpd
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2022-03-18 16:52:36 EDT; 11s ago

After doing those steps, you can start the service, but when you try to access the site from your personal system, you receive an unable to connect error. To verify that the service actually starts, use curl on the web server:

$ sudo curl http://localhost:1234 | head

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4481 100 4481 0 0 486k 0 --:--:-- --:--:-- --:--:-- 486k
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<title>Test Page for the HTTP Server on Red Hat Enterprise Linux</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<style type="text/css">
/*<![CDATA[*/
body {
background-color: #fff;

This result shows the service works and responds, but since your system isn't able to access it and because the service is running on a nonstandard port, there might be a firewall issue. That's the next item on the troubleshooting list.

Check the firewall

Check the status of the web server's firewall and use the firewall-cmd command to display the open ports:

$ sudo systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)

Active: active (running) since Fri 2022-03-18 16:01:32 EDT; 1h 6min ago
..….

$ sudo firewall-cmd --list-ports
1433/tcp

This result shows that the firewall is running but port 1234 is not allowed. To allow it, use:

$ sudo firewall-cmd --add-port=1234/tcp --permanent
success

$ sudo firewall-cmd --reload
success

Now that the firewall is open, try accessing the site again. It still fails to show the expected content, but at least now you're getting the default Red Hat Enterprise Linux Test Page content. That tells you that the httpd service is working and you can access it, but something is blocking you from viewing the expected content. To troubleshoot this, look at the logs while you try another attempt. As with journalctl -xef (where it ran while you did an action in another window), use the tail -f command to display the log files while accessing the site. The -f option causes the output to update automatically. The key files are:

• /var/log/audit/audit.log for SELinux
• /var/log/messages for Linux system messages
• /var/log/secure for general messages as well as SELinux messages
• /var/log/httpd/* for httpd-specific logs

$ sudo tail -f /var/log/audit/audit.log /var/log/messages /var/log/secure /var/log/httpd/*

When you refresh the web browser, the logs display:

==> /var/log/httpd/error_log <==

[Fri Mar 18 17:13:23.132201 2022] [core:error] [pid 1378103:tid 140619527722752] (13)Permission denied: [client 10.22.32.214:53322] AH00035: access to /index.html denied (filesystem path '/data/website/index.html') because search permissions are missing on a component of the path


==> /var/log/httpd/access_log <==

10.22.32.214 - - [18/Mar/2022:17:13:23 -0400] "GET / HTTP/1.1" 403 4481 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0"


==> /var/log/audit/audit.log <==

type=AVC msg=audit(1647638003.130:2855): avc: denied { getattr } for pid=1378103 comm="httpd" path="/data/website/index.html" dev="dm-0" ino=34897893 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=file permissive=0

[...]

And then, a few seconds later, /var/log/messages shows these entries:

Mar 18 17:13:32 rhel8prod setroubleshoot[1378616]: SELinux is preventing httpd from getattr access on the file /data/website/index.html. For complete SELinux messages run: sealert -l 79e16649-2ee6-4f25-956b-d8e7bda307cd

Mar 18 17:13:32 rhel8prod setroubleshoot[1378616]: SELinux is preventing httpd from getattr access on the file /data/website/index.html.#012#012***** Plugin catchall_labels (83.8 confidence) suggests *******************#012#012If you want to allow httpd to have getattr access on the index.html file#012Then you need to change the label on /data/website/index.html#012Do#012# semanage fcontext -a -t FILE_TYPE '/data/website/index.html'#012where FILE_TYPE is one of the following: NetworkManager_exec_t, NetworkManager_log_t,

This tells you that the issue is related to SELinux. It's time to look at SELinux again.

Back to SELinux troubleshooting

To investigate the SELinux issues, first look at those logs. The important things to note are the AVC entry and those slightly delayed /var/log/messages entries. Use the ausearch command again to look at the AVCs and then look at those semanage and sealert commands from the /var/log/messages logs. The command this time will be like before, but you can use -m avc to limit the display for events matching that AVC message:

$ sudo ausearch -m avc -c httpd | tail -n 20

----

time->Mon Mar 21 14:59:59 2022

type=PROCTITLE msg=audit(1647889199.608:3076): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44

type=SYSCALL msg=audit(1647889199.608:3076): arch=c000003e syscall=4 success=no exit=-13 a0=7fe48c041c50 a1=7fe48a7fb890 a2=7fe48a7fb890 a3=7fe48a7fc4f0 items=0 ppid=1378098 pid=1415603 auid=4294967295 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)

type=AVC msg=audit(1647889199.608:3076): avc: denied { getattr } for pid=1415603 comm="httpd" path="/data/website/index.html" dev="dm-0" ino=34897893 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=file permissive=0

[...]

Do you see it? These logs reveal that there's an issue accessing the /data/website/index.html file:

avc: denied { getattr } …. scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:default_t:s0

This entry tells you that SELinux doesn't allow httpd to access an unconfined file.

Look at the sealert and semanage commands from logs. First, the sealert command gives you information specific to the blocked event:

$ sudo sealert -l 79e16649-2ee6-4f25-956b-d8e7bda307cd

This output gives two options—either use the semanage command to change the context or create your own custom policy that allows httpd to access this unconfined file. You don't want httpd to be able to access unconfined files. The issue is that your website uses a nonstandard location, so the computer doesn't have a default SELinux policy yet for files there. To correct this, look at the suggested semanage command.

[ Free cheat sheet: Get a list of Linux utilities and commands for managing servers and networks. ]

The semanage command can change the SELinux policy so that files created in the /data/website directory receive a default SELinux context suitable for a web server. The command is a little confusing, so be sure to look at the man page for semanage-fcontext since that has the perfect example:

$ man semanage-fcontext | grep -A4 EXAMPLE
EXAMPLE
remember to run restorecon after you set the file context
Add file-context for everything under /web

# semanage fcontext -a -t httpd_sys_content_t "/web(/.*)?"

# restorecon -R -v /web

These commands add a record of type httpd_sys_content_t for all files in /web. You want to work in /data/website, so change the location and run the command:

[root@rhel8prod website]# ls -lZd /data/website/
drwxr-xr-x. 2 root root unconfined_u:object_r:default_t:s0 24 Mar 18 17:02 /data/website/

[root@rhel8prod website]# ls -lZ /data/website/index.html
-rw-r--r--. 1 root root unconfined_u:object_r:default_t:s0 29 Mar 18 17:02 /data/website/index.html

[root@rhel8prod website]# semanage fcontext -a -t httpd_sys_content_t "/data/website(/.*)?"

[root@rhel8prod website]# ls -lZ /data/website/index.html
-rw-r--r--. 1 root root unconfined_u:object_r:default_t:s0 29 Mar 18 17:02 /data/website/index.html

[root@rhel8prod website]# ls -lZ /data/website/index.html
-rw-r--r--. 1 root root unconfined_u:object_r:default_t:s0 29 Mar 18 17:02 /data/website/index.html

[root@rhel8prod website]# restorecon -R -v /data/website
Relabeled /data/website from unconfined_u:object_r:default_t:s0 to unconfined_u:object_r:httpd_sys_content_t:s0

Relabeled /data/website/index.html from unconfined_u:object_r:default_t:s0 to unconfined_u:object_r:httpd_sys_content_t:s0

[root@rhel8prod website]# ls -lZd /data/website/
drwxr-xr-x. 2 root root unconfined_u:object_r:httpd_sys_content_t:s0 24 Mar 18 17:02 /data/website/

[root@rhel8prod website]# ls -lZ /data/website/index.html
-rw-r--r--. 1 root root unconfined_u:object_r:httpd_sys_content_t:s0 29 Mar 18 17:02 /data/website/index.html

[root@rhel8prod website]#

Note that it wasn't until you ran the restorecon command that your /data/website and /data/website/index.html were relabeled.

Now that you corrected the SELinux permissions, refresh the web browser:

==> /var/log/httpd/access_log <==

10.22.11.165 - - [21/Mar/2022:15:50:10 -0400] "GET / HTTP/1.1" 200 29 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0"

The expected content is displayed.

Everything works!

This article provides one example of troubleshooting an SELinux issue. To summarize the steps,

1. You connected via SSH to the web server to check whether the service was running. You saw that it wasn't running because of an issue with port 1234.
2. You used systemctl status httpd and journalctl -xef to see what happens when the httpd service restarts. The issue was that SELinux was blocking access to port 1234.
3. You piped ausearch output to audit2allow and then used semodule to install that new module.
4. You could restart the httpd service but couldn't access the site from your personal computer. You could access it using curl from the actual web server, so that told you there could be a firewall in the way.
5. You used firewall-cmd to open port 1234/tcp in the firewall. You could then access the web server, but still not the expected content.
6. You again tailed the relevant logs and refreshed the web browser. The logs showed an SELinux error because the /data/website/index.html file had the wrong SELinux context.
7. You could access the index.html file on the web server on port 1234 after setting that with semanage fcontext and then restoring the newly configured default context for the files in /data/website.

This approach to troubleshooting issues you suspect are related to SELinux helps you narrow the scope of the problem to likely culprits and adjust system settings without introducing new vulnerabilities.