How to setup NRPE for client side monitoring

Nrpe, or Nagios Remote Plugin Executor, is the client side service of a monitoring setup. The monitoring server will send commands to the client, which listens passively when got no work to do. Upon incoming command, the nrpe checks it’s local configuration, and executes the plugin configured with the command, then sends back the results to the server for processing. You can read more about the server side installation in the Nagios installation guide, while this guide will focus on the client side.

In this tutorial you will learn:

How to install NRPE on Debian/Red Hat based distributions
How to configure NRPE to accept commands from the server
How to configure a custom check on the server and client side

NRPE – Nagios Remote Plugin Executor

Software Requirements and Conventions Used

Software Requirements and Linux Command Line Conventions
Category	Requirements, Conventions or Software Version Used
System	Ubuntu 18.04, Fedora 30
Software	Nagios 4.3.4, nrpe 3.2.1
Other	Privileged access to your Linux system as root or via the `sudo` command.
Conventions	# – requires given linux commands to be executed with root privileges either directly as a root user or by use of `sudo` command $ – requires given linux commands to be executed as a regular non-privileged user

Installing NRPE on Debian/Red Hat based distributions

Installing the required software is simple. We will cover Ubuntu, openSUSE, Fedora and RHEL.

Installing NRPE on Ubuntu

On Ubuntu, this process is a one-liner. The nrpe daemon’s package, called nagios-nrpe-server, is in the default repositories.

# apt-get install nagios-nrpe-server

In case of Ubuntu, the main configuration file is /etc/nagios/nrpe.cfg, the directory that is included by default is /etc/nagios/nrpe.d/, which can be used for drop-in configuration. The package also adds an empty local configuration file /etc/nagios/nrpe_local.cfg for convenience. This last one isn’t included in rpm based distributions.

Installing NRPE on openSUSE

On recent openSUSE versions, the nrpe software is also packaged in the default repositories. So the installation is a single linux command.

# zypper in nrpe

Unlike other distros, openSUSE places the main configuration file to the path /etc/nrpe.cfg.

Installing NRPE on Fedora

The Fedora Project also packages nrpe, and so it should be reachable from the default repositories. We’ll simply use dnf for installation.

# dnf install nrpe

The main configuration file will be /etc/nagios/nrpe.cfg, and the default included directory is /etc/nrpe.d/.

Installing NRPE on Red Hat Enterprise Linux

In case of RHEL, the nrpe package is not in the default repositories. You’ll need to enable the EPEL repository in order to install packages from there.

You can follow the steps described in the guide to enable EPEL repository, or import and publish the EPEL repositories contents, if you have a closed environment with internal software distribution. In either way, after the repository is available to the client machine, the installation process is quite the same as above.

# yum install nrpe

Configuration files are at the same place as in case of Fedora.

WARNING
Always do careful testing before enabling a new repository to a production environment. In this case, EPEL may contain packages that could be seen as updates for Red Hat packages, resulting in unexpected software changes on the system when running a full update.

Configuring NRPE to accept commands from the server

To configure the client service, we could use the main configuration file, but I recommend using a custom file and placing it into a directory that is included in the main configuration file. This way updates that came from a package upgrade on nrpe.cfg can be applied without changes to our custom configuration.

We can also include our own custom configuration file(s) in our custom packages, thus allowing updating client monitoring configuration in a centralized and automated way. Keeping that in mind, we’ll configure the client in /etc/nrpe.d/custom.cfg on all distributions in the following examples.

NRPE does not accept any commands other then localhost by default. This is for security reasons. To allow command execution from a server, we need to set the server’s IP address as an allowed address. In our case the server is a Nagios server, with IP address 10.101.20.34. We add the following to our client configuration:

allowed_hosts=10.101.20.34

Multiple addresses or hostnames can be added, separated by commas. Note that the above logic requires static address for the monitoring server. Using dhcp on the monitoring server will surely break your configuration, if you use IP address here. The same applies to the scenario where you use hostnames, and the client can’t resolve the server’s hostname.

Configuring a custom check on the server and client side

To demonstrate our monitoring setup’s capabilites, let’s say we would like to know if the local postfix system delivers a mail on a client for user root. The mail could contain a cronjob output, some report, or something that is written to the STDERR and is delivered as a mail by default. For instance, abrt sends a crash report to root by default on a process crash. We did not setup a mail relay, but we still would like to know if a mail arrives. Let’s write a custom check to monitor that.

Our first piece of the puzzle is the check itself. Consider the following simple bash script called check_unread_mail:
```
#!/bin/bash

USER=root

if [ "$(command -v finger >> /dev/null; echo $?)" -gt 0 ]; then
        echo "UNKNOWN: utility finger not found"
        exit 3
fi
if [ "$(id "$USER" >> /dev/null ; echo $?)" -gt 0 ]; then
        echo "UNKNOWN: user $USER does not exist"
        exit 3
fi
## check for mail
if [ "$(finger -pm "$USER" | tail -n 1 | grep -ic "No mail.")" -gt 0 ]; then
        echo "OK: no unread mail for user $USER"
        exit 0
else
        echo "WARNING: unread mail for user $USER"
        exit 1
fi
```
This simple check uses the finger utility to check for unread mail for user root. Output of the finger -pm may vary by version and thus distribution, so some adjustments may be needed.

For example on Fedora 30, last line of the output of finger -pm <username> is “No mail.”, but on openSUSE Leap 15.1 it would be “No Mail.” (notice the upper case Mail). In this case the grep -i handles this difference, but it shows well that when working with different distributions and versions, some additional work may be needed.
We’ll need finger to make this check work. The package’s name is the same on all distributions, so we can install it with apt, zypper, dnf or yum.
We need to set the check executable:
```
# chmod +x check_unread_mail
```
We’ll place the check into the /usr/lib64/nagios/plugins directory, the common place for nrpe checks. We’ll reference it later.
We’ll call our command check_mail_root. Let’s place another line into our custom client configuration, where we tell nrpe what commands we accept, and what need to be done when a given command arrives:
```
command[check_mail_root]=/usr/lib64/nagios/plugins/check_unread_mail
```

With this our client configuration is complete. We can start the service on the client with systemd. The service name is nagios-nrpe-server on Debian derivatives, and simply nrpe on other distributions.

# systemctl start nagios-nrpe-server
# systemctl status nagios-nrpe-server
● nagios-nrpe-server.service - Nagios Remote Plugin Executor
   Loaded: loaded (/lib/systemd/system/nagios-nrpe-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-09-10 13:03:10 CEST; 1min 51s ago
     Docs: http://www.nagios.org/documentation
 Main PID: 3782 (nrpe)
    Tasks: 1 (limit: 3549)
   CGroup: /system.slice/nagios-nrpe-server.service
           └─3782 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f

szept 10 13:03:10 mail-test-client systemd[1]: Started Nagios Remote Plugin Executor.
szept 10 13:03:10 mail-test-client nrpe[3782]: Starting up daemon
szept 10 13:03:10 mail-test-client nrpe[3782]: Server listening on 0.0.0.0 port 5666.
szept 10 13:03:10 mail-test-client nrpe[3782]: Server listening on :: port 5666.
szept 10 13:03:10 mail-test-client nrpe[3782]: Listening for connections on port 5666

Now we can configure the server side. If we don’t have one already, we can define a command that calls a remote nrpe instance with a command as it’s sole argument:

# this command runs a program $ARG1$ with no arguments
define command {
        command_name    check_nrpe_1arg
        command_line    $USER1$/check_nrpe -H $HOSTADDRESS$ -t 60 -c $ARG1$ 2>/dev/null
}

We also define the client as a host:

define host {
        use                     linux-server
        host_name               mail-test-client
        alias                   mail-test-client
        address                 mail-test-client
}

The address can be an IP address or hostname. In the later case we need to ensure it can be resolved by the monitoring server.

We can define a service on the above host using the Nagios side command and the client side command:
```
define service {
        use                        generic-service
        host_name                  mail-test-client
        service_description        OS:unread mail for root
        check_command              check_nrpe_1arg!check_mail_root
}
```
These adjustments can be placed to any configuration file the Nagios server reads on startup, but it is a good practice to keep configuration files tidy.
We verify our new Nagios configuration:
```
# nagios -v /etc/nagios/nagios.cfg
```
If “Things look okay”, we can apply the configuration with a server reload:
```
# systemctl reload nagios
```

Conclusion

If everything works, in a few minutes we should see our new client appear on the Nagios webpage, with it’s new service “OS: unread mail for root”, and with status as a green “OK” (that is, if there isn’t an unread mail for root).

The above scripts reports only warning if a new mail arrives on purpose: in the example environment it is not considered a critical issue, an application crash should have generate a critical error way before a mail arrives about it. In the background, the Nagios server passes the “check_mail_root” command to the client, where nrpe executes our custom script, which provides the output “OK: no unread mail for user root”, and the exit code 0 (which is translated by Nagios as “OK” state).

This simple setup aims to show the flow of commands and data in a Nagios+nrpe configuration, as well as explain the basic means of extending our monitoring capabilities. Countles checks (called plugins) are written in various languages for common usages, for example logfile parsing, database checks, webserver status information, and so on.

Many of them are also pre-packaged in the above mentioned repositories, and even more can be found on the official Nagios pages. While those are a great resource when we need to monitor something new, don’t take for granted that they will do exactly what you need out of the box. Tweaking their configuration and careful testing is needed in this case too, and if you find that a little modification may add some great feature/bugfix, don’t hesitate to contribute it back to the monitoring community. This is the way how it is built in the first place, after all.