Getting started guide to server management using Puppet

Introduction

Puppet is an open source configuration management utility allowing the user to automatically and if required also remotely manage multiple systems and its configuration. Puppet is declarative, which means that user needs to only request a state of the service or resource a do not really have think about how this state will be achieved.

In other words imagine that you are a system administrator managing hundreds of systems and need to make sure that that certain resource like hello package is installed. In order to achieve this in a traditional way of system administration the admin user will need to undergo multiple checks such as current state of the package installation, type of the operating system platform, installation command to be used before the actual package installation takes place. Being puppet a declarative, user only needs to define the state of the desired package and puppet will take care of the rest. In an event that our package “hello” is installed puppet will take no action, whereas if package is not installed it will install it.

Scenario

In our scenario we are not going to run hundreds of operating systems and attempt to manage them. Our goal will be much simpler than that. In fact we are going to run only two separate systems running puppet master and puppet agent. Thus through the master puppet server we will attempt to configure a remote node and install “hello” package using puppet agent. This will be done with a bare minimum configuration possible.

Terminology

  • puppet master – central server that hosts and compiles all of agent configuration manifests
  • puppet agent – a service which runs on node and periodically check a configuration status with master puppet server and fetches a current up to date configuration manifest
  • manifest – configuration file which is exchanged between puppet muster and puppet agent
  • node – an operating system that puppet service runs on

Scenario Settings

Throughout this tutorial I will refer to both hosts simply as master and node1. Operating system used on both master and node1 instances is Debian 8 Jessie. Ubuntu Linux can also be used as an alternative to follow this tutorial. The underlying network configuration is irrelevant. However, it is expected that node1 can resolve the master host by its name and both hosts are connected and proper firewall settings are applied to allow puppet master and node1 agent to communicate:

root@node1:/# ping -c 1 master
PING master (172.17.0.1): 56 data bytes
64 bytes from 172.17.0.1: icmp_seq=0 ttl=64 time=0.083 ms
--- master ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.083/0.083/0.083/0.000 ms

NOTE: Read appendix on how to setup the above scenario effortlessly with Docker.

Pupper Master installation and configuration

Let’s begin with the installation of puppet master:

root@master:~# apt-get install puppetmaster-passenger

The above command will install Puppet along side with Apache and Passenger. Thus instead of using typical WEBrick server we will involve Apache Passenger to run puppet master on port 8140. The default and auto generated Apache Passenger configuration file can be located under /etc/apache2/sites-available/puppetmaster.conf:

# This Apache 2 virtual host config shows how to use Puppet as a Rack
# application via Passenger. See
# http://docs.puppetlabs.com/guides/passenger.html for more information.

# You can also use the included config.ru file to run Puppet with other Rack
# servers instead of Passenger.

# you probably want to tune these settings
PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
# PassengerMaxRequests 1000
PassengerStatThrottleRate 120

Listen 8140


        SSLEngine on
        SSLProtocol             ALL -SSLv2 -SSLv3
        SSLCipherSuite          EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:+CAMELLIA256:+AES256:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!IDEA:!ECDSA:kEDH:CAMELLIA256-SHA:AES256-SHA:CAMELLIA128-SHA:AES128-SHA
        SSLHonorCipherOrder     on

        SSLCertificateFile      /var/lib/puppet/ssl/certs/master.pem
        SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/master.pem
        SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
        SSLCACertificateFile    /var/lib/puppet/ssl/certs/ca.pem
        # If Apache complains about invalid signatures on the CRL, you can try disabling
        # CRL checking by commenting the next line, but this is not recommended.
        SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
        # Apache 2.4 introduces the SSLCARevocationCheck directive and sets it to none
        # which effectively disables CRL checking; if you are using Apache 2.4+ you must
        # specify 'SSLCARevocationCheck chain' to actually use the CRL.
        # SSLCARevocationCheck chain
        SSLVerifyClient optional
        SSLVerifyDepth  1
        # The `ExportCertData` option is needed for agent certificate expiration warnings
        SSLOptions +StdEnvVars +ExportCertData

        # This header needs to be set if using a loadbalancer or proxy
        RequestHeader unset X-Forwarded-For

        RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
        RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
        RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

        DocumentRoot /usr/share/puppet/rack/puppetmasterd/public/
        RackBaseURI /
        
                Options None
                AllowOverride None
                Order allow,deny
                allow from all
        

Looking at the above configuration file we can notice a number of SSL certificates auto generated based on the system’s hostname. Confirm that all listed certificate paths point to a correct puppet SSL certificates. Otherwise new SSL certificates will need to be generated. If you need to generate new certificates first, remove current certificates:

root@master:~# rm -rf /var/lib/puppet/ssl

Next, run puppet in the foreground to see your new certificates to get generated. When finished, stop the process with CTRL+C key combination:

root@master:~# puppet master --verbose --no-daemonize
Info: Creating a new SSL key for ca
Info: Creating a new SSL certificate request for ca
Info: Certificate Request fingerprint (SHA256): FA:D8:2A:0F:B4:0B:91:8C:01:AD:71:B4:49:66:1F:B1:38:BE:A4:4E:AF:76:16:D2:97:50:C8:A3:8F:35:CC:F2
Notice: Signed certificate request for ca
Info: Creating a new certificate revocation list
Info: Creating a new SSL key for master
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for master
Info: Certificate Request fingerprint (SHA256): 43:67:42:68:64:73:83:F7:36:2B:2E:6F:06:20:65:87:AB:61:96:2A:EB:B2:91:A9:58:8E:3F:F0:26:63:C3:00
Notice: master has a waiting certificate request
Notice: Signed certificate request for master
Notice: Removing file Puppet::SSL::CertificateRequest master at '/var/lib/puppet/ssl/ca/requests/master.pem'
Notice: Removing file Puppet::SSL::CertificateRequest master at '/var/lib/puppet/ssl/certificate_requests/master.pem'
Notice: Starting Puppet master version 3.7.2

^CNotice: Caught INT; calling stop

Before we start our puppet master, we first need to create a default blank configuration manifest:

root@master:~# > /etc/puppet/manifests/site.pp

All is ready to enable puppet master to start after reboot:

root@master:~# systemctl enable apache2
Synchronizing state for apache2.service with sysvinit using update-rc.d...
Executing /usr/sbin/update-rc.d apache2 defaults
Executing /usr/sbin/update-rc.d apache2 enable

and start puppet master by starting apache webserver:

root@master:~# service apache2 start  
[ ok ] Starting web server: apache2.
root@master:~#

Confirm that puppet is running

# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  20228  2016 ?        Ss   11:53   0:00 /bin/bash
root      1455  0.0  0.0  98272  4600 ?        Ss   12:40   0:00 /usr/sbin/apache2 -k start
root      1458  0.0  0.0 223228  1920 ?        Ssl  12:40   0:00 PassengerWatchdog
root      1461  0.0  0.0 506784  4156 ?        Sl   12:40   0:00 PassengerHelperAgent
nobody    1466  0.0  0.0 226648  4892 ?        Sl   12:40   0:00 PassengerLoggingAgent
www-data  1476  0.0  0.0 385300  5116 ?        Sl   12:40   0:00 /usr/sbin/apache2 -k start
www-data  1477  0.0  0.0 450880  5608 ?        Sl   12:40   0:00 /usr/sbin/apache2 -k start
root      1601  0.0  0.0  17484  1140 ?        R+   12:44   0:00 ps aux

and listening on port 8140:

# netstat -ant              
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp6       0      0 :::8140                 :::*                    LISTEN     
tcp6       0      0 :::80                   :::*                    LISTEN     
tcp6       0      0 :::443                  :::*                    LISTEN

Puppet node configuration

At the moment our master server is running and expecting requests from puppet agent and therefore it is time to install our puppet agent on node1:

# apt-get install puppet

Next, we need to configure puppet to act as agent by removing any master server default directives from its configuration file /etc/puppet/puppet.conf:
FROM:

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
prerun_command=/etc/puppet/etckeeper-commit-pre
postrun_command=/etc/puppet/etckeeper-commit-post

[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN 
ssl_client_verify_header = SSL_CLIENT_VERIFY

TO:

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
prerun_command=/etc/puppet/etckeeper-commit-pre
postrun_command=/etc/puppet/etckeeper-commit-post

[agent]
server = master

The above directive server = master defines a master server to be connected to by the puppet agent. Where word master in our case as a hostname which resolves to master server’s IP address:

# ping -c 1 master
PING master (172.17.0.43): 56 data bytes
64 bytes from 172.17.0.43: icmp_seq=0 ttl=64 time=0.226 ms
--- master ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.226/0.226/0.226/0.000 ms

Installation part is done and what is left is to enable puppet to start after reboot and start puppet:

# systemctl enable puppet
Synchronizing state for puppet.service with sysvinit using update-rc.d...
Executing /usr/sbin/update-rc.d puppet defaults
Executing /usr/sbin/update-rc.d puppet enable
root@node1:/# service puppet start
[ ok ] Starting puppet agent.

Furthermore, by default the agent is disabled after installation on new uncofigured hosts. To enable puppet agent we need to run:

root@node1:/# puppet agent --enable

Signing Agent Certificate

Both hosts master and node1 are up and running. The last set of configuration required to get both master and agent talking is to sign node1‘s certificate request. After we have started puppet agent on node1 a certificate sign request was issued to master server:

root@master:/# puppet cert list
  "node1" (SHA256) 2C:62:B3:A4:1A:66:0A:14:17:93:86:E4:F8:1C:E3:4E:25:F8:7A:7C:FB:FC:6B:83:97:F1:C8:21:DD:52:E4:91

By default each certificate sign request must be signed manually:

root@master:/# puppet cert sign node1
Notice: Signed certificate request for node1
Notice: Removing file Puppet::SSL::CertificateRequest node1 at '/var/lib/puppet/ssl/ca/requests/node1.pem'

At this stage, our master should host two signed certificates:

 
root@master:/# puppet cert list --all
+ "master" (SHA256) EE:E0:0A:5C:05:17:FA:11:05:E8:D0:8C:29:FC:D2:1F:E0:2F:27:A8:66:70:D7:4B:A1:62:7E:BA:F4:7C:3D:E8
+ "node1"  (SHA256) 99:DC:41:BA:26:FE:89:98:DC:D6:F0:34:64:7A:DF:E2:2F:0E:84:48:76:6D:75:81:BD:EF:01:44:CB:08:D9:2A

Triggering puppet configuration request

It is time to create a first configuration manifest. As already mentioned above we are now going to make sure that package hello is available on node1. Open a default manifest /etc/puppet/manifests/site.pp file on the master hosts and add the following simplistic node configuration:

package { "hello":
    ensure => "installed"
}

Our agent on node1 is set by default to retrieve master’s configuration every 30 minutes. If we do not wish to wait we can trigger configuration request manually:

root@node1:/# hello
bash: hello: command not found

Package hello is currently unavailable on node1. Trigger new configuration request manually:

root@node1:/# puppet agent --test
Info: Caching certificate_revocation_list for ca
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for node1
Info: Applying configuration version '1434159185'
Notice: /Stage[main]/Main/Package[hello]/ensure: ensure changed 'purged' to 'present'
Info: Creating state file /var/lib/puppet/state/state.yaml
Notice: Finished catalog run in 4.00 seconds

From the above output we can see that new configuration was applied and package “hello” is now available:

root@node1:/# hello
Hello, world!

Conclusion

The above text shown a simplistic puppet configuration procedure. However, it should serve as a starting point for multi node deployments. To add more nodes simply re-visit above Puppet node configuration section and Signing Agent Certificate sections of this article.

Troubleshooting

apache2: Could not reliably determine the server’s fully qualified domain name, using 172.17.0.43. Set the ‘ServerName’ directive globally to suppress this message

# echo "ServerName `hostname`" >> /etc/apache2/apache2.conf

Notice: Skipping run of Puppet configuration client; administratively disabled (Reason: ‘Disabled by default on new or unconfigured old installations’);
Use ‘puppet agent –enable’ to re-enable.

root@node1:/# puppet agent --enable

Appendix

Quick scenario settings using Docker

The linuxconfig/sandbox is a docker image containing a base text editing and networking tools to help you configure and troubleshoot you puppet master and agent.
First start puppet master:

# docker run -it -h master --name=master linuxconfig/sandbox /bin/bash

Once the puppet master is up and running start node1:

# docker run -it -h node1 --name=node1 --link master:master linuxconfig/sandbox /bin/bash