Set up SSH access with session recording and containerized bastion servers: Part 1

With the vast majority of people working from home nowadays, remote access to systems is becoming the norm. Remotely managing and configuring servers securely is extremely critical for business continuity.

Most administrators rely on SSH (with or without a VPN) for remote administration. There has always been a demand for the ability to record user/administrator sessions for security and accountability reasons, as well as for knowledge-sharing purposes.

This series of posts covers out-of-the-box capabilities of Red Hat Enterprise Linux (RHEL) 8.2 (rootless containers with Podman, cgroups v2, systemd, ssh, tlog, and RHEL Identity Management) to put together a solution which helps administrators provide secure access to users. They can configure session-recording for some or all of the users along with centralized authentication (including two-factor authentication) and authorization (HBAC, centralized sudo) for the backend (target) servers.

Why containerized SSH for bastion servers?

Containers are generally considered suitable for stateless workloads, and SSH server is not one of the common workloads for containers. However, in the past, there have been some requests to the Red Hat support team to provide guidance to configure it.

Some of the benefits associated with this approach are:

• Run multiple SSH containers on the same VM to better utilize resources.
• Offer different SSH profiles and QoS to specific user groups.
• Provision/rebuild new SSH server containers on demand, whether for security (e.g., a new set of servers every day), availability (limited impact, smaller attack surface), or scalability (a sudden growth in the number of users needing access) reasons.
• Configure time-bound access to services for specific sets of users.
• Centrally record and retain user sessions and SSH access logs by utilizing persistent volumes without needing to make any changes on the backend servers.
• Multi-layer authentication structure for SSH with separate authentication and authorization mechanisms for bastion and backend servers.

Session recording

RHEL 8 introduced the capability to record users’ sessions by using tlog. It can be configured system-wide for all users or can be limited to specific users. This capability can be utilized within the bastion server containers to capture and record all user actions within the container and any subsequent servers accessed by the user via the containerized SSH server.

Why rootless containers?

Rootless containers are a new capability in RHEL 8. With the general availability of cgroups v2 in RHEL 8.2, some of the previous limitations associated with rootless containers have been mitigated. Running containers without root privileges enables improved security. In case of a compromised container, the attacker may have less access to the host, and the compromised container is easily replaced, or its credentials updated. With cgroups v2, containers also utilize systemd to start/stop services within rootless containers (previously, this only worked with containers running as root).

Utilizing systemd provides another advantage: admins can centrally capture all the logs (SSH access and user sessions) within the container via journald. These logs can be retained by mapping a persistent volume (/var/log/journal) from the host and can also be shipped to a central location (file-server or Elasticsearch) for later analysis. Session recording logs can be retained in a central location for the desired duration, and any of the user sessions can be replayed on demand.

User mapping and host-based access control with IDM

Red Hat IDM, based on the freeIPA project, is available free of charge to all Red Hat Enterprise Linux customers. It supports capabilities like centralized authentication and access management, host-based access control, AD integration, and two-factor authentication. This controls which users/groups can connect to which servers/groups-of-servers and what actions they can take using sudo, all from a central management plane.

By adding this access control as an additional layer behind the containerized bastion servers, administrators can not only control what actions users can take on a designated set of servers, but they can also have a complete audit trail with recorded user sessions.

High-level architecture

The following diagram depicts a proposed high-level architecture as a starting point for this solution. This design can be enhanced later using additional components, such as centralized file servers, Red Hat Satellite for patch management, and Ansible Tower for automating Day 2 Ops and jobs delegation.

Image

A minimum setup requires at least two RHEL 8.2 servers—one each to act as the bastion server and IDM server. Additionally, one or more RHEL (6, 7, 8) boxes are needed as the backend servers (target environment being managed or operated on).

Host preparation (bastion server)

As there is no pre-built image available for the containerized SSH server, we will create one using a custom Docker file. To build the image, a RHEL 8.2 server is required with the podman package installed and cgroups v2 enabled to use systemd with rootless containers.

The following steps highlight this process:

1. Check the operating system and Podman versions:

# cat /etc/redhat-release

# yum install -y podman slirp4netns containers-common

# podman version

2. Enable cgroups v2 on the RHEL server by adding the systemd.unified_cgroup_hierarchy=1 parameter to the boot command line and reboot the server, for example:

# grub2-editenv - set "kernelopts=root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap systemd.unified_cgroup_hierarchy=1"

3. After the reboot, confirm the cgroup v2 usage:

# findmnt -R /sys/fs/cgroup/

TARGET SOURCE

FSTYPE OPTIONS

/sys/fs/cgroup cgroup2 cgroup2 rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate

4. Set the container_manage_cgroup SELinux boolean to allow systemd to run properly in a container:

# setsebool -P container_manage_cgroup on

5. Increase the number of user namespaces:

# echo "user.max_user_namespaces=28633" > /etc/sysctl.d/userns.conf

# sysctl -p /etc/sysctl.d/userns.conf

6. Create a new user on the server and set the enable-linger option for it:

# useradd test2; id -u test2

# passwd test2

# loginctl enable-linger `id -u test2`

7. Validate that container storage is on the local filesystem. Remote file systems currently do not work well with unprivileged user namespaces. In this case, /home/test2 should be on a local file-system:

# podman info | grep GraphRoot

GraphRoot: /home/test2/.local/share/containers/storage

8. Log in to the Red Hat registry with your credentials. This should succeed:

# podman login https://registry.redhat.io -u <username> -p <password>

# podman pull registry.redhat.io/ubi8/ubi-init 

Wrap up

In this part of the series, we reviewed the reasons to set up the rootless container environment. We also examined the architecture, version requirements, and general technologies. In the next part of the series, we prepare the Docker file for the containerized SSH server, and test the container using it.

[ Getting started with containers? Check out this free course. Deploying containerized applications: A technical overview. ]