One of the things I both love and hate about my job is getting assigned new projects. They can be about anything and everything. A few months back I was given an assignment to create some chroot jails for a group of customers so that they could securely upload files with sftp. The requirement was that the customers needed to be able to upload file, but in a secure and private way. Customer One should not be able to see Customer Two's files, for example. And neither customer should be able to browse the filesystem of the server. I was also asked to define a process whereby our support staff could add new jails as needed.

I've used the chroot command many times, it's a very handy way of fixing servers that can't otherwise be fixed --- like when no one can remember the root password on some box that no one has touched for years. Simply boot the server with a live CD (Knoppix is a personal favorite, I have it loaded on a usb key that I carry with me everywhere), mount the main drive and then issue a command along the lines of:


chroot /mnt/hda1


Suddenly it's like you are root on the local filesystem and not on the live CD. I can then change passwords, run programs, and generally get myself into trouble. :-)

You can do a lot more with chroot than simply use it as part of your system recovery toolkit though. Chroot is _very_ useful on a day-to-day basis for jailing applications and users that should (or need) to be isolated from the rest of the system (and from each other).

To create a jail you create a folder that has a replication of the directory structure of a normal Linux box. The difference is that you only copy in the bare minimum of what you need into the directory structure. The way you find out what libraries and files an application needs is by using ldd or strace. If I can get away with it, I prefer using ldd since its output is much more readable, but there are times when both are needed to track down elusive libraries.

As an example, lets say I want to have a jail that contains the bash shell, just that and nothing more. It's a stupid example, but very easy.

I first need to create the directory that will house the jail:


daniel@bart:~$ mkdir /opt/jail


It's just an empty directory now, so we now need to add some apps to it. Bash lives in the /bin directory so I first recreate the /bin directory and then copy the bash binary into it like so:


daniel@bart:~$ mkdir /opt/jail/bin
daniel@bart:~$ cp /bin/bash /opt/jail/bin/


Then I run ldd on bash to find out what libraries it uses like so:


daniel@bart:~$ ldd /bin/bash
linux-gate.so.1 => (0xffffe000)
libncurses.so.5 => /lib/libncurses.so.5 (0xb7ec2000)
libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7ebe000)
libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7d7d000)
/lib/ld-linux.so.2 (0xb7f1f000)
daniel@bart:~$


I can ignore the linux-gate.so.1 file, it's a virtual object created by the Linux Kernel, but I need the others so I create the directories and then copy them over:


daniel@bart:~$ mkdir -p /opt/jail/lib/tls/i686/cmov
daniel@bart:~$ cp /lib/libncurses.so.5 /opt/jail/lib/
daniel@bart:~$ cp /lib/tls/i686/cmov/libdl.so.2 /opt/jail/lib/tls/i686/cmov/
daniel@bart:~$ cp /lib/tls/i686/cmov/libc.so.6 /opt/jail/lib/tls/i686/cmov/
daniel@bart:~$ cp /lib/ld-linux.so.2 /opt/jail/lib/


Now I can test the jail like so:


daniel@bart:~$ sudo chroot /opt/jail /bin/bash
bash-3.2#


The presence of the bash-3.2# prompt tells me I was successful.

This is a pretty useless jail right now, with only bash. I can 'cd' and 'pwd' and 'exit' and do other bash built-in commands, but I can't 'ls' or 'less' or 'cp' or 'mv' or anything else that is external to bash. Adding more binaries is the same as adding bash: copy the binary in and then ldd it to find its libraries and then copy them in. A lot of programs use the exact same libraries, so often you'll find that copying the binary in to an existing jail is all you need to do since the libraries will already be there.

Once I copy in a few binaries this could turn into a very nice sandbox. However, I had some specific needs when I started out building custom jails, and there are a few gotchas that the above jail doesn't come close to meeting:

1. Users assigned to a jail need to be automatically chrooted into their jail when they login
2. There should be no shell access --- we don't want them to be able to ssh in, they should only be able to use sftp and scp
3. This needs to be done yesterday, and it needs to be repeatable.

Number one is easy enough in theory, just create a false shell whose sole purpose is to chroot you into your jail. Yeah. Easy. Oh, and do it securely with no shortcuts that might open up security holes. Yup, definitely easy.

Number two actually is easy in theory and practice, just copy in the sftp and scp binaries and leave out the ssh binary and any shell binaries.

Number three is the real kicker. I needed a quick, repeatable process. Especially since once things are up and running the plan was to turn everything over to the support group so that they could create new jails as needed.

So what did I do? I went looking for something someone had already built and I found Jailkit.

There are other tools out there for automatically creating jails, so please do some searching and explore all options when choosing your own solution. Some of the things I like about jailkit are:

1. It has many pre-configured collections that you can mix and match to build the perfect jail. I call them jail-sets.
2. If none of the existing jail-sets meets your needs you can customize them or create new ones.
3. Creating multiple jails is easy and each jail can have multiple users assigned to it --- users can't belong to more than one jail though, but that's a good thing IMO.
4. Adding users to jails is very easy, almost to the point of "that's all there is to it?"
5. The fake shell that Jailkit uses to chroot users when they log in checks for potential security risks every time it is run and will disconnect the user if it notices anything bad (like the jail directory being setuid root, for example).
6. The Jailkit website has clear, easy to follow directions that walk you through several potential jail setups and it has a good troubleshooting section to help you when things don't work for some reason or other.

Here's what I did to create the jails on my server:

As far as the actual server is concerned, I didn't do much to it apart from installing a base ubuntu-server OS with the ssh-server option. Oh, and I also installed the build-essential meta-package so that I could compile and install Jailkit.

After downloading the tarball from the Jailkit website, Jailkit is installed from source like so:


daniel@bart:~$ wget http://olivier.sessink.nl/jailkit/jailkit-2.5.tar.gz
daniel@bart:~$ tar -zxvf jailkit-2.5.tar.gz
daniel@bart:~$ cd jailkit-2.5
daniel@bart:~/jailkit-2.5$ ./configure
daniel@bart:~/jailkit-2.5$ make
daniel@bart:~/jailkit-2.5$ sudo make install


The whole unpack-compile-install process is really quick. By default, Jailkit installs its binaries into /usr/sbin/ and the configuration and template files into /etc/jailkit/.

Once installed, creating jails is easy. As I said above, Jailkit comes with several default jail-sets. These are defined in /etc/jailkit/jk_init.ini and they include things like scp, sftp, apache, basicshell, and so on. As far as learning about what jail-sets are available and what each jail-set includes, the best thing I can say is to look through the /etc/jailkit/jk_init.ini file and familiarize yourself with them. This file is also where you can define your own custom sets.

You create a jail using the jk_init script and specifying where you want the jail to be located and what jail-sets you want included in the jail. The example below is what I use to create a jail that has sftp and scp functionality:


daniel@bart:~$ sudo jk_init -v /path/to/jail sftp scp jk_lsh


The jk_lsh jail-set provides a special shell that limits the binaries it will execute via a config file. That way, even if a user uploads their own binary file, they won't be able to execute it.

The jail at this point (assuming nothing went wrong) is technically a valid jail, but there aren't any users in it yet and because this jail has no 'shell', it is pretty useless to try and chroot into it manually to test it. The next order of business therefore is to create some users and add them to the jail. Once a user has been created and added to a jail using the method below, they will always be put in that shell whenever they log in. So you absolutely need to create special jail-only users. Do not put regular users into jails!


daniel@bart:~$ sudo groupadd companyname
daniel@bart:~$ sudo useradd -g companyname -m username
daniel@bart:~$ sudo passwd username
daniel@bart:~$ sudo jk_jailuser -m -j /path/to/jail username


The reason I create a group is because I am creating jails per client company on this server and it made sense to group (no pun intended) all the users that way. I also threw in the passwd command into the example above since the useradd command (on Ubuntu at least) doesn't set a password by default. Things may be different on your Linux distro of choice.

The jk_jailuser utility, when run as in the example above, changes the shell of an existing user to jk_chrootsh in the /etc/passwd file and it changes the home directory to /path/to/jail/./home/username. It also adds the user to a stripped down passwd file located at /path/to/jail/etc/passwd and adds the user's groups to a stripped down group file located at /path/to/jail/etc/group.

If you want to edit the two passwd files yourself you can do so. Just change the user's line in /etc/passwd to something resembling this:


username:x:1010:1010::/path/to/jail/./home/username:/usr/sbin/jk_chrootsh


and change the user's line in /path/to/jail/etc/passwd to something resembling this:


username:x:1010:1010::/home/username:/usr/sbin/jk_lsh


(I'm afraid of messing things up if I do it manually, so I use the jk_jailuser utility whenever possible.)

Now we need to allow the user to execute the scp and sftp programs. Edit the /path/to/jail/etc/jailkit/jk_lsh.ini file and add in some lines like so:


[group companyname]
paths= /usr/bin, /usr/lib/
executables= /usr/bin/scp, /usr/lib/sftp-server, /usr/lib/openssh/sftp-server, /usr/libexec/sftp-server


That's all there is to it. I now have a fully functioning jail and I can test it by connecting to the server using sftp or scp. When I sftp in and look around, I only see the directories below the jail directory, exactly the behavior I wanted.

Should you find later on that the jail needs to have some other binary in it, Jailkit also includes a handy copy command that you can use to copy additional files into jails like so:


daniel@bart:~$ sudo jk_cp /path/to/jail /path/to/command


Very simple, and it handles all of the locating of dependencies, libraries, and whatnot. The only gripe I have about the jk_cp command is that you have to specify jk_cp instead of doing it the way the cp command works, which is cp .

On occasion I have run into issues where Jailkit doesn't copy over everything properly, or it simply misses some libraries. In those cases I had to experiment with copying over additional libraries. Rather than reinvent the wheel, I'll just point you to the Jailkit website, which has a very nice page on troubleshooting jails. (it's in the links)

Once I had jailkit working on my jail server and I had a repeatable process for adding users, my next step was to create two _very_ simple shell scripts for creating jails and creating jailed users. These scripts are used by our support staff, who have limited sudo access to run only these scripts as root along with the passwd command (so that they can change jailed users' passwords, if need be).

The two scripts are called mkjail and jailuser. They're not all that pretty, and they could be considered dangerous if used by the wrong people. But for internal use by a select group of specifically authorized and trained people, they work fine.

With the server set up and the scripts in place and the staff trained my job was done. The server is under constant use and new jails are created whenever they are needed with no intervention needed on my part. Of course, the minute the project was completed I was immediately assigned three more . . . but that's what's great about working in technology, right? It's never boring, that's for sure. :-)

Admittedly, creating an sftp jail is probably not something that everyone will need to do, but jailkit has found its way into my sysadmin bag of tricks. Whenever I need to isolate a user or a process into a sandbox, and a Xen virtual machine is way too much overkill, the first thing I reach for is jailkit.