Apache in a Bubble

– By Jeffrey Costa –
According to legend, the name Apache

was an amalgam of the phrase “a
patchy server”. Today the true “patchy” server is
Microsoft Internet Information Server (IIS), due to the quantity of
hotfixes, rollup patches, and service packs issued for it. Apache, by
comparison, has been relatively immune from the patch parade in the
past. However, as its adoption accelerates,
the number of new vulnerabilities seems to be increasing
proportionally. To reduce the risk of getting stung by a new (and
initially unpatched) vulnerability, you can run Apache in a “bubble”,
safely insulated from the other components of your web server (often
the true target of an attacker). This practice is often referred to
as running in a “chroot jail,” after the command chroot

which provides the functionality.

The jail is simply a stripped-down
filesystem running within your existing server. It assumes that an
attacker has gotten past your perimeter defenses (firewalls, routers,
intrusion detection), and has gained a foothold on your server via an
Apache vulnerability. But the attacker does not realize that he has
actually broken into a room with no furniture, no windows, and no
exit (except for the way he came in). Without the ability to leave
the “room”, the attacker cannot exploit other files,
data, or commands on the server, limiting the amount of additional
damage he can do.

As with any facet of security, chroot
jails trade off increased protection for convenience. It requires
work to setup the jail (as described in this paper), and diligent
maintenance to ensure the integrity of the jail going forward.
However, you will find that the security advantages gained far
outweigh the work required.

Assumptions

This paper assumes the following
conditions are true:

• Mandrake Linux 9.0 (or equivalent
  ) is the OS in use.

• The reader has previous experience
  building Apache from source code.

• Apache 1.3.27 will be used with a
  minimal configuration (i.e. no ssl, php, perl).

• Additional Apache hardening steps
  must be taken (which are beyond the scope of this document)

Overview

Linux and most other UNIX variants (but
not Windows) have a program known as chroot, which stands for “change
root.” It allows the administrator to change the root point of
the file system as it appears to a given process. In this case, the
process is the Apache web server, and the new root directory will be
a tightly-controlled file system we create solely to support Apache.

While most system processes can access
the entire file system, a chrooted process is restricted to the
directory it starts in. This isolation prevents any rogue process (or
user impersonating a process) from stepping up the directory tree and
gaining unauthorized access to server resources. It will appear to an
attacker that the existing directory is the root directory,
when in reality it is merely a subset of the full server file system.
The only functionality available to a chrooted process must come from
files placed there by the (admittedly paranoid) system administrator.

Erecting a jail involves creating a
directory structure, compiling Apache, copying the Apache daemon and
required support files into the jail, cleaning up the jail, and
altering your startup scripts to launch Apache inside the jail at
boot.

Prepare
the directory structure

The first step is to create the
directories for the jail. The jail will somewhat mimic the existing
file system of the server, as Apache will expect to find certain
files in expected places. Start by becoming root, then change to the
real root directory of the server, and execute the following
commands:

# mkdir /chroot

# cd /chroot

# mkdir bin dev etc lib usr

# mkdir usr/local

Build
Apache

Download the latest version of Apache
1.3.x as source to the /usr/local/src directory. Then build it with
default settings so that it installs into /usr/local/apache. At this
stage, you might also consider trimming the number modules down to a
minimum set. This reduces the footprint of the httpd binary and
provides fewer potential avenues of attack against the server.

Test that your installation succeeded
by issuing the command:

/usr/local/apache/bin/apachectl
start

(Note: You will get a
ServerName directive error, but Apache will start. Edit the
httpd.conf file to add an appropriate ServerName entry. Then shut
down the server.)

Add
support files

Apache depends upon a number of system
files for its functionality. Since the jail will be pinched off from
the rest of the file system, it becomes necessary to copy these
required files into the jail. It is important to note that we are NOT
trying to reproduce the richness of a standard Apache runtime
environment in the jail. Rather, the goal here is minimalism, adding
only what Apache needs to run. As a rule of thumb, if you are
contemplating adding a file to the jail, consider whether it might be
(mis)used for malicious purposes before doing so.

/bin/sh

The first file we will move into the
jail is the sh binary. The apachectl startup (shell) script
will expect to find this file in its usual location in the /bin
directory. Note, however, that sh, in turn, depends upon
several library files for its own functionality. You will need to
discover these shared libraries, and copy them into the jail.

The ldd (“list dynamic
dependencies”) command will list the library dependencies of
any file. Running it against the /bin/sh executable yields the
following output:

ldd
/bin/sh

libtermcap.so.2
=> /lib/libtermcap.so.2 (0x40016000)

libdl.so.2
=> /lib/libdl.so.2 (0x4001a000)

libc.so.6
=> /lib/i686/libc.so.6 (0x4001d000)

/lib/ld-linux.so.2
=> /lib/ld-linux.so.2 (0x40000000)

(Note: Your output here
may vary depending on your Linux distribution.)

Now that you have discovered the
required dependencies, copy them from their current location into the
jail:

# cp /lib/libtermcap.so.2 /chroot/lib

# cp /lib/libdl.so.2 /chroot/lib

# cp /lib/i686/libc.so.6 chroot/lib

# cp /lib/ld-linux.so.2 /chroot/lib

Note: we are physically copying files
into the jail. You should not be creating hard or soft links
to these files outside the jail, as such links could break the
integrity of the jail.

/dev/null

A secure Apache httpd.conf file points
the old-style directives of ResouceConfig and AccessConfig to
/dev/null for added security. For this (and other reasons), we must
recreate /dev/null in our jail. The first step is to discover the
current parameters of this pseudo-file, and subsequently recreate the
device in the jail with the same parameters:

# ls –iL /dev/null

This command should return something
similar to the following:

crw-rw-rw- 1 root root 1,
3 Dec 31 1969 /dev/null

Given this information, use the mknod
command to create this character device:

# mknod /chroot/dev/null c 1 3

/etc files

As a security measure, Apache drops to
impersonate user “nobody” after invocation by user root.
The default “nobody” user is defined in the /etc/passwd
file, and thus will need to be present in the jail for Apache to run
correctly. In addition, the corresponding group and shadow files must
be transferred:

# cp /etc/passwd /chroot/etc

# cp /etc/group /chroot/etc

# cp /etc/shadow /chroot/etc

Apache

Use ldd once again to discover the
shared modules Apache depends upon:

:

ldd
/usr/local/apache/bin/httpd

libm.so.6
=> /lib/i686/libm.so.6 (0x40017000)

libcrypt.so.1
=> /lib/libcrypt.so.1 (0x4003a000)

libc.so.6
=> /lib/i686/libc.so.6 (0x40085000)

/lib/ld-linux.so.2
=> /lib/ld-linux.so.2 (0x40000000)

Note that some of the libraries listed
here were previously moved inside the jail to support the sh
binary. This saves us the trouble of copying them again at this
stage. Simply copy the remaining libraries into the jail:

# cp /lib/i686/libm.so.6 /chroot/lib

# cp /lib/libcrypt.so.1 /chroot/lib

# cp /lib/libnss_files.so.2 /chroot/lib
(Note: not listed, but you will need this library)

Now that the support files are in
place, copy the Apache distribution to the jail, while preserving its
file attributes during the copy process:

cp –rp /usr/local/apache/
/chroot/usr/local

With Apache tucked safely inside the
jail, start the server:

chroot /chroot
/usr/local/apache/bin/apachectl start

Now that Apache is running, let’s
verify that we are actually running inside the jail. Issue theps –ax command, and write down one of the Apache PID
values you find there. Then issue ls–al
/proc/YOUR_PID/exe at
the command line. This should yield
the path to therunning
Apache binary in the chroot jail:
/chroot/usr/local/apache/bin/httpd*

Cleanup

The /etc/passwd file contains a number
of accounts that are not necessary in the confines of a chroot jail.
We will remove all of these unneeded entries for added security (be
certain to make a backup copy first). Open the passwd file for
editing, and remove all entries except the “nobody” user.

Once completed, change the nobody user
login to /bin/false from /bin/sh. This change grants the nobody user
a pseudo-shell not present on disk. Since the Apache user should
never need to log on anyway, this change is yet another security
measure. Perform the same editing on the shadow file, removing all
but the nobody user entry.

The same procedure must now be
performed to the group file. Remove all but the “nogroup”

entry from this file. After doing this, stop and restart Apache
inside the jail to ensure everything is working correctly. If so,
delete the backup copies of you made for the above steps.

Now that Apache is now running inside
the jail, there is no need for the originally compiled version to
remain. Delete the files from the standard Apache installation
directory: rm –rf /usr/local/apache. You may also wish
to remove the expanded source directory and compressed source TAR
file.

Good security practice also mandates
that we tightly control access to the files we have placed within the
jail. Begin with the files in the /chroot/etc directory by allowing
only root to modify these sensitive configuration files:

# chmod 400 /chroot/etc/*

Now add the chattr ‘i’
attribute so that the files in /etc cannot be deleted, renamed,
linked, or written to:

# chattr +i /chroot/etc/*

Now make the sh file
execute-only for all users:

# chmod 111 /chroot/bin/sh

Set the null device to its standard
permission level:

#chmod 666 /chroot/dev/null

Protect the Apache distribution; begin
by making the binaries execute-only for all users:

# chmod 111
/chroot/usr/local/apache/bin/*

Now make the configuration file
immutable:

# chattr +i
/chroot/usr/local/apache/conf/httpd.conf

Remove world read permissions from the
Apache log files:

# chmod 750
/chroot/usr/local/apache/logs/*

Enable
the jail at boot time

The chrooted Apache instance we have
just created will not survive a reboot. You must add a shell script
for your desired runlevel or add a line to the rc.local file. Another
good idea is to set the umask of the Apache process to a more
conservative 077 in your startup script.

Conclusions

The steps above will give you a base
Apache web server running in a chroot jail. As you add additional
functionality (ex: modules, SSL, Perl) to your server, you must
ensure you have the proper files in the jail to support the
functionality you desire. Most problems with chrooted processes tend
to be the result of missing files in the jail. Use of the built-in
strace utility to help diagnose such problems by tracing the
execution path.

In this exercise, we built Apache from
source and copied it into the jail. While we could have easily
customized Apache to install directly into the jail, we opted for
ease and speed of deployment. Now that you know how to jail Apache
manually, take a look at the Jail Chroot Project
[http://www.gsyc.inf.uc3m.es/~assman/jail/index.html] for a
faster method. This group is building a tool that automatically
builds a chrooted environment for you.

Suggestions

• Use the strip –s httpd
  command to remove all symbol and debugging information from the
  httpd binary, thereby reducing its size (and memory footprint).

• You may need additional files in
  /chroot/etc, such as hosts, nsswitch.conf, and resolve.conf,
  depending upon your particular Apache configuration.

• Create a dedicated partition on a
  separate disk dedicated to the Apache jail. Locating Apache in a
  separate partition will add another barrier against server
  compromise, forcing a remote attacker to cross drive partitions in
  the event of a compromise. This will also halt hard linking to files
  outside the jail (as links cannot cross partitions.). Be certain to
  set the partition to read-only on boot, and make it only as large as
  Apache needs (to limit free diskspace an attacker could use to write
  files)

• Consider building Apache
  statically, negating the need for shared library files.

• Don’t allow binaries to be
  added or created inside the jail (especially compilers).

• Remove any files from the jail
  that are not being used by Apache.

• Delete any other instances of
  Apache on the server.

• Ensure no SUID binaries exist in
  the jail. Search for them with this command:

find /chroot
-perm -02000 -o -perm -04000 –ls

• Create an owner for the Apache
  document tree that is separate from the “nobody” userid
  and groupid under which Apache runs. This means creating another
  entry in the files under /etc/ in the chroot jail to enable this
  segregation.

• Consider removing any unused
  Apache utility files in /bin. This might include the removal of the
  benchmarking tool (ab), the log resolver, and the htdigest and
  htpasswd utilities if you are not employing user authentication. Do
  not remove httpd or apachectl from this directory.

• Remove the icons, man, and proxy
  directories, if not needed.

• Remove the sample index.html.*
  files in the /htdocs directory.

• Remove all unused configuration
  files from the /conf directory.

• Remove the cgi-bin directory if
  unused. If CGI is required, remove the sample files printenv and
  test-cgi found there.

• Ensure that root owns all files
  that do not change regularly.

References/Resources

• Chrooting apache – securing
  your webserver further. Jim Hewlett

• Apache chroot mini HOWTO

• Chroot BIND HOWTO

• Chrooting daemons and system
  processes HOWTO

• Securing and Optimizing Linux: Red
  Hat Edition

• How to set up a chroot environment
  with RedHat Linux 6.2. Harold Welte.

• NCSA HTTPd – A chroot
  example.

• NCSA HTTPd – How to Use
  chroot for Increased Security

• Using chroot Securely. Anton
  Chuvakinhow

• How to Secure Services by Running
  in a Chrooted Environment. Eric Larson.

• Chroot man pages.

• How to “chroot’ an
  Apache tree with Linux and Solaris

Appendix
1

If you want a glimpse of how the jail
would appear to a user, activate it and move into it:

chroot /chroot /bin/sh

This will create /chroot as the new
root directory, and run the sh shell interpreter found inside the
jail. Now try the following commands:

# pwd

# ls –al

Note that neither of these commands
work as expected. This is because we did not explicitly add the pwd
and ls files to the jail. If we had, the functionality would
be available.

Now try to step up the directory tree.
You will be unable to do so, as you are grounded in the root
directory. You are trapped, and can go nowhere else on the system.
Type exit to escape the jail.

This
document is copyright© 2002 for Jeffrey Costa and it is a FREE
document. You may redistribute it under the terms of the GFDL.