Make Peace With Your Processes: Part 4

5

The principle of basing as much as possible on Unix-like systems around files is a well-advised approach. It could be said that this principle also extends to the Process Table, which I have discussed in previous articles in this series. Consider, for example, the treasure trove of gems to be found if you delve deeply into the “procfs” pseudo-filesystem, located in root level “/proc” on your filesystem.

Everything Is A File

Elements of the innards of /proc can only be read from and not written to. The key file here is “/etc/sysctl.conf” where you can also change many tunable kernel settings so that they persist after a reboot. One not-so-trivial caveat is that, almost magically, any freshly entered parameters into /proc are usually set live instantly, so be careful!

Clearly, this approach has a number of advantages. There’s no messing about with stopping and starting daemons, but be warned that if you are the slightest bit unsure of making a change (especially to servers) then take a deep breath before doing so. Rest assured that a reboot will revert any changes that you make if they are not entered into the file “/etc/sysctl.conf”.

There are zillions of hidden corridors and secret rooms to explore inside /proc, and sadly we will only be able to look at a tiny percentage of them here. Needless to say, on a test virtual machine or development machine, you should spend a long time tweaking, fiddling, and breaking your current kernel’s procfs settings. If you’re like me, then you might even find such activity vaguely cathartic, and the immediacy of the changes will certainly appeal to the impatient.

You can, for example, look further into a particular process that you’ve found using the excellent ps command, as we’ve already seen. The path of Process ID 23022, for example, is simply “/proc/23022” in relation to /proc.

If we enter that directory, then we are shown (after some complaints that we don’t have access to parts of the directory structure if we’re not logged in as root) the contents is presented in Listing 1:

dr-xr-xr-x.   8 apache apache 0 Feb 26 03:15 .

dr-xr-xr-x. 144 root   root   0 Feb 11 13:31 ..

dr-xr-xr-x.   2 apache apache 0 Feb 26 04:03 attr

-rw-r--r--.   1 root   root   0 Feb 28 08:25 autogroup

-r--------.   1 root   root   0 Feb 28 08:25 auxv

-r--r--r--.   1 root   root   0 Feb 28 08:25 cgroup

--w-------.   1 root   root   0 Feb 28 08:25 clear_refs

-r--r--r--.   1 root   root   0 Feb 26 04:03 cmdline

-rw-r--r--.   1 root   root   0 Feb 28 08:25 comm

-rw-r--r--.   1 root   root   0 Feb 28 08:25 coredump_filter

-r--r--r--.   1 root   root   0 Feb 28 08:25 cpuset

lrwxrwxrwx.   1 root   root   0 Feb 28 08:25 cwd -> /

-r--------.   1 root   root   0 Feb 27 14:01 environ

lrwxrwxrwx.   1 root   root   0 Feb 28 08:25 exe -> /usr/sbin/apache2

dr-x------.   2 root   root   0 Feb 26 04:03 fd

dr-x------.   2 root   root   0 Feb 28 08:25 fdinfo

-r--------.   1 root   root   0 Feb 28 08:25 io

-rw-------.   1 root   root   0 Feb 28 08:25 limits

-rw-r--r--.   1 root   root   0 Feb 28 08:25 loginuid

-r--r--r--.   1 root   root   0 Feb 28 08:25 maps

-rw-------.   1 root   root   0 Feb 28 08:25 mem

-r--r--r--.   1 root   root   0 Feb 28 08:25 mountinfo

-r--r--r--.   1 root   root   0 Feb 28 08:25 mounts

-r--------.   1 root   root   0 Feb 28 08:25 mountstats

dr-xr-xr-x.   4 apache apache 0 Feb 28 08:25 net

dr-x--x--x.   2 root   root   0 Feb 28 08:25 ns

-r--r--r--.   1 root   root   0 Feb 28 08:25 numa_maps

-rw-r--r--.   1 root   root   0 Feb 28 08:25 oom_adj

-r--r--r--.   1 root   root   0 Feb 28 08:25 oom_score

-rw-r--r--.   1 root   root   0 Feb 28 08:25 oom_score_adj

-r--r--r--.   1 root   root   0 Feb 28 08:25 pagemap

-r--r--r--.   1 root   root   0 Feb 28 08:25 personality

lrwxrwxrwx.   1 root   root   0 Feb 28 08:25 root -> /

-rw-r--r--.   1 root   root   0 Feb 28 08:25 sched

-r--r--r--.   1 root   root   0 Feb 28 08:25 schedstat

-r--r--r--.   1 root   root   0 Feb 28 08:25 sessionid

-r--r--r--.   1 root   root   0 Feb 28 07:52 smaps

-r--r--r--.   1 root   root   0 Feb 28 08:25 stack

-r--r--r--.   1 root   root   0 Feb 26 03:15 stat

-r--r--r--.   1 root   root   0 Feb 26 03:15 statm

-r--r--r--.   1 root   root   0 Feb 26 04:03 status

-r--r--r--.   1 root   root   0 Feb 28 08:25 syscall

dr-xr-xr-x.   3 apache apache 0 Feb 27 11:41 task

-r--r--r--.   1 root   root   0 Feb 28 08:25 wchan

Listing 1: Inside “/proc/23022” we can see a number of pseudo files and directories for our web server.

You might want to think of this content as belonging to runtime system information. It has been said that /proc is a centralized config system for the kernel, and it’s easy to see that the directory contains a mountain of information for just one process. As suggested, rummaging through these directories and looking up which file does what might be described as therapeutic. Anyway, it’s well worth the effort.

Pseudo Filesystems

It’s hard to dismiss the power that /proc wields. Be aware, however, that there’s a lot going on inside your server when it is running, even if no one is hitting your website. As a result, wouldn’t it be sensible to separate the tricksy hardware settings from the kernel settings and Process Table?

Continuing with our “Everything Is A File” mantra, that’s exactly what Unix-type operating systems do. Step forward /dev.

When dealing with physical devices, whether they are connected to the machine or not, we turn to /dev and not /proc.

An abbreviated directory listing of /dev is shown in Listing 2.

drwxr-xr-x.  2 root root         740 Feb 11 13:31 block

drwxr-xr-x.  2 root root          80 Feb 11 13:31 bsg

lrwxrwxrwx.  1 root root           3 Feb 11 13:31 cdrom -> sr0

lrwxrwxrwx.  1 root root           3 Feb 11 13:31 cdrw -> sr0

drwxr-xr-x.  2 root root           2.5K Feb 11 13:31 char

crw-------.  1 root root            5,1 Feb 11 13:31 console

lrwxrwxrwx.  1 root root         11 Feb 11 13:31 core -> /proc/kcore

drwxr-xr-x.  4 root root          80 Feb 11 13:31 cpu

crw-rw----.  1 root root          10,  61 Feb 11 13:31 cpu_dma_latency

crw-rw----.  1 root root          10,  62 Feb 11 13:31 crash

drwxr-xr-x.  5 root root         100 Feb 11 13:31 disk

Listing 2: We can see an abbreviated list of some of the devices that /dev deals with.

What about another example of what “/dev” can do for us? Let’s take a look, for example, at the superb “lsof” utility. If you’re not familiar with lsof, then it’s unquestionably worth a look at. I’m a big fan. The abbreviation “lsof” stands for “list open files,” and its seemingly endless functionality is exceptionally useful.

Listing 3 shows output from “lsof” when looking up information relating to the /var/log directory. We can display this information by running the following command:


# lsof +D /var/log/


COMMAND PID   USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME

rsyslogd       1103   root    1w   REG  253,4     2743     19 /var/log/messages

rsyslogd       1103   root    2w   REG  253,4     1906     17 /var/log/cron

rsyslogd       1103   root    4w   REG  253,4      747     18 /var/log/maillog

rsyslogd       1103   root    5w   REG  253,4     1753     27 /var/log/secure

apache2       22856   root    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       22856   root    6w   REG  253,4        0 131104 /var/log/apache2/access_log

apache2       23022 apache    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       23022 apache    6w   REG  253,4        0 131104 /var/log/apache2/access_log

apache2       23024 apache    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       23024 apache    6w   REG  253,4        0 131104 /var/log/apache2/access_log

apache2       23026 apache    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       23026 apache    6w   REG  253,4        0 131104 /var/log/apache2/access_log

apache2       23027 apache    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       23027 apache    6w   REG  253,4        0 131104 /var/log/apache2/access_log

apache2       23028 apache    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       23028 apache    6w   REG  253,4        0 131104 /var/log/apache2/access_log

apache2       23029 apache    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       23029 apache    6w   REG  253,4        0 131104 /var/log/apache2/access_log

apache2       23030 apache    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       23030 apache    6w   REG  253,4        0 131104 /var/log/apache2/access_log

apache2       23031 apache    2w   REG  253,4      245 131095 /var/log/apache2/error_log

apache2       23031 apache    6w   REG  253,4        0 131104 /var/log/apache2/access_log

Listing 3: The output from the mighty “lsof” looks much like that from the ps command.

I am using this “lsof” example, because it highlights how a system weaves in and out referencing data from both /proc and /dev. I won’t pretend to understand the nuances.

From its manual, we learn that the versatile “lsof” transparently informs us of how it gathered such information about that directory, by telling us which files it references:

  • /dev/kmem — the kernel virtual memory device

  • /dev/mem — the physical memory device

  • /dev/swap — the system paging device

From what I can gather, these files change between varying Unix versions, but they should at least give you a taste of which file is responsible for which task.

As we can see /dev and /proc are useful for all sorts of things — including network information, devices (real or virtual), disks (loop disks and physical drives), and much more.

Next Time

So far, I’ve looked at the Process Table and pseudo filesystems, and I talked about /dev and /proc. Next time, in the final article of this series, I’ll examine some additional command-line tools that may come in very handy at some point in the future.

Read the previous articles in this series:

Part 1

Part 2

Part 3

Chris Binnie is a Technical Consultant with 20 years of Linux experience and a writer for Linux Magazine and Admin Magazine. His new book Linux Server Security: Hack and Defend teaches you how to launch sophisticated attacks, make your servers invisible and crack complex passwords.