The principle of basing as much as possible on Unix-like systems around files is a well-advised approach. It could be said that this principle also extends to the Process Table, which I have discussed in previous articles in this series. Consider, for example, the treasure trove of gems to be found if you delve deeply into the “procfs” pseudo-filesystem, located in root level “/proc” on your filesystem.
Everything Is A File
Elements of the innards of /proc can only be read from and not written to. The key file here is “/etc/sysctl.conf” where you can also change many tunable kernel settings so that they persist after a reboot. One not-so-trivial caveat is that, almost magically, any freshly entered parameters into /proc are usually set live instantly, so be careful!
Clearly, this approach has a number of advantages. There’s no messing about with stopping and starting daemons, but be warned that if you are the slightest bit unsure of making a change (especially to servers) then take a deep breath before doing so. Rest assured that a reboot will revert any changes that you make if they are not entered into the file “/etc/sysctl.conf”.
There are zillions of hidden corridors and secret rooms to explore inside /proc, and sadly we will only be able to look at a tiny percentage of them here. Needless to say, on a test virtual machine or development machine, you should spend a long time tweaking, fiddling, and breaking your current kernel’s procfs settings. If you’re like me, then you might even find such activity vaguely cathartic, and the immediacy of the changes will certainly appeal to the impatient.
You can, for example, look further into a particular process that you’ve found using the excellent ps command, as we’ve already seen. The path of Process ID 23022, for example, is simply “/proc/23022” in relation to /proc.
If we enter that directory, then we are shown (after some complaints that we don’t have access to parts of the directory structure if we’re not logged in as root) the contents is presented in Listing 1:
dr-xr-xr-x. 8 apache apache 0 Feb 26 03:15 . dr-xr-xr-x. 144 root root 0 Feb 11 13:31 .. dr-xr-xr-x. 2 apache apache 0 Feb 26 04:03 attr -rw-r--r--. 1 root root 0 Feb 28 08:25 autogroup -r--------. 1 root root 0 Feb 28 08:25 auxv -r--r--r--. 1 root root 0 Feb 28 08:25 cgroup --w-------. 1 root root 0 Feb 28 08:25 clear_refs -r--r--r--. 1 root root 0 Feb 26 04:03 cmdline -rw-r--r--. 1 root root 0 Feb 28 08:25 comm -rw-r--r--. 1 root root 0 Feb 28 08:25 coredump_filter -r--r--r--. 1 root root 0 Feb 28 08:25 cpuset lrwxrwxrwx. 1 root root 0 Feb 28 08:25 cwd -> / -r--------. 1 root root 0 Feb 27 14:01 environ lrwxrwxrwx. 1 root root 0 Feb 28 08:25 exe -> /usr/sbin/apache2 dr-x------. 2 root root 0 Feb 26 04:03 fd dr-x------. 2 root root 0 Feb 28 08:25 fdinfo -r--------. 1 root root 0 Feb 28 08:25 io -rw-------. 1 root root 0 Feb 28 08:25 limits -rw-r--r--. 1 root root 0 Feb 28 08:25 loginuid -r--r--r--. 1 root root 0 Feb 28 08:25 maps -rw-------. 1 root root 0 Feb 28 08:25 mem -r--r--r--. 1 root root 0 Feb 28 08:25 mountinfo -r--r--r--. 1 root root 0 Feb 28 08:25 mounts -r--------. 1 root root 0 Feb 28 08:25 mountstats dr-xr-xr-x. 4 apache apache 0 Feb 28 08:25 net dr-x--x--x. 2 root root 0 Feb 28 08:25 ns -r--r--r--. 1 root root 0 Feb 28 08:25 numa_maps -rw-r--r--. 1 root root 0 Feb 28 08:25 oom_adj -r--r--r--. 1 root root 0 Feb 28 08:25 oom_score -rw-r--r--. 1 root root 0 Feb 28 08:25 oom_score_adj -r--r--r--. 1 root root 0 Feb 28 08:25 pagemap -r--r--r--. 1 root root 0 Feb 28 08:25 personality lrwxrwxrwx. 1 root root 0 Feb 28 08:25 root -> / -rw-r--r--. 1 root root 0 Feb 28 08:25 sched -r--r--r--. 1 root root 0 Feb 28 08:25 schedstat -r--r--r--. 1 root root 0 Feb 28 08:25 sessionid -r--r--r--. 1 root root 0 Feb 28 07:52 smaps -r--r--r--. 1 root root 0 Feb 28 08:25 stack -r--r--r--. 1 root root 0 Feb 26 03:15 stat -r--r--r--. 1 root root 0 Feb 26 03:15 statm -r--r--r--. 1 root root 0 Feb 26 04:03 status -r--r--r--. 1 root root 0 Feb 28 08:25 syscall dr-xr-xr-x. 3 apache apache 0 Feb 27 11:41 task -r--r--r--. 1 root root 0 Feb 28 08:25 wchan
Listing 1: Inside “/proc/23022” we can see a number of pseudo files and directories for our web server.
You might want to think of this content as belonging to runtime system information. It has been said that /proc is a centralized config system for the kernel, and it’s easy to see that the directory contains a mountain of information for just one process. As suggested, rummaging through these directories and looking up which file does what might be described as therapeutic. Anyway, it’s well worth the effort.
Pseudo Filesystems
It’s hard to dismiss the power that /proc wields. Be aware, however, that there’s a lot going on inside your server when it is running, even if no one is hitting your website. As a result, wouldn’t it be sensible to separate the tricksy hardware settings from the kernel settings and Process Table?
Continuing with our “Everything Is A File” mantra, that’s exactly what Unix-type operating systems do. Step forward /dev.
When dealing with physical devices, whether they are connected to the machine or not, we turn to /dev and not /proc.
An abbreviated directory listing of /dev is shown in Listing 2.
drwxr-xr-x. 2 root root 740 Feb 11 13:31 block drwxr-xr-x. 2 root root 80 Feb 11 13:31 bsg lrwxrwxrwx. 1 root root 3 Feb 11 13:31 cdrom -> sr0 lrwxrwxrwx. 1 root root 3 Feb 11 13:31 cdrw -> sr0 drwxr-xr-x. 2 root root 2.5K Feb 11 13:31 char crw-------. 1 root root 5,1 Feb 11 13:31 console lrwxrwxrwx. 1 root root 11 Feb 11 13:31 core -> /proc/kcore drwxr-xr-x. 4 root root 80 Feb 11 13:31 cpu crw-rw----. 1 root root 10, 61 Feb 11 13:31 cpu_dma_latency crw-rw----. 1 root root 10, 62 Feb 11 13:31 crash drwxr-xr-x. 5 root root 100 Feb 11 13:31 disk
Listing 2: We can see an abbreviated list of some of the devices that /dev deals with.
What about another example of what “/dev” can do for us? Let’s take a look, for example, at the superb “lsof” utility. If you’re not familiar with lsof, then it’s unquestionably worth a look at. I’m a big fan. The abbreviation “lsof” stands for “list open files,” and its seemingly endless functionality is exceptionally useful.
Listing 3 shows output from “lsof” when looking up information relating to the /var/log directory. We can display this information by running the following command:
# lsof +D /var/log/
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 1103 root 1w REG 253,4 2743 19 /var/log/messages
rsyslogd 1103 root 2w REG 253,4 1906 17 /var/log/cron
rsyslogd 1103 root 4w REG 253,4 747 18 /var/log/maillog
rsyslogd 1103 root 5w REG 253,4 1753 27 /var/log/secure
apache2 22856 root 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 22856 root 6w REG 253,4 0 131104 /var/log/apache2/access_log
apache2 23022 apache 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 23022 apache 6w REG 253,4 0 131104 /var/log/apache2/access_log
apache2 23024 apache 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 23024 apache 6w REG 253,4 0 131104 /var/log/apache2/access_log
apache2 23026 apache 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 23026 apache 6w REG 253,4 0 131104 /var/log/apache2/access_log
apache2 23027 apache 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 23027 apache 6w REG 253,4 0 131104 /var/log/apache2/access_log
apache2 23028 apache 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 23028 apache 6w REG 253,4 0 131104 /var/log/apache2/access_log
apache2 23029 apache 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 23029 apache 6w REG 253,4 0 131104 /var/log/apache2/access_log
apache2 23030 apache 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 23030 apache 6w REG 253,4 0 131104 /var/log/apache2/access_log
apache2 23031 apache 2w REG 253,4 245 131095 /var/log/apache2/error_log
apache2 23031 apache 6w REG 253,4 0 131104 /var/log/apache2/access_log
Listing 3: The output from the mighty “lsof” looks much like that from the ps command.
I am using this “lsof” example, because it highlights how a system weaves in and out referencing data from both /proc and /dev. I won’t pretend to understand the nuances.
From its manual, we learn that the versatile “lsof” transparently informs us of how it gathered such information about that directory, by telling us which files it references:
-
/dev/kmem — the kernel virtual memory device
-
/dev/mem — the physical memory device
-
/dev/swap — the system paging device
From what I can gather, these files change between varying Unix versions, but they should at least give you a taste of which file is responsible for which task.
As we can see /dev and /proc are useful for all sorts of things — including network information, devices (real or virtual), disks (loop disks and physical drives), and much more.
Next Time
So far, I’ve looked at the Process Table and pseudo filesystems, and I talked about /dev and /proc. Next time, in the final article of this series, I’ll examine some additional command-line tools that may come in very handy at some point in the future.
Read the previous articles in this series:
Chris Binnie is a Technical Consultant with 20 years of Linux experience and a writer for Linux Magazine and Admin Magazine. His new book Linux Server Security: Hack and Defend teaches you how to launch sophisticated attacks, make your servers invisible and crack complex passwords.
