A fundamental design feature of Unix-like operating systems is that many of a system’s resources are accessible via the filesystem, as a file. For example the “procfs” pseudo-filesystem offers us access to all kinds of valuable treasures. In this series of articles, I’ll provide an overview of your system processes, explain how to use the “ps” command, and much more.

By querying files present in “/proc” you can quickly find out the intricacies of your network connections, the server’s CPU vendor, and look up a mountain of useful information, such as the command line parameters that were passed to your latest-and-greatest application when it fired up. This is because many of the functions of a server — such as a network connection — are really just another stream of data and in most cases can be presented as a file on your filesystem.

Let’s jump ahead for a moment in case you’re not too familiar with the “/proc” filesystem. If you knew that your Process ID was 16651, for example, then you could run this command to find out what was sent to the Puppet process to start it up:

# xargs -0 < /proc/16551/cmdline

The output from that command is:

/usr/bin/ruby /usr/bin/puppet agent

As you can see, Puppet’s agent is using the Ruby programming language in this case and the binary “/usr/bin/puppet” is passed the parameter “agent” to run it as an “agent” and not a “master”.

The “Everything Is A File” philosophy makes total sense if you think about it. The power harnessed within Unix’s standard tools (usually used for manipulating data held in the more common text files) such as “grep”, “awk” and “sed” are a major strength of the Operating System. But most importantly you can have a system’s components integrate very efficiently and easily if many things are simply files.

If you have you ever tried to look into a process running on a Unix-like machine then you’ll know that if anything the abundance of information adds confusion, rather than assists, if you don’t know where to look. There are all sorts of things to consider when you are eagerly trying to track down a rogue process on production machine.

In this series, I will attempt to offer a broad insight into how the Process Table can be accessed by the ps command and in combination with “/proc” and “/dev” how it can help you manipulate your systems.

Legacy

There are a few legacy stumbling blocks when it comes to looking up a process on different types of Unix boxes, but thankfully we can rely on the trusty “ps” command to mitigate some of these headaches automatically.

For example, Unix used the “ps” command by grouping its parameters together and prepending a hyphen. BSD, on the other hand, enjoyed grouping switches together but, for one reason or another, fell out with the hyphen entirely.

Throwing another spanner in the works, however, was good old GNU’s preference, in which its long options used two dashes. Now that you’ve fully committed those confusing differences to memory, let’s assume that the ps command does as much as it can by mixing and matching the aforementioned options in an attempt to keep everyone happy.

Be warned that occasionally sometimes oddities can occur, so keep an eye out for them just in case. I’ll try to offer alternative commands as we go along to act as a reminder that not all is to be taken exactly as read. For example, a very common use of the ps command is:

# ps -aux

Note,however, that this is indeed different from:

# ps aux

You might suspect, and would be forgiven for thinking as much, that this is purely to keep everyone on their toes. However, according to the ps command’s manual, this is apparently because POSIX and UNIX insist that they should cater to processes owned by a user called “x”. However, if I’m reading the information correctly, then if the user “x” does not exist, then “ps aux” is run. I love the last sentence of the manual’s definition and draw your attention to it as a gentle warning: “It is fragile, subject to change, and thus should not be relied upon.”

Process Tree

Enough eye strain for a moment; let’s begin by looking at the ps command and what it can help with in relation to querying the Process Table.

For starters (and I won’t be betting the ranch on this statement), it’s relatively safe to assume that upper- and lowercase mean the same thing.

If you’ve never seen the output of a Process Tree, then it might help with understanding “child” threads, which live under a “parent” process. In this case, the command is simply:

# pstree

The not-so-impossible-to-decipher output from that command is shown in Listing 1 (on a server not running “systemd” but good, old “init” (which is always Process ID (PID) number one, as an aside):

init-+-auditd---{auditd}

    |-certmonger

    |-crond

    |-dbus-daemon---{dbus-daemon}

    |-hald---hald-runner-+-hald-addon-acpi

    |                    `-hald-addon-inpu

    |-apache2---8*[apache2]

    |-master-+-pickup

    |        `-qmgr

    |-6*[mingetty]

    |-oddjobd

    |-rpcbind

    |-rsyslogd---3*[{rsyslogd}]

    |-sshd---sshd---sshd---bash---pstree

    |-udevd---2*[udevd]

Listing 1: Output from the “pstree” command showing parent processes and their children.

You can make your screen output much messier by adding the “-a” switch. Doing so will add command-line arguments (pulled from the /proc filesystem in the same way that our example did earlier). This is very useful, but you might want to do something like “grep” a specific process name from the output, as follows:

# pstree -ap | grep ssh


|-sshd,29076

 |   `-sshd,32365

 |       `-sshd,32370

 |               |-grep,8401 ssh

 |   |-sssd_ssh,1143 --debug-to-files

Listing 2: The command “pstree” showing only SSH processes with command line arguments and PIDs

As you can see from Listing 2, the command we are querying with is also shown (starting with “grep”) in the output so try not to let that trip you up. I’ve added the “-p” switch to display the PIDs, too.

One final look at this example is shown in Listing 3. Here, the all-pervasive “-Z” switch offers us any SELinux config associated with the parent and child detail displayed in our process table tree. That command for reference was:

# pstree -aZp | grep ssh


|-sshd,29076,`unconfined_u:system_r:sshd_t:s0-s0:c0.c1023'

 |   `-sshd,32365,`unconfined_u:system_r:sshd_t:s0-s0:c0.c1023'

 |       `-sshd,32370,`unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023'

 |               |-grep,8406,`unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023' ssh

 |   |-sssd_ssh,1143,`system_u:system_r:sssd_t:s0' --debug-to-files

Listing 3: The output now includes SELinux detail, command line arguments and PIDs

In this article, I provided a very brief introduction to the “ps” command. In the next several articles, I’ll show further options, examples, and details for using this powerful tool. Stay tuned.

Read Part 2 here.

Chris Binnie is a Technical Consultant with 20 years of Linux experience and a writer for Linux Magazine and Admin Magazine. His new book Linux Server Security: Hack and Defend teaches you how to launch sophisticated attacks, make your servers invisible and crack complex passwords.