Mystery infestation strikes Linux/Apache Web sites

According to a press release issued earlier this month by Finjan, a security research firm, compromised Web servers are infecting thousands of visitors daily with malware that turns their Windows machines into unwitting bots to do the bidding of an as yet unidentified criminal organization. Security firms ScanSafe and SecureWorks have since added their own takes on the situation, though with varying estimates on the number of sites affected. All reports thus far say the compromised servers are running Linux and Apache.

According to an article on ServerTune.com, the exploit involves a rootkit installed on the compromised server that replaces several system binaries with infected versions. When the system is booted, the infected binaries are executed, and as a result, dynamically created JavaScript payloads are randomly and intermittently served to site visitors. The malware JavaScript attempts to exploit vulnerabilites in Windows, QuickTime, and Yahoo! Messenger on the visitor’s machine in order to infect them.

We asked the Apache Software Foundation if it had any advice on how to detect the rootkit or cleanse a server when it’s found. According to Mark Cox of the Apache security team, “Whilst details are thin as to how the attackers gained root access to the compromised servers, we currently have no evidence that this is due to an unfixed vulnerability in the Apache HTTP Server.”

We sent a similar query to Red Hat, the largest vendor of Linux, but all its security team could tell us was that “At this point in time we have not had access to any affected machines and therefore cannot give guidance on which tools would reliably detect the rootkit.”

cPanel, a popular administration tool used by hosting companies that allows clients to manage their hosted sites, has posted a security note describing what the rootkit does after it’s installed, and suggests two ways to check a server for the rootkit.

According to cPanel, if you are unable to create a directory name beginning with a numeral — as in mkdir 1 — you’re infected. Another test is to monitor the packets from the server with the following tcpdump command:

tcpdump -nAs 2048 src port 80 | grep "[a-zA-Z]{5}.js'"

One great unknown thus far is how the servers come to be infected. Absent any forensic evidence of break-ins, the current thinking is that the malware authors gained access to the servers using stolen root passwords. The earliest known victims, according to quotes by researchers in this ComputerWorld story, were sites run by large hosting companies, which could give attackers root access to hundreds or even thousands of Web sites when compromised.

Other than using and safeguarding secure root passwords, not much can be done at this time to be proactive in preventing servers from being compromised, so searching techniques similar to the tcpdump command above, which check to see if a server has already been compromised, is probably the best course of action available to administrators. We haven’t found a good answer yet for disinfecting compromised servers, but a complete reinstall of Linux, Apache, and a new root password would certainly do the trick.