Linux DevCenter    
 Published on Linux DevCenter (http://www.linuxdevcenter.com/)
 See this if you're having trouble printing code examples


Scanning for Rootkits

by Oktay Altunergil
02/07/2002

In an earlier article, Understanding Rootkits, we talked about how to guard against system intrusions and rootkits. Although it is best to have a server that nobody can break into, in real life every server connected to a network is a potential target.

If a server is compromised, it is equally important to realize this before it's too late and take measures to survive the attack.

Is My Server Compromised?

Usually, the first sign that a server might be compromised is simple anomalies in the behavior of the server. One of the more common anomalies one might notice is a change in how one or more of the core system utilities behave. For instance, a command-line switch to 'netstat' or 'ps', which you used to use without a problem everyday, might start returning an error message. The reason for this is that intruders replace these utilities with versions designed to hide their malicious activities. The utility they replace your original one with might be a different version, or it could have been compiled with different options, and as a result, it does not have the same options you are used to.

Another anomaly that should raise a big red flag is a change in your bandwidth-usage patterns. If you or your hosting company routinely monitors your bandwidth usage, you might notice an increase in the amount of traffic your server is pushing compared with your normal traffic patterns. This is usually caused by intruders using your server to distribute copyrighted software, commonly known as 'warez'. Remember that you might be the target of a legal action as a result of such activities on your servers.

Ideally, a server administrator should not wait until all the alarms go off before a server is checked for signs of compromise, because the less time an intruder has the opportunity to spend on a server, the less damage he or she will be able to inflict. (Although a very malicious intruder can potentially wipe out the whole system seconds after gaining root-level access.) For this reason, it is important to conduct server-security audits periodically, and to know as soon as possible when a server is compromised.

Port Scanning

An intruder will almost always try to install a backdoor on a server in order to have easy access to it at a later point. Because the local system utilities will probably be infected, they can't be trusted to detect the intruder. However, a port scan initiated from an external machine might reveal open ports that should normally be closed. The de facto standard port scanner in the Unix world is nmap. A command line as well as a GUI version is available. Without going into the details of the tool itself, here's how to scan all ports on a server for listening services:

[root@leeloo root]# nmap -p 1-65535 192.168.0.1

Starting nmap V. 2.54BETA22 ( www.insecure.org/nmap/ )
Interesting ports on  (192.168.0.1):
(The 65531 ports scanned but not shown below are in 
  state: closed)
Port       State       Service
21/tcp     open        ftp
22/tcp     open        ssh
80/tcp     open        http
111/tcp    open        sunrpc


Nmap run completed -- 1 IP address (1 host up) scanned 
  in 169 seconds
[root@leeloo root]#

This particular variation of the command scans all available ports (1 to 65535 inclusive) on 192.168.0.1. While the open ports above can be trojaned versions of ftp, ssh, or http, a real trouble sign is when you have ports with numbers higher than a few thousand where very few legitimate services normally listen to incoming connections. It is possible, and advisable, to periodically scan your whole network and identify any differences there might be between the outputs for different time periods.

RPM as a Security Tool

RPM (Red Hat Package Manager) is a popular packaging system used by some of the major Linux distributors including Red Hat, Mandrake, and SuSE. Every time an RPM package is installed, the package manager puts information about the package and the individual files the package contains into a central RPM database. Although not foolproof, this information can be used for performing quick and easy system audits. You should not rely on the RPM method when it claims the system is clean, but there's little need to doubt when it reveals the system binaries have been modified.

RPM records and tests the checksums of all files in a package, including files that are expected to change over time such as various configuration files and other files such as passwd, shadow, and group files. This makes it hard to conduct a full check and make sense out of the results. You can make this task easier by running the test on a handful of important core packages. These packages include net-tools, fileutils, util-linux, procps, psmisc, and findutils, among others.

Here's how to check the signatures on an RPM package:

rpm -V procps

If there are no changes to the package files, you will be returned to the command line. If there are changes you will get an output like this:

S.5....T c /root/.bash_profile

The first column shows what kind of a change has been made to the particular file listed on the last column since the package was originally installed.

The meaning of the symbols are as follows (from the man page):

M   Mode differs (includes permissions and file type)
5   MD5 sum differs
D   Device major/minor number mis-match
L   readLink(2) path mis-match
U   User ownership differs
G   Group ownership differs
T   mTime differs

There's also an easy way to test all It can be used to detect changes in files not part of the base packages listed above. You can use rpm -Va > filename.txt to redirect the whole report to a file and try to identify any important changes using your own judgment.

You can also use the RPM utility to see if a particular file belongs to a package. This comes in handy when you find well-known binaries in unusual places.

rpm -qf /path/to/file will either tell you the file is not part of any package, or give you which package it is a part of. You can then run rpm -V on that particular package to detect changes.

Please keep in mind that you can only trust RPM when you get negative results. It is possible to modify the RPM binary, the operating system, or the RPM database so that the changes will not be detected. For this reason, it is a good idea to use the RPM method along with "chkrootkit".

chkrootkit Rootkit Scanner

chkrootkit is a collection of small utilities used to detect the presence of known rootkits on a system. Although it is possible to find stand-alone detection scripts for almost all rootkits, chkrootkit differentiates itself by its ability to detect a large number of different rootkits using a single application. In addition to detecting known rootkit signatures, it also runs some generic tests that might aid in discovering a rootkit that is not actually supported by the application.

Related Reading

Incident Response
By Kenneth R. van Wyk, Richard Forno

The chkrootkit package contains seven small applications. The main progam, chkrootkit, is used to detect known rootkit signatures. In addition to searching for a rootkit's default files, it also checks core system binaries for malicious content. The remaining applications in the package are ifpromisc, which helps to find out whether a network interface is in promiscuous mode (remember you can't trust netstat), chklastlog, chkwtmp, and check_wtmpx, all of which detect deletions in various log files, chkproc, which detects Loadable Kernel Module (LKM) trojans and hidden processes, and finally strings, which is a simple implementation of the Unix strings utility.

Before going into how to use chkrootkit, let's see how to obtain and install it. The official chkrootkit site is http://www.chkrootkit.org. Download the latest .tar.gz file from this Web site. After getting the file, put it in a temporary directory and decompress it (tar xfvz chkrootkit.tar.gz) and cd into the newly created directory (cd chkrootkit-0.35). Finally, type make sense to compile it. Almost always the application will compile without a problem, but if you get error messages try building the software on another, similar machine and moving the binaries to the server you want to scan. Although it is possible to copy the newly created binaries to a central location in order to install them, this is not recommended because the software is updated frequently with new capabilities and it's best to get a new copy each time.

After building the binaries, my personal preference is to run ./ifpromisc right away to see if any of the interfaces are in promiscuous mode, because in such a case I might want to pull the server off the network immediately. It is also a good idea to run ./chkproc right after ifpromisc to see if there are any hidden processes or LKM trojans on the system. After running these two stand-alone utilities, you can run the main program ./chkrootkit without parameters. This will perform the rootkit tests as well as the tests provided by the rest of the utilities in the package, including ifpromisc and chkproc, which we had run earlier.

Here's some sample output from chkrootkit:

[root chkrootkit-0.34]# ./ifpromisc
eth0 is not promisc
eth0:0 is not promisc
eth0:1 is not promisc
eth0:2 is not promisc
eth0:3 is not promisc
eth0:4 is not promisc
eth0:5 is not promisc
eth0:6 is not promisc
eth0:7 is not promisc
eth0:8 is not promisc
[root chkrootkit-0.34]#


[root chkrootkit-0.34]# ./chkrootkit
ROOTDIR is `/'
Checking `amd'... not found
Checking `basename'... not infected
Checking `biff'... not found
Checking `chfn'... not infected
Checking `chsh'... not infected
Checking `cron'... not infected
Checking `date'... not infected
Checking `du'... not infected
Checking `dirname'... not infected
Checking `echo'... not infected
Checking `egrep'... not infected
Checking `env'... not infected
Checking `find'... not infected
Checking `fingerd'... not found
Checking `gpm'... not infected
Checking `grep'... not infected
Checking `hdparm'... not infected
Checking `su'... not infected
Checking `ifconfig'... not infected
Checking `inetd'... not infected
Checking `inetdconf'... INFECTED
Checking `identd'... not found
Checking `killall'... not infected
Checking `login'... not infected
Checking `ls'... not infected
Checking `mail'... not infected
Checking `mingetty'... not infected
Checking `netstat'... not infected
Checking `named'... not infected
Checking `passwd'... not infected
Checking `pidof'... not infected
Checking `pop2'... not found
Checking `pop3'... not found
Checking `ps'... not infected
Checking `pstree'... not infected
Checking `rpcinfo'... not infected
Checking `rlogind'... not infected
Checking `rshd'... not infected
Checking `slogin'... not infected
Checking `sendmail'... not infected
Checking `sshd'... not infected
Checking `syslogd'... not infected
Checking `tar'... not infected
Checking `tcpd'... not infected
Checking `top'... not infected
Checking `telnetd'... not infected
Checking `timed'... not found
Checking `traceroute'... not infected
Checking `write'... not infected
Checking `aliens'... no suspect files
Searching for sniffer's logs, it may take a while... nothing found
Searching for t0rn's default files and dirs... nothing found
Searching for t0rn's v8 defaults... nothing found
Searching for Lion Worm default files and dirs... nothing found
Searching for RSHA's default files and dir... nothing found
Searching for RH-Sharpe's default files... nothing found
Searching for Ambient's rootkit (ark) default files and dirs... nothing found
Searching for suspicious files and dirs, it may take a while...
/usr/lib/perl5/site_perl/5.005/i386-linux/auto/mod_perl/.packlist /usr/lib/perl5/site_perl/5.005/i386-
linux/auto/MD5/.packlist /usr/lib/perl5/site_perl/5.005/i386-linux/auto/Quota/.packlist
/usr/lib/perl5/site_perl/5.005/i386-linux/auto/XML/Parser/.packlist /usr/lib/perl5/site_perl/5.005/i386-
linux/auto/Devel/Symdump/.packlist /usr/lib/perl5/site_perl/5.005/i386-linux/auto/Image/Magick/.packlist
/usr/lib/perl5/5.00503/i386-linux/.packlist /lib/... /lib/.../BitchX/.config.h
/lib/...
Searching for LPD Worm files and dirs... nothing found
Searching for Ramen Worm files and dirs... nothing found
Searching for Maniac files and dirs... nothing found
Searching for RK17 files and dirs... nothing found
Searching for Ducoci rootkit... nothing found
Searching for Adore Worm... nothing found
Searching for ShitC Worm... nothing found
Searching for Omega Worm... nothing found
Searching for anomalies in shell history files... nothing found
Checking `asp'... not infected
Checking `bindshell'... not infected
Checking `lkm'... ./chkrootkit: [: integer expression expected before -gt
not tested
Checking `rexedcs'... not found
Checking `sniffer'...
eth0 is not promisc
eth0:0 is not promisc
eth0:1 is not promisc
eth0:2 is not promisc
eth0:3 is not promisc
eth0:4 is not promisc
eth0:5 is not promisc
eth0:6 is not promisc
eth0:7 is not promisc
eth0:8 is not promisc
Checking `wted'... nothing deleted
Checking `z2'... user ap deleted or never loged from lastlog!

As you can see, on this system the interfaces are not promiscuous, but there is an infected file. Chkrootkit also detected a suspicious file in a '...' directory (/lib/.../BitchX/.config.h). It is common for intruders to create directories whose names start with a decimal dot character, which makes them invisible or hard to find. You should also be looking for ' .' (space dot) files, which are difficult to differentiate from a '.' (current dir) on a terminal screen. The error on the 'Checking lkm' line is nothing to worry about because that's the error message you get when your kernel does not support loadable kernel modules, and there's nothing wrong with that--it is an almost desirable configuration for a server. Finally, the last line shows that a user 'ap' has either not logged in or was deleted later on from the log. I strongly recommend reading the FAQ on the chkrootkit Web site for more information.

A Rootkit Was Detected, Now What?

The only way you can be 100 percent sure your system is clean after a rootkit infection is if you format your drive and reinstall the operating system again from the original media. However, real-life conditions can force you to keep an infected system running at least for a little while. In such a case you should at least clean up your server and get rid of the rootkits and trojans.

Although the chkrootkit output is clear and to the point, it doesn't give you any pointers as to how you get rid of any given rootkit or trojan once it's detected. You need to familiarize yourself with the way rootkits work. You can do so by frequenting security-related Web sites such as cert.org or sites that explain how a specific rootkit works. This article at Teep's site has a comprehensive report on the anatomy of a particular rootkit that also provides a lot of valuable information applicable to other rootkits. Another way to familiarize yourself with rootkits is to find and install them on a test environment and compare the effects by using before-and-after snapshots of the system. If you know what a rootkit does, which directories and files it creates, and which binaries it infects, you can go to the infected server and remove the trojan files and replace the infected binaries with clean ones.

On RPM-based systems you can identify the infected packages as described earlier in this article. Once you have a list of the infected RPM packages, you can reinstall them using your original source.

The following syntax will force an RPM package to be reinstalled, overwriting the old files in the process:

rpm -U --force  rpm_package_name.rpm

After reinstalling all infected RPM packages, run the RPM and chkrootkit checks again to see if there are still any unresolved issues. If your system doesn't use RPM, you can follow a similar procedure that applies to the particular operating system you're running. If you're not using any packaging system at all, you'll need to recompile those applications from source and overwrite the old ones.

After reinstalling packages that are supposed to be on your system, you need to clean up alien files that are introduced to your system by the intruder. The chkrootkit output will help you identify the location of these files. Also, after cleaning up core system utilities such as ifconfig, netstat, find, ls, top, and ps you can gather more accurate information that might reveal even more alien files and processes. You should go through and remove all these files and directories. If you take a few minutes to read the config files that might be included in the rootkits, you might find clues as to what other trojan files and processes exist on the server.

After removing all the alien files you can identify, run 'top' and 'ps' again to identify and kill undesired processes that are still running. You can also use 'netstat -nap' on most systems to see the process name and ID that started a daemon. You might also want to check your initialization scripts and make sure no alien process is started using those scripts. Intruders usually place their startup lines at the bottom of these scripts but they can also hide them in the middle of the file to make it harder to identify. Some common locations that intruders use to place their startup calls are the inetd.conf file, rc.local file, and any other file in the initialization directory for the run-level you run the server at. You should also pay special attention to helper function files such as /etc/rc.d/init.d/functions, which other initialization scripts include, thereby allowing any valid command in them to be executed indirectly when the system is rebooted.

When you're confident there are no sniffers or trojans left running, change all the passwords on the server. If this is not possible at least change the passwords for users who have root or equivalent access to the server. Also go through your password file and remove any users that you think shouldn't be there, especially those with root equivalency. You may optionally reboot the server and perform another security scan to see if you were able to clean up all startup scripts and infected binaries.

Conclusion

Related article:

Understanding Rootkits -- Hackers have many tools that allow them to remain undetected during an attack. Understanding these tools is key to recognizing and cleaning up after an attack.

In theory, it is possible to detect and clean up all effects of a rootkit or a trojan on a compromised machine. However, it is extremely important to secure a system so that rootkits and trojans can't be introduced to the system in the first place. Most of the intrusion activity on the Internet targets servers that respond positively to a network scan probing for a specific operating system or software weakness. If you keep up with software patches and updates, you are less likely to be targeted in the first place because a potential intruder will usually pick a server that he or she can get into with the least effort, and there's never a shortage of these weak servers on the Internet.

Furthermore, if your server still gets broken into, you must unplug that server from the network immediately, reinstall the OS, and apply all security patches before introducing it to the network again.

The information I provide is not meant to replace common security practices, but might prove useful when used in conjunction with these practices.

May your servers always be rootkit free.

Oktay Altunergil works for a national web hosting company as a developer concentrating on web applications on the Unix platform.


Return to the Linux DevCenter.

Copyright © 2009 O'Reilly Media, Inc.