Security Checks During Possible Compromise - Part 1

We are not living in a perfect world, and it is possible for a slice to get hacked. However, we can find the culprit and make sure it won't happen again. In this article we will learn some techniques and tools we can use to investigate our slices if we suspect they've been compromised.


Slices can be compromised as a result of various factors: weak passwords, weak iptables rules, older versions of software with known exploits, and more. If your slice has been compromised, do not panic. Panic will lead to poor decisions, and then the situation could become worse.

Instead, try to understand what happened and make sure your slice will not get compromised again in the same manner. The main idea of this article is simple: learn from your mistakes and don't make the same mistakes twice.

This article is the first in our "Slice Investigation" series. At the end of this series, we'll learn how to track intruders on a compromised slice, and to backup our data and packages.

In this first article we will cover the things we can do before going into rescue mode (which is covered in the second article).

The slice used for this article series was running Ubuntu 8.10. However, the steps demonstrated will be similar for other Linux distributions.

Important Warning

Before you do anything, you need to make an important decision—do you plan to involve law enforcement and prosecute the attacker? If the answer is yes, you should leave the compromised system alone and make no changes to it.

Any changes you make post-attack could complicate and taint the evidence. Because of that, a common policy is to power-off a system once a compromise is detected, and then to leave it off until law enforcement is ready to investigate.

Checking Network Connections

Let's start our investigation by checking our slice's network connections...


netstat -an

This command helps you check for any "backdoors" which have been opened on your slice.

netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q             Local Address                     Foreign Address            State
tcp        0               0                                    *                  LISTEN
tcp        0               0                                    *                  LISTEN
tcp        0               0                                    *                  LISTEN
tcp        0             284                                           ESTABLISHED

In this case we see port 6697 is open — that port is commonly used by IRC servers. That's not a good sign, unless we're running our own chat server. We can sniff any connections to that port using tcpdump. For more info on tcpdump, check here.

tcpdump src port 6697

This will capture all the packets with destination port 6697.

Using lsof

lsof is a command line utility which stands for "list open files". It is used in many Unix-like systems to report a list of all open files and the processes that opened them. By default Linux treats everything, including devices, as a file. This makes lsof a very powerful tool.

For example, we can use lsof to see what user has a particular file open:

sudo lsof /etc/passwd

If we discover the user name under the intruder's control, lsof can be used to display all his running processes:

sudo lsof -u hisUserName

lsof also helps us check our network connections. Investigating various aspects of our slice with multiple tools is important — if we suspect the system is compromised, we can't be sure which commands will provide reliable results. Also, lsof provides some options which netstat does not.

To list all the open IP sockets associated with your slice's SSH server run the following command:

sudo lsof -i:22


In this article we learned some techniques that can be used to discover backdoors and track intruders on our slice.

This will help us avoid a repeat of whatever situation or mistake led up to the compromise, so we're less likely to get hacked again in the same way.

In the second part of this series we will learn how to investigate our slice in rescue mode.


Article Comments:

James lopez commented Thu Mar 25 00:32:52 UTC 2010:

Great security article. I hope more articles are coming about investigating compromised slices

Ismail Guneydas commented Mon Mar 29 14:43:32 UTC 2010:

Thanks James. There will be more security articles about this for sure.

Ritesh commented Tue Mar 08 11:46:13 UTC 2011:

I have a specific addition to this list. I had only Apache2 running on my box when I realized that my bandwidth usage was close to 600 GB. It was funny because there was no website running on it. My only mistake, allowing mod_proxy on Apache to allow all connections in its configuration. My web server was used as a tunnel for illegal content on the web. Be safe, read up on Apache configuration properly.

Want to comment?

(not made public)


(use plain text or Markdown syntax)