RHEL –Shorewall Basic Troubleshooting

So far, we've got Shorewall running smoothly and we've learned how to work with port access, both with macros that Shorewall has provided and with our own macros that we've made ourselves. Now it's time to learn what to do when things don't go according to plan. In this article, we'll take a look at Shorewall logging and how to interpret it to help troubleshoot problems.


We're progressing nicely down the path toward Shorewall nirvana. If you've read through the previous Shorewall articles, you know that right about now is the time that I remind you to check out the previous articles before you read this one, starting with part 1. If you haven't... well.. you should check out the previous articles before you read this one, starting with part 1.

Now that that's out of the way, let's get to it. As it stands right now, we've got our slice nice and secure with Shorewall running on it. We can open any necessary ports and make macros to deal with special ports that may not be used as much as others. Yep, things are running pretty smoothly. And they'll always continue to run smoothly, right? Hm..

Let's just pretend for one second that that things didn't flesh out exactly as we had hoped. We can use our first example of when we wanted to open up the firewall for MySQL. Let's say that we weren't forward thinking systems administrators and we didn't configure the Shorewall to allow that traffic. Easy enough to simulate. We just turn that port off in our configuration.

First, we open our rules file for editing.

sudo nano /etc/shorewall/rules

Now, let's tell the Shorewall not to reference the MySQL macro. We could just remove the line from the rules file altogether, but a much easier way is to change that line in the file from one that is read into the configuration into a comment line that won't be read into the config, or in other words, comment it out. Remember, in the configuration files, any line that begins with a pound sign (#) is not read as a configuration line.

So let's find our line that references the MySQL macro...

# Accept connections from the net for mysql
MySQL/ACCEPT    net             $FW

… and comment it out.

# Accept connections from the net for mysql
# MySQL/ACCEPT    net             $FW

Save and exit. Now, we check to make sure that Shorewall is okay with the changes we made.

sudo /sbin/shorewall check

Verify that it isn't going to blow up.

Shorewall configuration verified

And, of course, restart the Shorewall.

sudo /etc/init.d/shorewall restart
Restarting shorewall:                                      [  OK  ]

Good. Now, let's pretend that we're an application trying to access our slice over the standard MySQL port of 3306. I think you can guess how that's going to work out. We can do this by, yes, opening another terminal window, and trying to connect on that port.

telnet rhelsandbox 3306
Trying 208.78.87.176...

Congratulations. We have now intentionally sabotaged our Shorewall.

Now let's pretend that Marty from the development team is now screaming at us over the phone because his application is breaking whenever it tries to query the MySQL server. Is is the firewall? Well, is it?!

Once we assure Marty that every available resource is being allocated to solving his problem, and then finishing the rest of our lunch (we never really did like Marty), let's find out if indeed it was the Shorewall.

Shorewall by default keeps its logs in /var/log/messages. Let's look on our slice and see what we can see in there.

sudo tail /var/log/messages

Yikes, that readout is pretty confusing. But hold on, there's some stuff there that looks kind of familiar. Let's take a line of the readout and pick it apart.

Aug 12 20:16:42 rhelsandbox kernel: Shorewall:net2fw:DROP:IN=eth0 OUT= MAC=40:40:a9:2b:7d:0e:00:18:8b:f9:6e:70:08:00 SRC=10.10.0.10 DST=192.168.1.100 LEN=60 TOS=0x10 PREC=0x00 TTL=48 ID=940 DF PROTO=TCP SPT=37629 DPT=3306 WINDOW=5840 RES=0x00 SYN URGP=0

Well, we see the system date and time and the hostname of the server. That's easy enough to understand. Then we see “Shorewall”. That's a good sign, we're looking at the right log line. Now, let's dissect the rest of the line and see what pertinent information we can get out of it.

net2fw:DROP

With this little bit here, we can see the zones that the line deals with and the action that was taken by the Shorewall. In this case, the traffic was going from the net zone to the firewall zone and the packet was dropped.

SRC=10.10.0.10

Here, we see where the traffic originated from. Looks like IP address 10.10.0.10. Ugh, Marty.

DST=192.168.1.100

Next, we have where the traffic was destined for. Yep, that's our slice.

PROTO=TCP SPT=37629 DPT=3306

And here's the meat of the traffic. Hm, looks like it was TCP protocol traffic. The source port of the traffic (SPT) was port 37629. And looks like the traffic was headed for destination port (DPT) 3306.

Wait, 3306? That's MySQL. Well, heck, looks like we shut Marty's application out of our server after all. Ah, we won't lose sleep over it, he needed to be reminded of who was running the show anyway.

Still, we should open the Shorewall for MySQL or Marty is going to be really difficult to deal with. Let's get back into our rules file and take the pound sign out from in front of our MySQL macro line so that it will be read into the configuration again.

sudo nano /etc/shorewall/rules

Find the line and remove the pound sign.

# Accept connections from the net for mysql
MySQL/ACCEPT    net             $FW

And you know the rest. Save and exit. Do a shorewall check. Restart the Shorewall. Really, you don't want me to type all that stuff again, do you? Just look at the beginning of the article if you forgot :)

So let's take a look. What happens when we try to get to the MySQL port now?

telnet rhelsandbox 3306
Trying rhelsandbox...
Connected to rhelsandbox.
Escape character is '^]'.

Excellent. Now, Marty should be happy. Well, happier anyway.

Conclusion

Now we know what to do and what to look for when things don't go exactly according to plan with our Shorewall. You now have a good basis for running and administering a potent firewall. There's much more to it, but with this basic framework, you can now be secure in your ability to keep your data and applications safe from the bad guys. And that's what it's really all about, isn't it?

Article Comments:

Brad commented Wed Jan 13 18:29:28 UTC 2010:

If MySQL setup uses "skip-networking" then he cannot connect to port 3306 using telnet anyway. So that service might not be the best one to use for testing?

KARTHIKEYAN commented Mon Jul 05 18:06:38 UTC 2010:

line and remove the pound sign is it?

Jered commented Mon Jul 05 18:24:25 UTC 2010:

I'm not sure what the question is, so I'll just clarify the bit about "comments" and hope that helps.

When you insert a "#" in front of a line in the file, it turns the line into a "comment". The application won't read anything from a comment line for its configuration. That means an easy way to disable a setting without deleting its entry is to make it a comment with the "#" symbol. When you want the line to be active again, remove the "#" symbol to "uncomment" the line.

Want to comment?


(not made public)

(optional)

(use plain text or Markdown syntax)