Netfilter firewall violation, how does it happen??

[Steven E. Protter]

Related: prior threads.

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=846932

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=764669

Here is a tcpdump on the network card:

21:48:16.433916 IP (tos 0x20, ttl 109, id 28083, offset 0, flags [DF], proto 6, length: 48) 218.56.241.227.4230 > shalom.investmenttool.com.http: S [tcp sum ok] 228348268:228348268(0) win 16384
21:48:25.242485 IP (tos 0x20, ttl 109, id 28786, offset 0, flags [DF], proto 6, length: 48) 218.56.241.227.4508 > shalom.investmenttool.com.http: S [tcp sum ok] 2307703548:2307703548(0) win 16384
21:48:28.158195 IP (tos 0x20, ttl 109, id 29128, offset 0, flags [DF], proto 6, length: 48) 218.56.241.227.4508 > shalom.investmenttool.com.http: S [tcp sum ok] 2307703548:2307703548(0) win 16384



21:49:27.306479 IP (tos 0x20, ttl 109, id 35930, offset 0, flags [DF], proto 6, length: 48) 218.56.241.227.1744 > shalom.investmenttool.com.http: S [tcp sum ok] 178981644:178981644(0) win 16384



21:50:17.606426 IP (tos 0x20, ttl 109, id 41167, offset 0, flags [DF], proto 6, length: 48) 218.56.241.227.2439 > dsl092-143-198.chi1.dsl.speakeasy.net.http: S [tcp sum ok] 4235782547:4235782547(0) win 16384


traffic inbound on a closed port hitting the httpd server on port 80

21:51:59.357513 IP (tos 0x20, ttl 109, id 54768, offset 0, flags [DF], proto 6, length: 48) 218.56.241.227.4306 > shalom.investmenttool.com.http: S [tcp sum ok] 3116407883:3116407883(0) win 16384
21:52:09.482462 IP (tos 0x20, ttl 109, id 55937, offset 0, flags [DF], proto 6, length: 48) 218.56.241.227.4747 > shalom.investmenttool.com.http: S [tcp sum ok] 3146051390:3146051390(0) win 16384
21:52:15.486099 IP (tos 0x20, ttl 109, id 56591, offset 0, flags [DF], proto 6, length: 48) 218.56.241.227.4747 > shalom.investmenttool.com.http: S [tcp sum ok] 3146051390:3146051390(0) win 16384
21:52:41.647339 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto 6, length: 48) dsl092-143-198.chi1.dsl.speakeasy.net.http > 218.56.241.227.1560: S [tcp sum ok] 3788662358:3788662358(0) ack 2830229211 win 5840
21:52:42.620355 IP (tos 0x0, ttl 64, id 5498, offset 0, flags [DF], proto 6, length: 40) dsl092-143-198.chi1.dsl.speakeasy.net.http > 218.56.241.227.1560: . [tcp sum ok] 1:1(0) ack 361 win 6432
21:52:42.635700 IP (tos 0x0, ttl 64, id 5500, offset 0, flags [DF], proto 6, length: 1400) dsl092-143-198.chi1.dsl.speakeasy.net.http > 218.56.241.227.1560: . 1:1361(1360) ack 361 win 6432
21:52:42.635713 IP (tos 0x0, ttl 64, id 5502, offset 0, flags [DF], proto 6, length: 1400) dsl092-143-198.chi1.dsl.speakeasy.net.http > 218.56.241.227.1560: . 1361:2721(1360) ack 361 win 6432
21:52:43.357529 IP (tos 0x0, ttl 64, id 5504, offset 0, flags [DF], proto 6, length: 1400) dsl092-143-198.chi1.dsl.speakeasy.net.http > 218.56.241.227.1560: . 2721:4081(1360) ack 361 win 6432
21:52:43.357538 IP (tos 0x0, ttl 64, id 5506, offset 0, flags [DF], proto 6, length: 1400) dsl092-143-198.chi1.dsl.speakeasy.net.http > 218.56.241.227.1560: . 4081:5441(1360) ack 361 win 6432
21:52:43.357543 IP (tos 0x0, ttl 64, id 5508, offset 0, flags [DF], proto 6, length: 599) dsl092-143-198.chi1.dsl.speakeasy.net.http > 218.56.241.227.1560: FP 5441:6000(559) ack 361 win 6432
21:52:44.614562 IP (tos 0x0, ttl 64, id 5510, offset 0, flags [DF], proto 6, length: 40) dsl092-143-198.chi1.dsl.speakeasy.net.http > 218.56.241.227.1560: . [tcp sum ok] 6001:6001(0) ack 362 win 6432

# The end part of the tcpdump is after the netfilter firewall is violated. The httpd server is manipulated to try and retrieve a page on a remote httpd server, one that pays for click referrals.

I am attaching the entire file for further analsys.

Notes:

Happens in Linux and HP-UX but much less often on HP-UX. This could simply be due to lower volume of traffic on the HP-UX servers.

I have run chkrootkit and every other tool known to administration to try and see if the system is being compromised. The system is not being compromised. Its a abuse of resources issue that has really annoyed me over the past few months. I would like to stop it.

Questions:
1) Based on the log, how is the firewall being violeted? If you simply try a telnet to the port that is being logged you will find the connection is denied.

2) Is there a way to filter packets to stop these firewall violations?

3) Are there tools for Linux and HP-UX that can be used to monitor and possibily defeat these violations. tcpwrapper, portsentry, nessus are examples. To get bunny a link or other identifying information is required.

Good answers to questions 1-3 above are going to find rabbits in their point boxes.

SEP

[Tim Sanko]

Steve,

I have some questions that I need answered first. Is the Firewall trying a reconnect on the same tcp port? Are you running kerebros or other security software? Is there a connection timeout on the firewall?

These are the first questions I need to know to know where to start.

Tim

[Steven E. Protter]

Seems that the connection is moving around on various ports until it finds one it likes. A single transmission will be denied. Most of the attempts prior to successful violation are denied by the firewall.

>>
Is there a connection timeout on the firewall?
<<
I don't think I set one explicitly. With netfilter/ipfilter how do I tell or set one? I will do some research and find out. Looking at the log, there seems to be a substantial period of time between probe and violation.

SEP

[Tim Sanko]

I dumped netfilter becuase of issues like these. I would use tcpwrappers anyway just as SOP. At N*S* we went to a custom socket program written to change ports and imbed encryption key and next socket in oob data.

Worked so well that NSA was unhappy.

Tim

[Steven E. Protter]

How would I use tcp_wrappers to stop these violations.

Also, it would seem that there is quite enough information that a portscan is going on to realize it and stop it. Wondering how I'd make the system smart enough to figure it out and deal with it without harming "good" traffic.

SEP

[Ron Kinner]

Check out snort:

http://www.snort.org/

Ron

[Tim Sanko]

I don't know that tcp wrappers wouldn't stop it because of the way the information is loaded into the wrapper. but how would it know a good application from a bad one? sometimes It has to be defined from an internal program. I would suggest a custom portmapper of sorts.

You write a program that accepts sockets and tracks ports and keeps the available list in a linked list of pointers to structures you malloc as necessary You do need previous and next structure. It allows you to "front end" the security and control the searching. It only requires one port to start distributing the available ports and sockets for those ports. If you embed oob data you have just destroyed 99.9999% of snoopers.

Tim

[Steve Lewis]

You keep a complex firewall config which is more likely to have holes in.

Harden your client/remote admin machines to prevent tunnelling or piggy-backing. Thats a sure way through the firewall.

Configure sshd to strictly enforce host based authentication for ssh logins and sftp - i.e. the public keys for your remote systems only.

Whatever you do, stop portmap(NFS) and Samba. Even if you tunnel traffic its probably inviting problems because you are still only as secure as the clients.

HIDS or tripwire is useful and I'd recommend one of those in your case. But beware it is very heavy on your disk i/o and affects webserver response times. Only use it with fast mirrored root disks.
http://www.tripwire.org

I run snort instead of portsentry. Its quite similar and came pre-configured on my Linux distribution with all kind of interesting warnings being logged.

Logsentry is another good idea.

Beware fragmented packets. Ensure that you use keyword related or else deny all fragmented packets through as a matter of policy.

If it is apache thats initiating outbound requests, then take it off your firewall server and block the initiation. I would have thought that apache should not initiate anything, ever - apart from possibly a database connection. I removed everything to do with vhosts.

A good recommendation is shorewall. A front-end to IP tables and IPfilter, the helpful documentation is at http://www.shorewall.net/two-interface.htm

[Steven E. Protter]

There is more than enough information in the log to prove conclusively that its not a ipfilter/iptables problem. If you attempt to connect on any port but the open ports, the connection failes.

try telnet hostname 1560

It will fail.

Yet with this sequence that seems to trigger the firewall to acknowledge the third connection attempt, the traffic gets through.

I did open a bug report at bugzilla but to date they have not moved on the issue.

SEP

[Steven E. Protter]

What is clearly obvious from the log is the traffic is not coming in on port 80.

Example.

21:52:09.482462 IP (tos 0x20, ttl 109, id 55937, offset 0, flags [DF], proto 6, length: 48) 218.56.241.227.4747 > shalom.investmenttool.com.http: S [tcp sum ok] 3146051390:3146051390(0) win 16384
21:52:15.486099 IP (tos 0x20, ttl 109, id 56591, offset 0, flags [DF], proto 6, length: 48) 218.56.241.227.4747 > shalom.investmenttool.com.http: S [tcp sum ok] 3146051390:3146051390(0) win 16384
21:52:41.647339 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto 6, length: 48) dsl092-143-198.chi1.dsl.speakeasy.net.http > 218.56.241.227.1560: S [tcp sum ok] 3788662358:3788662358(0) ack 2830229211 win 5840
21:52:42.620355 IP (tos 0x0, ttl 64, id 5498, offset 0, flags [DF], proto 6, length: 40) dsl092-143-198.chi1.dsl.speakeasy.net.http > 218.56.241.227.1560: . [tcp sum ok] 1:1(0) ack 361 win 6432
21:52:42.635700 IP (tos 0x0, ttl 64, id 5500, offset 0, flags [DF], proto 6, length: 1400) dsl092-143-198.chi1.dsl.speakeasy.net.http > 218.56.241.227.1560: . 1:1361(1360) ack 361 win 6432



Traffic is coming in on port 4747

Traffic goes out form the httpd server on port 80 outbound on port 1560.

Output has to allow traffic on most ports or the web server won't function.

It would seem that if a firewall was set up running no httpd server and it forwarded valid port 80 traffic that the exploit would stop.

This would also leave me with a valid DMZ.

So I'm working on building a firewall box based on a firewall distribution.

It still would be nice to not let the traffic in on port 4747 because that port is not supposed to accept inbound traffic.

SEP

[Steven E. Protter]

Interesting development.

The real problem here is the sysadmin. Me.

When this problem started there were holes in the iptables firewall.

There were a number of open holes in the firewall because I didn't understand completely how to use it. This issue and its three threads has made me at least competant in the firewall.

Once you become a target by having a leaky firewall, word gets around and there are always people knocking at the door trying to exploit.

The fireall config is now tight. It is possible to violate the firewall, but its relatively hard.

So what was going on here?

My first response was not to learn and fix the firewall, it was to write a script to gather up the violaters and add them to a banned list.

That led to seveal other scripts, one of which, shuts down the firewall read the new configuration file as it rebuilds the firewall.

That is a problem because if here are people knocking on the door trying to get in and you quickly open the door, fix a crack and then slam it shut again, they get in.

Once a persistant connect is established its no problem directing traffic through that connection for whatever purpose you want. All you have to do is keep the connection and its as if the firewall did not exist.

I figured this out because I saw IP addreses that I knew were on the firewall with persistant connections.

netstat -an | grep ":80 "

Since I left the Jewish United Fund I've had a little more time to work on things like this and a little more time to just think. Finally tonight I put two and two together.

Once a day, the system brings down the external network connection updates the violater list and bounces the firewall.

With the external connection down, nobody can achieve a persistant connection and the number of violations is under two per hour, which is quite reasonable.

Its actually possible from the firewall standpoint I fixed the problem weeks ago, but kept getting those persistant connections when I dropped the firewall several times an hour to bounce it.

There is a way to vioalte the firewall setup, which is actually quite simple, trimmed down a lot from the earlier thread.

I'm publishing this to make clear that no matter how many itrc points I have or how many years of Systems administration time I have in my past, I wasn't working smart. I worked hard and put a lot of time in on this, but overall I wasted a lot of time working on a problem that my methodology was making worse.

Bluntly, I did a bad job on this issue. Obviously this post is not an add for my next position, but for some reason I found it important to be honest with the crew here and admit my misakes.

I hope that counts for something.

Regards and many thanks for everyone's help in all of the threads on this issue.

SEP

[Steven E. Protter]

Clearly the biggest problem here was the systems administrator. There have been no firewall violations since the methodology as improved. The firewall is Linux and the configuration script and methodology will be made available to the open source community when its production.

SEP