Port 80 abuse, iptables question.

[Steven E. Protter]

My servers have been subjected to a continuous port 80 redirect attack.

The purpose of the attack seems to be to get the httpd server to send http requests to servers that I do not host. This appears to get the attacker credit for search engine clcks, essentially ripping off the search engine provider.

My question is this:

I get the following log output:
Mar 30 19:21:49 shalom kernel: IN=eth1 OUT= MAC=00:90:27:7b:22:e3:00:00:c5:a2:50:44:08:00 src=62.103.74.51 DST=66.92.143.194 LEN=48 TOS=0x00 PREC=0x20 TTL=113 ID=50476 DF PROTO=TCP SPT=4755 DPT=80 WINDOW=8192 RES=0x00 SYN URGP=0
Mar 30 19:21:49 shalom kernel: IN=eth1 OUT= MAC=00:90:27:7b:22:e3:00:00:c5:a2:50:44:08:00 src=62.103.74.51 DST=66.92.143.194 LEN=48 TOS=0x00 PREC=0x20 TTL=113 ID=50479 DF PROTO=TCP SPT=4756 DPT=80 WINDOW=8192 RES=0x00 SYN URGP=0
Mar 30 19:21:50 shalom kernel: IN=eth1 OUT= MAC=00:90:27:7b:22:e3:00:00:c5:a2:50:44:08:00 src=62.103.74.51 DST=66.92.143.194 LEN=48 TOS=0x00 PREC=0x20 TTL=113 ID=50486 DF PROTO=TCP SPT=4757 DPT=80 WINDOW=8192 RES=0x00 SYN URGP=0
Mar 30 19:21:51 shalom kernel: IN=eth1 OUT= MAC=00:90:27:7b:22:e3:00:00:c5:a2:50:44:08:00 src=62.103.74.51 DST=66.92.143.194 LEN=48 TOS=0x00 PREC=0x20 TTL=113 ID=50493 DF PROTO=TCP SPT=4758 DPT=80 WINDOW=8192 RES=0x00 SYN URGP=0
Mar 30 19:22:01 shalom kernel: IN=eth1 OUT= MAC=00:90:27:7b:22:e3:00:00:c5:a2:50:44:08:00 src=62.103.74.51 DST=66.92.143.194 LEN=48 TOS=0x00 PREC=0x20 TTL=113 ID=50507 DF PROTO=TCP SPT=4761 DPT=80 WINDOW=8192 RES=0x00 SYN URGP=0
Mar 30 19:23:04 shalom kernel: IN=eth1 OUT= MAC=00:90:27:7b:22:e3:00:00:c5:a2:50:44:08:00 src=62.103.74.51 DST=66.92.143.198 LEN=48 TOS=0x00 PREC=0x20 TTL=113 ID=50777 DF PROTO=TCP SPT=4806 DPT=80 WINDOW=8192 RES=0x00 SYN URGP=0
Mar 30 19:23:04 shalom kernel: IN=eth1 OUT= MAC=00:90:27:7b:22:e3:00:00:c5:a2:50:44:08:00 src=62.103.74.51 DST=66.92.143.198 LEN=40 TOS=0x00 PREC=0x20 TTL=113 ID=50782 DF PROTO=TCP SPT=4806 DPT=80 WINDOW=8760 RES=0x00 ACK URGP=0

I see a series of SYN and finally an ACK, and then the attack shows up on the access_log for the master web server.

Now, port 4806 is blocked inbound. I have tested this and it is true. Port 80 is open, because its a web server.

My interpretation is that the incoming request asked for traffic to go outbound on port 48XX.

I would like the output posted interpreted. If by chance there is a way to get apache to NOT do output on ports other than 80 and 443, I would like to know how to do that.

I can block outbound ports on the firewall, but this is not a real solution, though if the scammer gets no results, he/she/it may go elsewhere.

Bunny for interpretation. Bunny for apache httpd.conf or other fix.

SEP

[Jerome Henry]

Hi Steven,

Something is strange for me in your log.

- Source is always 62.103.74.51 (athedsla-2591.otenet.gr, a client from a greek provider), destination always 66.92.143.198 (investmenttool). So I do not see outbound traffic, but just inbound to your server.

- Delay between SYNs are a few seconds, then up to 10 minutes, which is quite long for a SYN flood or an automated attack tool, except in stealth mode (but in that cas, why always scan port 80 for an ACK ?)

- Source ports are 4755, 4756, 4757, 4758, then 4761, then 4806. As you know, on a standard connection socket, source port varies randomly over SPORT 1024. Here it looks like a windows explorer (source port increment 1 by 1 for each new window opened. Looks as if client was trying to get connection to investementtool with an ill working browser hanging often).

So what make you think that the guy is trying to force traffic outbound to port 4806 ? The fact that last packet is an ACK doesn't seem to me to have any clear meaning, as source is still the same, so it's still inbound, am I correct ?

Maybe a script kiddy trying to forge packets for a try ? Or once again an ill working browser ?

As far as your httpd.conf is concerned, the Listen directive is also in fact a reply directive. If you only allow ports 80 and 443, in your main Listen, your server name and your virtual hosts, then httpd won't reply from another port, except if there is a specific exploit which I'm not aware of by now (you may still check your httpd version for such exploit. BTW, on trying to get server banner from a 404 page, I get a strange page. Try http://www.investmenttool.com/drop.html for example).

Yours


Jerome

[Steven E. Protter]

I selected only the ip involved because that ip address sucessfully forced traffic through the httpd server.

Let me assure you that this is outbound traffic. The port is dead shut in bound. Totally.

I will give the possibility that the traffic originated on my server. The problem with that is the proximity to the target.

I believe and the tcpdump output agrees that this is outbound traffic designed to create money making log entries on the log files of search engines that pay for traffic.

Please accept that premise or tell me how to disprove it. I've been studying this for months.

Right now I have shut down a lot of the relavent outbound ports. Though the traffic will still show up on MY log, it will NEVER reach the target server. This will decrease the incentive to target my httpd server.

I'd still like to fix the httpd server so it won't trigger outbound traffic at all.

Thats why the traffic looks like it came from my server, because it did. The inbound traffic is triggering outbound traffic.

Good analysis, I'll read it carefully.

SEP

[Steven E. Protter]

Hmmm.

Re-read the post.

No outbound traffic.

Why am i worrying about it? Becasue if I leave it alone the web server eventually comes down, sometimes bringing the kernel with it.

Its real, its outbound and I don't no know it looks inbound.

Run your own test. Try and telnet in in 4608. It will stop, drop and die.

SEP

[Dave Falloon]

Do you have mod_proxy enabled, if the rules around a proxy connection were not locked down that could explain the outgoing traffic, also there was a buffer overflow in that same modules code, it should be fixed in new versions of apache, but its definately something to look at.

At the web hosting company I used to work at, we had a co-lo box throw 80 megs of traffic over the LAN at one of our managed servers because the colo admin uncommented the loadmodule mod_proxy line and had changed the defaults which restrict the list of sites that are proxied to be wide open. He was trying to use zope on a different port and wanted apache to forward the connections to zopes port. Are you attempting anything like this?

In apache's logs what is sent to the server when the attack happens, is this someone posting data to a form, does the hit show up at all or does the server not log the connection?

When I look at the posted network dump this is what it looks like to me:

client 62.103.74.51:4755 connects to 66.92.143.194:80 with a syn packet awaiting an ack from 66.92.143.194:80

the client attempts another connection, using a different source port

and again until it sends an ack. There is no outgoing connections shown in this log though.

That does not mean that these connection attempts are harmless, these could possibly be a buffer overflow? If the connection hasn't been three way hand shaked though, its likely not an apache problem, I suspect your logging methods have missed the outgoing portion of the conversation because the tcp kernel code has had a lot of eyeballs checking it over.

If your packet logs have missing info and its not the tcp connection causing the problems, try to find the information sent to your apache server you may need to turn on more verbosity in your network dump, so you can see the payload in the packets as well, if its a buffer overflow that allows arbitrary code execution you'll see shell args in the packet.

Check secunia to make sure you are fully patched http://secunia.com/

--Dave

[Steven E. Protter]

apache is 2.0.53 released with RH 4.

How do I check the mod file referred to in the prior post?

SEP

[Jerome Henry]

You can check mod_proxy lines in /etc/httpd/modules file.
But if you didn't change things in it, it's defaul configuration is quite safe.

Frankly speaking, I'm doing forensics quite often, and this doesn't look like an attack... Which cracker would be dummy enough to try the buffer overflow exploit (on 2.0.53, works on ssl, so not port 80) sending 3 SYN only per minute ? It would require at least 500 time more to succeed... unless you set up your httpd on a PDA (lol) !

As far as I see it, blocking port 4806 will just prevent an unlucky client to connect if his browser uses that port. I tried this morning, page not found... ok. Open another windows, I'm on the site.

What is odd is the lack of the SYN/ACK, but once again it proves nothing, and the rythm is too slow to look like a real attack. But if you have a SYN/ACK replied at some stage, then it's normal to receive the ACK from your greek friend, and then to have it's access logged by httpd... the handshake is indeed successful, and the transaction being complete the access is logged as successful.

As for engine clicks score, they rely on the get request that should be following the handshake, and we don't see it here (maybe do you have thsi packet in your log, but it can't be in the handshake itself, even forged, LEN is far too short).

I feel you don't need to worry on this exchange Steve, you are working so much already on many things ! Monitor the machine from time to time of course, but don't spend your night at it (how is it that you reply at 6 GMT !)

Jerome

[Jerome Henry]

BTW,

If you use RH4 w/ yum or up2date, httpd patches are backported, so you're imune to flood on this version...

Jerome

[Dave Falloon]

I totally agree with Jerome, nothing malicious in the handshake, not enough data in the packets, any mischief would have to happen after the handshake. If you can fully capture the packets following the handshake and you see anything like:

... ... ./bin/ksh -c

in the payload, that is a buffer overflow, the series of '.' is the no op sled, because when written into memory the program will slide through the no op's until it hits the mischief code, so an attacker will pad the data with these no ops so they don't have to 100% accurate regarding how much to overflow the buffer, otherwise they may start their attack code half way through which is effectively useless.

If you are still concerned about outgoing traffic from the machine, depending on your firewall/router setup you may be able to monitor all traffic outside of port 80 and 443.

Also I was curious about your success with the APC remote power administration, did you get that running? Any trouble?

--Dave

[Steven E. Protter]

The apc device is in full operation. The server is down right now due to an idiot firewall change I made and I've powered it on and off several times using the apc device.

Quite happy with that.

Reading and absorbing the other posts.

SEP

[Dave Falloon]

Woops, actually you won't see the /bin/ksh, it would be have to be machine code.

[Dave Falloon]

No wait I was right, the ascii decoding will probably pick up any text like /bin/ksh.

[Steven E. Protter]

Not doing any zope.

I can't see how I can ignore this. It fills up my access_log so traffic analysis shows other peoples websites as 95% of my traffic.

At the least this is theft of bandwidth. If its accidental, should it not be stopped because I'm paying for the T1 line?


SEP

[Jerome Henry]

Something worries me about what you write. You're far too smart to wonder about something that shouldn't need to be wondered about. I've been thinking about your posts last night :

What is strange is that it looks like inbound, source is outside and dest is you, and you say it's outbound. SO we have 2 possibilities here :

- those packets are forged, and designed to bypass your iptables. But designed as they are, they won't go anywhere, as next router will ignore them or send them back to you (as being dest), which may be the purpose. But how are they generated, if not from inside your box ? Try to check for backdoor/rootkit, with a tool like chkrootkit ( ).

- those are part of the packets, and the others are the SYN/ACK reply that don't appear in your post for it to be kept short. OK, this is the most logical solution.

But then, you write that if you leave it alone, the server finally comes down, which looks like a SYN flood. But in your post only 3 hits per minute are seen. Once again 2 possibilities :

- if this is enough to get the server down, then there is an issue in the server itself, and it would be interesting to understand why it cant fall with such a few hits. Whatesoever, we can assume that it does so because it waits for handshakes to complete taht never does. A way round to try would be to check your httpd.conf file. Are you on keepalive off ? If you need keepalives, shorten MaxKeepAliveRequests to a lower value (from 100 to 20, for example). Then lower your ServerLimit, say from 256 to 160, and, what is most important, lower your MaxRequestsPerChild dramatically, say from 4000 to 1500. Better not touch the MaxClients though, for the hacker no to prevent others to get in. I can explain the reasons of those if you need.

- There is a larger number of packets, and once again you only selected a few. Then it's a SYN flood looking like event. A way round I could suggest, insted of working on the port solution, would be to work on the IP address, as the dump shows it seems to be always the same in the flood sequence. Maybe could you set up a kind of hysteresis on your iptables, that would allow hits in, but in a limited flow, say 2 or 3 per minute (which is even quite few, as each clic on link in any page makes a new handshake, even in normal use of your site). It would be a line in your iptables like :
iptables -A FORWARD -p tcp --syn -m limit --limit 1/m -j ACCEPT

Hope this helps. If it doesn't, could you give more information on what makes you see it's outbound, how many packets are seen and so on ?

The worst case would of course be rootkit, as it would indicate stealth of bandwidth, storage space and deep compromission of the machine itself. Hope we're not there.

Yours

Jerome

[Jerome Henry]

Off topic, your target site was off this morning, it's back on now, but my advise would be to protect a directory, hum... the one that you use to chek what happens on your site, and that is well protected on your main site... see my point ? more details thru jhenry -leaving at- daclem -then a dot- com if you need.

Friendly yours

Jerome

[Dave Falloon]

Jerome, I don't quite follow, but this sounds interesting, could you give a little more detail through the forum?

I admin a lot of web servers and eagerly seek out any useful tricks.

[Steven E. Protter]

Thank you for your imput.

I've been trying to manage this situation in the context of a larger crisis. My employers disk array melted down last Tuesday with the loss of 1.3 TB of data.

Much of that data included data my HP-UX systems relied on. The past 9 business days have been spent rebuilding.

That being said, this situation is curious. I did a cold install of Red Hat on a secondary system. These log entries began as soon as I srarted the webserver.

What does this prove? That its not corruption of the local system. Its an inherent flaw in apache. Maybe its bugzilla time.

I will be looking at the firewall suggestions and trying to implement them. The websites themselves have been stable since I stopped playing with the firewall. Its not a firewall problem, nor will it be solved by the firewall I think.

SEP

[Steven E. Protter]

uploading iptables firwall config.

This should make clear one of two possibilities:

iptables has been bypassed by a new exploit

there is no way the log above is inbound.

Close port 80 on the firewall the abuse stops.

:-)

SEP

[Steven E. Protter]

This time the file WILL be attached.

[Jerome Henry]

Sorry if I'm looking slow to you but... I don't clearly get your network config. From your iptables it seems that your default policy is accept, right (or why do you -j drop all non accepted ports in detail) ?
66...194 seems to be your public ip address, right ? But why do you -s 66...194 -i eth1 okay ? Is this machine something else than your firewall ? Is it supposed to go inside your lan from eth1 ?
Where is your web server located ? In your LAN ? In a DMZ ? What is 66...194 and 66...197 for it ?

Tx

Jerome

[Steven E. Protter]

This server is a httpd/firewall with Internet exposure.

It was providing NAT services to the network, due to security problems this feature is being dropped.

Some of the code in question is legacy, and I may experiment with dropping it and implementing your other recommendations.

I had a test server that I inteneded to use for these changes without affecting web users. It died Saturday night. Disks recognized, won't boot grub worth a darn. Wondering if its the fact I tried to use LVM for the non-root paritions.

The secondary server had been showing reliability issues for a while. Now it is simiply unbootable.

Not wanting to experiment with a live server, I'm working on putting together an emergency replacement for the testbed server so that I can proceed with your recommendations.

SEP

[Steven E. Protter]

iptables v1.2.11: host/network `016.a.004.pth.iprimus.net.au' not found
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.2.11: host/network `172.141.78.83.cust.bluewin.ch' not found
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.2.11: host/network `213.213.212.72.brutele.be' not found
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.2.11: host/network `226.83.205.68.cfl.res.rr.com' not found

This output is from a script that picks up this activity and adds them to the iptables firewall.

Another reason to think this activity is nefarious. People are now bothering to spoof their true ip address.

Is there a way to force apache to do a lookup and deny access to those without valid lookup?

SEP

[Florian Heigl (new acc)]

SEP, could it be You have mod_proxy active on that server?

[Steven E. Protter]

All references to mod_proxy are commented out.

I am somewhat reluctant to post the httpd.conf file up, due to security concerns.

Later this evening, when I have better access, I will post some snippets.

To avoid the spoof issue, I've turned off lookups in httpd.conf. Now I get pure IP addresses. Though these can be spoofed, at least the errors in the cleanup program can be avoided.

A look at the iptables file I attached clearly shows port 4806 is shut tight.

SEP

[Jerome Henry]

Hi,

Clearly trying your closed ports blocks packets, and trying denied ports too. Forging packets to send you a SYN/ACK gets ACK back, which is normal. But I can't get to understand how to make you send a SYN without setting at first a backdoor inside your server. So for the advice of checking rootkits.
Then if I were a script kiddy I would think than reversing the packet order can allow me to bypass your firewall, as you won't prevent SYNs to go out.
But if source is you, then the next hop router will ignore your SYN, as dest is same as source (that is to say 66...194), so the packet won't go anywhere, unless the router was poisoned too, which gets harder.
Or it's a very clever guy and those packets wait for something from outside.

If you say that a clean install does the same, then something is going on that is not related to a backdoor. I did an install this morning, RHEL4 and httpd 2.0.53 standard setup, 2 virtual hosts for a check. Nothing special in my logs.

Maybe would it be time to try on a spare machien (if you have one left !) to clean iptables ? As far as I understand your config, a lot of things could be take off, as there is no 66...194 going input to eth1, except from loopback, which would imply a strange process action.
I juste requested an icmp timestamp while writing, and I get an anwser, which seems to imply that default policy is accept, or that your config allows somehow things inside, maybe thru this -s 66...194 -i eth1 accept rule.

I'm afraid I reached the end of what I can think about on it, I would suggest running httpd on a step by step rule to dump each exchange and get if it's coming from outside or inside.

As for lookup, you name it, hostnamelookups, which is quite heavy, is the solution to do it under apache. Maybe can it be linked with some mod_access rule to prevent access, but this is far beyond httpd's job. An IDS or PIDS would be more efficient and more at its place here.


Jerome