Something ugly is going on. DOS attack.

[Steven E. Protter]

Left my job after a quick fiber card swap on an rp5450. When I left all was well with my web servers.

I get to my private office and its whack city. The web server is unresponsive. A ton of httpd processes are out there with PPID 1 and can't be killed.

Pretty much looks like the kernel (Linux, not relavent) is whacked.

I'm forced to boot. To some degree its going on with apache 2.0.x on all servers, hp9000 and linux.

I see a lot of strange entries in the httpd access_log log.

220.160.43.32 - - [15/Dec/2004:23:30:12 -0600] "GET http://www.blazerunner.com/ppc/search.php?keywords=Nutrition+supplement&username=robertWyatt HTTP/1.0" 404 5926
222.135.120.122 - - [15/Dec/2004:23:30:13 -0600] "GET http://bee-search.com/search.php?AID=25&q=blackjack HTTP/1.0" 404 5871
60.208.230.145 - - [15/Dec/2004:23:30:14 -0600] "GET http://hpcgi1.nifty.com/trino/ProxyJ/prxjdg.cgi HTTP/1.0" 404 5893


I see the following in a tcpdump with the -f option for foreign addresses.

ip addresses are altered.

170 is httpd
168 is httpd
167 is supposedly mail.

23:22:22.923839 61.42.141.170.http > 220.160.43.32.3702: . 3848796859:3848798291(1432) ack 2301109696 win 6432 (DF)
23:22:22.976771 222.51.105.105.2420 > 61.42.141.168.http: S 2580438232:2580438232(0) win 16384 (DF) [tos 0x20]
23:22:22.976813 61.42.141.168.http > 222.51.105.105.2420: S 4050721642:4050721642(0) ack 2580438233 win 5840 (DF)
23:22:23.042141 220.160.43.32.1405 > 61.42.141.167.http: F 1:1(0) ack 1432 win 65535 (DF) [tos 0x20]
23:22:23.042175 61.42.141.167.http > 220.160.43.32.1405: . 1432:2864(1432) ack 1 win 6432 (DF)
23:22:23.042182 61.42.141.167.http > 220.160.43.32.1405: . ack 2 win 6432 (DF)
23:22:23.042187 61.42.141.167.http > 220.160.43.32.1405: . 2864:4296(1432) ack 2 win 6432 (DF)
23:22:23.303848 61.42.141.168.http > 220.160.43.32.1480: . 0:1432(1432) ack 1 win 6432 (DF)
23:22:23.454348 220.160.43.32.1215 > 61.42.141.168.http: R 2382819377:2382819377(0) win 0 (DF) [tos 0x20]

Here is the interesting part.

The IP addresses are blocked on the firewall. I've checked the configuration a few times and my response programs updated it properly. Yet the activity continues.

Questions:
does 220.160.43.32.1215
in tcpdump mean the traffic came in on 220.160.43.32 port 1215.

I hope not, becasue that port is blocked on the firewall and I have verified that from the outside.

I'm kind of wondering whats going on and how to deal with it. Hardening the firewall configuration seems to have slowed but not stopped the problem. The configuration was pretty solid before.

SEP

[Fred Ruffet]

Yes SEP. 220.160.43.32.1215 means 220.160.43.32 port 1215. If you activate name resolution on its output and port is defined in /set/services, you'll see it's a hostname.port format.

What is meant in this line
23:22:23.454348 220.160.43.32.1215 > 61.42.141.168.http: R 2382819377:2382819377(0) win 0 (DF) [tos 0x20]
if that 220.160.43.32 send a request to 61.42.141.168. But this is quite normal. If I understand, 61.42.141.168 is your public address. And 220.160.43.32 is the client. It accesses your server on port "http" (probably 80) and its local port is 1215. But that doesn't mean your 1215 port is oppened.

Regards,

Fred

[RAC_1]

I did not understand the all details, but how about putting additional entry in /var/adm/inetd.sec and deny all services to that ip address??
I did nslookup for all those ips, and did not get anything.

Anil

[Mark Greene_1]

Are you using a hardware or software firewall? And if hardware, how many network devices are between the firewall and the systems in question?

In your apache conf files you should also verify that the "MaxSpareServers" setting is as low as resonably possible. The out-of-the box setting is like 15 or 20; perhaps change it to be equal to the "MinSpareServers" value and then make sure that is set as low as practically possible. This won't prevent a DDOS attack, but at least it doesn't give hackers as much to work with.

mark

[Steven E. Protter]

itrc went unpostable last night when I tried to add further details.

I use ipfilter on the hp-ux apache server and iptables on the Linux one. 61.42.141.167 address is a public address but i picked it at random. Some of the inbound traffic is cominng in with my ip address hardcoded inside and I'm not in the mood to provide any assistance to these persons.

More details coming as time permits.

SEP

[Eric Antunes]

Hi SEP,

I've no experience in this kind of situations but when you wrote "after a quick fiber card swap on an rp5450" I remembered one basic principle I apply everytime I have a new problem:

Until when it was fine and what did I changed since that time??

Hope this will help you because I've no other ideas about your issue since I'm not an expert in security...

Kindest Regards,

Eric Antunes

[Steven E. Protter]

The problem does not effect the card swap. THe card swap was in a box that is thankfully not a public web server. That was sort of a "it was a sunny day on the way to work" type comment. Sorry to distract.

http://somesite.com/sproxy.php?ip= 61.42.141.168&port=80

I tried that myself in a browser. It did not go to my webserver and displayed some useless information about what ip addresss I'm at where I am currently sitting.

Does anyonw know why or how such stuff could be getting into my apache logs? I see a lot junk as it is, but this is impacting my servers reliability.

SEP

[Fred Ruffet]

I need a light : isn't 61.42.141.168 your IP ? What you mean is that you see traffic log from an unknown host to an unknown host ?

Regards,

Fred

[Steven E. Protter]

Hi Fred,

Its my ip.

I have changed all the records i post to confuse future hackers.

SEP

[Fred Ruffet]

So your real problem is that you see GET lines in access.log that does not correspond to your machines ? (tcpdump output seems normal to me). If this is the case, it may have been produced by a buggy DNS, resolving your IP for the wanting domain. Voluntary or not ? Hacker or not ? This is another problem.

Regards,

Fred

[Steven E. Protter]

Yes Fred.

I don't know where the get is coming from or how its triggered.

the tcpdump data should not be happening because the ip address is blocked.

I would say not voluntary to that questions.

I checked out some of the sites. blazerunner.com is a spyware laden cheap search engine. They don't have the url referenced in the log. Interestingly enough in the DNS realm, the name will not resolve by my browser is able to connect to the site no problem.

SEP

[Fred Ruffet]

SEP,

It will : you're not using the same DNS as the client does.

If the http client wants to access this site (blazerunner.com) and the DNS he refers to gives him your IP for this site, he will try to connect, issue the GET statement and you will have those logs. If you want to do the same, you can't, as long as you use a good DNS, refering your site as your correct IP (It's probably your DNS).

Fact is you should find DNS this IP uses. And that is probably a real challenge :-(

Regards,

Fred

[Steven E. Protter]

I have DNS resolution now on the site in question. I'm setting up my firewall to block all traffic inbound or outbound to that website.

The junk should show up in the log, but no more information will travel to that website.

This is but one of a dozen sites attempting the same type of abuse. So far all it seems to do is open up lots of extra httpd processes, which I control by restarting the httpd server.

Still there is a whole somewhere that needs to be plugged.

I may need to somehow limit where I accept DNS requests from. This is tricky because public websites must resolve their names.

SEP

[Steven E. Protter]

Discovered something very interesting.

The firewall was not configured properly.

A tiny little mistake while doing an system upgrade left the firewall database blocking ip addresses input from the internal network instead of the external network.

eth0 needed to be changed to eth1
lan0 needed to be lan1

So that explains why measures against the input were failing. They were not blocking on the correct NIC.

Will update and possbily close later.

SEP

[Eric Antunes]

As you see, there is always something we did wrong the last time we changed something... :)

Eric Antunes

[Steven E. Protter]

Once again had to develop a custom firewall solution to deal with the problem.

Detected the activity was coming in on port 80.

Blocked output on the attempted outbound ports.

Wrote a program to detect and add blocking entries to ipfilter and iptables to firewall configuration files.

SEP

[Steven E. Protter]

Thread closed.