Hacked again; forensics needed

[Vernon Brown_4]

For the third time now in about a week hackers have managed to kill my Apache server; twice before I reformatted, reinstalled, and recovered using backups. This time I was watching my server access_log with a tail -f when the attack came. I will later go through the reformat and recover process, but thought that now, with this damaged system limping along we might can find out what is happening.

Below is the beginning portion of a 15K access log record that came in. When I saw the attack in progress I immediately shut down the server and tried to reboot. But Lilo could not find the operating system. I rebooted using a boot floppy, and here I am.

Question: Has anyone seen this before; how does it work; what can be done to combat it; etc.

Points for your thoughts !!

66.41.166.198 - - [29/Mar/2004:18:00:16 -0600] "SEARCH
/ ± ± ± ± ± ± ± ± ± ± ± ± ± ± ±
± ±

[Stuart Browne]

Start with distribution and version of apache.

[Vernon Brown_4]

Thanks for your response ! This server is running RedHat distro 7.1 this time with the sendmail that came with the RedHat. The time before; day before yesterday; I was running sendmail freshly downloaded latest .gz but I didn't get the "run-as" jailhouse shell working before the crash.

Attack seems sendmail related because there were sendmail errors in maillog. "Loops back to me" MX problem. The mail recipients listed don't have accounts on the server. Normal mail was working before the mishap.

Vern

[Stuart Browne]

So I take it to mean you weren't using eratta apache (ssl in particular) or sendmail.

There have been MANY expliots found and fixed since RH7.1 was released.

The easiest thing to do is to grab the RH9 or FC1 sendmail src RPM and compile it. The apache is a little more difficult as it's a 1.3 series. Find the latest eratta, it should be around on RH's site, and install that as well (after you rebuild).

All that being said, it might be an idea for the next rebuild to update the RH version. Either go to 9 (or FC1), then do *ALL ERATTA* before putting it online, or go up to one of the RHE products (ES3 possibly?).

The updates are the important thing.

The sort of attack you got hit with could eitehr be a sendmail one, or one against mod_ssl apache (it was buggy in all RH7 releases, eratta releases fixed that).

[Vernon Brown_4]

Guess I'll do the upgrade; may be off line for awhile till I get it all going again.

Thanks for your help.

[Steven E. Protter]

Vernon.

If you must stay with Red Hat 7.x then at least go up to 7.3. Unless you are using some HP server that requires drivers that were never upgraded past 7.1 its got two major successor releases for a reason.

Apache must be upgraded with up2date to the lastest stable release at the very least.

Many hackers cover their tracks, some are quite stupid.

Check the last and lastb command and see from whence your hackers came.

They might have jumped on using traditional telnet, but an apache exploit needs to be epxlored.

If found an interesting situation on my Red Hat Server 7.3 before i closed the holes by running Bastille and shutting down telnet.

I found an account called haxor with user id zero.

I didn't put it there.

My hacker set up a form for relaying spam on the server and then stopped using the account, hoping I would not notice. It was the source of most of my spam problems.

I did notice. Look at /etc/passwd for accounts you did not put there. Especially those with root user id. Get rid of them.

You have been hacked very quickly, several times. If you have closed the apache hole look for a user account.

I recommend the following steps:
Installation and regular use of Bastille
Installation and regular reports with tripwire. Tripwire should be run nightly and will notify you of any change in system configuration.

Conduct regular reviews of /etc/passwd and eliminate accounts you can not account for. You can always put the account back if something breaks.

I had an account on my system called ftpuser. Someone was using that account to try and ftp a mailform to my server. This failed because he couldn't make the permissions executable. Once i reset the password on this account that silliness stopped.

I think the root of my problem was that I was using telnet to get on the box from remote locations, quite regularly. Someone sniffed for a password and got the root one, long enough to set up an account. Either that or they got privledges by patching an unupdated service.

The long term answer for me was Red Hat ES 3. It shipped much stronger on a security standpoint than RH 7.x. The only problem thus far is some hardware doesn't have drivers yet and tripwire won't work at all. Still, a lot of the vulnerabilities built into Red Hat 7.x are closed.

The upgrade is possible, and not too painful. You'll need to upgrade httpd.conf to handle the new syntax for your web servers and ssl is a slightly painful migration.

Still, being closer to current on the OS is worth it.

SEP

The junk in your access_log may be an exlpoit, or it may be just junk because apache wasn't patched up nicely.

[Vernon Brown_4]

Thanks Steven; today I'll try for the second time to upgrade the server. RedHat 9 is not available at the local Best Buy (Little Rock Arkansas) but they did have SuSE 9. So I bought that. SuSE 9 didn't come with a driver for my old eth1 10baseT card that interfaces Apache to my LAN. I'll get a new card today and try again.

Two reasons I suspect the unusual log record was part of a hack attack:

Lilo boot failed with "No Operating System Found" on my next boot up attempt. Had to use a boot floppy.

Email log file had MX loop back errors with mail addressed to what I call the "ShotGun Spam" where they hit you with a few hundred common names; tj@, bob@, sam@, billy@ etc.

Except these were outgoing attempts from my server and none of the names were valid accounts.

Password file seems Ok. Did notice a compiled cgi program, htsearch, in cgi-bin that I didn't put there. Maybe it comes with RH 7.1

Thanks everbody !! I'll be off line until I get SuSE running, or give up and put RH 7.1 back up.

Vern

[Marco van den Hout]

Also check your users' .bash_history for strange commands if the hacker/cracker was logged into a bash shell and didn't cover his tracks.

Maybe you can use 'find' to see which files were changed recently.

You might also want to check for the existence of so-called 'rootkits' on your system.

Be careful because standard programs like 'ls', 'ps' and 'top' may have been replaced as well, to hide certain files and processes.

I don't know why LILO stopped working, it's usually no use for hackers to break the system entirely.

[Jerome Henry]

Hi,

You have here a wonderfull buffer overflow example.
Passing escape strings to your search function prevents apache from filtering (as it's escape string) thus trying to interpret the instruction and conducting to a buffer overflow. On some plateforms, chroot limits the attack, I'm not sure it's the case on RH 7.1.
This attack is classical. Upgrading is the only solution.
BTW on overflowing, there is a big risk in affecting R/W to and from disk if the attack occurs close to kernel space, which is what happened to you, thus conducting some strategic exes to be erased, which is what happened to part of your kernel (so the lilo warning on reboot).

Hope it explains somehow...

Jerome Henry

[Vernon Brown_4]

Ok, I'm almost back on line with a freshly installed SuSE 9; so far only the server console (what I am on right now) has internet access.I have several issues to solve to get my system back on line. I'll start new threads as I work each issue.Thanks for your help so far !!!Vern

[U.SivaKumar_2]

Hi,

The information which you have provided does not suggest any use of http exploit. you have not provided maillog for finding any sendmail exploit patterns.

mail bombing or http dos attacks cannot produce a unbootable kernel. root compromise is necessary.

can you post the /var/log/messages lines logged at that time ?. check the root login logs before that incident ( internal staff also ).

Install Hybrid IDS in the server to get alerts and forensics on suspicious user and network activities.

regards,

U.SivaKumar.

[Vernon Brown_4]

Thanks for your response !!
Good advice and info:
I've restored and recovered now; didn't save the logs.

15K character length records consisting of all escape characters were in the access_log; example above just extend the garbage stuff out to 15,000 characters. That's only what was captured by access_log, no telling what else was sent.

The 66.41.166.198 IP was a different IP for each of the garbage hits but all began with 66. There were Email logged errors "mail loops back to me. MX config problem"
tj@myhost.mydomain ann@myhost.mydomain etc; what I call "Shotgun Spaming".

[Steven E. Protter · Accepted solution]

To completely and totally prevent a hacker from gaining root priviledgs via apache.

You should no matter your os and patch level run it in a chroot jail. Now HP ships its HP-UX depots with the configuiration files all ready to go. All you need to do is run one script and uncomment/modify one line in httpd.conf

If apache is a weapon being used against your system, take the weapon out of their hands.

Here is a procedure:
http://penguin.epfl.ch/chroot.html

Here is my google search:
http://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=chroot+apache+configuration

I foung the procedure I used there or at http://tldp.org

I have it on paper now and I'm 300 miles away from my papers. Bad syadmin.

SEP

[Stuart Browne]

Conceptually (read: not tried), you could just install the appropriate RPM's forcing a different --root= path to set one of these up..

It'd be easier than trying to recompile everything.

It'd also mean easier package updating upon eratta releases.

... Just conceptual thoughts here ...