Was my RH 7.0 box hacked? Can't log in either

[bob smith_6]

Just yesterday, my cable ISP shut me off (I'm on dial up now, ug!) because my IP has "been port scanning" and someone logged a complaint. *I* certainly am not doing this, so after turning on logging on my router, I see that my linux box is sending something every 1 minute 2 seconds. Example:

Date Time Src Src_Port Dest Dest_Port
07/23/2002 20:00:50 192.168.1.2 1034 207.110.0.52 6660
07/23/2002 20:01:52 192.168.1.2 1035 204.127.145.17 6660
07/23/2002 20:02:54 192.168.1.2 1036 154.11.89.164 6660
07/23/2002 20:03:56 192.168.1.2 1037 199.60.228.129 6660
07/23/2002 20:04:58 192.168.1.2 1038 205.188.149.3 6660
and so on

Up until yesterday, my HP pavilion machine has been happily running for 6 months, running the default redhat 7.0 install, with everything installed. I keep it up almost all the time because it's running a webserver.

When I try to log in to the console now, I get this error message:
Unable to load interpreter /lib/lb-linux.so.2

when I control-alt-del to reboot, most stuff shuts down [OK] except it seems that somethings aren't.
http and keytable didn't one time, eth0 another... etc?

This is redhat 7.0, kernel 2.2.16-22smp. Any ideas of

a) how to get back into my machine (console and telnet are both ignored)
b) figure out what happened
c) did someone hack my machine?
d) how to stop the port scanning originating from my machine
e) how to lock it down so I won't be hacked again.
f) if I have to rebuild, how can I without losing all the data on the drive

I'm not an expert. This isn't a commerical website, just running it for a friend's band.
Any ideas would be very helpful.

Bob

[I_M]

Hi

when you boot up, you boot up the system "single user mode" or "no runlevel".

1) boot: linux 1
boot: linux single

2) boot: linux init=/bin/bash

Or boot from rescue CD-ROM and mount the filesystem on HD.

http://lbt.linuxcare.com/download.epl

Check out your ps to find out "strange process" or check crontab to see the process start every one min.

good luck

[Misa]

Just going to spew ideas here. I'm not an expert either.

1. If you didn't do anything to secure your system, I'd say there's a reasonable chance you were hacked. I know of a Linux box that was put up a while back with the specific intent of seeing how long it took to hack it. I think the answer was 6 hours before it was compromised.

2. Suggest that once you recover, you give it your best try to secure the system more than has been done in the past. A good web site is http://online.securityfocus.com . Also, try the Linux version of Tripwire ( www.tripwire.com among other places). Keep up to date with security advisories. Putting a *separate* firewall between your cable modem and your http server is a good idea. Failing that, turn your http server into a firewall.

3. One surefire way to stop the port scans is to unplug the network. I've heard that you can experience trouble for things that other people do with your computer.

4. I'm not finding it right now, but I seem to recall that either on the ITRC or SecurityFocus, there used to be a checklist of "root compromise" for UNIX. It looked good. SecurityFocus doesn't look nearly as good as it used to, sorry. Search Google.

HTH.

[U.SivaKumar_2]

Hi,
it seems to be a clear case of hacking.
Some trojans may be planted in your linux server or hacker himself will be logged in. The
chances for latter one is very less. You have
not secured your linux server and the hacker might have exploited any network service in the
server and got in.

I would suggest a fresh OS installation as lot of daemons and excutables will replaced with hacker's trojans.

I hope you have the backup of the data.

After fresh installation ,
1)stop any unwanted network service.
2)Restrict root access to console only.
3)Put the server behind some firewall. And open
only HTTP , HTTPS ports in the firewall.
Have a look at this link.
http://www.phys.ufl.edu/docs/system/linux.html

regards,
U.SivaKumar

[Peter Kloetgen]

Hi Bob,

you can be sure, your box has been hacked in my opinion. And never forget, a hacker was in your system and he had root status. There is only one possibility to be sure that your computer is safe after:

--> do a new installation!

or are you sure, that you will find all security holes which where installed by your hacker on that box???

Allways stay on the bright side of life!

Peter

[Bob Horton]

Bob,

To be honest, I'd just unplug your host from the network and leave it unplugged. The internet is NOT a safe place to be anymore.

It's just a fad that will blow over soon anyway!

- Bob

[Mark Fenton]

port 6660 is possibly related to mIRC Chat. potentially a trojan is looking for IRC servers to trash.

If you don't have a baseline configuration to check against (with something like tripwire) then you really only have the option of turning the thing off, or wiping and re-installing the OS to be SURE your system isn't compromised.

Look at the Bastille project, also, for hardening tips/tools. It's available @
http://www.bastille-linux.org

hth
Mark

[Trever Furnish · Accepted solution]

Some random notes - hope they help.

If you haven't locked the box down AND kept it up to date with security updates, it should NOT be on the network.

Redhat's security updates are here:
http://www.redhat.com/apps/support/errata/

There have been a *lot* of them since 7.0 was released - when you re-install that system, you may want to start with 7.3 (for which there are also security updates already) to save yourself some update time.

I second the recommendation that you check out the Bastille project.

Your router probably can be used to filter incoming traffic from outside. If not, get a firewall. Allow from the outside only that traffic which you want to expose - it sounds like you should be exposing only http (tcp:80) and ssh (tcp:22).

Use an ssh client in place of telnet.

Disable all services on the system that you don't need. When you get done disabling services (and shutting them off), you should have very little output from the following command:

netstat -tupan

That command lists all the tcp and udp ports open and shows you what process is listening on each. If you don't recognize something, you should question it's presence.

I'd also suggest using either ipchains or iptables (assuming you reinstall with a more recent version than 7.0). RH 7.2 at least includes lokkit to facilitate creating a firewall config.

If you don't need to expose httpd or ssh to all networks, but rather to just a few trusted networks, both apache and sshd allow restricting connections to only those from explicitely listed source addresses. For services started by xinetd, you can use the "only_from" directive within the configuration files for xinetd. See man xinetd.conf.

Do NOT trust your existing system for anything other than raw data files. You can boot off of the CD or a rescue disk (search freshmeat.net for 'rescue' if you don't have one), then copy your files over the network to somewhere else if you need to pull data off the system. Alternatively, you can format only the system partitions during your install if all of your data is on other partitions. It's not enough to re-install over the system without reformatting the system partitions - it easily possible that the system would still be compromised after you complete your re-install if you don't format the system partitions.

[bob smith_6]

Thanks for all the great replies. I have not had time to try to fix this problem (my cable ISP still hasn't even turned my access back on) though I understand that I'll need to keep the box off the network until I fix it.

A few interesting notes... my machine IS behind a linksys router, and I have only ports 80, 21, and 22 available. I have never telneted to the box from outside, only ssh, and then only as a normal user, then su to root.

Knowing this, what vulnerability might someone have used to get access? I'm sure ALL available services are (well, were) running on that box, but how could they use anything but ports 80, 21 and 22? Perhaps a hole in Apache?

Thanks again,
Bob

[Trever Furnish]

Well, again, it's still just an assumption that your system was hacked. The problem is that it's always a safer assumption than assuming that it *wasn't* hacked.

I think many times the first thing to jump into our heads when something unexplained happens is "someone else did it", but often (usually?) it's just that the admin himself did something he did. Um... had that situation myself yesterday. :-)

I don't know what your typical pattern of administration for this system is, so don't take this as an implication that you've screwed up. Just an example...

Example:
- Installing untrusted (or inappropriately trusted) packages. If installing anything on a production system other than a package from a completely trusted vendor (yourself or someone who will be very unhappy when you become unhappy) doesn't make you at least exercise extreme caution (double-check your backups, do a md5sum of your system files for comparison, background-check the application, etc), then you are being either niave or lazy. I've been guilty of both and sorry for it later.

- Just because you have traffic going out doesn't mean it's hacker traffic (again though, it's the safe assumption). Grab a hub and use a sniffer (ethereal, tcpdump, whatever) to determine exactly what the traffic is if you have time.

- Apache has had quite a few security updates, and there were remote exploits possible...

- There was the big zlib compression library bug a few months back that affected lots of components, including the kernel. Again, updates available from most vendors.

- There are always going to be holes the blackhats know about before the whitehats. Maybe you were unlucky. And the cable networks seem to get disproportionately hammered by script kiddies...

I'll stop rambling - I'm certainly no expert. :-)

[K.C. Chan]

Bob,
if your flavor of *nix is BSD, then most likely the hacker may have exploited apache. There was a know exploited on apache running under BSD (Not sure of what version). But to be sure you can check out apache.org under security advisory.

[bob smith_6]

FYI, using the advice above, I was able to log in to the box and collect all my important files (via floppy). I wiped the drive and installed RH 7.3 (seemed much easier to install than 7.0). I've also read up on RPMs and have been keeping the machine up-to-date as new fixes come out. Hopefully, I'll avoid getting hacked in the future : )

Thanks for all the advice,
Bob