Server hack through perl

[karimb]

Hello all,

I have a webserver that was overloaded yesterday. I did hardware reset because it was not answering anymore.
After that, I did a # ps aux. I've append the result at the bottom of this message.
What I could see is bunches of perl processes and script, so I did a # find / -name "afg.cgi" but there was no result, same this for all other perl scripts listed by ps !
I've searched all perl scripts modified during the last 2 days but there was nothing weired about the result.
The only solution I had was to rename /use/bin/perl by /usr/bin/perl.old and to kill all processes containing ".pl", ".cgi" and "perl".
The server is ok now but I'd like to reactivate perl. I tried to reactivate it last night but the same problem happened this morning so I deactivated it again.

If you have any answer, it'd be of a great help.

Thank you all.

Herer is the "ps aux"'s result :
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 2036 648 ? Ss Oct08 0:02 init [3]
root 2 0.0 0.0 0 0 ? S< Oct08 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S< Oct08 0:00 [migration/0]
root 4 0.0 0.0 0 0 ? S< Oct08 0:01 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< Oct08 0:03 [events/0]
root 6 0.0 0.0 0 0 ? S< Oct08 0:00 [khelper]
root 85 0.0 0.0 0 0 ? S< Oct08 0:09 [kblockd/0]
root 87 0.0 0.0 0 0 ? S< Oct08 0:00 [kacpid]
root 88 0.0 0.0 0 0 ? S< Oct08 0:00 [kacpi_notify]
root 149 0.0 0.0 0 0 ? S< Oct08 0:00 [ata/0]
root 150 0.0 0.0 0 0 ? S< Oct08 0:00 [ata_aux]
root 151 0.0 0.0 0 0 ? S< Oct08 0:00 [ksuspend_usbd]
root 156 0.0 0.0 0 0 ? S< Oct08 0:00 [khubd]
root 159 0.0 0.0 0 0 ? S< Oct08 0:00 [kseriod]
root 195 0.0 0.0 0 0 ? S Oct08 0:04 [pdflush]
root 196 0.0 0.0 0 0 ? S Oct08 0:05 [pdflush]
root 197 0.0 0.0 0 0 ? S< Oct08 0:06 [kswapd0]
root 198 0.0 0.0 0 0 ? S< Oct08 0:00 [aio/0]
root 203 0.0 0.0 0 0 ? S< Oct08 0:00 [xfslogd/0]
root 204 0.0 0.0 0 0 ? S< Oct08 0:00 [xfsdatad/0]
root 205 0.0 0.0 0 0 ? S< Oct08 0:00 [xfs_mru_cache]
root 884 0.0 0.0 0 0 ? S< Oct08 0:00 [scsi_eh_0]
root 886 0.0 0.0 0 0 ? S< Oct08 0:00 [scsi_eh_1]
root 904 0.0 0.0 0 0 ? S< Oct08 0:00 [mtdblockd]
root 926 0.0 0.0 0 0 ? S< Oct08 0:00 [kpsmoused]
root 937 0.0 0.0 0 0 ? S< Oct08 0:00 [rpciod/0]
root 939 0.0 0.0 0 0 ? S< Oct08 0:04 [kjournald]
root 1024 0.0 0.0 2120 596 ? Sroot 2325 0.0 0.0 0 0 ? S< Oct08 0:24 [kjournald]
root 2842 0.0 0.0 1692 572 ? Ss Oct08 0:07 syslogd -m 0
root 2845 0.0 0.0 1644 388 ? Ss Oct08 0:00 klogd -x
dbus 2878 0.0 0.0 2712 876 ? Ss Oct08 0:00 dbus-daemon --system
root 2923 0.0 0.1 10644 1280 ? Ssl Oct08 0:02 pcscd
root 2966 0.0 0.1 6140 1028 ? Ss Oct08 0:00 /usr/sbin/sshd
root 2979 0.0 0.1 9900 1956 ? Ss Oct08 0:00 cupsd
root 2993 0.0 0.0 2676 864 ? Ss Oct08 0:03 xinetd -stayalive -pidfile /var/run/xinetd.pid
root 3139 0.0 0.0 1872 464 ? Ss Oct08 0:00 gpm -m /dev/input/mice -t exps2
root 3609 0.0 0.1 5460 1108 ? Ss Oct08 0:00 crond
root 3634 0.0 0.0 2208 412 ? Ss Oct08 0:00 /usr/sbin/atd
root 3647 0.2 3.0 43560 30116 ? R Oct08 2:08 /usr/bin/python /usr/sbin/yum-updatesd
68 3660 0.0 0.5 7092 5332 ? Ss Oct08 0:04 hald
root 3661 0.0 0.1 3108 1044 ? S Oct08 0:00 hald-runner
root 3711 0.0 0.0 1904 336 ? S Oct08 0:00 /usr/sbin/smartd -q never
root 3714 0.0 0.0 1628 440 tty1 Ss+ Oct08 0:00 /sbin/mingetty tty1
root 3715 0.0 0.0 1628 440 tty2 Ss+ Oct08 0:00 /sbin/mingetty tty2
root 3717 0.0 0.0 1628 440 tty3 Ss+ Oct08 0:00 /sbin/mingetty tty3
root 3719 0.0 0.0 1628 440 tty4 Ss+ Oct08 0:00 /sbin/mingetty tty4
root 3722 0.0 0.0 1628 440 tty5 Ss+ Oct08 0:00 /sbin/mingetty tty5
root 3727 0.0 0.0 1628 440 tty6 Ss+ Oct08 0:00 /sbin/mingetty tty6
10032 8265 1.6 0.4 6856 4708 ? Rs Oct08 10:25 /usr/bin/perl -w d.pl
10032 8412 1.6 0.4 7156 4736 ? Rs Oct08 10:17 /usr/bin/perl -w ew
10032 8464 3.0 0.4 6936 4852 ? Rs Oct08 18:58 /usr/bin/perl -w avg.cgi
10032 8965 3.2 0.4 7008 4916 ? Rs Oct08 20:07 /usr/bin/perl -w avg.cgi
10032 8986 1.7 0.4 6892 4712 ? Rs Oct08 10:52 /usr/bin/perl -w ihfpugm.pl
10032 8993 1.6 0.4 6916 4696 ? Rs Oct08 10:14 /usr/bin/perl -w n
named 9995 0.3 0.3 32300 3544 ? Ssl 08:40 0:00 /usr/sbin/named -u named -c /etc/named.conf -u named -t /var/named/run-root
root 10072 0.0 0.1 4692 1156 ? S 08:40 0:00 /bin/sh /usr/bin/mysqld_safe --defaults-file=/etc/my.cnf --pid-file=/var/run/mysqld/mys
mysql 10108 0.1 2.0 121928 20720 ? Sl 08:40 0:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql
qmails 10137 0.0 0.0 1680 480 ? S 08:40 0:00 qmail-send
qmaill 10138 0.0 0.0 1640 460 ? S 08:40 0:00 splogger qmail
root 10143 0.0 0.0 1668 368 ? S 08:40 0:00 qmail-lspawn | /usr/bin/deliverquota ./Maildir
qmailr 10144 0.0 0.0 1664 384 ? S 08:40 0:00 qmail-rspawn
qmailq 10147 0.0 0.0 1628 336 ? S 08:40 0:00 qmail-clean
root 10173 0.0 0.0 5820 744 ? S 08:40 0:00 /usr/lib/courier-imap/couriertcpd -address=0 -stderrlogger=/usr/sbin/courierlogger -std
root 10176 0.0 0.0 4668 892 ? S 08:40 0:00 /usr/sbin/courierlogger imapd
root 10183 0.0 0.0 5820 744 ? S 08:40 0:00 /usr/lib/courier-imap/couriertcpd -address=0 -stderrlogger=/usr/sbin/courierlogger -std
root 10186 0.0 0.0 4668 892 ? S 08:40 0:00 /usr/sbin/courierlogger imapd-ssl
root 10191 0.0 0.0 5820 756 ? S 08:40 0:00 /usr/lib/courier-imap/couriertcpd -address=0 -stderrlogger=/usr/sbin/courierlogger -std
root 10194 0.0 0.0 4668 952 ? S 08:40 0:00 /usr/sbin/courierlogger pop3d
root 10200 0.0 0.0 5820 744 ? S 08:40 0:00 /usr/lib/courier-imap/couriertcpd -address=0 -stderrlogger=/usr/sbin/courierlogger -std
root 10203 0.0 0.0 4668 892 ? S 08:40 0:00 /usr/sbin/courierlogger pop3d-ssl
root 10216 0.4 2.6 31492 26656 ? Ss 08:40 0:01 /usr/bin/spamd --username=popuser --daemonize --nouser-config --helper-home-dir=/var/qm
popuser 10217 0.0 2.5 31492 25332 ? S 08:40 0:00 spamd child
popuser 10220 0.0 2.5 31492 25292 ? S 08:40 0:00 spamd child
root 10244 0.1 1.9 42176 19676 ? Ss 08:40 0:00 /usr/sbin/httpd
apache 10254 0.0 0.9 30124 9112 ? S 08:40 0:00 /usr/sbin/httpd
apache 10255 0.0 1.5 42636 15644 ? S 08:40 0:00 /usr/sbin/httpd
apache 10258 0.0 1.3 42176 13876 ? S 08:40 0:00 /usr/sbin/httpd
apache 10260 0.0 1.3 42176 13876 ? S 08:40 0:00 /usr/sbin/httpd
apache 10261 0.0 1.3 42176 13876 ? S 08:40 0:00 /usr/sbin/httpd
apache 10262 0.0 1.3 42176 13876 ? S 08:40 0:00 /usr/sbin/httpd
apache 10263 0.0 1.3 42176 13876 ? S 08:40 0:00 /usr/sbin/httpd
apache 10264 0.0 1.3 42176 13876 ? S 08:40 0:00 /usr/sbin/httpd
apache 10265 0.0 1.3 42176 13876 ? S 08:40 0:00 /usr/sbin/httpd
root 10276 0.0 0.7 43652 7016 ? Ss 08:40 0:00 /usr/local/psa/admin/bin/httpsd
psaadm 10280 0.1 1.1 48076 11188 ? S 08:40 0:00 /usr/local/psa/admin/bin/httpsd
psaadm 10398 0.0 0.3 43652 3644 ? S 08:41 0:00 /usr/local/psa/admin/bin/httpsd
drweb 10405 0.0 2.9 37008 29424 ? Ss 08:41 0:00 /opt/drweb/drwebd
drweb 10406 0.0 2.9 37008 29356 ? S 08:41 0:00 /opt/drweb/drwebd
drweb 10407 0.0 2.9 37008 29360 ? S 08:41 0:00 /opt/drweb/drwebd
drweb 10420 0.0 2.9 37008 29360 ? S 08:41 0:00 /opt/drweb/drwebd
drweb 10421 0.0 2.9 37008 29360 ? S 08:41 0:00 /opt/drweb/drwebd
root 10926 0.2 0.3 11024 2992 ? Ss 08:43 0:00 sshd: root@pts/0
root 10967 0.2 0.1 4868 1476 pts/0 Rs 08:43 0:00 -bash
root 11010 0.0 0.2 10260 2224 ? S 08:44 0:00 crond
root 11019 0.0 0.1 4732 1092 ? S 08:44 0:00 /bin/sh /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t
root 11020 0.0 0.0 4736 632 ? S 08:44 0:00 /bin/sh /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t
root 11021 0.0 0.0 3916 424 ? S 08:44 0:00 cat
root 11024 0.0 0.0 4212 588 ? S 08:44 0:00 tee -a /var/tmp/mail.send
root 11027 0.0 0.0 1684 452 ? S 08:44 0:00 bin/qmail-inject -H --
drweb 11028 0.0 0.1 2164 1152 ? S 08:44 0:00 bin/qmail-queue
root 11121 0.0 0.0 4424 864 pts/0 R+ 08:44 0:00 ps aux
10032 12495 3.1 0.4 7024 4900 ? Rs Oct08 27:42 /usr/bin/perl -w avg.cgi
10032 12563 3.0 0.5 7272 4980 ? Rs Oct08 26:31 /usr/bin/perl -w avg.cgi
10032 12623 2.4 0.4 7020 4936 ? Rs Oct08 21:43 /usr/bin/perl -w qysgm
10032 12630 3.0 0.4 6980 4904 ? Rs Oct08 26:15 /usr/bin/perl -w afg.cgi
10032 12686 0.7 0.4 6484 4040 ? Rs 06:45 0:53 dixfk.pl
10032 12690 1.1 0.4 6812 4612 ? Rs 06:45 1:19 /usr/bin/perl -w avg.cgi
10032 12693 1.1 0.4 6764 4652 ? Rs 06:45 1:19 /usr/bin/perl -w afg.cgi
10032 12702 1.1 0.4 6720 4612 ? Rs 06:45 1:20 /usr/bin/perl -w adf.cgi
10032 12709 1.0 0.4 6728 4576 ? Rs 08:02 0:26 /usr/bin/perl -w afg.cgi
10032 12712 1.0 0.4 6952 4628 ? Rs 08:02 0:26 /usr/bin/perl -w avg.cgi
10032 12724 2.5 0.4 7140 4872 ? Rs Oct08 21:44 /usr/bin/perl -w fn.pl
10032 12734 2.5 0.4 6960 4852 ? Rs Oct08 21:47 /usr/bin/perl -w k
10032 12736 2.6 0.4 7060 4956 ? Rs Oct08 22:39 /usr/bin/perl -w fviam.pl
10032 12745 3.2 0.4 7080 4968 ? Rs Oct08 28:05 /usr/bin/perl -w afg.cgi
10032 12787 0.8 0.4 7100 4752 ? Rs 06:46 0:59 /usr/bin/perl -w iafqvoe.pl
10032 12791 1.1 0.4 6992 4656 ? Rs 06:46 1:18 /usr/bin/perl -w avg.cgi
10032 12801 1.0 0.4 6868 4676 ? Rs 08:03 0:26 /usr/bin/perl -w aubhvtx
10032 12809 1.1 0.4 6908 4640 ? Rs 06:46 1:20 /usr/bin/perl -w afg.cgi
10032 12810 1.0 0.4 6996 4628 ? Rs 08:03 0:26 /usr/bin/perl -w ryo.pl
10032 12812 1.0 0.4 6884 4628 ? Rs 08:03 0:25 /usr/bin/perl -w avg.cgi
10032 12823 3.1 0.5 7188 5032 ? Rs Oct08 27:22 /usr/bin/perl -w avg.cgi
10032 12870 1.2 0.4 6912 4736 ? Rs 06:47 1:26 /usr/bin/perl -w rva
10032 12885 1.1 0.4 6760 4608 ? Rs 06:47 1:17 /usr/bin/perl -w avg.cgi
10032 12892 1.2 0.4 6972 4704 ? Rs 06:47 1:26 /usr/bin/perl -w oogrrsn
10032 13113 1.1 0.4 6832 4652 ? Rs 06:50 1:18 /usr/bin/perl -w avg.cgi
10032 13195 1.1 0.4 6836 4636 ? Rs 06:51 1:16 /usr/bin/perl -w avg.cgi
10032 13256 3.1 0.4 7020 4940 ? Rs Oct08 27:23 /usr/bin/perl -w avg.cgi
10032 13270 1.1 0.4 6752 4620 ? Ss 06:52 1:17 /usr/bin/perl -w afg.cgi
10032 13348 3.3 0.4 7288 4952 ? Rs Oct08 28:59 /usr/bin/perl -w avg.cgi
10032 13361 2.4 0.4 7040 4864 ? Rs Oct08 21:28 /usr/bin/perl -w jpxs
10032 13514 1.1 0.4 6836 4632 ? Rs 06:54 1:16 /usr/bin/perl -w adf.cgi
10032 13536 3.3 0.5 7132 5000 ? Rs Oct08 28:37 /usr/bin/perl -w afg.cgi
10032 13552 2.4 0.4 7096 4936 ? Rs Oct08 21:01 /usr/bin/perl -w xfnwcsb.pl
10032 13766 0.7 0.4 6484 4040 ? Rs 06:57 0:47 ufr.pl
10032 13858 1.1 0.4 6736 4620 ? Ss 06:58 1:14 /usr/bin/perl -w avg.cgi
10032 13869 1.1 0.4 6828 4636 ? Rs 06:58 1:13 /usr/bin/perl -w afg.cgi
10032 13871 1.2 0.4 6880 4620 ? Rs 06:58 1:22 /usr/bin/perl -w q
10032 13874 1.1 0.4 6896 4628 ? Rs 06:58 1:12 /usr/bin/perl -w afg.cgi
10032 13888 1.0 0.4 6804 4588 ? Rs 08:05 0:24 /usr/bin/perl -w avg.cgi
10032 14036 0.8 0.4 6932 4700 ? Rs 07:01 0:52 /usr/bin/perl -w b.pl
10032 14468 1.3 0.4 6844 4684 ? Rs 07:06 1:17 /usr/bin/perl -w jscwjmzk
10032 14535 1.2 0.4 6804 4604 ? Rs 07:06 1:10 /usr/bin/perl -w adf.cgi
10032 14556 3.1 0.4 7032 4944 ? Rs Oct08 27:14 /usr/bin/perl -w avg.cgi
10032 14585 2.2 0.4 6976 4848 ? Rs Oct08 19:18 /usr/bin/perl -w rpixphc
10032 14620 5.4 0.5 7572 5312 ? Rs Oct08 47:01 hyd.pl
10032 14857 2.2 0.4 6996 4740 ? Rs Oct08 19:27 /usr/bin/perl -w sc.pl
10032 14912 3.1 0.5 7256 4976 ? Rs Oct08 27:06 /usr/bin/perl -w afg.cgi
10032 14999 0.7 0.4 6484 4040 ? Rs 07:09 0:41 vifhdp.pl
10032 15194 3.1 0.5 7148 5040 ? Rs Oct08 27:23 /usr/bin/perl -w afg.cgi
10032 15335 2.2 0.4 6940 4748 ? Rs Oct08 19:11 /usr/bin/perl -w z.pl
10032 19420 0.8 0.4 7016 4728 ? Rs 07:15 0:45 /usr/bin/perl -w qjz.pl
10032 21502 0.0 0.1 3752 1608 ? R 08:15 0:00 /usr/bin/perl -w check.cgi
root 21669 0.0 0.2 7932 2956 ? Rs 08:15 0:00 proftpd: graphicmedia - 87.229.26.206: IDLE
root 29324 0.0 0.2 7932 2956 ? Rs 08:23 0:00 proftpd: graphicmedia - 204.12.216.50: IDLE
10032 31524 1.2 0.4 6876 4724 ? Rs 07:32 0:55 /usr/bin/perl -w zguzfo
10032 32450 0.0 0.1 3752 1520 ? R 08:27 0:00 /usr/bin/perl -w hozrqiw.pl
10032 32456 0.0 0.1 3752 1496 ? R 08:27 0:00 /usr/bin/perl -w check.cgi
10032 32457 0.0 0.0 2096 228 ? R 08:27 0:00 /usr/bin/perl -w ebdq

[Steven E. Protter]

Shalom,

It would appear you have undergone a denial of service attack.

Looks like sloloris, but I'm not sure.

Probably what is being exploited is bad perl code.

Steps:

1) Update the system, in case a perl vulnerability is being seen.
2) update httpd software and mysql server if in use.
3) Take a look at the logs in /var/log/httpd to see where this activity is happening.
4) Get a list of every perl script on your system.

Looks to me like the attacker is trying to run his or her own perl script.

perl -w is a debugging switch used to diagnose problems in perl programs. Either the attacker is looking for vulnerabilities in one of your perl scripts, or trying to debug his own code. Thats why I said take a look a the httpd server.

Lastly look at your html content for perl use, there could be a security hole created by bad content. Maybe a bad sendmail form.

Data on slowloris
http://ha.ckers.org/slowloris/

SEP

[Steven E. Protter]

Shalom,

I believe you are definitely being hit by slowloris

iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 100 -j DROP

That might help but you may need to set that limit lower. Very hard to do with high volume sites.

Source website:
http://www.funtoo.org/en/security/slowloris/

SEP