Need guidance locating source of massive file deletions

[Ray Ball]

We have an HP-UX 11.0 K360 that has been the victim of what appears to be a scheduled, programatic attack that results in a massive file deletion. The contents of many of the filesystems in the /dev/vg00 volume group have been deleted. The top level directories remain and are uniformly time-stamped but all the sub-directories and files below them have been removed. For example, /stand is virtually empty, as is /home & /opt & much of /etc. Many commands in /usr/bin are gone. This has left us with a system that you can't login to (no /etc/password file) and cannot be rebooted (no kernels in /stand). There is no evidence in the syslog, nor any logons revealed by the 'last' command to point to a particular source of the attack.

This system is one of dozens of UNIX servers behind the corporate Internet firewall and we have NO reason to believe this attack is coming from outside the corporation; we believe it to be an inside job done via a Trojan Horse that has suddenly sprung itself upon us (twice in 2 days).

Fortunately we had a session logged in from the system console, thereby affording us the opportunity to recover from our latest make_recovery tape. Once that had been accomplished, we used NetBackup to restore the /home filesystem contents from the very latest full & incremental backups. Of course, this wipes out any evidence that might have remained from the attack.

After the recovery we swept the system (in fact all systems) with a "find / -perm -4000 -type f" looking for obvious setuid programs but no easily recognizible candidates can be seen (which is what I guess one should expect from a hacker).

After having the event occur 24 hours and 30 minutes after the first attack, we are not inclined to recover in the same manner but instead rebuild after a cold install. However, while that might eliminate the source of the problem, I'll probably never know for sure what caused it. So, in summation, I guess I'm looking for advice on how I might locate the source of the comprimised system before I wipe everything out and start from scratch.

[Stefan Farrelly · Accepted solution]

1. Turn on auditing! - and set on audit events for file deletion. This way after it happens again you can find the user/pid of the process doing it.

2. Have a job which runs constantly doing a ps -elf and sending the output to a file on a remote system. You could also grep for rm or find commands and if these come up issue an immediate alarm (echo > console). Run it every 30 secs or so. This is easier than using auditing and should help you catch the culprit in the act.

Good luck and let us know what you find please.

[Paula J Frazer-Campbell]

Hi
Have you looked at the .sh_history and all cron jobs.

Have a look at lastb and last for high level users.

Do netstat -r and look strange routes.

Do a who who -b -r and look at time stamps.

HTH

Paula

[David Lodge]

This sounds very much like an rm gone mad, I'd check your system admins and systems scripts - especially any new ones that have gone in, just in case of a mangled statement, (eg something like:
rm -rf ${DeleteDir}
When DeleteDir may potentially be empty)

I think this is more likely to be due to human/program error than due to a malicious attack...

dave

[paul courry]

A humble suggestion..............

encapsulate the rm command in a script that checks the number of files to be deleted. Anything over x number of files triggers a page to your beeper, logging all appropriate info and a request for password to complete the operation.

[harry d brown jr]

Many moons ago (and I mean many - in the first days of AIX), I came into work early, only to find the operators in a state of panic. One of our servers was not responding, so they rebooted it and nothing happened. I was able to do a "cat *", but nothing was really left on the system. So I instructed the operators to do a recovery.

Everything was fine with the recovery, as was everything else, until that next morning. Again everything went poof. Again we did a recovery.

This time though, I knew we had a very bad job running that was toasting our filesystems. I finally found it, and I was actually the one that caused it to trigger it in the first place. Three days prior, I was deleting garbage from the /tmp directory (well I thought most of it was garbage). It appears I deleted a directory a few levels below /tmp that a cron was trying to "cd" to, to clean up temporary files. But, because I had removed that sub-directory, the 'cd' failed, and the "rm" started at "/" and started deleting files. Therefore, you might want to look at cron and "at" jobs that have "rm" in them, but you also need to look at all scripts that might be executing "rm"'s.

good luck!

[harry d brown jr]

Many moons ago (and I mean many - in the first days of AIX), I came into work early, only to find the operators in a state of panic. One of our servers was not responding, so they rebooted it and nothing happened. I was able to do a "cat *", but nothing was really left on the system. So I instructed the operators to do a recovery.

Everything was fine with the recovery, as was everything else, until that next morning. Again everything went poof. Again we did a recovery.

This time though, I knew we had a very bad job running that was toasting our filesystems. I finally found it, and I was actually the one that caused it to trigger it in the first place. Three days prior, I was deleting garbage from the /tmp directory (well I thought most of it was garbage). It appears I deleted a directory a few levels below /tmp that a cron was trying to "cd" to, to clean up temporary files. But, because I had removed that sub-directory, the 'cd' failed, and the "rm" started at "/" and started deleting files. Therefore, you might want to look at cron and "at" jobs that have "rm" in them, but you also need to look at all scripts that might be executing "rm"'s.

good luck!

Also, if it's not a big deal, rename "rm" to "RM".

[Ray Ball]

It took us three times but the root cause was (as some of you suspected) a self-inflicted wound; a benign remove file script that suddenly turned malignant when someone deleted a directory on Friday morning. The loss of this obsolete directory caused a change directory command to fail, leaving the job in the / directory where it then deleted files older than 5 days.

Thank you all for your insightful replies. We've learned many lessons from this unfortunate episode, including not to prejudge a situation. Indeed, we looked at the script that was the culprit, but we were looking for evidence of tampering and ignored the need to look for a logical way it could suddenly turn against us purely by happenstance.

Thanks again...Ray

[Mark Crosbie]

Hi Ray,

Install IDS/9000, HP's host-based intrusion detection system. It will monitor your system and can tell you immediately when someone deletes even a single file from a critical directory (for your definition of critical).

You run an agent on the server in question and a GUI monitoring the agent on the same or a different machine. Alerts will be sent to the GUI, and logged to a local alert log file. The alerts will contain the following information:
- who did it
- when did they do it
- what file did they touch
- what program did they run
- how was it invoked
- what arguments were passed to the program

In addition, you can configure an automated response script to kick in as soon as IDS/9000 detects that someone deleted a file. The script (for version 1.0) lives in /opt/ids/bin/ids_alertResponse. The script is passed the details of the alert on the command line.

In this example you could instantly kill the process that is deleteing files, or you can restore the files/dirs from a CD as the attacker deletes them!

Email me at mark_crosbie@hp.com for more information.

IDS/9000 is available from http://software.hp.com (product J5083AA) and details are at http://www.hp.com/security/products/ids

Version 2.0 is coming out soon...

Regards
Mark
--
Mark Crosbie IDS/9000 Product Architect
http://www.hp.com/security/products/ids