What to check for if system security is compromised

[Tim Adamson_1]

If you suspect your system has been compromised, what sort of things would you check to prove one way or another your suspicions?

[Michael Tully · Accepted solution]

Hi Tim,

I guess perhaps if it is a who or a what ... But I'm sure you've worked most if not all of these out already .... Are we talking OS only, partial or through the web?

This posting below from a couple of years back, is quite interesting.
http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x7ba08b55dfd6d4118fef0090279cd0f9,00.html

suspicous entries in '/var/adm/sulog'. I use a very simple script below to monitor it.

# cat /var/adm/sulog | awk '$4 == "-" {print $0}' | grep 'root$'

/var/adm/syslog/syslog.log
/var/adm/sudo.log (or wherever it's kept)

extra entries in /etc/passwd that have a UID of 0
strange entries in the last and lastb outputs. Strange entries in /etc/utmp (use /usr/sbin/acct/fwtmp to read them)

syslog as mentioned in the posting
check root's cron entries

I also recommend having a look at this piece of software:
http://www.software.hp.com/cgi-bin/swdepot_parser.cgi/cgi/displayProductInfo.pl?productNumber=B6849AA

Cheers
Michael

[Pete Randall]

Tim,

I think the best plan for this sort of thing is to keep a current Ignite tape handy. If you suspect something is amiss, use it! Restore the system to a known, safe state. You could spend hours or days checking for a multitude of nefarious hackings - all while the damage is occurring and escalating.


Pete

[Michael Tully]

Hi Pete,

Having an ignite tape is fantastic, but you can't always drop a system and reboot off an ignite tape. This type of fix requires a plan, especially if it is a production server. I am sure my management will tell me to mininise the risks, change the root password, minimise the people who use the system until an approprite WOW is presented to take things back a little.

cheers
Michael

[Pete Randall]

Hi Michael,

I know my answer was a bit on the simplistic side - but that's me! Keep it simple.

You're right that, in the real world, it's not always possible to keep it that simple - but I'd still like to have that Ignite tape nearby!


Cheers,

Pete

[Andrew Cowan]

Remember before you take any corrective actions, ensure that there are no legal implications. Do you have any "duty of care" issues with customer data?

The best course of action is to disconnect the server from the LAN and contact the relevant authorities for advice before you remove any evidence.

I don't wish to sound heavy or paranoid, but you can end-up in a lot of trouble if you don't do this properly, and it later lands in court.

[Steven E. Protter]

Take an Ignite backup of the entire system. That is a valid snapshot that you can turn in to authorities.

Change the root password.

Get a notice to all users that their passwords are going to change.

passwd -f every user on the system.

Now all user priviledges are secure.

Check /etc/passwd

Are there any new users created by root or users with uid zero. Get rid of them.

Check the btmp and wtmp files with lastb

... go crazy checking everything else ...

I think Pete is right. If you've truly been compromised, the only way to be sure you have a clean system is to roll vg00 back to a safe state. If the system is laid out correctly this hould not effect customer data or software installation. I do all that in vg01. I keep my systems in a constant state that can let me roll vg00 back months if need be without much effect on the user community other than password issues.

The thing is, before you lay down Ignite you need to figure out how you were compromised so you can get the Ignite tape with the right date and then take action to plug the hole.

Some things to consider:
Two thirds of compromises are from the inside. Disgruntled but way to smart employees, hackers that got menial jobs so they could work from the inside. People who did the old over the shoulder in operations when the looked up the root password. Stuff like that.

Actions part ii.


get Batille http://www.software.hp.com/cgi-bin/swdepot_parser.cgi/cgi/displayProductInfo.pl?productNumber=B6849AA
and install it on your system.

Get a copy of the book Practical Unix and Internet Security by Farfinkel and Spaford and read it. Implement what it recommends.

If there is money in the budget attend HP's Practial Network Security Course and its Internet Security Course.

SEP

[Con O'Kelly]

Hi Tim

Thats an interesting question, though its often very difficult to determine if a system has been comprised as a good hacker will disguise it extremely well. For example entries in log files will be deleted, commnads such as ps, ls etc may be replaced by the intruders own so that the intrusion is disguised.

A few other things thing that might be worth checking:
- Check ps output for any unusual daemons that might be running.

- Check file modification times etc on files in /usr/bin & /usr/sbin & /sbin to try & determine if a standard binary has been replaced.

CERT provides a checklist for determing if your system has been compromised:
http://unix.about.com/gi/dynamic/offsite.htm?site=http%3A%2F%2Fwww.cert.org%2Ftech_tips%2Fintruder_detection_checklist.html

Cheers
Con

[Tim Adamson_1]

Thanks guys. Sorry for the delay in assigning points. I was told my PC was infected by that virus yet all scans were clean.

The suggestions were excellent. I wanted a list of some files to check, commands to run, etc. The things I came up with based on responses and own thoughts (and I have missed a few things):

Check following files for permissions, ownership, cksum, date modifiction, suspicious entries, etc:

- /etc/passwd
- /etc/group
- /etc/inittab
- /etc/services
- /etc/inetd.conf
- /etc/profile
- /var/adm/inetd.sec
- /.profile
- /.rhosts (shouldn't have any)
- /.sh_history
- /var/adm/syslog/syslog.log

Run a few commands:

pwck
grpck
swverify
last
lastb
at -l
crontab -l

Check crontabs directory.
Check for modem entries, changes, etc

Check certain directories and the contents for recent changes or modifications (/sbin, /bin, etc).

Disconnect server from network.
Change passwords, etc.


Not a bad little list!


Tim

[Michael Tully]

Hi Tim,

In regards to the list of files, why not use 'swverify' to tell you what the permissions are supposed to be on those files. On second thought using 'bastille' should be able to tell you what the ownerships and permissions should be anyway.

Cheers
Michael

[Tim Adamson_1]

Mike,

I used swverify, but once I saw the output in swagent.log, I fainted.

Actually I couldn't install any new software but having a development server proved handy. I used tusc to determine what system calls were used and what files were opened when certain commands were run.

Another important command was mentioned by Con. The ps command is also very useful.


Cheers

[Pete Randall]

Tim,

Also check for suid scripts. If you have any that are supposed to be there, you should know exactly which ones there are and exactly what they should contain. Look for links that would point you to an altered version of a command. Check your PATH(s) carefully to make sure that there's no avenue for a trojan to be snuck in.

I still feel that if you suspect a compromise, your best bet is to get the machine isolated immediately so as to prevent further damage.


Pete

[Steven Sim Kok Leong]

Hi,

One tool you wouldn't want to miss out is chkrootkit.

If your HP-UX has been compromised by an auto-rooter right under your nose by replacing your legitimate binaries with trojaned ones, you can no longer trust the utilities you use e.g. netstat, ifconfig, ls, find etc. to name just a few.

chkrootkit will check for all the signs and alert you, which tends to save quite a fair bit of the manual checking effort required.

During incident handling of Unix or Linux cases, I tend to use it as a starting point of investigation before analyzing deeper.

For more details, access the chkrootkit site at http://www.chkrootkit.org.

Hope this helps. Regards.

Steven Sim Kok Leong

[W.C. Epperson]

As suggested by others, the best time to react to a compromise is before it happens. Having an Ignite tape is excellent insurance for being able to put your system back in reliable working order. You should also have tape backups of the "data" files that change daily. I understand that it can be a difficult business decision to shut down and restore a server, but it's even riskier to continue operating one that's been compromised in most scenarios.

We also take regular checksums of executable files and stash them on a secured server for use in back-checking. Or you can use Tripwire for this:
www.tripwire.org

[Volker Borowski]

Hi,

old thread, but still valid stuff....
http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xd5945220af9bd5118ff10090279cd0f9,00.html

Volker