How to secure servers from previous administrators

[Gail59]

My company is in the process of outsourcing our data center. The question I have is how do I secure our HP machines and to check to see if there are any security holes that might have been opened by previous administrators.

Are there any white papers that would help me effectively secure our systems? Do you have any utilities to examine our existing servers?

We want to make sure that any previous administrator doesn't have a back door into our systems. We want to make sure that we are the only ones that have root rights.

Any information would be helpful and appreciated. Thank You...

[Steven E. Protter · Accepted solution]

Shalom,

Great question.

1) Change the root password and make it complex
2) Check /etc/passwd for other accounts with user id 0 and get rid of them or change password
3) Scan your system for suid programs, especially shells. A shell in a user directory with suid permissions can give a regular user root permissions
4) Run Bastille on the system and close down vulnerabilities.
5) Look at logs for unusual events that only and admin can do.
6) Do a google search on this topic with the site:itrc.hp.com directive and read them carefully.

SEP

[Patrick Wallek]

Back doors are really hard to check for.

Things I do if/when administrator leaves:

1) Change the root password
2) Expire or remove their personal ID
3) Verify access removed from sudoers
4) Verify remote access is removed

You might also want to see if they have anything like VNC running on the systems.

If you use SSH, you might want to change roots public/private key pair.

[Gail59]

Thank you both for the information and your quick replies.

I kinda suspected that about the back door question. I will check these things.

[Steven E. Protter]

Shalom again,

Further advice if applicable. Assuming the admin wasn't fired, it pays to maintain contact in some way with the admin.

Many of us are proud types and when we get talking tend to brag about our abilities and what we've done. Sometimes beer loosens the proud admin's tongue.

I don't do back doors. Console access is a good enough back door to get me into any system that I can boot into single user mode.

In the short run, if there is console access you might wish to password protect the single user mode if the person might have access to the console.

How much work you do here depends on the circumstances under which the guy left and the level of trust.

If the person did and internal transfer and you are concerned the person might undermine you or make you look bad, make your system a fortress and make sure access to any master password documents is revoked.

I live in the middle east. You can't be too paranoid here.

SEP

[A. Clay Stephenson]

If I were going to do this, I would choose a setuid root program that must have the setuid bit set to function normally (e.g. passwd, sendmail) and create a backdoor version that emulates the original perfectly except when supplied with a special argument. If it were the sendmail deamon, for example, it could email me the password hashes for all of the users which could then be attacked with crack.

The way that you protect against this sort of thing is have a database that stores the cksum, timestamps, and length(s) of known good versions and compares with the current version. That's difficult to do after the fact and it's one reason well-written backdoors are so hard to find.

You aren't going to like my answer but the safest approach if this is a real concern is to load the box from scratch but even that is not foolproof because your normal restores could easily restore the backdoor'ed versions.

[A. Clay Stephenson]

Oh, and to really scare you, I'll mention one other technique that I tried once just to see how dangerous it would be. Consider shared libraries. Now consider "improving" one standard function (that might be used thousands of times per day) that has the side effect when it discovers that the effective UID = 0 and when a certain obscure file does not exist that if copies a portion of itself which looks like data but is actually a complete functional statically linked program to this obscure file --- or if I wanted to more devious it would replace a standard setuid utility.

I really can't imagine a more difficult situation than trying to prove than a machine once administered by a knowledgeable but suspect person is actually "clean". That's why building from scratch is really the way to go -- if this is a truly significant concern.

[Bill Hassell]

You really need to get the HP-UX Security book by Chris Wong. The first concern is as mentioned: are the previous admins willing to help or were they fired? If they were fired and had plenty of time to get things in order, you may be better off reinstalling the entire OS rather than looking for holes. Now all of this assumes that these systems are accessible outside the data center. If not, the firewall will help the most, but there may be other holes that the outsourcing company needs to address in network protection.

A simple backdoor is any to look for duplicate user IDs. Run this command:

logins -d

Any duplicates for root should be viewed as an active attack on your system--remove that userID immediately and check the user's HOME directory for suspicious files. NOTE: some admins (wrongfully) assume that having another root user is a good thing so it may be an innocent user ID, but it is one of the first areas that a hacker will try to accomplish.

There is no single set of tests or tools that will answer the question...perhaps the machines were very poorly secured before the outsourcing, and if so, I would download Bastille from the HP software.hp.com website and use that to secure your systems. One of the great features of Unix is flexibility but it is also one of the difficulties in securing these systems. You'll need to spend a lot of time in the Security book. You'll also need to assess the impact of higher security on the system. Sometimes, the changes may disable an application or make the user interface more difficult to use.

[inventsekar_1]

Hi Gail,
Nice and important question.

i am not that much good to answer but i can give some suggestions..

1. As SEP told, maintain a GOOD contact with him. Some good help u can get from him.

2. regarding setuid root program.

3. and configure the ftp logging, and always analyse the log files...

Hope this gave u some idea.
Sekar

[DCE]

A couple of additional suggestions

Check for world writable files. If a file is world writable anyone can modify it to do interesting things when executed.

If you are paranoid - force all of the users to change their passwords - the previous admins may have knowledge of users password - because of password reser requests (or passwords that were never changed from the defaults)

If you are using sudo (or a similar program) examine it's configuration for who is allowed root access.