Auditing changes to a single file

[Hash_1]

Was wondering if it's possible to audit changes to a file such as /etc/group

Been having issues where more users are being deleted than they should, and need to pinpoint what was the last few processes/users that modified that file.

Removing write permissions from /etc/group isn't possible.

[whiteknight]

Hash,

you can consider HIDS

http://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPUX-HIDS

WK

[Hash_1]

Hi, thanks for the HIDS suggestion, but the current way we have things setup is that a certain application team have been allowed (via E-trust) to run a script that removes users from the system as well as from E-trust.

Can't find anything unusual within the script, so am after a way of finding out what was the last few processes/users that did modify that file.

i.e. Basically if there was a way to do exactly the following:

1. Intercept any change on /etc/group
2. cp /etc/group /etc/group.`date +%d/%m/%Y:%H:%M:%S`
3. Log which process/user requested change
4. Allow change to happen

Is there a list of HP-UX binaries that modify the /etc/group file?

So the idea is to allow the change to happen, but to also focus all logging attention to the one file.

[Dennis Handly]

>Removing write permissions from /etc/group isn't possible.

What did you mean by this? That it is impossible because root bypasses this security?
Or you have it wide open to normal users?

>Can't find anything unusual within the script

Sometimes logging may show you something you can't see.

>if there was a way to do exactly the following:
>1. Intercept any change on /etc/group

I doubt it. You might be able to replace the following executables by a tusc wrapper or another wrapper that does 2..4.

>3. Log which process/user requested change

Any user would be root.

>Is there a list of HP-UX binaries that modify the /etc/group file?

Well there is vi. And I assume sam.
useradd/userdel/usermod all modify that file but shouldn't delete groups.
Then there are groupadd/groupdel/groupmod.

[Hash_1]

> What did you mean by this?
We have it as:
# ll /etc/group
-rw-r--r-- 1 root root 10102 Dec 24 12:17 /etc/group

Write permission only by root, and the file is E-trusted. Only a few users have access to the file (sesu) and a handful of application scripts that have been given access by e-trust rules.

The file will randomly "null" itself. Have checked output of application script logs and E-trust logs haven't revealed anything, which is why I'm after a more OS level logging.

>I doubt it. You might be able to replace the following executables by a tusc wrapper or another wrapper that does 2..4.

Any links regarding this? I'm not sure what "tusc wrapper" is...

Would love something like PostgreSQL triggers :-) to monitor accesses to files.

[Dennis Handly]

>Write permission only by root,

root doesn't really need write permission, it can write anyway. But it does give an idea who can write it.

>Only a few users have access to the file (sesu) and a handful of application scripts

I assume it uses su to become root to do the changes?

>The file will randomly "null" itself. Have checked output of application script logs and E-trust logs haven't revealed anything

That's strange. Do the logs indicate the file was opened at all?

If the file's EOF was set to 0, you should have a modification time.

>Any links regarding this? I'm not sure what "tusc wrapper" is...

You run tusc to log all system calls for a program. You might be able to do it as:
1) Rename foo as foo.real
2) move wrapper script into place as foo.

Script for useradd would be something like:
#!/sbin/sh
# run tusc on useradd
path-to-tusc/tusc -a tusc.ua.out -fp useradd.real "$@"

This would log all system calls in useradd to the file tusc.ua.out.

[Hash_1]

> I assume it uses su to become root to do the changes?

Yes, it is an etrust command, which is similar to su.

> Do the logs indicate the file was opened at all?

Nothing in either logs. Nothing in the su log.

> If the file's EOF was set to 0, you should have a modification time.

Have a modification time, but modified by what root level process is what we can't tell.

[Dennis Handly]

>Have a modification time, but modified by what root level process is what we can't tell.

See if that time matches any in your logs. Or in the output of logins from last(1) or any log like sudo.

[Hash_1]

> See if that time matches any in your logs. Or in the output of logins from last(1) or any log like sudo.

Have checked the log, nothing un-ordinary. Our servers are running a lot of batch jobs so sulog is large, but consistent with previous entries (hence nothing unusual).

It has been happening around once a month at irregular intervals.

I have been playing around with auditing system on a test box. I turned on all syscalls for auditing, quickly ran:
cat /dev/null > ~/test_file
Then turned auditing off and went through the output.

I also did:
1. Turn on auditing.
2. vi ~/test_file
3. Delete all lines + save + quit.
4. audsys -f (turn off auditing)

From the audisp of the output files, I was able to see:
cat, /dev/null
vi, test_file

and found the best way to trace what command actually did the change to test_file was to:
1. Search for the PID of "Event=open" that refers to "test_file" as a parameter.
2. Using the PID from step 1, search for "Event=execve" with the same PID. This will show the cat, /dev/null or vi, test_file

Unless anyone has better suggestions I can use?

I realize that since this issue isn't reproducible and happens only like once a month, I will have to leave auditing on for a while with some cron scripts to remove old audfiles once they get filled.

Looks like a lot of effort :-)

[Hash_1]

Hmm, looks like I'll have to monitor Event=admin too.

[Hash_1]

Can someone explain the following:

1. Enabled auditing of all system calls.
2. groupdel testgroup
3. audsys -f

Looking through the audit file, I cannot see any reference to "/etc/group" file.

Wouldn't an Open syscall need the full path name of that file?

[Dennis Handly]

>so sulog is large, but consistent with previous entries (hence nothing unusual).

You don't just look for unusual, you look for something happening that covers the time frame of the corruption.

>From the audisp of the output files, I was able to see:
cat, /dev/null

This shows you why should use the following instead of cat: > ~/test_file

>2. Using the PID from step 1
>Unless anyone has better suggestions I can use?

This looks fine. Unless there is a creat(2) syscall vs open??

>2. groupdel testgroup
>I cannot see any reference to "/etc/group" file.
>Wouldn't an Open syscall need the full path name of that file?

I would expect so. I wouldn't cd to /etc to open just "group".

You may want to use "tusc -fp" to see the actual syscalls. Then match up with your audit file.

[Roberto Arias]

hello try rcs control. please see man page of rcsintro. You can make alias for vi with ci -l and vi for other users

regards

[Bill Hassell]

There are many ways to null the group file:

> /etc/group
cat /dev/null > /etc/group
vi /etc/group

ans so on. Because only root can change the contents of the file, only root users would be suspect. The above list does not list applications (such as eTrust) or scripts run by root. I think you'll find that because only /etc/group is affected that anything having to do with user account maintenance is suspect. You'll have to turn on paranoia logging for eTrust as well as looking at all root user logins. Run a file size checker every minute in cron and when it falls to zero, immediately dump these items into a logfile:

date
ps -ef | grep root
ll /home/*/.sh_history
tail /home/*/.sh_history

The meticulously compare the time stamp with every log in /var/adm and /var/adm/syslog (not just syslog). Also verify that .sh_history lists from above do not have sudo or su's to root. Look specifically at users where .sh_history has a timestamp similar to the failure period.

You may have to remove eTrust's ability to add/delete users until you solve the problem. Use SAM or useradd/userdel in the meantime.

[Hash_1]

Regarding:

> You may want to use "tusc -fp" to see the actual syscalls. Then match up with your audit file.

and

> try rcs control. please see man page of rcsintro. You can make alias for vi with ci -l and vi for other users

Basically, I have no idea what command/process could have nulled /etc/group, could've been vi, cat, etc...


> You'll have to turn on paranoia logging for eTrust as well as looking at all root user logins.

Yeah, will try and get our Unix Security people to do that, but since we have so many batch jobs that need root access, it's gonna be a nightmare for them.

> Run a file size checker every minute in cron and when it falls to zero, immediately dump these items into a logfile: date; ps -ef | grep root ; ll /home/*/.sh_history ; tail /home/*/.sh_history

I would like to do that, but is there any way I can make sure it doesn't get logged into cronlog? I want other cronjobs logged, but not something running every minute (since account team are really cheap when it comes to storage space), especially since we have experienced the issue around once a month (not consistent times however).

[Dennis Handly · Accepted solution]

>I have no idea what command/process could have nulled /etc/group, could've been vi, cat,

I guess I was suggesting to use it on some sample programs, vi, cat and groupdel to see how the file is opened and changed. Then look for these in your audit logs. So if groupdel issues system calls that aren't even logged, you would know why you don't find "/etc/group".

>but is there any way I can make sure it doesn't get logged into cronlog?

Instead of starting every minute, you can just put a sleep there and have the job run continuously:

while true; do
HOUR=$(date +"%H")
# Ignore from 2000 to 0700
if [[ $HOUR -lt 07 || $HOUR -gt 20 ]]; then
exit 0
fi
Do Bill's stuff here
# wait for 1 min
sleep $((1*60))
done

You can remove the time of day stuff, if you want.

[Roberto Arias]

hi again:


Please check history of user's comands.

If the file is modified by a process p`lease senbd us swlist with sw installed

[Hash_1]

Have checked history, but this system has a lot of users constantly being added and removed weekly (however, group names stay fairly constant).