missing files

[Michael O'brien_1]

Hi,

I have had a very strange problem on a server running 11.11, files are going missing mostly from /usr/bin and /opt. I'm then having to recovery the machine using ignite and a backup tool. This weekend is the second time it's happen over the last six weeks. Apart from someone deleting the files which is always a possibility but I hope unlikely.

This weekend files such as cat, bdf, ll, vi, more and serveral others were missing from /usr/bin

Also all the contents /opt/hpservices/lib were missing and all the files apart from the links were missing from /opt/dce/bin

The problems comes to light following a reboot of the server.

Any ideas what might be happening would be most welcome.

Thanks
Michael

[Ganesan R]

Hi,

Check the history file if somebody removed the file unknowingly using rm as root user.

Also check if any script doing that in the background.

[Pete Randall]

Other than "someone deleting the files", the only thing I can think of would be a malformed cron job. Typically admins use a cron'd find command to remove unnecessary files, something like "find /tmp /var/tmp -mtime +7 -exec rm {} \;". Do a crontab -l and take a look for anything similar.


Pete

[James R. Ferguson]

Hi Michael:

A properly configured system isn't going to allow a non-root user to remove files from directories like '/usr/bin' and //opt'. Given that, if you have any scripts that you run as the root user, you might want to look at those scripts for 'rm' commands.

Pay attention to any script that assumes a change directory operation works and/or assumes that variables are always defined. For example:

# cd ${SOMEPATH}
# rm ${LIST_OF_FILES}

Scripts like this should have :

# set -u

...to expose undefined variables

Too, make sure you trap 'cd' failures, like:

# cd SOMEPATH || { exit 1; }

The 'root' user should always have a HOME directory that isn't just '/'. Either '/root' or '/home/root' are appropriate. In this way, as simple 'cd' or 'cd ${SOMEPATH}' [when "SOMEPATH" isn't defined] doesn't leave you sitting in '/' when you don't want to be!

This all said, i would run :

# swverify \*

...to help expose the extent of your damage. You may have to rebuild your server from a good Ignite backup if the damage is severe.

Regards!

...JRF...
Regards!

...JRF...

[Michael O'brien_1]

Thanks you for your responses, the two things which crossed my mind was a malformed cron job and user error etc.. I still investigating these possibilities.

The one thing which is bothering me is the randomness of the files which are missing and also their doesn't seem to be a pattern.
Somtimes serveral files from /usr/bin directory and other instances all the files from the directory are missing. But in the /opt/dce/bin directory the binaries we missing but the symbolic links remain untouched? Could this be an inode issue?

Only vg00 seems to be effect the application SAN filesystems are okay

thanks
Michael

[James R. Ferguson]

Hi (again) Michael:

> The one thing which is bothering me is the randomness of the files which are missing and also their doesn't seem to be a pattern.

You might look for a 'find' command gone wrong.

Regards!

...JRF...

[Dennis Handly]

>JRF: You might look for a 'find' command gone wrong.

You forgot some descriptive words "horribly horribly wrong". :-)