/bin and /lib links disappearing...

[Lee Harris_5]

Hi,

We've been seeing a strange problem just lately where by the symbolic links of /bin and /lib usually present in the root filesystem and linking to /usr/bin and /usr/lib respectively just disappear.

I am pretty certain these are not being removed by human hands (nothing shows up in the .sh_history to support this theory, and often no superuser is even logged onto the system at the time), but nevertheless these links seem to just vanish! It is not a major problem, we simply recreate the link, but it can cause problems with applications being unable to execute until we replace the relevant links. The other concern is that if something is deleting these files, then could it turn its attention to other more critical files and delete those?

We've only seen this on a few boxes out of our many hundreds, but there does not seem to be a consistent pattern. We've seen it on 11.11 and 10.20 boxes for example.

The most recent occurence was yesterday with a 10.20 box. We had reports of performance issues, and while investigating found / to be 100% full. We cleared some space from the filesystem, but at the time also noticed that a command we were trying to run was complaining that it could not find /usr/lib/dld.so...this turned out to be because the link to /usr/lib had gone. The other recent occurence was on an 11.11 box and this happened twice within a few days, both bin and lib links disappeared, but in this case the root filesystem never even got close to filling up.

Has anyone ever come across anything like this before???

Many Thanks - Lee

[Steve Steel]

Hi

Sounds like the ghost in the machine but do you have any scripts running via cron at the time they go which do cleanup work.

Something is removing or unlinking the things
I have never heard of it as a function.

I would suggest checking cron activity

If not move rm to another name and make an rm wrapper which notes all parameters before using the renamed rm to do the remove

Then you will know


Steve Steel

[Alex Lavrov.]

Hmm, interesting. I had it once, that /lib->/usr/lib disappeared, I recreated it and evrything works.

To be honest, I never investigated it.

[renarios]

Hi Lee,

Please check your syslog. What does your it say?

Cheerio,

Renrios

[Lee Harris_5]

Hi,

I have checked the syslog on one of the boxes where this has occurred. We worked out that the links must have disappeared around 7am local time.

However there is nothing of any use in the syslog...

May 9 00:02:00 dekbzu01 su: + tty?? root-sched
May 9 06:53:53 dekbzu01 login: get_pwhist_config)): Password history is not con
figured in the /etc/default/security file. Default password history depth 1 is
used.
May 9 07:55:53 dekbzu01 su: + td root-mbsysop
May 9 07:56:05 dekbzu01 su: - td root-mbsysop

This would have been noticed by someone attempting to switch to the mbsysop user. When logging in, this userID runs a menu system which relies upon these links being present.

[Steve Steel]

Hi

Is there any restoring of data via ignite or something when the links disappear.

Is it every day or occasionally

Steve Steel

[Cesare Salvioni]

Hi,
not a real guess on the problem but a just a tip: remember that in order to delete a file what is really necessary is the writing permission on the directory owning the file.
So chek permission on / to see if any user can remove files like /usr or /lib.
Also if sticky bit (t on other execute permision) is set on / only owner user can delete a file.
Once you are sure that only root can delete files from / I would agree with other before me: check in deep crontab work and job executed automatically for mistakes as some variable not well configured or wrong commands executed in certain situations.
Last point /lib and /bin are in the transition links list. That is to say links that HP will remove in a future and created only for backward comatibility. It could be that installing some patch or package these links are removed and must be recreated.
see man tllist

hope it helps

[Lee Harris_5]

Steve - There are no file restores etc going on at the time of the problems. I have, in relation to your first reply however, written an rm wrapper script which I intend to deploy at some of the servers where we've seen the problem.

I've basically got it to log what happens then pass the script arguments to the original rm command for processing.

Should be interesting to see what we get! Thanks for all your input guys...I'll let you know of any further developments.

Regards - Lee

[sreejith_4]

Hi,

Its a good idea to run fsck to verify the disk is OK. we had this issue in Solaris box and there was a disk error.


Thanks
Sreejith M