Out of space on /

[doug mielke]

Some auditors ran a script out of / and ended up filling / with the output. Now mostly cleaned up, I am understandably having some minor weird problems. I understand that filling / can result some tasks not completing, file corruption and more.

I plan on rebooting server in a couple of days, and am cleaning up things like broken profiles and histories. Lp seems very slow, and a few files in roots home are missing (.profile, for instance)

I've never had this happen on HPUX, and My question is what should I look for before and after reboot to make sure I've caught any damage done.
This is an N class running 11.00 and Oracle Financials.

All hints appreciated.

[MarkSyder]

What form of backup do you use?

If you have an fbackup available, you can run frecover to put back files like .profile without overwriting any files which are still there.

If you haven't got an fbackup, then I'm sorry but I can't be of any further help.

Mark Syder (like the drink but spelt different)

[Sridhar Bhaskarla]

Hi,

I wonder if anything happened during the cleanup of / filesystem that might have accidentally erased your .profile etc.,

I don't see any reason why lp would become slow. What is the load on the server before and after the / fill-up?. What do you see in glance. If you don't have glance, how about top and sar?. How about memory utilization?. Any swapping done?.

Reboot is not necessarily a solution if the system is really having problem.

The precaution I would take before I reboot the system is to prepare make_tape_recovery tape and|or make_net_recovery image.

I would also check if boot disks, mirroring is fine before I reboot the box.

-Sri

[Pete Randall]

Doug,

I'll be (one of) the first to say it: The best way to guarantee recovery from just this sort of situation is to have a current make_tape_recovey tape handy and use it. Without it, you're going to be searching and destroying odd little quirks for quite some time, I'm afraid.


Pete

[Paula J Frazer-Campbell]

Doug

Do you a have an ignite tape? or a backup of / files? If so look at their root files and recover what is missing.

Who were these auditors - I would string them up and ensure that they never get access to anything more than a PS2.

Can your company invoice them for cleanup time and lost productivity?


Paula

[Kent Ostby]

Doug --

Some helpful tips:

find . -xdev -size +1000000c

The "xdev" keeps find from looking on mounted file systems.

check out /var/tmp and /tmp and /dev for large files that don't look write as well.

within each of those directories, you can do:

ll | grep ^\- | awk '$5>100000'

Best regards,

Kent M. Ostby

[Helen French]

Unless you are very careful, cleaning up root (/) file system will result in strange behaviour of the system. If you suspect you 've accidently deleted some files which were important (like profiles, configuration files, binary files etc), I would recommend doing a recovery from recent make_tape_recovery tape.

This would save you lot of time and should be easy and safe. You can also try recovering files from your recent backups if that's more easy.

[doug mielke]

I have ignite tapes, Omniback tapes and simple rcp -rp / to an NFS array. I could copy back with a no overwrite I suppose.

The reboot window was previously scheduled to load some new SW. The system load seems normal in sar, vmstat and glance.

I'm still searching around.

Paula,
I'm sure this will be noted by the auditors, as

Risk: Allowed foolish auditors to crap on system.
(no extra charge for this "test")

[Elmar P. Kolkman]

As for the printing: it might be some things got messed up in /etc/lp ...
You might stop your printer scheduler and then restore /etc/lp too.

[Steven E. Protter]

If you have a recent make_tape_recovery backup of vg00, that might be the fastest route to a working system again.

SEP

[Michael Tully]

Once lesson to be learned (yes I know it's too late)

Never ever load a program without first reviewing the contents, particularly unknown scripts. (test server comes in handy)
Another thing I would make sure of is, that you make sure that these baboons go through your change control mechanism (besides being a little nasty) as this also is a test of your skills. (to say no, wait etc) It actually makes you look good. If you can test this on a test server first and you have this result, who looks bad then ...