grace way to rescue the system

[YLTan]

I encountered a situation where the root'/' filesystem is 100% due to a bug in my backup scripts(i am using fbackup) which causes a superlarge file to be created in /dev/rmt and it cause me unable to login into the server even as root from telnet, console, or using GSP. I have to use another virtual partition to vparboot it to single user to rescue the system.

Qns:
1. Is there a much grace way to rescue this situation other than vparbooting, such as login in as root and manually remove the superlarge file?

2. When the server finally booted to single-user, the filesystem seems to restored back to is original size. Does fbackup remove the superlarge files or does the system automatically remove it when it enters into single user mode?

[S.K. Chan]

When / hits 100% it's very difficult to be able to log in even as root on console. To get that window for root to "squeeze in" and get a shell is almost impossible. However you should still be able to access GSP because it is not part of the OS. To avoid this kind of "accident" it is always a good practice to put in place some pro-active monitoring scripts or tools. In your case you may want to consider monitoring your FS space and when they hit 90% or more, alert you (eg: send email or page), that way you can catch potential problem much earlier. For question 2, it's a normal behavior for that large file to be deleted automatically after a reboot because fbackp hasn't got the chance to complete its task yet. It would be a different story if fbackup completes and fills up your / to say 99%. And if that happens a reboot would not take care of that file it has to be manually removed.

[Tom Geudens · Accepted solution]

Hi,
We've ran into this very same problem (a large corefile in the / filesystem) several times. We've now installed a system which is called "clearroot". It uses the following steps :
0. Allocate file
prealloc /.clearroot 1000000

1. In /etc/services
clearroot 10666/tcp

2. In /etc/inetd.conf
clearroot stream tcp nowait root /opt/clearroot/clearroot clearroot

3. Contents of /opt/clearroot/clearroot
#!/sbin/sh
/usr/bin/rm -f /.clearroot
exec /usr/lbin/telnetd

Now, when you run into the the problem, you telnet to port 10666 of the system. Clearroot "clears" a little space ... allowing you to login (be quick about this). Don't forget to reallocate the file afterwards ;-) !

Hope this helps,
Tom Geudens

[T G Manikandan]

What you can do now is

go into GSP

by pressing Cntl-B.

From there do a RS (reset) on the system.

when you boot again interrupt the boot process interact with IPL
then boot into single user mode and do the needful.

IPL>hpux -is