Root filesystem full preventing login

[John Eaton]

I tried to find an open posting on this, as I wanted to post an interesting resolution, but there were no recent ones. So I'll just post the solution.
On an 11.11 rp7420 (in trusted mode, or TCB), / became 100% full due to a scopeux core file (sound familiar?). Since there were no active logins, we were stuck. You used to be able to ftp in (10.20, trusted), and look around and remove problem files, but not any more.
We found lots of posts with the same answer - crash the box and go to single user mode. Now that is fine on a crash and burn system, but would your top brass want to hear that you crashed the Oracle Financials System, and the DB is hosed? I doubt it! (that's where it happened to us) Hey HP, maybe you could find a way to help us out here...
Now for the interesting solution-
If you are using Omniback (ie DataProtector), you can use it to view the / filesystem, and locate the problem files (pretend you're doing a backup, drill down into / and look around). Once you've done that and found the problem, create a shell script on another server with the command to remove the problem files (make sure your ownership and permissions are set right!), back it up, and then restore it to the broken server. If your pre/post scripts are set to /opt/omni/lbin, then make sure you put it there. Once you have it restored, run a backup of anything (just pick 1 file) on the problem machine, and specify the file you created and restored to it, as the post exec. Run the backup and Viola! the file(s) gone, and you can login with out crashing the server.
I don't have a clue if this works with other backup software. One could be hopeful...

[Prashant Zanwar_4]

Do you have console access. try logging in at console which shall be possible.

Another thing could be

remsh to the box

remsh boxname date
remsh boxname 'rm /path/filename'

Last option can be just taking the box down to single user mode or just rebooting also may clear your problem

Thanks
Prashant

[Pete Randall]

John,

My compliments. That was absolutely brilliant! I hope I'm never in the situation to utilize your tip but thanks for sharing it.


Pete

[RAC_1]

I could not really get what you are trying to say. Backup the file?/ and zero it on post execute script??

The real concern should be, Root is a static FS and should not grow.

Anil

[Prashant Zanwar_4]

I am just sorry...i just thought of solution..Please discard the above...

Sorry again..

Thanks a lot for share
Prashant

[John Poff]

Hi John,

Nice save! We run DataProtector and Oracle Financials also, so I can understand your situation.

I'll definitely remember this one for my little bag of tricks. Thanks for sharing it with us.

JP

[Sridhar Bhaskarla]

John,

That's quite a solution. I used to use OpC (now VP0) to do that stuff in my previous experiences. Simply push a monitor that does the job for you. But using Omniback to do it is a new thing. Even you can recover a lost root password using these solutions.

-Sri

[john korterman]

Hi John,

congratulations on an excellent solution. I have beeen faced with the problem a couple of times - each time on the same, very remote server. And every time the solution has been to pay HP for booting the machine; they suggested no other alternatives.
Unfortunately, we do not have omniback on the machine in question, but as it is due for tsm, it might enable a solution similar to what you have suggested.
Thanks for posting it.


regards,
John K.

[Rita C Workman]

..hmm...I too like your 'little trick'.

But since I'm a stickler for leaving at least one session open on my production boxes (just so I have one), I've never hit your problem.

I'll keep your little tip though...cause you never know when the next guy might accidently kill my session & I'll need this.

You might want to add to your list to keep that one session on production boxes-open.

Rgrds & Thanks,
Rita

[Steven E. Protter]

for future reference

The following entry in /etc/profile will prevent core files

ulimit -Sc 0

SEP

[John Eaton]

Interesting responses. Thanks to all.

Regarding Rita's comment- we NEVER keep root sessions open when not in use. Not even at the console. I'm guessing you don't have an agressive security policy in place.

[Sundar_7]

John,

Interesting thread :-). I was just trying to simulate root full on my test workstation with 11.11.

HP-UX hptest B.11.11 U 9000/782 (ta)

login: root
Password:
Last successful login for root: Mon Oct 25 12:16:44 PST8PDT 2004 on pts/ta
Last unsuccessful login for root: Mon Oct 25 08:56:28 PST8PDT 2004 on pts/ta

msgcnt 24 vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)

Wait for login exit: ..

This is what I get.

But guess what, using CDE, I was able to login in FAIL SAFE mode.

I was not able to login to CDE in the regular mode. Logged in successfully using the CDE fail safe mode, did housekeeping and now I am able to login as root from the command line.

Am I missing something here ? - have you tried this ?

- Sundar

[John Eaton]

Sundar,
Did not try that, as this is a server (and only has root and oracle accounts), it is not running CDE.
Is your workstation converted to trusted?

John

[Rita C Workman]

Mr. Eaton,

Greetings....I never keep my console logged on in the computer room. Too many potential hands that can touch it...

But in my office - I do.

Rgrds,
Rita

[Sundar_7]

John,

Yes, the workstation is trusted. ( You could see the successful/unsuccessful login times. It is displayed only when the system is trusted).

Refer the attachment.

- Sundar

[Thiyagarajan.s]

Hi John and Sundar

Thanks for help
I could use this> in my system which cant be rebooted
If i could have posted the thread i could have given 10 pts to all :)
Thank you very much for all

Thiyagarajan.S

[Senthil Kumar .A_1]

Hi guys,

On this pretext i would like to share my thought in some of these workarounds.

....

This workaround can be used , if any of the other HPUX in your network has SD-OV(see the note at the end). installed...then the job becomes that much simpler.

1) Create a script to query whatever you want to query from the system under concern in your case its the files that has filled root . swpackage the written script as a post install.. When installed this product on the target machine..you will be able to read the output of the script you have written remotely.

2) Then write a remedial script to remove the offending files and swpackage it as post install script and install on the target machine and voila...there you go,,. the / size is restored to normal...

Note: --
SD-UX commands manage software on a local host only. To install and manage software simultaneously on multiple remote hosts (including PCs) from a central controller, you must purchase the HP OpenView Software Distributor (HP Prod. No. B1996AA) which provides extended software management and multi-site software distribution capabilities.

[CAS_2]

Yes, Eaton.

BackUp tool is also used to solve other situations without reboot as forgotten root passwords.

[IE Admins]

Great solution John! At 3am this morning the root filesystem on our backup/monitoring tools server filled up with debug logs (one of the admins forgot to mount a filesystem for debugging).

I followed your steps and was able to cleanup enough space to login and address the root cause. Saved me a 1 hour drive and a potential reboot!

Adam