Noobie in need of help from someone with experience

[Greg Gibson]

Hi folks,

I'm an AS/400 specialist who's been put in charge of an HP-UX box (HP3000 series K) running v.11.0.0. Zero training of any kind on the HP - just enough knowledge to be somewhat dangerous. The only app we have running is Oracle.

A rogue process running on the operator console has gone defunct and the console is now hung without a prompt of any kind and I cannot regain control of the console.

The rogue process was an fbackup command that was hung by another unfinished fbackup running from the previous evening that was undetected until the console hung up.

How do I regain control of the operator console, so I can run a shutdown -r there? I'm afraid to try the shutdown -r from a telnet session, for fear it will really mess things up.

I know enough that Oracle must be brought down in an orderly fashion before the shutdown -r is performed, but the real issue is getting control of the console back in my hands, so I can get rid of my rogue processes via "kill -9" and start afresh so I can resume getting nightly backups done.

Thanks in advance for any help you can offer,
Greg

[Florian Heigl (new acc) · Accepted solution]

- probably, hopyfully, You can simply suspend the job by pressing ctrl+z.
- there is no real difference in if You reboot from the console or via a remote session.
- You should first try to get rid of the processes, also I'm courious as to *why* they are hanging - You might be facing a corrupt filesystem.
- at first try shutting down oracle via it's shutdown script in the directory /sbin/init.d
if the shutdown doesn't complete within ~10 minutes, there might still be user sessions active in the database. in that case You either have to gracefully disconnect them or
it's time for a visit to sqlplus
there You'd have to issue "connect internal;" and "shutdown immediate;" (not risky)
or "shutdown abort;" (will cost some performance when restarting oracle, but our DBAs state it is not riskier, still I don't like using it). after that the databases are shutdown to a synced state.

Please also take the time to enter
dmesg (search for recent hardware errors there)
and
more /var/adm/syslog/syslog.log (both application and os errors could be found here, if fbackup hangs You should find at least sometyhing)

also, please take care to remove the tape before rebooting, if rebooting appears neccesary.

during a reboot any messages will be logged to a file called /etc/rc.log - You should read and check it in whole. at the beginning You'll see messages for filesystem mounting - have a sharp look, if any file system get's it's log replayed - this might be a candidate for further checking.

[Greg Gibson]

Thanks for your reply Florian!

I looked in dmesg and saw that my problem was a hardware error due to a power failure that resolved itself, but apparently left the fbackup process that was running at failure-time at a hung state, and causing all successive fbackup command processes and associated fbackuprdr and fbackupwtr processes to also hang.

I am currently running the shutdown -r, invoked from a telnet session, which has since been disconnected during the shutdown. I will go back to the console and monitor events from here on in.

Thanks again, Florian!

Warmly,
Greg

[Greg Gibson]

I now have control of my console and all my rogue processes have been flushed by the reboot.

I am now closing the thread.

Thanks again to Florian for his kind and very prompt assistance! 8^)

Greg

[Greg Gibson]

Closing thread per above post.