things went bad when do this fuser -kc /var

[hpuxhelp]

I did that, and now i tried to login on..it won't let me in...from the console.... what 's wrong ?

[hpuxhelp]

it kept saying wait for login retry....

[harry d brown jr]

You killed every process that had /var open. Which on my 11i machine is about 39 root processes.

You probably need to reboot.

live free or die
harry

[Patrick Wallek · Accepted solution]

You shouldn't have done that.

You should have just rebooted the machine into single user mode.

The /var filesystem is used by LOTS of OS processes and by issuing the 'fuser -kc' you just killed a lot of VERY IMPORTANT HP-UX processes.

You may have to now do a CTRL-B and at the prompt do either a TOC or an RS to restart the system. If you have another session logged in somewhere, try rebooting the system from there before you do the above.

[John Poff]

Hi,

I think you just sent the SIGKILL signal to every process that had the /var filesystem open. Ouch!

What problem were you trying to solve?

JP

[Uday_S_Ankolekar]

It looks like you killed everything mounted from /var

Does bdf shows you /var mounted??
If not try mounting /var again and try to login.

[Patrick Wallek]

John,

I think he was trying to kill all the processes in /var so he could extend it. See this post:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x4d95afe90f1cd71190050090279cd0f9,00.html

By the way, do NOT do the 'fuser -kc' command on /tmp. If you do, you are likely to run into some of the same problems you have right now as a lot of root processes access /tmp as well.

[hpuxhelp]

i meant to type ku .......is there a way i can do this without have to shutdown....
there must be a way to recover this....
i can't reboot right now..

[Jeff Schussele]

Well - that should go right to the top of your
"Things to Never Do Again" list.

Without OnLineJFS, the ONLY way to extend /var is in sigle-user mode. There's simply NO way around this fact.

Rgds,
Jeff

[Patrick Wallek]

I really don't think you are going to have a choie in the matter. I think a reboot may be your only option.

If you have a support contract on your machine, Call HP and tell them what has happened. They may be able to help you.

To see what processes are using a mountpoint you should do:

# fuser -cu /mountpoint

[hpuxhelp]

yeah..learn my lesson..
but strange things..if I come in as a user and su - root
and it is ok for me to do ..but i can't do the same from the console work ???
now since i am in as root, is there a way i can repair the damage ???

[Martin Johnson]

You say you can't reboot right now. You don't have a choice. You have inadvertantly killed over half the OS. Your system can barely function. I would question the integrity of any transactions occurring on the system.

Reboot now!

HTH
Marty

[John Poff]

So you are on a 10.20 box? I just did an 'fuser' on my /var and I have 221 processes that have /var open, out of a total of 526 processes. Here are some of the root processes that you probably just killed:

/usr/lbin/cmsnmpd
/usr/sbin/syslogd
/opt/omni/lbin/rds
/usr/sbin/swagentd
/opt/dce/sbin/rpcd
/usr/sbin/netfmt
/usr/sbin/ptydaemon
/usr/sbin/rbootd
/usr/lbin/ntl_reader
/opt/perf/bin/perflbd
/opt/perf/bin/alarmgen
sendmail:
/usr/sbin/cron
/opt/perf/bin/midaemon
/usr/lbin/cmcld
/usr/sbin/envd
/opt/omni/lbin/mmd
/usr/lbin/cmsrvassistd
/opt/perf/bin/rep_server
/opt/perf/bin/ttd
/opt/perf/bin/scopeux
/opt/omni/lbin/crs
/usr/dt/bin/dtlogin
/opt/perf/bin/agdbserver


Now, if you are really adventurous and motivated [and you sure seem to be so far!], and if you have a root connection open to the box, you COULD try to restart these daemons via /sbin/init.d somehow. Maybe.

Or you could just reboot to single user mode and expand the filesystems as you will need to do anyway.

In situations like this I've found it to be quite convenient to trip over the power cord. ;)

JP

[hpuxhelp]

i do have access to root now...
how do i start this using the daemon

[John Poff]

Alright! That's the spirit! Rebooting is the easy way out. Let's see if we can really recover this system.

Let me login to one of my 10.20 boxes and see what scripts we can run in /sbin/init.d that might get things restarted.

JP

[hpuxhelp]

ok...thank you so much.... i hate if i have to reboot this box... i am in dead water...now ...

[John Poff]

For starters, let's see if we can get the first three processes going by using /sbin/init.d scripts. Try these:

/sbin/init.d/envd start
/sbin/inid.d/ptydaemon start


See if those two get the envd and ptydaemon daemons going. If so, we might be able to pull this off. I won't claim to know which daemons are the most important, but we'll start with those.

JP

[hpuxhelp]

/sbin/init.d/envd start
Environmental daemon started
# /sbin/init.d/ptydaemon start
Starting the ptydaemon

yes..it seems the2 daemon is working at this point..so waht's next??
i am so nervous

[hpuxhelp]

i did ps -ef
and i saw that lvmkd is not running...and it doesn't look so good

[John Poff]

No lvmkd daemon. That's a bigger problem. You are closer to having to reboot now. I don't think there is a way to restart it manually.

http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000062942884

JP

[John Poff]

Yep, it doesn't look good. I can think of a couple more things to try, but if they don't work they will probably cause your system to reboot, which is what needs to happen anyway.

My advice? Go to your boss and tell them what happened, and that the system needs to be rebooted now. The daemons that were killed have probably brought the box to a stand still anyway. In cases like this I've always found that honesty is the best policy and will serve you better than anything else. Tell them what you did, what you learned, and that it won't happen again.

Or, you could take my earlier advice and trip over the power cord. Tomorrow is January 1st. You could blame it on a Y2K+3 bug. You never know. Someone might actually buy it. ;)

JP

[hpuxhelp]

I hit CtrolB on the node, and should i do reboot from there or from a open telnet session...

[John Poff]

If you still have a root telnet session open, I'd suggest doing a shutdown from there. At least that way the system will try to shutdown the remaining processes gracefully.

Good luck, and Happy New Year!

JP

[hpuxhelp]

thanks for all your help....