A question about startup on L1k

[Wang,MinJie]

Hi every one
I've configured a cluster and one node of it down because of a failing of the package and it never startup any more,when it startup it stops at the following information:
HP-UX Start-up in progress
__________________________

Configure system crash dumps ........... OK
VxVM device node check .................. OK
VxVM general startup ................. OK
VxVM reconfiguration recovery ......... OK
Mount file systems ...................... OK
Update kernel and loadable modules ........ N/A
Initialize loadable modules .............. OK
Setting hostname ................... OK
Start Kernel Logging facility ....... N/A
Set privilege group ......... N/A
Display date ....................... N/A
Copy processor logs to /var/tombstones.... OK
Save system crash dump if needed .............
and then there is no response any more
Can you help me?

[Patrick Wallek]

How long do you let it sit at the "Save system crash dump if needed...."? If the machine crashed, or was TOC'ed by MC/SG, then it is likely that a crash dump was generated.

Depending on the amount of RAM you have, saving of the crash dump can take a while.

[RAC_1]

Did your server crash? It looks that it is saving the crash dump to a file system. Just wait and it should boot.

[Wang,MinJie]

I'm sorry but the "Save system crash dump if needed " message has lasted for one whole day and no reponse occured
What can I do now?

[Wang,MinJie]

No one can help me?
I have to up the thread to call your attention

[RAC_1]

If it is taking so much time, then abort it. (shift+|) Once system boots, you can save crash with savecrash command. First let's see if it boots or not.

[Wang,MinJie]

I'm sorry RAC
You mean I can interrupt this process by pressing "shift+|"?

But I don't think it work.
Any other idea to interrupt the process?

[Wang,MinJie]

Now it boot successfully!
And it reports the following errors:
/var is full
So can I remove all the files below /var/tomestone?
Thx in advance

[RAC_1]

Do not delete files under /var/tombstones.
First check shutdownlog (in /etc) and check why it crashed if it did.

We would need crash dump for analysis later.

[Bill Hassell]

It sounds like your machine did indeed crash and a crash dump was saved in /var/adm/crash. It also sounds like you have a LOT of memory (16Gb or more?). That is why /var is full and as you've seen by now, /var is critical to system operation (cannot be full).

Since you want to know why the system crashed, start a backup of the /var/adm/crash directory. To see how big it is, use this command:

di -ksx /var/adm/crash

That will be the size on Kbytes. Once the tape has finished (or you could move the files to another disk if you have the space), you can delete the files and directories under /var/adm/crash (leave the crash directory). Later, you can contact HP to determine the reason for the crash.

[Wang,MinJie]

How can I analyse the crash information in /var/adm/crash!
That's very complicate to understand
Can you help me?

[RAC_1]

Looks like in your case the full crash dump is not saved. (because, looks like you have not configured /etc/rc.config.d/savecrash and /etc/rc.config.d/crashconf and have not checked with crashconf -v)

Let's try to save crash to some other location. (Make sure that you have enough space there. At least 2xRAM)

savecrash -rv /some_dir_with_enough_free_space

[Wang,MinJie]

Hey RAC
I've got the complete crash dump in /var/adm/crash directory and the total size of /var/adm/crash is 2GB while my physical memory size is 512MB.
And now I want to know is how to process these information and when I can remove all these huge files
Thx

[RAC_1]

cd /dir_where_crash_is_saved.

adb -m /var/adm/crash/crash.0/vmunix

Once on adb prompt, do msgbuf+8/s

This should give you some information.

[Wang,MinJie]

Hey RAC
I pressed "adb -m /var/adm/crash/crash.0/vmunix
it output the following errors:
PA-64 adb ($h help $q quit)
crash dump: No such file or directory
Error on open of corfil = core.
But the "/var/adm/crash/crash.0/vmunix" file actually exits
Do you know why?