System Crash

[David Child_1]

I have an HP V2600 that crashes every so often.

The dates are sporadic, but it always starts at 03:32 with the following messages: "vxfs: mesg 010: vx_ialloc - /var file system inode 27598 not free" and "vxfs: mesg 016: vx_ilisterr - /var file system error reading inode 9293" (the directory isn't always /var and the inodes are not always the same).

What has made this more difficult is that the system was never set up correctly concerning dump space, so a crash dump was not saved.

There is an OmniBack backup job that takes place at that time.

I have attached an output from the test stations console log. The addresses and stack are the same every time.

I was hoping someone has come across a similar problem and may be able to point me in the correct direction.

We are running HP-UX 11.00, Mar2000 patch level.

[Vincenzo Restuccia]

Check /var/adm/crash with q4.

[MARTINACHE]

Hi,

I suppose that your filesystems are VXFS.

You need to fsck them.

Reboot in single user (hpux -is)

One each unmounted FS do :

/sbin/fsck -F vxfs -o full -y /dev/vg00/r...

For example, this error can appear when you defragment a busy FS (fsadm -e -d ...)

A patch correct this.

Regards,

Patrice.

[melvyn burnard · Accepted solution]

Couple of things here.
1: You appear to have some form of corruption on a file system, in this case /var, which will need fsck'ing in single user mode with -o full option. I would suggest doing it for all of the file systems in vg00 bar / and /stand
2: you need to look at the/etc/shutdownlog to see what type of system panic (if one occurred) you have had. You could also post it here.
3: You need to get the dump story configured and if this persists, get HP to analyse the dump
4: You should update the patchiung level to the latest bundle if you are happy to do this.
HTH

[Steffi Jones_1]

Hi David,

I agree with Patrice. The only other thing... these type of crashes with the see message can also be caused by a bad disk.

So if after a newfs of the filesystem the box crashes again you need to replace the disk.

It is up to you to deteremine how production impacting a crash is and if you risk another downtime. If not, you might want to have a hw person check the disk now and cover both ends.

Good luck,

Steffi Jones

[David Child_1]

Thanks for the quick replies.

Yes, the file system is VXFS (OnlineJFS).

We are currently testing the latest patch bundle on a test box and when that is done we will roll it out to our production and development servers.

I will definately fsck all of vg00 when I can get some downtime.

I did attach the console output to my original message -- the system panic was due to a data page fault

Thanks,
David
p.s. I will assign points in a while :)

[Mark Vollmers]

David-

I had a somewhat similar experience, whether it will help or not. I have 10.20. I had, for a long time, crashes when my backup was running, normally taking out /home. There were scsi and bad inode notes in the syslog (we have a RAID drive). Finally, I shut off and restarted the RAID and then restarted the server. When it ran fsck, it finally tagged the inodes as bad and since has been working fine. Something about the backup was hitting that inode and instead of moving on, it was flipping out and crashing. You can try to shut down everything (drives included) and bring it all back up, watching the start up to see if it finds bad areas on the drives. I kept rebooting and fsck didn't actually tag the inode as bad until I restarted the drive. It kept saying that it fixed everything but the problem kept reappearing.

Mark