K570 automatically reboots 2 to 3 time a week

[Alex Chau]

K570 automatically reboots 2 to 3 time a week.
It give me a warning during a reboot about the memory. Have any idea where I could look? Thanks for you help in advance. AC


/etc/rc.log does not show anything.

/var/adm/shutdownlog shows
Reboot after panic: , isr.ior = 0'1034001c.0'1284b160

/var/tombstones shows nothing.

/var/adm/crash/crash.4/INDEX shows the following:
comment savecrash crash dump INDEX file
version 2
hostname jupiter
modelname 9000/899/K570
panic , isr.ior = 0'1034001c.0'1284b160
dumptime 992976157 Tue Jun 19 13:42:37 CDT 2001
savetime 992977333 Tue Jun 19 14:02:13 CDT 2001
release @(#)B2352B/9245XB HP-UX (B.11.00) #1: Wed Nov 5 22:38:19 PST 1997

memsize 2147483648
chunksize 67108864
module /stand/vmunix vmunix 18022656 647652307

I have also attached the file in /var/adm/crash/crash.4/ana.out
It was zipup due to the size.

[Patrick Wallek]

Check you /var/adm/syslog/OLDsyslog.log for any messages at the end of the log which should right before the crash.

Have a look at this link. This should help you in determining what happened. If it turns out to be a HPMC (High priority machine check) you will need to get the response center involved as it will probably require some new parts on your system.

Here's the link:
http://us-support.external.hp.com/cki/bin/doc.pl/sid=b8e8deaf09b880b691/screen=ckiDisplayDocument?docId=200000054932003

[KapilRaj]

hi,

if you hv multiple CPU's , remove one by one and trace the faulty CPU. once we faced the prob with CPU. The frequency was 2-3 in a day

kaps

[melvyn burnard]

As per your ana.out file, crash event 0 is an HPMC.
Get a harwdare call placed and have this analyzed by your hardware support.

[Javier Ortiz Guajardo]

hi Alex. Maybe you nedd to install one of those patches for system panic and crash.

PHKL_18799 1.0 PANIC in diskquota while running sendmail
PHKL_14026 1.0 mtstart() Page fault,tape0 Hang,Panic patch
PHKL_14765 1.0 Fixes HFS Panic in alloccgblk
PHKL_17043 1.0 Panic possible on heavily fragmented vxfs fs
PHKL_20965 1.0 Incomplete Selective Dump, TOC/Panic Failure

Ask to your HP administrative for more help

i hope it helps

regards

[Magdi KAMAL]

Hi Alex,

You have an excessive errors on I/O subsystems.
Try to use xstm to check which component is bringing this situation.
You got also "File system full" on
/dev/vg03/lvol2 and /dev/vg01/lvol2 ( seems to be the swap area !!! )

You got also an error "Fibre channel driver recieved an interrupt indicating a link failure" : Please verify the connecting cables and Online terminators.

Please can you tell me how much swap space do you have on this server ? Thanks

[Rita C Workman]

I agree with Melvyn, you definately have an HPMC hardware issue.
But I also noted in your syslog files you have errors regarding your fiber connections and timeouts on your disk drives. You may want to check if there is a problem with your fiber cards (I had to replace several some months ago..) and your system is showing the same symptoms. I know you could extend the timeout on your disk..but if the fiber card is going bad...you'll need to replace it.

Just a thought,
Rita

[Alex Chau]

Thanks for all of you for your suggestions.

Magdi, we have 1.5GB RAM and 2.0GB swap.
Magdi and Rita, the system has two Fibre channel cards. We are only using one. I think this may be the reason for the errors.

Yesterday, while running top, I notice only one CPU is being display. For some reason It did not see the other 5 CPUs. (total 6 CPUs)

Today, the display adapter was removed. The system was completely shutdown and restarted. All cpu's is now accounted for. However, an error message show up while the machine is booting up. The message was at the LCD display on the system. ( Warning 7702 Memory ) Does anyone know what it means?

I have enclosed two files. It was the results of running cstm. Thanks again for all of you.

[Patrick Wallek]

The WARN 7702 is not an error. It is just telling you that the memory was not test, it was initialized only.

Have a look at this document for all front panel codes on your system:



http://us-support2.external.hp.com/bhp/bin/doc.pl/screen=bhpDisplayDoc/sid=c2f5019e1c8fd45a01?docId=22969dhtm&docType=HW-DETAILS&infoType=Troubleshooting&DOC_ID=30631wwI&PRODUCT_CATEGORY=HP+Servers%3A%3AHP+9000+Servers%3A%3AK-Class+Enterprise+Server%3A%3AK5xx&PRODUCT_FAMILY=K5xx&PRODUCT=HP+9000+K570+Server+%28A3641A%29&INFO_TYPE=

[Patrick Wessel]

Alex,

Your crash was caused by a runway bus parity error. It is a very nasty HPMC, because you just know that someone flipped a bit on the bus, but not who. (The runway bus connects the CPUs Master Memory Controller and IO converter)
My first step would be an upgrade of the PDC before touching any hardware. Sounds funny to correct an error on the bus with firmware, don?t you think? But the older PDC versions had a problem with the driver for the runway bus, which has been fixed. Also the misleading warnings in the CPU info will disappear with the new PDC.

[Timothy Czarnik]

Alex,

I had this same problem on a K460, with the reboots and the fibre card errors. In my case, it turned out that the fibre card was bad (naughty card!) and was causing the reboots. The reboots ceased immediately after replacing the defective card with a new one.

Hope this helps :)

-Tim

[Angus Crome]

Going along with Timothy and others above. The HSC expansion card on our K460 was flaky.
It took a lot of detailed analysis by our hardware rep on the tombstones to figure this out. We actually replaced the fibre card once prior and it disappeared for about two weeks. We then decided to try the expansion card instead. It has been error free for about 6 months now (knock on wood).

[Magdi KAMAL]

Hi Alex,

For the LCD warrning, you may check in the xstm which will give you the text of this warrning.

You said you have 1.5 GB memory and 2 GB swap, I would increase the swap space to three or four times the memory space to avoid future problems ( Have a look in the book : Essential System Administration. O'Reilly - Aeleen Frisch - page 315 under Managing Paging Space).

[Magdi KAMAL]

Hi again,


####
####

Attention :

Could be the room temprature !!!

[Steve Tasker]

Generally memory errors are single bit errors. I think you are not getting single bit errors.
Most of ios.ir errors are either concerned to System board and (or) CPU's.
It is always better to turn the "Power Key" to service mode. If you keep key at service mode, you can see error messages either on console if connected or in syslog.log file.

Regards
Srini