HP 9000
cancel
Showing results for 
Search instead for 
Did you mean: 

HP 9000 L-Class rebooting at will

SOLVED
Go to solution
Gary_O
Frequent Advisor

HP 9000 L-Class rebooting at will

Hey guys,
My HP 9000 L-Class keeps rebooting for no apparent reason. I cant figure it out. It
seems to boot like 2 or 3 times in a row...

On December 5th it rebooted 9 TIMES?
Dec 5 00:38:44 acp /sbin/init.d/sendmail[1124]: #### rebooted ####
Dec 5 04:31:50 acp /sbin/init.d/sendmail[1125]: #### rebooted ####
Dec 5 09:04:12 acp /sbin/init.d/sendmail[1126]: #### rebooted ####
Dec 5 10:06:10 acp /sbin/init.d/sendmail[1127]: #### rebooted ####
Dec 5 11:50:03 acp /sbin/init.d/sendmail[1128]: #### rebooted ####
Dec 5 12:16:40 acp /sbin/init.d/sendmail[1129]: #### rebooted ####
Dec 5 13:25:53 acp /sbin/init.d/sendmail[1130]: #### rebooted ####
Dec 5 13:42:46 acp /sbin/init.d/sendmail[1131]: #### rebooted ####
Dec 5 13:52:47 acp /sbin/init.d/sendmail[1132]: #### rebooted ####

Any idea what I can look at to figure out why the machine keeps rebooting.

I look in /etc/rc.log and /etc/rc.log.old and dont see anything good.
I also look in /var/adm/syslog/syslog.log
and OLDsyslog.log - same story....

Is there a way to view the crash / dump info?
50 REPLIES
Bill Hassell
Honored Contributor

Re: HP 9000 L-Class rebooting at will

Look at /etc/shutdownlog. Also look in /var/adm/crash.


Bill Hassell, sysadmin
Gary_O
Frequent Advisor

Re: HP 9000 L-Class rebooting at will

no files in /var/adm/crash
Do I need to turn something on to collect crash data?

and last entry in shutdownlog is Nov 29th.
No entries for the mystery reboots.
Shibin_2
Honored Contributor

Re: HP 9000 L-Class rebooting at will

What is your settings in /etc/rc.config.d/savecrash ?

If your SAVECORE=1 and SAVECORE_DIR=/var/adm/crash , then it will generate crash dump, if system crashed.

If SAVECORE=0 then change it to 1
Regards
Shibin
Gary_O
Frequent Advisor

Re: HP 9000 L-Class rebooting at will

Hi Shibin,
Both SAVECRASH and SAVECRASH_DIR were commented out.
They are now:
SAVECRASH=1
SAVECRASH_DIR=/var/adm/crash

I think all other options are also commented out.
Gary_O
Frequent Advisor

Re: HP 9000 L-Class rebooting at will

By changing /etc/rc.config.d/savecrash do I need to restart the machine to pick up the new options?
Shibin_2
Honored Contributor

Re: HP 9000 L-Class rebooting at will

No need to reboot system. You can restart these services.

/sbin/init.d/crashconf
/sbin/init.d/savecrash

Regards
Shibin
Shibin_2
Honored Contributor

Re: HP 9000 L-Class rebooting at will

Please check /etc/rc.config.d/crashconf

It should be something like this.

CRASHCONF_READ_FSTAB=1


CRASHCONF_REPLACE=0


Read man pages of crashconf and savecrash for more details :)
Regards
Shibin
cnb
Honored Contributor

Re: HP 9000 L-Class rebooting at will

Hi Gary,

Take a look in the GSP logs to see if any chassis code events are logged that may lend a clue to the reason for the reboot. Dump out all of the events prior to and after the reboot and post.

Telnet into the GSP and do a control B

^B



GSP> sl


SL

Select Chassis Code Buffer to be displayed:
Incoming, Activity, Error, Current boot or Last boot? (I/A/E/C/L) e
e

Set up filter options on this buffer? (Y/[N])


The first entry is the most recent Chassis Code
Type + CR and CR to go up (back in time),
Type - CR and CR to go down (forward in time),
Type Q/q CR to quit.


Log Entry # 0 :
[]
Type CR for next entry, Q CR to quit.


Rgds,

Andrew Rutter
Honored Contributor

Re: HP 9000 L-Class rebooting at will

hi,

as cnb says look at the GSP logs and post the last 6-10 entries if you can

Also in STM look at the pim data for the cpu's, this should tell you if it crashed with a cpu or io error

select cpu in stm and run infolog tool on it

Andy
Gary_O
Frequent Advisor

Re: HP 9000 L-Class rebooting at will

Hello Andrew,
I have attached the cstm output for the cpu.
Not sure what to make of it.

I have never run GSP, how is it invoked?

Thanks,
Gary
cnb
Honored Contributor

Re: HP 9000 L-Class rebooting at will

On the system console or telnet into the GSP lan port and then press Control B and follow the example above.


Rgds,
cnb
Honored Contributor

Re: HP 9000 L-Class rebooting at will

Your system firmware looks like it needs to be updated; 39.46 is old for the L-class systems. I would also take a look at the GSP Firmware level. Random reboots were noted on GSP cards with out dated Firmware.

How long has the system been exhibiting these reboots?


Which model L-class is this?


# model

Rgds,


Gary_O
Frequent Advisor

Re: HP 9000 L-Class rebooting at will

Hi CNB,

The model is: 9000/800/L1000-36

I've never updated firmware before..
Now how do you do that, and where do you get the firmware image. lol

By the way, how do you figure out the lan port
for GSP. I dont want to use the console,
cant sweep out the info.

Thanks,
Gary
cnb
Honored Contributor

Re: HP 9000 L-Class rebooting at will

Hi Gary,

See these guides to configure the GSP:

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01868384/c01868384.pdf


http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01404143/c01404143.pdf

Before upgrading the firmware, check the chassis codes to make sure there isn't something else causing the reboots.


The L-Class systems can have two variants of GSP modules (A & B). Usually the rp5400 (L1000) contains the A version, but you should verify.

System Firmware can be updated via HP-UX patches.

System Firmware:
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=322770&prodTypeId=15351&prodSeriesId=2512017&swLang=13&taskId=135&swEnvOID=7


However GSP Firmware cannot be updated via a patch. It has to be done offline and via an FTP server on the same subnet as the GSP module.

Depending upon the version you're GSP is at it may have to be upgraded in stages to get to the lastest version.

For the A revision, see this thread to download the firmware. It's attached in a post from Robert-Jan Goossens:
http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1291837179158+28353475&threadId=1029709

For the B revision:
See this document first:
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c00713649〈=en&cc=us&taskId=135&prodSeriesId=2512032&prodTyp
eId=15351

Firmware here:
ftp://ftp.hp.com/pub/softlib/software7/PF-CCANGSPB0220/pf-41402-1/PF_CCANGSPB0221.tar.gz


Rgds,

Gary_O
Frequent Advisor

Re: HP 9000 L-Class rebooting at will

Hello CNB,

I was able to get into GSP.
The Hardware Revision is 8, and the Firmware
is A.01.06 Aug 1999.

Does this mean it is the "A" version module?

Thanks,
Gary
Andrew Rutter
Honored Contributor

Re: HP 9000 L-Class rebooting at will

Gary,

yes you have revision A gsp, this is also an older version, and could do with updating but not to fix your issue

in the cstm output, it definatly looks like the system had a HPMC which crashed the system. HP is the best to analyze this, but you did suffer a fault in one of the memory dimms which was uncorrectable. This would crash the system.

So, check the infolog of the memory in cstm as well.

Also now that you are in the gsp, look at the logs as instructed and post the last 6-10 if you can. Sometimes more info can be picked out from this.

Andy
Gary_O
Frequent Advisor

Re: HP 9000 L-Class rebooting at will

Hello All,

Looking at cstm for the memory, it has nothing of interest. i.e. The memory error log is empty.

I have attached the latest 5 error logs from GSP.

Indicates
SOURCE: 6 = platform
SOURCE DETAIL: 7 = power monitor SOURCE ID: 0
PROBLEM DETAIL: 4 = internal failure

Although I dont believe we took a power hit....

Thanks for all your help!

cnb
Honored Contributor
Solution

Re: HP 9000 L-Class rebooting at will

It looks like your Platform Monitor card is having problems:

SYSTEM NAME: aphp14_gsp
DATE: 12/05/2010 TIME: 20:37:43
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 6 = platform
SOURCE DETAIL: 7 = power monitor SOURCE ID: 0
PROBLEM DETAIL: 4 = internal failure

CALLER ACTIVITY: 2 = operation STATUS: F
CALLER SUBACTIVITY: 00 = unspecified
REPORTING ENTITY TYPE: 2 = power monitor REPORTING ENTITY ID: 00

0x002000246700200F 00000000 00000000 type 0 = Data Field Unused
0x582008246700200F 00006E0B 0514252B type 11 = Timestamp 12/05/2010 20:37:43

Chassis Code 0x002000246700200F:

0x002008846700200F CODE_HIT_DEFAULT_CASE
0x002008846700200F CODE_EXECUTION_ERROR
0x002008A46700200F ERROR_EXTERNAL_READ

From the Service Guide-

www.dectrader.com/docs/set2/A5191-96019.pdf


6 - Platform 7 - Power Monitor N/A N/A Power
Monitor Replace Platform Monitor card.

You might want to go further back in time to see if any other events are being logged. Or try reseating the Platform Monitor module.


Rgds,
Gary_O
Frequent Advisor

Re: HP 9000 L-Class rebooting at will

Booted twice on the 20th and 7 times on the 21st? I see a bunch of messages from today the 22nd, but it didnt appear to reboot today? system firmware issues?
Nothing in /var/adm/crash? Do you know why?
aargh.....


Dec 20 10:53:22 acp /sbin/init.d/sendmail[1118]: #### rebooted ####
Dec 20 14:02:03 acp /sbin/init.d/sendmail[1137]: #### rebooted ####
Dec 21 00:49:13 acp /sbin/init.d/sendmail[1138]: #### rebooted ####
Dec 21 14:42:55 acp /sbin/init.d/sendmail[1139]: #### rebooted ####
Dec 21 17:58:15 acp /sbin/init.d/sendmail[1140]: #### rebooted ####
Dec 21 18:33:53 acp /sbin/init.d/sendmail[1141]: #### rebooted ####
Dec 21 20:03:22 acp /sbin/init.d/sendmail[1142]: #### rebooted ####
Dec 21 22:19:52 acp /sbin/init.d/sendmail[1143]: #### rebooted ####
Dec 21 23:53:22 acp /sbin/init.d/sendmail[1144]: #### rebooted ####

Could still be power monitor but also seeing
Robert_Jewell
Honored Contributor

Re: HP 9000 L-Class rebooting at will

I would agree you have a flaky power monitor board and should have it replaced. The system is simply resetting, which is why you have limited logs at the OS level. Likely the PM board is not holding a stable voltage at some level and at times this just triggers a log, while others it forces a reset of the server.

Each time this happens you stand chance of having your root filesystem corrupted so I would not delay the replacement if you have a choice.

Good luck!

-Bob
----------------
Was this helpful? Like this post by giving me a thumbs up below!
Gary_O
Frequent Advisor

Re: HP 9000 L-Class rebooting at will

Dumb question, but is the platform monitor card and the power monitor card the same thing.

I found these 2 parts for the platform monitor card, but wondering if there really is a power monitor card.

New Part # Exch. Part #
A5191-60010 A5191-69010 Platform Monitor Board
Andrew Rutter
Honored Contributor

Re: HP 9000 L-Class rebooting at will

hi,

yes thats it

it could be A5191-60110 or 60210 as well, depending upon the revision

-60210 is the latest

Andy
cnb
Honored Contributor

Re: HP 9000 L-Class rebooting at will

Hi Gary,

I came across another L-class server doing the exact same thing as yours, but without the power monitor failure messages.

Did you replace the Platform Monitor and did it resolve the intermittent reboot problem?

Best Regards,


-cnb
Gary_O
Frequent Advisor

Re: HP 9000 L-Class rebooting at will

Hi cnb,
I did 2 things.
I pulled the server apart and reseated
the power monitor card a couple times.
I looked at the contacts on the card, they
seemed to be good / clean.

When I booted up the system, I rebuilt my vmunix. I wasnt sure with all the reboots if something wasnt getting corrupted.

In any case, I have NOT seen the power monitor
error since.

I think I got lucky. Good luck to you!!