STM CPU load issue

[marius etsebeth_1]

Hi there,

We have an RP8420 running HPUX11.11 with a number of VPARs. One of the hosts have the following diagnostics software:

OnlineDiag B.11.11.21.02 HPUX 11.11 Support Tools Bundle, December 2008
Version A.61.000

It has the December 2008 patch bundle (including patches listed by swainv).

When the system boots and diagnostics is started, some STM processes that collects hardware info seems to hang (see attachment) and the CPU load keeps on gradually increasing over a period until the machine has to be rebooted again.

Oddly enough, it also seems to affect the vparstatus command, which once run, gets stuck. You cannot kill any of the STM or VPARSTATUS processes.

Another host on this NPAR running older patches & STM (Dec2006 and STM Version A.53.00) has no such problem.

Could this be a firmware issue? I know the firmware is outdated (also in the attachment), but it puzzles me why the host with the newer patches & software is causing this.

Any suggestions? (Other than doing a fimware upgrade ;)

Marius

[Torsten. · Accepted solution]

""Any suggestions? (Other than doing a fimware upgrade ;)"



Not really.

Your FW version is bundle 2.0 from 2004.

There are sooo many fixes and enhancement until today ... I suggest to contact HP and let them update the firmware first.

[Torsten.]

Read the details:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=2512350&swItem=pf-53620-2&prodNameId=392498&swEnvOID=54&swLang=13&taskId=135&mode=4&idx=1

[marius etsebeth_1]

Hi Torsten,

Thanks for the reply. Just another question. What is the upgrade path? I have seen special warnings relating to versions prior to V.3.20 and that's where we are.

I quote:
-----
CAUTION: When updating from a version prior to 3.2, read the version 3.2 release notes carefully. Failure to update FPGAs with 48V power on, followed by updating Firmware files with 48V power off may place your system in an unrecoverable state!
-----

Can you jump straight from our version (V2.00) to the latest, or is there a specific path to follow e.g. apply one after another until the latest or can I at least skip some?

Marius

[marius etsebeth_1]

Hi Torsten,

Thanks for that. We did the upgrade and it didn't go as smoothly as we'd have liked, but the HP engineer eventually got it there after a second attempt (on another day).

However, this created a HUGE problem. Since this upgrade, my VPARs can now only see one CPU each instead of 4 each!

They are running VPAR version A.03.05.07. This version requires iCAP, but we have all the pre-requisites installed (iCAP B9073BA version B.11.11.08.03.01.03), patches - December 2008 plus swainv-patches etc.)

Any idea why we went from a fully functional machine with 4 CPU's per VPAR to 1 quarter of that?

Is there a difference between a normal CPU and an iCAP CPU? We didn't ask for iCAP, but this is forced upon you as it is a pre-requisite for VPAR A.03.05.07...

This is a huge corporation with enterprise wide licensing. This has been fully paid for.

HP now cannot generate us the necessary codeword that iCAP seemingly needs and we have lost the usage of CPU's we paid for!

I see a court case coming up which HP won't win :( This latter is just my personal opinion...

Any ideas on how to resolve this?

Marius

[Torsten.]

Strange. I would like to see

# icapstatus

# parstatus

# vparstatus

# vparstatus -A

[Duncan Edmonstone]

If this system has no iCAP CPUs in it, then the iCAP software should effectively do nothing... it is a pre-req just because the 2 are interlinked in functionality.

>> I see a court case coming up which HP won't win :( This latter is just my personal opinion...

If this is the case I hope you have a call open with HP and have asked for it to be escalated to an appropriste level (as well as getting your HP account manager involved).

The support organisation can't fix stuff if they don't know about it...

HTH

Duncan

[marius etsebeth_1]

Torsten,

As requested.

Duncan,

Yes, I agree with you that the system should not be affected. I tried to generate a codeword myself and the record of the system that was returned to me, indicated 0 iCAP CPU's. Unless HP's records and reality differs...

We have escalated it to our account manager and he has escalated it to engineering, but today is day 4 and people are getting edgy.

Thanks for the support so far.

Marius

[Duncan Edmonstone]

OK interesting points here:

the system seems to think that 8 of your cores (a cell boards worth) are iCAP cores.

so has the hardware spec of this system changed since it came from the factory? (did it come straight from the factory or from a reseller?)

There's a possibility (remote I think), that this system *always* had iCAP CPUs on it, but that cos there was no iCAP software on the OS, you were able to use the cores anyway - that's the way iCAP works...

that said I can find no trace of the systems serial number in the iCAP portal.

looking at the output you could try at least raising the number of CPUs in your vPars with:

icapmodify -s 4

in each one - but if things are as they look from the icapstatus output, I bthink you will find that this fails after the first vPar.

It would be interesting to look at the output of /var/adm/icap.log for each vPar

HTH

Duncan

[marius etsebeth_1]

Duncan,

Log file attached as requested.

Yes, it is entirely possible that the system was delivered with iCAP CPU's and that we only picked it up now with the firmware upgrade (and iCAP sotfware installed - a few months ago with the VPAR software upgrade).

I have tried to do the mod, but I get an error:
---
The Instant Capacity software is in an inoperable state. A software upgrade has been started, but requires the application of an upgrade codeword .....
---

But, good news is that HP ordered 2 new cell boards for us last night.

Marius

[marius etsebeth_1]

Duncan, herewith the attachment. Apologies.

[Duncan Edmonstone]

Marius,

That log file makes no sense to me if you only just installed iCAP, how come all the dates in the file are for Dec 08?

When you say you installed iCAP, are you sure you don't mean you upgraded iCAP?

Also this seems to indicate you only ever had 8 cores active on this cell board, not 16....

Can you tell me what state you *think* the system should be in (how many cores per vPar?)

As for adding new cell boards... well I hope that works out - with the iCAP software still in this state of waiting for a codeword, you might find you can't use any other cores you add... (but don't quote me on that)

I take it you have also tried:

vparmodify -p nbsch028 -m cpu::4

HTH

Duncan

[marius etsebeth_1]

Duncan,

No. You have to go back to the beginning of the thread to see that it all started with the firmware upgrade.

We upgraded the VPAR software (when we had to install iCAP as pre-requisite to VPAR A.03.05.07) some time ago, but didn't have issues with iCAP (well, one small one which I side-stepped).

It was only after the firmware upgrade that this came up. Keep in mind that our firmware was WAAAY old. The newer firmware obviously caters for iCAP (I saw some relationship with PDHC version 24.x?? somewhere).

Anyway, the system had 4 CPU's per VPAR and it was working OK. It's only that they disappeared AFTER the firmware upgrade.

Now, good news is that HP has supplied me with a codeword, which I have applied and I can once again see all CPU's (and they are still there after rebooting all the VPARs :).

Thus, I'm planning on closing this thread, but one final question...

How do I know whether CPU's are iCOD CPU's i.e. how do I distinguish between normal CPU's and iCAP CPU's? (We have already seen that HP's records of our systems won't necessarily be correct in this regard.)

I have another system which needs upgrading soon, and I fear that I'd be in the same position when I upgrade that VPAR and install iCAP.

Marius

P.S. I will assign points when I close the thread.

[Duncan Edmonstone]

Marius,

Good to see you're getting resolution. I did point some folks internally to this thread, and they were asking for the case ID, so they could take a look themselves... could you possibly post that here - they may still be interested even though the case is now apparently fixed...

WRT identifying iCAP processors - its the output of icapstatus you need to be looking at... the same line I identified on the previous post:

Number of cores without usage rights: 8


this should be 0 for a system with no iCAP processors.

Physically there's no difference between an active and iCAP processore - its all down to the iCAP software and whats written into NVRAM on the cell boards.

HTH

Duncan

[marius etsebeth_1]

Duncan,

OK, here's the case number: 4600146131.

The problem is that on VPAR 03.04.06 no iCAP software is installed (we didn't), as it is not a pre-requisite, so I won't be able to identify the CPU's.

Seems like I'll be in for more surprises with the next upgrades.

Thanks all for the help. I'm closing this thread now.

Marius

[marius etsebeth_1]

Resolution was that HP issued a codeword. See above for all the details.