topic Re: High Interrupt CPU when shadow copy in Operating System - OpenVMS

High Interrupt CPU when shadow copy

Cass Witkowski — Sun, 11 Feb 2007 02:50:11 GMT

We have a 4 node ES40 cluster running OpenVMS V7.3. The disks are on HSG80s
When performing shadow copy operations I see high CPU usage. On a single ES40 node with 4 CPUs I see the following:

CPU Interrupt stack %
0 84
1 1
2 89
3 1

CPU 2 is the fastpath for the FGA0 device.

Do these Interrupt stack values seem within reason?

SHADOW_MAX_COPY = 2
SHAD$MERGE_DELAY_FACTOR = 400

Thanks

Cass

Re: High Interrupt CPU when shadow copy

Jur van der Burg — Sun, 11 Feb 2007 03:31:37 GMT

That's a known issue. I fixed this just before I left HP, it has to do with pool management by SYS$DKDRIVER. It allocates and zeroes huge packets in nonpaged pool, and because of the size and if you have a fragmented free list it has to walk that list over and over again, causing the high interrupt stack time. The latest version has some smarts in the buffer management which makes a real difference. You can verfiy the freelist size by going into SDA and do a CLUE MEM/FREE. If it's more than a couple of 1000's you will start to notice this.

Upgrade to at least V7.3-2 and apply the latest fibre_scsi patchkit.

Jur.

Re: High Interrupt CPU when shadow copy

Cass Witkowski — Sun, 11 Feb 2007 04:04:05 GMT

Would this also affect normal operations? I did the clue mem/free and had over 38,000 elements

Re: High Interrupt CPU when shadow copy

Jur van der Burg — Sun, 11 Feb 2007 05:09:16 GMT

Sure, with 38000 it's amazing that the system still runs. Make sure you have the system parameters NPAG_GENTLE and NPAG_AGGRESSIVE set to 100 (default in V8.2).

If you want to play then I have a program to fragment nonpaged pool even worse :-)

Jur.

Re: High Interrupt CPU when shadow copy

Ian Miller. — Sun, 11 Feb 2007 05:13:02 GMT

Cass,
do you have npag_aggressive set to 100, and npag_gentle to set 100? You should as this turns off some pool management algorithms which can lead to fragmented pool.

Re: High Interrupt CPU when shadow copy

Volker Halle — Sun, 11 Feb 2007 08:45:38 GMT

Cass,

with that many packets on the free packet list, you are quite likely to run into CPUSPINWAIT crashes, when one of the CPUs needs to traverse that list to obtain or release a nonpaged pool packet and another CPU is trying to acquire the POOL spinlock.

Volker.

Re: High Interrupt CPU when shadow copy

Hoff — Sun, 11 Feb 2007 10:01:23 GMT

Above and beyond the fix in the ECO that Jur points to, an upgrade to V7.3-2 is going to show multiple other performance benefits. Benefits that can be quite significant. The shadowing- and SMP-related performance updates may well assist with your goals and requirements here, for instance. Various HP customer folks have publicly reported significant speed-ups.

V7.3-2 is a PVS release.

As it is not a major upgrade for OpenVMS Alpha (and was centrally release-numbered to track OpenVMS I64 release numbering), I'd also look at an upgrade to the current release; to V8.3. The OpenVMS Alpha V8.3 release is in the roadmap as being the current release and then probably (if the current HP CVS/PVS process holds) being PVS-eligible (ECOs) through 2011 or so.

Re: High Interrupt CPU when shadow copy

Cass Witkowski — Sun, 11 Feb 2007 23:40:44 GMT

NPAG_GENTLE was 85 and NPAG_AGGRESSIVE was 50.

I have set these 100. Will this help defragement the NPAGE pool or will I have to reboot?

Re: High Interrupt CPU when shadow copy

Cass Witkowski — Sun, 11 Feb 2007 23:53:34 GMT

Hoff,

As for upgrading to a newer version, we are working with our customer to that end.

It is not my job to run the train,
or even ring the bell.
But watch this baby jump the tracks,
and see who catches hell!

Re: High Interrupt CPU when shadow copy

Jur van der Burg — Mon, 12 Feb 2007 01:56:34 GMT

If you changed the parameters you have to reboot to get it fixed.

Jur.

Re: High Interrupt CPU when shadow copy

Cass Witkowski — Wed, 14 Feb 2007 13:26:25 GMT

Thanks again for the quick and helpful responses

Solution is to upgrade to at least OpenVMS V7.3-2 plus Fibre_SCSI patches and set NPAG_GENTLE and NPAG_AGRESSIVE SYSGEN parameters to 100