SCA REPORT very slow on Itanium

[Brian Reiter]

Hi folks,

We have a mixed archtitecture cluster in place (2x RX2660s running VMS 8.3-1H1 and 2x DS15As running VMS 8-3) which is used for software development in OpenVMS Pascal.

We run the same build scripts on both architectures amd after the build completes construct the internals report and LSE package.

The Alpha 8-3 buold of the system (debug and non debug versions) took 37 minues, the Itanium version took 75 minutes. In the past, builds without the SCA reporting have taken roughly the same time. SHO PROC/CONT was used to verify what the command procedure was executing.

The SCA version in both cases is v5.1-01


Hope someone can shed some light. Note, its not really a problem, I'm just very curious as to why it should take so long.


Regards

Brian Reiter

[Ian Miller.]

Are there lots of alignment faults?

Have you look at the performance metrics of the process running SCA ?

SCA 5.1-1 appears to be the latest version (from DECset 12.8 eco2)

[Brian Reiter]

Hmmm, looks like it.

During the build (MMS and Pascal) the alignments faults were in the low hundreds (200 or so), now I'm seeing a max of 382236 with an average of arounf 180000.

Seems very excessive.

I'm assuming there are no workarounds.

[P Muralidhar Kini]

Hi Brain,

>> During the build (MMS and Pascal) the alignments faults were in the low
>> hundreds (200 or so), now I'm seeing a max of 382236 with an average of
>> arounf 180000.
Now that you know that there are alignment faults, next step would be identify
which process is causing the alignment faults.

You can use the FLT trace to find out which process is causing a lot of
alignment faults. Check the following thread -
http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1276597674260+28353475&threadId=1425319
This talks about a similar problem and also talks about using the FLT trace to
find out which process is causing lot of alignment faults.

It might be intresting to know from where those alignment faults are coming.

Regards,
Murali

[Brian Reiter]

Most of the faults appear to be coming out of the batch process doing the build.

Anyway, I've attached a text file showing the results of the FLT tracing.

At the time the batch process was running SCA$MAIN. There are thousands of faults reported by SDA$SHARE+FF300 with the process ID of the batch job. Is this normal? Or is it a side-effect of the tracing.


cheers

Brian

[Ian Miller.]

a pointer to the SDA FLT documentation for anyone that's interested

http://h71000.www7.hp.com/doc/82final/6549/6549pro_030.html#sda_flt

[P Muralidhar Kini]

Hi Brain,

>> this normal? Or is it a side-effect of the tracing.
The "Exception PC" is getting symbolized to SDA$SHARE+... because you are within SDA process.

In this case, you need to look at the EPID 216010B0 (picked up from one entry).
Get in to the process context using "SET PROC" command and then issue the
SHOW TRACE command again. The symbolization would be more informative as it
would be with respect to the correct process. (i.e. EPID 216010B0).

Check the following link -
http://groups.google.com/group/comp.os.vms/browse_thread/thread/ebf9c3ae63a6d14e

Regards,
Murali

[Brian Reiter]

Hi Murali

I tried that. However it made little difference, the SCA$SHARE execption PC is still the most common. For a 5 second trace it accounted for 141432 faults. LIB$GET_VMS was next with a rate of 5366.


cheers

Brian

[Hoff]

There is a DECset ECO resident in the patch area, though I've not pulled that apart to see if an SCA update was included in the patch kit. If (when) you get around to escalating this to HP, they'll want to verify that kit is installed.

Also see if the PC monitoring tool available within SDA helps, or instrument your own builds; get SCA out of the loop. With some basic tracking and a brute-force RMS file, you can trend your build times, both in general and for specific components.

In addition to the alignment faults, looking at other areas of system activity during the builds would be appropriate. While piles of alignment faults will certainly slow your processing (substantially), it is not yet certain what the limiting factor might be here. Builds can tend to be CPU and disk bound, and can suck down copious memory. Differences in fragmentation or allocation or any number of other factors can be in play here.

That 37 minute number on the Alpha implies a very large pile of source code, or some potential for improvement, or both. I'm aware of some Very Large source code builds that conclude in about an hour; builds that are almost certainly far larger than your builds here.

[Brian Reiter]

DecSet has ECO2 applied to it which seems to be the latest.

The systems do contain 2 decades worth of crap, but the 37 minutes was for a normal build followed by a debug build.

I wonder if the issue is to do with the way SCA compiles the reports, it seems to be relying on TPU to perform the bulk of the work.

However, as per usual, I am unable to escalate this to HP. Management are uwilling to to invest in any form of software support contract.

Regards

Brian

[Ian Miller.]

Brian - ensure the cost of this issue (and others) is visible as a cost of not having a HP support contract.

You can report issues informally to HP via email. I have posted a note with a reference to this itrc thread.

[P Muralidhar Kini]

>> Brian - ensure the cost of this issue (and others) is visible as a cost of not
>> having a HP support contract.
Thats a very good point by Ian. The management should know the impact of
various problems faced. Who knows couple more problems like this and they
might change their decision!

Any query that you on OpenVMS, you send mail qeury to OpenVMS.Programs@hp.com.
This way the mail query would get routed to the appropriate VMS Engineering team.

>> You can report issues informally to HP via email.
>> I have posted a note with a reference to this itrc thread.
Looks like Ian has already done that on your behalf.

Regards,
Murali

[Volker Halle]

Brian,


However it made little difference, the SCA$SHARE execption PC is still the most common. For a 5 second trace it accounted for 141432 faults. LIB$GET_VMS was next with a rate of 5366


This exactly demonstrates the problem: the addresses were being symbolized as SDA$SHARE, because they were P0 space addresses and your process context was your SDA process. Once you've set process context to the process running SCA$MAIN, the PC values reported by the FLT trace were correctly being symbolized to be happening in SCA$SHARE !

This is a massive problem in SCA, generating that many alignment faults.

Volker.

[Hoff]

>Thats a very good point by Ian. The management should know the impact of various problems faced.
>Who knows couple more problems like this and they
might change their decision!

Yeah. They might.

What tends to occur when these issues and these technical problems are brought to visibility among customer or ISV management and opened up for more general consideration and discussion usually involves a cost comparison of support with the other available options and other vendors.

One of those other options is a port off of VMS.

I've seen and worked these ports. Sometimes to Windows. Sometimes to Unix or Linux. These customers and these vendors? They're often gone from VMS.

HP certainly has to balance its investments and revenues here. Support has costs. Accepting (and resolving) bug reports can be expensive. Conversely, porting means that the customer is (barring cases where HP-UX is the porting target) gone, and that's expensive both for the customer and for the vendor.

And interestingly, the reverse can also be true. Not accepting bug reports - requiring paid support - has its costs. To the vendor.

A vendor can get blind-sided if unaware of the frequency of a problem, and when the customers don't know about ECOs or don't apply the ECOs. The vendor can be leaving useful information unconsidered, and that's a bad move in any game. The perception that an operating system is unstable or that ECO kits are buggy is difficult to overcome, and rush-fixes for paying (support) customers can be technically challenging and politically expensive. Knowing that SCA is hammering out massive quantities of alignment faults is useful information for a vendor, regardless of its source.

This situation is not fun. But then customer and ISV and vendor management are respectively paid The Big Bucks to make these trade-offs; to schedule and to trade-off and to fund the various and usually-conflicting business requirements; including vendor software support contracts.

[Brian Reiter]

Many thanks for all the replies. To give some hint as to the problem, I repeated the build, once without the SCA reporting and again with.

The build without the SCA reporting completed in 6 minutes 41 seconds (both debug and release builds), the build with SCA reporting completed in 1 hour, 16 minutes and 6 seconds. The build was performed on a RX2660 with 2 CPUS and 4 Gbyte of memory.

Its a large difference which is tricky to
explain away. Thankfully it is only a problem with the builds, the rest of the applications appear not too cause too many alignment faults. In the short to medium term builds will be performed without SCA reporting during the day, if SCA reports are required then the build will be performed overnight.

At least I now have a good trace from ANALYSE/SYS which I will email to HP via the mechanism Murali suggested.

As for the support contract, the management view is that given that the issues tend not to occurr too often and can normally be worked around, hence the contract is not essential. Any savings can then be passed to the customer.

Fortunately we're unlikely to port off of VMS, there are too many years of investment in the systems (millions of pounds and many many years over 2 decades) for our customer to suggest we move off.

Anyway, thanks for all your help


Regards


Brian

[Brian Reiter]

For what its worth, the equivalent non SCA build on the Alpha took 8 minutes to complete, the build with full SCA reports took 30 minutes.

[P Muralidhar Kini]

Hi Brian,

>> The build without the SCA reporting completed in 6 minutes 41 seconds
>> (both debug and release builds), the build with SCA reporting completed
>> in 1 hour, 16 minutes and 6 seconds. The build was performed on a
>> RX2660 with 2 CPUS and 4 Gbyte of memory.
The problem has been narrowed down to the SCA reporting.
In any case the margin is huge.

>> if SCA reports are required then the build will be performed overnight.
right. until the problem is fixed, this would be the workaround that needs to be in place.

>> At least I now have a good trace from ANALYSE/SYS which I will email to HP
>> via the mechanism Murali suggested.
Yes, thats the way forward. Hope the problem gets resolved.

Regards,
Murali