High kernel mode from SAS

[James Becker]

System info: OpenVMS V7.3-2, ES45 Model 2, 2 EV68 CPUs, 9 GB memory, HSG80

We've used SAS (a statistical analysis language from SAS Institute) on OpenVMS for years. We were finally going to upgrade from SAS 8.2 to 9.1.3, but 9.1.3 has some nasty performance problems on our system.

We've been working with SAS tech support, but the problem remains unexplained and unsolved.

My question for this group is whether the pattern of problems (listed below) closely resembles problems you've encountered before.

SAS programs run well under version 8.2 (our current production version).

Our test cases generally run 1/3 faster under version 9.1.3 than under 8.2, but only when we run them one at a time.

When we run two or more SAS test cases concurrently under 9.1.3, kernel mode time shoots through the roof: often >100% and generally much greater than user mode time.

In the worst example, a job that took less than 2 minutes when it ran by itself took 4.5 hours when we ran two instances concurrently (writing to separate output volumes on the HSG80). The single job ran in about 18 CPU seconds, 00:01:52 elapsed, 17K BIO, 47K DIO, 1K page faults. The two jobs running concurrently averaged 04:15:44 CPU, 04:25:38 elapsed, 17K BIO, 47K DIO, 1K page faults.

When SAS 9.1.3 triggers the high kernel mode, the compute queue runs into the teens (because other people are also on the system), and response time is terrible for everyone.

SAS tech support has run a couple of our test cases, but they can't duplicate the problem.

We've run other tests to see if we can reproduce the problem outside of SAS 9.1.3. Basically, all other tests pass with flying colors. We use DFO routinely, btw.

In short, it seems that something SAS 9.1.3 is doing reacts very badly with something in VMS or RMS that yields huge amounts of kernel mode time.

Any thoughts?

P.S. If we have to give up on SAS on OpenVMS, that's basically the end of OpenVMS at our site.

[Robert Brooks_1]

I have never used SAS, so keep that in mind whilst reading my reply . . .

These are mostly "from-the-hip" musings . . .

1) try turning off one CPU; that'll remove any spinlock contention (it's possible that would show up as interrupt stack time, rather than kernel mode -- that's not my area of expertise . . .) $ STOP/CPU will do the trick.

2) What does "has run a couple of our test cases, but they can't duplicate the problem."
mean? How close is their test environment to your production environment?

3) RMS runs primarily in Exec mode, so it's not likely that RMS is relevant here. It's also not likely an I/O thing, so while running DFO is good, it's also not likely relevant.

4) What patch kits have been installed? While I can't point to any specific reason why that would be interesting, I'd compare your operating environment to that of SAS's, given that they couldn't reproduce the issue in-house.

5) Log a support call, or have SAS log a support call, in case VMS Engineering needs to get involved with both parties.

[Hein van den Heuvel]

Sounds like an interesting problem.

The kernel time it probably spend in locking.

What monitoring tools do you have running?
You may want to get a nice wide T4 data collection going to see what activities correlate with the kernel time.

Do you have a kernel profile tool (PCA, DCPI)

http://h71000.www7.hp.com/openvms/journal/v2/articles/dcpi.html

Also, if the lock manager plays a role, then the SDA LCK extention may be interesting.

Hope this help some,
Regards,
Hein.

[Volker Halle]

James,

using T4 and correlating your kernel mode usage to other activity by using the TLVIZ tool might be the fastest way to find out, where the kernel mode time is spent. As you seem to be able to easily reproduce this, you have about 4 hours time to look at the system while those 2 jobs are running ;-)

As the 2 jobs shows the same amount of BIO,DIO and pagefaults, you don't have to worry about paging and IO. Here are some MONITOR screens to look at look for 'high' rates:

$ MONI IO
- logical name translations
- file opens
- INSWAP

$ MONI LOCK
- ENQ or DEQ

$ MONI FCP
FCP Call Rate
CPU tick rate

$ MONI FILE
any high attempt rate with low hit %

OpenVMS V7.3-2 includes the PCS$SDA extension (PC Sampling), which will allow you to find out, where the system is spending most of it's time:

$ ANAL/SYS
SDA> PCS ! will show you some help
SDA> PCS LOAD
SDA> PCS START TRACE ! let it run for a while
SDA> PCS STOP TRACE
SDA> PCS SHOW TRACE/STAT
SDA> PCS UNLOAD

This would at least tell you, in which modules/routines the system (and/or your processes) are spending most of their time and allow you a more educated guess what the problem may be.

Volker.

[Wim Van den Wyngaert]

Did high water marking of the disk get turned on where it was off before the upgrade ?

SAS is very memory depending. Did the wsextent get decreased of the user ? Or is SAS itself using a lot more and thus requires an increase of wsextent ? Other quota's sufficent ?

Changes in config (SAS and VMS system params) ?

Wim

[Robert Gezelter]

Jim,

High kernel time is suspicious, but it is often the result of a rather mundane situation. The fact that SAS technical support would suggest that the problem is related to some specific configuration on your system, not a generic problem affecting all configurations.

One thing that does come to mind is a dramatic difference in paging rates.

The first step is to look, in detail, at the resources used and configurations of both the old and new systems, and identify what is different between them. File locking, as has been mentioned, is also a possible issue. In all cases, this is most likely an application (e.g. SAS) mis-use of the API, rather than an underlying OpenVMS problem.

It is hard to troubleshoot these remotely, but a detailed review normally identifies the source of the problem.

- Bob Gezelter, http://www.rlgsc.com

[Terry Lykens]

We ran into performance issues as well when we upgraded from 8.2 to 9.1.3 of SAS. We had to increase the user account pglflquo substancially. In addition we ran into a sort problem. All of the user procedures had to include sortsize=0 as an option. Also the data files had to be converted from 8.2 to 9.1.3 formats. This version of SAS is extremely resource intensive. If you can, I'd add more memory. There's also a service pack out for 9.1.3 that you should install as well.

-Terry

[James Becker]

Many thanks for all the considered replies. I'll report back when I've done some more testing. Ah, the wonderful game of Spot The Pattern.

Someone asked for our installed patches: VMS732_PCSI V1.0, VMS732_UPDATE V5.0, VMS732_DRIVER V2.0, VMS732_GRAPHICS V3.0, VMS732_SYS V7.0, VMS732_XFC V2.0, VMS732_DCL V4.0, VMS732_MANAGE V5.0.

For SAS 9.1.3, we're at service pack level 3 and we installed the D9ST02 patch (which applies to the SAS/STAT module).

[Phillip Thayer]

I have seen this type of performance problem on systems where a new disk is configured and initialized and the High-Water marking was turned on (As Wim suggested.) Check the SH DEV/FULL on all the disk drives being used by SAS and see if this alleviates the performance problem.

Phil