topic Re: What is the performance impact of reporting alignment faults? in Operating System - OpenVMS

What is the performance impact of reporting alignment faults?

Glenn Wolf — Tue, 24 Jan 2012 21:37:53 GMT

We are porting an application from Alpha to Itanium, and want to keep an eye on alignment faults due to their performance impact. The critical part of the application consists of a number of transaction server processes that field requests from other hosts on the network. We already have instrumented these servers to track the transaction duration and a few other parameters; it seemed logical to add alignment fault count to our instrumentation.

To do this , we changed our server processes so that before it processes a transaction, it now calls SYS$START_ALIGN_FAULT_REPORT, and after the transaction has been processed, it calls SYS$GET_ALIGN_FAULT_DATA, followed by SYS$STOP_ALIGN_FAULT_REPORT. All of this work is bracketed by a call to LIB$INIT_TIMER, and calls to LIB$STAT_TIMER to collect elapsed and CPU time, direct and buffered I/O and page fault counts. This data, along with the number of fault entries returned by SYS$GET_ALIGN_FAULT_DATA is logged for this transaction.

The issue that we have is that adding the code to collect the number of alignment faults drastically increases the execution duration of the transaction, as reported by LIB$STAT_TIMER. In a simple, isolated test, a transaction that without aliignment fault reporting takes 5 msec, executing the alignment fault reporting and collecting calls increases this time to almost 400msec. In this case, 486 alignment faults were reported. We added the enabling and disabling of the alignment fault reporting via a logical name that is checked before each transaction is serviced, and the test results are very repeatable - Call the alignment fault reporting services, the transaction duration is about 400msec. Don't call the alignment fault services, and the transaction duration is about 5msec.

This increase in elapsed time will not work in production - our application currently runs at about a 21 transactions/second rate, and can be very bursty.

I've tried both the AFR$C_BUFFERED and AFR$C_EXCEPTION approaches - both have similar performance impacts (the exception approach seems marginally worse).

My questions are:

1) Is the above implementation of these system services the way the are intended to work? Or should I just poll at the start of the transaction to clear any residual faults, and then poll at the end to get the count, leaving the reporting enabled the entire time the server is running?

2) Would I have better results with the SYS$PERM_REPORT_ALIGN_FAULT approach?

3) Has anyone else seen this kind of performance impact from using these system services?

By the way, our test environment is an HP rx2620 (1.60GHz/6.0MB) with 16GB of memory, running OpenVMS V8.3. We see similar behavior on our intended production environment: HP BL860c i2 (1.60GHz/5.0MB) 16 cores with 32GB of memory, running OpenVMS V8.4.

Re: What is the performance impact of reporting alignment faults?

John Gillings — Tue, 24 Jan 2012 22:24:15 GMT

Glenn,

System service calls are potentially expensive. I suspect your observed problem is due to the START/STOP added to each transaction, not the tracing itself.

What happens if you move the START and STOP higher up the stack and just exectute the GET on the transaction? Don't worry about "clearing", just keep track of the previous value to calculate the cost per transaction.

You could also build a simple test program that measures the cost of the START/GET/STOP cycle, possibly with and without dummy loads with known numbers of alignment faults.

My assumption is these services are really only of use during development to find sources of alignment faults. There's therefore no real incentive to make the services themselves very fast. One would hope that in a stable, production system, the number would be more or less constant, and not interesting enough to continue to track. Do you see any variation in transactions?

Can you identify what's generating the 486 faults you're seeing, eliminate them and then remove or disable the sampling code for production?

Re: What is the performance impact of reporting alignment faults?

John McL — Tue, 24 Jan 2012 22:42:44 GMT

Is there any reason why you would expect the alignment faults to vary much when the code is on Production? If not I'd take some typical data back to your development system and use it to refine the code, and then not include your diagnostic code when you go to Production.

Also, I wouldn't use LIB$INIT_TIMER and LIB$STAT_TIMER, the latter I ssume is called 4 times to get all the data you want. Use SYS$GETTIM and SYS$GETJPI, the latter with an item list that includes all the requested values, and calculate the difefrences yourself.

Minimising calls to LIB$ and SYS$ routines is a good idea because there's overheads like checking that you have write access to memory where output values will be written, and of course all the internal code in those routines to deal with different input.

Re: What is the performance impact of reporting alignment faults?

John Gillings — Tue, 24 Jan 2012 23:22:12 GMT

Glenn,

Simple experiment. attached. The routine "bad" performs 1000 unaligned additions. The first loop repeats the routine 100000 times. The second repeats it (to remove the influence of the image activation pagefault). The third does the same thing surrounded by your START/GET/STOP, the fourth moves the START and STOP outside the loop.

Here are the results.

ELAPSED:    0 00:00:01.02 CPU: 0:00:01.03 BUFIO: 0 DIRIO: 0 FAULTS: 1
ELAPSED:    0 00:00:01.02 CPU: 0:00:01.06 BUFIO: 0 DIRIO: 0 FAULTS: 0
ELAPSED:    0 00:00:01.36 CPU: 0:00:01.35 BUFIO: 0 DIRIO: 0 FAULTS: 0
ELAPSED:    0 00:00:00.94 CPU: 0:00:00.93 BUFIO: 0 DIRIO: 0 FAULTS: 0

As expected, the START/GET/STOP loop is more expensive, but curiously the one with just the GET is actually faster than the loops with no monitoring. Note that I'm seeing less than 350msec for 100000 repeats of the START/GET/STOP cycle, so I don't know where your numbers are coming from.

        .title timealign
$AFRDEF
reps=100000
        .psect data,rd,wrt,noexe,quad
l: .LONG
   .BYTE
a: .LONG
.align quad
BufSiz=1024
b: .BLKB BufSiz

        .psect code,rd,nowrt,exe
        .entry start,^M<R2>
        MOVL #reps,R2
        CALLS #0,G^LIB$INIT_TIMER
loop:   CALLS #0,bad
        SOBGTR R2,loop
        CALLS #0,G^LIB$SHOW_TIMER

        MOVL #reps,R2
        CALLS #0,G^LIB$INIT_TIMER
loop0:  CALLS #0,bad
        SOBGTR R2,loop0
        CALLS #0,G^LIB$SHOW_TIMER

        MOVL #reps,R2
        CALLS #0,G^LIB$INIT_TIMER
loop1:    PUSHL #BufSiz
          PUSHAL b
          PUSHL #AFR$C_BUFFERED
          CALLS #3,G^SYS$START_ALIGN_FAULT_REPORT
          CALLS #0,bad
          PUSHAL l
          PUSHL  #BufSiz
          PUSHAL b
          CALLS #3,G^SYS$GET_ALIGN_FAULT_DATA
          CALLS #0,G^SYS$STOP_ALIGN_FAULT_REPORT
        SOBGTR R2,loop1
        CALLS #0,G^LIB$SHOW_TIMER

        MOVL #reps,R2
          PUSHL #BufSiz
          PUSHAL b
          PUSHL #AFR$C_BUFFERED
          CALLS #3,G^SYS$START_ALIGN_FAULT_REPORT
        CALLS #0,G^LIB$INIT_TIMER
loop2:    CALLS #0,bad
          PUSHAL l
          PUSHL  #BufSiz
          PUSHAL b
          CALLS #3,G^SYS$GET_ALIGN_FAULT_DATA
        SOBGTR R2,loop2
        CALLS #0,G^LIB$SHOW_TIMER
        CALLS #0,G^SYS$STOP_ALIGN_FAULT_REPORT
        RET
        .ENTRY bad,^M<R3>
        MOVL #1000,R3
badloop: INCL a
        SOBGTR R3,badloop
        RET
        .END Start

Re: What is the performance impact of reporting alignment faults?

Dennis Handly — Wed, 25 Jan 2012 04:48:47 GMT

On HP-UX, this can be 1 to 2 orders of magnitude since reporting requires a user signal handler and the emulation is done in the kernel or even in the CPU.

Also on HP-UX it isn't as bad, since the default is natural alignment and it aborts if not. So most applications are compiled with the right options to prevent the alignment traps in the first place.

Re: What is the performance impact of reporting alignment faults?

Hoff — Wed, 25 Jan 2012 12:52:53 GMT

That's not a default with that abort-penalty for an unaligned reference Dennis, that's a requirement.

The default on all of the OpenVMS compilers is natural alignment, which may or may not be the best model.

With OpenVMS, the programmer has to disable that alignment via directive, via compiler qualifier (switch), or via some code construct that the compiler writers either hadn't handled or hadn't foreseen.

If there was an option to abort on unaligned references (with some decent traceback), I suspect some OpenVMS folks would use it. Particularly given the (large) penalty for unaligned references within the Itanium architecture.

Re: What is the performance impact of reporting alignment faults?

jreagan — Wed, 25 Jan 2012 20:52:23 GMT

Hoff, look closer at the defaults for COBOL and BASIC. Not as aligned as you think. However, the compiler will generate multiple instructions to load the data in pieces to avoid alignment faults.

Back to the question at hand. Yes, there is certainly a Heisenberg effect with asking for fault reporting.

From the SRM Vol 1, User Mask

ac flag : 0: unaligned data memory references may cause an Unaligned Data Reference fault.

1: all unaligned data memory references cause an Unaligned Data Reference fault

Note the "may" in the 0 case. The Itanium architecture allows implementations to handle some unaligned references totally 'on-chip'. I suspect unaligned references inside the same cache line (or perhaps just within a 64-bit quadword) are handled 'on-chip' and VERY fast.

However, when you use the system services that you mention, the 'ac' flag gets set in the user mask. So now EVERY unaligned memory reference causes a fault, traps to OpenVMS, which in turn takes out a spinlock in order to fix up the unaligned reference, and then in turn signals the unaligned access fault to the program. So just by looking, you increase the overhead.

The same holds true for using SET BREAK/UNALIGNED in the debugger or even the SDA FLT support.

Re: What is the performance impact of reporting alignment faults?

abrsvc — Wed, 25 Jan 2012 20:58:47 GMT

It would seem to me that the time cost of the monitoring is irrelevant. I would expect that the developers use the instrumented code to locate the alignment faults in order to "fix" them. Once "fixed" there is no longer a need for the instrumentation and thus no cost on the production system. Lets face it, the goal is to avoid the alignment faults. Once this has been accomplished during the testing phase, the instrumentation (specific to these faults) should be disabled.

Just my approach...

Dan

Re: What is the performance impact of reporting alignment faults?

jreagan — Wed, 25 Jan 2012 22:25:14 GMT

I agree with Dan 100%

Re: What is the performance impact of reporting alignment faults?

Glenn Wolf — Thu, 26 Jan 2012 00:56:20 GMT

John:

Thanks for taking the time to provide the example; it's very similar to the C example from Eight-Cubed, which I had already exercised. It provided the assurance that I needed - yes, I was doing something wrong, and forced me to look closer at my code to find it.

It turns out that when changing my alignment fault reporting from exception based to buffered, I introduced a bug that caused it to continue to use the exception approach anyway, thus the "similarity" of my test results between the exception based and what-I-thought-was-buffered. I should have verified that my code was really doing what I expected before posting.

Now that I've corrected THAT oversight, I see results very similar to yours - a millisecond or two to start the alignment fault collection, and nothing measurable to report it.

It is the exception processing that was taking the extra time. Yet another thing to be wary of (as has been reported in other posts) when moving to Itanium.

To answer some of the other questions, yes we may see some variability in the number of alignment faults by transaction. The transactions process requests that are presented in a "query language" (think SQL, but not too hard), and we will see the query usage change over time. Today our transactions vary in duration from a few milliseconds to hundreds of milliseconds, based on what they are doing, so I am quite willing to pay a millisecond or two to be able to correlate a rise in alignment faults to a particular transaction signature. And, as it is coded today, we can easily turn on or off the alignment fault reporting in real time if it gets in the way.

We will certainly pursue the reported alignment faults and eradicate as many as possible, but I'm pretty certain that we won't get to zero. This data will help us understand how close to zero we can get.

Thank you to all who responded. As always, this forum is of immense help in finding the solutions that I need.

Glenn

Re: What is the performance impact of reporting alignment faults?

Dennis Handly — Thu, 26 Jan 2012 06:26:23 GMT

>That's not a default with that abort-penalty for an unaligned reference Dennis, that's a requirement.

I don't think I understand that you said? From the subject of this topic, I assume the default on OpenVMS is to continue on unaligned accesses and be very slow.

>The default on all of the OpenVMS compilers is natural alignment, which may or may not be the best model.

I don't think there is anything wrong with it. :-)

But this means the topic author isn't taking the default and is cheating and the compiler doesn't provide help.

>the 'ac' flag gets set in the user mask. So now EVERY unaligned memory reference causes a fault, traps to OpenVMS, which in turn takes out a spinlock in order to fix up the unaligned reference

This sounds completely backwards and not how HP-UX does it. There the default is set to 1 and the process aborts on an alignment trap.

If you set it to 0, then the kernel emulates the instruction, if the chip doesn't handle it.

This gives applications a choice and allows the chip to handle it.

Re: What is the performance impact of reporting alignment faults?

H.Becker — Thu, 26 Jan 2012 09:05:10 GMT

>>> By the way, our test environment is an HP rx2620 (1.60GHz/6.0MB) with 16GB of memory, running OpenVMS V8.3. We see similar behavior on our intended production environment: HP BL860c i2 (1.60GHz/5.0MB) 16 cores with 32GB of memory, running OpenVMS V8.4.

>>> It is the exception processing that was taking the extra time. Yet another thing to be wary of (as has been reported in other posts) when moving to Itanium.

You should see much better exception performance on 8.4.

Re: What is the performance impact of reporting alignment faults?

Hoff — Thu, 26 Jan 2012 13:58:02 GMT

:>That's not a default with that abort-penalty for an unaligned reference Dennis, that's a requirement.
:
:I don't think I understand that you said? From the subject of this topic, I assume the default on OpenVMS is to :continue on unaligned accesses and be very slow.

If I interprete "default" as indicating what happens when the compiler doesn't correctly predict data alignment, and when the run-time is "unforgiving" about that, then that is what I'd consider a "requirement" and not a "default".

If I interprete "default" as what happens when the compiler doesn't correctly predict data alignment" and the run-time is 'unforgiving', but you can switch either or both the compiler and (particularly) the run-time 'forgiving'", then that's a different design choice.

The VMS run-time is 'forgiving'.

Whether the compiler misdetected the alignment is certainly debateable; generating the extra instructions unnecessarily is certainly bad, but mispredicting unaligned data is bad, too.

When the compiler misses on the data alignment (often enough), application performance craters.

Equivalent to 10,000 to 15,000 instructions per alignment fault.

Whether that performance hit really matters depends on various factors, including the numbers of faults and what else the code is doing. (There are other things that will slow down application performance and to varying degrees, such as cache misses or most anything that'll trigger a bubble, I/O, etc. And off the top, I don't recall the analogous "instruction" hit for I/O, but that's huge, too.)

VMS doesn't have anything that's really comparable to Vtune, unfortunately.

Handling unaligned references in a compiler with the insertion of "extra" instructions is normal. Having to add extra instructions probably happened with VAX (been a while since I looked at that) and those extra instructions were commonplace on Alpha (both for data (un)alignment, and for dealing with instruction subsets), and it's a necessity on Itanium. Having the compiler and the run-time trigger an application-level unaligned-data error (on VMS) isn't.

Itanium is variously perceived as either a limited-use architecture, and as being sufficiently weird that the typical code-scheduling optimizations implemented for other architectures can't automatically be reused, so there aren't many open-source compilers that target the architecture. gcc is one of the few, and I don't know how actively that's being maintained and extended for Itanium. One of the HP folks stated "the need for improved gcc performance for Itanium is urgent", and that was from around five years ago. The LLVM folks report that that tool-chain has yet lower performance than gcc on Itanium, and there wasn't any activity with that back-end with Itanium when last I checked.

But if past code-generation improvements with compilers other architectures are any guide, I wouldn't expect a huge improvement even with a large investment in the Itanium compilers. Barring degenerate cases - such as massive numbers of alignment faults, as can arise here - the performance-improvement percentages just aren't there with the compilers, as compared with other changes in computer designs.

Re: What is the performance impact of reporting alignment faults?

jreagan — Thu, 26 Jan 2012 18:29:01 GMT

The VAX compilers accessed the memory without regard to alignment. The VAX microcode did all work. On machines like the 11/780, aligned vs unaligned didn't make much difference. On later model VAXen, it did help to a small degree. The biggest help was to longword align code labels.

On Alpha, the first revision of the architecture ONLY had longword and quadword memory accesses. The compilers were required to fetch larger chunks of memory and mask/shift. There is also the _U forms that mask the effective-address to fetch quadwords above or below (helpful in fetching unaligned data). Note that fixing alignment faults on Alpha can help performance too. The PAL code fixup isn't free. The defaults on the Alpha compilers are still EV4-era instruction sets (ie, no word, no byte). You need /ARCH=HOST or something to enable the newer instructions.

Re: What is the performance impact of reporting alignment faults?

Dennis Handly — Thu, 26 Jan 2012 19:48:09 GMT

>If I interpret "default" as indicating what happens when the compiler doesn't correctly predict data alignment, and when the run-time is "unforgiving" about that, then that is what I'd consider a "requirement" and not a "default".

(Unfortunately, I no longer know what the last "that" is referring.)

On the HP-UX compilers, there is no predicting. The compilers only do what they are told and will assume natural alignment unless options, pragmas, attributes or typedefs are used to indicate misaligned data.

>If I interpret "default" as what happens when the compiler doesn't correctly predict data alignment" and the run-time is 'unforgiving', but you can switch either or both the compiler and (particularly) the run-time 'forgiving'", then that's a different design choice.

On HP-UX, it really isn't the "runtime" since the work is done in the kernel. Or on PA-RISC, in a bolted on signal handler library.

And on PA-RISC it is so slow, that reporting of the faults doesn't have to be much slower.

Re: What is the performance impact of reporting alignment faults?

Hoff — Thu, 26 Jan 2012 20:50:42 GMT

So that compiler implementation transfers the Itanium alignment requirements and particularly pushes at least this aspect of the architecture from the compiler's analysis processing onto the programmer, and HP-UX enforces unalignment data errors at run-time.

Certainly makes for simpler compiler designs, and vastly simpler code analysis. And a far larger incentive for code coverage testing by programmers targeting HP-UX, certainly.

If there is no means to switch on code analysis and run-time handling of unaligned data, then the behavior is not really what I'd consider a "default". This behavior is a requirement. ("Default" implies there are alternatives. And based on what I'm reading here, there aren't. Unaligned data causes an HP-UX application to receive (what I presume is probably) a signal. That the application would then fail.

(It's conceivably that sufficiently gonzo programming requirements could lead to the implementation an application-level fix-up and some sort of a recovery, but I'd doubt most HP-UX programmers would bother.)

The VMS compilers do rather more here, and the run-time (which is also implemented in privileged code) fixes up cases where the data isn't alighed.

The OP has the answer, and certainly I've learned rather more about the design and code analysis features in the HP-UX compilers. Thanks.

Re: What is the performance impact of reporting alignment faults?

jreagan — Fri, 27 Jan 2012 05:38:18 GMT

Yeah, the VAX strongly influenced the compiler's ability to tightly pack data. Partially for PDP-11 compatibility and partially since the first 11/780s shipped with 256KB of memory. :) [I suspect that Dave Cutler had some hand in the decision as well. Next time any of us sees Andy G, ask him.]

For instance, VAX Pascal will let you put a 32-bit integer on a BIT boundary!

PACKED RECORD

F1: BOOLEAN; (* One bit big, starting at offset 0 *)

F2: INTEGER; (* 32bits bit, starting at offset 1 *)

END;

Even on VAX, you have to extract from memory into a register, operate on the register, and insert it back.

When we ported to Alpha, we changed all the defaults so that F2 field would now aligned on at least a byte boundary for a PACKED record and 32-bit boundary for an "unpacked" record. [We still have qualifier to get VAX packing rules in case you have old VAX binary data on disk.]

Re: What is the performance impact of reporting alignment faults?

Dennis Handly — Fri, 27 Jan 2012 21:06:57 GMT

>and HP-UX enforces unaligned data errors at run-time.

(Actually the hardware does this.)

>And a far larger incentive for code coverage testing by programmers targeting HP-UX, certainly.

It depends. To get misaligned data, you usually have to tell the compiler you are cheating for struct layout. (Of course you can add non-portable casts that the compiler will warn about.)

>If there is no means to switch on code analysis and run-time handling of unaligned data

The code analysis warnings are always on. You can optionally reset the flag to get the hardware and kernel to handle them.

>then the behavior is not really what I'd consider a "default". This behavior is a requirement. ("Default" implies there are alternatives.

(Ah, ok. So you don't believe in "you can have any color you want, as long as it is black"?

But in this case, you can change the default by looking at the fine print. :-)

>Unaligned data causes an HP-UX application to receive a signal. That the application would then fail.

This is the default but it can be changed. Either by enabling belts and suspenders code generation and/or passing the buck to the hardware and kernel.

>could lead to the implementation an application-level fix-up and some sort of a recovery

You could say this was what was done on PA-RISC.

>The VMS compilers do rather more here

I wouldn't necessarily say it does more, it's that its default is different. ;-)

You have to order the HP-UX compilers to use different alignments and packing and then there are options to handle the non-typesafe coding cases.

Re: What is the performance impact of reporting alignment faults?

Hoff — Fri, 27 Jan 2012 13:50:08 GMT

OK; now I'm completely confused.

I know what Itanium is (not) doing, but whatever HP-UX is doing here seems, well, just as differently-odd as Itanium itself.

Off to read some HP-UX compiler documentation.

Re: What is the performance impact of reporting alignment faults?

Hoff — Fri, 27 Jan 2012 14:07:59 GMT

John: based on a quote from DEC for a VAX-11/780 to VAX-11/785 upgrade, eight megabytes of VAX-11/780 memory cost US$36,000 back about 1985. (That was MS780-E series) We ended up doing that upgrade with used memory (MS780-C series), and that was "only" US$10,000 for that part of the upgrade. At the time, an 8 MB VAX-11/785 was a very large VAX box.

I'm well aware of the data packing and padding on VAX, Alpha and Itanium running OpenVMS, and the related compiler pragmas and such. (That's for C, other languages vary.) And its history, for that matter.

It's the "completely different world" of HP-UX and its compilers that's the oddity (to me) here.

But then HP-UX and its compilers on Itanium is presently only a curiosity here (for me), for reasons not germane to this discussion. And as mentioned in another reply of mine, I'll be scrounging for technical papers on the compiler and the operating system platform, if anyone has suggestions.