PA-RISC space registers - hardware vs. software access

[Neil Armstrong]

Hi,

I've heard from a few people in the past that there were differences between PA-RISC 1.0 and 2.0 (can't remember if 1.1 was different) with how memory/space registers are accessed.

From what I understand, 1.0 had something like 6 hardware-accessed registers (for code, stack, shared mem, etc)...and any further registers (eg another shared memory segment) were software-accessed, thereby taking a performance hit. Then with 2.0, the number of hardware registers increased, so additional memory segments could be used with no real performance loss.

Where can I find information on this? I've looked through the PA-RISC 2.0 64bit architecture documents (which make mention of 6 registers) but this doesn't explain it too well.

I really just want to check whether we are running some of our database shared memory segments in a software-accessed mode, as we are experiencing strange performance issues since we upgraded to a 64bit database engine.

(sorry, some of my terminology might be wrong!)

thanks!
neil.

[James Murtagh]

Hi Neil,

I've noticed a few other questions you have posted have been inclined towards what we refer to as "The Internals" of HPUX. If you need to know more about these topics, the best bet would be to enrol in the official HP course "Inside the HPUX Kernel - HPUX Internals". As a general guide, the course covers :

System Architecture
Process Management
Memory Management
Filesystems
LVM
System Initialisation
IO Subsystem
IPC Subsysytem
Kernel Services

Its the lowest level (and probably most intense) course HP offers in HPUX.

As to your main question, I'll answer as far as I understand it.

All recent PA-RISC chips have 8 space registers, organised as follows :

SR0 : Link Code Space id
SR1 : General Use
SR2 : General Use
SR3 : General Use
SR4 : Space id Quadrant 1
SR5 : Space id Quadrant 2
SR6 : Space id Quadrant 3
SR7 : Space id Quadrant 4

As 32 & 64bit implementations both use 4 quadrants, the number of registers are the same. In reality the space registers don't determine what the kernel is accessing, the offset does. The space and offset are joined to form the virtual address.

How it ties in - 64 bit example :

The space register is 64 bits, of which only 22 bits is used. (so 2^22 = 4194304 - compare this with nprocs maximum in /usr/conf/master.d/core-hpux)

The offset is 64 bits, of which only 44 bits are used. Now 2^44 = 16TB. Divide that by your quadrants so that is why there is a restriction in the quadrant size. So if an offset address = 5GB then that would be in Quadrant 2 so for a 64 bit process it would be accessing text.

Of course PA-RISC 2.0 expands other registers, but not the space registers. Is this what you were asking?

Cheers,

James.

[Bruno Vidal]

Hi,

I think there nothing to add from previous answer. But you can find a lots of docs on the web page of the puffin group:
http://www.parisc-linux.org/documentation/index.html

It is the linux port to pa-risc systems.

Bye.

[Neil Armstrong]

Hi James,

That info is pretty much what i've gleaned so far...i'm trying to determine what portions of a process are loaded into what register.

From a reply from an Informix group:


It is a problem that any software will encounter if running on HP and is allocating shared memory segments.
The problem is actually caused by the way that HP manages shared memory segments and utilizes the space registers. Basically when more than four memory segments are constantly being referenced, then there will beperformance degredation. This is because HP actually only bases four segmentsat any given time.
If there are actually five, then when an access to amemory address is made to a segment that is not currently based, then a check is made to see if it *could* be resolved by one of the un-based segments. If it could, then
that segment's basing information is loaded into one of the space registers and the access is retried. Needless to say, this could causea bit of overhead.
OK - lets look at where the space registers are used. One is always basing the text
portion of the program. A Second is basing the shared object library. That leaves two for shared memory.


Does that mean then that registers S1-S3 are used for the code, data and stack/heap portions of a process, and S4-S7 used for the remainder? I'm afraid that we might be trying to utilise an additional memory segment that's causing the above performance hit (our Informix implementation runs 3 shm segments - resident, message and virtual [user space], as well as the normal data, stack, uarea and text portions).
Is there a way to determine what registers are used by what for a process, and whether we are pushing the boundaries?

cheers,
neil.

[Neil Armstrong]

Hi James (again!),

Could you clarify something that you stated earlier:


The offset is 64 bits, of which only 44 bits are used. Now 2^44 = 16TB. Divide that by your quadrants so that is why there is a restriction in the quadrant size. So if an offset address = 5GB then that would be in Quadrant 2 so for a 64 bit process it would be accessing text.


If 64bit gives you 16TB available (4 Quadrants of 4TB each), wouldn't an offset of 5GB for a process mean that it's still in Quadrant 1? Or did you mean 5TB (which makes sense)?
Or have I really got it ass-about-face? ;o)

cheers,
neil.

[rick jones]

i've never heard of anyone talking about space register thrashing. the odd ldsid instruction here and there and that is not a big deal

i _have_ heard of people discussing and I have experienced thrashing on protection ids, for which there are a fixed number of registers, and the number of registers there did (iirc) change between PA1.0/1.1 and 2.0.

perhaps there is some confusion out there as to what registers are thrashing with large numbers of memory segments?

I don't think there should be much protection id thrashing with truly "shared" memory since those are generally given the same "public" protection id.

if protection id thrashing is happening, it can be seen in a kernel profile - the routine that gets executed when a protection id trap happens would be visible. the prospect profile tool at the developer's portal can be used to take a kernel profile. to get to the dspp i typically start at devresource.hp.com because I learned that straightforward URL first.

[Mike Stroyan · Accepted solution]

Rick is on the right track. The potential for thrashing occurs with the protection IDs cached in control registers CR8,CR9,CR12 and CR13. PA-RISC 2.0 has two protection IDs in each of those registers, for a total of 8. The registers are listed in
http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,959!19!241,00.html
The use of the registers is discussed at
http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,959!28!244,00.html

The system does commit a couple of those protection ID registers to IDs for non-shared memory regions, so PA1.1 had two "registers" for shared memory protection IDS. PA2.0 has six "registers" for shared memory protection IDS.
(HP-UX 10.20 actually required a kernel patch to use all of the protection ID registers. Since you are using a 64 bit database you must be using at least 11.00, which did not need any patch for that.)
The public protection ID which Rick referred to can only be used for shared memory segments that have the same permissions for all users that tan map that segment. It is used for the text areas of shared libraries. It can't be used for shmat or normal mmap(MAP_SHARED) regions. The attached shmthrash example shows the slow down when cycling rapidly through accesses to too many shared memory segments.
The prospect profiler won't show how much time you spend handling protection ID faults. The trap handler is at too high of a priority and is never caught by prospect's sample interrupt.
The only good way to know that you are not protection ID thrasing is to have six or less shared memory segments so you know any access pattern will fit in the available registers. You can see the shared memory segments in the putput of the attached procvm
program. The segments to watch out for would have pst_flags and pst_type that show the segment is shared and is not a shared library-
pst_flags: 0x4 PS_SHARED
pst_type: 0x8 PS_MMF
or
pst_flags: 0x4 PS_SHARED
pst_type: 0x5 PS_SHARED_MEMORY

[Neil Armstrong]

Hi all,

Proctection IDs - yes! I thought I might've been on the wrong track with Space IDs...

Are the 6 IDs available per process, or system-wide?

The procvm program is interesting - although it reports 101 PS_SHARED_MEMORY segments (doesn't seem to line up with an ipcs -bom though).

neil.

[James Murtagh]

Hi Neil,

Yes, it was a very good spot by Rick and Mike.

The six registers are available per processor, so it depends on how many CPUs you have. Again, the CPU can only process instructions for one process at a time, so each process can use the six registers.

To expand on Mikes comments regarding the trap handler, running a timex on the shmthrash program shows some interesting results :

# timex ./shmthrash
0.457450 seconds for 2 segments
0.455498 seconds for 3 segments
0.455555 seconds for 4 segments
0.572718 seconds for 5 segments
0.633060 seconds for 6 segments
35.297755 seconds for 7 segments

real 37.89
user 37.27
sys 0.03

So only 0.03 seconds system time is reported. This is because the kernel has two routines to handle protection id misses. The "normal" routine checks the processes pregion list. As it does this it builds up a cached list that can be used to handle subsequent misses. The fast version uses this list, and it is here that the normal accounting checks are not done as it is at too high a priority.

The kernel has a variable that controls the building of this list - fast_resolvepid_on. Check this using adb :

# echo fast_resolvepid_on/D|adb -k /stand/vmunix /dev/kmem
fast_resolvepid_on:
fast_resolvepid_on: 1

As for the procvm program, you have to count all the NATTCH entries.

# ./procvm > /tmp/procvm.out
# cat /tmp/proc.out |grep -c PS_SHARED_MEMORY
19
# ipcs -mob|awk '{print $7}'|grep -v "0"
NATTCH

1
1
3
6
6
2

So that is 19.

Cheers,

James.

[harry d brown jr]

I think we need a PHD forum for this topic.

When I'm able to get back to work, I plan on printing this and studying it.

live free or die
harry