Kernel Tuning - Anybody got some insights into sysv_hash_locks

[Rita C Workman]

Here's the deal.....
Running FileNet on my RP7410...we see quick periods where the system chokes. If it chokes long enough then FileNet RPC commands will time out and then a process gets stuck in it's queue. Now FileNet is a FIFO system...so when something gets stuck in it's queue everything behind it just waits....
Or sort of waits...continuing requests get resources allocated and they in turn wait behind that stuck process. All of this can fill up pretty quickly ...and in a matter of a few short minutes the whole system is hosed.
There really is no super-user that can get to the top of the 'tree/priority' like root, so if the resources within FileNet are taken up, you can't get in to clear the stuck process.
....arghghghgh....

Anyway, FileNet has suggested increasing parms. But so far every suggestion on more semaphores has met and exceeded their "Best Practices".....the only parm they don't mention is sysv_hash_locks. And FileNet relies heavily on SysV.....so does anybody have any insights on what this might need to be set at...
Currently it's at the default of 128...and I'm thinking it should be set at the same as semmni.

Can somebody share some information or guidance on this elusive little parm...I have read the general parm info under hp.docs.

Open to info on this....
Thanks,
Rita

[A. Clay Stephenson · Accepted solution]

Rita:

None of my System V documentation (and it's AT&T and thus "real" System V) mentions sysv_hash_locks or hash_locks. I suppose I would double the value and see if it changes anything but this truly sounds like a shot in the dark. I would first use Glance -> Systems Tables Report to see if any limits are being approached under heavy load.

It will also be a good idea to use tusc to determine what system calls are being hit and what the system is actually waiting on.

I assume that the FileNet boys and girls have been less than stellar Mr. GoodWrench'es in helping to resolve this.

[Rita C Workman]

Well Hello Dear Sir !

Stellar...hmmm...nope.

We're gonna take a look at using tusc on this, right now we're just trying to do some network tracing, cause their blaming on the network performs. < Get this...we switched to the GigE backbone, and they still complain about network. Talk about "that dog don't hunt">
But when they started complaining I needed to up semaphores I been a watchin and lo and behold they NEVER hit was I have allocated. But got to reading and sysv_hash_locks was in the list of semaphore relations and based on what it said, I got to wondering about it's setting.

A shot in the dark....yeah ! But with FileNet your always hitting moving targets in the dark...and my hearing ain't what it used to be. But I smells something....

Hope you make it to Chicago (me to 4 that matter !!)
Your Friend,
Rita

[Paula J Frazer-Campbell]

Rita

Spinlock Pool Parameters
This was "borrowed" from the web page on Configurable Kernel Parameters.

The following parameters, all related to spinlock pools for multi-processor computers, are used similarly and are documented together here. Each parameter allocates the specified number of spinlocks for the corresponding system resource:

bufcache_hash_locks

Buffer-cache spinlock pool
chanq_hash_locks

Channel queue spinlock pool
ftable_hash_locks

File-table spinlock pool
io_ports_hash_locks

I/O-port spinlock pool
pfdat_hash_locks

Pfdat spinlock pool
region_hash_locks

Process-region spinlock pool
sysv_hash_locks

System V Inter-process-communication spinlock pool
vnode_cd_hash_locks

Vnode clean/dirty spinlock pool
vnode_hash_locks

Vnode spinlock pool
These parameters are for use by advanced users only who have a thorough understanding of how spinlocks are used by multiple processors and how the number of spinlocks needed are related to system size and complexity. Do not change these from their default value unless you understand the consequences of any changes. In general, these values should not be altered without the advice of HP support engineers who are thoroughly familiar with their use.

Setting these parameters to inappropriate values can result in severe performance problems in multi-processor systems.

Following is a list of acceptable values. All of these parameters have the same minimum and maximum values. Only the defaults are different as indicated:

Minimum

64
Maximum

4096
Default

64 (ftable_hash_locks, io_ports_hash_locks)
Default

128 (bufcache_hash_locks, pfdat_hash_locks, region_hash_locks, sysv_hash_locks, vnode_hash_locks, vnode_cd_hash_locks)
Default

256 (chanq_hash_locks)
Specify a value that is an integer exponent of 2. If you specify any other value, SAM or the kernel itself changes the parameter value to the next larger integer exponent of two (for example, specifying 100 results in the spinlock-pool value of 128.

Description

In simple terms, spinlocks are a mechanism used in multiple-processor systems to control the interaction of processors that must be held off while waiting for another processor to finish a task so the results can be passed to the waiting processor. Spinlocks control access to file-system vnodes, I/O ports, buffer cache, and various other resources.

Earlier HP-UX versions allocated a fixed number of spinlocks for all resources, but beginning at HP-UX 11.0, spinlocks can be allocated for each resource type to accommodate very large and complex systems.

In general, if the system is encountering lock contention problems that are associated with one of these hashed pools, first identify the resource spinlock pool that is associated with the contention, then increase the spinlock-pool parameter for that resource. Spinlock pools are always an integer power of two. If you specify a value that is not, the kernel always allocates a value that is the next larger integer exponent of two.

As stated above, these parameters are for use by experienced, knowledgeable system administrators only. They should not be altered unless you are quite certain that what you are doing is the correct thing to do.



From :-
http://www.unixadm.net/hp/hp_tuning.html


HTH

Paula

[Rita C Workman]

Hey Paula,

Yep...read that. Mine are sitting at the current default values. Untouched by these foolish mortal fingers.

As I mentioned the software involved is FileNet, a Records Management Imaging System.
It uses SysV...and hence relies heavily on IPC/RPC communication. I've tuned the kernel and had been getting reasonably good performance. But notice that when we have a problem...it seems to be due to a timeout in some kind of RPC communication. Once the process timeouts the RPC communication gets hung, and hence FileNet begins to queue up to the max and is totally stuck. Reboot is the only thing that fixes it.
That's when I started reading, and thought since FileNet is SysV...possibly these hash values need to be 'expertly tuned', since they are all related in the semaphore realm.

But so far....nothing concrete from the vendor. Was hoping someone else out there in ITRC Land might have hit upon this. But alas...not yet !

If I get anything further on this I'll post it for posterity.

Rgrds,
Rita

[Jim Butler]

Check your patch level on pthreads. -
There is a problem that I ran into with clearcase when I recently updated my patches, and specifically, it was a conflict with phkl_23902 and the latest bdf patch.

I removed the latest bdf patch, and problem solved. (my disclaimer- don't know if filenet is using pri queing via pthreads)

good luck

[ASTSZ-YICT]

Hi Rita ,
How about the FileNet issue ?I met same trouble in my rp7400,I want to know how you fix your FileNet issue ,Thanks


ED

[Rita C Workman]

Greetings A-Y

FileNet...is a challenge no matter what you do. For the FIFO thing..there is no way around it. There are tricks if you catch it fast enough, but you have to watch it every second to do that (and who has that kind of time).

We put in some newer patches, moved data around on disk. Our parms were already far more than FileNet every asked for, and more than monitoring showed being used.
Found that some of the patches addressed SysV issues (RPC timeouts) and I believe this helped. But probably the biggest help was taking a look at the way FileNet loaded up everything. It was a case of who was controlling things...FileNet, our FileNet Admin, or somebody who would look at trying to fix the performance.. Remember it's raw....and when I went down checking every lvol, I found they just drop it at the next space available. So there 'oracle' binaries, & logs were on the same disk...[ouch]..and on one disk I found binaries, logs and yes..data...[double ouch].
So I mapped the 'little darling' for every lvol out there and gave it some more disk, scheduled downtime, and pvmoved to my hearts content.

Timeouts reduced...performance so improved that folks are actually 'pleased'.

FileNet Tip...when all else fails, drop back to the basics......

Rgrds,
Rita

[Fred Ruffet]

About RPC timeouts, have a look at this lines in /etc/rc.config.d/nfsconf :
NUM_NFSD=16
NUM_NFSIOD=16
I know this doesn't seem to be linked, but it worked for me with PFS problems wich is also based on RPC. I had the same messages and when I increased this parameters, and it became better.

Regards,

Fred

[Rita C Workman]

Ladies & Gents...

Please Consider This Post Closed...

Thanks,
Rita