Sockets in /var/spool/sockets/pwgr

[David Gourley_1]

We have a large number of supposedly identical servers both in the field and in a lab environment. (at least they are all running the same variant of HPUX 11.0 with similar patches etc.) Whilst investigating a problem on a field server, we encountered an interesting situation which we've not been able to find documentation or information on.

On most of our servers, there appear to be around 300 sockets in /var/spool/sockets/pwgr. Using tusc and lsof, I can relate these to the running processes on the server (with more or less a 1:1 mapping).

However, on two machines (one in the lab, and one production system) there are literally thousands of sockets in this direction. Many of these were preserved over a server reboot (my guess is that they may be cleaned out using 'rm' which couldn't cope with the number of parameters it was supplied due to command line limitations). We'd like to understand what's causing these to be created on these two servers (but not on around 20 other servers that I've checked), all ostensibly running the same software (they certainly have similar numbers of processes and threads).

Our first step was to disable pwgrd; however, the sockets continued to be created (possibly at a slightly slower rate than before).

The unaccounted sockets appear to be owned by root (although as a member of a variety of groups including sys and root).

Anyone got any idea:

1)What causes these sockets to be created?
2)Whether there are any limits on how many will end up being created (or could they end up causing us to run out of inodes?)
3)What(if anything) purges them on reboot

Thanks in advance

Dave

[Ross Zubritski]

Dave,

pwgr is the password caching daemon, it does not need to be enabled unless you are processes thousands of logins, i.e. webserver.

Kind Regards,

RZ

[Tim Sanko]

Dave,

Ah, the PWGR problem. If your system is busy enough you can find these.

The password/group hashing it does can greatly improve system performance.

1. Each time an app opens and starts to check for the appropriate permissions,
pwgrd accepts the socket, and passes the perms back to the app as though it had checked through a file list and compared them. When the app is active, the socket should remain to service the permissions/ownership resolution. It can reduce poorly written codes CPU/DISK metrics by about 30% (We see 26% on our production server.)

Pwgrd can hang. do a stop to stop and a /sbin/init.d/pwgr start to start it. run pwgr_stat to see the status.

We have had such problems with pwgr that we got a program that we put in bin
named get_pwgr_stats.

It will allow you to see if your pwgr daemon is being uncommunicative. you can output this to a file, and
grep for "Unable to communicate" and then issue the /sbin/init.d/pwgr start
command.

We check this on our production box every 10 minutes.

Then you have the sockets cleaned up and the performance boost of pwgrd.

2. Is there a limit to how many get created. yes 32768, I was told.

3. pwgrd must clean them up.

Tim

[Tim Sanko]

I pesonnally would prefer to run a machine more efficiently so that financial resources can be used for something more important than hardware (my raise.)

The number of users, logins are important. We have over 3000 users and always except in a 10:00 - 10:30 P.M. Window at least 200 users. Impromptu sessions, Cognos, Oracle, Informix, Sybase, and other applications get a big performance boost from pwgrd.

Tim

[David Gourley_1]

Thanks for the answers so far.

A couple of things I'm still slightly confused about:

1) I understand what pwgrd does. As the password file on our systems is small (typically in the range of 20-30 users) I believe we should not see a major performance hit from disabling it (indeed we've done some testing and don't notice the difference).

2) With pwgrd disabled, why do we still get these sockets created? Will anything tidy up?

[David Gourley_1]

Better late than never: we eventually discovered that sockets continued to be created when entries were written to syslog.

To tidy up completely we had to remove /var/spool/pwgr/status; it appears that some parts of the system use presence of this file to indicate whether or not pwgrd is used (so if you subsequently disable it you also have to remove the file).