"IpcFindEmptyIpcHandle: IpcHandleTable is full" message in DP log

[Ana M. García Olivencia]

Hi all.

One of our DP backups is failing and the information in the media agent (OpenVMS v7.3-2) debug.log is, summarizing, the following repeated messages:

.....
11/06/06 11:56:03 BMA-NET.5680.0 ["lib/ipc/ipc.c/main/dp55/blr_ovmsdp55_beta/blr_ovmsdp55_oracle/4":4647] A.05.50 OpenVMS A.05.50 (74) [IpcAllocShmem] shmget(1783634710, 2080768, IPC_CREAT|IPC_EXCL|IPC_SHMEM_PERM) failed (err=65535)
....
IpcFindEmptyIpcHandle: IpcHandleTable is full (64)
.....

This messages are Unix-like, relationed to shared memory and I was trying to relation it with OpenVMS.

What I first thought was that this had something to do with what could be similar to shared memory in OpenVMS, global sections and global pages, but starting the backup and monitoring the free gblsects and global pages, I could see they were not the problem.

I have searched through Internet and the few topics I found about ipcallocshmem has to do with the modification of shm* parameters in UNIX environments.

I have attached an extract of the media agent 'debug.log' file corresponding to the backup session. The DP version is A5.50.

Although this topic should be posted at Data Protector forum, I'd like to ask OpenVMS people if those messages have something to do with some OpenVMS parameters that should be adjusted because the rest of backups works well and it seems that the only difference is that the failing backup has more save sessions than the rest.

Thank you very much in advance.

Regards.

Ana

[Volker Halle]

Ana,

I have not been sucessful in trying to find OpenVMS related documentation of the shmget call, which seems to be at the bottom of this problem.

They key questions would be:

- what does an errno of 65535 mean on a shmget call ?
- how is shmget implemented on OpenVMS ($CRMPSC maybe) ?
- do the shmget errors repeat 64 times and then finish with the IpcHandleTable is full (64) error ?

If you want to experiment with parameters without yet understanding the underlying problem, PGFLQUOTA comes to mind and maybe PROCSECTCNT.

But a call to HP may be more helpful.

Volker.

[Ian Miller.]

errno of 65535 means a VMS specific condition code has been returned. It it returned in another variable which unfortunaly no reported.

[Ana M. García Olivencia]

Volker, Ian

From your information about the $CRMPSC system service I have seen that another parameter involved is GBLPAGFIL, that was lower than the rest of the systems (although I executed an autogen with feedback and this parameter was not changed).

Anyway, I have raised its value and rebooted. Moreover, as our network has firewalls, we were suspecting that perhaps was a problem of reconnections and has made some modifications in the network configuration.

With both changes, we have started the failing backup and, by the moment, is ok. If it finishes ok, I'll modify the GBLPAGFIL again to the old value to see if the problem arises, to know what the solution to the problem is.

I'll let you know the results.

Thank you very much.

Regards.

Ana

[labadie_1 · Accepted solution]

If you want to monitor the remaining gblpagfil, use the following program: it will look every 60 seconds (adjust it for your use) and define a logical name min_gblpagfil in the system table.

#include
#include
#include
#include
#include

int Mingb = 0;
$DESCRIPTOR(TblNameD, "LNM$SYSTEM_TABLE");
$DESCRIPTOR(LogNameD, "MIN_GBLPAGFIL");
char Value[32];
$DESCRIPTOR(LogValD, Value);

globalref int mmg$gl_gblpagfil;

main() {
int s;

Mingb = mmg$gl_gblpagfil + 1000;
for (;;) {
if (Mingb > mmg$gl_gblpagfil) {
Mingb = mmg$gl_gblpagfil;
sprintf(Value, "%d", Mingb);
LogValD.dsc$w_length = strlen(Value);
s = lib$set_logical(&LogNameD, &LogValD, &TblNameD);
if (! $VMS_STATUS_SUCCESS(s)) exit(s);
}
sleep(60);
}

[Volker Halle]

Ana,

you can also check the remaining global pagefile quota with SDA:

$ ANAL/SYS
SDA> CLUE MEM/STAT
...
Global Pagefile Quota 16821 GBLPAGFIL (SYSGEN) Limit 17440

SDA> exa sgn$gl_gblpagfil ! sysgen parameter value
SGN$GL_GBLPAGFIL: 00000000.00004420 " D......"
SDA> exa mmg$gl_gblpagfil ! remaining free global pagefile pages
MMG$GL_GBLPAGFIL: 00000000.000041B5 "µA......"

Volker.

[Volker Halle]

Ana,


although I executed an autogen with feedback and this parameter was not changed


GBLPAGFIL is not an AUTOGEN FEEDBACK parameter.

Volker.

[Ana M. García Olivencia]

Thanks you both for the information about how to get the free global pages remaining.

I have entered SDA (now the backup is running) and the information given is:

Global Pagefile Quota 54723 GBLPAGFIL (SYSGEN) Limit 196608

This means that is using 141631 (the difference between both figures) that is greater than the old GBLPAGFIL value (98304).

So, it seems that the lack of GBLPAGFIL could be the reason of the failing backup, but I'll wait for the backup to finish to check the values when there is no backup running.

I'll periodically execute the program to know the GBLPAGFIL needs.

Tomorrow I'll let you know what happens.

Thank you very much again.

Regards.

Ana

[Ana M. García Olivencia]

Hi all.

The test we made yesterday was ok but when we have repeated the same test changing the paralellism to 8 (a DP concept -number of sessions by device and we are using four devices-) with client systems in both behind and in front of the firewall, the backup failed, very early. With a paralellism of 4 it also fails.

Well, to determine if it's a firewall problem, we are now doing a backup only with clients not going through the firewall and, by the moment, is ok. Of course, there is less number of clients and, as a result, a less number of needed resources. When this backup finishes, we'll make another one only with clients behind the firewall. With these tests we want to discard network problems.

We raised the GBLPAGFIL quota to 1000000

Depending on our tests results, we'll focus on a resources problem, a firewall problem or both.

In the meantime, I have been checking the accounting information of the OMNIADMIN procesess of yesterday tests (the one with paralellism of 8) and found a figure that I don't understand. One of the OMNIADMIN processes has a 'peak pagefile quota' of 4194448 when its uaf quota is PGFLQUOTA=1000000.

I thought that the 'peak page file quota' should be <= PGFLQUOTA. Is this ok?. If not, what's the meaning of that figure?.

Well, I'll continue with the research and I'll let you know the results of our tests.

Thank you very much.

Regards.

Ana

[labadie_1]

Ana

You should get pquota.exe from
http://h71000.www7.hp.com/freeware/freeware40/pquota/
copy it to a disk:[dir]

then do

$ pquota :== $ disk:[dir]pquota
and then
$ pquota 204002B3
to look at all the quotas of the process id 204002B3

Now you can have a coffee, you will even have a file pquota*.out summing up all that.

A quota heavily used will be blinking.

Look at the last right column to see which quota is used at more than, say 80 %

[Volker Halle]

Ana,

accounting shows 'peak pagefile' usage, the actual pagefile quota of the process may be higher than it's UAF PGFLQUOTA. Check the running process with SHO PROC/ID=xxx/QUOTA (it will show you the currently remaining pagefile quota).

$ ANAL/SYS
SDA> READ SYSDEF
SDA> SET PROC omni...
SDA> FORMAT JIB
...
FFFFFFFF.8338F940 JIB$L_PGFLQUOTA 00007A12
FFFFFFFF.8338F944 JIB$L_PGFLCNT 0000774F
...

will show the initial PGFLQUOTA and the remaining quota.

Volker.

[Ana M. García Olivencia]

The last test we did yesterday was ok. So, it seems, at a first glance, that it's not a firewall or network problem, at least, with the number of savesets backuped. Or perhaps the network problem begins when the number of resources to save raise.

We'll gradually increment the number of save sets to save, keeping a paralellism of 8, and will check, with all the information given by you, the resources consumption "to chase" the quota or parameter failing.

I'll let you know.

Thank you very much.

Regards.

Ana

[Ana M. García Olivencia]

Hi all.

With the information you gave me we have discovered that we should raise some parameters (MIN_GBLPAGFIL, MIN_PQL_DPGFLQUOTA y MIN_PQL_DWEXTENT) and, of course, increment the pagefile size until 5GB.

However, under certain circunstances, the backups fails, although I think that it has something to do with a misconfiguration of Data Protector parameters, without discarding more changes in sysgen parameters.

I leave the note open for a few days because we are going to do more tests and, if we find the solution, I'll let you know.

Thank you very much.

Regards.

Ana

[Ana M. García Olivencia]

Hi all.

It seems that it is a Data Protector problem and there is a patch for it (patch DPWIN_00234). The problem reports:

If 60-70 Volume Backup Disk Agents (VBDA) are started, the Backup Media Agent (BMA-NET) fails with the following error:

"IPC Cannot Create Shared Memory Segment"

Although I haven't still tested it, it seems the same problem we had and the fact that, depending on the number of clients backuped, sometimes worked and other didn't.

Anyway, there was also a parameter and quota problem that, thanks to your help, seems solved.

The fact that I haven't been able to find that information is because I work in a Linux environment and the HP page with Data Protector patches doesn't work with my browsers (it's a problem reported to HP more than a year ago and with no answer from them by the moment), and I haven't been able to open a HP call due to a burocratic problem with our contract. All the searches about that error message didn't give any reference to that patch page.

Therefore, I think that the post can be closed because I don't exactly know when we are going to install the patch and do the tests.

Thank you very much for all your help.

Regards.

Ana