Netbackup 6.0 mmap errors and core dumps

[Gregory Marchese_1]

We have recently upgraded from netbackup 5 to 6 mp4. We have an issue with one of our netbackup media servers. The server in question runs hpux 11.00 it has 12gb of memory and 8 750mhz cpu's. When we hit the peak of our work day on this server with over 400 users we start to see issues with ssh connections failing...
--------syslog-------------
Feb 22 11:46:28 achilles sshd[18427]: fatal: mmap(1310720): Not enough space
--------syslog-------------
and as soon as we shutdown the netbackup daemons we are fine. If we try to start netbackup again we get core dumps for every process it tries to start and similar mmap errors.

# /usr/openv/netbackup/bin/nbdbms_start_stop[18]: 26472 Abort(coredump)

NetBackup Database Server started.

/usr/lib/dld.sl: Call to mmap() failed - TEXT /usr/openv/lib/libvxicudata.sl

/usr/lib/dld.sl: Not enough space

/usr/openv/netbackup/bin/goodies/netbackup[124]: 26550 Abort(coredump)

NetBackup Notification Service started.

/usr/lib/dld.sl: Call to mmap() failed - TEXT /usr/openv/lib/libvxTAO.sl.3

/usr/lib/dld.sl: Not enough space

/usr/openv/netbackup/bin/goodies/netbackup[132]: 26639 Abort(coredump)

NetBackup Enterprise Media Manager started.

/usr/lib/dld.sl: Call to mmap() failed - TEXT /usr/openv/lib/libvxicudata.sl

/usr/lib/dld.sl: Not enough space

/usr/openv/netbackup/bin/goodies/netbackup[140]: 26734 Abort(coredump)

NetBackup Resource Broker started.

/usr/openv/netbackup/bin/goodies/netbackup[163]: 26865 Abort(coredump)

Media Manager daemons started.

/usr/openv/netbackup/bin/initbprd[13]: 26933 Abort(coredump)

NetBackup request daemon started.

/usr/lib/dld.sl: Call to mmap() failed - TEXT /usr/openv/lib/libvxicudataST.sl

/usr/lib/dld.sl: Not enough space

/usr/openv/netbackup/bin/goodies/netbackup[183]: 27078 Abort(coredump)

NetBackup compatibility daemon started.

/usr/lib/dld.sl: Call to mmap() failed - TEXT /usr/openv/lib/libvxicudata.sl

/usr/lib/dld.sl: Not enough space

/usr/openv/netbackup/bin/goodies/netbackup[191]: 27140 Abort(coredump)

NetBackup Job Manager started.

/usr/lib/dld.sl: Call to mmap() failed - TEXT /usr/openv/lib/libvxTAO.sl.3

/usr/lib/dld.sl: Not enough space

/usr/openv/netbackup/bin/goodies/netbackup[199]: 27327 Abort(coredump)

NetBackup Policy Execution Manager started.

/usr/lib/dld.sl: Call to mmap() failed - TEXT /usr/openv/lib/libvxicudata.sl

/usr/lib/dld.sl: Not enough space

/usr/openv/netbackup/bin/goodies/netbackup[207]: 27603 Abort(coredump)

NetBackup Service Layer started.

/usr/lib/dld.sl: Call to mmap() failed - TEXT /usr/openv/lib/libvxTAO.sl.3

/usr/lib/dld.sl: Not enough space

/usr/openv/netbackup/bin/goodies/netbackup[234]: 27753 Abort(coredump)

NetBackup Service Monitor started.

[Dennis Handly]

>/usr/lib/dld.sl: Call to mmap() failed - TEXT /usr/openv/lib/libvxicudata.sl
/usr/lib/dld.sl: Not enough space

This indicates you have badly fragmented or leaked shared memory area. There is only 2 Gb.

What does this show?
$ ipcs -ma

What is the version of your dld/linker patch?

[Gregory Marchese_1]

Thanks for the reply.

$ ipcs -m
IPC status from /dev/kmem as of Fri Feb 23 13:18:28 2007
T ID KEY MODE OWNER GROUP
Shared Memory:
m 0 0x411c0627 --rw-rw-rw- root root
m 1 0x4e0c0002 --rw-rw-rw- root root
m 2 0x41200649 --rw-rw-rw- root root
m 31749 0x0c6629c9 --rw-r----- root root
m 6 0x06347849 --rw-rw-rw- root root
m 3079 0xffffffff --rw-r--rw- root root
m 111627 0x0000cace --rw-rw-rw- root root
m 142349 0x00000000 --rw-rw-r-- xxxx xxx
m 2062 0x00000000 --rw-rw-r-- xxxx xxx
m 2063 0x00000000 --rw-rw-r-- xxxx xxx
m 2064 0x00000000 --rw-rw-r-- xxxx xxx
m 2065 0x00000000 --rw-rw-r-- xxxx xxx
m 2066 0x00000000 --rw-rw-r-- xxxx xxx
m 2067 0x00000000 --rw------- xxxx xxx
m 2068 0x00000000 --rw------- xxxx xxx
m 21 0x00000000 --rw-rw-r-- xxxx xxx
m 22 0x00000000 --rw------- xxxx xxx
m 23 0x00000000 --rw------- xxxx xxx
m 24 0x435dce60 --rw-rw-rw- root root
---------------------------------------------
# swlist -l product | grep -i link
Networking B.11.00 HP-UX_10.0_Lanlink_Product
PHCO_16885 1.0 pax(1) restoring hard and soft links
PHKL_20178 1.0 Updated linkorder files
PHKL_24201 1.0 JFS;stickybit;quota access;vx_maxlink tunabl
PHSS_19866 1.0 ld(1) and linker tools cumulative patch
PHSS_24303 1.0 ld(1) and linker tools cumulative patch
PHSS_30048 1.0 ld(1) and linker tools cumulative patch
PHSS_30965 1.0 ld(1) and linker tools cumulative patch
PHSS_33032 1.0 ld(1) and linker tools cumulative patch
PHSS_33034 1.0 ld(1) and linker tools cumulative patch

[Gregory Marchese_1]

This looks less cryptic. :) Thanks again.

$ ipcs -ma
IPC status from /dev/kmem as of Mon Feb 26 09:13:10 2007
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
Shared Memory:
m 0 0x411c0627 --rw-rw-rw- root root root root 0 348 1046 1046 6:18:08 6:18:08 6:18:02
m 1 0x4e0c0002 --rw-rw-rw- root root root root 1 31040 1046 1046 6:18:04 6:18:08 6:18:02
m 2 0x412006b0 --rw-rw-rw- root root root root 1 8192 1046 1058 6:18:04 6:18:02 6:18:02
m 11267 0x0c6629c9 --rw-r----- root root root root 2 19508728 2410 2406 6:19:28 no-entry 6:19:28
m 4 0x06347849 --rw-rw-rw- root root root root 1 77384 2410 2506 6:19:31 6:19:28 6:19:28
m 1029 0xffffffff --rw-r--rw- root root root root 0 22908 2406 2406 6:19:29 6:19:29 6:19:29
m 320518 0x00000000 --rw-rw-r-- xxxxxxxx xxxx root xxxx 240 134217728 10961 27729 9:12:41 9:10:29 6:24:01
m 7 0x00000000 --rw-rw-r-- xxxxxxxx xxxx root xxxx 240 134217728 10961 27729 9:12:41 9:10:29 6:24:01
m 8 0x00000000 --rw-rw-r-- xxxxxxxx xxxx root xxxx 240 134217728 10961 27729 9:12:41 9:10:29 6:24:01
m 9 0x00000000 --rw-rw-r-- xxxxxxxx xxxx root xxxx 240 134217728 10961 27729 9:12:41 9:10:29 6:24:01
m 10 0x00000000 --rw-rw-r-- xxxxxxxx xxxx root xxxx 240 134217728 10961 27729 9:12:41 9:10:29 6:24:01
m 11 0x00000000 --rw-rw-r-- xxxxxxxx xxxx root xxxx 240 307200 10961 27729 9:12:41 9:10:29 6:24:01
m 12 0x00000000 --rw------- xxxxxxxx xxxx xxxxxxxx xxxx 1 3092 11445 11445 6:24:44 no-entry 6:24:44
m 13 0x00000000 --rw------- xxxxxxxx xxxx xxxxxxxx xxxx 1 3092 11450 11450 6:24:44 no-entry 6:24:44
m 14 0x00000000 --rw-rw-r-- xxxxxxxx xxxx root xxxx 26 9822208 12164 26190 9:10:01 9:10:01 6:25:05
m 15 0x00000000 --rw------- xxxxxxxx xxxx xxxxxxxx xxxx 1 852 12684 12684 6:25:23 no-entry 6:25:23
m 16 0x00000000 --rw------- xxxxxxxx xxxx xxxxxxxx xxxx 1 3092 12690 12690 6:25:23 no-entry 6:25:23
m 17 0x435dce60 --rw-rw-rw- root root root root 1 4008 12924 12924 6:25:31 no-entry 6:25:31

[Dennis Handly]

You have several large segments with same creation time. Do you know if they have been leaked?
m 11267 0x0c6629c9 --rw-r----- root root root root 2 19508728 2410 2406 6:19:28 no-entry 6:19:28
m 320518 0x00000000 --rw-rw-r-- xxxxxxxx xxxx root xxxx 240 134217728 10961 27729 9:12:41 9:10:29 6:24:01
m 7 0x00000000 --rw-rw-r-- xxxxxxxx xxxx root xxxx 240 134217728 10961 27729 9:12:41 9:10:29 6:24:01
m 8 0x00000000 --rw-rw-r-- xxxxxxxx xxxx root xxxx 240 134217728 10961 27729 9:12:41 9:10:29 6:24:01
m 9 0x00000000 --rw-rw-r-- xxxxxxxx xxxx root xxxx 240 134217728 10961 27729 9:12:41 9:10:29 6:24:01
m 10 0x00000000 --rw-rw-r-- xxxxxxxx xxxx root xxxx 240 134217728 10961 27729 9:12:41 9:10:29 6:24:01
m 14 0x00000000 --rw-rw-r-- xxxxxxxx xxxx root xxxx 26 9822208 12164 26190 9:10:01 9:10:01 6:25:05

[Sheriff Andy]

Gregory,

Sorry you are having the problems. One more reason not to go to NBU 6.0 right now.

Are you properly patched?

I would post this on the Veritas NBU forums as well.