TSM error

John J Burke — Tue, 15 Apr 2008 18:20:11 GMT

I got the following error backing up two mountpoints on a hpux 11.0 server with TSM 5.2

ANS1030E System ran out of memory. Process ended.

ANS1999E Incremental processing of '/workfs' stopped.
Total number of objects inspected: 29,026
Total number of objects backed up: 125
Total number of objects updated: 1
Total number of objects rebound: 0
Total number of objects deleted: 0
Total number of objects expired: 0
Total number of objects failed: 0
Total number of bytes transferred: 124.00 KB
Data transfer time: 0.00 sec
Network data transfer rate: 102,657.09 KB/sec
Aggregate data transfer rate: 1.33 KB/sec
Objects compressed by: 0%
Elapsed processing time: 00:01:33
ANS1030E System ran out of memory. Process ended.

I have enabled the following dsm.sys parameter:
MEMORYEFFICIENTBACKUP YES

The problem is still occuring. There are over 1000,000 files in this mountpoint and 950,000

#################

Memory info

##################

* Tunable parameters

STRMSGSZ 65535
bufpages 102400
maxfiles 2048
maxfiles_lim 2048
maxswapchunks 3072
maxuprc ((NPROC*9)/10)
maxusers 1000
maxvgs 80
msgmax 32768
msgmnb 65535
msgmni (NPROC)
msgseg 7168
msgssz 128
msgtql 250
nfile 555000
nflocks (NPROC)
ninode (8*NPROC+2048)
nproc ((MAXUSERS*3)+64)
npty (MAXUSERS)
nstrpty (MAXUSERS)
nstrtel (MAXUSERS)
semmni (NPROC*5)
semmns (SEMMNI*2)
semmnu (NPROC-4)
semume 64
semvmx 32768
shmmni 512
st_san_safe 1
swapmem_on 0
unlockable_mem (MAXUSERS*10)
root@dbnhpu01:/tmp >ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 65536
stack(kbytes) 8192
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 2048

###

ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 65536
stack(kbytes) 8192
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 2048

###

glance -m

ProcList CPU Rpt Mem Rpt Disk Rpt NextKeys SlctProc Help Exit
B3692A GlancePlus C.03.72.00 20:09:57 dbnhpu01 9000/800 Current Avg High
----------------------------------------------------------------------------------------------------
CPU Util S SN NAU U | 54% 54% 54%
Disk Util F F | 83% 83% 83%
Mem Util S SU UB B | 69% 69% 69%
Networkil R R | 47% 47% 47%
----------------------------------------------------------------------------------------------------
MEMORY REPORT Users= 17
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 398 398 284.2 284.2 284.2
Page In 88 88 62.8 62.8 62.8
Page Out 1 1 0.7 0.7 0.7
KB Paged In 40kb 40kb 28.5 28.5 28.5
KB Paged Out 4kb 4kb 2.8 2.8 2.8
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 3 3 2.1 2.1 2.1
VM Writes 2 2 2.1 1.4 1.4

Total VM : 666.7mb Sys Mem : 357.8mb User Mem: 3.39gb Phys Mem: 6.00gb
Active VM: 386.7mb Buf Cache: 400.0mb Free Mem: 1.87gb

###

swapinfo -mat
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 6144 0 6144 0% 0 - 1 /dev/vg00/lvol2
reserve - 2885 -2885
total 6144 2885 3259 47% - 0 -

#########################
#########################

I am not familiar with tsm and unsure of what to change to make this work, would appreciate any advice

Re: TSM error

Don Morris_1 — Tue, 15 Apr 2008 20:18:12 GMT

The ulimit data implies that maxdsiz is only 64Mb. You still have over 3Gb of swap free [and if you'd set swapmem_on to 1 you'd have pseudo-swap from your 6Gb (around 4.5Gb or so on a quick in-my-head calculation) ].

So I think it is fair to assume your maxdsiz could be higher -- and that the failure point that caused the message was likely an allocation failure from malloc() when you hit 64Mb [due to the larger file set]. Being 11.0, unfortunately this will require a reboot to be sure, but raising maxdsiz and enabling swapmem_on would be my approach here. (How much to raise it is difficult to say without knowing the program and if you're worried about other programs eating up swap space... if you aren't worried about rogue swap reservation exhaustion, push up to 1Gb [32bit default architectural maximum]. Otherwise, you could try more gradual increases (128Mb / 256Mb) and check the VSZ of tsm as it runs to attempt to determine if it is hitting the limits.

Re: TSM error

John J Burke — Tue, 20 Oct 2009 13:39:40 GMT

changes to kernel parms did not fix this issue, this is an old version of tsm running on an old OS, we backed up the problem directories individually, as number of files plus depth of directory trees were causing the issue.

topic Re: TSM error in Operating System - HP-UX

TSM error

Re: TSM error

Re: TSM error