Problems with TSM 5.1 under hpux 11.0

[Peer-Joachim Koch]

Hi,

we are running one file server on hpux 11.00
which all patches for nearly one year now. It's
a N-class with 4 CPU's and 4GB RAM.

Now the tsm doesn't want to backup the file systems any more. We got a ANS1030E error (out of memmory).
When I run dsmc incremental
from the prompt eveything is fine !

bash-2.03# ulimit -a
core file size (blocks) 2097151
data seg size (kbytes) 524288
file size (blocks) unlimited
max memory size (kbytes) unlimited
open files 100
pipe size (512 bytes) 16
stack size (kbytes) 8192
cpu time (seconds) unlimited
max user processes 129
virtual memory (kbytes) unlimited

We also use the following settings within
the dsm.sys:
memoryefficientbackup yes
resourceutilization 2

Any idea ?

Thank's a lot, Peer

[RAC_1]

Does it give any oss error code??
Is there any more detailed message?

check following.

vmstat 2 5
Check for free column. (values are in pages. one page=4096k)
swapinfo -mat
glance -m

[Peer-Joachim Koch]

Hi,

I see no other system messages indicating
any problems. The last time we used the "normal" kernel, and this causes similiar
problems. But we used the settings from
the release nodes of dsm and everything was OK.

Currently only the interactive backup is running:
glance -m
vent Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 0 180 0.0 8.5 168.5
Page In 0 29 0.0 1.3 32.8
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 124kb 0.0 5.9 120.0
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Reactivations 5 25 0.9 1.1 14.2
Deactivations 5 25 0.9 1.1 7.1
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 0 7 0.0 0.3 5.7
VM Writes 0 0 0.0 0.0 0.0

Total VM : 855.4mb Sys Mem : 215.6mb User Mem: 1.76gb Phys Mem: 4.00gb
Active VM: 113.0mb Buf Cache: 2.00gb Free Mem: 34.3mb
bash-2.03# swapinfo -mat
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 218 806 21% 0 - 1 /dev/vg00/lvol2
reserve - 732 -732
memory 3150 2955 195 94%
total 4174 3905 269 94% - 0 -
bash-2.03# vmstat 2 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
0 2 0 29677 9768 0 0 0 0 0 0 0 2897 1772 526 1 2 97
0 2 0 29677 9709 3 0 0 0 2 0 0 1398 8652 287 1 0 98
0 2 0 27523 9709 1 0 0 0 0 0 0 1410 8719 318 13 6 80
0 2 0 27523 9709 0 0 0 0 0 0 0 1371 8295 234 0 2 98
1 1 0 48949 9709 0 0 0 0 0 0 0 1394 8749 283 13 6 81

[Victor BERRIDGE]

Hi Peer,

What I see from your last posting is:
You do actually have memory contention because you left the Buf Cache kernel parameter with the default value (50% 2GB!)which is far to high...
Start by compiling a new kernel with
dbc_max_pct 15

That should release enough memory (more than a GB...)

All the best
Victor

[Kent Ostby]

Peer --

Two issues.

#1) You have 4 gigabytes of physical ram but only 1 gigabyte of disk swap space. HP's default recommendation is still to have 2x ram worth of disk swap. Up your swap space and your system should run better.

#2) As mentioned by the above poster, you have max_dbc_pct set to 0 which tells the system to default to 50%. Change this to 20 and you will have plenty of DBC, but free up memory.

[Peer-Joachim Koch]

Hi, thank's to all for the hints!

1) I'm not sure about the kernel parameter:
>Buf Cache kernel parameter with the default >value (50% 2GB!)which is far to high...
>Start by compiling a new kernel with
>dbc_max_pct 15

From the ulimit output I can not see the relation to this parameter ?!?! Which one
of the ulimit.

It's not so easy to reboot one of the main file server in the moment, so I can not check the hint.

I'll see, if I can extend the swap space. But I have only space on an external raid lun.

But why can I use the smc command interactive ?
Bye the way the system is only running samba and nfs ...

Bye, Peer

[Victor BERRIDGE]

Hi again,
here is an example (real case...)
this box has 3GB of RAM , runs heavy applications (taxes..) writen in cobol with quite a lot of files (140 GB)
$ swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 2048 0 2048 0% 0 - 1 /dev/vg00/lvol2
dev 1000 0 1000 0% 0 - 1 /dev/r5vg01/lvol2
dev 1000 0 1000 0% 0 - 1 /dev/r5vg02/lvol2
reserve - 1160 -1160
total 4048 1160 2888 29% - 0 -


Total VM : 1.90gb Sys Mem : 871.6mb User Mem: 1.02gb Phys Mem: 3.00gb
Active VM: 1.66gb Buf Cache: 460.8mb Free Mem: 694.5mb

Look at Free Mem just above (694 Mb)
from the kernel:
$ cat /stand/system|grep dbc
dbc_max_pct 15


What do you have?
Total VM : 855.4mb Sys Mem : 215.6mb User Mem: 1.76gb Phys Mem: 4.00gb
Active VM: 113.0mb Buf Cache: 2.00gb Free Mem: 34.3mb
Free Mem 34.3 Mb


All the best
Victor

[Peer-Joachim Koch]

Hi,

thank's a lot for the describtion, which are causing two new questions ;)

1) Are there other kernel parameter as well
to have a closer look at this time ?
Sure, it's hard to say for an unknow system,
but the basic parameter are here.
4 CPU, 4 GB RAM, 2 FC-HBA, 5 TB diskspace
running SAMBA and NFS

2) Can I prepare a kernel and install everything, but without rebooting the machine ?
We have next week 10h down time (the main
power supply of the institute is getting checked), but I'm on vaction and can not build a new kernel after rebooting all systems .....
So I want to prepare something for my friends here.

Bye, Peer

[Victor BERRIDGE]

Hi again,

Well for that we would need to see what your current tunable kernel are set to...
For sure I imagine maxswapchunks would do with some tuning allowing you to be able to create more swap if needed e.g.
maxswapchunks 2025

Do a more of /stand/system and send us what follows the line * Tunable parameters

And Yes you can prepare a new kernel using SAM and answer reboot later... that will be used at next reboot (do a man kmupdate)

All the best
Victor

[Dr. Peer-Joachim Koch]

Hi,

* Tunable parameters

STRMSGSZ 65535
maxdsiz 0X020000000
maxfiles 100
maxswapchunks 512
maxtsiz 0X08000000
maxuprc 128
maxusers 256
nflocks 4096
ninode ((NPROC+32+MAXUSERS)+32+(2*NPTY))
nstrpty 60

thanks for the hint's !!

Bye, Peer

[Victor BERRIDGE]

maybe the output of sar -v 1 ?

Victor (waiting...)

[Victor BERRIDGE]

Well,
I would use these values on a HPUX-11.00 for what you say the box is used for:

STRMSGSZ 65535
maxdsiz 0X020000000
dbc_max_pct 12
maxfiles 160
maxfiles_lim 2048
maxswapchunks 2048
maxtsiz 0X08000000
maxuprc 128
maxusers 320
nflocks 4096
ninode ((NPROC+32+MAXUSERS)+32+(2*NPTY))
nkthread 2048
nproc (256+8*(MAXUSERS))
npty 128
nstrpty 72

It should let you add extra swap, leave you plenty of RAM, plus enough file descriptors etc...

All the best
Victor

[John Payne_2]

Are there a large number of files on this system?

We have one machine that started having this problem. (We eventually got to the point that it looked like there were just too many files on the system for TSM to handle) The "fix" was to set

MEMORYEFFICIENTBACKUP YES

in the dsm.opt file. (And restart tsm.)

This causes the backup to run a lot longer, but it connects.

We have planned on splitting that backup into pieces, one piece for parts of the entire system, but we have been busy and haven't had time to worry about it.

Hope it helps

John

[Dr. Peer-Joachim Koch]

Hi,

thank you Victor ! I'll try the values.
John - we have already added this key in the
sam.opt file. The system has to backup
~7 Mil files.

Bye, Peer

[Dr. Peer-Joachim Koch]

Hi,

the sar output:
bash-2.03# sar -v 1

HP-UX gaia B.11.00 U 9000/800 10/13/05

08:27:15 text-sz ov proc-sz ov inod-sz ov file-sz ov
08:27:16 N/A N/A 190/2068 0 0/2508 0 2663/4146 0

[Victor BERRIDGE]

Good morning Peer,
What I see here is you might reach nfile limits, so to respect the your tuning philosophy (using formula in order to change one value that affects automatically the others, so you just need for instance to increase maxusers value...)
I would recommend to modify nfile's formula to:
(4*nproc + 2048)
That should give nfile (max open files on system) around 12000 instead 4146 currently (with at the time of your sar 2663 open...)
You should be safe for some time after that...
Good luck


All the best
Victor