Performance Problem

[Rene Mendez_4]

Enviroment:
HPUX 11.11 with LVM File System
Sybase 12.5
Cobis 3
Wsphere 6.0 with Row devices database
Storage EVA4000
Securepath Active-Active
Phisycal Memory: 14GB
When execute the online exits performance problem, the S.O command "bdf, ls, more" and new telnet conection no allow.

#sar -v 1 1

15:39:31 text-sz ov proc-sz ov inod-sz ov file-sz ov
15:39:32 N/A N/A 341/16404 0 2572/18620 0 3708/30130 0

#kmtune |grep maxfile
maxfiles 3072 - 3072
maxfiles_lim 1024 Y 1024

hysical memory = 3656960 14.0g 100%
Free memory = 169178 660.9m 5%
User processes = 2173697 8.3g 59% details with -user
System = 1304295 5.0g 36%
Kernel = 572903 2.2g 16% kernel text and data
Dynamic Arenas = 309137 1.2g 8% details with -arena
M_TEMP = 170304 665.2m 5%
M_IOSYS = 39368 153.8m 1%
M_SYSVMSGB = 16384 64.0m 0%
M_SPINLOCK = 12924 50.5m 0%
LVM_PBUF = 12446 48.6m 0%
Other arenas = 57711 225.4m 2%
Super page pool = 4635 18.1m 0%
Static Tables = 213580 834.3m 6%
pfdat = 83550 326.4m 2%
nbuf = 57984 226.5m 2% bufcache headers
htbl2_0 = 32768 128.0m 1%
pfn_to_virt = 13925 54.4m 0%
bufhash = 10240 40.0m 0% bufcache hash headers
Other tables = 15113 59.0m 0% Buffer cache = 731392 2.8g 20% details with -bufcache
UFC file mrg = 0 0.0b 0%
UFC meta mrg = 0 0.0b 0%

How to determine the problem ?

How to verity the use de maxfiles parameter ?

Regards
RM

[Michael Steele_2]

Dear Rene:

Please provide the following:

sar -uM 5 5
sar -d 5 5
sar -b 5 5
swapinfo -tam
vmstat 5 5

[Rene Mendez_4]

The sar and information

[Bill Hassell]

The maxfiles parameters are backwards. The maxfiles parameter limits the number of files a single process can open at the same time. Normally it would be set low (about 60) to prevent bad programs from using all the file descriptors in the system. 3072 is a very large number and very unusual. Do you know if you are running a programs that opens 3000 files at one time?

Now if you do have such a program, you would set maxfiles to 60 or maybe 100, then for the special program that needs a lot of files, you can raise the limit by including a ulimit command in the startup script (or have the programmer use the setrlimit system call) to increase the maxfiles allowed for the program.

By having maxfiles_lim smaller than maxfiles, your current limit is maxfiles_lim. Note that this is not a performance issue, it is just a limit to prevent runaway programs from consuming too many resources.

[Bill Hassell]

From your sar disk information, something is wrong with seveeral disk LUNs. There are several disks that are extremely slow. avwait and avserv are far too long and the avqueue is over 10000! The sar report also confirms that 75% of the time is spent waiting for I/O to complete. The bad disk LUNs are at:

c29t0d2
c31t0d2
c15t0d2
c17t0d2

Check your syslog file for disk error messages. Also run the dmesg command to see if there are additional error messages.

As far as kernel parmameters, the values for:

dbc_max_pct
nfile
nproc
ninode

are way too large for your current system. But the disks need fixing first.

[Arunkumar.B]

Hi

There is something wrong with the disk.Pls chk the dmesg , syslog & eventlog for any disk error.

Cheers
Arunkumar.B