topic Re: SuperDome x Informix in Operating System - HP-UX

SuperDome x Informix

Carlos Roberto Schimidt — Mon, 26 Jun 2006 07:43:59 GMT

This problem occur only with SuperDome Server PA-RISC:

u125:/logs/conf# model
9000/800/SD32A

u125:/logs/conf# uname -a
HP-UX u125 B.11.23 U 9000/800 441251589 unlimited-user license

We try use HP-UX 11.11 or HP-UX 11.23.

We are trying migrate Informix from a server AIX to a server HP-UX, after migration the user complain about system

perfomance.
Then, some output commands were captured and migration was canceled.

Now, we developed an stress java program wich simulates the problem about 100 users accessing table with 210000 lines. We

have about 4000 users actually using the production system in a AIX box.

we have IDS running in the PA-RISC processor.
u122 (u122) /home/informix$onstat -
IBM Informix Dynamic Server Version 7.31.FD9PORT -- On-Line -- Up 19:46:19 -- 10922728 Kbytes

We use the same IDS configuration (NUMCPUVPS, BUFFERS, KAIO, ....) on both
servers (RP and Superdome) including kernel parameters.

We installed IDS 7.31 FD9 in the RP-4440 server and problem doesnt occurred. Then I reinstall the SuperDome Server like

RP-4440 server.

The problem doesn´t occur with RP-4440 using SAN Storage ( HDS or Shark).

Informix Support Team (IBM) and HP are working on the problem, the 7.31.FD9PORT developed by Informix solve the problem only

if we use chunks in the internal disks. If we use chunks in the storage, the problem happen again.

We have two SuperDome Servers each one are connect for differents Storages Box (IBM Shark / Hitachi HDS).

Informix Support Team (IBM) and HP continue working together to solve the problem.

The same problem occurs with IDS 9.40. This is the reason we are looking for people who has IDS running into SuperDome Server

to know it was possible to solve the problem. This problem only occurs in the SuperDome Machine.

See output commands below, wich "S-" on second column represents mutex contention.

u122 (u122) /home/informix$onstat -u | grep S
IBM Informix Dynamic Server Version 7.31.FD9PORT -- On-Line -- Up 19:52:40 -- 10922728 Kbytes
c00000011ae1f268 S--PR-- 34 dsvcona - c00000011c3a8c28 0 1 829 1232
c00000011ae20048 S--PR-- 35 dsvcona - c00000011c3a8c28 0 1 462 1232
c00000011ae20738 S--PR-- 36 dsvcona - c00000011c3a8c28 0 1 180 1120
c00000011ae21518 S--PR-- 38 dsvcona - c00000011c3a8c28 0 1 21 1176
c00000011ae21c08 S--PR-- 39 dsvcona - c00000011c3a8c28 0 1 21 1176
c00000011ae222f8 S--PR-- 40 dsvcona - c00000011c3a8c28 0 1 19 1064
c00000011ae229e8 S--PR-- 41 dsvcona - c00000011c3a8c28 0 1 15 840
c00000011ae230d8 S--PR-- 42 dsvcona - c00000011c3a8c28 0 1 8 448
c00000011ae245a8 S--PR-- 45 dsvcona - c00000011c3a8c28 0 1 5 280
c00000011ae26168 S--PR-- 49 dsvcona - c00000011c3a8c28 0 1 2 112
c00000011ae26f48 S--PR-- 51 dsvcona - c00000011c3a8c28 0 1 2 112
c00000011ae27638 S--PR-- 52 dsvcona - c00000011c3a8c28 0 1 2 112
c00000011ae27d28 S--PR-- 53 dsvcona - c00000011c3a8c28 0 1 2 112
c00000011ae28418 S--PR-- 54 dsvcona - c00000011c3a8c28 0 1 1 56
c00000011ae28b08 S--PR-- 55 dsvcona - c00000011c3a8c28 0 1 1 56
c00000011ae291f8 S--PR-- 56 dsvcona - c00000011c3a8c28 0 1 1 56
c00000011ae298e8 S--PR-- 57 dsvcona - c00000011c3a8c28 0 1 1 56
c00000011ae29fd8 S--PR-- 58 dsvcona - c00000011c3a8c28 0 1 1 56
c00000011ae2a6c8 S--PR-- 59 dsvcona - c00000011c3a8c28 0 1 1 56
c00000011ae2bb98 S--PR-- 62 dsvcona - c00000011c3a8c28 0 1 1 56
c00000011ae2c978 S--PR-- 64 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae2d068 S--PR-- 65 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae2de48 S--PR-- 67 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae2e538 S--PR-- 68 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae2ec28 S--PR-- 69 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae2f318 S--PR-- 70 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae2fa08 S--PR-- 71 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae300f8 S--PR-- 72 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae307e8 S--PR-- 73 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae31cb8 S--PR-- 76 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae32a98 S--PR-- 78 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae33878 S--PR-- 80 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae34658 S--PR-- 82 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae34d48 S--PR-- 83 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae35438 S--PR-- 84 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae35b28 S--PR-- 85 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae36218 S--PR-- 86 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae36ff8 S--PR-- 88 dsvcona - c00000011c3a8c28 0 1 0 0
c00000011ae376e8 S--PR-- 89 dsvcona - c00000011c3a8c28 0 1 0 0
u122 (u122) /home/informix$onstat -g lmx

IBM Informix Dynamic Server Version 7.31.FD9PORT -- On-Line -- Up 19:52:45 -- 10922728 Kbytes

Locked mutexes:
mid addr name holder lkcnt waiter waittime

u122 (u122) /home/informix$onstat -g lmx

IBM Informix Dynamic Server Version 7.31.FD9PORT -- On-Line -- Up 19:52:47 -- 10922728 Kbytes

Locked mutexes:
mid addr name holder lkcnt waiter waittime

u122 (u122) /home/informix$onstat -g lmx

IBM Informix Dynamic Server Version 7.31.FD9PORT -- On-Line -- Up 19:52:48 -- 10922728 Kbytes

Locked mutexes:
mid addr name holder lkcnt waiter waittime
2054 c00000011c3a8c28 pt_c0002a 205 0 105 21

u122 (u122) /home/informix$onstat -g lmx

IBM Informix Dynamic Server Version 7.31.FD9PORT -- On-Line -- Up 19:52:49 -- 10922728 Kbytes

Locked mutexes:
mid addr name holder lkcnt waiter waittime
2054 c00000011c3a8c28 pt_c0002a 140 0 153 11
150 12
128 15
107 21
159 11
125 15
125 15
181 8
176 9
160 11
135 14
144 12
110 21
103 22
171 9
130 14
186 8
166 10
154 11
161 10
128 15

Anybody been down this road before?

Tks

Schimidt

Re: SuperDome x Informix

Steve Lewis — Tue, 27 Jun 2006 04:41:28 GMT

Please provide your kernel parameters. I know that we bumped up semmnu quite a lot and semume to ~200, which increases the kernel size a lot.
I run 8 instances on 9.40.FC3 on a 2 npar SD32 (16 in each npar), plus xp12000, using raw, sym-linked logical volumes, hp-ux 11.11, 64 bit. No problems at all, but I never had more than 600 sessions in each instance on that box.
I have never before heard of anyone using 7.31 on a SD.

Re: SuperDome x Informix

Steve Lewis — Tue, 27 Jun 2006 04:51:25 GMT

Table size is not an issue at all until you get to 32Gb and fragmentation. E.g. I have about 100 databases on my SD. About 20 of these have tables with between 130-150 million rows in each and 2-3 indexes (about 28Gb actual tablespace size per table). No, your problems are most likely IPC or, multi-user related.

Another idea I have is this: how big are your shared memory segments and how many do you have? Keep this number low, see this, my favourite thread:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=147349

Re: SuperDome x Informix

Carlos Roberto Schimidt — Tue, 27 Jun 2006 05:33:50 GMT

Hi Steve,

IÂ´m attach file with kernel parameter.

We monitor systems calls from process oninit who create segment shared memory .

Here, if we are using internal disks, the problem doesnÂ´t occur:

/var/tmp/hp/without_hang/t.out
u122:/var/tmp/hp/without_hang# more t.out
( Attached to process 10000 ("oninit -v") [64-bit] )
1150977768.245431 [10000]{55517} #1 semop(34, 0xc00000011c6032b0, 1) [sleeping]
1150977768.834259 [10000]{55517} #1 semop(34, 0xc00000011c6032b0, 1) = 0
1150977769.844237 [10000]{55517} #1 semop(34, 0xc00000011c6032b0, 1) = 0
1150977770.334164 [10000]{55517} #1 Received signal 14, SIGALRM, in semop(), [caught], no siginfo
1150977770.334353 [10000]{55517} #1 semop(34, 0xc00000011c6032b0, 1) ERR#4 EINTR
1150977770.334592 [10000]{55517} #1 kill(10000, SIG#0) ... = 0
1150977770.334775 [10000]{55517} #1 times(0xc00000011c603b68) = 5908523
1150977770.334951 [10000]{55517} #1 alarm(3) ............. = 0
1150977770.854234 [10000]{55517} #1 semop(34, 0xc00000011c6032b0, 1) = 0
1150977771.864208 [10000]{55517} #1 semop(34, 0xc00000011c6032b0, 1) = 0
1150977772.874210 [10000]{55517} #1 semop(34, 0xc00000011c6032b0, 1) = 0
1150977773.344126 [10000]{55517} #1 Received signal 14, SIGALRM, in semop(), [caught], no siginfo
1150977773.344257 [10000]{55517} #1 semop(34, 0xc00000011c6032b0, 1) ERR#4 EINTR
1150977773.344438 [10000]{55517} #1 kill(10000, SIG#0) ... = 0
1150977773.344610 [10000]{55517} #1 times(0xc00000011c603b68) = 5908824
1150977773.344781 [10000]{55517} #1 alarm(3) ............. = 0
1150977773.884200 [10000]{55517} #1 semop(34, 0xc00000011c6032b0, 1) = 0
1150977774.894186 [10000]{55517} #1 semop(34, 0xc00000011c6032b0, 1) = 0
1150977775.904178 [10000]{55517} #1 semop(34, 0xc00000011c6032b0, 1) = 0
1150977776.354098 [10000]{55517} #1 Received signal 14, SIGALRM, in semop(), [caught], no siginfo
1150977776.354232 [10000]{55517} #1 semop(34, 0xc00000011c6032b0, 1) ERR#4 EINTR
1150977776.354409 [10000]{55517} #1 kill(10000, SIG#0) ... = 0

Here, if we are using external disks, when the problem occurs:

u122:/var/tmp/hp/with_hang# more t.out
( Attached to process 10000 ("oninit -v") [64-bit] )
1150978315.441252 [10000]{55517} #1 semop(16415, 0xc00000011efbe530, 2) [running]
1150978315.441783 [10000]{55517} #1 semop(16415, 0xc00000011efbe530, 2) = 0
1150978315.442272 [10000]{55517} #1 semop(16415, 0xc00000011efbe530, 2) = 0
1150978315.442734 [10000]{55517} #1 semop(16415, 0xc00000011efbe530, 2) = 0
1150978315.443078 [10000]{55517} #1 semop(34, 0xc00000011c6032b0, 1) = 0
1150978315.443591 [10000]{55517} #1 semop(34, 0xc00000011c6032b0, 1) = 0
1150978315.443928 [10000]{55517} #1 semop(34, 0xc00000011c6032b0, 1) = 0
1150978315.444411 [10000]{55517} #1 semop(16415, 0xc00000011efbe530, 2) = 0
1150978315.444838 [10000]{55517} #1 semop(16415, 0xc00000011efbe530, 2) = 0
1150978315.445309 [10000]{55517} #1 semop(16415, 0xc00000011f3d6530, 2) = 0

IÂ´m try undestand about the other link you post here.

Re: SuperDome x Informix

Carlos Roberto Schimidt — Tue, 27 Jun 2006 05:46:35 GMT

Hi Steve,

This is output from my system:

u122:/# ipcs -mob
IPC status from /dev/kmem as of Tue Jun 27 07:23:10 2006
T ID KEY MODE OWNER GROUP NATTCH SEGSZ
Shared Memory:
m 0 0x411c1146 --rw-rw-rw- root root 0 348
m 1 0x4e0c0002 --rw-rw-rw- root root 1 61760
m 2 0x41201627 --rw-rw-rw- root root 1 8192
m 163843 0x06347849 --rw-rw-rw- root root 2 65544
m 98308 0x0c6629c9 --rw-r----- root root 2 34906344
m 131077 0x4914b152 --rw-r--rw- root root 0 22908
m 98310 0x00a5c581 --rw------- sfmdb users 1 10493952
m 17891335 0x52574801 --rw-rw---- root informix 50 4294705152
m 1376264 0x52574802 --rw-rw---- root informix 50 4294705152
m 1245193 0x52574803 --rw-rw---- root informix 50 2594529280
m 1114122 0x52574804 --rw-rw-rw- root informix 28 1155072