How to reduce memory requirement.

[Massimo Bianchi]

File table full means that you shuold increase your nfile/nflocks kernel parameter. Double them as a start.

Have a look also at the syslog.log, i thought that you had already gave it a glance...

Massimo

[John Flanagan]

I checked out the file table full errors and the most recient is more than 3 weeks old.


This is the ana.out text for FUNC section of a quick process.

stack trace for process at 0x0'08a7b040 (pid 2318), thread at 0x0'0764a800 (tid 271961)
process was not running on any processor
FUNC PC SP FMP SSP
_sleep+0x214 0.0xb0f90 0x22e8400.0x0'7fff1070 0x22e8400.0x0'7fff1050 n/a
read_sleep+0x18c 0.0xc9b64 0x22e8400.0x0'7fff0f58 0x22e8400.0x0'7fff0f38 n/a
hpstreams_read_tty+0x43c 0.0x51c484 0x22e8400.0x0'7fff0f00 0x22e8400.0x0'7fff0ee0 n/a
hpstreams_read_int+0x7c8 0.0xca814 0x22e8400.0x0'7fff0df0 0x22e8400.0x0'7fff0dd0 n/a
hpstreams_read+0x38 0.0x523178 0x22e8400.0x0'7fff0cc8 0x22e8400.0x0'7fff0ca8 n/a
spec_rdwr+0x108 0.0x6b9f8 0x22e8400.0x0'7fff0c88 0x22e8400.0x0'7fff0c68 n/a
vno_rw+0x180 0.0x242104 0x22e8400.0x0'7fff0b80 0x22e8400.0x0'7fff0b60 n/a
read+0x104 0.0xf410c 0x22e8400.0x0'7fff0ae8 0x22e8400.0x0'7fff0ac8 n/a
syscall+0x1bc 0.0xdea80 0x22e8400.0x0'7fff0a60 0x22e8400.0x0'7fff0a40 n/a
$syscallrtn+0x0 0.0x34578 0x22e8400.0x0'7fff09a0 0x22e8400.0x0'7fff0980 n/a

___________________________________________
and for an oracle process


_sleep+0x214 0.0xb0f90 0x7006400.0x0'7fff0d08 0x7006400.0x0'7fff0ce8 n/a
fifo_rdwr+0x200 0.0x125790 0x7006400.0x0'7fff0bf0 0x7006400.0x0'7fff0bd0 n/a
vno_rw+0x180 0.0x242104 0x7006400.0x0'7fff0b80 0x7006400.0x0'7fff0b60 n/a
read+0x104 0.0xf410c 0x7006400.0x0'7fff0ae8 0x7006400.0x0'7fff0ac8 n/a
syscall+0x1bc 0.0xdea80 0x7006400.0x0'7fff0a60 0x7006400.0x0'7fff0a40 n/a
$syscallrtn+0x0 0.0x34578 0x7006400.0x0'7fff09a0 0x7006400.0x0'7fff0980 n/a

[John Flanagan]

Most frequent wait channesl

waitchannel # threads longest shortest
--------------------------- ----------- ----------------- ----------------
ksleep_one()+0x2b4 45 218396465 7947
per_processor_selects 34 218397340 293
FIFO_read 23 218394127 288
hpstreams_read_tty()+0x43c 22 1990319 317
()+0x 21 218394429 304
getmsg_subr()+0x504 20 2721070 1208
vx_worklist_thread_sv 15 604 304
wait1()+0x2a0 13 218388779 691943
ksleep_one()+0x13c 9 218396504 737
pm_sigwait()+0x1dc 8 218396509 362
lvmkd_q 6 5002 5002
0 4 218392636 287
thread_wait()+0x1a4 3 218396464 218392414
lbolt 2 315 315
socket_receive 2 218392636 218392497
SD$#var#tmp#_0_15007hpnfs_k 1 218399101 218399101

Ticks since boot: 218407425

[John Flanagan]

Most frequent wait channesl

waitchannel # threads longest shortest
--------------------------- ----------- ----------------- ----------------
ksleep_one()+0x2b4 45 218396465 7947
per_processor_selects 34 218397340 293
FIFO_read 23 218394127 288
hpstreams_read_tty()+0x43c 22 1990319 317
()+0x 21 218394429 304
getmsg_subr()+0x504 20 2721070 1208
vx_worklist_thread_sv 15 604 304
wait1()+0x2a0 13 218388779 691943
ksleep_one()+0x13c 9 218396504 737
pm_sigwait()+0x1dc 8 218396509 362
lvmkd_q 6 5002 5002
0 4 218392636 287
thread_wait()+0x1a4 3 218396464 218392414
lbolt 2 315 315
socket_receive 2 218392636 218392497
SD$#var#tmp#_0_15007hpnfs_k 1 218399101 218399101

Ticks since boot: 218407425


Regards,

John.

[Kent Ostby · Accepted solution]

John -- I'm sorry to say that the above stack and wait channel does not point to any definite known problem.

In doing some general research on processes hanging on 11.11, I found a few calls that pointed at a group of networking / process management patches that you might want to consider.

The patches in question start with this group (although three are superceeded):

PHNE_25642 -- ARPA
PHNE_25840 -- PSets
PHNE_25842 -- Thread Abort
PHNE_25871 -- socket close
PHNE_25084 -- STREAMS

The short description above are all from the Patch Description so you can check what you have by doing (for instance):

swlist -l product | grep ARPA

The latest versions of these patches are:

PHNE_29885 -- ARPA
PHNE_27317 -- PSets
PHNE_25842 -- Thread Abort
PHNE_25995 -- socket close
PHNE_28476 -- STREAMS

Again, these patches dont apply directly to your stack trace, but did fix some other process hangs that other customers have seen.

The other question that I wanted to ask you was whether the Q4 run you did was during one of the problem periods with the system ?

The data needs to be from when the system is having problems since its dynamic and will only really show us the issue if we run Q4 while its occurring.

Best regards,

Kent M. Ostby

[John Flanagan]

Kent,

It is difficult to run q4 when the problem is happening. Due to my limited no of users (25 concurrent) and the number of them that are active at any one time I would be very lucky to copy the kernel at the exact time.

I will go with installing the patches for now and I will post success or fail on this page towards the end of the week.


Everybody,

!!!Thanks for your time and effort!!!

Regards,

John.

[John Flanagan]

I have installed the recommended patches and all seems to be OK. I have not received a user complaint in the last 3 working days. I am hoping my problem is solved.

Thanks everybody for your help!

Regards,

john.