Sybase dataserver hangs and can't be killed

[David Child_1]

I know this issue has been hashed over numerous times, but I'm trying to find out what might be causing the initial hang.

System Info:
- V-2250 (4x8)
- HP-UX 11i (patched within the last 6 months)

Storage:
- EMC Symmetrix 8830
- SAN attached via two DS-24M2 (McData) switches

Applications:
- 16 Sybase databases (mix of 11.9.3 and 12.5).

Senario:
Approximately every 4 to 10 days one or more of the Sybase dataservers hangs. Its not always the same ones and there doesn't seem to be any pattern in the date/time. When the databases hang the DBAs try to shut it down, but end up trying to kill it. Then they come to me. By this time the processes are already owned by init (PPID=1). This server was recently upgraded from 11.00 to 11i (actually a cold install) and the DBAs are claiming this is the problem. I have performed this upgrade on 9 servers so far and this is the only one where this problem is occuring.

I am aware that the process is most likely waiting on some resource, but I don't know what.

Here is some additional info:
- Priority=168 (SLEEPING)
- I have attached the output of 'lsof -p '
- There are no NFS mounts
- there are no 'pfs' mounts (or daemons running)
- the DBAs claim they had not been writing to the tape drive or performing any backups (I'm trying to verify this)
- the system is running at full memory capacity and swapinfo currently shows 3% in use (swapmem_on=0).

I know a reboot will fix it, but it will happen again and again unless I can find out what these processes are waiting for. I have already asked the DBAs to come to me immediately when the hang happens (BEFORE trying to kill it) in the hopes that I might be able to get some more info (using tusc, etc.).

In the meantime, any sugguestions on how to determine what they are waiting for?

Thanks,
David

[David Child_1]

Sorry, this was created under "Linux" by mistake. I have moved it to the "HP-UX" section. Please see that post instead.

David