topic Re: 11.31 system pausing in Operating System - HP-UX

11.31 system pausing

Andrew Scott_3 — Tue, 31 Jul 2007 13:01:04 GMT

This is an rx4640 running 11.31 and Oracle 10g.

We've run into a big user-interaction problem. It appears that anything that requires I/O to vg00 is causing a wait. It does not seem to affect Oracle. It's running on a SAN presented volume in a different volume group.

This problem comes and goes, its very frustrating.

There are no hardware errors in the chassis log. Nothing unusual gets written to the syslog. The only thing I can find that might be a clue is an entry at the bottom of dmesg:

"Warning: The validity of the tunable values could not be completely verified, because the value of the tunable 'process_id_max' will not be known until the system is booted. The tunable values will be verified during boot. Please check the console messages during boot to see if there are any tunable value errors."

I checked process_id_max in kcweb, it was set to 30000. I upped it to 65535. Problem persists.

I've checked all the logical volumes on vg00, all the extents are current and vgsync completed with no errors on it, so I do not think that I have a bad disk at this time.

this is a production system, the earliest I can reboot it would be 4am tomorrow morning. Anything I can look at before then?

Re: 11.31 system pausing

James R. Ferguson — Tue, 31 Jul 2007 13:13:30 GMT

Hi Andrew:

Upon what are you basing your statement that "...anything that requires I/O to vg00 is causing a wait"?

Do you have 'sar' output? 'glance' data?

Are you running a very large buffer cache? What mountpoint options do you have for your Oracle filesystems?

Is this a new behavior? If so, what has changed?

Regards!

...JRF...

Re: 11.31 system pausing

Andrew Scott_3 — Tue, 31 Jul 2007 13:25:09 GMT

My statement is based purely on the observation that:

Logging in takes a long time. username prompt pops immediately, it is at least 30 seconds before you get a password prompt.

Running a program that hasn't been run in awhile, like sar or top, takes at least 30 seconds after you hit enter at the command line, then it loads and runs as normal.

Glance isn't available. I lost that battle with the bean counters and the network managmement guy that thinks his SNMP system can do everything.

This is sar output from a period when it hung when I hit "q" to exit top in another terminal. I hit "q" 14:19:57. The shell returned at 14:22:20 (marked with asterisks). There's an interruption because I guessed wrong on the n parameter and sar exited.

# sar 3 50

HP-UX prepora1 B.11.31 U ia64 07/31/07

14:19:03 %usr %sys %wio %idle
14:19:06 3 1 15 81
14:19:09 1 0 4 95
14:19:12 3 1 2 94
14:19:15 1 0 1 98
14:19:18 1 1 2 96
14:19:21 0 1 0 99
14:19:24 0 0 0 100
14:19:27 1 1 10 88
14:19:30 0 0 1 98
14:19:33 5 0 0 95
14:19:36 0 0 1 99
14:19:39 0 0 0 100
14:19:42 0 0 5 95
14:19:45 1 0 1 98
14:19:48 0 0 1 99
14:19:51 0 0 0 99
14:19:54 0 0 0 99
*14:19:57 0 0 0 100
14:20:00 1 0 7 92
14:20:03 4 2 6 88
14:20:06 0 0 0 99
14:20:09 2 1 8 89
14:20:12 5 7 16 72
14:20:15 1 3 14 83
14:20:18 3 1 4 92
14:20:21 2 0 5 92
14:20:24 3 1 3 93
14:20:27 3 1 10 85
14:20:30 3 1 4 93
14:20:33 9 1 8 82
14:20:36 4 1 2 92
14:20:39 3 0 3 93
14:20:42 3 1 25 71
14:20:45 3 1 6 90
14:20:48 3 0 2 95
14:20:51 3 0 2 95
14:20:54 2 4 5 89
14:20:57 3 0 3 93
14:21:00 3 1 5 91
14:21:03 4 1 5 91
14:21:06 6 1 8 84
14:21:09 4 1 5 90
14:21:12 4 1 3 93
14:21:15 3 1 2 93
14:21:18 6 2 5 86
14:21:21 4 1 2 94
14:21:24 3 0 2 94
14:21:27 19 2 3 76
14:21:30 28 5 7 61
14:21:33 30 4 8 58

Average 4 1 5 90
#
sh: ^[[A^[[B: not found.
#
# sar 3 100

HP-UX prepora1 B.11.31 U ia64 07/31/07

14:21:59 %usr %sys %wio %idle
14:22:02 5 1 27 67
14:22:05 7 2 15 76
14:22:08 5 2 3 90
14:22:11 4 1 2 93
14:22:14 3 0 3 94
14:22:17 4 0 3 93
*14:22:20 7 4 7 82
14:22:23 4 2 9 85

Over two minutes to exit from top to a command prompt on a system that was very lightly loaded at the time, according to these numbers.

Re: 11.31 system pausing

Andrew Scott_3 — Tue, 31 Jul 2007 14:18:09 GMT

If by buffer cache, you mean filecache_max, its set at 4070219776, and is currently 95.2% utilized according to kcweb.

The oracle filesystems are mounted with ioerror=mwdisable,largefiles,delaylog.

Yes, something did change. We installed 11iV3. This did not happen on the same hardware with 11iV2.

We've increased maxuprc, nproc, nkthread, ninode, semmns, and shmmax from the stock values. Our list of tunables looks like this:

filecache_max 3885748224 4070219776
maxdsiz 24961024 1073741824
maxdsiz_64bit 4280320 4294967296
maxfiles_lim 308 4096
maxssiz 86016 8388608
maxssiz_64bit 692224 268435456
maxtsiz 864256 100663296
maxtsiz_64bit 79478784 1073741824
maxuprc 81 8192
max_thread_proc 31 1101
msgmni 2 512
msgtql 0 1024
nflocks 313 4096
ninode 1367 8192
nkthread 457 16484
nproc 247 16384
npty 1 60
nstrpty 2 60
nstrtel 0 60
nswapdev 1 32
nswapfs 0 32
semmni 23 2048
semmns 463 4096
shmmax 1610620928 4096000000
shmmni 6 400
shmseg 3 300

Re: 11.31 system pausing

Torsten. — Wed, 01 Aug 2007 03:16:06 GMT

Please some more hardware details!

Internal boot disks?
What "SAN" (array)?

Anything connected to the core IO card?

For the beginning, an "ioscan -fn" would be fine.

Re: 11.31 system pausing

Andrew Scott_3 — Wed, 01 Aug 2007 08:15:06 GMT

The external storage is a SAN attached EVA4000. The fabric is 4GB using HP Director san switches.

# ioscan -fn
Class I H/W Path Driver S/W State H/W Type Description
==============================================================================
root 0 root CLAIMED BUS_NEXUS
ioa 0 0 sba CLAIMED BUS_NEXUS System Bus Adapter (1229)
ba 0 0/0 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (122e)
tty 0 0/0/1/0 asio0 CLAIMED INTERFACE PCI SimpleComm (103c1290)
/dev/diag/mux0 /dev/mux0 /dev/tty0p0
tty 1 0/0/1/1 asio0 CLAIMED INTERFACE PCI Serial (103c1048)
/dev/GSPdiag1 /dev/mux1 /dev/tty1p2
/dev/diag/mux1 /dev/tty1p0 /dev/tty1p4
OO 0 0/0/2/0 UsbOhci CLAIMED INTERFACE PCI SerialBus (10330035)
OO 1 0/0/2/1 UsbOhci CLAIMED INTERFACE PCI SerialBus (10330035)
OO 2 0/0/2/2 UsbEhci CLAIMED INTERFACE PCI SerialBus (103300e0)
sideba 0 0/0/3/0 side_multi CLAIMED INTERFACE CMD IDE controller
ext_bus 0 0/0/3/0.0 side CLAIMED INTERFACE IDE Primary Channel
/dev/side0
target 30 0/0/3/0.0.0 tgt CLAIMED DEVICE
disk 2 0/0/3/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-C
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
ext_bus 1 0/0/3/0.1 side CLAIMED INTERFACE IDE Secondary Channel
/dev/side1
graphics 0 0/0/4/0 gvid_core CLAIMED INTERFACE PCI Display (10025159)
ba 1 0/1 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (122e)
slot 0 0/1/1 pci_slot CLAIMED SLOT PCI Slot
ext_bus 2 0/1/1/0 mpt CLAIMED INTERFACE SCSI Ultra320 A6961-60011
/dev/mpt2
target 0 0/1/1/0.0 tgt CLAIMED DEVICE
disk 0 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 73.4GMAU3073NC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
/dev/dsk/c2t0d0s1 /dev/rdsk/c2t0d0s1
/dev/dsk/c2t0d0s2 /dev/rdsk/c2t0d0s2
/dev/dsk/c2t0d0s3 /dev/rdsk/c2t0d0s3
target 1 0/1/1/0.1 tgt CLAIMED DEVICE
disk 1 0/1/1/0.1.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c2t1d0 /dev/rdsk/c2t1d0
/dev/dsk/c2t1d0s1 /dev/rdsk/c2t1d0s1
/dev/dsk/c2t1d0s2 /dev/rdsk/c2t1d0s2
/dev/dsk/c2t1d0s3 /dev/rdsk/c2t1d0s3
ext_bus 3 0/1/1/1 mpt CLAIMED INTERFACE SCSI Ultra320 A6961-60011
/dev/mpt3
slot 1 0/1/2 pci_slot CLAIMED SLOT PCI Slot
lan 0 0/1/2/0 iether CLAIMED INTERFACE HP AB352-60001 PCI/PCI-X 1000Base-T Dual-port Core
lan 1 0/1/2/1 iether CLAIMED INTERFACE HP AB352-60001 PCI/PCI-X 1000Base-T Dual-port Core
ba 2 0/2 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (122e)
slot 2 0/2/1 pci_slot CLAIMED SLOT PCI Slot
lan 2 0/2/1/0 igelan CLAIMED INTERFACE HP A6825-60101 PCI 1000Base-T Adapter
ba 3 0/3 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (122e)
slot 3 0/3/1 pci_slot CLAIMED SLOT PCI Slot
slot 4 0/3/2 pci_slot CLAIMED SLOT PCI Slot
fc 0 0/3/2/0 td CLAIMED INTERFACE HP Tachyon XL2 Fibre Channel Mass Storage Adapter
/dev/td0
fcp 0 0/3/2/0.13 fcp CLAIMED INTERFACE FCP Domain
ext_bus 7 0/3/2/0.13.16.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
target 4 0/3/2/0.13.16.0.0.0 tgt CLAIMED DEVICE
ctl 3 0/3/2/0.13.16.0.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/rscsi/c7t0d0
ext_bus 6 0/3/2/0.13.16.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 3 0/3/2/0.13.16.255.0.0 tgt CLAIMED DEVICE
ctl 2 0/3/2/0.13.16.255.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/rscsi/c6t0d0
ext_bus 5 0/3/2/0.13.17.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
target 6 0/3/2/0.13.17.0.0.0 tgt CLAIMED DEVICE
ctl 1 0/3/2/0.13.17.0.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/rscsi/c5t0d0
ext_bus 4 0/3/2/0.13.17.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 5 0/3/2/0.13.17.255.0.0 tgt CLAIMED DEVICE
ctl 0 0/3/2/0.13.17.255.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/rscsi/c4t0d0
ext_bus 9 0/3/2/0.13.18.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
target 11 0/3/2/0.13.18.0.0.0 tgt CLAIMED DEVICE
disk 6 0/3/2/0.13.18.0.0.0.1 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c9t0d1 /dev/rdsk/c9t0d1
disk 7 0/3/2/0.13.18.0.0.0.2 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c9t0d2 /dev/rdsk/c9t0d2
disk 16 0/3/2/0.13.18.0.0.0.3 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c9t0d3 /dev/rdsk/c9t0d3
disk 17 0/3/2/0.13.18.0.0.0.4 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c9t0d4 /dev/rdsk/c9t0d4
disk 26 0/3/2/0.13.18.0.0.0.5 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c9t0d5 /dev/rdsk/c9t0d5
ext_bus 8 0/3/2/0.13.18.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 10 0/3/2/0.13.18.255.0.0 tgt CLAIMED DEVICE
ctl 4 0/3/2/0.13.18.255.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/rscsi/c8t0d0
ext_bus 11 0/3/2/0.13.19.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
target 9 0/3/2/0.13.19.0.0.0 tgt CLAIMED DEVICE
disk 8 0/3/2/0.13.19.0.0.0.1 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c11t0d1 /dev/rdsk/c11t0d1
disk 9 0/3/2/0.13.19.0.0.0.2 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c11t0d2 /dev/rdsk/c11t0d2
disk 22 0/3/2/0.13.19.0.0.0.3 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c11t0d3 /dev/rdsk/c11t0d3
disk 23 0/3/2/0.13.19.0.0.0.4 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c11t0d4 /dev/rdsk/c11t0d4
disk 27 0/3/2/0.13.19.0.0.0.5 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c11t0d5 /dev/rdsk/c11t0d5
ext_bus 10 0/3/2/0.13.19.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 8 0/3/2/0.13.19.255.0.0 tgt CLAIMED DEVICE
ctl 6 0/3/2/0.13.19.255.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/rscsi/c10t0d0
ext_bus 12 0/3/2/0.13.25.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 7 0/3/2/0.13.25.255.0.0 tgt CLAIMED DEVICE
ctl 8 0/3/2/0.13.25.255.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV110 (C)COMPAQ
/dev/rscsi/c12t0d0
ext_bus 13 0/3/2/0.13.26.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 2 0/3/2/0.13.26.255.0.0 tgt CLAIMED DEVICE
ctl 9 0/3/2/0.13.26.255.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV110 (C)COMPAQ
/dev/rscsi/c13t0d0
ba 4 0/4 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (122e)
slot 5 0/4/1 pci_slot CLAIMED SLOT PCI Slot
slot 6 0/4/2 pci_slot CLAIMED SLOT PCI Slot
fc 1 0/4/2/0 td CLAIMED INTERFACE HP Tachyon XL2 Fibre Channel Mass Storage Adapter
/dev/td1
fcp 2 0/4/2/0.3 fcp CLAIMED INTERFACE FCP Domain
ext_bus 29 0/4/2/0.3.1.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
target 27 0/4/2/0.3.1.0.0.0 tgt CLAIMED DEVICE
ctl 25 0/4/2/0.3.1.0.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV110 (C)COMPAQ
/dev/rscsi/c29t0d0
ext_bus 28 0/4/2/0.3.1.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 26 0/4/2/0.3.1.255.0.0 tgt CLAIMED DEVICE
ctl 24 0/4/2/0.3.1.255.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV110 (C)COMPAQ
/dev/rscsi/c28t0d0
ext_bus 31 0/4/2/0.3.5.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
target 29 0/4/2/0.3.5.0.0.0 tgt CLAIMED DEVICE
ctl 27 0/4/2/0.3.5.0.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV110 (C)COMPAQ
/dev/rscsi/c31t0d0
ext_bus 30 0/4/2/0.3.5.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 28 0/4/2/0.3.5.255.0.0 tgt CLAIMED DEVICE
ctl 26 0/4/2/0.3.5.255.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV110 (C)COMPAQ
/dev/rscsi/c30t0d0
fcp 1 0/4/2/0.5 fcp CLAIMED INTERFACE FCP Domain
ext_bus 22 0/4/2/0.5.16.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
target 23 0/4/2/0.5.16.0.0.0 tgt CLAIMED DEVICE
disk 10 0/4/2/0.5.16.0.0.0.1 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c22t0d1 /dev/rdsk/c22t0d1
disk 11 0/4/2/0.5.16.0.0.0.2 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c22t0d2 /dev/rdsk/c22t0d2
disk 24 0/4/2/0.5.16.0.0.0.3 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c22t0d3 /dev/rdsk/c22t0d3
disk 25 0/4/2/0.5.16.0.0.0.4 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c22t0d4 /dev/rdsk/c22t0d4
disk 28 0/4/2/0.5.16.0.0.0.5 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c22t0d5 /dev/rdsk/c22t0d5
ext_bus 21 0/4/2/0.5.16.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 22 0/4/2/0.5.16.255.0.0 tgt CLAIMED DEVICE
ctl 17 0/4/2/0.5.16.255.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/rscsi/c21t0d0
ext_bus 24 0/4/2/0.5.17.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
target 25 0/4/2/0.5.17.0.0.0 tgt CLAIMED DEVICE
disk 12 0/4/2/0.5.17.0.0.0.1 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c24t0d1 /dev/rdsk/c24t0d1
disk 13 0/4/2/0.5.17.0.0.0.2 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c24t0d2 /dev/rdsk/c24t0d2
disk 20 0/4/2/0.5.17.0.0.0.3 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c24t0d3 /dev/rdsk/c24t0d3
disk 21 0/4/2/0.5.17.0.0.0.4 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c24t0d4 /dev/rdsk/c24t0d4
disk 29 0/4/2/0.5.17.0.0.0.5 sdisk CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/dsk/c24t0d5 /dev/rdsk/c24t0d5
ext_bus 23 0/4/2/0.5.17.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 24 0/4/2/0.5.17.255.0.0 tgt CLAIMED DEVICE
ctl 19 0/4/2/0.5.17.255.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/rscsi/c23t0d0
ext_bus 20 0/4/2/0.5.18.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
target 21 0/4/2/0.5.18.0.0.0 tgt CLAIMED DEVICE
ctl 16 0/4/2/0.5.18.0.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/rscsi/c20t0d0
ext_bus 19 0/4/2/0.5.18.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 20 0/4/2/0.5.18.255.0.0 tgt CLAIMED DEVICE
ctl 15 0/4/2/0.5.18.255.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/rscsi/c19t0d0
ext_bus 16 0/4/2/0.5.19.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
target 14 0/4/2/0.5.19.0.0.0 tgt CLAIMED DEVICE
ctl 12 0/4/2/0.5.19.0.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/rscsi/c16t0d0
ext_bus 15 0/4/2/0.5.19.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 13 0/4/2/0.5.19.255.0.0 tgt CLAIMED DEVICE
ctl 11 0/4/2/0.5.19.255.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV111 (C)COMPAQ
/dev/rscsi/c15t0d0
ext_bus 26 0/4/2/0.5.21.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
target 17 0/4/2/0.5.21.0.0.0 tgt CLAIMED DEVICE
ctl 22 0/4/2/0.5.21.0.0.0.0 sctl CLAIMED DEVICE HP HSV200
/dev/rscsi/c26t0d0
ext_bus 25 0/4/2/0.5.21.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 16 0/4/2/0.5.21.255.0.0 tgt CLAIMED DEVICE
ctl 21 0/4/2/0.5.21.255.0.0.0 sctl CLAIMED DEVICE HP HSV200
/dev/rscsi/c25t0d0
ext_bus 18 0/4/2/0.5.22.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
target 19 0/4/2/0.5.22.0.0.0 tgt CLAIMED DEVICE
ctl 14 0/4/2/0.5.22.0.0.0.0 sctl CLAIMED DEVICE HP HSV200
/dev/rscsi/c18t0d0
ext_bus 17 0/4/2/0.5.22.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 18 0/4/2/0.5.22.255.0.0 tgt CLAIMED DEVICE
ctl 13 0/4/2/0.5.22.255.0.0.0 sctl CLAIMED DEVICE HP HSV200
/dev/rscsi/c17t0d0
ext_bus 27 0/4/2/0.5.25.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 15 0/4/2/0.5.25.255.0.0 tgt CLAIMED DEVICE
ctl 23 0/4/2/0.5.25.255.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV110 (C)COMPAQ
/dev/rscsi/c27t0d0
ext_bus 14 0/4/2/0.5.26.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 12 0/4/2/0.5.26.255.0.0 tgt CLAIMED DEVICE
ctl 10 0/4/2/0.5.26.255.0.0.0 sctl CLAIMED DEVICE COMPAQ HSV110 (C)COMPAQ
/dev/rscsi/c14t0d0
ba 5 0/5 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (122e)
slot 7 0/5/1 pci_slot CLAIMED SLOT PCI Slot
lan 3 0/5/1/0 igelan CLAIMED INTERFACE HP A6825-60101 PCI 1000Base-T Adapter
processor 0 120 processor CLAIMED PROCESSOR Processor
processor 1 122 processor CLAIMED PROCESSOR Processor
processor 2 123 processor CLAIMED PROCESSOR Processor
processor 3 124 processor CLAIMED PROCESSOR Processor
ba 6 250 pdh CLAIMED BUS_NEXUS Core I/O Adapter
ipmi 0 250/0 ipmi CLAIMED INTERFACE IPMI Controller
/dev/ipmi
acpi_node 0 250/1 acpi_node CLAIMED INTERFACE Acpi Hardware

Re: 11.31 system pausing

Andrew Scott_3 — Wed, 01 Aug 2007 08:15:12 GMT

Re: 11.31 system pausing

Torsten. — Wed, 01 Aug 2007 08:29:01 GMT

My thought is that the HSV200 is causing the trouble. It is zoned, but there are no disks assigned. Based on experience, the driver will looking for disks and this takes a lot of time ... consider to remove this EVA from the zone.
Switch Domain ID 5 - ports 21 and 22.

Re: 11.31 system pausing

Andrew Scott_3 — Wed, 01 Aug 2007 08:39:40 GMT

So the driver will constantly search for luns, causing I/O hangs? Is there a way to disable that behavior, so that it only searches for new luns when an ioscan is issued?

Re-doing our zoning is going to be messy. We can do it, but I'd rather look down other avenues first.

Re: 11.31 system pausing

Torsten. — Wed, 01 Aug 2007 08:41:05 GMT

Of course a problem in the SAN could also produce such behaviour, but either the removel of the HSV200 from the zone or the creation and presentation of a "test-vdisk" would be a good point for start.

Re: 11.31 system pausing

Torsten. — Wed, 01 Aug 2007 09:00:47 GMT

BTW, the diagnostic will issue an ioscan every ~20 min. This could really cause your issue.
(Don't think about disabling diags now!)

Try to create a small vdisk and present it - just to test this.

Re: 11.31 system pausing

Andrew Scott_3 — Wed, 01 Aug 2007 09:43:43 GMT

We rebooted the system, and the problem has gone away for now.

I see the possibility of the system scanning for luns causing some pauses, but would they last two to twenty minutes? I'm not being resistant, I'm just being thorough. Getting that HSV200 out of there isn't as simple as removing it from the zone, and just presenting a test volume also has some non-technical obstacles right now, plus that same HSV200 is visible to several other heavily-laden systems with no LUNs presented, and they aren't exhibiting this behavior.

Looking through KCtune, when the pauses started yesterday, the filecache was at 102%, over 4GB on this machine. The volumes oracle is running on are not mounted in a way to bypass the filecache. Could that cause some issues with I/O pausing?

Re: 11.31 system pausing

Duncan Edmonstone — Wed, 01 Aug 2007 09:46:30 GMT

Andrew,

I think what Torsten was trying to say is that there's 3 EVAs visible from this host:

HSV110 - that's an EVA5000 with the old A/P firmware - you don't appear to have any disks presented from this.

HSV111 - that's an EVA5000 with the new A/A firmware installed - this appears to be the disk you are using.

HSV200 - this is an EVA4000 - doesn't look like you have any disk presented from this either.

So if you don't use any disks on those two EVAs, why can you see them? Better to just have them zoned out in your fabric.

I'm not convinced this will solve your problem though, as it appears to be related to your local (internal) disk.

But I'd clear this up first before proceeding.

HTH

Duncan

Re: 11.31 system pausing

Andrew Scott_3 — Wed, 01 Aug 2007 09:54:39 GMT

Way back in the beginning of SAN at this organization, the decision was made to zone by OS, not host, so this box is in our HP-UX zone. All our EVAs are in the zone, because all our EVAs talk to at least one HP-UX box.

This is why I can't just zone it out right now, I'm working my way through the system to get the zoning changed, but I'm hoping there are some other things to look at too, because I know they're going to ask if I've checked anything else.

Re: 11.31 system pausing

MSD Unix Admin — Wed, 01 Aug 2007 19:22:21 GMT

We are seeing the same problem with a 11.31 server tha tis SAN attached, with our backups, some of the vg00 lvols take forever to backup, and one of our SAN attached disks takes a long time to respond. We are also seeing the login issue as well, where when running a backup it can take 30 seconds to login to the server.

No clues as to what is causing this, but thought you'd like to know you aren't alone.

Re: 11.31 system pausing

Andrew Scott_3 — Thu, 02 Aug 2007 08:30:44 GMT

Woo-hoo!

I'm currently opening a support call, but not on this issue. In an effort to isolate this machine and its EVA/LUNs in a zone by themselves, I've found that when I create a new Zone in my HP Director, put this equipment in it, remove it from the old zone, save and enable the config, it loses all connectivity.

Once that problem is resolved, I'll have these machines isolated in zones and we can move forward.

On another front, I reduced the filecache_max on our DEV system (the problem is in pre-production) from 50% to 10%, and then had the DBAs hammer on the thing for about four hours solid. There was no performance degradation, and the pausing did not manifest itself.

Re: 11.31 system pausing

Torsten. — Thu, 02 Aug 2007 08:34:15 GMT

Did you add the new zone to the default config?

I forgot this once, but now I remember it everytime.
;-)

Re: 11.31 system pausing

Paul Rafferty — Wed, 05 Dec 2007 21:25:14 GMT

We are experiencing a similar problem
with 11.31 were you able to determine the cause.

We have no san attached just internal disks on an rx6600.

Moderate i/o activity even listing a directory with thousands of files is causing the system to pause.

I would be interested to know if you did resolve your issue and if so how.

Thanks.

Re: 11.31 system pausing

Geoff Wild — Wed, 05 Dec 2007 21:55:47 GMT

This is a very interesting thread - one that is very close to me.

I have a RX6600 with Integrity Virtual Machines - so the Host is 11.23 and the 5 Guests are 11.31. All the guests are SAN attached disk including boot.

So far, I have seen no issues with I/O. (probably because the Host is 11.23).

Also have an Ignite server - RX3600 - it has SAN attatched and internal(for boot) - running 11.31 - and no issues.

We are about to roll out a new environment for our Prod and QA SAP systems on RX8640's - using vpars. All will have boot from SAN except for the first vpar.

BTW - all our SAN is on EMC DMX's....

Rgds...Geoff

Re: 11.31 system pausing

Hasan Atasoy — Wed, 05 Dec 2007 21:59:13 GMT

hi raul and andrew :

if your system has to many files in a filesystem ; then you are facing a problem 11.31.
please modify vxfs_ifree_timelag to -1 and reboot to system and monitor if you encounter or not.

Hasan.