Disk Failure

nasmel — Tue, 09 Nov 2010 07:08:01 GMT

Hi Experts

We are running HP service gurad cluster on two servers using CFS packages.
Yesterday we had a single physical disk failure on our EVA8000, and the cluster server connected to it also hanged at that time,we were not able to restart the cluster.
Then we rebooted the server and everything is working fine then.
Please check the below logs

Nov 8 02:05:00 hq-nps1 su: + tty?? root-oracle
Nov 8 02:06:43 hq-nps1 vmunix: LLT INFO V-14-1-10019 delayed hb 1025001 ticks from 2 link 1 (lan1)
Nov 8 02:06:43 hq-nps1 vmunix: LLT INFO V-14-1-10023 lost 20501 hb seq 35348299 from 2 link 1 (lan1)
Nov 8 02:05:10 hq-nps1 su: + tty?? root-oracle
Nov 8 02:06:43 hq-nps1 vmunix: LLT INFO V-14-1-10019 delayed hb 1025001 ticks from 2 link 3 (lan3)
Nov 8 02:06:43 hq-nps1 vmunix: LLT INFO V-14-1-10023 lost 20501 hb seq 35348299 from 2 link 3 (lan3)
Nov 8 02:06:43 hq-nps1 vmunix: LLT INFO V-14-1-10019 delayed hb 1025001 ticks from 2 link 2 (lan2)
Nov 8 02:06:43 hq-nps1 vmunix: LLT INFO V-14-1-10023 lost 20501 hb seq 35348299 from 2 link 2 (lan2)
Nov 8 02:19:38 hq-nps1 su: + tty?? root-oracle
Nov 8 02:19:39 hq-nps1 su: + tty?? root-oracle
Nov 8 05:45:00 hq-nps1 su: + tty?? root-oracle
Nov 8 05:54:14 hq-nps1 ftpd[26329]: Data port : 20
Nov 8 05:54:14 hq-nps1 ftpd[26329]: FTP server (Revision 1.1 Version wuftpd-2.6.1(PHNE_34698) Fri Nov 10 10:21:03 GMT 2006) ready.
Nov 8 05:54:14 hq-nps1 ftpd[26329]: FTP LOGIN FROM 172.16.1.23 [172.16.1.23], oper
Nov 8 05:54:34 hq-nps1 ftpd[26329]: FTP session closed
Nov 8 07:15:53 hq-nps1 vmunix: LLT INFO V-14-1-10019 delayed hb 1854973 ticks from 2 link 3 (lan3)
Nov 8 07:15:53 hq-nps1 vmunix: LLT INFO V-14-1-10023 lost 37099 hb seq 35385399 from 2 link 3 (lan3)
Nov 8 07:15:53 hq-nps1 vmunix: LLT INFO V-14-1-10019 delayed hb 1854973 ticks from 2 link 1 (lan1)
Nov 8 07:15:53 hq-nps1 vmunix: LLT INFO V-14-1-10023 lost 37099 hb seq 35385399 from 2 link 1 (lan1)
Nov 8 07:15:53 hq-nps1 vmunix: LLT INFO V-14-1-10019 delayed hb 1854973 ticks from 2 link 2 (lan2)
Nov 8 07:15:53 hq-nps1 vmunix: LLT INFO V-14-1-10023 lost 37099 hb seq 35385399 from 2 link 2 (lan2)
Nov 8 08:27:55 hq-nps1 vmunix: LLT INFO V-14-1-10019 delayed hb 432190 ticks from 2 link 1 (lan1)
Nov 8 08:27:55 hq-nps1 vmunix: LLT INFO V-14-1-10023 lost 8641 hb seq 35394041 from 2 link 1 (lan1)
Nov 8 08:27:55 hq-nps1 vmunix: LLT INFO V-14-1-10019 delayed hb 432190 ticks from 2 link 3 (lan3)
Nov 8 08:27:55 hq-nps1 vmunix: LLT INFO V-14-1-10023 lost 8641 hb seq 35394041 from 2 link 3 (lan3)
Nov 8 08:27:55 hq-nps1 vmunix: LLT INFO V-14-1-10019 delayed hb 432190 ticks from 2 link 2 (lan2)
Nov 8 08:27:55 hq-nps1 vmunix: LLT INFO V-14-1-10023 lost 8641 hb seq 35394041 from 2 link 2 (lan2)
Nov 8 10:37:02 hq-nps1 vmunix: LLT INFO V-14-1-10019 delayed hb 774750 ticks from 2 link 1 (lan1)
Nov 8 10:37:02 hq-nps1 vmunix: LLT INFO V-14-1-10023 lost 15494 hb seq 35409536 from 2 link 1 (lan1)
Nov 8 10:37:02 hq-nps1 vmunix: LLT INFO V-14-1-10019 delayed hb 774750 ticks from 2 link 3 (lan3)
Nov 8 10:37:02 hq-nps1 vmunix: LLT INFO V-14-1-10023 lost 15494 hb seq 35409536 from 2 link 3 (lan3)
Nov 8 10:37:02 hq-nps1 vmunix: LLT INFO V-14-1-10019 delayed hb 774750 ticks from 2 link 2 (lan2)
Nov 8 10:37:02 hq-nps1 vmunix: LLT INFO V-14-1-10023 lost 15494 hb seq 35409536 from 2 link 2 (lan2)
Nov 8 12:05:00 hq-nps1 su: + tty?? root-oracle
Nov 8 13:01:20 hq-nps1 LVM[5682]: lvlnboot -v /dev/vg00/
Nov 8 15:05:00 hq-nps1 su: + tty?? root-oper
Nov 8 15:15:53 hq-nps1 vmunix: LLT INFO V-14-1-10019 delayed hb 1673051 ticks from 2 link 3 (lan3)
Nov 8 15:15:53 hq-nps1 vmunix: LLT INFO V-14-1-10023 lost 33462 hb seq 35442999 from 2 link 3 (lan3)
Nov 8 15:15:53 hq-nps1 vmunix: LLT INFO V-14-1-10019 delayed hb 1673051 ticks from 2 link 1 (lan1)
Nov 8 15:15:53 hq-nps1 vmunix: LLT INFO V-14-1-10023 lost 33462 hb seq 35442999 from 2 link 1 (lan1)
Nov 8 15:15:53 hq-nps1 vmunix: LLT INFO V-14-1-10019 delayed hb 1673051 ticks from 2 link 2 (lan2)
Nov 8 15:15:53 hq-nps1 vmunix: LLT INFO V-14-1-10023 lost 33462 hb seq 35442999 from 2 link 2 (lan2)
Nov 8 22:19:53 hq-nps1 vmunix:
Nov 8 22:19:53 hq-nps1 vmunix: AUTOPATH : Path 0xbc1c0200 failed! Rerouting to alternate path
Nov 8 22:19:53 hq-nps1 vmunix: AUTOPATH : Path 0xbc1c0300 failed! Rerouting to alternate path
Nov 8 22:19:59 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-112 disabled path 31/0xc0200 belonging to the dmpnode 0/0x10
Nov 8 22:19:59 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-112 disabled path 31/0x160200 belonging to the dmpnode 0/0x10
Nov 8 22:19:59 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-112 disabled path 31/0xd0200 belonging to the dmpnode 0/0x10
Nov 8 22:19:59 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-112 disabled path 31/0x170200 belonging to the dmpnode 0/0x10
Nov 8 22:19:59 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-112 disabled path 31/0x180200 belonging to the dmpnode 0/0x10
Nov 8 22:19:59 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-112 disabled path 31/0x1a0200 belonging to the dmpnode 0/0x10
Nov 8 22:19:59 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-112 disabled path 31/0x1b0200 belonging to the dmpnode 0/0x10
Nov 8 22:19:59 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-112 disabled path 31/0x1c0200 belonging to the dmpnode 0/0x10
Nov 8 22:19:59 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-111 disabled dmpnode 0/0x10
Nov 8 22:19:59 hq-nps1 vmunix: WARNING: VxVM vxio V-5-0-2 Subdisk c4t0d2-01 block 216496: Uncorrectable write error
Nov 8 22:19:59 hq-nps1 vmunix: vxfs: WARNING: msgcnt 1 mesg 038: V-2-38: vx_dataioerr - /dev/vx/dsk/ora-software/vol1 file system file data
write error in dev/block 0/216496
Nov 8 22:19:59 hq-nps1 vmunix: vxfs: WARNING: msgcnt 2 mesg 031: V-2-31: vx_disable - /dev/vx/dsk/ora-software/vol1 file system disabled
Nov 8 22:19:59 hq-nps1 vmunix:
Nov 8 22:20:00 hq-nps1 above message repeats 10 times
Nov 8 22:20:00 hq-nps1 vmunix: WARNING: VxVM vxio V-5-0-2 Subdisk c4t0d2-01 block 0: Uncorrectable read error
Nov 8 22:20:05 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-112 disabled path 31/0xc0300 belonging to the dmpnode 0/0x20
Nov 8 22:20:05 hq-nps1 vmunix:
Nov 8 22:20:05 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-112 disabled path 31/0x160300 belonging to the dmpnode 0/0x20
Nov 8 22:20:05 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-112 disabled path 31/0xd0300 belonging to the dmpnode 0/0x20
Nov 8 22:20:05 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-112 disabled path 31/0x170300 belonging to the dmpnode 0/0x20
Nov 8 22:20:05 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-112 disabled path 31/0x180300 belonging to the dmpnode 0/0x20
Nov 8 22:20:05 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-112 disabled path 31/0x1a0300 belonging to the dmpnode 0/0x20
Nov 8 22:20:05 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-112 disabled path 31/0x1b0300 belonging to the dmpnode 0/0x20
Nov 8 22:20:05 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-112 disabled path 31/0x1c0300 belonging to the dmpnode 0/0x20
Nov 8 22:20:05 hq-nps1 vmunix: NOTICE: VxVM vxdmp V-5-0-111 disabled dmpnode 0/0x20
Nov 8 22:20:05 hq-nps1 vmunix: WARNING: VxVM vxio V-5-0-2 Subdisk c4t0d3-01 block 13556768: Uncorrectable write error
Nov 8 22:20:05 hq-nps1 vmunix: vxfs: WARNING: msgcnt 3 mesg 038: V-2-38: vx_dataioerr - /dev/vx/dsk/ora-data/vol1 file system file data writ
e error in dev/block 0/13556768
Nov 8 22:20:05 hq-nps1 vmunix: vxfs: WARNING: msgcnt 4 mesg 031: V-2-31: vx_disable - /dev/vx/dsk/ora-data/vol1 file system disabled
Nov 8 22:20:05 hq-nps1 vmunix:
Nov 8 22:20:05 hq-nps1 above message repeats 8 times
Nov 8 22:20:05 hq-nps1 vmunix: Synchronous Page I/O error occurred while paging to/from disk
Nov 8 22:20:05 hq-nps1 vmunix: device 0x010003e8, 1K block #19134672, 0 bytes of 16384 bytes not paged, page addr =a3c8431.4000000009e34000,
bflag =1f, bp = e000000188dc9980
Nov 8 22:20:06 hq-nps1 vmunix: device 0x010003e8, 1K block #14656175, 0 bytes of 16384 bytes not paged, page addr =33124431.4000000000220000
, bflag =1f, bp = e000000188dc9980
Nov 8 22:20:06 hq-nps1 vmunix: device 0x010003e8, 1K block #5147167, 0 bytes of 1024 bytes not paged, page addr =fbe0431.4000000000000000,
bflag =1f, bp = e000000188dc20c0
Nov 8 22:20:06 hq-nps1 vmunix: device 0x010003e8, 1K block #50813292, 0 bytes of 28672 bytes not paged, page addr =3bfc0031.c000000011c3a000
, bflag =1f, bp = e000000188dc9980
Nov 8 22:20:06 hq-nps1 vmunix: device 0x010003e8, 1K block #3834643, 0 bytes of 1024 bytes not paged, page addr =3bfc0031.c00000001f67a000,
bflag =1f, bp = e000000188dc9980
Nov 8 22:20:06 hq-nps1 vmunix: WARNING: VxVM vxio V-5-0-2 Subdisk c4t0d3-01 block 0: Uncorrectable read error
Nov 8 22:20:06 hq-nps1 syslog: Waiting for filesystem containing /u01/crs/oracle/product/10.2.0/app/bin/crsctl.
Nov 8 22:20:06 hq-nps1 vmunix: Synchronous Page I/O error occurred while paging to/from disk
Nov 8 22:20:06 hq-nps1 above message repeats 4 times
Nov 8 22:20:06 hq-nps1 cmclconfd[2527]: Updated file /etc/cmcluster/cmclconfig.tmp for node hq-nps1 (length = 96544).
Nov 8 22:20:06 hq-nps1 cmclconfd[2527]: Updated file /etc/cmcluster/cmclconfig.tmp for node hq-nps1 (length = 0).
Nov 8 22:20:06 hq-nps1 cmcld[2494]: Sending file $SGCONF/cmclconfig (47504 bytes) to file assistant daemon.
Nov 8 22:20:06 hq-nps1 vmunix: WARNING: VxVM vxio V-5-0-2 Subdisk c4t0d3-01 block 0: Uncorrectable read error
Nov 8 22:20:07 hq-nps1 cmfileassistd[21127]: Updated file /etc/cmcluster/cmclconfig (length = 47504).
Nov 8 22:20:06 hq-nps1 syslog: Waiting for filesystem containing /u01/crs/oracle/product/10.2.0/app/bin/crsctl.
Nov 8 22:20:10 hq-nps1 cmclconfd[21155]: Error opening /etc/cmcluster/cmknowncmds
Nov 8 22:20:10 hq-nps1 cmclconfd[21158]: Error opening /etc/cmcluster/cmknowncmds
Nov 8 22:20:11 hq-nps1 cmcld[2494]: Sending file $SGCONF/cmclconfig (46736 bytes) to file assistant daemon.
Nov 8 22:20:11 hq-nps1 cmfileassistd[21127]: Updated file /etc/cmcluster/cmclconfig (length = 46736).
Nov 8 22:20:12 hq-nps1 cmcld[2494]: Sending file $SGCONF/cmclconfig (46644 bytes) to file assistant daemon.
Nov 8 22:20:12 hq-nps1 cmfileassistd[21127]: Updated file /etc/cmcluster/cmclconfig (length = 46644).
Nov 8 22:20:16 hq-nps1 vmunix:

Nov 8 22:20:16 hq-nps1 vmunix:
Nov 8 22:20:16 hq-nps1 vmunix: AUTOPATH : Path 0xbc160600 failed! Rerouting to alternate path
Nov 8 22:20:16 hq-nps1 vmunix: LVM: WARNING: VG 64 0x020000: LV 1: Some I/O requests to this LV are waiting
Nov 8 22:20:16 hq-nps1 vmunix: indefinitely for an unavailable PV. These requests will be queued until
Nov 8 22:20:16 hq-nps1 vmunix: the PV becomes available (or a timeout is specified for the LV).
Nov 8 22:20:20 hq-nps1 cmclconfd[21223]: Error opening /etc/cmcluster/cmknowncmds
Nov 8 22:20:20 hq-nps1 cmclconfd[21226]: Error opening /etc/cmcluster/cmknowncmds
Nov 8 22:20:29 hq-nps1 vmunix: WARNING: WARNING: VVxxVVMM vvxxiioo VV--55--00-2 Subdisk c4t0d2-01 block 9104528: Uncorrectable write error
-2 Subdisk c4t0d2-01 block 2921984: Uncorrectable write error
Nov 8 22:20:29 hq-nps1 vmunix:
Nov 8 22:20:29 hq-nps1 cmclconfd[2527]: Updated file /etc/cmcluster/cmclconfig.tmp for node hq-nps1 (length = 93288).
Nov 8 22:20:29 hq-nps1 cmclconfd[2527]: Updated file /etc/cmcluster/cmclconfig.tmp for node hq-nps1 (length = 0).
Nov 8 22:20:29 hq-nps1 cmcld[2494]: Sending file $SGCONF/cmclconfig (45872 bytes) to file assistant daemon.
Nov 8 22:20:30 hq-nps1 cmfileassistd[21127]: Updated file /etc/cmcluster/cmclconfig (length = 45872).
Nov 8 22:20:30 hq-nps1 cmcld[2494]: Sending file $SGCONF/cmclconfig (45100 bytes) to file assistant daemon.
Nov 8 22:20:30 hq-nps1 syslog: Oracle CSSD failure 137.
Nov 8 22:20:30 hq-nps1 cmfileassistd[21127]: Updated file /etc/cmcluster/cmclconfig (length = 45100).
Nov 8 22:20:30 hq-nps1 syslog: Oracle CRS failure. Rebooting for cluster integrity.
Nov 8 22:20:30 hq-nps1 vmunix: Synchronous Page I/O error occurred while paging to/from disk
Nov 8 22:20:30 hq-nps1 vmunix: device 0x010003e8, 1K block #464200, 0 bytes of 1024 bytes not paged, page addr =b4a4431.4000000000000000, b
flag =1f, bp = e000000188dc9980
Nov 8 22:20:30 hq-nps1 syslogd: restart
Nov 8 22:20:30 hq-nps1 syslog: Oracle clsomon failed with fatal status 138.
Nov 8 22:20:30 hq-nps1 syslog: Oracle CRS failure. Rebooting for cluster integrity.
Nov 8 22:20:30 hq-nps1 syslogd: restart
Nov 8 22:20:30 hq-nps1 vmunix: 0/6/1/0: Device at device id 0x690005 has disappeared from Name Server GPN_FT
Nov 8 22:20:30 hq-nps1 vmunix: (FCP type) response, or its 'Port World-Wide Name' has changed.
Nov 8 22:20:30 hq-nps1 vmunix: device id = loop id, for private loop devices
Nov 8 22:20:30 hq-nps1 vmunix: device id = nport ID, for fabric/public-loop devices
Nov 8 22:20:30 hq-nps1 vmunix: System won't be able to see LUNs behind this port.
Nov 8 22:20:30 hq-nps1 vmunix:
#

Nov 8 22:32:18 hq-nps1 cmcld[2517]: 2 nodes have formed a new cluster, sequence #1
Nov 8 22:32:18 hq-nps1 cmcld[2517]: The new active cluster membership is: hq-nps1(id=1), hq-nps2(id=2)
Nov 8 22:32:18 hq-nps1 cmlvmd[2528]: Clvmd initialized successfully.
Nov 8 22:32:18 hq-nps1 cmcld[2517]: Package SG-CFS-DG-1 cannot run on this node because the following dependencies are not running: SG-CFS-p
kg
Nov 8 22:32:18 hq-nps1 cmcld[2517]: Package SG-CFS-MP-1 cannot run on this node because the following dependencies are not running: SG-CFS-D
G-1
Nov 8 22:32:18 hq-nps1 cmcld[2517]: Package SG-CFS-DG-2 cannot run on this node because the following dependencies are not running: SG-CFS-p
kg
Nov 8 22:32:18 hq-nps1 cmcld[2517]: Package SG-CFS-MP-2 cannot run on this node because the following dependencies are not running: SG-CFS-D
G-2
Nov 8 22:32:18 hq-nps1 cmcld[2517]: Package SG-CFS-DG-3 cannot run on this node because the following dependencies are not running: SG-CFS-p
kg
Nov 8 22:32:18 hq-nps1 cmcld[2517]: Package SG-CFS-MP-3 cannot run on this node because the following dependencies are not running: SG-CFS-D
G-3
Nov 8 22:32:18 hq-nps1 cmcld[2517]: Package SG-CFS-DG-4 cannot run on this node because the following dependencies are not running: SG-CFS-p
kg
Nov 8 22:32:18 hq-nps1 cmcld[2517]: Package SG-CFS-MP-4 cannot run on this node because the following dependencies are not running: SG-CFS-D
G-4
Nov 8 22:32:18 hq-nps1 cmcld[2517]: Package SG-CFS-DG-5 cannot run on this node because the following dependencies are not running: SG-CFS-p
kg
Nov 8 22:32:18 hq-nps1 cmcld[2517]: Package SG-CFS-MP-5 cannot run on this node because the following dependencies are not running: SG-CFS-D
G-5
Nov 8 22:32:18 hq-nps1 cmcld[2517]: Request from node hq-nps1 to start package SG-CFS-pkg on node hq-nps1.
Nov 8 22:32:18 hq-nps1 cmcld[2517]: Executing '/etc/cmcluster/cfs/SG-CFS-pkg.sh start' for package SG-CFS-pkg, as service PKG*23553.
Nov 8 22:32:18 hq-nps1 cmcld[2517]: Sending file $SGCONF/cmclconfig (45008 bytes) to file assistant daemon.
Nov 8 22:32:18 hq-nps1 cmclconfd[2534]: Updated file /etc/cmcluster/cmclconfig.tmp for node hq-nps1 (length = 90016).
Nov 8 22:32:18 hq-nps1 cmclconfd[2534]: Updated file /etc/cmcluster/cmclconfig.tmp for node hq-nps1 (length = 0).
Nov 8 22:32:18 hq-nps1 cmfileassistd[2902]: Updated file /etc/cmcluster/cmclconfig (length = 45008).
Nov 8 22:32:18 hq-nps1 cmcld[2517]: Sending file $SGCONF/cmclconfig (45008 bytes) to file assistant daemon.
Nov 8 22:32:18 hq-nps1 cmcld[2517]: Sending file $SGCONF/cmclconfig (44236 bytes) to file assistant daemon.
Nov 8 22:32:18 hq-nps1 cmclconfd[2534]: Updated file /etc/cmcluster/cmclconfig.tmp for node hq-nps1 (length = 90016).
Nov 8 22:32:18 hq-nps1 cmfileassistd[2902]: Updated file /etc/cmcluster/cmclconfig (length = 44236).
Nov 8 22:32:18 hq-nps1 cmfileassistd[2902]: Updated file /etc/cmcluster/cmclconfig (length = 45008).
Nov 8 22:32:18 hq-nps1 cmclconfd[2923]: Error opening /etc/cmcluster/cmknowncmds
Nov 8 22:32:18 hq-nps1 cmclconfd[2534]: Updated file /etc/cmcluster/cmclconfig.tmp for node hq-nps1 (length = 0).
Nov 8 22:32:18 hq-nps1 cmclconfd[2929]: Error opening /etc/cmcluster/cmknowncmds
Nov 8 22:32:18 hq-nps1 cmclconfd[2936]: Error opening /etc/cmcluster/cmknowncmds
Nov 8 22:32:18 hq-nps1 cmcld[2517]: Sending file $SGCONF/cmclconfig (43464 bytes) to file assistant daemon.
Nov 8 22:32:19 hq-nps1 cmfileassistd[2902]: Updated file /etc/cmcluster/cmclconfig (length = 43464).
Nov 8 22:32:19 hq-nps1 cmclconfd[2955]: Error opening /etc/cmcluster/cmknowncmds
Nov 8 22:32:19 hq-nps1 cmclconfd[2534]: Updated file /etc/cmcluster/cmclconfig.tmp for node hq-nps1 (length = 88472).
@

Anbody,Please tell me how this issue happend ?

Re: Disk Failure

Mohamed Saleem_1 — Tue, 09 Nov 2010 09:31:01 GMT

Hi Mate..

From your above system logs what i can understand is....

1. you have 2 node cluster running atleast 2 packages. a) SG-CFS-DG-1 which depends on b) SG-CFS-pkg

2. You have Veritas file system instead of hpux lvm.

3. One of your disk failed. Path 0xbc1c0200
& 0xbc1c0300.

4. It looks like ur multipath is not configured or not working. So the server was not able to route the data to alternate path leading to the unavailability of the filesystem ( /dev/vx/dsk/ora-software/vol1 = /u01 ).

5. Also you have oracle RAC run Cluster ing. Because of the above FS missing, Oracle Ready service ( CRS )rebooted your node - hq-nps1 to maintain data integrity .

6. After reboot node hq-nps1 came up and both the packages were started.

Hope this helps..

Cheers,
Saleem

topic Disk Failure in Operating System - HP-UX

Disk Failure

Re: Disk Failure