HP-UX 11.00 took long time to boot and performance

[rizuane]

Hello,

I've recently been working with this very old HP UX server and im not to familiar with the Unix operating system.
Hopefully someone here can help me out or point me in the right direction to troubleshoot the problem...
We using the server for application (form & report server 6i). We have other server for database Oracle 9i.
Usually the boot was done in 20 minutes and I was able to telnet into the system and start the apache and reports then the application was work fine.
Since last month the performance of this HP-UX become very bad with many latency in the application and sometime hang or freeze and need to reboot....and when reboot took almost 4h or more hours to boot.

below are some information related to this server:

sena #uname -a
HP-UX sena B.11.00 U 9000/800 1530786577 unlimited-user license
============================================================================

Total Configured Memory : 4096 MB

sena #grep Physical /var/adm/syslog/syslog.log
Sep 9 09:21:46 sena vmunix: Physical: 4194304 Kbytes, lockable: 3832648 Kbytes, available: 3905684 Kbytes
============================================================================

Total disk i think 150 gb
============================================================================

The bundle in HP-UX:

sena #swlist -l bundle

A5230A B.11.00.05 100BT/9000 PCI
B2491BA B.11.00 MirrorDisk/UX
B5456CA C.01.18.04 HP-UX Development Kit for Java*
B5725AA B.3.7.95 HP-UX Installation Utilities (Ignite-UX)
B8110AA 1.2.2.08.00 Java 2 SDK for HP-UX (700/800), PA1.1 + PA2.0 Add On
B8111AA 1.2.2.08.00 Java 2 RTE for HP-UX (700/800), PA1.1 + PA2.0 Add On
B8342AA B.11.00.05 Netscape Communicator 4.75
B8724AA A.01.05 CIFS/9000 Client
B8725AA A.01.05 CIFS/9000 Server
B9098AA 1.2.2.08.00 Java 2 Plugin for HP-UX (700/800)
B9415AA 01.03.12.03.01 HP Apache-based Web Server with Strong (128bit) Encryption
HPUXEng64RT B.11.00.01 English HP-UX 64-bit Runtime Environment
HWE1100 B.11.00.0203.5 Hardware Enablement Patches for HP-UX 11.00, March 2002
Ignite-UX-11-00 B.3.7.95 HP-UX Installation Utilities for Installing 11.00 Systems
J2559C D.06.21 Hewlett-Packard JetAdmin for Unix Utility
OnlineDiag B.11.00.26.07 HPUX 11.00 Support Tools Bundle, Sep 2003
UXCoreMedia B.11.00.02 HP-UX Media Kit (Reference Only. See Description)
UnlimUserLic B.11.00.02 HP-UX Unlimited-User License
XSWGR1100 B.11.00.52.2 HP-UX General Release Patches, March 2001
XSWHWCR1100 B.11.00.52.3 HP-UX Hardware Enablement and Critical Patches, March 2001
============================================================================

The syslog (/var/adm/syslog/syslog.log) it's full of errors (almost like this: vmunix: SCSI: Reset detected -- lbolt: 615545, bus: 4).

Below im showing other errors in the syslog.log:

vmunix: SCSI: Reset detected -- lbolt: 615545, bus: 4
vmunix: scb->cdb: 12 00 00 00 80 00
vmunix: SCSI: Reset detected -- lbolt: 636410, bus: 4
vmunix: DIAGNOSTIC SYSTEM WARNING:
vmunix: The diagnostic logging facility is no longer receiving excessive
vmunix: errors from the I/O subsystem. 70 I/O error entries were lost.
vmunix: scb->cdb: 12 00 00 00 80 00
vmunix: I/O hardware probe timed out
vmunix: Last successful probe at path 0/4/0/0.13
vmunix: mpc_bindlwp: Overriding conflicting mandatory binding!
vmunix: mpc_bindlwp: Migrating process 1914 from processor 1 to processor 0!
above message repeats 159 times
vmunix: SCSI: Reset detected -- lbolt: 669592, bus: 4
vmunix: SCSI: Unexpected Disconnect -- lbolt: 670169, dev: cb04f002, io_id: 400062e
vmunix: scb->cdb: 12 00 00 00 80 00
vmunix: SCSI: Unexpected Disconnect -- lbolt: 130987780, dev: cb04f000, io_id: 40b5251
vmunix: scb->cdb: 12 00 00 00 80 00
vmunix: LVM: vg[1]: pvnum=0 (dev_t=0x1f040000) is POWERFAILED
vmunix: LVM: vg[1]: pvnum=1 (dev_t=0x1f041000) is POWERFAILED
vmunix: LVM: vg[1].lv[1] has 2 fault disks on : pvnum=0 (dev=0x1f040000) pvnum=1 (dev=0x1f041000)
vmunix: LVM: PV 1 has been returned to vg[2].
LVM[2692]: /sbin/vgsync /dev/vg03
LVM: PV 0 has been returned to vg[1].
vmunix: LVM: PV 1 has been returned to vg[1].
vmunix: LVM: Failed to automatically resync PV 1f043000 error: 5
vmunix: LVM: vg[2]: pvnum=0 (dev_t=0x1f042000) is POWERFAILED
vmunix: LVM: vg[2]: pvnum=1 (dev_t=0x1f043000) is POWERFAILED
vmunix: LVM: vg[3]: pvnum=0 (dev_t=0x1f04d000) is POWERFAILED
vmunix: DIAGNOSTIC SYSTEM WARNING:
vmunix: The diagnostic logging facility has started receiving excessive
vmunix: errors from the I/O subsystem. I/O error entries will be lost
vmunix: until the cause of the excessive I/O logging is corrected.
vmunix: If the diaglogd daemon is not active, use the Daemon Startup command
vmunix: in stm to start it.
vmunix: If the diaglogd daemon is active, use the logtool utility in stm
vmunix: to determine which I/O subsystem is logging excessive errors.
vmunix: LVM: vg[2].lv[1] has 2 fault disks on : pvnum=0 (dev=0x1f042000) pvnum=1 (dev=0x1f043000)
: su : + tty?? root-oracle9
vmunix: DIAGNOSTIC SYSTEM WARNING:
vmunix: The diagnostic logging facility is no longer receiving excessive
vmunix: LVM: vg[2].lv[1] has 2 fault disks on : pvnum=0 (dev=0x1f042000) pvnum=1 (dev=0x1f043000)
vmunix: errors from the I/O subsystem. 1 I/O error entries were lost.
vmunix: LVM: PV 0 has been returned to vg[1].
vmunix: LVM: PV 1 has been returned to vg[1].
vmunix: LVM: PV 0 has been returned to vg[2].
vmunix: LVM: PV 1 has been returned to vg[2].
vmunix: LVM: PV 0 has been returned to vg[3].
vmunix: LVM: PV 1 has been returned to vg[2].
vmunix: LVM: Failed to automatically resync PV 1f043000 error: 5
vmunix: LVM: PV 0 has been returned to vg[1].
vmunix: LVM: Failed to automatically resync PV 1f04d000 error: 5
vmunix: lv_readvgdats: Could not read VGDA 1 header & trailer from disk H/W path 0/4/0/0.3.0 (error = 5)
------------------

rpcbind: check_netconfig: Found CLTS loopback transport
rpcbind: check_netconfig: Found COTS loopback transport
rpcbind: check_netconfig: Found COTS ORD loopback transport
rpcbind: init_transport: check binding for udp
rpcbind: init_transport: check binding for tcp
rpcbind: init_transport: check binding for ticlts
rpcbind: svc_tli_create: t_getname failed; T_IDLE
rpcbind: svc_dg_create: could not get transport information
rpcbind: init_transport: Could not create rmtcall fd for ticlts
rpcbind: init_transport: check binding for ticotsord
rpcbind: init_transport: check binding for ticots
--------------------------

sena automount[745]: Automount abort!!!
automount[745]: Possible cause:
sena automount[745]:
automount[745]: 1)No map provided
automount[745]: 2)Using the same map as the running automount
automount[745]: 3)The maps contain invalid mount points
-------------------------------

/usr/sbin/envd[1257]: restart, and read the configuration file
--------------------------------

/usr/sbin/nfsd[1535]: Setting STREAMS-HEAD high water value to 65536.
/usr/sbin/nfsd[1540]: nfsd do_one mpctl succeeded: ncpus = 2.
/usr/sbin/nfsd[1540]: nfsd do_one pmap 2
/usr/sbin/nfsd[1540]: nfsd do_one pmap 3
/usr/sbin/nfsd[1540]: nfsd do_one bind 1
/usr/sbin/nfsd[1541]: nfsd do_one bind 0
/usr/sbin/nfsd[1540]: Return from t_optmgmt(XTI_DISTRIBUTE) 0
/usr/sbin/nfsd[1541]: Return from t_optmgmt(XTI_DISTRIBUTE) 0
/usr/sbin/nfsd[1545]: nfsd 1 0 sock 4
/usr/sbin/nfsd[1546]: nfsd 0 0 sock 4
/usr/sbin/nfsd[1541]: nfsd 0 1 sock 4
/usr/sbin/nfsd[1540]: nfsd 1 1 sock 4
------------------------------

EMS [1829]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/storage/events/enclosures/ses_enclosure/0_7_0_0.15.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 119865346 -r /storage/events/enclosures/ses_enclosure/0_7_0_0.15.0 -n 119865345 -a
EMS [1829]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/storage/events/enclosures/ses_enclosure/0_7_0_0.15.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 119865346 -r /storage/events/enclosures/ses_enclosure/0_7_0_0.15.0 -n 119865346 -a
EMS [1721]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/storage/events/disks/default/0_4_0_0.3.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 112787478 -r /storage/events/disks/default/0_4_0_0.3.0 -n 112787457 -a
EMS [1721]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/storage/events/disks/default/0_4_0_0.1.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 112787470 -r /storage/events/disks/default/0_4_0_0.1.0 -n 112787458 -a
EMS [1721]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/storage/events/disks/default/0_4_0_0.0.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 112787466 -r /storage/events/disks/default/0_4_0_0.0.0 -n 112787459 -a
EMS [1721]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/storage/events/disks/default/0_4_0_0.2.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 112787474 -r /storage/events/disks/default/0_4_0_0.2.0 -n 112787460 -a
============================================================================
_______________________________________________________________________________________
 Other info from the system:
_______________________________________________________________________________________

sena #bdf

Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 143360 84501 55244 60% /
/dev/vg00/lvol1 99669 35833 53869 40% /stand
/dev/vg00/lvol7 1142784 773724 347653 69% /var
/dev/vg00/lvol6 1216512 523934 649329 45% /usr
/dev/vg00/lvol5 307200 2279 286518 1% /tmp
/dev/vg01/lvol2 15675392 10353642 5162832 67% /oratbs2
/dev/vg02/lvol1 17772544 16133060 1618300 91% /oratbs1
/dev/vg00/lvol4 552960 513588 36927 93% /opt
/dev/vg00/lvol8 20480 9194 10661 46% /home
/dev/vg00/lvol9 5017600 4749937 251003 95% /developer
/dev/vg01/cemdemo 2097152 1153510 886923 57% /cemdemo
/dev/vg00/lvol10 1536000 1334293 190003 88% /cem
/dev/vg03/arcsenaprimary
10485760 579288 9596950 6% /arcsenaprimary
-----------------

sena #swapinfo -tam

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1536 0 1536 0% 0 - 1 /dev/vg00/lvol2
reserve - 888 -888
memory 3328 590 2738 18%
total 4864 1478 3386 30% - 0 -
-----------------

sena #cat /etc/fstab

/dev/vg00/lvol3 / vxfs delaylog 0 1
/dev/vg00/lvol1 /stand hfs defaults 0 1
/dev/vg00/lvol4 /opt vxfs delaylog 0 2
/dev/vg00/lvol5 /tmp vxfs delaylog 0 2
/dev/vg00/lvol6 /usr vxfs delaylog 0 2
/dev/vg00/lvol7 /var vxfs delaylog 0 2
/dev/vg00/lvol8 /home vxfs delaylog 0 2
/dev/vg00/lvol9 /developer vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
/dev/vg02/lvol1 /oratbs1 vxfs delaylog,nodatainlog,largefiles,rw,suid 0 2
/dev/vg01/lvol2 /oratbs2 vxfs delaylog,nodatainlog,largefiles,rw,suid 0 2
/dev/vg00/lvol10 /cem vxfs rw,suid,nolargefiles,delaylog,nodatainlog 0 2
/dev/vg01/cemdemo /cemdemo vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
/dev/vg03/arcsenaprimary /arcsenaprimary vxfs rw,suid,largefiles,delaylog,datainlog 0 2
-----------------

sena #lvlnboot -v

Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c1t2d0 (0/0/1/1.2.0) -- Boot Disk
/dev/dsk/c2t2d0 (0/0/2/0.2.0) -- Boot Disk
Boot: lvol1 on: /dev/dsk/c1t2d0
/dev/dsk/c2t2d0
Root: lvol3 on: /dev/dsk/c1t2d0
/dev/dsk/c2t2d0
Swap: lvol2 on: /dev/dsk/c1t2d0
/dev/dsk/c2t2d0
Dump: lvol2 on: /dev/dsk/c1t2d0, 0

lvlnboot: LIF information corrupt or not present on "/dev/dsk/c5t2d0".
Use the "mkboot" command to initialize the LIF area.
lvlnboot: LIF information corrupt or not present on "/dev/dsk/c5t3d0".
Use the "mkboot" command to initialize the LIF area.
Boot Definitions for Volume Group /dev/vg02:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c4t2d0 (0/4/0/0.2.0)
/dev/dsk/c4t3d0 (0/4/0/0.3.0)
/dev/dsk/c5t2d0 (0/7/0/0.2.0) -- Boot Disk
/dev/dsk/c5t3d0 (0/7/0/0.3.0) -- Boot Disk
No Boot Logical Volume configured
No Root Logical Volume configured
No Swap Logical Volume configured
No Dump Logical Volume configured
------------------------

sena #vgdisplay -v

--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 10
Open LV 10
Max PV 16
Cur PV 2
Act PV 2
Max PE per PV 2500
VGDA 4
PE Size (Mbytes) 4
Total PE 4338
Alloc PE 4070
Free PE 268
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg00/lvol1
LV Status available/syncd
LV Size (Mbytes) 100
Current LE 25
Allocated PE 50
Used PV 2

LV Name /dev/vg00/lvol2
LV Status available/syncd
LV Size (Mbytes) 1536
Current LE 384
Allocated PE 768
Used PV 2

LV Name /dev/vg00/lvol3
LV Status available/syncd
LV Size (Mbytes) 140
Current LE 35
Allocated PE 70
Used PV 2

LV Name /dev/vg00/lvol4
LV Status available/syncd
LV Size (Mbytes) 540
Current LE 135
Allocated PE 270
Used PV 2

LV Name /dev/vg00/lvol5
LV Status available/syncd
LV Size (Mbytes) 300
Current LE 75
Allocated PE 150
Used PV 2

LV Name /dev/vg00/lvol6
LV Status available/syncd
LV Size (Mbytes) 1188
Current LE 297
Allocated PE 594
Used PV 2

LV Name /dev/vg00/lvol7
LV Status available/syncd
LV Size (Mbytes) 1116
Current LE 279
Allocated PE 558
Used PV 2

LV Name /dev/vg00/lvol8
LV Status available/syncd
LV Size (Mbytes) 20
Current LE 5
Allocated PE 10
Used PV 2

LV Name /dev/vg00/lvol9
LV Status available/syncd
LV Size (Mbytes) 4900
Current LE 1225
Allocated PE 1225
Used PV 2

LV Name /dev/vg00/lvol10
LV Status available/syncd
LV Size (Mbytes) 1500
Current LE 375
Allocated PE 375
Used PV 2

--- Physical volumes ---
PV Name /dev/dsk/c1t2d0
PV Status available
Total PE 2169
Free PE 0

PV Name /dev/dsk/c2t2d0
PV Status available
Total PE 2169
Free PE 268

VG Name /dev/vg01
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 2
Open LV 2
Max PV 16
Cur PV 4
Act PV 4
Max PE per PV 2171
VGDA 8
PE Size (Mbytes) 4
Total PE 8680
Alloc PE 8678
Free PE 2
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg01/cemdemo
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 512
Allocated PE 1024
Used PV 3

LV Name /dev/vg01/lvol2
LV Status available/syncd
LV Size (Mbytes) 15308
Current LE 3827
Allocated PE 7654
Used PV 4

--- Physical volumes ---
PV Name /dev/dsk/c4t0d0
PV Status available
Total PE 2170
Free PE 0

PV Name /dev/dsk/c4t1d0
PV Status available
Total PE 2170
Free PE 0

PV Name /dev/dsk/c5t0d0
PV Status available
Total PE 2170
Free PE 0

PV Name /dev/dsk/c5t1d0
PV Status available
Total PE 2170
Free PE 2

VG Name /dev/vg02
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 1
Open LV 1
Max PV 16
Cur PV 4
Act PV 4
Max PE per PV 2171
VGDA 8
PE Size (Mbytes) 4
Total PE 8678
Alloc PE 8678
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg02/lvol1
LV Status available/stale
LV Size (Mbytes) 17356
Current LE 4339
Allocated PE 8678
Used PV 4

--- Physical volumes ---
PV Name /dev/dsk/c4t2d0
PV Status available
Total PE 2170
Free PE 0

PV Name /dev/dsk/c4t3d0
PV Status available
Total PE 2170
Free PE 0

PV Name /dev/dsk/c5t2d0
PV Status available
Total PE 2169
Free PE 0

PV Name /dev/dsk/c5t3d0
PV Status available
Total PE 2169
Free PE 0

VG Name /dev/vg03
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 1
Open LV 1
Max PV 16
Cur PV 2
Act PV 2
Max PE per PV 8683
VGDA 4
PE Size (Mbytes) 4
Total PE 17362
Alloc PE 5120
Free PE 12242
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg03/arcsenaprimary
LV Status available/stale
LV Size (Mbytes) 10240
Current LE 2560
Allocated PE 5120
Used PV 2

--- Physical volumes ---
PV Name /dev/dsk/c4t13d0
PV Status available
Total PE 8681
Free PE 6121

PV Name /dev/dsk/c5t13d0
PV Status available
Total PE 8681
Free PE 6121
------------------------------------

sena #ioscan -fnkCdisk

Class I H/W Path Driver S/W State H/W Type Description
======================================================================
disk 0 0/0/1/1.2.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
disk 1 0/0/2/0.2.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c2t2d0 /dev/rdsk/c2t2d0
disk 2 0/0/2/1.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 305
/dev/dsk/c3t2d0 /dev/dsk/cdrom /dev/rdsk/c3t2d0
disk 3 0/4/0/0.0.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c4t0d0 /dev/rdsk/c4t0d0
disk 4 0/4/0/0.1.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c4t1d0 /dev/rdsk/c4t1d0
disk 5 0/4/0/0.2.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c4t2d0 /dev/rdsk/c4t2d0
disk 6 0/4/0/0.3.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c4t3d0 /dev/rdsk/c4t3d0
disk 19 0/4/0/0.13.0 sdisk CLAIMED DEVICE SEAGATE ST136403LC
/dev/dsk/c4t13d0 /dev/rdsk/c4t13d0
disk 11 0/7/0/0.0.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c5t0d0 /dev/rdsk/c5t0d0
disk 12 0/7/0/0.1.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c5t1d0 /dev/rdsk/c5t1d0
disk 13 0/7/0/0.2.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c5t2d0 /dev/rdsk/c5t2d0
disk 14 0/7/0/0.3.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c5t3d0 /dev/rdsk/c5t3d0
disk 20 0/7/0/0.13.0 sdisk CLAIMED DEVICE SEAGATE ST136403LC
/dev/dsk/c5t13d0 /dev/rdsk/c5t13d0
-----------------

sena #dmesg

Feb 20 12:04
...
etected -- lbolt: 138432670, bus: 4
SCSI: Reset detected -- lbolt: 138433006, bus: 4
SCSI: Reset detected -- lbolt: 138433400, bus: 4
SCSI: Reset detected -- lbolt: 138436567, bus: 4
SCSI: Unexpected Disconnect -- lbolt: 138437109, dev: cb04f000, io_id: 40c1acc
scb->cdb: 12 00 00 00 80 00
SCSI: Reset detected -- lbolt: 138437466, bus: 4
SCSI: Reset detected -- lbolt: 138437804, bus: 4
SCSI: Reset detected -- lbolt: 138438150, bus: 4
SCSI: Reset detected -- lbolt: 138438490, bus: 4
SCSI: Reset detected -- lbolt: 138438899, bus: 4
SCSI: Reset detected -- lbolt: 138439320, bus: 4
SCSI: Reset detected -- lbolt: 138444705, bus: 4
SCSI: Reset detected -- lbolt: 138445083, bus: 4
SCSI: Reset detected -- lbolt: 138445495, bus: 4
SCSI: Reset detected -- lbolt: 138445872, bus: 4
SCSI: Unexpected Disconnect -- lbolt: 138446414, dev: cb04f000, io_id: 40c1acc
scb->cdb: 12 00 00 00 80 00
SCSI: Reset detected -- lbolt: 138446751, bus: 4
SCSI: Reset detected -- lbolt: 138447101, bus: 4
SCSI: Reset detected -- lbolt: 138447497, bus: 4
SCSI: Reset detected -- lbolt: 138447830, bus: 4
SCSI: Reset detected -- lbolt: 138448162, bus: 4
SCSI: Reset detected -- lbolt: 138448493, bus: 4
SCSI: Reset detected -- lbolt: 138448823, bus: 4
SCSI: Reset detected -- lbolt: 138449160, bus: 4
SCSI: Reset detected -- lbolt: 138455825, bus: 4
SCSI: Reset detected -- lbolt: 138456155, bus: 4
SCSI: Reset detected -- lbolt: 138456487, bus: 4
SCSI: Reset detected -- lbolt: 138456817, bus: 4
SCSI: Reset detected -- lbolt: 138460879, bus: 4

[Torsten.]

There is something wrong in the SCSI system, either a bad component or a bad disk.

Consider to start here:

EMS [1829]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/storage/events/enclosures/ses_enclosure/0_7_0_0.15.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 119865346 -r /storage/events/enclosures/ses_enclosure/0_7_0_0.15.0 -n 119865345 -a
EMS [1829]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/storage/events/enclosures/ses_enclosure/0_7_0_0.15.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 119865346 -r /storage/events/enclosures/ses_enclosure/0_7_0_0.15.0 -n 119865346 -a
EMS [1721]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/storage/events/disks/default/0_4_0_0.3.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 112787478 -r /storage/events/disks/default/0_4_0_0.3.0 -n 112787457 -a
EMS [1721]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/storage/events/disks/default/0_4_0_0.1.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 112787470 -r /storage/events/disks/default/0_4_0_0.1.0 -n 112787458 -a
EMS [1721]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/storage/events/disks/default/0_4_0_0.0.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 112787466 -r /storage/events/disks/default/0_4_0_0.0.0 -n 112787459 -a
EMS [1721]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/storage/events/disks/default/0_4_0_0.2.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 112787474 -r /storage/events/disks/default/0_4_0_0.2.0 -n 112787460 -a

Either a single bad disk freezing the bus or another component:

vmunix: LVM: vg[2]: pvnum=0 (dev_t=0x1f042000) is POWERFAILED
vmunix: LVM: vg[2]: pvnum=1 (dev_t=0x1f043000) is POWERFAILED
vmunix: LVM: vg[3]: pvnum=0 (dev_t=0x1f04d000) is POWERFAILED

Consider to post a full

ioscan -fn

[rizuane]

Thank you for your attention, help and reply.

Here is the full ioscan -fn:

sena #ioscan -fn
Class I H/W Path Driver S/W State H/W Type Description
============================================================================
root 0 root CLAIMED BUS_NEXUS
ioa 0 0 sba CLAIMED BUS_NEXUS System Bus Adapter (582)
ba 0 0/0 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
lan 0 0/0/0/0 btlan3 CLAIMED INTERFACE PCI Ethernet (10110019)
/dev/diag/lan0 /dev/ether0
ext_bus 0 0/0/1/0 c720 CLAIMED INTERFACE SCSI C896 Fast Wide Single-Ended
target 0 0/0/1/0.3 tgt CLAIMED DEVICE
tape 0 0/0/1/0.3.0 stape CLAIMED DEVICE HP C1537A
/dev/rmt/0m /dev/rmt/0mnb /dev/rmt/c0t3d0BESTn /dev/rmt/c0t3d0DDSb
/dev/rmt/0mb /dev/rmt/c0t3d0BEST /dev/rmt/c0t3d0BESTnb /dev/rmt/c0t3d0DDSn
/dev/rmt/0mn /dev/rmt/c0t3d0BESTb /dev/rmt/c0t3d0DDS /dev/rmt/c0t3d0DDSnb
target 1 0/0/1/0.7 tgt CLAIMED DEVICE
ctl 0 0/0/1/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c0t7d0
ext_bus 1 0/0/1/1 c720 CLAIMED INTERFACE SCSI C896 Ultra Wide Single-Ended
target 2 0/0/1/1.2 tgt CLAIMED DEVICE
disk 0 0/0/1/1.2.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
target 3 0/0/1/1.7 tgt CLAIMED DEVICE
ctl 1 0/0/1/1.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c1t7d0
ext_bus 2 0/0/2/0 c720 CLAIMED INTERFACE SCSI C875 Ultra Wide Single-Ended
target 4 0/0/2/0.2 tgt CLAIMED DEVICE
disk 1 0/0/2/0.2.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c2t2d0 /dev/rdsk/c2t2d0
target 5 0/0/2/0.7 tgt CLAIMED DEVICE
ctl 2 0/0/2/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c2t7d0
ext_bus 3 0/0/2/1 c720 CLAIMED INTERFACE SCSI C875 Ultra Wide Single-Ended
target 6 0/0/2/1.2 tgt CLAIMED DEVICE
disk 2 0/0/2/1.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 305
/dev/dsk/c3t2d0 /dev/dsk/cdrom /dev/rdsk/c3t2d0
target 7 0/0/2/1.7 tgt CLAIMED DEVICE
ctl 3 0/0/2/1.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c3t7d0
tty 0 0/0/4/0 asio0 CLAIMED INTERFACE PCI Serial (103c1048)
/dev/GSPdiag1 /dev/mux0 /dev/tty0p1 /dev/tty1p0
/dev/diag/mux0 /dev/tty0p0 /dev/tty0p2
tty 1 0/0/5/0 asio0 CLAIMED INTERFACE PCI Serial (103c1048)
/dev/GSPdiag2 /dev/diag/mux1 /dev/mux1 /dev/tty1p1
ba 1 0/1 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 2 0/2 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 3 0/3 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
lan 1 0/3/0/0 btlan5 CLAIMED INTERFACE HP A5230A/B5509BA PCI 10/100Base-TX Addon
/dev/diag/lan1 /dev/ether1 /dev/lan1
ba 4 0/4 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ext_bus 4 0/4/0/0 c720 CLAIMED INTERFACE SCSI C895 Ultra2 Wide LVD
target 19 0/4/0/0.0 tgt CLAIMED DEVICE
disk 3 0/4/0/0.0.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c4t0d0 /dev/rdsk/c4t0d0
target 20 0/4/0/0.1 tgt CLAIMED DEVICE
disk 4 0/4/0/0.1.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c4t1d0 /dev/rdsk/c4t1d0
target 21 0/4/0/0.2 tgt CLAIMED DEVICE
disk 5 0/4/0/0.2.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c4t2d0 /dev/rdsk/c4t2d0
target 8 0/4/0/0.3 tgt CLAIMED DEVICE
disk 6 0/4/0/0.3.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c4t3d0 /dev/rdsk/c4t3d0
target 9 0/4/0/0.7 tgt CLAIMED DEVICE
ctl 4 0/4/0/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c4t7d0
target 10 0/4/0/0.13 tgt CLAIMED DEVICE
disk 19 0/4/0/0.13.0 sdisk CLAIMED DEVICE SEAGATE ST136403LC
/dev/dsk/c4t13d0 /dev/rdsk/c4t13d0
target 11 0/4/0/0.15 tgt CLAIMED DEVICE
ctl 6 0/4/0/0.15.0 sctl CLAIMED DEVICE HP A5272A
/dev/rscsi/c4t15d0
ba 5 0/5 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 6 0/6 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 7 0/7 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ext_bus 5 0/7/0/0 c720 CLAIMED INTERFACE SCSI C895 Ultra2 Wide LVD
target 12 0/7/0/0.0 tgt CLAIMED DEVICE
disk 11 0/7/0/0.0.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c5t0d0 /dev/rdsk/c5t0d0
target 13 0/7/0/0.1 tgt CLAIMED DEVICE
disk 12 0/7/0/0.1.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c5t1d0 /dev/rdsk/c5t1d0
target 14 0/7/0/0.2 tgt CLAIMED DEVICE
disk 13 0/7/0/0.2.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c5t2d0 /dev/rdsk/c5t2d0
target 15 0/7/0/0.3 tgt CLAIMED DEVICE
disk 14 0/7/0/0.3.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c5t3d0 /dev/rdsk/c5t3d0
target 16 0/7/0/0.7 tgt CLAIMED DEVICE
ctl 7 0/7/0/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c5t7d0
target 17 0/7/0/0.13 tgt CLAIMED DEVICE
disk 20 0/7/0/0.13.0 sdisk CLAIMED DEVICE SEAGATE ST136403LC
/dev/dsk/c5t13d0 /dev/rdsk/c5t13d0
target 18 0/7/0/0.15 tgt CLAIMED DEVICE
ctl 9 0/7/0/0.15.0 sctl CLAIMED DEVICE HP A5272A
/dev/rscsi/c5t15d0
memory 0 8 memory CLAIMED MEMORY Memory
processor 0 160 processor CLAIMED PROCESSOR Processor
processor 1 166 processor CLAIMED PROCESSOR Processor

[Bill Hassell]

You have disk failures.
In addition, your system may never boot up again. lvlnboot shows corrupted boot areas on both disks:

lvlnboot: LIF information corrupt or not present on "/dev/dsk/c5t2d0".
lvlnboot: LIF information corrupt or not present on "/dev/dsk/c5t3d0".

You'll need to call HPE  for a hardware service call. 
If you don't have anyone familiar with HP-UX, you'll need see if HPE can also provide consulting to get the new disks configured for your system and fix any other problems.

[rizuane]

Hi Hassel,

I think the boot have mirror in  /dev/vg00/lvol1 

/dev/dsk/c1t2d0 (0/0/1/1.2.0) -- Boot Disk
/dev/dsk/c2t2d0 (0/0/2/0.2.0) -- Boot Disk

The corrupted boot areas on other disks is:

/dev/dsk/c5t2d0 (0/7/0/0.2.0) -- Boot Disk
/dev/dsk/c5t3d0 (0/7/0/0.3.0) -- Boot Disk

How can i sort this boot disk since i have mirror in other disk ???

sena #lvlnboot -v

Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c1t2d0 (0/0/1/1.2.0) -- Boot Disk
/dev/dsk/c2t2d0 (0/0/2/0.2.0) -- Boot Disk
Boot: lvol1 on: /dev/dsk/c1t2d0
                     /dev/dsk/c2t2d0
Root: lvol3 on: /dev/dsk/c1t2d0
                      /dev/dsk/c2t2d0
Swap: lvol2 on: /dev/dsk/c1t2d0
                       /dev/dsk/c2t2d0
Dump: lvol2 on: /dev/dsk/c1t2d0, 0

lvlnboot: LIF information corrupt or not present on "/dev/dsk/c5t2d0".
Use the "mkboot" command to initialize the LIF area.
lvlnboot: LIF information corrupt or not present on "/dev/dsk/c5t3d0".
Use the "mkboot" command to initialize the LIF area.

Boot Definitions for Volume Group /dev/vg02:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c4t2d0 (0/4/0/0.2.0)
/dev/dsk/c4t3d0 (0/4/0/0.3.0)
/dev/dsk/c5t2d0 (0/7/0/0.2.0) -- Boot Disk
/dev/dsk/c5t3d0 (0/7/0/0.3.0) -- Boot Disk
No Boot Logical Volume configured
No Root Logical Volume configured
No Swap Logical Volume configured
No Dump Logical Volume configured

[Bill Hassell]

Based on the output from lvlnboot, you have what appears to be a bootable disk set:

/dev/dsk/c1t2d0 (0/0/1/1.2.0) -- Boot Disk
/dev/dsk/c2t2d0 (0/0/2/0.2.0) -- Boot Disk

You also have two disks that were partially configured, probably by mistake):

/dev/dsk/c1t2d0 (0/0/1/1.2.0) -- Boot Disk
/dev/dsk/c2t2d0 (0/0/2/0.2.0) -- Boot Disk

But as mentioned before, the SCSI lbolt/timeout errors are disk failures that must be fixed. You'll a service call from HPE to repair the issues.