ARMServer cannot find 12H

[Tim Krego_2]

I have a K460 running 11i with an AutoRAID 12H connected over SCSI controllers X/Y. The 12H has (12) 9GB hard drives installed. I'm pretty sure the firmware is current version HP62.

I'm getting an error in syslog with MAJORWARNING. When I type the output of the command I get the info below.

I tried restarting the armmon and ARMServer processes with no luck. The arraydsp -i command shows no know arrays.

I also included the output of ioscan -fun below as it does not look right. I'm going to check the cables and termination but that is going to require scheduling downtime.





ARCHIVED MONITOR DATA:

Event Time..........: Fri Dec 9 07:40:30 2005
Severity............: MAJORWARNING
Monitor.............: armmon
Event #.............: 4
System..............: mozzer.healthnetstaffing.com

Summary:
Disk Array at hardware path :
The computer can not connect to the ARMServer. The status on the disk
array is not currently available.




Description of Error:


This event message is displayed when a connection cannot be made to
the ARMServer. The resource monitor will continue to attempt to
connect to the ARMServer at regular polling intervals. When the
ARMServer is restored, the connection will succeed and normal
operation will continue. Until the ARMServer is operating normally,
it is not possible to obtain the current status of any of the disk
arrays.

Possible Causes Include:
1. The ARMServer is not running.
2. The wrong version of the ARMServer is running.
3. The correct ARMServer is running, but is hung.



Probable Cause / Recommended Action:


Restart ARMServer, or if still running, stop and restart the ARMServer.

Additional Event Data:
System IP Address...: 192.168.1.68
Event Id............: 0x43997b3e00000000
Monitor Version.....: B.01.01
Event Class.........: I/O
Client Configuration File...........:
/var/stm/config/tools/monitor/default_armmon.clcfg
Client Configuration File Version...: A.01.01
Qualification criteria met.
Number of events..: 1
Associated OS error log entry id(s):
None
Additional System Data:
System Model Number.............: 9000/889
EMS Version.....................: A.04.00
STM Version.....................: A.47.00
Latest information on this event:
http://docs.hp.com/hpux/content/hardware/ems/armmon.htm#4

v-v-v-v-v-v-v-v-v-v-v-v-v D E T A I L S v-v-v-v-v-v-v-v-v-v-v-v-v



Product/Device Data:

Logger ID.........: (not available/applicable)
Product Identifier: (not available/applicable)
Product Qualifier.: (not available/applicable)
SCSI Target ID....: (not available/applicable)
SCSI LUN..........: (not available/applicable)

# ioscan -fun

Class I H/W Path Driver S/W State H/W Type Description
======================================================================
ext_bus 0 8/8 c720 CLAIMED INTERFACE GSC add-on Fast/Wide SCSI Interface
disk 9 8/8.1.0 sdisk CLAIMED DEVICE HP C5447A
disk 10 8/8.1.1 sdisk CLAIMED DEVICE HP C5447A
ctl 0 8/8.6.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c0t6d0
ext_bus 1 8/12 c720 CLAIMED INTERFACE GSC add-on Fast/Wide SCSI Interface
ctl 1 8/12.0.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c1t0d0
disk 2 8/12.1.0 sdisk CLAIMED DEVICE HP C5447A
/dev/dsk/c1t1d0 /dev/rdsk/c1t1d0
disk 3 8/12.1.1 sdisk CLAIMED DEVICE HP C5447A
/dev/dsk/c1t1d1 /dev/rdsk/c1t1d1
disk 11 8/12.2.0 sdisk CLAIMED DEVICE HP C5447A
disk 12 8/12.2.1 sdisk CLAIMED DEVICE HP C5447A
disk 13 8/12.3.0 sdisk CLAIMED DEVICE HP C5447A
disk 14 8/12.3.1 sdisk CLAIMED DEVICE HP C5447A
disk 15 8/12.4.0 sdisk CLAIMED DEVICE HP C5447A
disk 16 8/12.4.1 sdisk CLAIMED DEVICE HP C5447A
disk 17 8/12.5.0 sdisk CLAIMED DEVICE HP C5447A
disk 18 8/12.5.1 sdisk CLAIMED DEVICE HP C5447A
disk 19 8/12.6.0 sdisk CLAIMED DEVICE HP C5447A
disk 20 8/12.6.1 sdisk CLAIMED DEVICE HP C5447A
disk 21 8/12.7.0 sdisk CLAIMED DEVICE HP C5447A
disk 22 8/12.7.1 sdisk CLAIMED DEVICE HP C5447A
disk 23 8/12.8.0 sdisk CLAIMED DEVICE HP C5447A
disk 24 8/12.8.1 sdisk CLAIMED DEVICE HP C5447A
disk 25 8/12.9.0 sdisk CLAIMED DEVICE HP C5447A
disk 26 8/12.9.1 sdisk CLAIMED DEVICE HP C5447A
disk 27 8/12.10.0 sdisk CLAIMED DEVICE HP C5447A
disk 28 8/12.10.1 sdisk CLAIMED DEVICE HP C5447A
disk 29 8/12.11.0 sdisk CLAIMED DEVICE HP C5447A
disk 30 8/12.11.1 sdisk CLAIMED DEVICE HP C5447A
disk 31 8/12.12.0 sdisk CLAIMED DEVICE HP C5447A
disk 32 8/12.12.1 sdisk CLAIMED DEVICE HP C5447A
disk 33 8/12.13.0 sdisk CLAIMED DEVICE HP C5447A
disk 34 8/12.13.1 sdisk CLAIMED DEVICE HP C5447A
disk 35 8/12.14.0 sdisk CLAIMED DEVICE HP C5447A
disk 36 8/12.14.1 sdisk CLAIMED DEVICE HP C5447A
disk 37 8/12.15.0 sdisk CLAIMED DEVICE HP C5447A
disk 38 8/12.15.1 sdisk CLAIMED DEVICE HP C5447A
ext_bus 2 10/0 c720 CLAIMED INTERFACE GSC built-in Fast/Wide SCSI Interface
tape 1 10/0.2.0 stape CLAIMED DEVICE QUANTUM DLT7000
/dev/rmt/1m /dev/rmt/1mn /dev/rmt/c2t2d0BEST /dev/rmt/c2t2d0BESTn
/dev/rmt/1mb /dev/rmt/1mnb /dev/rmt/c2t2d0BESTb /dev/rmt/c2t2d0BESTnb
disk 4 10/0.3.0 sdisk CLAIMED DEVICE SEAGATE ST34371W
/dev/dsk/c2t3d0 /dev/rdsk/c2t3d0
disk 5 10/0.4.0 sdisk CLAIMED DEVICE SEAGATE ST34371W
/dev/dsk/c2t4d0 /dev/rdsk/c2t4d0
disk 6 10/0.5.0 sdisk CLAIMED DEVICE SEAGATE ST34371W
/dev/dsk/c2t5d0 /dev/rdsk/c2t5d0
disk 7 10/0.6.0 sdisk CLAIMED DEVICE SEAGATE ST34371W
/dev/dsk/c2t6d0 /dev/rdsk/c2t6d0
ctl 2 10/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c2t7d0
tty 0 10/4/0 mux2 CLAIMED INTERFACE MUX
/dev/diag/mux0 /dev/diag/tty0p1 /dev/mux0 /dev/tty0p1
/dev/diag/tty0p0 /dev/diag/tty0p7 /dev/tty0p0 /dev/tty0p7
ext_bus 4 10/12/0 CentIf CLAIMED INTERFACE Built-in Parallel Interface
/dev/c4t0d0_lp
ext_bus 3 10/12/5 c720 CLAIMED INTERFACE Built-in SCSI
tape 0 10/12/5.0.0 stape CLAIMED DEVICE HP C1533A
/dev/rmt/0m /dev/rmt/0mnb /dev/rmt/c3t0d0BESTn /dev/rmt/c3t0d0DDSb
/dev/rmt/0mb /dev/rmt/c3t0d0BEST /dev/rmt/c3t0d0BESTnb /dev/rmt/c3t0d0DDSn
/dev/rmt/0mn /dev/rmt/c3t0d0BESTb /dev/rmt/c3t0d0DDS /dev/rmt/c3t0d0DDSnb
disk 8 10/12/5.2.0 sdisk CLAIMED DEVICE TOSHIBA CD-ROM XM-5401TA
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
ctl 3 10/12/5.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c3t7d0
lan 1 10/12/6 lan2 CLAIMED INTERFACE Built-in LAN
/dev/diag/lan1 /dev/ether1
ps2 0 10/12/7 ps2 CLAIMED INTERFACE Built-in Keyboard/Mouse
/dev/ps2_0 /dev/ps2_1 /dev/ps2kbd /dev/ps2mouse

[Sameer_Nirmal]

Hi,

Looking at the EMS data, there is no device file shown for the disk array. This is the reason the "arreydsp -i" command is couldn't able to communicate with array controller. In "ioscan" , the device files for the controllers are missing as well.

Does something happened recently , any changes with the setup etc???
Do you see any error on the front panel of the array?
Check for SCSI connections , cables, termination , SCSI IDs of the host HBAs and array controllers for ID conflicts.
Check if you see the respective devices files under /dev/rdsk and /dev/dsk for LUNs.

What you see in
/var/adm/syslog/syslog.log , dmesg?
Run following commands in order and post outputs.
# ioscan -fn
# ps -ef |grep ARM
# arraydsp -R
# arraydsp -i
# /usr/sbin/hpC2400/arrayscan
Post contents of /etc/hpC2400/HPARRAY.INFO file.
# ll /dev/diag/diag0
# lssf /dev/diag/diag0
# logprint -d /var/opt/hparray/log -a

[Tim Krego_2]

I checked the syslog.log and this appears to be happening after a reboot on 11/22/05 in the early morning.

This is our production box at a remote hosting facility so I'm concerned about doing anything remotely. The hosting colo is only 10 minutes away so I can make a trip there if necessary. I would need to schedule downtime as it is used 24/7.

I do have a test sandbox that is identical so I've been able to try the commands you listed there first.

# ioscan -fn

Class I H/W Path Driver S/W State H/W Type Description
========================================================================
bc 0 root CLAIMED BUS_NEXUS
bc 1 8 ccio CLAIMED BUS_NEXUS I/O Adapter
ext_bus 0 8/8 c720 CLAIMED INTERFACE GSC add-on Fast/Wide SCSI Interface
target 0 8/8.1 tgt CLAIMED DEVICE
disk 9 8/8.1.0 sdisk CLAIMED DEVICE HP C5447A
disk 10 8/8.1.1 sdisk CLAIMED DEVICE HP C5447A
target 1 8/8.6 tgt CLAIMED DEVICE
ctl 0 8/8.6.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c0t6d0
ext_bus 1 8/12 c720 CLAIMED INTERFACE GSC add-on Fast/Wide SCSI Interface
target 2 8/12.0 tgt CLAIMED DEVICE
ctl 1 8/12.0.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c1t0d0
target 3 8/12.1 tgt CLAIMED DEVICE
disk 2 8/12.1.0 sdisk CLAIMED DEVICE HP C5447A
/dev/dsk/c1t1d0 /dev/rdsk/c1t1d0
disk 3 8/12.1.1 sdisk CLAIMED DEVICE HP C5447A
/dev/dsk/c1t1d1 /dev/rdsk/c1t1d1
target 4 8/12.2 tgt CLAIMED DEVICE
disk 11 8/12.2.0 sdisk CLAIMED DEVICE HP C5447A
disk 12 8/12.2.1 sdisk CLAIMED DEVICE HP C5447A
target 5 8/12.3 tgt CLAIMED DEVICE
disk 13 8/12.3.0 sdisk CLAIMED DEVICE HP C5447A
disk 14 8/12.3.1 sdisk CLAIMED DEVICE HP C5447A
target 6 8/12.4 tgt CLAIMED DEVICE
disk 15 8/12.4.0 sdisk CLAIMED DEVICE HP C5447A
disk 16 8/12.4.1 sdisk CLAIMED DEVICE HP C5447A
target 7 8/12.5 tgt CLAIMED DEVICE
disk 17 8/12.5.0 sdisk CLAIMED DEVICE HP C5447A
disk 18 8/12.5.1 sdisk CLAIMED DEVICE HP C5447A
target 8 8/12.6 tgt CLAIMED DEVICE
disk 19 8/12.6.0 sdisk CLAIMED DEVICE HP C5447A
disk 20 8/12.6.1 sdisk CLAIMED DEVICE HP C5447A
target 9 8/12.7 tgt CLAIMED DEVICE
disk 21 8/12.7.0 sdisk CLAIMED DEVICE HP C5447A
disk 22 8/12.7.1 sdisk CLAIMED DEVICE HP C5447A
target 10 8/12.8 tgt CLAIMED DEVICE
disk 23 8/12.8.0 sdisk CLAIMED DEVICE HP C5447A
disk 24 8/12.8.1 sdisk CLAIMED DEVICE HP C5447A
target 11 8/12.9 tgt CLAIMED DEVICE
disk 25 8/12.9.0 sdisk CLAIMED DEVICE HP C5447A
disk 26 8/12.9.1 sdisk CLAIMED DEVICE HP C5447A
target 12 8/12.10 tgt CLAIMED DEVICE
disk 27 8/12.10.0 sdisk CLAIMED DEVICE HP C5447A
disk 28 8/12.10.1 sdisk CLAIMED DEVICE HP C5447A
target 13 8/12.11 tgt CLAIMED DEVICE
disk 29 8/12.11.0 sdisk CLAIMED DEVICE HP C5447A
disk 30 8/12.11.1 sdisk CLAIMED DEVICE HP C5447A
target 14 8/12.12 tgt CLAIMED DEVICE
disk 31 8/12.12.0 sdisk CLAIMED DEVICE HP C5447A
disk 32 8/12.12.1 sdisk CLAIMED DEVICE HP C5447A
target 15 8/12.13 tgt CLAIMED DEVICE
disk 33 8/12.13.0 sdisk CLAIMED DEVICE HP C5447A
disk 34 8/12.13.1 sdisk CLAIMED DEVICE HP C5447A
target 16 8/12.14 tgt CLAIMED DEVICE
disk 35 8/12.14.0 sdisk CLAIMED DEVICE HP C5447A
disk 36 8/12.14.1 sdisk CLAIMED DEVICE HP C5447A
target 17 8/12.15 tgt CLAIMED DEVICE
disk 37 8/12.15.0 sdisk CLAIMED DEVICE HP C5447A
disk 38 8/12.15.1 sdisk CLAIMED DEVICE HP C5447A
bc 2 10 ccio CLAIMED BUS_NEXUS I/O Adapter
ext_bus 2 10/0 c720 CLAIMED INTERFACE GSC built-in Fast/Wide SCSI Interface
target 18 10/0.2 tgt CLAIMED DEVICE
tape 1 10/0.2.0 stape CLAIMED DEVICE QUANTUM DLT7000
/dev/rmt/1m /dev/rmt/1mn /dev/rmt/c2t2d0BEST /dev/rmt/c2t2d0BESTn
/dev/rmt/1mb /dev/rmt/1mnb /dev/rmt/c2t2d0BESTb /dev/rmt/c2t2d0BESTnb
target 19 10/0.3 tgt CLAIMED DEVICE
disk 4 10/0.3.0 sdisk CLAIMED DEVICE SEAGATE ST34371W
/dev/dsk/c2t3d0 /dev/rdsk/c2t3d0
target 20 10/0.4 tgt CLAIMED DEVICE
disk 5 10/0.4.0 sdisk CLAIMED DEVICE SEAGATE ST34371W
/dev/dsk/c2t4d0 /dev/rdsk/c2t4d0
target 21 10/0.5 tgt CLAIMED DEVICE
disk 6 10/0.5.0 sdisk CLAIMED DEVICE SEAGATE ST34371W
/dev/dsk/c2t5d0 /dev/rdsk/c2t5d0
target 22 10/0.6 tgt CLAIMED DEVICE
disk 7 10/0.6.0 sdisk CLAIMED DEVICE SEAGATE ST34371W
/dev/dsk/c2t6d0 /dev/rdsk/c2t6d0
target 23 10/0.7 tgt CLAIMED DEVICE
ctl 2 10/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c2t7d0
bc 3 10/4 bc CLAIMED BUS_NEXUS Bus Converter
tty 0 10/4/0 mux2 CLAIMED INTERFACE MUX
/dev/diag/mux0 /dev/diag/tty0p1 /dev/mux0 /dev/tty0p1
/dev/diag/tty0p0 /dev/diag/tty0p7 /dev/tty0p0 /dev/tty0p7
lan 0 10/4/8 btlan1 UNCLAIMED UNKNOWN HP HP-PB 100 Base TX card
ba 0 10/12 bus_adapter CLAIMED BUS_NEXUS Core I/O Adapter
ext_bus 4 10/12/0 CentIf CLAIMED INTERFACE Built-in Parallel Interface
/dev/c4t0d0_lp
ext_bus 3 10/12/5 c720 CLAIMED INTERFACE Built-in SCSI
target 24 10/12/5.0 tgt CLAIMED DEVICE
tape 0 10/12/5.0.0 stape CLAIMED DEVICE HP C1533A
/dev/rmt/0m /dev/rmt/0mnb /dev/rmt/c3t0d0BESTn /dev/rmt/c3t0d0DDSb
/dev/rmt/0mb /dev/rmt/c3t0d0BEST /dev/rmt/c3t0d0BESTnb /dev/rmt/c3t0d0DDSn
/dev/rmt/0mn /dev/rmt/c3t0d0BESTb /dev/rmt/c3t0d0DDS /dev/rmt/c3t0d0DDSnb
target 25 10/12/5.2 tgt CLAIMED DEVICE
disk 8 10/12/5.2.0 sdisk CLAIMED DEVICE TOSHIBA CD-ROM XM-5401TA
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
target 26 10/12/5.7 tgt CLAIMED DEVICE
ctl 3 10/12/5.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c3t7d0
lan 1 10/12/6 lan2 CLAIMED INTERFACE Built-in LAN
/dev/diag/lan1 /dev/ether1
ps2 0 10/12/7 ps2 CLAIMED INTERFACE Built-in Keyboard/Mouse
/dev/ps2_0 /dev/ps2_1 /dev/ps2kbd /dev/ps2mouse
processor 0 32 processor CLAIMED PROCESSOR Processor
processor 1 34 processor CLAIMED PROCESSOR Processor
processor 2 36 processor CLAIMED PROCESSOR Processor
processor 3 38 processor CLAIMED PROCESSOR Processor
memory 0 49 memory CLAIMED MEMORY Memory



# ps -ef | grep ARM

root 16568 1 1 18:30:14 ? 6:24 ARMServer
root 11330 7940 2 14:13:12 pts/1 0:00 grep ARM


# arraydsp -R
Rescanning for supported arrays ...

# arraydsp -i
Arrays known to the ARMServer:
No arrays were found by the ARMServer


# /usr/sbin/hpC2400/arrayscan
arrayscan: scanning I/O system for disk arrays...
arrayscan: Cannot stat device file. Please ensure all device files are installed

/etc/hpC2400/HPARRAY.INFO file contents

Mon Oct 24 16:29:47 EDT 2005
Killing disk array monitor daemon.
Killing /opt/hparray/bin/ARMServer process.

Wed Nov 2 11:07:15 EST 2005
arrayscan: scanning I/O system for disk arrays...

arrayscan: HP C5447A SCSI Disk Array at SCSI address [ 8/8.0.0 ]
arrayscan: HP C5447A SCSI Disk Array at SCSI address [ 8/8.0.1 ]
arrayscan: HP C5447A SCSI Disk Array at SCSI address [ 8/12.1.0 ]
arrayscan: HP C5447A SCSI Disk Array at SCSI address [ 8/12.1.1 ]

Starting disk array monitor daemon

Starting ARMServer daemon

Tue Nov 22 06:22:55 EST 2005
Killing disk array monitor daemon.
Killing /opt/hparray/bin/ARMServer process.

Tue Nov 22 07:15:20 EST 2005
arrayscan: scanning I/O system for disk arrays...

arrayscan: Cannot stat device file. Please ensure all device files are installed
SCAN FOR DISK ARRAY'S ENCOUNTERED UNEXPECTED ERRORS.
Take corrective action then execute "/usr/sbin/hpC2400/arrayscan" again.


Fri Dec 16 13:11:39 EST 2005
arrayscan: scanning I/O system for disk arrays...

arrayscan: Cannot stat device file. Please ensure all device files are installed
SCAN FOR DISK ARRAY'S ENCOUNTERED UNEXPECTED ERRORS.
Take corrective action then execute "/usr/sbin/hpC2400/arrayscan" again.


Fri Dec 16 13:12:22 EST 2005
Killing /opt/hparray/bin/ARMServer process.

Fri Dec 16 13:12:31 EST 2005
arrayscan: scanning I/O system for disk arrays...

arrayscan: Cannot stat device file. Please ensure all device files are installed
SCAN FOR DISK ARRAY'S ENCOUNTERED UNEXPECTED ERRORS.
Take corrective action then execute "/usr/sbin/hpC2400/arrayscan" again.

# ll /dev/diag/diag0
crw------ 1 bin bin 28 0x000080 Mar 23 2005 /dev/diag/diag0

# lssf /dev/diag/diag0
pseudo driver diag0 minor 0x000080 /dev/diag/diag0

I've attached a zip file of the output for logprint and the above files.

On 11/22/2005 at 7A we shutdown the K460/12H to mount into a rack. Maybe the cables are loose or maybe I've had a problem before that and just now found it. I admit I'm not very experienced with this stuff. It doesn't help that we don't have HP support.

Any help will be appreciated as I think I'm on borrowed time. Our Oracle 10g database files are on the 12H and I'm trying to fix this before we have a major outage.

[Sameer_Nirmal · Accepted solution]

Hi,

Well it is quite similar problem in your sandbox as well. The controller could not be communicated because there are no device files.
As the info log ( *SANDBOX* only ) shows , the last sucessful scan of the array was done on Nov 2 for 4 device paths and trouble started from Nov 22. You have 2 arrays right?

How those arrays are connected physically?
Verify the connections and through old "ioscan" output if you have any.
Now provided the connections are OK, you can create device file using
# insf -e -v

Moreover the "demsg" show SCSI bus reset errors pointing to HP C1533A DAT drive at h/w path c3t0d0. It may be normal if you are switching off the drive before making it offline etc.
Is it working OK? if not,how it is physically connected?

I doubt the problem could be physical connection.

It is good that you have sandbox to make it work before any troubleshoting attempt on production boxes. Make sure you make sandbox working first.

[Tim Krego_2]

Sameer,

Thanks for the help.

The output I posted was from the production box. I was using the sandbox to test your commands before trying on the production box.

The 12H is connected using two SCSI cables to the HSC SCSI channels.

I did the insf -eC disk and got back device files for the luns but I still have many more drives listed.

For instance in SAM I should have two 22500MB LUNS showing up. For some reason SAM shows that pair 12 times. I know that shouldn't be correct.

The DAT drive is internal on the K460 so it is alwasy powered up. I was trying to make a make_tape_recovery tape the last day or so and I haven't tried to verify it yet to see if it worked.

We have shut down the box a few times in November which may coincide with those dates in the files.

Now after insf I am able to see the array in ARMServer. I wonder if I need some patches. Though the sandox has a fresh install of 11.11 and no patches yet.

Any other suggestions?

[Sameer_Nirmal]

Tim,
I am glad I could put some pointers useful to you.

The most probable cause of those extra disks in the "ioscan" and "SAM" is SCSI Bus not terminated. In your case, the SCSI bus is
"ext_bus 1 8/12 c720 CLAIMED INTERFACE GSC add-on Fast/Wide SCSI Interface " as shown in ioscan. Make sure it is terminated correctly at the end.

The latest/last patch for ARMserver on HP-UX 11.11 is PHCO_23263 which I guess must be installed already. The controller firmware HP62 is latest. Maybe you can check the disks firmware now as ARMserver is working now. From OS side , you need to ensure you have latest patch bundles installed for OS which takes care of SCSI, RPC etc. patches related to such array.