lbolt value error message.

[Aamir14]

Hi,

Yesterday I found some error message in syslog file.

prodata2#tail -10000000 /var/adm/syslog/syslog.log |grep CRITICAL

Aug  6 20:08:37 prodata2 EMS [3457]: ------ EMS Event Notification ------   Value: "CRITICAL (5)" for Resource: "/adapters/events/raid_adapter/0_6_0_0_0_0_4_0_0_0"     (Threshold:  >= " 3")    Execute the following command to obtain event details:   /opt/resmon/bin/resdata -R 226557954 -r /adapters/events/raid_adapter/0_6_0_0_0_0_4_0_0_0 -n 226557953 -a
prodata2#date
Tue Aug  7 13:02:04 IST 2012

prodata2#/opt/resmon/bin/resdata -R 226557954 -r /adapters/events/raid_adapter/0_6_0_0_0_0_4_0_0_0 -n 226557953 -a

CURRENT MONITOR DATA:

Event Time..........: Mon Aug  6 20:08:37 2012
Severity............: CRITICAL
Monitor.............: dm_raid_adapter
Event #.............: 43
System..............: prodata2

Summary:
     Adapter at hardware path 0/6/0/0/0/0/4/0/0/0 : CISS: Logical Drive
     transitioned state.



Description of Error:


 lbolt value: 181737283

    CISS: Logical Drive Status:
    CISS: Logical Drive 0 transitioned from OK to INTERIM RECOVERY
    Drive Type : RAID 1+0 - Disk Mirroring


Probable Cause / Recommended Action:

Informational message.

Additional Event Data:
     System IP Address...: 192.168.4.6
     Event Id............: 0x501fd6ed00000000
     Monitor Version.....: B.01.00
     Event Class.........: I/O
     Client Configuration File...........:
     /var/stm/config/tools/monitor/default_dm_raid_adapter.clcfg
     Client Configuration File Version...: A.01.00
          Qualification criteria met.
               Number of events..: 1
     Associated OS error log entry id(s):
          0x501fd6ed00000001
     Additional System Data:
          System Model Number.............: ia64 hp server rx6600
          OS Version......................: B.11.31
          EMS Version.....................: A.04.20.31.02
          STM Version.....................: D.04.00
     Latest information on this event:
          http://docs.hp.com/hpux/content/hardware/ems/dm_raid_adapter.htm#43

v-v-v-v-v-v-v-v-v-v-v-v-v    D  E  T  A  I  L  S    v-v-v-v-v-v-v-v-v-v-v-v-v



I/O Log Event Data:

     Driver Status Code..................: 0x0000002B
     Length of Logged Hardware Status....: 0 bytes.
     Offset to Logged Manager Information: 0 bytes.
     Length of Logged Manager Information: 184 bytes.

Manager-Specific Information:

Raw data from the SCSI RAID SA Controller CISS driver:
00000007 0AD51743 00000001 00000003 00000000 00000000 11165880 02000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000




prodata2#

Why this error generated.

How to resolve the problem

Thanks

Aamir

[rariasn]

Hi:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c00992076

rgs.

[Ken Grabowski]

Use the sasmgr command to view the status of disks and logical drives on your SAS controller.  Going from OK to interim recovery is a little strange. You normally recover from a failed state.  Are there any other messages in the log about your drive state?  You may have a disk that's starting to go bad, or a badly seated drive.

[Torsten.]

You most likely have a smartarray P400 controller installed. One disk of the RAID0+1 has failed.

Use ioscan to get the device file, e.g. /dev/ciss0

# ioscan -fnCciss

Then use sautil to get the status

# sautil /dev/ciss0

pull out the bad disk, insert the replacement and check again with sautil.

[Aamir14]

Hi,

This is my sautil output

prodata2#ioscan -fnC /dev/ciss0
prodata2#sautil /dev/ciss0

******************************************************************************
****                                                                      ****
****             S A U T I L   S u p p o r t   U t i l i t y              ****
****                                                                      ****
****             for the HP SmartArray RAID Controller Family             ****
****                                                                      ****
****                           version A.02.13                            ****
****                                                                      ****
****  (C) Copyright 2003-2007 Hewlett-Packard Development Company, L.P.   ****
******************************************************************************



---- DRIVER INFORMATION ------------------------------------------------------

  Driver State........................ READY

---- CONTROLLER INFORMATION --------------------------------------------------

  Controller Product Number........... P400
  Controller Product Name............. HP PCIe SmartArray P400
  Hardware Path....................... 0/6/0/0/0/0/4/0/0/0
  Device File......................... /dev/ciss0
  Hardware Revision................... 'E'
  Boot Block Revision................. 0.03
  Firmware Revision (running)......... 5.20
  Firmware Revision (in ROM).......... 5.20
  Firmware Revision (inactive)........ 5.22
  # of Logical Drives................. 1
  # of Physical Disks Configured...... 2
  # of Physical Disks Detected........ 1
  Logical Drive Rebuild Priority...... 0   (low)
  Array Capacity Expansion Priority... 0   (low)
  Auto-Fail Missing Disks at Boot..... disabled
  SCSI Transfer Detection Mode........ Auto Detect
  Controller Path Selection Method.... default


---- ARRAY ACCELERATOR (CACHE) INFORMATION -----------------------------------

  Array Accelerator Board Present?.... yes
  Cache Configuration Status.......... cache enabled
  Cache Ratio......................... 25% Read / 75% Write
  Total Cache Size (MB)............... 208
    Read Cache........................ 052
    Write Cache....................... 156
    Transfer Buffer................... 000
  Battery Pack Count.................. 1
  Battery Status (pack #1)............ ok


---- LOGICAL DRIVE SUMMARY ---------------------------------------------------

       #   RAID      Size        Status

       0   1+0      139979 MB    USING INTERIM RECOVERY MODE


---- SAS/SATA DEVICE SUMMARY -------------------------------------------------

  Location  Ct Enc Bay       WWID           Type        Capacity Status

  internal  2I   1   8  0x500000e01e9a0362  DISK       146.8 GB  OK
  N/A       2I   1   7  0x500000e01e9a1dd2  N/A        N/A       FAILED


---- SAS/SATA ENCLOSURE SUMMARY ----------------------------------------------

  Location  Ct   Enc  Expander_count    Bay_count     SEP_count

  internal  1I   1    0                 4             1
  internal  2I   1    0                 4             1


---- LOGICAL DRIVE 0 ---------------------------------------------------------

  Logical Drive Device File........... c0t0d0
  Fault Tolerance Mode................ RAID 1+0 (Disk Mirroring)
  Logical Drive Size.................. 139979 MB
  Logical Drive Status................ USING INTERIM RECOVERY MODE
  # of Participating Physical Disks... 2

  Participating Physical Disk(s)...... Ct:Enc:Bay:WWID
                                       2I:1:8:0x500000e01e9a0362
                                       2I:1:7:0x500000e01e9a1dd2 <-- NOT RESPONDING

  Participating Spare Disk(s)......... Ct:Enc:Bay:WWID
                                       none

  Stripe Size......................... 128 KB
  Logical Drive Cache Status.......... cache enabled
  Configuration Signature............. 0xA9CAF724
  Media Exchange Detected?............ no


---- SAS/SATA DEVICE 2I:1:8:0x500000e01e9a0362 [DISK] ------------------------

  Connector Location............................... internal
  Connector........................................ 2I
  Enclosure........................................ 1
  Bay.............................................. 8
  WWID............................................. 0x500000e01e9a0362
  Device Type...................................... DISK
  Disk Capacity.................................... 146.8 GB
  Device Status.................................... OK
  Device Vendor ID................................. HP
  Device Product ID................................ DG146BABCF
  Device Firmware Version.......................... HPD6
  Reserved Area (cfg/status info).................. 33554.4 KB (33.6 MB)
  Block Size (bytes/sector)........................ 512 bytes
  M&P Data Stamped?................................ yes
  Last Failure Reason.............................. none

  Physical Disk Flags:
    Disk present and operational................... yes
    Non-disk device detected....................... no

    S.M.A.R.T. supported........................... yes
    S.M.A.R.T. errors (in factory M&P data)........ no
    S.M.A.R.T. enabled............................. yes
    S.M.A.R.T. errors (in powerup M&P data)........ no
    Attached to external connector................. no
    Configured in a logical drive.................. yes
    Configured as a spare disk..................... no
    Disk write cache enabled at spin up............ no

    Supports redundant controller operation........ yes
    Disk write cache enabled in current page....... no
    Disk write cache disabled in default page...... yes





******************************************************************************
****                         End of SAUTIL Output                         ****
********************************************************

How to recover failed disk.

and how to find this disk is mirror or Primary disk.

Thanks 

Aamir

[Dennis Handly]

>How to recover failed disk?  and how to find this disk is mirror or Primary disk.

Participating Physical Disk(s)...... Ct:Enc:Bay:WWID
                                       2I:1:8:0x500000e01e9a0362
                                       2I:1:7:0x500000e01e9a1dd2 <-- NOT RESPONDING

You replace it.  I wouldn't think it matters which disk is "primary" for an array, especially when one failed.

[Aamir14]

Thanks for soonest reply

O.K I down the server

Than Insert new 146.8 GB disk.

one question ?

When insert a new disk on server its automatically mirroring,

or I will do some task or run some commands .

Thanks

Aamir

[Aamir14]

Hi,

I am only one engineer on my company to handle HP-UX servers

On my town and my company no one Knowing the HP-UX basic Knowledge

That's why I told you some basic questions .

Sorry some trouble.

Thanks

aamir

[Torsten.]

Do not shutdown the server - not needed!

Look at this:

 Location Ct Enc Bay WWID Type Capacity Status

internal 2I 1 8 0x500000e01e9a0362 DISK 146.8 GB OK
N/A 2I 1 7 0x500000e01e9a1dd2 N/A N/A FAILED

If you have a llok at your server front, you see a disk in slot 7 and in slot 8.

Disk in slot 7 is bad.

Pull the disk and insert the new disk.

Then check again with sautil - you should see the drive recovering.

[Aamir14]

I had shudown the server

during startup time its told press f1 to recover mode

after long process

 system started and its contained error message on syslog file.

tail -100000 /var/adm/syslog/syslog.log |grep error
Aug  8 00:06:24 prodata2 vmunix:       abort due to a write error on replacement disk 5:
prodata2#

[Torsten.]

Did you replace the disk?

With server powered off?

What is the "sautil" output now?

[Aamir14]

Hi,

This is my sautil output.

prodata2#sautil /dev/ciss0 |more

******************************************************************************
****                                                                      ****
****             S A U T I L   S u p p o r t   U t i l i t y              ****
****                                                                      ****
****             for the HP SmartArray RAID Controller Family             ****
****                                                                      ****
****                           version A.02.13                            ****
****                                                                      ****
****  (C) Copyright 2003-2007 Hewlett-Packard Development Company, L.P.   ****
******************************************************************************



---- DRIVER INFORMATION ------------------------------------------------------

  Driver State........................ READY

---- CONTROLLER INFORMATION --------------------------------------------------

  Controller Product Number........... P400
  Controller Product Name............. HP PCIe SmartArray P400
  Hardware Path....................... 0/6/0/0/0/0/4/0/0/0
  Device File......................... /dev/ciss0
  Hardware Revision................... 'E'
  Boot Block Revision................. 0.03
  Firmware Revision (running)......... 5.20
  Firmware Revision (in ROM).......... 5.20
  Firmware Revision (inactive)........ 5.22
  # of Logical Drives................. 1
  # of Physical Disks Configured...... 2
  # of Physical Disks Detected........ 1
  Logical Drive Rebuild Priority...... 0   (low)
  Array Capacity Expansion Priority... 0   (low)
  Auto-Fail Missing Disks at Boot..... disabled
  SCSI Transfer Detection Mode........ Auto Detect
  Controller Path Selection Method.... default


---- ARRAY ACCELERATOR (CACHE) INFORMATION -----------------------------------

  Array Accelerator Board Present?.... yes
  Cache Configuration Status.......... cache enabled
  Cache Ratio......................... 25% Read / 75% Write
  Total Cache Size (MB)............... 208
    Read Cache........................ 052
    Write Cache....................... 156
    Transfer Buffer................... 000
  Battery Pack Count.................. 1
  Battery Status (pack #1)............ ok


---- LOGICAL DRIVE SUMMARY ---------------------------------------------------

       #   RAID      Size        Status

       0   1+0      139979 MB    USING INTERIM RECOVERY MODE


---- SAS/SATA DEVICE SUMMARY -------------------------------------------------

  Location  Ct Enc Bay       WWID           Type        Capacity Status

  internal  2I   1   8  0x500000e01e9a0362  DISK       146.8 GB  OK
  N/A       2I   1   7  0x500000e01e9a1dd2  N/A        N/A       FAILED


---- SAS/SATA ENCLOSURE SUMMARY ----------------------------------------------

  Location  Ct   Enc  Expander_count    Bay_count     SEP_count

  internal  1I   1    0                 4             1
  internal  2I   1    0                 4             1


---- LOGICAL DRIVE 0 ---------------------------------------------------------

  Logical Drive Device File........... c0t0d0
  Fault Tolerance Mode................ RAID 1+0 (Disk Mirroring)
  Logical Drive Size.................. 139979 MB
  Logical Drive Status................ USING INTERIM RECOVERY MODE
  # of Participating Physical Disks... 2

  Participating Physical Disk(s)...... Ct:Enc:Bay:WWID
                                       2I:1:8:0x500000e01e9a0362
                                       2I:1:7:0x500000e01e9a1dd2 <-- NOT RESPONDING

  Participating Spare Disk(s)......... Ct:Enc:Bay:WWID
                                       none

  Stripe Size......................... 128 KB
  Logical Drive Cache Status.......... cache enabled
  Configuration Signature............. 0xA9CAF724
  Media Exchange Detected?............ no


---- SAS/SATA DEVICE 2I:1:8:0x500000e01e9a0362 [DISK] ------------------------

  Connector Location............................... internal
  Connector........................................ 2I
  Enclosure........................................ 1
  Bay.............................................. 8
  WWID............................................. 0x500000e01e9a0362
  Device Type...................................... DISK
  Disk Capacity.................................... 146.8 GB
  Device Status.................................... OK
  Device Vendor ID................................. HP
  Device Product ID................................ DG146BABCF
  Device Serial Number............................. BS05P8C0S3800851
  Device Firmware Version.......................... HPD6
  Reserved Area (cfg/status info).................. 33554.4 KB (33.6 MB)
  Block Size (bytes/sector)........................ 512 bytes
  M&P Data Stamped?................................ yes
  Last Failure Reason.............................. none

  Physical Disk Flags:
    Disk present and operational................... yes
    Non-disk device detected....................... no

    S.M.A.R.T. supported........................... yes
    S.M.A.R.T. errors (in factory M&P data)........ no
    S.M.A.R.T. enabled............................. yes
    S.M.A.R.T. errors (in powerup M&P data)........ no
    Attached to external connector................. no
    Configured in a logical drive.................. yes
    Configured as a spare disk..................... no
    Disk write cache enabled at spin up............ no

    Supports redundant controller operation........ yes
    Disk write cache enabled in current page....... no
    Disk write cache disabled in default page...... yes

Thanks

Aamir

[Torsten.]

If not done already, replace the disk now online, post sautil output after the replacement.

[Aamir14]

Hi Torsen,

Another server i have Seven disk

and all disk are used but my client says findout the one disk on the server.and formated the perticular

disk.

after formated replace the disk to faulty server disk.

So,Please see my another server details.

and help me  it is possible without harddisk formated.

appsdev1#ioscan -funC disk
Class     I  H/W Path     Driver S/W State   H/W Type     Description
=====================================================================
disk      4  0/6/0/0/0/0/4/0/0/0.0.0  sdisk   CLAIMED     DEVICE       HP      LOGICAL VOLUME
                         /dev/dsk/c0t0d0     /dev/rdsk/c0t0d0
                         /dev/dsk/c0t0d0s1   /dev/rdsk/c0t0d0s1
                         /dev/dsk/c0t0d0s2   /dev/rdsk/c0t0d0s2
                         /dev/dsk/c0t0d0s3   /dev/rdsk/c0t0d0s3
disk      3  0/6/0/0/0/0/4/0/0/0.0.1  sdisk   CLAIMED     DEVICE       HP      LOGICAL VOLUME
                         /dev/dsk/c0t0d1   /dev/rdsk/c0t0d1
disk      5  0/6/0/0/0/0/4/0/0/0.0.2  sdisk   CLAIMED     DEVICE       HP      LOGICAL VOLUME
                         /dev/dsk/c0t0d2   /dev/rdsk/c0t0d2
disk     11  255/1/0.0.0  sdisk   CLAIMED     DEVICE       TEAC    DVD-ROM DW-224EV
                         /dev/dsk/c1t0d0   /dev/rdsk/c1t0d0
appsdev1#
appsdev1#ioscan -m dsf
Persistent DSF           Legacy DSF(s)
========================================
/dev/rdisk/disk0         /dev/rdsk/c0t0d0
/dev/rdisk/disk0_p1      /dev/rdsk/c0t0d0s1
/dev/rdisk/disk0_p2      /dev/rdsk/c0t0d0s2
/dev/rdisk/disk0_p3      /dev/rdsk/c0t0d0s3
/dev/rdisk/disk6         /dev/rdsk/c0t0d1
/dev/rdisk/disk7         /dev/rdsk/c0t0d2
/dev/rdisk/disk12        /dev/rdsk/c1t0d0

                                                                     ****
****  (C) Copyright 2003-2007 Hewlett-Packard Development Company, L.P.   ****
******************************************************************************

---- DRIVER INFORMATION ------------------------------------------------------

  Driver State........................ READY

---- CONTROLLER INFORMATION --------------------------------------------------

  Controller Product Number........... P400
  Controller Product Name............. HP PCIe SmartArray P400
  Hardware Path....................... 0/6/0/0/0/0/4/0/0/0
  Device File......................... /dev/ciss0
  Hardware Revision................... 'E'
  Boot Block Revision................. 0.03
  Firmware Revision (running)......... 5.20
  Firmware Revision (in ROM).......... 5.20
  Firmware Revision (inactive)........ 5.22
  # of Logical Drives................. 3
  # of Physical Disks Configured...... 7
  # of Physical Disks Detected........ 7
  Logical Drive Rebuild Priority...... 0   (low)
  Array Capacity Expansion Priority... 0   (low)
  Auto-Fail Missing Disks at Boot..... disabled
  SCSI Transfer Detection Mode........ Auto Detect
  Controller Path Selection Method.... default

---- ARRAY ACCELERATOR (CACHE) INFORMATION -----------------------------------

  Array Accelerator Board Present?.... yes
  Cache Configuration Status.......... cache enabled
  Cache Ratio......................... 25% Read / 75% Write
  Total Cache Size (MB)............... 208
    Read Cache........................ 052
    Write Cache....................... 156
    Transfer Buffer................... 000
  Battery Pack Count.................. 1
  Battery Status (pack #1)............ ok

---- LOGICAL DRIVE SUMMARY ---------------------------------------------------

       #   RAID      Size        Status

       0   1+0      139979 MB    OK
       1   1+0      279964 MB    OK
       2   0        286070 MB    OK

---- SAS/SATA DEVICE SUMMARY -------------------------------------------------

  Location  Ct Enc Bay       WWID           Type        Capacity Status

  internal  1I   1   4  0x5000c5000fc99599  DISK       146.8 GB  OK
  internal  1I   1   3  0x5000c5000fc9e20d  DISK       146.8 GB  OK
  internal  1I   1   2  0x5000c5000a998e6d  DISK       300.0 GB  OK
  internal  2I   1   8  0x5000c5000fc9af65  DISK       146.8 GB  OK
  internal  2I   1   7  0x5000c5000fc9f46d  DISK       146.8 GB  OK
  internal  2I   1   6  0x5000c5000fc9f051  DISK       146.8 GB  OK
  internal  2I   1   5  0x5000c5000fc9eb3d  DISK       146.8 GB  OK

---- SAS/SATA ENCLOSURE SUMMARY ----------------------------------------------

  Location  Ct   Enc  Expander_count    Bay_count     SEP_count

  internal  1I   1    0                 4             1
  internal  2I   1    0                 4             1

---- LOGICAL DRIVE 0 ---------------------------------------------------------

Thanks

aamir

[Torsten.]

You want to take a disk from another server?

Which disks?

Your "sautil" output is incomplete, please send again.

[Aamir14]

Hi,

Please see the full sautil output attached file.

Thanks

aamir

[Torsten.]

As you can see all disks are used, but disk in slot 2 has a RAID0 only.

Dou you want to remove this disk?

So you should first delete the LVM configuration and the RAID on this disk.

Is this what you want to do?

[Aamir14]

hi,

But
 bay 2 or slot 2 has 300 gb disk

and failed disk size is 146gb

but its size is different.

Enclosure........................................ 1
  Bay.............................................. 2
  WWID............................................. 0x5000c5000a998e6d  
  Device Type...................................... DISK       
  Disk Capacity.................................... 300.0 GB  
  Device Status.................................... OK        
  Device Vendor ID................................. HP

Thanks

Aamir

[Torsten.]

I don't know what you want to do ... please clarify.

[Aamir14]

Hi,

my question is that slot 2 disk is 300gb

and faulty server disk size is 146gb

does size are matter?

and how to delete raid configuration on particular disk.

Thanks

aamir

[Aamir14]

Hi,

I want to delete raid disk (300gb)

Please see the command below:--

this is right command

saconfig /dev/ciss0 -D 2

after deleting what i will do ?

[Torsten.]

first you should delete the lvm structures from c0t0d2.

# strings /etc/lvmtab

# vgdisplay -v ...

# lvremove ...

# vgremove ...

then delete the logical drive, then pull the disk.


You will waste the space, better to ue a disk with same size.

[Aamir14]

I have another server

disk details are:--

appsdev2.mpmkvvcl.com#
appsdev2.mpmkvvcl.com#sautil /dev/ciss0 |more

sautil: Unrecognized device file specified: "/dev/ciss0"

sautil: Error - File open failed
sautil: No such file or directory, errno=2


******************************************************************************
****                                                                      ****
****             S A U T I L   S u p p o r t   U t i l i t y              ****
****                                                                      ****
****             for the HP SmartArray RAID Controller Family             ****
****                                                                      ****
****                           version A.02.13                            ****
****                                                                      ****
****  (C) Copyright 2003-2007 Hewlett-Packard Development Company, L.P.   ****
******************************************************************************


  RAID SA device files (/dev/cissX) found using "/usr/sbin/ioscan -kfnd ciss":


  Run "sautil run_startup_script" if you suspect that the device file is missing or corrupted.
  Run "sautil" to view usage screen.

appsdev2.mpmkvvcl.com#ioscan -m dsf
Persistent DSF           Legacy DSF(s)
========================================
/dev/rdisk/disk9         /dev/rdsk/c0t2d0
/dev/rdisk/disk10        /dev/rdsk/c0t4d0
/dev/rdisk/disk11        /dev/rdsk/c0t1d0
/dev/rdisk/disk12        /dev/rdsk/c0t3d0
/dev/rdisk/disk14        /dev/rdsk/c0t5d0
/dev/rdisk/disk15        /dev/rdsk/c0t6d0
/dev/rdisk/disk15_p1     /dev/rdsk/c0t6d0s1
/dev/rdisk/disk15_p2     /dev/rdsk/c0t6d0s2
/dev/rdisk/disk15_p3     /dev/rdsk/c0t6d0s3
appsdev2.mpmkvvcl.com#
appsdev2.mpmkvvcl.com#
appsdev2.mpmkvvcl.com#diskinfo /dev/rdisk/disk10
SCSI describe of /dev/rdisk/disk10:
             vendor: HP
         product id: DG0300BALVP
               type: direct access
               size: 292968750 Kbytes
   bytes per sector: 512
appsdev2.mpmkvvcl.com#

disk 10 have a lot of space

Process to delete lvm----

# strings /etc/lvmtab

# vgdisplay -v ...

# lvremove ...

# vgremove ...

than i Think no p400 raid controller installed on the server

So,

only delete the LVM and pull the disk ?

thanks

aamir

[Aamir14]

Hello Torsen,

I am awaiting of your precious reply.

Thanks

Aamir

[Torsten.]

It is a bit confusing - again a different server?