topic Re: lbolt value error message. in Operating System - HP-UX

lbolt value error message.

Aamir14 — Tue, 07 Aug 2012 08:03:22 GMT

Hi,

Yesterday I found some error message in syslog file.

prodata2#tail -10000000 /var/adm/syslog/syslog.log |grep CRITICAL

Aug 6 20:08:37 prodata2 EMS [3457]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/adapters/events/raid_adapter/0_6_0_0_0_0_4_0_0_0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 226557954 -r /adapters/events/raid_adapter/0_6_0_0_0_0_4_0_0_0 -n 226557953 -a
prodata2#date
Tue Aug 7 13:02:04 IST 2012

prodata2#/opt/resmon/bin/resdata -R 226557954 -r /adapters/events/raid_adapter/0_6_0_0_0_0_4_0_0_0 -n 226557953 -a

CURRENT MONITOR DATA:

Event Time..........: Mon Aug 6 20:08:37 2012
Severity............: CRITICAL
Monitor.............: dm_raid_adapter
Event #.............: 43
System..............: prodata2

Summary:
     Adapter at hardware path 0/6/0/0/0/0/4/0/0/0 : CISS: Logical Drive
     transitioned state.

Description of Error:

lbolt value: 181737283

    CISS: Logical Drive Status:
    CISS: Logical Drive 0 transitioned from OK to INTERIM RECOVERY
    Drive Type : RAID 1+0 - Disk Mirroring

Probable Cause / Recommended Action:

Informational message.

Additional Event Data:
     System IP Address...: 192.168.4.6
     Event Id............: 0x501fd6ed00000000
     Monitor Version.....: B.01.00
     Event Class.........: I/O
     Client Configuration File...........:
     /var/stm/config/tools/monitor/default_dm_raid_adapter.clcfg
     Client Configuration File Version...: A.01.00
          Qualification criteria met.
               Number of events..: 1
     Associated OS error log entry id(s):
          0x501fd6ed00000001
     Additional System Data:
          System Model Number.............: ia64 hp server rx6600
          OS Version......................: B.11.31
          EMS Version.....................: A.04.20.31.02
          STM Version.....................: D.04.00
     Latest information on this event:
          http://docs.hp.com/hpux/content/hardware/ems/dm_raid_adapter.htm#43

v-v-v-v-v-v-v-v-v-v-v-v-v    D E T A I L S    v-v-v-v-v-v-v-v-v-v-v-v-v

I/O Log Event Data:

     Driver Status Code..................: 0x0000002B
     Length of Logged Hardware Status....: 0 bytes.
     Offset to Logged Manager Information: 0 bytes.
     Length of Logged Manager Information: 184 bytes.

Manager-Specific Information:

Raw data from the SCSI RAID SA Controller CISS driver:
00000007 0AD51743 00000001 00000003 00000000 00000000 11165880 02000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000

prodata2#

Why this error generated.

How to resolve the problem

Thanks

Aamir

Re: lbolt value error message.

rariasn — Tue, 07 Aug 2012 12:20:25 GMT

Hi:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c00992076

rgs.

Re: lbolt value error message.

Ken Grabowski — Tue, 07 Aug 2012 13:21:29 GMT

Use the sasmgr command to view the status of disks and logical drives on your SAS controller. Going from OK to interim recovery is a little strange. You normally recover from a failed state. Are there any other messages in the log about your drive state? You may have a disk that's starting to go bad, or a badly seated drive.

Re: lbolt value error message.

Torsten. — Tue, 07 Aug 2012 14:36:20 GMT

You most likely have a smartarray P400 controller installed. One disk of the RAID0+1 has failed.

Use ioscan to get the device file, e.g. /dev/ciss0

# ioscan -fnCciss

Then use sautil to get the status

# sautil /dev/ciss0

pull out the bad disk, insert the replacement and check again with sautil.

Re: lbolt value error message.

Aamir14 — Thu, 09 Aug 2012 08:05:38 GMT

Hi,

This is my sautil output

prodata2#ioscan -fnC /dev/ciss0
prodata2#sautil /dev/ciss0

******************************************************************************
****                                                                      ****
****             S A U T I L   S u p p o r t   U t i l i t y              ****
****                                                                      ****
****             for the HP SmartArray RAID Controller Family             ****
****                                                                      ****
****                           version A.02.13                            ****
****                                                                      ****
**** (C) Copyright 2003-2007 Hewlett-Packard Development Company, L.P.   ****
******************************************************************************

---- DRIVER INFORMATION ------------------------------------------------------

Driver State........................ READY

---- CONTROLLER INFORMATION --------------------------------------------------

Controller Product Number........... P400
Controller Product Name............. HP PCIe SmartArray P400
Hardware Path....................... 0/6/0/0/0/0/4/0/0/0
Device File......................... /dev/ciss0
Hardware Revision................... 'E'
Boot Block Revision................. 0.03
Firmware Revision (running)......... 5.20
Firmware Revision (in ROM).......... 5.20
Firmware Revision (inactive)........ 5.22
# of Logical Drives................. 1
# of Physical Disks Configured...... 2
# of Physical Disks Detected........ 1
Logical Drive Rebuild Priority...... 0   (low)
Array Capacity Expansion Priority... 0   (low)
Auto-Fail Missing Disks at Boot..... disabled
SCSI Transfer Detection Mode........ Auto Detect
Controller Path Selection Method.... default

---- ARRAY ACCELERATOR (CACHE) INFORMATION -----------------------------------

Array Accelerator Board Present?.... yes
Cache Configuration Status.......... cache enabled
Cache Ratio......................... 25% Read / 75% Write
Total Cache Size (MB)............... 208
    Read Cache........................ 052
    Write Cache....................... 156
    Transfer Buffer................... 000
Battery Pack Count.................. 1
Battery Status (pack #1)............ ok

---- LOGICAL DRIVE SUMMARY ---------------------------------------------------

       #   RAID      Size        Status

       0   1+0      139979 MB    USING INTERIM RECOVERY MODE

---- SAS/SATA DEVICE SUMMARY -------------------------------------------------

Location Ct Enc Bay       WWID           Type        Capacity Status

internal 2I   1   8 0x500000e01e9a0362 DISK       146.8 GB OK
N/A       2I   1   7 0x500000e01e9a1dd2 N/A        N/A       FAILED

---- SAS/SATA ENCLOSURE SUMMARY ----------------------------------------------

Location Ct   Enc Expander_count    Bay_count     SEP_count

internal 1I   1    0                 4             1
internal 2I   1    0                 4             1

---- LOGICAL DRIVE 0 ---------------------------------------------------------

Logical Drive Device File........... c0t0d0
Fault Tolerance Mode................ RAID 1+0 (Disk Mirroring)
Logical Drive Size.................. 139979 MB
Logical Drive Status................ USING INTERIM RECOVERY MODE
# of Participating Physical Disks... 2

Participating Physical Disk(s)...... Ct:Enc:Bay:WWID
                                       2I:1:8:0x500000e01e9a0362
                                       2I:1:7:0x500000e01e9a1dd2 <-- NOT RESPONDING

Participating Spare Disk(s)......... Ct:Enc:Bay:WWID
                                       none

Stripe Size......................... 128 KB
Logical Drive Cache Status.......... cache enabled
Configuration Signature............. 0xA9CAF724
Media Exchange Detected?............ no

---- SAS/SATA DEVICE 2I:1:8:0x500000e01e9a0362 [DISK] ------------------------

Connector Location............................... internal
Connector........................................ 2I
Enclosure........................................ 1
Bay.............................................. 8
WWID............................................. 0x500000e01e9a0362
Device Type...................................... DISK
Disk Capacity.................................... 146.8 GB
Device Status.................................... OK
Device Vendor ID................................. HP
Device Product ID................................ DG146BABCF
Device Firmware Version.......................... HPD6
Reserved Area (cfg/status info).................. 33554.4 KB (33.6 MB)
Block Size (bytes/sector)........................ 512 bytes
M&P Data Stamped?................................ yes
Last Failure Reason.............................. none

Physical Disk Flags:
    Disk present and operational................... yes
    Non-disk device detected....................... no

    S.M.A.R.T. supported........................... yes
    S.M.A.R.T. errors (in factory M&P data)........ no
    S.M.A.R.T. enabled............................. yes
    S.M.A.R.T. errors (in powerup M&P data)........ no
    Attached to external connector................. no
    Configured in a logical drive.................. yes
    Configured as a spare disk..................... no
    Disk write cache enabled at spin up............ no

    Supports redundant controller operation........ yes
    Disk write cache enabled in current page....... no
    Disk write cache disabled in default page...... yes

******************************************************************************
****                         End of SAUTIL Output                         ****
********************************************************

How to recover failed disk.

and how to find this disk is mirror or Primary disk.

Thanks

Aamir

Re: RAID drive failed error message

Dennis Handly — Tue, 07 Aug 2012 17:15:21 GMT

>How to recover failed disk? and how to find this disk is mirror or Primary disk.

Participating Physical Disk(s)...... Ct:Enc:Bay:WWID
2I:1:8:0x500000e01e9a0362
2I:1:7:0x500000e01e9a1dd2 <-- NOT RESPONDING

You replace it. I wouldn't think it matters which disk is "primary" for an array, especially when one failed.

Re: RAID drive failed error message

Aamir14 — Tue, 07 Aug 2012 17:24:14 GMT

Thanks for soonest reply

O.K I down the server

Than Insert new 146.8 GB disk.

one question ?

When insert a new disk on server its automatically mirroring,

or I will do some task or run some commands .

Thanks

Aamir

Re: RAID drive failed error message

Aamir14 — Tue, 07 Aug 2012 17:51:20 GMT

Hi,

I am only one engineer on my company to handle HP-UX servers

On my town and my company no one Knowing the HP-UX basic Knowledge

That's why I told you some basic questions .

Sorry some trouble.

Thanks

aamir

Re: RAID drive failed error message

Torsten. — Tue, 07 Aug 2012 18:02:26 GMT

Do not shutdown the server - not needed!

Look at this:

Location Ct Enc Bay WWID Type Capacity Status

internal 2I 1 8 0x500000e01e9a0362 DISK 146.8 GB OK
N/A 2I 1 7 0x500000e01e9a1dd2 N/A N/A FAILED

If you have a llok at your server front, you see a disk in slot 7 and in slot 8.

Disk in slot 7 is bad.

Pull the disk and insert the new disk.

Then check again with sautil - you should see the drive recovering.

Re: RAID drive failed error message

Aamir14 — Tue, 07 Aug 2012 18:30:05 GMT

I had shudown the server

during startup time its told press f1 to recover mode

after long process

system started and its contained error message on syslog file.

tail -100000 /var/adm/syslog/syslog.log |grep error
Aug 8 00:06:24 prodata2 vmunix: abort due to a write error on replacement disk 5:
prodata2#

Re: RAID drive failed error message

Torsten. — Tue, 07 Aug 2012 18:37:19 GMT

Did you replace the disk?

With server powered off?

What is the "sautil" output now?

Re: RAID drive failed error message

Aamir14 — Fri, 10 Aug 2012 05:59:44 GMT

Hi,

This is my sautil output.

prodata2#sautil /dev/ciss0 |more

******************************************************************************
****                                                                      ****
****             S A U T I L   S u p p o r t   U t i l i t y              ****
****                                                                      ****
****             for the HP SmartArray RAID Controller Family             ****
****                                                                      ****
****                           version A.02.13                            ****
****                                                                      ****
**** (C) Copyright 2003-2007 Hewlett-Packard Development Company, L.P.   ****
******************************************************************************

---- DRIVER INFORMATION ------------------------------------------------------

Driver State........................ READY

---- CONTROLLER INFORMATION --------------------------------------------------

Controller Product Number........... P400
Controller Product Name............. HP PCIe SmartArray P400
Hardware Path....................... 0/6/0/0/0/0/4/0/0/0
Device File......................... /dev/ciss0
Hardware Revision................... 'E'
Boot Block Revision................. 0.03
Firmware Revision (running)......... 5.20
Firmware Revision (in ROM).......... 5.20
Firmware Revision (inactive)........ 5.22
# of Logical Drives................. 1
# of Physical Disks Configured...... 2
# of Physical Disks Detected........ 1
Logical Drive Rebuild Priority...... 0   (low)
Array Capacity Expansion Priority... 0   (low)
Auto-Fail Missing Disks at Boot..... disabled
SCSI Transfer Detection Mode........ Auto Detect
Controller Path Selection Method.... default

---- ARRAY ACCELERATOR (CACHE) INFORMATION -----------------------------------

Array Accelerator Board Present?.... yes
Cache Configuration Status.......... cache enabled
Cache Ratio......................... 25% Read / 75% Write
Total Cache Size (MB)............... 208
    Read Cache........................ 052
    Write Cache....................... 156
    Transfer Buffer................... 000
Battery Pack Count.................. 1
Battery Status (pack #1)............ ok

---- LOGICAL DRIVE SUMMARY ---------------------------------------------------

       #   RAID      Size        Status

       0   1+0      139979 MB    USING INTERIM RECOVERY MODE

---- SAS/SATA DEVICE SUMMARY -------------------------------------------------

Location Ct Enc Bay       WWID           Type        Capacity Status

internal 2I   1   8 0x500000e01e9a0362 DISK       146.8 GB OK
N/A       2I   1   7 0x500000e01e9a1dd2 N/A        N/A       FAILED

---- SAS/SATA ENCLOSURE SUMMARY ----------------------------------------------

Location Ct   Enc Expander_count    Bay_count     SEP_count

internal 1I   1    0                 4             1
internal 2I   1    0                 4             1

---- LOGICAL DRIVE 0 ---------------------------------------------------------

Logical Drive Device File........... c0t0d0
Fault Tolerance Mode................ RAID 1+0 (Disk Mirroring)
Logical Drive Size.................. 139979 MB
Logical Drive Status................ USING INTERIM RECOVERY MODE
# of Participating Physical Disks... 2

Participating Physical Disk(s)...... Ct:Enc:Bay:WWID
                                       2I:1:8:0x500000e01e9a0362
                                       2I:1:7:0x500000e01e9a1dd2 <-- NOT RESPONDING

Participating Spare Disk(s)......... Ct:Enc:Bay:WWID
                                       none

Stripe Size......................... 128 KB
Logical Drive Cache Status.......... cache enabled
Configuration Signature............. 0xA9CAF724
Media Exchange Detected?............ no

---- SAS/SATA DEVICE 2I:1:8:0x500000e01e9a0362 [DISK] ------------------------

Connector Location............................... internal
Connector........................................ 2I
Enclosure........................................ 1
Bay.............................................. 8
WWID............................................. 0x500000e01e9a0362
Device Type...................................... DISK
Disk Capacity.................................... 146.8 GB
Device Status.................................... OK
Device Vendor ID................................. HP
Device Product ID................................ DG146BABCF
Device Serial Number............................. BS05P8C0S3800851
Device Firmware Version.......................... HPD6
Reserved Area (cfg/status info).................. 33554.4 KB (33.6 MB)
Block Size (bytes/sector)........................ 512 bytes
M&P Data Stamped?................................ yes
Last Failure Reason.............................. none

Physical Disk Flags:
    Disk present and operational................... yes
    Non-disk device detected....................... no

    S.M.A.R.T. supported........................... yes
    S.M.A.R.T. errors (in factory M&P data)........ no
    S.M.A.R.T. enabled............................. yes
    S.M.A.R.T. errors (in powerup M&P data)........ no
    Attached to external connector................. no
    Configured in a logical drive.................. yes
    Configured as a spare disk..................... no
    Disk write cache enabled at spin up............ no

    Supports redundant controller operation........ yes
    Disk write cache enabled in current page....... no
    Disk write cache disabled in default page...... yes

Thanks

Aamir

Re: RAID drive failed error message

Torsten. — Tue, 07 Aug 2012 18:56:51 GMT

If not done already, replace the disk now online, post sautil output after the replacement.

Re: RAID drive failed error message

Aamir14 — Fri, 10 Aug 2012 06:00:24 GMT

Hi Torsen,

Another server i have Seven disk

and all disk are used but my client says findout the one disk on the server.and formated the perticular

disk.

after formated replace the disk to faulty server disk.

So,Please see my another server details.

and help me it is possible without harddisk formated.

appsdev1#ioscan -funC disk
Class     I H/W Path     Driver S/W State   H/W Type     Description
=====================================================================
disk      4 0/6/0/0/0/0/4/0/0/0.0.0 sdisk   CLAIMED     DEVICE       HP      LOGICAL VOLUME
                         /dev/dsk/c0t0d0     /dev/rdsk/c0t0d0
                         /dev/dsk/c0t0d0s1   /dev/rdsk/c0t0d0s1
                         /dev/dsk/c0t0d0s2   /dev/rdsk/c0t0d0s2
                         /dev/dsk/c0t0d0s3   /dev/rdsk/c0t0d0s3
disk      3 0/6/0/0/0/0/4/0/0/0.0.1 sdisk   CLAIMED     DEVICE       HP      LOGICAL VOLUME
                         /dev/dsk/c0t0d1   /dev/rdsk/c0t0d1
disk      5 0/6/0/0/0/0/4/0/0/0.0.2 sdisk   CLAIMED     DEVICE       HP      LOGICAL VOLUME
                         /dev/dsk/c0t0d2   /dev/rdsk/c0t0d2
disk     11 255/1/0.0.0 sdisk   CLAIMED     DEVICE       TEAC    DVD-ROM DW-224EV
                         /dev/dsk/c1t0d0   /dev/rdsk/c1t0d0
appsdev1#
appsdev1#ioscan -m dsf
Persistent DSF           Legacy DSF(s)
========================================
/dev/rdisk/disk0         /dev/rdsk/c0t0d0
/dev/rdisk/disk0_p1      /dev/rdsk/c0t0d0s1
/dev/rdisk/disk0_p2      /dev/rdsk/c0t0d0s2
/dev/rdisk/disk0_p3      /dev/rdsk/c0t0d0s3
/dev/rdisk/disk6         /dev/rdsk/c0t0d1
/dev/rdisk/disk7         /dev/rdsk/c0t0d2
/dev/rdisk/disk12        /dev/rdsk/c1t0d0

****
**** (C) Copyright 2003-2007 Hewlett-Packard Development Company, L.P. ****
******************************************************************************

---- DRIVER INFORMATION ------------------------------------------------------

Driver State........................ READY

---- CONTROLLER INFORMATION --------------------------------------------------

Controller Product Number........... P400
Controller Product Name............. HP PCIe SmartArray P400
Hardware Path....................... 0/6/0/0/0/0/4/0/0/0
Device File......................... /dev/ciss0
Hardware Revision................... 'E'
Boot Block Revision................. 0.03
Firmware Revision (running)......... 5.20
Firmware Revision (in ROM).......... 5.20
Firmware Revision (inactive)........ 5.22
# of Logical Drives................. 3
# of Physical Disks Configured...... 7
# of Physical Disks Detected........ 7
Logical Drive Rebuild Priority...... 0 (low)
Array Capacity Expansion Priority... 0 (low)
Auto-Fail Missing Disks at Boot..... disabled
SCSI Transfer Detection Mode........ Auto Detect
Controller Path Selection Method.... default

---- ARRAY ACCELERATOR (CACHE) INFORMATION -----------------------------------

Array Accelerator Board Present?.... yes
Cache Configuration Status.......... cache enabled
Cache Ratio......................... 25% Read / 75% Write
Total Cache Size (MB)............... 208
    Read Cache........................ 052
    Write Cache....................... 156
    Transfer Buffer................... 000
Battery Pack Count.................. 1
Battery Status (pack #1)............ ok

---- LOGICAL DRIVE SUMMARY ---------------------------------------------------

# RAID Size Status

       0   1+0      139979 MB    OK
       1   1+0      279964 MB    OK
       2   0        286070 MB    OK

---- SAS/SATA DEVICE SUMMARY -------------------------------------------------

Location Ct Enc Bay WWID Type Capacity Status

internal 1I   1   4 0x5000c5000fc99599 DISK       146.8 GB OK
internal 1I   1   3 0x5000c5000fc9e20d DISK       146.8 GB OK
internal 1I   1   2 0x5000c5000a998e6d DISK       300.0 GB OK
internal 2I   1   8 0x5000c5000fc9af65 DISK       146.8 GB OK
internal 2I   1   7 0x5000c5000fc9f46d DISK       146.8 GB OK
internal 2I   1   6 0x5000c5000fc9f051 DISK       146.8 GB OK
internal 2I   1   5 0x5000c5000fc9eb3d DISK       146.8 GB OK

---- SAS/SATA ENCLOSURE SUMMARY ----------------------------------------------

Location Ct Enc Expander_count Bay_count SEP_count

internal 1I 1 0 4 1
internal 2I 1 0 4 1

---- LOGICAL DRIVE 0 ---------------------------------------------------------

Thanks

aamir

Re: RAID drive failed error message

Torsten. — Wed, 08 Aug 2012 04:40:24 GMT

You want to take a disk from another server?

Which disks?

Your "sautil" output is incomplete, please send again.

Re: RAID drive failed error message

Aamir14 — Wed, 08 Aug 2012 06:28:40 GMT

Hi,

Please see the full sautil output attached file.

Thanks

aamir

Re: RAID drive failed error message

Torsten. — Wed, 08 Aug 2012 06:43:36 GMT

As you can see all disks are used, but disk in slot 2 has a RAID0 only.

Dou you want to remove this disk?

So you should first delete the LVM configuration and the RAID on this disk.

Is this what you want to do?

Re: RAID drive failed error message

Aamir14 — Wed, 08 Aug 2012 07:15:23 GMT

hi,

But
bay 2 or slot 2 has 300 gb disk

and failed disk size is 146gb

but its size is different.

Enclosure........................................ 1
Bay.............................................. 2
WWID............................................. 0x5000c5000a998e6d
Device Type...................................... DISK
Disk Capacity.................................... 300.0 GB
Device Status.................................... OK
Device Vendor ID................................. HP

Thanks

Aamir

Re: RAID drive failed error message

Torsten. — Wed, 08 Aug 2012 07:30:39 GMT

I don't know what you want to do ... please clarify.

Re: RAID drive failed error message

Aamir14 — Wed, 08 Aug 2012 07:39:14 GMT

Hi,

my question is that slot 2 disk is 300gb

and faulty server disk size is 146gb

does size are matter?

and how to delete raid configuration on particular disk.

Thanks

aamir