HP Ultrium 5 LTO drives dropping offline using Backup Exec 2012 SP3

[Justin Williams_2]

We have 2 x HP LTO Ultirum 5 drives attached to a DL385p G8 server via 2 separate H222 SAS cards.

Server is running Windows 2008 R2 and the latest Proliant Service Pack from Sep 2013. All device firmare is updated to the latest version.

Backup Exec 2012 SP3 is installed and was running fine, however now the backups are failing and reporting the drives are offline.

The problem started when we had only 1 drive installed, so a new drive, card & cable was purchased to rule out a hardware fault but the problem occurs with the new drive too.

I have followed the Symantec TN http://www.symantec.com/business/support/index?&page=content&id=TECH61192&locale=en_US and also logged a call with Symantec Support.

They have reviewed the adamm.log which shows the drives going offline and have suggested I contact HP to check the status of the hardware. I have run the HP Tape Tools and the drives appear to be healthy, although I am occaisionally seeing an LSA_SAS2 Event ID 11 logged on the server - The driver detected a controller error on \Device\RaidPort1.

I have attached the output from the HP Tape Tools utility and also the BackupExec adamm.log.

Any assistance troubleshooting this issue would be appreciated as this is a production system running backups for a regional site.

Many thanks,

Justin Williams

[Justin Williams_2]

Not sure the attachments worked, so here is the relevant section from the adamm.log

Scsi Address
Prt :Bus :Tar :Lun   Attributes
-------------------  -------------------------------------------------------------------------------

0004:0000:0000:0000  "HpCISSs2", (Port Driver)

-------------------  -------------------------------------------------------------------------------

0005:0000:0000:0000  "LSI_SAS2", (Port Driver)
0005:0000:0000:0000  "HP      Ultrium 5-SCSI  Z64D", (Tape Device), "\\.\Tape0"

-------------------  -------------------------------------------------------------------------------

0006:0000:0000:0000  "LSI_SAS2", (Port Driver)
0006:0000:0000:0000  "HP      Ultrium 5-SCSI  Z64D", (Tape Device), "\\.\Tape1"

-------------------  -------------------------------------------------------------------------------

[4932] 01/10/14 00:18:10.653 Read NDMP Server Records - start

[4932] 01/10/14 00:18:10.668 Read NDMP Server Records - end

[4932] 01/10/14 00:18:10.668 Read OST Server Records - start

[4932] 01/10/14 00:18:10.699 Read OST Server Records - end

[4932] 01/10/14 00:18:10.699 DeviceIo Discovery - start

[4932] 01/10/14 00:18:18.324 DeviceIo Discovery - end

[4932] 01/10/14 00:18:18.324 DeviceIo Devices:

Scsi Address
Prt :Bus :Tar :Lun   Attributes
-------------------  -------------------------------------------------------------------------------

0005:0000:0000:0000  DeviceIo[01]: "HP      Ultrium 5-SCSI  HU1342YRVU", Cert=Y, "\\.\Tape0"
0006:0000:0000:0000  DeviceIo[02]: "HP      Ultrium 5-SCSI  HU1319VM2V", Cert=Y, "\\.\Tape1"

-------------------  -------------------------------------------------------------------------------

[4932] 01/10/14 00:18:18.324 PnP Device Paths:

Scsi Address
Prt     :Bus     :Tar     :Lun       Device Path
-----------------------------------  ---------------------------------------------------------------

00000006:00000000:00000000:00000000  \\?\scsi#sequential&ven_hp&prod_ultrium_5-scsi#5&2dd6f7d7&0&000000#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}
00000005:00000000:00000000:00000000  \\?\scsi#sequential&ven_hp&prod_ultrium_5-scsi#5&3aca94ce&0&000000#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}

-----------------------------------  ---------------------------------------------------------------

[4932] 01/10/14 00:18:18.324 Read Device Inquiries - start

[4932] 01/10/14 00:18:18.324 Device: \\.\Tape0
[4932] 01/10/14 00:18:18.714 Device: \\.\Tape1

[4932] 01/10/14 00:18:18.730 Read Device Inquiries - end

[4932] 01/10/14 00:18:18.730 Serialize Devices:

[4932] 01/10/14 00:18:18.761 Device Discovery:

[4932] 01/10/14 00:18:18.761 Device Driver Status:

Scsi Address
Prt :Bus :Tar :Lun   Status
-------------------  -----------------------------------------------

0005:0000:0000:0000  HP      Ultrium 5-SCSI  Z64D [HP      Ultrium 5-SCSI  HU1342YRVU] (White List User Mode Driver)
0006:0000:0000:0000  HP      Ultrium 5-SCSI  Z64D [HP      Ultrium 5-SCSI  HU1319VM2V] (White List User Mode Driver)
-------------------  -----------------------------------------------

Scsi Address
Prt :Bus :Tar :Lun   Attributes
-------------------  -------------------------------------------------------------------------------

0005:0000:0000:0000  Device Name             "\\.\Tape0"
                     Secondary Name          "\\?\scsi#sequential&ven_hp&prod_ultrium_5-scsi#5&3aca94ce&0&000000#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}"
                     Primary Inquiry         "HP      Ultrium 5-SCSI  Z64D"
                     Serial Number           "HP      Ultrium 5-SCSI  HU1342YRVU"
                     Device Flags            UMD, SCSI, SN(TYPE 0)
                     Device State            2, Disabled

                     Device IDs              1011, {4B258A92-B6F3-40DC-9CEA-594B1D7D4917}
                     Device Name             "Tape drive 0001"
                     Device Type             134283265, "LTO 64K (64K,10,0,E,HU)"
                     Device Features         0x005FFA7F: PB,SFB,SRB,WFM,SFF,SRM,PEOD,EL,LU,E,SBS,DC,SDC,TA,HU,DR,RR,W,EN
                     Device Element          0, 0
                     Device Block Limits     512 min, 1048576 max
                     Device Hard Errors      0 write, 0 read
                     Device Soft Errors      0 write, 0 read

-------------------  -------------------------------------------------------------------------------

0006:0000:0000:0000  Device Name             "\\.\Tape1"
                     Secondary Name          "\\?\scsi#sequential&ven_hp&prod_ultrium_5-scsi#5&2dd6f7d7&0&000000#{53f5630b-b6bf-11d0-94f2-00a0c91efb8b}"
                     Primary Inquiry         "HP      Ultrium 5-SCSI  Z64D"
                     Serial Number           "HP      Ultrium 5-SCSI  HU1319VM2V"
                     Device Flags            UMD, SCSI, SN(TYPE 0)
                     Device State            2, Disabled

                     Device IDs              1012, {164E1CE9-48D3-46BB-A541-5875EC39FBCC}
                     Device Name             "Tape drive 0002"
                     Device Type             134283265, "LTO 64K (64K,10,0,E,HU)"
                     Device Features         0x005FFA7F: PB,SFB,SRB,WFM,SFF,SRM,PEOD,EL,LU,E,SBS,DC,SDC,TA,HU,DR,RR,W,EN
                     Device Element          0, 0
                     Device Block Limits     512 min, 1048576 max
                     Device Hard Errors      0 write, 0 read
                     Device Soft Errors      0 write, 24 read

-------------------  -------------------------------------------------------------------------------

[4932] 01/10/14 00:18:18.902 Start Rsm Support:

[4932] 01/10/14 00:18:18.902 Device Discovery End:

[4932] 01/10/14 00:18:18.933 Disk mapping for Win32:

Win32 name               NT internal name
------------------------ -------------------------------------------------------------------------------

C:                      \Device\HarddiskVolume2
D:                      \Device\CdRom0
E:                      \Device\HarddiskVolume4
PhysicalDrive0          \Device\Harddisk0\DR0
PhysicalDrive1          \Device\Harddisk1\DR1
Harddisk0Partition1     \Device\HarddiskVolume1
Harddisk0Partition2     \Device\HarddiskVolume2
Harddisk1Partition1     \Device\HarddiskVolume3
Harddisk1Partition2     \Device\HarddiskVolume4

------------------------ -------------------------------------------------------------------------------

[StorageManager::RemoveUnusedLocalDisksFromDB] Results:
0 non-present unconfigured local disks were deleted
0 local disk parent objects with no local disks were deleted

[8668] 01/10/14 00:40:06.727 DeviceIo: 05:00:00:00 - Device error 55 on "\\.\Tape0", SCSI cmd 1a, 1 total errors
[8668] 01/10/14 00:40:13.330 PvlDrive::DisableAccess() - ReserveDevice failed, offline device
       Drive = 1011 "Tape drive 0001"
       ERROR = 0x0000001F (ERROR_GEN_FAILURE)

[8668] 01/10/14 00:40:13.337 PvlDrive::UpdateOnlineState()
       Drive = 1011 "Tape drive 0001"
       ERROR = The device is offline!

       Changer    = {00000000-0000-0000-0000000000000000} ""
       Drive      = {4B258A92-B6F3-40DC-9CEA594B1D7D4917} "Tape drive 0001"
       Slot       = 0
       Media      = {D46256F5-4326-4B47-BFABEF9E3EECF8FF} "LTO000031"
       ERROR = 0xE000810C (E_PVL_DRIVE_NOT_AVAILABLE)

[5972] 01/10/14 00:40:18.732 PvlDrive::DisableAccess() - ReserveDevice failed, offline device
       Drive = 1011 "Tape drive 0001"
       ERROR = 0x0000001F (ERROR_GEN_FAILURE)

[5972] 01/10/14 00:40:18.738 PvlDrive::UpdateOnlineState()
       Drive = 1011 "Tape drive 0001"
       ERROR = The device is offline!

[5972] 01/10/14 00:40:18.739 Begin dump of device's SCSI history

[5972] 01/10/14 00:40:18.739 End dump of device SCSI history

[Dennis Handly]

>Not sure the attachments worked

No, only known suffixes are allowed.  Try renaming it to .txt.

[Threonine]

We are having this same kind of issue with an H222 controller attached to an LTO6 drive in a VMware ESXi 5.5 host (HP custom ISO).  The tape drive is externally attached with SAS.  The virtual machine has the tape drive added to its configuration (not using DirectPath I/O) and the VM is running Windows Server 2012 R2 and Backup Exec 2014.  We have to reboot the host to get the tape drive to come back online.

Anyone else get anywhere with similar symptoms?

[Curtis Ballard]

Regarding the most recent posting, I haven't looked at this in a couple of years but the last time I did we spent a lot of time and were unable to find any reliable way of using tape drives with VMWare.  At the time VMWare claimed that tape would work only when connected to specific Adaptec parallel SCSI controllers.  I just ran a quick search and the latest document I could find still has that claim:

• The tape hardware must be connected to a VMWare-supported Adaptec SCSI controller.

That is from: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1016407

I once worked with an engineer that ran a lot of testing and was able to get tape drive connected and seeming to work but always ran into random problems with things like losing the tape drive from the guest OS, random I/O errors, etc.  SCSI analyzers didn't show any issues with external traffic but somewhere in the system stack things would occasionally not work correctly.

I just pulled the latest copy of the EBSMatrix which shows HP's compatibility (http://www.hp.com/go/ebs) and confirmed that SAS on VMWare is not listed as a supported configuration.  The matrix shows direct attached [parallel] SCSI as working but provides a qualifier that HP does not support that configuration.  SAS and FC attach are both shown as 'No'.

[RbBsmn]

HP SAS Tape drives are not supported in VMWare.
There is an advisory about that, will try to dig it up once I'm on my pc.

@Justin Williams, could you generare a support ticket and drive assessment result log and upload these logs (if this issue is still ongoing)?