topic Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount in StoreEver Tape Storage

LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

CLEB — Mon, 14 Sep 2009 15:07:56 GMT

I have two DL380G5 servers with SC44Ge HBA cards connected to StorageWorks 1760 1U rackmount tape drives.

Both servers fail to backup using NT Backup or ARCserve 12.0 SP2.

The event log shows Error Event Source: Lsi_sas
Event Category: None
Event ID: 11
Description:
The driver detected a controller error on \Device\RaidPort1.

I have upgraded the HBA driver to v1.28.2.1 (B) and the firmware to 06.18.05.00 (1.23.43.00A)

Tape drive firmware is U52D

I have disabled all insight agent services too.

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

CLEB — Fri, 18 Sep 2009 10:22:02 GMT

Tape drive has been replaced.

Still get lsi_sas errors in event log and LTT read/write test fails.

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

CLEB — Thu, 24 Sep 2009 13:54:17 GMT

The HBA has been replaced also.

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

GustavoT — Thu, 24 Sep 2009 20:01:56 GMT

What version of OS are you running? See if you are also getting event IDs 129 along with the Event ID 11. That and if you are running Windows 2003 Server SP2 you may want to check the storport driver. All sp2 systems are required to run storport KB945119. Upgrading the storport driver could help you fix this problem.

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

CLEB — Fri, 25 Sep 2009 08:45:14 GMT

OS is Windows 2003 R2 SP2.

I am only getting EventID 11.

I have tried every hotfix for storport.sys bar this one!

I have another server which is running the same hardware but the firmware version and scsi driver have never been updated since install. The storport.sys is SP2 version though.

Unfortunately I cannot downgrade the firmware on the hba as it does not allow that.

I will try this storport.sys, thank you for your help.

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

Curtis Ballard — Fri, 25 Sep 2009 19:42:53 GMT

Post the full DWord format binary details from the event log.

The 1U rack mount enclosure has an active interface card internal to the box. Has that been replaced or the cables?

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

CLEB — Sat, 26 Sep 2009 15:46:17 GMT

The cables were replaced and the HBA. I believe just the actual drive was shipped. This server is the other side of the world to me.

If the internal interface card is faulty then this would make sense as my local technician swapped the complete rackmount unit with another at the DR site. Perhaps there is an issue with this model?

I have since tried all newer firmware and driver revisions.

Event Type: Error
Event Source: Lsi_sas
Event Category: None
Event ID: 11
Date: 25/09/2009
Time: 8:50:55 AM
User: N/A
Computer: CANFP1
Description:
The driver detected a controller error on \Device\RaidPort1.

0000: 0018000f 00680001 00000000 c004000b
0010: 31130000 00000000 00000000 00000000
0020: 00000000 00000000 00000000 00000000
0030: 00000000 c004000b 00000000 00000000

Event Type: Error
Event Source: Lsi_sas
Event Category: None
Event ID: 11
Date: 15/05/2009
Time: 9:04:01 PM
User: N/A
Computer: CANFP1
Description:
The driver detected a controller error on \Device\RaidPort1.

Data:
0000: 0010000f 00680001 00000000 c004000b
0010: ad0f1600 00000000 00000000 00000000
0020: 00000000 00000000 00000000 00000000
0030: 00000000 c004000b

First message was before the tape drive was swapped.

I see now that HP have closed my support case even though I told them I still cannot perform a backup successfully.

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

Curtis Ballard — Mon, 28 Sep 2009 13:29:14 GMT

This error record dump looks familiar. I think you may have posted it on another thread where we were looking at a different issue.

The issues that we have identified and fixed in the various firmware was for an abort after a command was sent to the device. In generic terms a "hang" condition.

This is a different failure and the error data is reporting that the host issued an abort before the HBA had even processed the command to send on to the target. I'm not certain what might cause that.

You mention that the CE replaced the rack enclosure at your DR site. Is that the site that is having problems or was that another site that previously had problems but is now working?

You may have mentioned it before but I can't see most of your comments during a reply. What length cables are you using? I assume you are using the standard skinny "tape" cable. If you have a normal full width SAS cable it might be worth trying that just to eliminate the possibility that somehow you got multiple bad cables. There is nothing unique about the special tape cable except that it is a little cheaper than the standard cables because it doesn't need all the extra SAS pairs.

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

CLEB — Mon, 28 Sep 2009 13:48:03 GMT

The tape drive and HBA have definitely been replaced. I'm told it was the actual drive that was swapped. The 1U enclosure is still the same.

I have an exact same setup at DR site and the SAS cable was taken from there to try. The cable being used is the one that came with the tape unit.

Both sites still have issues, with my primary site getting progressively worse. This morning there is lots of these events:

Event Type: Error
Event Source: PlugPlayManager
Event Category: None
Event ID: 12
Date: 28/09/2009
Time: 7:21:02 AM
User: N/A
Computer: CANFP1
Description:
The device 'Hewlett Packard LTO Ultrium-4 drive' (SCSI\Sequential&Ven_HP&Prod_Ultrium_4-SCSI&Rev_U52D\5&2f1e44f4&0&000500) disappeared from the system without first being prepared for removal.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 00 00 ....

I have just checked and the tape drive is showing in device manager again.

I will run a test and post the results.

Unfortunately this site is in Canada, I am in the UK and my local support person is away for the next few weeks. I trying to get hold of an alternative technician.

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

CLEB — Mon, 28 Sep 2009 14:15:09 GMT

What storport version is required? Someone suggested 5.2.3790.4189. I have tried this but have since reverted to 5.2.3790.3959.

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

Curtis Ballard — Mon, 28 Sep 2009 15:38:25 GMT

I'm not aware of any Storport driver dependencies. Later drivers introduce a "feature" that requires a new registry entry for the behavior when a command can't be sent immediately.

See:
http://support.microsoft.com/kb/932755

You should set the BusyRetryCount to at least 75 and higher wouldn't hurt. If you have that registry entry then there shouldn't be any problem with the most recent Storport drivers.

The failure your system is logging might be consistent with what would happen if that registry entry wasn't set as the OS could give up and abort the command without it ever getting sent out. I don't know exactly what the event log signature is for the failures caused by this registry entry not being set correctly.

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

CLEB — Tue, 29 Sep 2009 07:52:12 GMT

Great I'll give this a go.

I've check and there isn't a key at all for this. Perhaps I should create it?

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

CLEB — Tue, 29 Sep 2009 08:05:46 GMT

I've installed that hotfix.

When I rebooted the server the event log filled up with these messages:

Event Type: Error
Event Source: Lsi_sas
Event Category: None
Event ID: 11
Date: 29/09/2009
Time: 3:00:00 AM
User: N/A
Computer: CANFP1
Description:
The driver detected a controller error on \Device\RaidPort1.

Data:
0000: 0f 00 18 00 01 00 68 00 ......h.
0008: 00 00 00 00 0b 00 04 c0 .......Ã
0010: 00 00 13 31 00 00 00 00 ...1....
0018: 00 00 00 00 00 00 00 00 ........
0020: 00 00 00 00 00 00 00 00 ........
0028: 00 00 00 00 00 00 00 00 ........
0030: 00 00 00 00 0b 00 04 c0 .......Ã
0038: 00 00 00 00 00 00 00 00 ........

Event Type: Error
Event Source: PlugPlayManager
Event Category: None
Event ID: 12
Date: 29/09/2009
Time: 3:00:00 AM
User: N/A
Computer: CANFP1
Description:
The device 'Hewlett Packard LTO Ultrium-4 drive' (SCSI\Sequential&Ven_HP&Prod_Ultrium_4-SCSI&Rev_U52D\5&2f1e44f4&0&000500) disappeared from the system without first being prepared for removal.

Data:
0000: 00 00 00 00 ....

The drive is appearing and disappearing from Device Manager constantly.

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

Curtis Ballard — Tue, 29 Sep 2009 13:20:42 GMT

If you installed a hotfix from the Microsoft KB article I pointed to you probably want to remove that. The information about the new registry entry that is required is in the document but the hotfix is old and the changes have been rolled into the standard distribution.

The "BusyRetryCount" registry entry has to be created and then set to at least 75.

It sounds like things are getting worse. You are getting errors on boot up? Any of the known issues really should be infrequent and usually only during activity. If you are having errors during the boot phase that sounds like there must be some faulty hardware. I would be a bit suspicious of the SAS signal conditioner in the 1U enclosure. That is P/N 403721-002. Your CE should be aware of it.

If there is any way you could get a loaner standalone drive that would be a really interesting test.

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

CLEB — Tue, 29 Sep 2009 15:12:16 GMT

After a power cycle the tape drive is showing a hardware error now. LTT reports the drive should no longer be used. I have an open support case for this so it's really upto HP how they want to progress now.

I have installed hotfix KB957910, storport.sys version 4485, onto the other server I had the same issues with. I downgraded the StorageWorks 1760 firmware to U29D also. I've been able to run several successful backups using ARCserve.

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

CLEB — Tue, 29 Sep 2009 15:20:55 GMT

Yes it looks like I have a stable backup on my DR server now.

SC44Ge FW 1.23.43.0, Driver 1.28.2.1
Ultrium 1760 FW U29D, Driver 1.0.5.2

Currently getting a throughput of 5600MB/minute.

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

CLEB — Tue, 29 Sep 2009 15:28:06 GMT

Forgot to mention Proliant Support Pack 8.30 was installed and Storport.sys version 5.2.3790.4485, no registry changes were made.

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

Curtis Ballard — Tue, 29 Sep 2009 15:49:07 GMT

I wouldn't recommend leaving a 1760 at U29D with the SC44Ge controller long term. Eventually if you load it heavily there will be an error but U52D can recover from that error.

I'm glad to hear that there has been some improvement.

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

CLEB — Wed, 30 Sep 2009 07:01:56 GMT

I downgraded from U52D. Backup would fail a minute or two into NT Backup job. ARCserve failed almost immediately.

Re: LSI_SAS event ID 11 with SC44Ge and StorageWorks 1760 1U Rackmount

CLEB — Wed, 30 Sep 2009 09:32:46 GMT

I was able to do several backups on my 2nd server. I have done a scan and merge of the tape this morning and there is an error in the log. There is a LSI_SAS error in the system log also:

Event Type: Error
Event Source: Lsi_sas
Event Category: None
Event ID: 11
Date: 30/09/2009
Time: 1:53:41 AM
User: N/A
Computer: CANFP2
Description:
The driver detected a controller error on \Device\RaidPort1.

Data:
0000: 0f 00 18 00 01 00 68 00 ......h.
0008: 00 00 00 00 0b 00 04 c0 .......Ã
0010: 00 00 19 31 00 00 00 00 ...1....
0018: 00 00 00 00 00 00 00 00 ........
0020: 00 00 00 00 00 00 00 00 ........
0028: 00 00 00 00 00 00 00 00 ........
0030: 00 00 00 00 0b 00 04 c0 .......Ã
0038: 00 00 00 00 00 00 00 00 ........

ARCserve seems to be quite happy to carry on though and I was able to restore a 100GB file.

We have an option to change the drive to a StorageWorks 1840 SAS. Would this model suffer from the same issue?

My boss is suggesting we should throw money at the problem now as we have not had a decent backup for so long.