topic Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2 in Operating System - OpenVMS

DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Joe Trimble — Fri, 18 Jul 2008 16:45:31 GMT

Hello, fellow VMS folks...

I just installed a new MSA1000 integrated system for use with OpenVMS. I created 4 new luns, gave them LUN ID numbers, and can seem them on my system as $1$DGA1 thru $1$DGA4.

When I try to initialize or use the new drives with VMS, I am getting hundreds of errors, some random in nature, during the INIT command or while copying (or using backup to copy) file to the new devices.

Here are some notes;
- OpenVMS V8.2
- All relevent patches applied, including VMS82A_FIBRE_SCSI V6.0.
- MSA100 upgraded to firmware version 7
- MSA100 in active/active mode
- 3 luns created using ADG (RAID6)
- 1 lun created using RAID1
- 8 drives (out of 14) hosting the 4 luns
- luns are configured using disks 1-4 and 8-11 to optimize I/O accross the scsi buses.
- each MSA1000 controller has 512MB cache, configured 50% read, 50% write.
- caching is enabled on the configured luns

When I initialize the drives, they will randomly fail as the following example shows:

$ init /system /share /headers=64000 /structure=5 $1$dga3: disk3n
%INIT-F-DATACHECK, write check error

When the drive does init correctly, and then you mount and copy files to the device, these errors are routinely seen.

%BACKUP-E-OPENOUT, error opening DISK3N:[IS$DISK.MISTY.SQL]PZVDEDN.SQL;3 as output
-RMS-E-CRE, ACP file create failed
-SYSTEM-F-DATACHECK, write check error
%BACKUP-E-OPENOUT, error opening DISK3N:[IS$DISK.OPS.MB]MB901S.RPT;1 as output
-RMS-E-CRE, ACP file create failed
-SYSTEM-F-DATACHECK, write check error
%BACKUP-E-OPENOUT, error opening DISK3N:[IS$DISK.OPS.MB]MB911S.RPT;2 as output
-RMS-E-CRE, ACP file create failed
-SYSTEM-W-BADIRECTORY, bad directory file format
%BACKUP-E-CREDIRERR, error creating directory DISK3N:[IS$DISK.OPS.MB.STORED]
-SYSTEM-W-BADIRECTORY, bad directory file format

This is only a small sample.

It seems that copying small files presents the problem more than copying large files. I have successfully copied large (20G) files to a device, and also restored a backup saveset containing several large files to the disk successfully. When I copy lots of small files, like moving user directories over to the new device, then I get hundreds of errors in a short time.

I'm likely going to call HP on this, but wanted to see what your experience and ask for your help.

Thank you!
Joe

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Uwe Zessin — Fri, 18 Jul 2008 16:53:06 GMT

Joe,

check the connections on the MSA1000 and make sure they are working with the "OpenVMS" profile.

CLI> show connections
...

CLI> add connection ALPHA1_PGA0 wwpn=10000000-C9244321 profile=OpenVMS

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Joe Trimble — Fri, 18 Jul 2008 17:59:34 GMT

I have double-checked. All connections are designated profile=OpenVMS

Here's a thought...

I may have switched the names of the two connections between the MSA's and the HBA's in my system since VMS was last booted 2 days ago. The connections are called flash-1 and flash-2. I switched them in the MSA configuration so they would match the order of the HBA cards on the PCI bus in my system. The LUNs and ACLs were created after that switch. Could that make a difference? Could the system be confused? Should I try reinitializing the Alphaserver?

Thanks.

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Jon Pinkley — Fri, 18 Jul 2008 20:51:33 GMT

Joe,

Disclaimer: I have never used an MSA.

This is only a guess, but because you notice the problems with small files, I would suspect the cache. Does the MSA have any diagnosics it can run to test its memory?

Does the problem go away if you turn off caching?

Summary: My guess is that it is a hardware issue that needs to be fixed.

Jon

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Joe Trimble — Fri, 18 Jul 2008 22:14:10 GMT

Thanks, Jon.

I thought about the cache, and that is still on the table, as far as I'm concerned.

I ended up opening a call with HP OpenVMS Support this afternoon. I'm going to get some additional information for the support rep tomorrow morning, and power-cycle the MSA and my AlphaServer system. Unfortunately, today was a work-at-home day, so I'm not physically with the equipment today.

I mentioned the cache to the support rep. He indicated it was a potential problem, but needs additional information, thus my trip to the office early Saturday.

Thanks for the replies so far.. I'll update this thread again when I have more information or questions.

Joe

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Rob Leadbeater — Sat, 19 Jul 2008 22:51:42 GMT

Hi Joe,

If you've just installed the MSA1000, it's possible that the RAID initialisation is still going on in the background...

This *shouldn't* affect the hosts connected to it, but I guess it might have an effect.

If you're talking to HP support, they'll probably have you hook up the CLI cable to front of the MSA, and do a "show techsupport" (IIRC). Looking through that output, might highlight some issues.

You might also want to check that the firmware on the MSA is current. (5.20 or 7.00 depending on whether you're active/standby or active/active.)

Cheers,

Rob

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Rinkens — Tue, 22 Jul 2008 07:14:53 GMT

Are these disk write enabled on the controller
maybe a stupid question.

$ init /system /share /headers=64000 /structure=5 $1$dga3: disk3n
%INIT-F-DATACHECK, write check error

here is goes wrong already, you have should first solve this problem.

Check the setting on your msa100

writeback cache
read cache and so on

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Joe Trimble — Tue, 22 Jul 2008 13:41:24 GMT

Here is an update on this situation. No resolution yet, so it is still being worked by HP Support.

Currently, the Storage team is still examining several log files created over the weekend during several hours of testing. Today it will likely be passed over to the AlphaServer team.

Storage team's initial conclusion is that the MSA1000 is setup and configured correctly. Firmware (V7.0) looks good on both MSA1000's, and is in active/active mode.

I found over the weekend that when I create and connect additional LUNS to another VMS server (DS25, identical VMS V8.2 and patches applied), then that system can write files without errors. So the errors appear to be a problem with the ES40 connection to MSA1000 only.

To answer recent suggestions:
- the initialization is complete on all LUNs; it made no difference in the errors.
- I have tried LUNs with cache both on and off. There seems to be fewer errors with cache turned off, but still plenty of errors are encountered. (From the DS25 test system, no DATACHECK errors occurred regardless of the cache setting.)

I now believe there is some configuration or compatibility issue between my ES40 system and the MSA1000 integrated box. The ES40 has FCA2684 HBA cards installed (same as the DS25), and the firmware is updated to the latest rev (TS1.91X6).

Thanks for your continued assistance and questions. I'll update again later.

Joe

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Rob Leadbeater — Tue, 22 Jul 2008 18:26:33 GMT

Hi Joe,

Are you using the embedded 2/8 switch in the back of the MSA1000 ?

If so, has the version of FabricOS been checked ?

Cheers,

Rob

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Joe Trimble — Tue, 22 Jul 2008 20:40:45 GMT

Hi Rob,

Yes, we are using the embedded 2/8 switches.

No, I have not checked the firmware levels there. Do you think that might make a difference? The DS25 is newer hardware than the ES40. Could there be some incompatibility with the ES40?

The HP Storage team has not asked about the switches, at least not yet. I'll try to find out more information on this.

Thanks!
Joe

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Rob Leadbeater — Wed, 23 Jul 2008 08:12:45 GMT

Hi,

If the firmware on the switches is too old, it might cause incompatibility issues with the newer firmware on the HBAs...although if the DS25's HBAs are also at the same firmware revision, that should rule that out...

I'm surprised that Storage Support haven't asked you to look at the switch port status, as that might indicate some faults, either with a HBA or cable etc.

Cheers,

Rob

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Joe Trimble — Wed, 23 Jul 2008 10:33:42 GMT

The switches show firmware as follows. Both are identical, so only switch1 is displayed here.

MSA-VMS-1-switch1:admin> version
Kernel: 5.4
Fabric OS: v3.2.1b
Made on: Fri Jul 28 14:42:33 PDT 2006
Flash: Fri Jul 28 14:43:15 PDT 2006
BootProm: Mon Jul 8 18:35:44 PDT 2002

A quick check on the HP download site shows this is one minor revision behind on the fabric OS. The latest download is v3.2.1c, dated in 2007.

Joe

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Rob Leadbeater — Wed, 23 Jul 2008 12:17:46 GMT

That version shouldn't cause any problems...

Does a "portErrShow" indicate any issues on the ES40's ports ?

Cheers,
Rob

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Joe Trimble — Wed, 23 Jul 2008 19:12:38 GMT

Rob,

I'll attach a small file showing output from a few commands on both 2/8 switches, including porterrshow. I don't know how to interpret most of the stats.

I've sent the same information to HP Storage Support as part of my open call with them.

Note the ES40 is connected to port 1 on each switch; the DS25 is connected to port 2.

Thanks,
Joe

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Khairy — Thu, 24 Jul 2008 03:19:05 GMT

hi joe,

can you provide the show tech_support output from msa controller. I know you've checked this but just want to understand more the configuration and the problem you have.

I've installed dual controller MSA1000 (14 x 72GB) with 2 x DS25 running openvms 7.3.2 last year without no problem. The only thing i encounter was patch issue. I upgraded both SRM to rule out firmware problem and it works for me.

> show tech_support

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Jon Pinkley — Thu, 24 Jul 2008 05:59:50 GMT

Joe,

No guarantees, but have you tried a complete power cycle (complete removal of power cords) on the ES40?

We installed DS-A5132-AA (370426-B21) PCI-X 64BIT 133MHZ 2Gb-ALPHA LP10000, FCA2684 HBAs in ES40 M2 4 6/667 systems, and had problems until I did a firmware upgrade on the ES40, using the manual procedure and upgraded everything, including RMC, followed by a complete power cycle (per the firmware
upgrade instructions). For more details see this thread:

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1140804

The storage was EVA on HP (Brocade) 4/16 switch fabrics. The error we saw was problems in autoconfig, not exactly the same as you are seeing, but I would still try, it only cost time.

Your porterrshow output looks fine. The "enc out" errors are related to auto-negotiation and they are normal. If you really dislike them, you can lock the speed, but in my opinion, that tends to come back to haunt in the future when you plug something else into that port (perhaps for debugging).

The MSA works with your DS25, which tends to point toward the ES40 as the problem location.

If you are going to need to power cycle, I would take the opportunity to do a firmware update on the ES40, and I would use the manual procedure to update everything, including RMC, followed by a complete power disconnect (not just turning off the power at the front panel. I shut my systems down, then disconnected the cords for a minute (perhaps overboard, but it isn't something you need to do frequently).

If that doesn't fix the problem, the next thing I would do is swap the FC HBA in the problem box. But I would try the firmware upgrade/power cycle first before introducing other changes.

Good luck,

Jon

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Joe Trimble — Thu, 24 Jul 2008 14:05:06 GMT

I'm attaching a file containing the show tech_support command from controller 1, captured last Saturday.

Regarding ES40 firmware -- the latest firmware updates (7.3) were installed before the MSA1000 was connected. We needed to upgrade because our old firmware was not up to the correct level to recognize the MSA devices.

Current firmware revisions (taken from LFU capture last Saturday during a review):

Abios v5.71
SRM v7.3-1
pga0 TS1.91X6
pgb0 TS1.91X6
rmc V2.8
srom V2.22-G
tig 10

I have power-cycled the ES40 multiple times, but only by turning off the switch on the front panel. The LED's on the HBA's do go dark when the system is powered off. Do you think pulling the power cables will make a real difference?

Thanks,
Joe

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Jon Pinkley — Thu, 24 Jul 2008 17:55:59 GMT

Is the firmware on the HBA's up to data?

The first time I did my firmware upgrade, I didn't cycle power and I didn't update the RMC (I hadn't used the manual update, which is still automated).

Whether it makes a difference? We have had several flakey problems on the ES40s (in this case a fan was being reported as bad, when it appeard to be working). A Field Service engineer replaced the fan, it worked, but the show power command continued to call out a bad fan. He was about to replace the motherboard (he had even ordered it), but before we did, he said he wanted to cycle power, with a complete disconnect.

That solved the problem.

So in at least those two cases, cycling power all completely, seems to have had an effect. The ES40 is never "completely off" when you use the fromt panel.

I think there is even something in the firmware update documentation that says you need to do a complete power cycle after appliying certain firmware updates (for example the RMC).

Like I said before, no guarantees. But I would try it. And if you haven't updated your RMC, I would do that too.

If you looked at the thread I referenced before, you will see we were getting errors. Have you analyzed your errlog with SEA?

Jon

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Jon Pinkley — Thu, 24 Jul 2008 21:25:13 GMT

This thread is about a priblem that sounds exactly like the one we had that was solved by a complete power cycle. My guess is they really didn't need to replace anything.

http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1146545

Re: DATACHECK errors with MSA1000 DGA Devices on OpenVMSV8.2

Jon Pinkley — Thu, 24 Jul 2008 21:40:47 GMT

Here's another recent thread involving ES40s.

http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1247808

What "fixed" the problem was moving the CPU boards and memory banks. So the question remains, was it the complete power cycle (assuming they really did remove power when moving the modules), or was it the reseating of the components? If you don't try a complete power cycle before changing something, you never will know for sure what the fix was.

So I always do the complete power cycle first and verify that the problem still exists before starting to change anything else.

Jon