topic Re: Shadowing problem in Operating System - OpenVMS

Shadowing problem

dschwarz — Fri, 29 Aug 2008 07:08:09 GMT

On one node of our OpenVMS Cluster we observe
growing error counter for a single member shadow
set without errors on the member disk.

Output of 'SHOW DEVICE D' looks like this:

Device Device Error
Name Status Count
DSA100: Mounted 24
...
$1$DGA100: (BUCL01) ShadowSetMember 0 (member of DSA100:)

Cluster consists of 2 DS20E, OpenVMS Version is 7.3-2,
Disk is MSA1000 unit, shadow set is mounted on both cluster members.

DIAGNOSE shows:
...
IOSB x000000000000005C
DATACHECK - write check error
...

Any suggestions ?

Re: Shadowing problem

Jon Pinkley — Fri, 29 Aug 2008 08:07:35 GMT

dschwarz,

Please provide output (in plain text attachment) from the following:

$ mcr sysman set env/cluster
SYSMAN> do show device/ful dsa100

(Cut and paste output to notepad, save with .txt extension, and save as attachment with reply.)

Jon

Re: Shadowing problem

dschwarz — Fri, 29 Aug 2008 08:18:57 GMT

hi Jon,

attached you will find sysman output
(at least I hope so).

Re: Shadowing problem

Jon Pinkley — Fri, 29 Aug 2008 08:39:05 GMT

I don't see any obvious problems, and I can't explain the errors on the DSA virtual unit, at least assuming that DSA100 has been as single member shadowset for the duration of the boot.

Are other members being added and removed, for instance for "backup snapshots"?

What device did the DATACHECK write check error appear under?

Jon

P.S. I had no trouble reading the attachment.

Re: Shadowing problem

dschwarz — Fri, 29 Aug 2008 09:11:58 GMT

At the time in question here, we didn't add/remove memebrs to/from the shadow set.

All errors appear under DSA100 as you can see in the attached DIAGNOSE extract.

This system has another seven(!) single member shadow sets mounted. None of them has ever shown errors on th dsaxxx device.

Re: Shadowing problem

Hoff — Fri, 29 Aug 2008 15:17:46 GMT

ANALYZE /DISK /SHADOW the shadow master?

IIRC, this command arrived via shadowing ECO kit in V7.3-2. And on the topic of ECOs, if you're not current on your ECOs and your firmware, get there.

If you're current on your patches and firmware, it might also pay off to engage your hardware support organization here, as there's clearly something a little wonky here.

Re: Shadowing problem

Volker Halle — Sun, 31 Aug 2008 05:48:18 GMT

You've set your DSA100: shadowset virtual unit for DATA CHECK on all WRITE-operations (SET VOLUME/DATA_CHECK=WRITE DSA100:), this is quite unusual. Did you change that recently ? Why ?

Do those errors disappear, if you turn it off ? Does shadowing correctly handle this setting ?

Volker.

Re: Shadowing problem

dschwarz — Mon, 01 Sep 2008 11:49:35 GMT

Hoff,

ANA/DISK/SHADOW compares the disk with itself. As expected there were no errors.

We are not completely up to date with firmware and patches. I shall install the latest firmware and patches as soon as possible.

Volker,

DATA_CHECK on write operations has been enabled in 2002 because there were mysterious problems with Oracle Rdb. It didn't help solving those problems but nobody turned it off. We will try this now, because this device is in fact the only with DATA_CHECK turned on.

But what's wrong with DATA_CHECK enabled except slowing down I/O ?

Re: Shadowing problem

Wim Van den Wyngaert — Mon, 01 Sep 2008 12:05:07 GMT

Seems a bug in shadowing to me.
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=110&prodSeriesId=459922&prodTypeId=18964&prodSeriesId=459922&objectID=c01462103 so turn the write check off.

Also try a anal/dis/read of your disk (will read all data on disk !) to see if there are parity errors (bad blocks). It shouldn't give the errors because there are no hardware errors but it could give more info about what is wrong.

Wim

Re: Shadowing problem

Jur van der Burg — Tue, 02 Sep 2008 05:25:13 GMT

Ah, yes I remember. An application may cause this by issueing an async datacheck i/o and clobbering the buffers before the datacheck completes.

Fwiw,

Jur.

Re: Shadowing problem

debu_17 — Tue, 02 Sep 2008 08:56:13 GMT

I feel that the errors due "Data Check on write" may have something to do with , the write cache for this disk, as the cache may be shared with the main disk for which dsa100 is the shadow volume.
2. as already suggested the async operations
may modify the data before the write check completes.

Re: Shadowing problem

dschwarz — Tue, 02 Sep 2008 09:54:27 GMT

Your answers helped me understanding what is happening on the system. I now have an idea why this error occurs and why it doesn't happen very often.

Thank you very much.

I turned DATA_CHECK off.

I hope this will solve the problem, but as this is not seen very often, it will take some months maybe years until we can be quite sure.

Re: Shadowing problem

John Hockett — Wed, 03 Sep 2008 18:44:49 GMT

Curious - to seeing this problem were there any kits installed - you can send the PRODUCT SHOW HISTORY to john.hockett@hp.com

Thanks