topic Re: HBMM %SYSTEM-F-SHADFEATNOMNT, in Operating System - OpenVMS

HBMM %SYSTEM-F-SHADFEATNOMNT,

Benjamin Levy — Sat, 06 Mar 2010 20:05:12 GMT

Hello

On a VMS V7.3-2 Alpha cluster, the SYPAGSWPFILES.COM contains
Set Shadow/Enable=HBMM Sys$Sysdevice:
after a complete cluster shutdown, the first node to reboot executes this command and gives:

%SYSTEM-F-SHADFEATNOMNT, device not mounted on system that supports requested Shadowing feature

HELP/MESSAGE does not seem to know about this error. Any ideas?

No obvious actual adverse impact on the system, other than the error message.

Thank you

Re: HBMM %SYSTEM-F-SHADFEATNOMNT,

Steven Schweda — Sat, 06 Mar 2010 20:16:48 GMT

I know nothing, but around here:

alp $ help /mess SHADFEATNOMNT

SHADFEATNOMNT, device not mounted on system that supports requested
shadowing feature

Facility: SET, SET Command and SET Utility

Explanation: You tried to enable an enhanced shadowing feature on the
specified shadow set, but the shadow set is mounted on at
least one system in the cluster that does not support that
feature.

User Action: Dismount the shadow set on the system that does not support
the requested enhanced shadowing feature; then try the
operation again.

alp $ write sys$output "''f$getsyi( "version")' ''f$getsyi( "arch_name")'"
V8.3 Alpha

What's mounted where?
SHOW DEVICE D

Re: HBMM %SYSTEM-F-SHADFEATNOMNT,

Robert Brooks_1 — Sun, 07 Mar 2010 00:46:16 GMT

We'll need a bit more detail, like what is the policy that's in use. Is there a default policy defined for the cluster? If so, are you sure that it's defined before this attempt at using HBMM?

-- Rob

Re: HBMM %SYSTEM-F-SHADFEATNOMNT,

Benjamin Levy — Mon, 08 Mar 2010 01:16:08 GMT

The command immediately proceeding the attempt to enable HBMM on the system disk is:

$ Set Shadow/Name=Default -
/Policy=HBMM=((MASTER_LIST=(*)),RESET_THRESHOLD=50000)

I think this may be due to the 2nd node booting into the cluster being not very far behind the 1st node. Possibly the 2nd node is far along enough in the boot for the shadow driver to count it as a node, but not far enough for the shadow driver to enable HBMM on that other node??

Re: HBMM %SYSTEM-F-SHADFEATNOMNT,

Robert Brooks_1 — Mon, 08 Mar 2010 03:37:31 GMT

Possibly the 2nd node is far along enough in the boot for the shadow driver to count it as a node, but not far enough for the shadow driver to enable HBMM on that other node??

--
That's a possibility, although I'll need to think about this a bit. The SHADOW_SERVER process is required for ANY shadow set to be mounted, even if HBMM isn't used. Thus, it's hard to imagine (but not impossible) a window where enough of the system was running to allow the mounting of the shadowed system disk
on the 2nd node, but yet not enough to have
gone through the locking protocol to allow it to run any of the enhanced features (HBMM or dissimilar device shadowing).

-- Rob

Re: HBMM %SYSTEM-F-SHADFEATNOMNT,

Benjamin Levy — Tue, 09 Mar 2010 05:15:18 GMT

OK - looked back a little more closely. Have seen the issue twice in the last month. In both cases the first node was starting to boot, and in SYPAGSWPFILES.COM. The second node had not yet started to boot - was still at >>> prompt. The second node did start to boot a little later - before the first node had finished booting - but well after the suspicious shadow driver message.

Re: HBMM %SYSTEM-F-SHADFEATNOMNT,

Ian Miller. — Tue, 09 Mar 2010 10:12:12 GMT

Are both nodes booting from the same system disk and what shadowing patches do you have installed?

Re: HBMM %SYSTEM-F-SHADFEATNOMNT,

Benjamin Levy — Tue, 09 Mar 2010 15:55:45 GMT

both nodes on the same system disk

PROD SHOW HISTORY show the last major round of patches was done in 2004. (I know - a long time ago.)

DEC AXPVMS VMS732_SYS V7.0 Patch Install 10-JUN-2005
DEC AXPVMS VMS732_RMS V2.0 Patch Install 09-MAR-2005
DEC AXPVMS VMS732_VMSMUP V1.0 Patch Install 08-MAR-2005
DEC AXPVMS VMS732_DRIVER V1.0 Patch Install 08-MAR-2005
DEC AXPVMS VMS732_LINKER V1.0 Patch Install 08-MAR-2005
DEC AXPVMS TCPIP_ECO V5.4-154 Patch Install 08-MAR-2005
DEC AXPVMS VMS732_SYSINI V1.0 Patch Install 08-MAR-2005
DEC AXPVMS VMS732_FIBRE_SCSI V4.0 Patch Install 08-DEC-2004
DEC AXPVMS VMS732_SYS V6.0 Patch Install 08-DEC-2004
DEC AXPVMS VMS732_TRACE V2.0 Patch Install 01-DEC-2004
DEC AXPVMS VMS732_UPDATE V3.0 Patch Install 01-DEC-2004
DEC AXPVMS VMS732_F11X V3.0 Patch Install 19-OCT-2004
DEC AXPVMS DNVOSIECO01 V7.3-2 Patch Install 13-OCT-2004
DEC AXPVMS VMS732_HBMM V2.0 Patch Install 12-OCT-2004
DEC AXPVMS VMS732_BACKUP V2.0 Patch Install 12-OCT-2004
DEC AXPVMS VMS732_IPC V1.0 Patch Install 12-OCT-2004
DEC AXPVMS VMS732_AUDSRV V1.0 Patch Install 12-OCT-2004
DEC AXPVMS VMS732_TRACE V1.0 Patch Install 12-OCT-2004
DEC AXPVMS VMS732_F11X V2.0 Patch Install 12-OCT-2004
DEC AXPVMS VMS732_XFC V1.0 Patch Install 12-OCT-2004
DEC AXPVMS VMS732_RPC V3.0 Patch Install 12-OCT-2004
DEC AXPVMS VMS732_SYS V5.0 Patch Install 25-AUG-2004
DEC AXPVMS VMS732_BACKUP V1.0 Patch Install 25-AUG-2004
DEC AXPVMS VMS732_RMS V1.0 Patch Install 20-AUG-2004
DEC AXPVMS VMS732_PTHREAD V1.0 Patch Install 20-AUG-2004
DEC AXPVMS VMS732_MOUNT96 V1.0 Patch Install 20-AUG-2004
DEC AXPVMS VMS732_SYS V4.0 Patch Install 22-JUL-2004
DEC AXPVMS VMS732_SYSLOA V2.0 Patch Install 22-JUL-2004
DEC AXPVMS VMS732_LIBRTL V1.0 Patch Install 22-JUL-2004
DEC AXPVMS VMS732_TDF V2.0 Patch Install 22-JUL-2004
DEC AXPVMS DCE V3.1 Full LP Install 21-JUL-2004
DEC AXPVMS DCE V3.0 Full LP Remove 21-JUL-2004
DEC AXPVMS DCE030U01 V1.0 Patch (MUP) Remove 21-JUL-2004
DEC AXPVMS VMS732_MANAGE V2.0 Patch Install 19-JUL-2004
DEC AXPVMS VMS732_DCL V2.0 Patch Install 19-JUL-2004
DEC AXPVMS VMS732_UPDATE V2.0 Patch Install 19-JUL-2004
DEC AXPVMS VMS732_SHADOWING V1.0 Patch Install 19-JUL-2004
DEC AXPVMS VMS732_SYS V3.0 Patch Install 19-JUL-2004
DEC AXPVMS VMS732_FIBRE_SCSI V3.0 Patch Install 19-JUL-2004
DEC AXPVMS TCPIP_ECO V5.4-152 Patch Install 16-JUL-2004
DEC AXPVMS VMS732_F11X V1.0 Patch Install 05-APR-2004
DEC AXPVMS TCPIP_ECO V5.4-151 Patch Install 30-MAR-2004
DEC AXPVMS VMS732_FIBRE_SCSI V1.0 Patch Install 26-MAR-2004
CPQ AXPVMS CPQSNVIEW V2.0-3 Full LP Install 25-MAR-2004
CPQ AXPVMS CPQSNVIEW V2.0-1 Full LP Remove 25-MAR-2004
DEC AXPVMS VMS732_LAN V1.0 Patch Install 25-MAR-2004
DEC AXPVMS FORTRAN V7.6-1 Full LP Install 23-MAR-2004
DEC AXPVMS FORTECO V7.5-2630 Patch Remove 23-MAR-2004
DEC AXPVMS FORTRAN V7.5-1 Full LP Remove 23-MAR-2004
DEC AXPVMS FORRTL V7.6-1 Full LP Install 23-MAR-2004
DEC AXPVMS FORRTL V7.5-1 Full LP Remove 23-MAR-2004
DEC VMS AVAIL_MAN V2.3-1 Full LP Install 22-MAR-2004
DEC VMS AVAIL_MAN V2.3 Full LP Remove 22-MAR-2004
DEC AXPVMS VMS732_SYS V2.0 Patch Install 22-MAR-2004
DEC AXPVMS VMS732_SYS V1.0 Patch Install 22-MAR-2004
DEC AXPVMS VMS732_MANAGE V1.0 Patch Install 16-MAR-2004
DEC AXPVMS COBOL V2.8-1286 Full LP Install 11-MAR-2004
DEC AXPVMS COBOL V2.7-1262 Full LP Remove 11-MAR-2004
DEC AXPVMS COBRTL V2.8-670B Full LP Install 11-MAR-2004
DEC AXPVMS COBRTL V2.7-641B Full LP Remove 11-MAR-2004
DEC AXPVMS VMS732_PCSI V1.0 Patch Install 10-MAR-2004
DEC AXPVMS VMS732_UPDATE V1.0 Patch Install 10-MAR-2004
DEC AXPVMS VMS732_GRAPHICS V2.0 Patch Install 10-MAR-2004
DEC AXPVMS VMS732_RPC V1.0 Patch Install 10-MAR-2004
DEC AXPVMS VMS732_DCL V1.0 Patch Install 10-MAR-2004
CPQ AXPVMS CDSA V2.0-109 Full LP Install 08-MAR-2004
DEC AXPVMS DECNET_OSI V7.3-2 Full LP Install 08-MAR-2004
DEC AXPVMS DWMOTIF V1.3-1 Full LP Install 08-MAR-2004
DEC AXPVMS OPENVMS V7.3-2 Platform Install 08-MAR-2004
DEC AXPVMS TCPIP V5.4-15 Full LP Install 08-MAR-2004
DEC AXPVMS VMS V7.3-2 Oper System Install 08-MAR-2004

Re: HBMM %SYSTEM-F-SHADFEATNOMNT,

Robert Brooks_1 — Tue, 09 Mar 2010 16:57:10 GMT

OK - looked back a little more closely. Have seen the issue twice in the last month. In both cases the first node was starting to boot, and in SYPAGSWPFILES.COM. The second node had not yet started to boot - was still at >>> prompt.

--

Then this makes no sense, because what you effectively have is a one-node cluster. For grins, how 'bout placing a $ SHOW CLUSTER command before attempting to enable HBMM. It almost seems like there is a rogue node hanging around that is voting to prevent the use of HBMM.

Some background -- in order for "enhanced shadowing features" to be used, there is vote taken of all known shadowing drivers in the cluster. The lock manager is used to conduct this poll, and it's something that we spent a lot of time making sure that we got right. I don't remember the exact details (it was almost seven(!) years ago that we did this), but it was designed in such a way that a shadowing driver that had no knowledge of this locking protocol would wind up implicitly withholding their vote, making the ballot measure fail (the only way the votes "passes" is if all nodes in the cluster vote "yes". Any abstention or explicit "no" causes the vote to fail.

It's *possible* that some modifications were added after the initial release such that a newer UPDATE kit would help . . .

-- Rob

Re: HBMM %SYSTEM-F-SHADFEATNOMNT,

Hoff — Tue, 09 Mar 2010 17:25:49 GMT

What sort of device and what sort of interconnect?

What is the general cluster configuration here?

What's happening that you're rebooting the boxes like this? Backups?

How are the boxes being shut down? Direct console HALT, or a controlled shutdown followed by a console HALT?

Issue a SHOW DEVICE SYS$SYSDEVICE /FULL in addition to the SHOW CLUSTER command that Robert suggests, too.

For grins and giggles, I'd stick a WAIT 00:00:05 into the startup between the SET SHADOW and the MOUNT commands, too. This on the off (and wild) chance that HBVS is getting its knickers tangled if the SET SHADOW happens to blocks the MOUNT for some reason.

Yes, loading a newer UPDATE might be an auspicious approach. Six years is a long time.

Re: HBMM %SYSTEM-F-SHADFEATNOMNT,

Benjamin Levy — Wed, 10 Mar 2010 02:26:30 GMT

Re: HBMM %SYSTEM-F-SHADFEATNOMNT,

Robert Brooks_1 — Thu, 11 Mar 2010 05:33:55 GMT

OK - this is a cluster with crossover ethernet cables - pretty straightforward there.

Based on the responses, I added the "SHOW CLUSTER" and the 5-second wait in SYPAGSWPFILES.COM - "SHOW CLUSTER" first, then the 5-second wait. Will monitor the situation and see if any change or interesting output.

--

The five second delay may mask the problem, but make no mistake -- something is quite wrong here, and it's likely in the shadowing code -- either in SHDRIVER itself or the SHADOW_SERVER process.

There were five engineers who worked on the HBMM project; only two of us remain at HP now, and neither one of us is still in VMS Engineering. I conferred with the other remaining engineer, after pointing him to this thread, and he didn't have any other suggestions, other than to remind me that the shadowing driver does have some system disk-specific code that is not exercised for data disks, and it's possible that there is a latent bug, even though we did have an exhaustive list of test cases that involved system disks and HBMM.

I'd suggest contacting HP and opening a call (assuming you have prior version support), but even if you do, the first suggestion will be to upgrade to a more current UPDATE kit.

-- Rob