topic Re: shutdown/turn off/stop/start utility for FGx device? in Operating System - OpenVMS

shutdown/turn off/stop/start utility for FGx device?

Rich Hearn — Tue, 31 Jul 2007 14:36:38 GMT

Hi all,

I've done searches on anything I can think of to try to locate a way to shutdown/turn off/stop an FGx device - much like you would with a lan device (mcr scacp stop lan ewa
/ mcr scacp start lan ewa) using a utility. Am I missing "the obvious?" I haven't found anything like that - does it exist or am I forgetting what the name is? I want to upgrade the fibre switch firmware and make sure VMS doesn't use that particular path. I prefer to shut the path down (at the hba) as opposed to "pull the plug" on VMS - even tho' it will "adjust" if I do that.

Tnx for any thoughts,
Rich

Re: shutdown/turn off/stop/start utility for FGx device?

labadie_1 — Tue, 31 Jul 2007 14:39:52 GMT

If there is something, maybe it is with the sda extension fc
$ mc sda
fc
and look at the help

but I am unaware of such a thing (but I have not had such a config recently).

Re: shutdown/turn off/stop/start utility for FGx device?

Robert Brooks_1 — Tue, 31 Jul 2007 20:21:49 GMT

I'm pretty sure that there is no supported way to do that. The closest you can get (and this is rather ugly) is to use the command $ SET DEVICE /PATH = /NOENABLE.

(Yeah, it's /NOENABLE, not /DISABLE)

You'd need to do that for every device that has a path to the switch in question, for every path to that switch. Also, that command won't work on a device if the path you're trying to disable is the current path.

If it was me, I'd trust multipath to do the right thing, so I'd just shut down the switch and let the paths settle down naturally, or if you are a bit paranoid, you could manually switch any devices whose current path is on that switch to a different path. Of course, I'd do this during a maintenance window, not during production hours.

Also, in the future, please mention what version of VMS, as the fibre channel stuff varies dramatically from V7.2 to V8.3, especially for the above-mentioned FC SDA extension.

-- Rob (ex-Multipath engineer)

Re: shutdown/turn off/stop/start utility for FGx device?

Volker Halle — Wed, 01 Aug 2007 00:30:03 GMT

Rich,

you could also manually issue $ SET DEV/PATH=xxx/SWITCH commands for each device to move it's path away from the HBA, which connects to the FC switch you want to upgrade.

Volker.

Re: shutdown/turn off/stop/start utility for FGx device?

The Brit — Wed, 01 Aug 2007 05:38:49 GMT

Rich,
I think you should also know that "$ mc scacp stop lan ews" doesn't actually turn off the ewa device, it just disables it for SCS Cluster communications.

(Could be wrong though!)

I frequently used to execute this command even though my primary network adaptor was an EWA device, when I didn't want the cluster to use the main network for cluster traffic.

Dave.

Re: shutdown/turn off/stop/start utility for FGx device?

Rich Hearn — Wed, 01 Aug 2007 09:11:11 GMT

Whew! Thank you all for the thoughts!

Labadie, yes, the fc command is available, but it doesn't allow you to disable/shutdown/stop/noenable an fg device - that I could find.

Rob, you're right - I should have put the version #'s in - *I* knew what they were, didn't everyone :^) (actually, never thought of it when I wrote - my mistake)

SDA>
SDA> fc show dev
FGA0: operational firmware revision CS3.93A0
port_name(adapter_id) = 1000-0000-C935-C185, node_name(host_id) = 2000-0000-C935-C185
SDA>
SDA> exit
CACHE1::DISK$INFSYS:[RJHEARN]_>uptime
OpenVMS V7.3-2 on node CACHE1 1-AUG-2007 09:43:17.20 Uptime 261 23:35:27
CACHE1::DISK$INFSYS:[RJHEARN]_>

Trusting multipath is not a problem, just trying to "cleanly" make the change in paths before I shut off the switch port. Doing a /noenable against 30 devices is a bit "less elegant" than I had hoped for, but could be done. I was hoping for a single cmd in case I needed to turn it back on in a hurry (assuming the upgrade is complete - bit "sticky" if needed in the middle of the upgrade :^)

Volker, I had done a "set dev/switch/path=PGA0.5000-1FE1-5000-C0Fx $1$DgaXX" - (for all my devices) which worked, but around midnight, the system automatically changed it back to the other path because it was available. I was looking to temporarily, "permanently" disable the other path so I could do the upgrade to the switch.

Dave, the distinction you point out is understood - I'm basically just looking for a way to tell vms "ignore" fg device X

Thank you *all* for your thoughts & perspectives - I do appreciate them.

Rich

Re: shutdown/turn off/stop/start utility for FGx device?

Robert Brooks_1 — Wed, 01 Aug 2007 09:29:45 GMT

Volker, I had done a "set dev/switch/path=PGA0.5000-1FE1-5000-C0Fx $1$DgaXX" - (for all my devices) which worked, but around midnight, the system automatically changed it back to the other path because it was available.

--

That's not how multipath works.

While multipath *will* automatically switch away from the MSCP path under certain conditions, multipath doesn't move away from a local path unless provoked by an I/O error or DCL command.

Perhaps there was some disruption in your SAN around midnight?

-- Rob

Re: shutdown/turn off/stop/start utility for FGx device?

Rich Hearn — Wed, 01 Aug 2007 09:48:25 GMT

Rob,

Thanks for the info - I'll go back & see if I can find any msgs in the console logfile - I have (at other times in the past) seen pga0/pgb0 errors, but have not been able to track them down to anything specific. Hp suggested it could be a timing issue that we're seeing. no real answer, but plausible - and difficult to prove.

Tnx,
Rich

Re: shutdown/turn off/stop/start utility for FGx device?

Robert Brooks_1 — Wed, 01 Aug 2007 11:46:12 GMT

Hp suggested it could be a timing issue that we're seeing. no real answer, but plausible - and difficult to prove.

--

I'm not sure what my fellow HP colleague means by that. Without going into a long, boring explanation as to how multipath works, suffice it to say that a switch from a local path to another local path requires one of two things:

1) the DCL command to change paths

or

2) an I/O error that provokes mount verification. Multipath (usually) works in the context of mount verification to do its work. Note that there can be transient disruptions in a SAN such that an I/O can fail, yet succeed immediately upon retry. In that case, however, multipath will attempt to stay on the current path, and not switch paths.

Also note that with V7.3-2, the concept of suppressed mount verification messages was introduced; by default, certain messages are not emitted to OPA0: and OPERATOR.LOG

-- Rob

Re: shutdown/turn off/stop/start utility for FGx device?

Rich Hearn — Thu, 02 Aug 2007 08:01:51 GMT

I went back and checked regarding what I wrote - using the console buffer - I have to make a correction. It was pea0/peb0 that I was thinking of - not pga0/pgb0. My apologies to your colleagues for my memory. Here's what I was thinking of:

%PEA0, Excessive packet losses on LAN Path from EIA to EIA on REMOTE NODE IDXT2

%PEA0, Excessive packet losses on LAN Path from EWA to EWA on REMOTE NODE IDXT2

%CNXMAN, Lost connection to system IDXT2
%SHADOW-I-VOLPROC, DSA2: is offline. Volume Processing in progress.

%SHADOW-I-VOLPROC, DSA2: is offline. Volume Processing in progress.

%CNXMAN, Re-established connection to system IDXT2
%SHADOW-I-VOLPROC, DSA2: has completed volume processing.

%PEA0, Excessive packet losses on LAN Path from EWA to EWA on REMOTE NODE IDXT2

%PEA0, Excessive packet losses on LAN Path from EWA to EWA on REMOTE NODE IDXT2

%%%%%%%%%%% OPCOM 1-AUG-2007 00:01:00.55 %%%%%%%%%%%

This "view" is from IdxT1's console. Would the "Lost Connection" to IdxT2 be indicative of a mount verify that would cause the multipath switch over? How 'bout a pointer to where I can find info to turn on mount verify msgs - doesn't appear to be in the Mount command - or I'm just missing it...

attached is a .txt file detailing our connectivity.

Rich

Re: shutdown/turn off/stop/start utility for FGx device?

Volker Halle — Thu, 02 Aug 2007 08:25:30 GMT

Rich,

suppression of mount-verification messages is controlled by SYSGEN parameters:

$ MC SYSGEN HELP SYS_P MVSUPMSG

There seems to have been some problem with both network pathes (EWA and EIA) between IDXT1 and IDXT2.

A 'Lost connection' message is shown, if the local node does NOT receive any cluster hello multicast message from the remote node within about 9 seconds. Each node sends such a message about every 3 seconds.

This can indicate a network problem or a problem in the remote node. Check the console of the other cluster members to see, which node had lost connection to which other node. This can help you find out, what may have been happening.

Volker.

Re: shutdown/turn off/stop/start utility for FGx device?

Volker Halle — Thu, 02 Aug 2007 08:34:09 GMT

Rich,

you can also control the suppression of mount-verification messages individually per disk device with:

$ SET DEV /MV_INTVL=n /MV_NUM=n

AFAIK, multipath switched messages will not be suppressed - but Rob will correct me if I'm wrong.

Volker.

Re: shutdown/turn off/stop/start utility for FGx device?

Rich Hearn — Thu, 02 Aug 2007 09:21:32 GMT

Volker,

Thank you for the information on the mount verify msg. I've placed it away for our next downtime.

Rich

Re: shutdown/turn off/stop/start utility for FGx device?

Volker Halle — Thu, 02 Aug 2007 09:24:27 GMT

Rich,

these are DYNAMIC parameters...

Volker.

Re: shutdown/turn off/stop/start utility for FGx device?

Rich Hearn — Thu, 02 Aug 2007 09:36:15 GMT

Volker,

Thank you for that information... I haven't had a chance to go "visit" sysgen yet to see whether the parameters were dynamic or not, but now I know I don't have to wait 'til November :^)

tnx agn,
Rich

Re: shutdown/turn off/stop/start utility for FGx device?

Robert Brooks_1 — Thu, 02 Aug 2007 19:07:01 GMT

AFAIK, multipath switched messages will not be suppressed - but Rob will correct me if I'm wrong.

--

Volker, as always, is correct; it's only the mount verification begin/end messages that are suppressed. The multipath-generated messages signaling a path switch will alway be emitted.

-- Rob

Re: shutdown/turn off/stop/start utility for FGx device?

Volker Halle — Fri, 03 Aug 2007 00:40:56 GMT

Rich,

now that you've learned about mount verification message suppression, go back to your console logs or OPERATOR.LOG and try to find out WHEN the multipath switch has taken place in your system. You should see the manual path switch (from your SET DEV/PATH=.../SWITCH) and also any automatic path switch, which changed the path you had set.

Volker.

Re: shutdown/turn off/stop/start utility for FGx device?

Robert Brooks_1 — Fri, 03 Aug 2007 18:36:27 GMT

If you can stomach the mount verification messages, I highly recommend disabling the suppression. It's a real pain to troubleshoot an issue (or to determine if there really is a problem) with MV suppression enabled.

Fortunately, the MV suppression can be disabled dynamically.

-- Rob

Re: shutdown/turn off/stop/start utility for FGx device?

Rich Hearn — Thu, 09 Aug 2007 09:43:58 GMT

Volker & Robert,

Sorry for the delay, I've been doing exactly what you've both suggested; checking on what's happening. I've turned on mount verification msgs & the only time they're appearing is when I turn off EWA0 every night to check the counters & reset them. I MSCP serve each systems system disk to the other system and those are the disks that "mount verify". Since I'm not specifically checking & reassigning cluster comm to EWB0, I figure it's probably the reason for it. Please let me know if I'm wrong on that.

Rich

Re: shutdown/turn off/stop/start utility for FGx device?

Volker Halle — Thu, 09 Aug 2007 09:59:53 GMT

Rich,

what makes you turn off a SCA-speaking LAN device at midnight ? How many SCA channels do you have (MC SCACP SHOW CHAN) between those nodes ?

According to the console log provided earlier, both EIA-EIA and EWA-EWA lost communications at that time, causing a connection lost event. This will throw the disks into mount-verification.

Volker.