shutdown/turn off/stop/start utility for FGx device?

[Rich Hearn]

Hi all,

I've done searches on anything I can think of to try to locate a way to shutdown/turn off/stop an FGx device - much like you would with a lan device (mcr scacp stop lan ewa
/ mcr scacp start lan ewa) using a utility. Am I missing "the obvious?" I haven't found anything like that - does it exist or am I forgetting what the name is? I want to upgrade the fibre switch firmware and make sure VMS doesn't use that particular path. I prefer to shut the path down (at the hba) as opposed to "pull the plug" on VMS - even tho' it will "adjust" if I do that.

Tnx for any thoughts,
Rich

[labadie_1]

If there is something, maybe it is with the sda extension fc
$ mc sda
fc
and look at the help

but I am unaware of such a thing (but I have not had such a config recently).

[Robert Brooks_1]

I'm pretty sure that there is no supported way to do that. The closest you can get (and this is rather ugly) is to use the command $ SET DEVICE /PATH = /NOENABLE.

(Yeah, it's /NOENABLE, not /DISABLE)

You'd need to do that for every device that has a path to the switch in question, for every path to that switch. Also, that command won't work on a device if the path you're trying to disable is the current path.

If it was me, I'd trust multipath to do the right thing, so I'd just shut down the switch and let the paths settle down naturally, or if you are a bit paranoid, you could manually switch any devices whose current path is on that switch to a different path. Of course, I'd do this during a maintenance window, not during production hours.

Also, in the future, please mention what version of VMS, as the fibre channel stuff varies dramatically from V7.2 to V8.3, especially for the above-mentioned FC SDA extension.

-- Rob (ex-Multipath engineer)

[Volker Halle]

Rich,

you could also manually issue $ SET DEV/PATH=xxx/SWITCH commands for each device to move it's path away from the HBA, which connects to the FC switch you want to upgrade.

Volker.

[The Brit]

Rich,
I think you should also know that "$ mc scacp stop lan ews" doesn't actually turn off the ewa device, it just disables it for SCS Cluster communications.

(Could be wrong though!)

I frequently used to execute this command even though my primary network adaptor was an EWA device, when I didn't want the cluster to use the main network for cluster traffic.

Dave.

[Rich Hearn]

Whew! Thank you all for the thoughts!

Labadie, yes, the fc command is available, but it doesn't allow you to disable/shutdown/stop/noenable an fg device - that I could find.

Rob, you're right - I should have put the version #'s in - *I* knew what they were, didn't everyone :^) (actually, never thought of it when I wrote - my mistake)

SDA>
SDA> fc show dev
FGA0: operational firmware revision CS3.93A0
port_name(adapter_id) = 1000-0000-C935-C185, node_name(host_id) = 2000-0000-C935-C185
SDA>
SDA> exit
CACHE1::DISK$INFSYS:[RJHEARN]_>uptime
OpenVMS V7.3-2 on node CACHE1 1-AUG-2007 09:43:17.20 Uptime 261 23:35:27
CACHE1::DISK$INFSYS:[RJHEARN]_>

Trusting multipath is not a problem, just trying to "cleanly" make the change in paths before I shut off the switch port. Doing a /noenable against 30 devices is a bit "less elegant" than I had hoped for, but could be done. I was hoping for a single cmd in case I needed to turn it back on in a hurry (assuming the upgrade is complete - bit "sticky" if needed in the middle of the upgrade :^)

Volker, I had done a "set dev/switch/path=PGA0.5000-1FE1-5000-C0Fx $1$DgaXX" - (for all my devices) which worked, but around midnight, the system automatically changed it back to the other path because it was available. I was looking to temporarily, "permanently" disable the other path so I could do the upgrade to the switch.

Dave, the distinction you point out is understood - I'm basically just looking for a way to tell vms "ignore" fg device X

Thank you *all* for your thoughts & perspectives - I do appreciate them.

Rich

[Robert Brooks_1]

Volker, I had done a "set dev/switch/path=PGA0.5000-1FE1-5000-C0Fx $1$DgaXX" - (for all my devices) which worked, but around midnight, the system automatically changed it back to the other path because it was available.

--

That's not how multipath works.

While multipath *will* automatically switch away from the MSCP path under certain conditions, multipath doesn't move away from a local path unless provoked by an I/O error or DCL command.

Perhaps there was some disruption in your SAN around midnight?


-- Rob

[Rich Hearn]

Rob,

Thanks for the info - I'll go back & see if I can find any msgs in the console logfile - I have (at other times in the past) seen pga0/pgb0 errors, but have not been able to track them down to anything specific. Hp suggested it could be a timing issue that we're seeing. no real answer, but plausible - and difficult to prove.

Tnx,
Rich

[Robert Brooks_1]

Hp suggested it could be a timing issue that we're seeing. no real answer, but plausible - and difficult to prove.

--

I'm not sure what my fellow HP colleague means by that. Without going into a long, boring explanation as to how multipath works, suffice it to say that a switch from a local path to another local path requires one of two things:

1) the DCL command to change paths

or

2) an I/O error that provokes mount verification. Multipath (usually) works in the context of mount verification to do its work. Note that there can be transient disruptions in a SAN such that an I/O can fail, yet succeed immediately upon retry. In that case, however, multipath will attempt to stay on the current path, and not switch paths.

Also note that with V7.3-2, the concept of suppressed mount verification messages was introduced; by default, certain messages are not emitted to OPA0: and OPERATOR.LOG

-- Rob

[Rich Hearn]

I went back and checked regarding what I wrote - using the console buffer - I have to make a correction. It was pea0/peb0 that I was thinking of - not pga0/pgb0. My apologies to your colleagues for my memory. Here's what I was thinking of:

%PEA0, Excessive packet losses on LAN Path from EIA to EIA on REMOTE NODE IDXT2

%PEA0, Excessive packet losses on LAN Path from EWA to EWA on REMOTE NODE IDXT2

%CNXMAN, Lost connection to system IDXT2
%SHADOW-I-VOLPROC, DSA2: is offline. Volume Processing in progress.

%SHADOW-I-VOLPROC, DSA2: is offline. Volume Processing in progress.

%CNXMAN, Re-established connection to system IDXT2
%SHADOW-I-VOLPROC, DSA2: has completed volume processing.

%PEA0, Excessive packet losses on LAN Path from EWA to EWA on REMOTE NODE IDXT2

%PEA0, Excessive packet losses on LAN Path from EWA to EWA on REMOTE NODE IDXT2

%%%%%%%%%%% OPCOM 1-AUG-2007 00:01:00.55 %%%%%%%%%%%

This "view" is from IdxT1's console. Would the "Lost Connection" to IdxT2 be indicative of a mount verify that would cause the multipath switch over? How 'bout a pointer to where I can find info to turn on mount verify msgs - doesn't appear to be in the Mount command - or I'm just missing it...

attached is a .txt file detailing our connectivity.

Rich

[Volker Halle · Accepted solution]

Rich,

suppression of mount-verification messages is controlled by SYSGEN parameters:

$ MC SYSGEN HELP SYS_P MVSUPMSG

There seems to have been some problem with both network pathes (EWA and EIA) between IDXT1 and IDXT2.

A 'Lost connection' message is shown, if the local node does NOT receive any cluster hello multicast message from the remote node within about 9 seconds. Each node sends such a message about every 3 seconds.

This can indicate a network problem or a problem in the remote node. Check the console of the other cluster members to see, which node had lost connection to which other node. This can help you find out, what may have been happening.

Volker.

[Volker Halle]

Rich,

you can also control the suppression of mount-verification messages individually per disk device with:

$ SET DEV /MV_INTVL=n /MV_NUM=n

AFAIK, multipath switched messages will not be suppressed - but Rob will correct me if I'm wrong.

Volker.

[Rich Hearn]

Volker,

Thank you for the information on the mount verify msg. I've placed it away for our next downtime.

Rich

[Volker Halle]

Rich,

these are DYNAMIC parameters...

Volker.

[Rich Hearn]

Volker,

Thank you for that information... I haven't had a chance to go "visit" sysgen yet to see whether the parameters were dynamic or not, but now I know I don't have to wait 'til November :^)

tnx agn,
Rich

[Robert Brooks_1]

AFAIK, multipath switched messages will not be suppressed - but Rob will correct me if I'm wrong.

--

Volker, as always, is correct; it's only the mount verification begin/end messages that are suppressed. The multipath-generated messages signaling a path switch will alway be emitted.

-- Rob

[Volker Halle]

Rich,

now that you've learned about mount verification message suppression, go back to your console logs or OPERATOR.LOG and try to find out WHEN the multipath switch has taken place in your system. You should see the manual path switch (from your SET DEV/PATH=.../SWITCH) and also any automatic path switch, which changed the path you had set.

Volker.

[Robert Brooks_1]

If you can stomach the mount verification messages, I highly recommend disabling the suppression. It's a real pain to troubleshoot an issue (or to determine if there really is a problem) with MV suppression enabled.

Fortunately, the MV suppression can be disabled dynamically.

-- Rob

[Rich Hearn]

Volker & Robert,

Sorry for the delay, I've been doing exactly what you've both suggested; checking on what's happening. I've turned on mount verification msgs & the only time they're appearing is when I turn off EWA0 every night to check the counters & reset them. I MSCP serve each systems system disk to the other system and those are the disks that "mount verify". Since I'm not specifically checking & reassigning cluster comm to EWB0, I figure it's probably the reason for it. Please let me know if I'm wrong on that.

Rich

[Volker Halle]

Rich,

what makes you turn off a SCA-speaking LAN device at midnight ? How many SCA channels do you have (MC SCACP SHOW CHAN) between those nodes ?

According to the console log provided earlier, both EIA-EIA and EWA-EWA lost communications at that time, causing a connection lost event. This will throw the disks into mount-verification.

Volker.

[Rich Hearn]

Apologies for the delay and confusion; I was responding on the 9th, when the site "disappeared" & I'm just getting back to it now...

A long time ago, we were getting errors on our sca comms, so I put together a dcl .com to check the counters, reset the counters, stop EWA, then restart EWA. Also the same for EWB.

This was done to show the uptime for the interfaces. I've left the .com running at 04:01 just to have it keep track of error counters. I did have a wait 00:00:01 between the start of EWA0 & the stop of EWB0 - perhaps not enough time? I don't shut down the EIA interfaces, so I'm not sure why they both would be. Obviously, I must missing an integral piece of info in my understanding. Any assistance in this regard would be appreciated.

The attached .txt is the output of mcr scacp show chan for both clusters.

Thanks for your thoughts,
Rich

[Volker Halle]

Rich,

what problem are we chasing now ?

The console output from the IDXT1/2 cluster provided earlier was from midnight, not 04:01. The EIA-EIA channel in this cluster is up since 10-MAY. Did the VC drop ($ MC SCACP SHOW VC) ?

Volker.

[Rich Hearn]

Volker,

We are not chasing a different problem - I had been responding to your question regarding the sca circuits. Please do not feel obligated regarding this path my question has gone down. It was not my intention to have it be this confusing. I was, originally, trying to shut off my hba interface because multipath switched automatically around midnight and I didn't know why. That lead me to the rest of this note. My apologies for digressing.

I appreciate your time & willingness already given, I'm not wishing to impose on it any further. We can let this stand as is.

Thanks again,
Rich

[Volker Halle]

Rich,

thanks for the clarification.

Could you show both the multi-path switched messages and the CNXMAN messages from the same time ? If they are not happening at the same time, I would see no correlation between them.

Volker.

[Rich Hearn]

Volker,

Thanks for your persistance. I'm delayed in responding because I took yesterday to set up the same situation as originally, but with the mount verify msgs enabled. The two *are* unrelated. The Auto-switch occurred at approx 00:01 and the CNXMAN msgs occurred on the other cluster at approx 04:01. No CNXMAN msgs occurred on the cluster with the auto-switch.

I'll just accept the fact that it'll auto-switch and go ahead & do the upgrades as I had originally wanted

btw: the last few days, getting a response from this site's been slooow - is it just me or have you seen it too?

Tnx,
Rich