ioscan/insf not discovering devices (MC/SG)

[David Child_1]

I'm having a bit of an issue concerning the discovery of new devices. Basically whenever I create new LUNs on an EMC DMX1000 and present them to our hosts, an ioscan/insf does not discover them. If I reboot, the devices do show up as expected.

Servers: 32 way SuperDome nPartitions running MC/SG OPS Edition (supporting Oracle 9i RAC).

There are 4 nPars in two separate clusters and all 4 are experiencing this problem. The other 4 nPars do not have MC/SG on them and appear to work correctly when running ioscan/insf.

I have attached patch list and kernel parms for one of the nPars.

Any suggestions would be greatly appreciated.

Thanks,
David

[Hai Nguyen_1]

Dave,

How did you run ioscan and insf? Like this:

# ioscan -fnC disk
# insf -e

Hai

[David Child_1]

I tried all of the following:

ioscan
ioscan -f
ioscan -fnCdisk
ioscan -fCdisk

After each I tried 'insf' and 'insf -e'

I don't believe its procedural since I use the same method on the non-MC/SG nPars and they work fine. I have not experienced this issue on other MC/SG servers we have either (those are V-2250s running 11.00 connected to an EMC 8830).

[Robert Holliday_1]

Dave,

Contact EMC and make sure the disk are enabled internally. If they are not enabled you will not see the them on your system.

[Denver Osborn]

I think it's important that we point out David said

"If I reboot, the devices do show up as expected."

So the key is... what's not happing with the box, fibre or emc after the LUN is created. Wish I had a clue, still thinking! :)

We're also comparing two differnent configs. David said it works on the V-class w/ an EMC 8830 @ 11.0, but doesn't on the rp8400 w/ DMX1000 @ 11i.

You have the latest FC driver, right?

Have you looked at the fc switch? Maybe the f/w on the switch??

Might be able to rule out the switch if it's the same/similar to what's being used with the V and 8830.

-denver

[Stuart Abramson_2]

On the client servers run EMC commands:

symcfg discover
fpath refresh / symmask refresh

then run:

ioscan
insf -e

then run, again:

symcfg discover

powermt config

[David Child_1]

Thanks for all the replies.

Robert: As Denver pointed out "If I reboot, the devices do show up as expected."

Denver: The one difference between the V-class servers and the SD32000 is that the V-classes are using A5158A HBAs and the SD32000 nPars are using A6795As. The driver version are current. We also have recently upgraded the firmware on the directors (McData). I am not going to rule out anything in the firmware/driver area yet.

Stuart: Yes, I have already tried symcfg discover, symmask refresh without luck. This server doesn't have PowerPath (don't ask me why we didn't spring for it considering how much $$ was put into it)

This issue will most likely have something to do with MC/ServiceGuard. There are two 32 way domes and each has 4 nPars. 2 nPars on each 32 way cabinet have MC/SG running (clustered with the nPars on the other cabinet) and 2 non-MC/SG npars. All nPars are running the same OS and patch set and are connected to the same SAN directors and EMC frame. Only the MC/SG nPars are experiencing the issue. I can discover newly assigned LUNs on the non-MC/SG nPars easily.

[Denver Osborn]

That's interesting that the non MC/SG nPars don't experience the problem. I would've assumed that the app wouldn't affect the system's ability to see a new fc attached lun. Unless patches/drivers diff from MC/SG is it...

Can you pick apart other similarities between an MC/SG nPar and a non-MC/SG nPar. Take a look at the kernel and drivers, patches, etc.

-denver

[Jeff Schussele]

Hi David,

Proper ioscan command syntax would be:

ioscan -fnC disk

Then if you see devices reported w/o actual device file, run:

insf

You don't need the -e.

HTH,
Jeff

[Jeff Schussele]

I should add that I navigate to /dev/dsk & then do an ll -t to see the newest files up top.
If you see files with current date/times, you're in business.

Rgds,
Jeff

[David Child_1]

Just an update on this in case anyone runs into this problem.

In the end the problem was a bug in the EMC WideSky code. It was hard to determine because the problem was not consistent. Some hosts would see the newly assigned storage while others would not. The EMC WideSky code in question was version 5.1 on the host where the VCM database updates where made.

Basically there is a bug in WideSky 5.1. When you make changes to the VCM database via 'symmask' commands (or via ECC) on a DMX Symmetrix, ioscan on the host where the LUNs were assigned does not pick up the new LUNs.

To fix this problem, upgrade WideSky on the Symmetrix agent host (where ECC Symmetrix is running or you run 'symmask' commands from) to version 5.3.

David

[Daryl Much]

I ran into a similar problem also, having to reboot to acquire new emc luns. one work around for us was to use rad to disable/enable the fc adapters, first one then the other. powerpath was not bothered by the brief disconnection to one path and I doubt if pv links would care either (but it should switch links). here's the sequence of commands I used:

ioscan -fnC fcp #get the path of the fibre cards
fcp 3 0/0/8/0/0.97 fcp CLAIMED INTERFACE FCP Domain
fcp 2 1/0/6/0/0.33 fcp CLAIMED INTERFACE FCP Domain

rad -q #get slot associated with path
rad -s 0-0-0-1 #suspend driver on the slot
rad -o 0-0-0-1 #power off the slot
rad -i 0-0-0-1 #power on the slot
rad -r 0-0-0-1 #resume the driver
insf -v #install the device files

[David Child_1]

In the end the problem was a bug in the EMC WideSky code. Please refer to my second to last post for details.

Thanks,
David