ioscan -fnC disk Latency

[lhsu]

Hello,
We have trouble with ioscan. "ioscan -fnC disk" takes 8.5 mins to output all the disks.
Our environment consists several rx2620 (Itanium, hpux 11.23) and HSV111 (EVA 5000) SAN. These Itaniums connect to SAN via Febric Switch. The ioscan issue only happens to 2 machines. The other 4 machines only take seconds to report the result. All these machines have the same patch bundles. I even changed cables, that did not help...

Anyone have any idea?

Thanks,

Lynn

[Sandman!]

Check if those two machines are zoned properly if they are zoned at all. Talk to your SAN admins and verify that.

[Torsten.]

Did you present vdisks to the servers in question already?

[Sp4admin]

Hi Lynn,

The ioscan has to scan for all the disk and tape SAN. so if you have a large SAN some times it can take a while. However 8.5 mins is a long time. We have one server that take about 5 mins and we were unable to ever corrrect this issue. He also had HP Support look into this with no results.

sp,

[Pete Randall]

You might want to consider adding the -k switch to your ioscan command so it looks at the kernel structures rather than physically looking at all those devices.

ioscan -kfnCdisk


Pete

[Bill Hassell · Accepted solution]

And if you really want to track down the problem to a given LUN or HBA, issue ioscan -fH and the hardware path. Start with the hardware path to the HBA and run through each one until it slows way down. Then (depending on your free time) keep trying specific LUNs...if all of them are slow, zoning and/or HBA/fabric problems exist. Be sure to run fcmsutil to find any errors on the HBA.

[lhsu]

Hi Everyone,
Thanks for the help. To answer your question - The zoning on the SAN is at Switch level and there are 3 Itaniums in
that storage group. They are all seeing the
same disks. Itm1 has no problem and itm3
and itm4 have ioscan delay. We must use -fnC
for ioscan because our software using that output to do something...

Anyway, our SAN admin told me that in the
past these machines saw "ghost" devices and
they had to clean out the HW path. In my case (my second day in this job:-), I ignited itm3 and itm4 with a hpux 11.23 image. I'd think the "ghost" path should not
present in a new installed machine.

These disks are not configured. Should I try to delete these disks path and rescan in?

The other unknown things are Serviceguard & Cluster if they are interfering with ioscan? I have attached a proc list here. Please take a look and see if you find anything that may cause ioscan delay...

Thanks,

Lynn

[Steven E. Protter]

Shalom Lynn,

Few things to think about.

1) Any machine that does not have LUN0 presented to it may detect the disk array itself as a disk. Except its not and can slow down ioscan and cause EMS errors. Check EMS output to confirm.

2) ioscan can be slowed down by unterminated scsi connections. Check the cables in back of the boxes.

Your secondary questions:


These disks are not configured. Should I try to delete these disks path and rescan in?

Yes.

The other unknown things are Serviceguard & Cluster if they are interfering with ioscan?

Normally Servicegaurd could not interfere with ioscan. I suppose if it had a hung package trying to activate a volume group it might slow things down but not to this degree. Easy enough to test. cmhaltnode and re-run your ioscan.

I have attached a proc list here. Please take a look and see if you find anything that may cause ioscan delay...

If any of your processes have really heavy i/o it could slow down ioscan.

SEP

[Patrick Wallek]

>> We must use -fnC for ioscan because our software using that output to do something...

Well, ideally you should rewrite your software. 'ioscan -fnC disk' will ALWAYS probe ALL buses for disks. It can be very slow if you have a server with lots of FC or SCSI buses or lots of disks.

The better way to do it would be to use 'ioscan -kfnC disk' so that you just query the kernel to get the output of what the kernel knows about. You could also try 'ioscan -funC disk' and see if that helps.

[lhsu]

Hi Bill & everyone,
I ran ioscan to the hw path, every path
gave answer really quick. I don't see errors in EMS. Since I don't have much exp with fcmsutil, please show me how to use that to trouble shoot HBA? how about clear/reset HBA? I attached ioscan output here.

Thanks,

Lynn

[Steven E. Protter]

Shalom,

Check my theory more thoroughly then. A device that the system thinks a disk but is not a disk is probably causing the delay.

SEP

[Sandman!]

Check with your SAN team that the "Host Type" is set to HP-UX (not Windows) on the SAN side. This mis-configuration could also cause delays in the ioscan runtime. Could you post the output of ioscan on the good system? thanks.

[Torsten.]

If you run an "ioscan -H ..." it is of course quick by nature, because if the path is not a SBA or LBA the ioscan will use the "-k" flag automagicly. This means, the result is not a real scan, it comes from the kernel.

See

http://docs.hp.com/en/B2355-60105/ioscan.1M.html

"-H hw_path Restrict the scan and output listing to those devices connected at the specified hardware path. The hardware path must be a bus path. Scanning below the bus level will not probe the hardware and may produce incorrect results"


The scan may be slow if there are device without a presented device to the system, e.g. an EVA without a presented vdisk to this server.

[Torsten.]

From your attached ioscan I can see there are disks presented from the EVA5000, but no disks from the EVA4000 - nad BTW, what is this "MendoCnoArrayCTL" device?

From this 2 storage systems you see only control devices - no disks, this may be the reason for the delay.

[Sandman!]

From the ioscan listing you have posted there are 11 FCP Domains. The HPUX host can see that only the one on hardware path 0/2/1/0.3 is tied to disks.

# ioscan -funC fcp

Ask your SAN admins if they can zone this host only to that one hardware path. IMHO this should substantially reduce your ioscan runtime. Could you attach the ioscan listing from the good system.

~hope it helps

[lhsu]

Hi Torsten & Sendman,
I think we are getting close to the bottom. The good system only shows 1 FCP domain and no HSV 200 ctl. I am having our
SAN admin to look into this now. attach is
an ioscan output from a good system.

Thanks,

Lynn

[Sandman!]

The fact that the good system has only 1 FCP Domain while the very slow one has 11 FCP Domains indicates a zoning issue. More so since the storage is being presented on just one FCP Domain.

Is the good system connected to the same storage array as the bad one? In your last post you mentioned that the bad system is missing the HSV 200 controller. This suggests that the "Host Mode" setting is not set correctly on the SAN side.

Make sure that the SAN admins check the zoning issue and the "Host Mode" setting on the storage array which imho are likely causes of the ioscan latency.

~hope it helps