ioscan

[antonio difruscia]

I'm running HP-UX 11.23 with all the updated patches on an rx2600. when I boot or run ioscan it takes a long time (8 minutes to complete an ioscan).. after the ioscan completes all devices are claimed, I even ran
dd to all the devices with no problems.I am connected to an EVA3000 with luns presented to this node, and there is no hardware failures. also in the SAN there are 2 other EVAs with no luns presented to the node. Is there a way to run ioscan in a verbose mode so I can see what the node getting hung on during the scan ?

thanks
TonyD

[My_Server]

You can run the last config by excute ioscan -fnkC disk .
it's take just 1 or 2 sc

[Andrew Merritt_2]

You could run it under tusc, and see where it is taking the time, e.g:

tusc -rall -wall -l -f -p -a -E -v -T%T /usr/sbin/ioscan -o /tmp/tusc.out

Those options should be enough to let you see where the time is being taken, and which device it is happening for.

Andrew

[antonio difruscia]

running funC doesn't help me because the systems run reboot tests and the system hangs when it does a ioscan on bootup, or when you run ioscan when they system is up
so I need to see what hangs the ioscan. The system does not have tusc loaded on it, so i'll see if I can locate tusc and try running that, it would be nice if they had
a verbose flag for ioscan.

thanks
TonyD

[Torsten.]

Hi,

on a large SAN the process could take some time. If you don't use the other 2 EVAs, you should prevent the access by adjusting your zoning and make the zone as small as possible.
This could speed up the discover process.

Another point to look at are failures in transmitting the data. You should look at your switch logs and search for high error rates.

[antonio difruscia]

there are no zones my san setup is a
2 core switches (4/16 brocade) with 8 edge
switches (2/16 brocade) the only error being
reported on the switches is a arptnew error
which I think is related to the switch trying to do a reverse arp to a system that is on a network and the switch can not reach
that node.

thanks
TonyD

[spex]

Hi Tony,

You could partition the ioscan by scanning a particular device or subset of devices at a time. Then, whichever scan(s) take(s) an inordinate amount of time is/are most likely the culprit(s).

For example, scan one hardware path at a time:
# ioscan -fnH 10/16/12.0.0
# ioscan -fnH 10/16/12.1.0
# ioscan -fnH 10/16/12.2.0

Or scan one device class at a time:
# ioscan -fnC ctl
# ioscan -fnC ext_bus
# ioscan -fnC lan

For additional clues, observe system tables in Glance during the scans.

PCS

[Andrew Merritt_2]

One place to get tusc is here:
http://hpux.connect.org.uk/hppd/hpux/Sysadmin/tusc-7.8/

Andrew

[Michael Steele_2]

Dear Tony:

This is a familiar problem to me. I have a rx2600 attached to a SAN / EMC disk array and this periodically happens. And it is always related to the disk array or SAN.

Run a 'insf -e' and you should be able to id the culprit device. It will be a fairly long report so be able to capture a couple of hundred lines.

[Andrew Young_2]

Hi Tony

Please check if you have patch PHKL_33361 installed. We have had similar problems with an rx2620 and rx4640 connecting to MSA-1500's, one a direct connect and the other through a SAN, but both with the same problem.

Removing that patch solve these problems.

Alas it will mean a reboot.

Regards

Andrew Y

[Andrew Young_2]

You might also want to look at this thread:

https://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1031711

[Sp4admin]

Hello,

some times if you have exteranl disk this can cause the ioscan to take some time. does. all the devices come back caime

[Torsten.]

Look at the thread mentioned by Andrew.

The solution says
"Problem solved by modifying zone set configuration"

Especially in a large SAN you should consider zoning.

Without zoning, almost every devices talks to every other device. There is a lot of communication (let's call it "noise") in your SAN!

Every zone should only include a server and his storage (if possible). This help to make the SAN more easy, robust, fast and safe.

[antonio difruscia]

running ioscan -fnH on each hardware path
ran fine:

# time ioscan -fnH 0/5/1/0.5.15.0.0.0.1
Class I H/W Path Driver S/W State H/W Type Description
===========================================================================
disk 84 0/5/1/0.5.15.0.0.0.1 sdisk CLAIMED DEVICE HP HS0

real 0m0.03s
user 0m0.00s
sys 0m0.03s

but if I run ioscan -fnC lan or disk it takes over 8 minutes.

TonyD

[Torsten.]

ioscan -fnH does in fact display the kernel information (like "-k"), unless the target is a LBA or SBA.

See "man ioscan" for more information.

This is fast by default, you cannot compare this with an "life" scan ;-)

[antonio difruscia]

using the tusc command I get: out put in
attachment so it is readable. does anyone know what:
ioctl(4, IOCONF_IO_SCAN, 0x7ffff540)
points to ?

thanks
TonyD

[antonio difruscia]

I do have patch PHKL_33361 installed:

Is it documented anywhere about the removal of this patch ?

thanks
TonyD

[Michael Steele_2]

Dear Tony:

You say this only happens on lan and disk ioscans, are you running NFS?

Can you id any tape issues? Try ioscan -fnC tape.

Please attach your complete 'ioscan' and these:

sar -d 5 5
sar -uM 5 5
insf -e > file

[Andrew Young_2]

Hi.

Details on patch PHKL_33361 appear here:

http://www5.itrc.hp.com/service/cki/patchDocDisplay.do?patchId=PHKL_33361

It seems that you need to ensure that at least PHKL_31507 is installed.

However PHKL_33361 IS required if you are running a Superdome connected to EMC storage. Note that this patch fixes a memory leak in ioscan so running ioscan too often without this patch could cause problems.

So the debate around this patch is... if you need to run ioscan regularly are you prepared to deal with the memory leak vs performance.

Regards

Andrew Y

[antonio difruscia]

Mike, there are no tapes. files are attached
I have also found that more than just this
system showing the same problem. its when
ever you run ioscan without reading from kernel or on a bootup it takes 8 minutes.

thanks
TonyD

[Michael Steele_2]

Since its happening on other servers you've probably got a SAN switch or zone problem.

a) If happening on all then disk array.

Here's some trouble shooting guidelines for a Brocade Switch. Note: I'm not sure what kind of switch you have, it just a suggestion.

Table 10-1 Common Troubleshooting Problems and Tools

Problem Area
Fabric â ¢ Missing devices


Investigate
â ¢ Marginal links (unstable
connections)
â ¢ Incorrect zoning configurations
â ¢ Incorrect switch configurations

Tools

â ¢ Switch LEDs
â ¢ Switch commands
switchShow or
diagnostics
â ¢ Web or GUI-based
and management


Storage Devices â ¢ Physical issues between switch
and devices
â ¢ Incorrect storage software
configurations


Tools
â ¢ Device LEDs
â ¢ Storage diagnostic
â ¢ Switch commands
switchShow or
diagnostics

[antonio difruscia]

We have no zones, so any node can get to any
storage (we have 2 EVA3000s and 1 EVA8000)
so it doesn't matter where you plug in. the
san design is out of the san design handbook. have 2 core switches (brocade 4/16) 8 edge switches (brocade 2/16) all with updated firmware. the EVAs do not show
any error, the only error the switches show is a arptnew error, which I believe is related to it trying a reverse arp that is failing. when the node does finish a bootup,
no errors are displayed and all disks are
good, ran a dd test for all drives, no issues. I was also watching perf monitor on the EVA and nothing was getting overloaded. I am going to setup up Fabric Manager and see if that can spot anything.

[Sandman!]

Unplug the external storage cables and see how long the ioscan takes...to isolate the problem. How long does the "ioscan -funC disk" command takes?

[antonio difruscia]

something is running on the system so I can not unplu it right now, but I will when I get a chance. ioscan -funC comes back quickly, no problems when reading from the kernel

TonyD

[Sandman!]

"ioscan -funC disk" is not reading the kernel unless you supply the -k switch to it as in "ioscan -funkC disk". If "ioscan -funC disk" is fine try "ioscan -funC lan"