HPE Community read-only access December 15, 2018
This is a maintenance upgrade. You will be able to read articles and posts, but not post or reply.
Hours:
Dec 15, 4:00 am to 10:00 am UTC
Dec 14, 10:00 pm CST to Dec 15, 4:00 am CST
Dec 14, 8:00 pm PST to Dec 15, 2:00 am PST
System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

insf is slow on some systems but not on others...

 
SOLVED
Go to solution
Kevin Lister
Frequent Advisor

insf is slow on some systems but not on others...

Hi Gang,

I've noticed that using "insf -e -H" on some of my HP systems is very slow while on others it's very fast. On a typical system each insf command takes less than 1 second. On some systems it takes several seconds. This occurs when I'm installing device files for hardware paths coming from my va7400's.

All of my OS's are 11.23. My servers consist of N4000's and A500's. The slow behavior happens on both models. So it isn't OS specific or hardware specific.

Anyone else see this behavior?

Thanks in advance for taking the time to answer. ;o)

kev
14 REPLIES
Mark McDonald_2
Trusted Contributor

Re: insf is slow on some systems but not on others...

Are the slow boxes heavily utilised by databases or apps?
Kevin Lister
Frequent Advisor

Re: insf is slow on some systems but not on others...

Nope. This problem is not I/O performance related. The boxes are essentially quiescent as I am in the process of setting up storage on them. So, nobody is using them but me when I'm setting them up.

kev
Avinash20
Honored Contributor

Re: insf is slow on some systems but not on others...

Hi Kevin,

Is ioscan is having any devices in NO_HW, we have a known issue

Please check if you are having the patch PHCO_34780 or superseeded one(PHCO_37930) installed on the server

( QX:QXCR1000546202 SR:8606396757 CR:JAGaf56740 )
On IPF platforms, even if disk class devices are already
known to be in NO_HW state (H/W State of devices as seen
with ioscan(1M)), insf(1M) can take a long time to complete.

"Light travels faster than sound. That's why some people appear bright until you hear them speak."
Avinash20
Honored Contributor

Re: insf is slow on some systems but not on others...

Yo..
"I have assigned points to 0 of 15 responses to my questions."

Please assign point.
If not aware please have a look at
http://forums11.itrc.hp.com/service/forums/helptips.do?#33

Its just a way of saying Thanks !!
"Light travels faster than sound. That's why some people appear bright until you hear them speak."
Kevin Lister
Frequent Advisor

Re: insf is slow on some systems but not on others...

Hi Avinash,

There's no problem with seeing the lun's I created on the va's. I would be trying to install the device files if ioscan failed to see the hardware.

Let me be clear that I'm not having trouble getting disk devices set up. Everything is working. I simply don't see why some of the systems are slow when installing the device files and other systems are not.

It seems quite odd. I through maybe there was a problem with one of the va's. There isn't. I have three va's connected to two different Brocade FC switches. The slow behavior happens through both switches from all three va's on the one system but not on another. Strange.

kev
Avinash20
Honored Contributor

Re: insf is slow on some systems but not on others...

Ok.. so you hav a working and a non-working server

Check with
# timex dd if=/dev/rdsk/ of=/dev/null bs=1024 count=1024

Check if both provide the same output

Also please provide the output of

# timex insf .. ,,the command you are typing

from both working and non-working server.
"Light travels faster than sound. That's why some people appear bright until you hear them speak."
Kevin Lister
Frequent Advisor

Re: insf is slow on some systems but not on others...

Hi Avinash,

All of the servers are working. There are no non-working servers. I've done performance testing on on filesystems after getting them created. They are fine. And before anyone asks... No! It's not a question of using the redundant path to the arrays as opposed to the performance paths. It doesn't matter which path or which array. I can time the execution of the insf command but I don't see any value in that. It takes 5 or so to run. Normal execution time is way less than 1 second. That is what I would expect. I can check the patch levels but I typically perform the exact same install on all servers. I do have a web page I use to compare patch levels on different systems. I wrote a Perl cgi-script that grabs the patch info from each system and then renders the comparison in a simple format that makes any differences stand out very clearly. I'll check that out tomorrow. For now I'm hitting the hay! Talk to you tomorrow! ;o)

kev
Robert-Jan Goossens
Honored Contributor

Re: insf is slow on some systems but not on others...

Kevin,

Could you compare the /etc/loadmods file on your systems, or compare the output of the "kmadmin -s" command and look for the dynamically configurable loadable module krm.

Regards,
Robert-Jan
Kevin Lister
Frequent Advisor

Re: insf is slow on some systems but not on others...

Hi RJ,

I'd love to provide you with the info you asked for but I don't have the kmadmin command or the loadmods file on my systems.

Those must be from a different rev of the OS.

kev
Bill Hassell
Honored Contributor

Re: insf is slow on some systems but not on others...

for DIR in rmt rdsk dsk rscsi
do
print "\t/dev/$DIR"
LIST=$(lssf /dev/$DIR/* | awk '/\?\?\?/ {print $NF}' )
[ "$LIST" != "" ] \
&& EX rmsf -a $LIST \
|| print "(none)"
done
print "\nDone"


Bill Hassell, sysadmin
Bill Hassell
Honored Contributor
Solution

Re: insf is slow on some systems but not on others...

Sorry, I missed the surrounding text to explain the code:

======================

Missing paths due to SAN switch or WWN name changes will cause lengthy delays, whether using insf, PowerPath or other path optimization tools. The fibre channel protocol is quite complex and requires a lot of negotiations just to find out that there is nothing on that LUN. You can see the problem more clearly by running this command:

lssf /dev/dsk/* | grep "???"

These are device files that exist but no longer point to a valid hardware path. You can feed the output into a loop and use rmsf to properly remove them, something like this:

for DIR in rmt rdsk dsk rscsi
do
print "\t/dev/$DIR"
LIST=$(lssf /dev/$DIR/* | awk '/\?\?\?/ {print $NF}' )
[ "$LIST" != "" ] \
&& EX rmsf -a $LIST \
|| print "(none)"
done
print "\nDone"

The above will clean both the dsk and rdsk directories as well as the tape device directory rmt. If the devices are presented to the server again, insf will create the required device files.


Bill Hassell, sysadmin
Michael Steele_2
Honored Contributor

Re: insf is slow on some systems but not on others...

Update your HBA firmware and drivers. This is often missed. And of course, this would mean also checking the same on your switch and disk array.

On the server side you'll get both HBA product number and firmware version with this command:

ioscan -funC fc
-or-
ll /dev/fc*
-or-
ll /dev/td*
then
fcmsutil /dev/fcd? vpd

Since you're running N4000 and A500 you've got some old equipment there, probably using the older style tachyon devices and probably never upgraded.

Also refer to swlist :

swlist -l fileset | grep -i -e fibre - tach
Support Fatherhood - Stop Family Law
Kevin Lister
Frequent Advisor

Re: insf is slow on some systems but not on others...

Hi Bill,

Thanks for the script. I also appreciate the subsequent explanation even though it wasn't necessary. I'm well versed in HP commands.

There aren't any missing paths and there have been no WWN changes due to changes in the SAN topology.

Here's an example scenario that describes what I'm talking about:

1) Install 2 Tachyon FC cards in a server.
2) Install 11.23 on the server.
3) Create a set of 4 lun's 50 GB each on the va7400.
4) Set up the Lun Security table on the va7400 to allow only the new server to see those lun's.
5) Run ioscan -fnCdisk > ioscan.out on the server.
6) Get the list of hardware paths from the ioscan.out file that do not yet have device files.
7) Run "insf -e -H " on each of the hardware paths from step 6.

In some cases insf command is very slow and in other cases it is very fast. The I/O performance itself doesn't suffer. So, it's not a big deal. I don't really consider this a problem that needs to be fixed. I would like to know what causes it though.

kev



Kevin Lister
Frequent Advisor

Re: insf is slow on some systems but not on others...

Hi Michael,

Hmm, I suppose I could check the FW levels on the HBA's. I believe however that the same HBA that has exhibited abnormal behavior has also exhibited normal behavior.

BTW, all of my FC cards and the Tachyon cards. So, all my dev's are /dev/tdn. I don't have any of the /dev/fcdn cards.

I'm phasing out the A500's. I don't like them since thay don't have redundant power. There's still plenty of life in the ol' N4000's! They are big, but they are cheap to maintain. Good performance too! I'm replacing two of them with rp4440's. Pretty cool little systems. I can't wait to get the production databases up and running on them.

I'll look into in FW and driver revs on my various systems to see if there is any correlation. I'll post a reply in a day or two.

Thanks for the suggestions Michael and everyone else!!! ;o)

kev