Avarage disk queue

[Lennart Johansson]

Hi all,
I have proliant dl380 g4 running w2k3 sp1 and Lotus dominio, the server is connected to a msa1000. Now to my issue, we do have some slow performance and the notes developers think it might have to do with slow disk access. When i run perfomance monitor checking counters for avg disk read queue/physical disk for the data disk I got an avarage of 4, I have read it should not be higher than 2. Is there somethings I should check before trying to get management to invest in a new server, like disk space ussage defragmentation etc. I can imaging that this is good things to do but does anybody else have eny suggestions?
Btw all firmware and drivers are up to date

Best regards
Lennart johansson

[John Kufrovich]

Lennart,

At the MSA cli prompt, issue the commands

>start perf
This will start the MSA performance counters. I will use very little controller cpu cycles. Let this run for a while to average out the counters.

>show perf logical
This will show the r/w latency, from the MSA to drives. Anything over 25ms, should consider making some changes. Faster drives, move drives, different stripe size.

Average 4 Q's isn't bad.

jk

[Lennart Johansson]

Hi,
Thanks for your reply, perhaps a stupid question but what is the proper way to determin the right blocksize? I quess it got somthings to do with the avarage file size on the disks.

Best regards
Lennart

[John Kufrovich]

There are numerous factors in determining a good stripe size. With a SAN storage you want the drives to be the slowest part of the chain but not saturated. If the drives saturate then everything is backs up. This goes back to my statement of 25ms.

Issue the following MSA commands,
>show cacheinfo

This cmd takes a snapshot of your cache useage. Cycle through the command a few times, you can use the up/down arrows on the Keypad. Look at "lines not set" Depending on your cache ratio settings, if you see a high number of "LINES NOT SET" Then your MSA isn't busy or it is running efficiently.


Likewise, you can use the
>show taskstats
This cmd takes a snapshot of the commands the MSA is currently working on. Cycle the cmd.


How does everything look when you run those commands.

[Urban Petry]

Now that the MSA part has been covered I qould also suggest looking at some things on the windows side:
1) take a look at the performance counter "PhysicalDisk"/"Avg. Disk sec/Write" and "PhysicalDisk"/"Avg. Disk sec/Read" (for the disk in question); this gives you the average time a write or read request takes to complete (from the OS point of view). As long as this value is not getting to high, you should be ok
2) what HBA brand do you use (Emulex or QLogic) and which driver model (storport or SCSIport)? Check the HBA settings for QueueDepth/QueueTarget (Emulex) or ExecutionTrhottle/NumberOfRequests (QLogic) that they are set as expected. We had several occasions where these settings we too low and changing them increased performance (although the performance was not too bad before either)

Urban

[Lennart Johansson]

Hi all,
Next basic question, how do I connect to the cli on msa1000?
As soon I know that then I will answare the rest of the questions

/Lelle

[Jefferson Humber]

For this you will need to use the MSA serial cable (259992-001) plugged in to the front of the controller, and a terminal session with the following settings 19200/8/N/1.

[Lennart Johansson]

Hi all,

Thanks for all your help, I have added a text file with the captured text from the msa1000 controller, but here is som quick info this is ½ H after i started monitoring on the controller
show perf logical gave all 0, thats good I quess.

show taskstats alot of zeros and
INITIATOR INDEX QUEUE DEPTH
0004 0512

show cacheinfo
00014307 LINES: NOT SET
00000007 LINES: LINE_LOCK
Minimum DMA CDB count since last check: 1022

The HBA is a Emulex LP1050 scsiport

Queuedepth: current=0x20 min=0x1 max=0xff default=0x20 dynamic=yes
queuetarget: current=0x1 min=0x0 max=0x0 default=0x0 dynamic=restart

[John Kufrovich]

Lennart,

You need to issue
>start perf

To use,
>show perf
>show perf logical
>show perf physical
>clear perf
>stop perf

From the cacheinfo and taskstats your systems is sitting idle.

[Lennart Johansson]

Hi John,
I have, the output I send was captured ½ hour after I had enabled monitoring by issuing start perf and after that issued show perf logical
I will run
show perf logical
show perf physical
to morrow again and post the output
btw you did see the attached file?
Cheers

Lelle

[Lennart Johansson]

Hi all again
Here is a updated file, I did run the command this morning and I quess our San doing very little or dont report what it does, pleas read and comment

Best Regards

Lelle

[John Kufrovich]

Lennart,

Please use the cli on the Active controller.

You can spot the active controller by looking at the LEDs on the controller.

MSA LEDs
15, 14, 13, 12, 11, 10, 9, 8
7, 6, 5, 4, 3, 2, 1, 0

LED 9 or bit 9 represents Active Controller.

Then run through the exercise again.

[Lennart Johansson]

*Blushing*
Of course it was wrong controller, here is a new file with counters (and this looks completly diffrent) perhaps the msa1000 has to be upgraded.

/Lelle

[John Kufrovich]

Can you provide a >show tech_support

Let the performance counters run. The only thing I see is the read latency is a little high. But we should let the perf counters run and take another reading after everything averages out.

I see at least 3-4 servers attached to the MSA. They must not all be working at the same time. Because your >show taskstats is low and your using very little MSA cache.

[Lennart Johansson]

Hi,

I have attached 3 files describing the msa luns and servers + the output from msa cli
As far as I can see has lun 1,6 and 9 most write and all other luns has more read.
Should I make the stripesize bigger on the "read" volumes?
defraging all volumes I quess should be a good thing to do, remove the unused LUN and make all volumes at least have 30% free space. Does you have any other suggestions?

Best regards
Lelle

[Lennart Johansson]

And attachment 2
This is the output from cli

[Lennart Johansson]

file 3, this is about luns
/Lelle

[John Kufrovich · Accepted solution]

Good Morning Lennart,

Thank you for gathering the information. Now, the performance counters have run for a while, everything has leveled off.

I suspect one maybe two LUNs are impacting the rest of your luns latency. Your LUNs are suffering from drive contention. LUN1 and LUN7 are the most active. Adding a MSA30 shelf and a few drives, separate 3NOTESCLUNODE cluster servers and their LUNs. Give that cluster, its own disk array and LUNs would do wonders for your configuration.

[Lennart Johansson]

Good afternoon John, (afternoon in Sweden anyway)
Have I understand you right if I say that you recomend
connecting a msa30 to msa1000 via scsi cable, add a few disks to it and configure 1 array and move the 2 clustred servers and there luns to it?
Best regards
Lennart Johansson

[John Kufrovich]

Exactly, add MSA30 self to the MSA1000, with some additional drives.

Separate your two clusters
Cluster 1, create one disk array and your LUNs.

WWWCluster, create one disk array and your LUNs.

This should help with the read performance. Very soon, we will be introducing A/A FW for the 1000.

[Lennart Johansson]

Thanks John for your exelent help and expertice
Best regards

Lennart Joahnsson