Disk Queue Length

[Wim Van den Wyngaert]

Hi,

I found that 1 disk in my cluster had at one moment a queue length of 90. At 6:00 it was 30, at 6:02 it was 90 and at 6:04 it was 5.

The event is repeated every day at the same time.

The active image is dataserver (Sybase).

Top hot files reports high Non Virtual QIO (about 30% of all IO, mainly writes). This indicates file system activity ?

What can this be ?
(VMS 7.3, GS160, HSG80, MA8000, FDDI)

[Volker Halle]

Wim,

why not try to collect some info about those IOs with my favourite tool: SDA.

If this happens predictably at the same time every day, just submit a batch job to run at 06:00 and include the following commands:

$ ANAL/SYS
SDA> SET OUT/NOINDEX file1.lis
SDA> SHOW DEV
SDA> EXIT

This should give you the list of IOs in the queue for the device and you can find out, what they are alike, who issued them etc.

Volker.

[Wim Van den Wyngaert]

That's an option Volker. In TNG I found some extra info :
1) both cluster nodes have a peak at the same moment : 70 for 1 and 20 for the other.
2) thruput was about 4 MB/sec at the moment of the peak
3) there was also a peak of 35 in "credit waits"

Wim

[Wim Van den Wyngaert]

Correction : thruput was about 9 MB/sec on 1 node, about 3.5 on the other. Thus about at its maximum.

cluster_credits is at 128. Too low ?
Since the disk is shadowed via FDDI : mscp_buffer is at 16384 and mscp_credits at 128.

Wim

[Volker Halle]

Wim,

CLUSTER_CREDIT = 128 is the maximum value.

Is anything unusual happening with the disk, pathes to the disk or HSG80 ? Mount-verification ? Path switches ?

Check the FC counters with SDA> FC STDT
(QF seen or Seq TMO > 0 ?).

Is there a specific job, which always starts at 06:00 ?

Volker.

[Jan van den Ende]

Wim,


Since the disk is shadowed via FDDI : mscp_buffer is at 16384 and mscp_credits at 128.


Maybe a superfluous question, but since you use FDDI, did you set NISCS_MAX_PKTSZ to 4486?

Proost.

Have one on me.

Jan

[Wim Van den Wyngaert]

Jan, yes it is at 4468 (Set by Johan Michiels, so, should be ok).

But I did mon clu and added cr_waits.
There are a few thousand of them but all nodes I checked have them, even in other companies.

Volker : nothing special active. Just an application peak.

Other thing : the FDDI is shared with another cluster. This one had a thruput of about 2-4 MB at the moment of the problem.

Wim

[Volker Halle]

Wim,

here is an article explaining non-virtual QIOs as reported by TNG/PSDC

http://h18000.www1.hp.com/support/asktima/operating_systems/CHAMP_SRC931006004627.html

In your case, this would point to 'database', probabyl doing Logical-IO to some of it's files. And if it is happening every day at 06:00, there must be some 'time-released' job in the application causing this.

Did you check the FC counters ?

Volker.

[Wim Van den Wyngaert]

Now I took a look at a wider interval.

It seems that there are also peaks at other hours. And the peak at 6:00 has been lowered because the interval is 30 min instead of 2.

Between 01:06 and 2:15 several disks have peak queues of 10-70. And at this moment backup is active reading about 15 MB/sec. The SCS traffic during this interval is almost 0. I also saw backup doing lots of IO but not doing any thruput (during 20 minutes) generating queue length of 70 continuously.

Wim

So I guess it is normal behaviour when reading or writing too fast.

[Volker Halle]

Wim,

are you running your BACKUP jobs from an account with a very high DIOLM ? You may be overloading your HSG80...

The SDA> FC STDT/ALL counters would give an indication (QF seen).

Volker.