SAR -d issue with storage

[Gigi_]

Hello forum, 

I'm newbie on this topic and this forum.

I've an issue (?) with my system and/or IO monitoring. In particular my system is an HP Superdome2 with HP-UX 11.31 OS.

In order to monitor IO usage we utilize SAR, out-of-the-box with the OS.

What we are noting during the last days is a quite strange SAR -d behaviour. That is two or three times per day the output of SAR -d is so far away from the physical range normal IO behaviour: we have r+w/s near 35 million and blks/s over 650 millions.

The SAR version is:

/usr/sbin/swlist -l file |grep -i "/usr/sbin/sar"
  OS-Core.SYS2-ADMIN: /usr/sbin/sar                    
  PHCO_42120.SYS2-ADMIN: /usr/sbin/sar     

Below there is a sample of the described output:

<CMD-015;20140210120209;xxx;/usr/bin/sar -d 30 1 >

HP-UX xxx B.11.31 U ia64    02/10/14

12:02:09   device   %busy   avque   r+w/s  blks/s  avwait  avserv
12:02:39   disk12    1.13    0.50      38     667    0.00    0.35
           disk13    0.00    0.50       0       0    0.00    0.15
           disk14    1.00    0.50      34     648    0.00    0.36
          disk305    0.00    0.50       0       0    0.00    2.33
          disk306  100.00    0.50  480183 10214909    0.00    1.83
          disk307    0.00    0.50       0       0    0.00    2.53
          disk308    0.03    0.50       0       0    0.00    4.54
          disk309    0.03    0.50       0       0    0.00    5.64
          disk310    0.00    0.50       0       1    0.00    1.18
          disk311    0.13    0.50       0      11    0.00    3.98
          disk312    0.07    0.50       0       4    0.00    4.46
          disk313    0.00    0.50       0       0    0.00    2.51
          disk314    0.07    0.50       0       1    0.00    7.97
          disk315    0.03    0.50       0       0    0.00    2.38
          disk316    0.07    0.50       0       9    0.00    9.98
          disk317    0.10    0.50       1      98    0.64    7.49
          disk318    0.03    0.50       0       0    0.00    7.23
          disk319    0.00    0.50       0       0    0.00    2.37
          disk320    0.00    0.50       0       0    0.00    0.92
          disk321  100.00    0.50   59491 7240524    0.01    5.02
          disk322    0.00    0.50       0       0    0.00    2.35
          disk323    0.03    0.50       0       0    0.00    2.45
          disk324    0.03    0.50       0       0    0.00    4.55
          disk325    0.03    0.50       0       0    0.00    2.30
          disk326    0.00    0.50       0       0    0.00    2.39
          disk327    0.03    0.50       0       0    0.00    2.08
          disk328    6.23    0.51     128    1275    0.01    0.79
          disk329    0.00    0.50       0       0    0.00    2.74
          disk330    0.00    0.50       0       0    0.00    1.82
          disk331    0.03    0.50       0       0    0.00    2.73
          disk332    0.03    0.50       0       0    0.00    1.98
          disk333    0.03    0.50       0       0    0.00    1.99
          disk334    0.00    0.50       0       0    0.00    2.80
          disk335    0.00    0.50       0       0    0.00    2.97
          disk336    0.17    0.50       2      64    0.00    2.50
          disk337    1.03    1.31      16     115    0.61    1.18
          disk338    0.03    0.50       0       0    0.00    3.80
          disk339    0.00    0.50       0       0    0.00    3.88
          disk340    0.00    0.50       0       0    0.00    2.03
          disk341    0.03    0.50       0       0    0.00    4.28
          disk342    0.03    0.50       0       0    0.00    5.40
          disk343    0.03    0.50       0       0    0.00    5.21
          disk344    0.00    0.50       0       0    0.00    2.33
          disk345    0.00    0.50       0       0    0.00    2.17
          disk346    0.00    0.50       0       0    0.00    2.40
          disk347    0.03    0.50       0       0    0.00    5.47
          disk348    0.00    0.50       0       0    0.00    3.49
          disk349    0.00    0.50       0       0    0.00    5.41
          disk350    0.03    0.50       0       0    0.00    2.13
          disk351    0.00    0.50       0       0    0.00    8.02
          disk352    0.03    0.50       0       0    0.00    8.38
          disk353    0.00    0.50       0       0    0.00    3.83
          disk354    0.00    0.50       0       0    0.00    4.16
          disk355    0.03    0.50       0       0    0.00    5.39
          disk356    0.00    0.50       0       0    0.00    2.25
          disk357    0.00    0.50       0       0    0.00    2.09
          disk358    0.00    0.50       0       0    0.00    2.13
          disk359    0.00    0.50       0       0    0.00    4.26
          disk360    0.00    0.50       0       0    0.00    3.60
          disk361    0.03    0.50       0       0    0.00    5.20
          disk362  100.00    0.50   50281 17233818    0.02    4.46

 My question is: it could be a SAR bug for this version ? Otherwise could I investigate more then storage and hd ?

Thanks in advance

[likid0]

Hi,

Check if you have the latest sar cumulitive patch for 11.31.

There are several bugs fixed related to "-d"

http://h20565.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/mostViewedDisplay?javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken&javax.portlet.prp_efb5c0793523e51970c8fa22b053ce01=wsrp-navigationalState=docId%3Dpdb_na-PHCO_43002-1%7CdocLocale%3Den_US&javax.portlet.tpst=efb5c0793523e51970c8fa22b053ce01&sp4ts.oid=3367813&ac.admitted=1392389208...

[Gigi_]

Hi likid0,

the current cumulative patch installed is PHCO_42120: it seems to be the last one realeased by HP. 

Moreover I've verified that this patch could solve so many issues with sar -d, it introduces new problem with sar indeed.

Another server with the same OS version, patch and application doesn't give this kind of problem.

It is very frustrating

Luigi

[likid0]

If you have the latest and greatest in sar cumulative patch, I would try and compair the sar results with another tool, if you have glance or Performance Agent, leave them running and compair the results with sar outputs.

[Gigi_]

Hi,

unfortunately I don't have another tool to monitor IO resource, and in general all resoruces. In any case I can stated that there is a inaccuracy issue related to SAR output values, these values are not physical acceptable for any storage devices: IOPS 10 million and Throughput over 250M MB/s.

I suppose that there is an issue concerning SAR and/or OS. For that I'm asking if anyone also did face this situation ?