Performance monitoring on Tru64 Unix

[Abhijit_8]

I need to check the performance on my 3 node GS140 server cluster running Tru64 Unix V5.1A.

Also i want the output that can come in a graphical format.Anybody knows about any tool(Free or licensed ) that can help me in the same.

I have already used PMGR and collect tools.

[Hein van den Heuvel]

There is, best I know, no standard tool that integrates performance data accross clusternode in Tru64. This is actaully, IMHO, a weakness in the the Tru64 cluster software offering.

I once created a perl script to go over the collect data and organize the IO throughput activity in MB/sec by node and disk. This is still pretty much work in progress, but it seems to give a nice clusterwide view.

See attachment for that script

Under consideration:
- produce alternative output in csv format for easy excell graphing.
- validate that the collect intervals from multiple file remain sufficiently in sync
- understand the impact of interval clock skew.
- combine IO and CPU in a single time line (not enough columns!?)
- think about IO/sec versus MB/sec as primary metric
- think about Read-Write ratios.
- remove hard-coded 4-nodes in final output (already mostly dynamic).
- fix semi-deliberate one-off error on timestamp.

Current usage: Just provide a (wildcarded) collection of compressed collect files.
Current example: ./io_sum /06141230_db?.dmp.cgz
Reduced output for that collection:

MB/sec psftdb1 psftdb2 psftdb3 psftdb4 psftdb1 psftdb2 psftdb3 psftdb4
---------------------------------------------------------------------------
:
4.0 13:43:42 0.9 0.7 0.6 1.8 data8 data10 data12 data2
34.1 13:43:52 34.0 0.0 0.0 0.1 data8 data1 data1 data1
64.7 13:44:02 54.3 0.4 0.1 9.9 data8 data1 data1 data7
100.2 13:44:12 86.6 0.0 0.0 13.5 data3 data1 data1 data7
59.7 13:44:22 42.8 0.0 1.0 15.9 data8 data1 data2 data7
34.0 13:44:32 18.4 0.0 0.0 15.6 data8 data1 data1 data7
31.6 13:44:42 22.8 0.2 4.8 3.8 data8 data2 data12 data13
20.5 13:44:52 19.7 0.6 0.1 0.0 data8 data2 data1 data1
21.5 13:45:02 19.9 1.5 0.0 0.0 data8 data2 data1 data1
67.2 13:45:12 19.9 14.4 23.4 9.5 data8 data10 data12 data7
118.5 13:45:22 25.8 42.4 41.9 8.5 data8 data10 data12 data13
209.9 13:45:32 20.6 77.3 101.4 10.6 data5 data4 data3 data7
122.7 13:45:42 15.3 65.3 28.7 13.4 data5 data1 data12 data7
:
0.7 14:08:52 0.2 0.4 0.1 0.0 data1 data1 data1 data1
0.3 14:09:02 0.2 0.1 0.1 0.0 data1 data1 data1 data1

Top MB/sec rates:
Cluster wide 209.9 13:45:32
Disk / Cluster 73.6 13:49:02 data1
Disk / Node 38.7 13:47:32 psftdb3 data1
Single Node 101 13:45:22 psftdb3

[Jean-Pierre Denis]

Hi,

we use cacti (Open Source) to monitor all our unix servers. http://www.raxnet.net/products/cacti/

I have build a dedicated linux server that gather data from a mix of hp-ux, tru64 and linux. Our stats server run apache,mysql,nfsd.
If you can't have a dedicated stats server you should be able to install it on the GS140. We decided to do it on a seperated box to make it easier for backup/restore.

Shell scripts that run thru crontab on all the servers gather the data and send the output to a shared nfs filesystem on the linux stats server. This filesystem have one folder for every server. Then cacti take those results and build graph.

Here is an example of the output :
http://stats.webspacesolutions.com/graph_view.php?action=tree

It is also very easy to implement. Maybe is sound like a lot of work but a good unix admin can set this up in 2-3 days...

Thanks,
JP

[Stephen Carpenter]

You may find my cluiomon tool useful:

cluiomon(local) cluiomon(local)



NAME

cluiomon - cluster I/O monitor

SYNOPSIS

cluiomon [-d] [-r] [-w] [-m] [-u] [-i interval] [domain | disk ...]

OPTIONS

-d Displays individual disks rather than domains.

-r Displays read statistics only. The default action is to display
combined read and write statistics.

-w Displays write statistics only.

-m Lists members by memberid rather than by hostname.

-u Disables the sorting of domain or disk names.

-i interval
Specifies the interval time in seconds between display updates.
The default interval is 1 second.

domain ...
Display just the specified domains. If no domain is specified
cluiomon displays all mounted domains in the cluster.

disk ... Display just the specified disks. If no disk is specified
cluiomon displays all disks in the cluster.

DESCRIPTION

The cluiomon command displays the I/O transfer rate in kilobytes per second
of AdvFS domains or disks across all members of a TruCluster Server confi-
guration.




EXAMPLES

1. To display cluster I/O per AdvFS domain every 5 seconds, enter:
# cluiomon -i 5
dogbert dilbert
cluster_root 0 0
cluster_usr 0 0
cluster_var 0 0
nsr_domain 0 0
ris_domain 0 0
root1_domain 10 0
root2_domain 0 3
share_domain 0 0
src_domain 0 0
users_domain 513 0
www_domain 0 72

2. To display cluster I/O per disk, enter:
# cluiomon -d
tom dick harry
dsk0 0
dsk1 0 0 12
dsk2 0 912 0
dsk3 0 0 0
dsk4 0 0 0
dsk5 0 0 436
dsk6 0 0 0
dsk7 80 0 0
dsk8 0 0 0
dsk9 0 0 52
dsk10 0 0 0
dsk11 0 0 0
dsk12 23 0 0
dsk13 0
dsk14 0

RESTRICTIONS

I/O data is gathered on a per disk basis so all AdvFS partitions on a disk
will show the same figures.

No data is shown for AdvFS domains containing LSM volumes.

SEE ALSO

Commands: iostat(1), drdmgr(8)

[Stephen Carpenter]

Oops, forgot the Makefile!