topic Re: Graphing data collected by T4 in Operating System - OpenVMS

Graphing data collected by T4

Jeremy Begg — Fri, 05 Jun 2009 03:41:42 GMT

Hi,

We have had T4 running on some OpenVMS V7.3-2 AlphaServers for about a year, and we want to start generating some graphs. (OK, we've been wanting to for about a year, I just now got around to it!)

What we'd like is a monthly graph of CPU usage (for example), and also a yearly one. Using CSVPNG we produced the attached graph showing CPU usage one one machine last month.

That's (barely) acceptable for a single month but the format is not going to be much useful to spot a trend over a year (unless it's a really strong trend). For one thing it needs to be a much smoother line; there's way too much detail.

So I was wondering what *automated* tools people are using for producing useful graphs from T4 data? Pointers to freeware tools and procedures would be much appreciated.

Thanks,
Jeremy Begg

Re: Graphing data collected by T4

Murali L.R. — Fri, 05 Jun 2009 06:08:06 GMT

Hi,

TLViz: A timeline viewer (visualizer) which analyzes T4 files, it gives comprehensive report.With TLViz you can look at complex performance time-series histories containing hundreds or even thousands of variables. TLViz lets you look at individual metrics one at a time in rapid succession, or quickly look at the relationship between two or more metrics.

It can be downloaded from below URL,

http://h71000.www7.hp.com/openvms/products/t4/index.html
or

ftp://ftp.hp.com/pub/openvms/t4/TLVIZ_1609_KIT.ZIP
Regards,
Murali

Re: Graphing data collected by T4

Jeremy Begg — Fri, 05 Jun 2009 06:21:19 GMT

Unfortunately TLViz does not run on MacOS, and it's not particularly suited to automation either. I want something that I can "set and forget" on the VMS side of things, to run as a regular job each month.

Thanks,
Jeremy Begg

Re: Graphing data collected by T4

Ian Miller. — Fri, 05 Jun 2009 07:20:03 GMT

Jeremy,
are you up to date with CSVPNG?
The latest version is at
http://www.trendsthatmatter.com/nbu/t4-tools/index.html

It allows quite a lot of customisation.

Re: Graphing data collected by T4

Karl Rohwedder — Fri, 05 Jun 2009 10:03:03 GMT

< Unfortunately TLViz does not run on MacOS

Pls. note, that Martin VorlÃ¤nder has rewritten TLViz using Java, that should run on Mac also.

Link: http://www.pdv-systeme.de/users/martinv/JTLviz.jar

regards Kalle

Re: Graphing data collected by T4

Hein van den Heuvel — Fri, 05 Jun 2009 14:58:21 GMT

Jeremy,

There is a good few ways to skin that cat.
I like using a PERL script to go over a wildcarded list of *COMP.CSV files and gather stuff. First step is always to transform the nasty dd-mmm-yyyy file names to a list sorted by ymd.

For some actions I access the raw data, for others I prefer to have CSVPNG help me.
For example to EXTRACT to a smaller file with a larger sample interval.

In the the attached example I just use AVG and MAX values and times from tsvpng, for a certain time window in the production day, and spit out a record for each day, starting with the averages for each selected measurement for handy plot, followed by the rate/max/time columns for details.

Excel (or any other tool) will be happy to plot that data.

Typically I offload the T4 data from VMS production systems onto some Windows system, and run my scripts there, but you can run perl and csvpng on the OpenVMS systems as well.

Hope this helps some,
Hein van den Heuvel
HvdH Performance Consulting

Re: Graphing data collected by T4

Martin Vorlaender — Fri, 05 Jun 2009 20:13:35 GMT

Slight correction WRT to Kalle's post:

That URL isn't valid any more; it now resides at http://vms.pdv-systeme.de/users/martinv/JTLviz.html

But JTLviz, like TLviz, is an interactive tool and not suited for automated processes (though using some of the classes would ease the creation of some custom Java code).

cu,
Martin

Re: Graphing data collected by T4

James Viswasam — Tue, 23 Jun 2009 17:38:11 GMT

Jeremy

I have been trying to get a monthly graph for CPu using csvpng and not able to get like the one you posted.
I did a merge of the daily comp.csv files for one month using the /sample=5184000
and then generated htm and png using the following
csvpng /para="CPU Busy","MP synch","Interrupt State",direct,buffered /i /l /border=1 -
/width=1200 /height=400 /z /u=t4-may-node.htm t4-may-node.csv;
All I got was average of the whole month for each time 60sec interval
Can you please post the csvpng commands ?
Thanks
James

Re: Graphing data collected by T4

John Gillings — Tue, 23 Jun 2009 20:46:48 GMT

Jeremy,

I've used CSVPNG to automate extraction of data from a (large) T4 archive our data covers 24x7 at 10 second granularity across I don't know how many systems, going back at least 3 years. My problem is the reverse of yours, I usually want to look at a very short time frame, maybe only 2-3 minutes, but the principles are much the same. I use CSVPNG to extract time slices, then merge multiple nodes together. For example, I have a procedure which takes a time period and list of node names, out the other end pops a CSVPNG stack of graphs and CSV files. It knows how to find, all the files, both horizontal and vertical, expand them, merge and slice. (Unfortunately it's proprietary, so I can't just post it)

Look at the qualifiers for CSVPNG, there are LOTS of them! Different combinations can do some very slick things. In particular /MERGE, /SAMPLE, /DUPLICATES and /MISSING.

"Smoothing" is a very subjective thing! Your sample graph looks fine to me. There's a clear band showing the normal load, with periodic drops to zero and spikes up to 100%. Most smoothing algorithms (arithmetic mean, moving average, quadratic or spline interpolation, least squares etc...) would flatten the spikes. It might LOOK prettier, but it would obscure possibly important information.

On the other hand, if you can think of a generally useful function that CSVPNG doesn't do now, propose it as an enhancement. Although Pat is now retired, he told me he may still be interested in tinkering with CSVPNG.

If that were my system, I'd be much less interested in broadening the view, than zooming in and working out what the features are. In particular, what's with the incredibly regular drops to zero?

Contact me off line if you want to discuss it in more detail.

Re: Graphing data collected by T4

John Gillings — Tue, 23 Jun 2009 21:15:45 GMT

Just notices James' question:

>I did a merge of the daily comp.csv files for
>one month using the /sample=5184000

>All I got was average of the whole month

You got what you asked for! A sample period of 5 million seconds, so CSVPNG has averaged the entire month into one sample.

Try again with (say) /sample=3600 (hourly). That may be more useful, BUT it will obscure spikes, which are probably more interesting than the average.

Re: Graphing data collected by T4

Jeremy Begg — Wed, 24 Jun 2009 00:54:56 GMT

Thanks for all the comments, in particular the JTLviz tool which I'll try to set up on my Mac.

We've just about got our graphs set up now after a lot of tinkering with CSVPNG but John's given me food for thought (as always!)

Regards,
Jeremy Begg