Operating System - Linux
1748169 Members
4205 Online
108758 Solutions
New Discussion юеВ

Re: How can I gather performance history for redhat linux

 
SOLVED
Go to solution
westb
Advisor

Re: How can I gather performance history for redhat linux

I tried ksar but it chokes when importing large text files; at about 2 weeks of sar -A data. I think it remains useful if one needs to look at a particular day or even a week.

For my systems, I have a script that runs just before midnight that gets the sar data for that day and keeps appending it to another text file that I was importing into ksar. Worked okay until the file gets too large.

The same script also generates another text file with daily averages for cpu and i/o. This gets imported into Excel and works well. All systems write their sar data to an nfs share which is also a samba share.

The goal here is to be able to generate reports that show performance over a long period of time, say a year or more. The daily average provides a nice summary of things and if one needs to drill into it further then the sar -A text file can be viewed.

ksar is nice for generating reports but in my opinion there is too much info which will generate a lot of questions. I tried adding specific graphs and also tried only generating pdf's for specific metrics but seems only 2 can be used ??

Ideally, something that can auto generate the graphs would be nice, guess I will have to look at gnuplot or something for that.

I'm also going to further explore collectl.
MarkSeger
Frequent Advisor

Re: How can I gather performance history for redhat linux

Trying to keep/report performance data over a long period of time is something I'm not personally interested in as my (and collectl's) focus is fine grained very accurate data. At the very least I think anyone running sar should change their sampling interval to 10 seconds, otherwise it's not telling you anything useful.

That said, if you want longer term data my assumption is you don't care about accuracy. After all, if you look at a 10 minute or 1 hour sample (which I think you'd have to do it you want a year's worth of data) and see an average network load of 30%, you'd never realize it might have been pegged at 100% for multiple minutes and that you're bandwidth starved. However if that's what you want, maybe you want to load your data into rrd. That's a db technology which ages/aggregates data so that the more recent data is finer grained while the older data is courser. It also has a nice plotting tool - every see ganglia

Of course, my vote would be to use collectl to gather the data rather than sar, but I'm also admittedly a little biased. ;-)

Hope this helps.

-mark

Re: How can I gather performance history for redhat linux

My name is Bill Parkinson and I have been at this for a long time, see my web site analinux.com. I use sar, iostat, and ps at 20 second intervals. I am starting to look at the work marc has done especially in the ps area. I have done Openvms work for many years and I use csvpng for web or html serving of graphs. I use Tlviz for drilling down on complicated issues. I recently completed graphs on demand from the fancy new admin menu but have not updated the web site to reflect the added feature yet. With menu driven feature all graphs are available within the minute for sar and iostat, and the rule driven graphs are also available. Tar gzip will give you 14 to 1 compression, you can keep reference days, its nice to have a base line. While my web site is still a work in progress, it is already packed with a lot of information. I want to make it shareware but I will do one free evaluation for you after I send you the collection bash script I use. Fill out the contact us section at analinux.com and you'll get one free evaluation. If your really interested read the Case studies section.
MarkSeger
Frequent Advisor

Re: How can I gather performance history for redhat linux

Just thought I'd let anyone who is still following this thread that there is a new blog called HP Cluster Edge where you can read about many things HP is doing in High Performance Computing. Check it out at http://www.communities.hp.com/online/blogs/hpcclusteredge/default.aspx
-mark
John McNulty_2
Frequent Advisor

Re: How can I gather performance history for redhat linux

I use Zabbix. Sample intervals are tunable per item with the default to keep fine grained data for 90 days and course grained data for 1 year. After the 90 days it drops the fine grained data and only retains hourly stats.

It's a sledgehammer to crack a walnut though if you're only interested in a couple of systems, as it's designed to collate data for hundreds and thousands of systems/devices, and alert/trigger on events.

It's I/O stats collection on Linux sucks a bit, but I've worked round that by adding a small script to take named devices from /dev/mapper and then:

- get the major/minor numbers

- grep the major/minor numbers from /sys/block/*/dev to find the dm device name

- process the associated dm data from /proc/diskstats and graph what I'm interested in.

I find that being able to graph user friendly multipath device names like these:

asm-db, asm-dbp1, asm-fra, asm-frap1, home, ocr1, system-oracle, system-root, system-swap, usr-local, vote1

..is a dam sight more meaningful than

dm-0, dm-1, dm-2, dm-3, dm-4, dm-5, dm-6, dm-7, dm-8, dm-9, dm-10.

Not to mention that dm devices are not guaranteed to be the same, and change more often between reboots than they should with an iSCSI cluster, making them useless for collecting trend data.

That was the only thing that stopped me from using collectl ... hint hint ;-). Coming from a tru64 background I'm a big fan of collect.
MarkSeger
Frequent Advisor

Re: How can I gather performance history for redhat linux

When you say I/O reporting sucks, are you specifically talking about device mapper naming or do you have somethign else in mind.

As for DM naming, I think I follow what you're doing but I'm not sure where you're getting names like "asm-db, asm-dbp1, asm-fra, asm-frap1, etc".

Perhaps I can add what you're looking for to collectl if not too tough and general enough...

-mark
John McNulty_2
Frequent Advisor

Re: How can I gather performance history for redhat linux

Well, the suck statement was really referring to the default stats collection that comes with Zabbix: a small blot on an otherwise superb product. The rest of what I said though referred to my frustration with tools in general that report stats using dm device names, and my work around to get something more meaningful.

The names "asm-db, asm-dbp1, asm-fra, asm-frap1, etc" come from multipathd. By default it aggregates device names that present different paths to the same device (/dev/sdc, /dev/sdd, etc) into a single device name e.g: /dev/mapper/mpath0. The fancy names come from multipathd's user friendly names feature, where you bind the WWID of a device you know, to a name you want to use and declare it in /etc/multipath.conf.

So for example if I know (from the LUN and from using the command "multipath -ll") that /dev/mapper/mpath0 is a disk I'm using for Oracle ASM data, and that it has a WWID of 16465616462656166353a3500000000000000000000000000 then I can rename mpath0 to something more meaningful by adding this to multipath.conf

multipaths {
multipath {
wwid 16465616462656166353a3500000000000000000000000000
alias asm-db
}
}

Then /dev/mapper/mpath0 becomes /dev/mapper/asm-db. I can treat that device then just like I would a regular disk block device, with the added benefit that the new name appears in a "multipath -ll" listing. Unfortunately it doesn't appear in /proc/diskstats, so I have to find its associated dm device first.

Does that help?

John


MarkSeger
Frequent Advisor

Re: How can I gather performance history for redhat linux

So if I understand you correctly, I can add my own aliases to multipath.conf - something I've never done before, but that doesn't say I can't try. When I look at my conf file most of the entries are commented out, including the one for multipaths. Is the wwid listed there the real one? Do I just uncomment and add an alias as you did? Here's what's in that stanza on my system:

#multipaths {
# multipath {
# wwid 3600508b4000156d700012000000b0000
# alias yellow
# path_grouping_policy multibus
# path_checker readsector0
# path_selector "round-robin 0"
# failback manual
# rr_weight priorities
# no_path_retry 5
# }
# multipath {
# wwid 1DEC_____321816758474
# alias red
# }
#}

tell me what it should look like and where the alias will appear and I'll see what I can do, maybe even get it into the next release of collectl!

-mark
John McNulty_2
Frequent Advisor

Re: How can I gather performance history for redhat linux

Ah, well you can't use this for direct attached disks that just hang off a local scsi/sata bus and only have one path to them. Multipathd can only be used to manage disks that have multiple IO paths. Like for example if you had two FibreChannel HBA cards linked via dual FC Switches to the 4 ports of an EVA over a SAN, in which case you would get 4 physical paths to each device. Or another example, where you're presenting a disk from one system to another via iSCSI, and you have dual NICs and present the disk over both of them, in which case you'd get 2 paths.

Tell you what, forget about multipathd for a minute and look at LVM, because this applies to LVM Logical Volumes just as well as multipath devices, and that's probably going to be more accessible to you without dedicated hardware.

On my systems I like to name what would normally be the default LVM volume group: VolGroup00 to be "system" and the two LVs that sit on that as "root" and "swap". device-mapper creates the following two devices for me:

/dev/mapper/system-root
/dev/mapper/system-swap

I think Ubuntu does this by default anyway, but I mostly use Redhat or CentOS.

But these devices don't appear in /proc/diskstats either. So I treat them just like I would user friendly multipath device names. I pick them out of /dev/mapper, and use the major/minor number pair to find the dm device files and then record the disk stats against system-root instead of dm-0 and system-swap instead of dm-1. If you can solve that, then you'll automatically solve the multipath user friendly names scenario as well :)

John