What commands needed to sanity check and performance eval a linux box?

[Michael Steele_2]

Comments/

[Ivan Ferreira · Accepted solution]

>>> sanity check

Maybe logwatch or Insight Agents

>>> performance eval

collectl, sar, vmstat, iostat, top, pmap, ipcs, free

Benchmark tools

http://lbs.sourceforge.net/

[Gerardo Arceri]

May i suggest also looking for errors on the IML (use hplog if you are running PSP).
I can also suggest you use stress and leave it running for an entire day.
http://weather.ou.edu/~apw/projects/stress/
(Use google to look for rpms if you are lazy to compile it from source)

Another option would be to use StressLinux

http://www.stresslinux.org/sl/
From their homepage:

What is stresslinux ?

stresslinux is a minimal linux distribution running from a bootable cdrom, usb, vmware or via PXE (wip).

stresslinux makes use of some utitlities available on the net like: stress, cpuburn, hddtemp, lm_sensors ...

stresslinux is dedicated to users who want to test their system(s) entirely on high load and monitoring the health.

[Michael Steele_2]

I guess I'm looking for more information about the internals of the /proc file system, which seems to me to be the obvious place to go since o/s commands like sar are also going to refer to /proc.

Or am I missing something?

Question: Why don't other Linux Admins use /proc to eval linux?

[Gerardo Arceri]

I think we don't use /proc because it's cumbersome to lookup info there, it's much easier to use system tools to do it.
For example, sar output can be easily plotted to a chart with excel or other tools and quickly discover any trends.

[Steven E. Protter]

Shalom,

I think sar is a great way to go.

For years I've been meaning to port http://hpux.ws/system.perf.sh to Linux.

That is where I'd start.

I'd also consider hobbit monitoring or Openview.

SEP

[Alzhy]

Hey Mike...

On a UNIX-away path too eh? Yeppers -- the new Nehalems and Magny Cours X86-64 systems are just awesome and can compete with the UNIX big irons.

But I digress... as a UNIX person who's also crossing to the Penguin side (at work -- been using at for "magny" years now at home and NGOs/small-biz shops) -- most of the UNIX tools you've been accustomed to are available on LINUX ... there are also tons of OSS (open source tools) available out on the internet wild...

You can be at home rather quicly man... and most stuff are actually simpler and easier on Linux.

[Prasanth V Aravind]

For performance i would like to suggest a nice tool, which is available free.

NMON...

you can create graphs in xls with nmon analyzer. Hav a look on attached sample output.

http://www.ibm.com/developerworks/aix/library/au-analyze_aix/index.html

GUdluck
Prasanth

[Michael Steele_2]

What happened to the days when you weren't considered a sys admin if you couldn't get what you wanted from the command line?

I guess I'm old fashioned. Using applications with embedded command line operations isn't being a sys admin. Its something a lot less.

[Alzhy]

Mikhail,

If you manage HP-UX systems (or other UNIX Flavours) -- the tools you've been accustomed be SHOULD be available "including" the commercial one$ -- you know, OVPA/Insight/GlancePlus, Director (hmm thats Blue), Patrol/BMC, CA...etceterum.

IF it is a DB machine... then for "sanity/performance" checks/tests -- I always go with:

- IOZONE for I/O
- ORION (an Oracle Tool) for simulating DB I/O Load (and characterisation)
- and finally SwingBench


Ch33rs.

[MarkSeger]

you want command line? then you definitely want collectl. It has so many switches there are multiple switches to show different subsets of switches. ;)

BUT there is a reason to my madness - since it essentially monitors just about everything as all the other tools combined in a consistent output format, you need all those switches to tell it what you want to do.

If you really feel you must dive into /proc, and believe me it's not all that exiting, 'man proc' will tell you quite a lot. If you want to know more, you can always look at the collectl source as it's all written in perl.

-mark

[Ivan Ferreira]

Thinking about the /proc, some files used are:

/proc/mdstat (Status if using software raid)

/proc/net/bonding/bond* (Status if using software network bonding)

/proc/scsi/qla2xxx/* (If using Qlogic adapters)

Also the commands:

netstat -ni (collisions/errors)
ethtool (status/speed/duplex)
multipath -ll (md multipath)

[TwoProc]

cat /proc/swaps
cat /proc/diskstats
cat /proc/loadavg
cat /proc/meminfo
cat /proc/mount (interesting stuff here)
cat /proc/fb (screen info)
cat /proc/cmdline (shows what you booted)

if you having network issues and you want to see if you're having issues - ck out:
/proc/sys/net/ip4
look at *retries*

One of my favorite ways to see what a particular program is doing (perf, troubleshoooting, etc) is to get the process id of a process,

Run top in a window somewhere

pgrep top

use that process id and go to

cd /proc/

cat sched

This info is useful - particularly the wait stats, IO wait stats, nr_switches (voluntary and involuntary). Really neat stuff in here.

However, if you want command line tools, and not necessarily to be playing around in /proc - I like this how to guide.

http://www.softpanorama.org/Commercial_linuxes/Performance_tuning/troubleshooting_linux_performance_issues.shtml

[MarkSeger]

re /proc - virtually all monitoring tools read data from here, but none to my liking which is why I wrote collectl. Think sar on steriods! It's available on SourceForge and its unique name makes it really easy to find with google.

So what makes collectl 'special'? For one, it can report a lot more info than sar. For another it can do very find-grained monitoring at the sub-second level when you need it and believe me, there ARE times you need it. And don't forget low overhead. Using less than 0.1% of the CPU, you just turn it on and forget it. It will take samples every 10 seconds (not the 10 minute default for sar which I still think is pretty silly).

It has lots of different output formats depending on your needs at the time and a companion kit called collectl-utils, contains a tool to let you plot the data via a browser.

I had developed collectl years ago based on the Tru64 tool collect (the 'l' stands for linux) and we'd been using it internally for about 5 years before it was released as open source. It's currently part of fedora, suse and in the process of getting into the debian distro. A number of others as well.

-mark

[Alzhy]

Still going insanely agogo-agaga on thine Linux Adventures? Then your Google SkillZ (#1 Skill set in my staff right now) likely needs polishing amigo.

If thou art running RedHat - there is actually a Hardware Validation Testsuite that you can optionally install Bos. Just enable that Channel on RHN and yum away. It also includes the command line "stress" command Bos.

Performance? Magny people have mentioned already practivally ALL of the tools you use on HP-UX are the SAME and available on LeeNochs sire.

No points plz coz I knows there is something that ticks ya on my posties.