topic Re: Need help with sar in Operating System - Linux

Need help with sar

Debbie Fleith — Fri, 10 Nov 2006 10:35:22 GMT

We have a new Linux server running 2 database applications. The server appears to have plenty of CPU (its running 98 - 99% idle) and memory resources are plenty, yet users are complaining that there is lag/delay moving between screens.

I have sar installed, but I'm not familiar with how to analyze it or what I should look for. Can anyone tell me what screen delays may be pointing to and what sar parameters I should run to analyze?

Re: Need help with sar

Victor Semaska_3 — Fri, 10 Nov 2006 11:07:41 GMT

Debbie,

If CPU & memory looks OK then check your network card and disks.

For the network card I would use the command:

sar -n DEV

and see if you saturated your NIC, rxbyt/s + txbyt/s. Is your NIC 100Mb or 1Gb (1,000 Mb)? My rule of thumb is the max. no. of bytes a NIC can handle is Mb (Megabits) / 10 so a 100Mb NIC can handle at most 10MBs/sec, a 1Gb NIC can handle at most 100 MBs/sec.

For disks, if you have it, I would use the iostat command to see if they're the problem. The command would be:

iostat -x

I check the avgqu-sz column to see if I/O requests are getting queued up because the disk is too busy.

Vic

Re: Need help with sar

Debbie Fleith — Fri, 10 Nov 2006 12:41:47 GMT

When I ran sar -n DEV, and looked at the end "Average" numbers, it showed 318.64 rxbyt/s and 283.06 txbyt/s for my eth0 NIC. This is a 1 GB NIC. Are you saying that average rxbyt/s plus txbyt/s should be less than 100 combined?

Re: Need help with sar

Mike Hassell — Fri, 10 Nov 2006 14:13:32 GMT

Deb,

When you say users are complaining that there is a lag/delay when moving between screens, are you talking about an application that connects to a database on your linux server or a terminal based application that connects directly via SSH, etc??

I would start by checking your basic network settings. You say you have a GbE, but is the switch GbE as well? Or only 100/Mbs? If so you'll want to review your duplex settings and such, make sure the NIC matches the switch port that it is connected to.

What version of Linux are you running? Here's some info to start with for RHEL:

http://www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/ref-guide/s1-modules-ethernet.html

I've had all kinds of problems in the past with duplex that causes a delay in application response even though the server is performing above average.

HTH.

Peace,
Mike

Re: Need help with sar

Mike Hassell — Fri, 10 Nov 2006 14:16:53 GMT

Deb,

I reread your message and I see that you're referring to a database server. I'm assuming therefor that the users are connecting with a client on Win32.

Check the duplex as stated above, but I would also look at the following:

1. Check with your DBA on the status of the database performance. Are tables optimized as well as SQL queries?

2. Has the database grown in size recently that would cause slow performance?

3. Was the system running well in the past? How long has this problem been going on?

Answer those and we'll move from there ;-)

Peace,
Mike

Re: Need help with sar

Debbie Fleith — Fri, 10 Nov 2006 14:53:57 GMT

Yes, this is a database server and users are connecting with a client on Win 32. Network settings have been verified to match the switch. This is a RH ES 3.0 server, just replaced an HP-UX 11 server. Progress databases were migrated over. DB's have not grown in size since the migration. The problem was noticed from the first day on the Linux box. We are looking at DB performance, but all the DB settings are the same as they were on the HP.

Re: Need help with sar

Vitaly Karasik_1 — Sun, 12 Nov 2006 03:09:49 GMT

Is it possible in your environment to run client application on the DB server itself - in this case we'll avoied network issues.

In most cases disk performance is a bottleneck for DB. Check your RAID controller options for best performance. Many DB vendors suggest some kernel tuning, filesystem tuning and so on - so check release nites for your DB.

Re: Need help with sar

Victor Semaska_3 — Sun, 12 Nov 2006 08:41:46 GMT

Debbie,

The rxbyt/s plus txbyt/s are in bytes so with a 1Gb NIC, you could see no.s up in the tens of millions. Your sample of 318.64 rxbyt/s and 283.06 txbyt/s shows the NIC is basically idle.

My estimate of a 1Gb NIC handling up to 100 MBs seems a bit hit according to others. More realistically, the no. would be 70-75 MBs. When you add the no.s together they should go up to 70,000,000 to 75,000,000 when the NIC is saturated.

Vic

Re: Need help with sar

Karsten Breivik_1 — Mon, 13 Nov 2006 07:16:50 GMT

Perhaps interesting to do a low level network analysis - to see what runs on the wire.

For bandwidth and latency tests there are a few open source tools available: ttcp netperf netpipe and nTop.

To see what actuall runs on the wire for various protocols check out the open source tool Ethereal (would think you could find your Progres protocols there).

Also, I have recently seen Opnet ACE and Compuware Vantage in action - those are really imressing diagnostics packages, but they are not really cheap.

Re: Need help with sar

Alpha977 — Mon, 13 Nov 2006 07:59:15 GMT

Hello!

How much is the "load" of the server?

You can see with "top".

Sometimes the cpu and memory have a lot of free space, but the load is high.
In Linux, when the load is between 0.00 and 1.50 is ok, but when it go from 3.00 and over, there is porbabily and imminent crash or slow machine.

In hp-ux i see a load over 30.00 but the connection in telnet was impossible and the application was instable.
For resolve i had restarted Oracle.
do you have java application running too?