Idle time using sar is 0

[Kathleen]

What are some things I can check for this?System is responding slower than normal for login, bdf and some other things. Sar shows idle time is 0 and the rest of the output is listed below....
10:02:18 %usr %sys %wio %idle
10:02:22 13 10 78 0
10:02:26 18 13 69 0
10:02:30 13 10 77 0
10:02:34 15 14 71 0
10:02:38 10 8 82 0
10:02:42 7 8 85 0
10:02:46 20 12 68 0
10:02:50 10 9 81 0
10:02:54 17 19 63 0
10:02:58 10 11 79 0

Average 13 11 75 0

[Vijeesh CTK]

hi

idle time 0 means ur CPU is 100% utilised fm sar o/p it is is teling some process is waiting for IO/ so please check what is the process which is using CPU using 'top'
if ur having glance it will give good picture

CTK

[Mark Greene_1]

You are I/O bound. What sort of disk is on the system, an array or JBOD? Are you seeing any disk related errors in the syslog?

Do an "iostat 1 5" to see which disks are getting pounded. You may have a run-away application process. If your system is hosting a database, you may have had someone doing a large query, or perhaps they typo'd the input and are selecting a data range that includes all the data in the tables they are querying against instead of just a subset.

HTH
mark

[A. Clay Stephenson]

Simply run 'top' and look for the hogs. You may simply have a very busy system or you may have a few runaway processes.

[Helen French]

Hi Kathleen:

The idle time 0% means the system CPU is fully utilized this time. The sar output shows a high I/O waiting too ! I would suggest you to check the system with GlancePlus:

# gpm

Also run 'top' for better process/process status. These are the tools which you could use:

# top
# vmstat
# swapinfo
# sar -d 1 10

HTH,
Shiju

[Kathleen]

The output of top is attached....

[Kathleen]

The output of
# vmstat
# swapinfo
# sar -d 1 10

is attached....

[Sanjay_6]

Hi Kathleen,

The top output you have attached is the first page only. There are 54 more pages like that.

Hope this helps.

Regds

[Vijeesh CTK]

hi

Now the top o/p is showing normal. nothing extraordinary.this time i assume that your system is working fine.

see what you have to do is when u feel that ur system is really slow. just check the again the top output and do a sar -u 1 100 so that you can find out what is the process which is loading the CPU (waiting for IO).

CTK

[Bill Hassell]

The output of top shows that the machine is barely being used. That's the problem with simple tools like top or sar--they take a snapshot of a very complex system and for the picture given in the top output, the system is 25% idle. Now it does indicate 40% system overhead which is directly related to I/O and other system calls but no classic Unix tools will show this data very well.

The original sar output with 0% idle may have meant that 2-3 users all decided to run du / at the same time, causing massive I/I ans system overhead to calculate disk usage.

You really need a better tool like Glance to show you what the system is really doing. It's on your Application CDs.

[Vijeesh CTK]

hi

ur sar -d is showing something is doing some IO.just find out what is that process..
If u r having glance , u will get a clear picture which process is doing that much IO operation.
You can find the trial version of glance in the application CD

CTK

[Helen French]

Hi Kathleen:

Again, the sar -d output showing a very high load on the disk /dev/dsk/c1t2d0. This causes the I/O waiting on the system.

Also the 'top' output looks okay now, eventhough the load is high. I would suggest you to load GlancePlus and check the disk activities, processes running, resource utilization etc.

Also check this kernel parameter - dbc_max_pct. If it is set to the default 50%, then reduce it to a lower value (15% ?)

HTH,
Shiju

[Sandip Ghosh]

Does the system is having only one disk? In sar -d 2 10 it shows that it is 100% busy. I think some heavy read/write activity is going on the disk. That is why you are getting 0% idle on Sar.

Sandip

[Kathleen]

3 disks

[Sandip Ghosh · Accepted solution]

First thing it is swapping too much. Look at the dbc_max_pct whether it is at default 50%. If it is so, bring it down to 15% by doing the kernel parameter setting.

Look at the filesystems which are on c1t2d0. If possible move some of the filesystem to other disk.

Sandip

[Vijeesh CTK]

hi


just find out which VG is this disk c1t2d0 included and find out what the LV's in that disk. if any filesystem find which file is getting updated.

CTK

[MANOJ SRIVASTAVA]

Hi


What is running on c1t2d0 ? And there are lot of openmail processes running , the load is also too high like 7.43 . The system is waiting for 1/0 as this disk is being used the most more thatn 99 % . Does this belong to vg00 ?

I if possible can you please also attach a ps -ef list and answer to which vg this disk belongs too and may be try to search for the process which is making the disk busy .

Manoj Srivastava

[Mark Greene_1]

if you run fuser /directory/* where directory is the top-level directory for the file system on the affected disks, you will get the PIDs for the processes writing to that disk. You can then grep the ps -ef listing for those PIDs and see who is running what. You may have duplicate processes contending for the same disk, or simply an excessive number of processes running.

HTH
mark

[Kathleen]

I was able to get glance installed. This is the process list and memory report.....

[PIYUSH D. PATEL]

Hi,

Check what is installed on c1t2d0. I think that a swap volume is installed on it ??

Give the details. I think taht ur system is busy swapping.

HTH,
Piyush

[Jeff Schussele]

Hi Kathleen,

Appears to me you're short on memory.
Definitely check dbc_max_pct in kernel params & if @ 50% bring it down to no more than 20-25%. Remember this value is the max % of SYSTEM memory to use for buffering, so if buffers use their max you only have 256 Mb of memory for everything else.
Also if you have a high OpenMail user count on this box, I'd strongly recommend you bring memory up to at lease 1Gb from the 512Mb you currently have.
Your high disk usage is directly tied to having to page in/out when memory is exhausted.
Another recommendation would be to get another disk in their & swap to it instead of the primary disk. If you can't get the memory, this would be the next best way to help this system & probably cheaper to boot.

HTH,
Jeff

[Sandip Ghosh]

It seems that your Server is running short of memory. You need to add some memory on the system. Due to the shortage of memory it is swapping too much. If you look at the glance output you can see that swapper and vhand is taking most of the resources.

Another thing you can do. Set the priority of both the swap as same i.e. 1. Then it will swapp in both the device. And if possible devide the swap into two disks, not in one disk.

The above things can increase your performance marginally. To improve the performance you have to install more memory.

Sandip

[Kathleen]

Thanks so much for the help. I will start with the kernel parameter and then work on getting swap on another drive. Then increase the memory if the others do not seem to help. Thanks SO much for all of the help!!