Do we pay with %SYS CPU good I/O throughput ?

[Nicolas Dumeige]

Hello,

We have a dreamy SAN (EMC DMX 2000) with an amazing I/O throughput.

However, system statistics show hight %SYS CPU consomption.

Hence, we seems to pay with %SYS CPU good I/O throughput. Is that so ?

Thx

Nicolas

[harry d brown jr]

What system processes are consuming CPU?

live free or die
harry

[Sridhar Bhaskarla]

Hi Nicolas,

If your system does a lot of reads/write IO and other system calls, then your %sys would obviously be higher. Howver, I would use glance to find it out. Run glance and type "Y" to find out the system call (Global System Calls). Depending on what you find, you may have to finetune your application.

-Sri

[Nicolas Dumeige]

Thanks for you answers,

Top CPU consumers are Oracle Process.
But how do I get the %Sys comsumption at the process level without glance ?

Thx

[Carlo Montanari]

pstat_getproc, check the field pst_stime

bye, Carlo

[Florian Heigl (new acc)]

What model of tachyon adapters are You using?
(ioscan -fnkCfc)
What mount options do You use for Your filesystems?
(grep -v vg00 /etc/fstab | grep vxfs)
what are Your scsi settings for the emc² luns?
man scsictl (immediate_report=1 and queue_depth=as high as emc² allows)
furthermore look at the max_fcp_reqs kernel option.

but probably it's simply a too old adapter without fancy protocol offloading.

I'd try to see how high %sys goes when dd'ing the rawdevice of an emc² lun. (throughput should be 50-300MB/s :)
(time dd if=/dev/rdsk/c1t2d3 of=/dev/null bs=1024k count=3000)

Also check that Your databases indexes don't share physical disks inside the emc² with the tablespaces. This would easily cause a lot of I

[Nicolas Dumeige]

OK,

I found my way to get the %Sys of processes.

I any case, my initial question was : is normal to see a lot of %Sys time when using a SAN (like with NFS server) ?

[Carlo Montanari]

How much is "a lot"? :-)
If you have found the processes consuming sys cpu I would try executing tusc -cp (pid) to understand which system call are consuming more time.

ye, Carlo

[Florian Heigl (new acc)]

It is *not* normal, but it depends.

It should be low with sequential I/O, unless data rates are >200MB/s, it might get quite a bit higher on random I/O at least if You saw oracle processes sit and wait.

*if* You see a lot of sys time while simply reading sequentially from a device etc., there probably is an adapter/firmware/driver problem.

even at full speed the I/O subsystem should not consume >20% cpu time.

but probably it's simply Your db waiting for random I/Os.
-> check data distribution in emc
-> check cache allocation in emc

[Steve Lewis]

The answer to your original question often is YES because of these reasons:

1. Your system is capable of shifting a lot more data, so any program will run quicker,l spending less time blocked on i/o and more time processing it.

2. Your database is essentially an i/o bound application, that now goes a lot quicker, so the server is capable of more of the following system calls per second:
read
write
send
semctl
switch

Using Glance to check the system call rate will confirm this. It is certainly true in my case, especially when I increased the number of cpu processes that service database requests.

[Steve Lewis]

I meant select(), not switch().

[Nicolas Dumeige]

But %Sys is not "productive" from the user point of view, are they ?

I mean, it is said that a lot of %Sys, mean that the system take a lot of time managing himself instead of serving the users. Is that correct ?

[Steve Lewis]

Define 'productive'. It could be that lower i/o times make the whole system more productive.

You could have a patch issue relating to i/o through fibre. HP will want your patch level if you logged a call over this.

Tuning your system (more memory, buffers or SGA), to reduce the amount of disk i/o calls required in the first place will reduce your sys% time and increase the available cpu for user processes. But you may still get a high system call rate for semctl(), send(), recv() and select(), because Oracle is heavy on these calls.

At the end of the day, you just have to decide whether there is a performance problem or not.

[Bill Hassell · Accepted solution]

It's really important to understand that %sys is not a bad thing if it is expected. Yes, the CPU will be spending some time in the system to perform I/O. If you reduced %sys to zero, the computer will halt. In other words, you can't do anything without the kernel's help. Every I/O requires some setup, then driver code and finally end-of-I/O processing.

For a slow disk, the kernel will spend a very small percentage of time processing the I/O requests but change the disk to a very fast array with massive I/O cache and not the disk requests complete very quickly which translates into many more I/O's per second. So a slow disk might produce 10 I/O's per second but your DMX I/O's can hit 100 or more. This would mean about 10x more system overhead per unit of time since the driver is much busier. Perfectly normal and expected. So you are not 'paying' for fast I/O with more system overhead, it is perfectly normal.

On the other hand, if you are running a very old (in computer terms) computer, the CPU speed will limit how fast these I/O's can be completed in the kernel. It is false economy to take an old computer (perhaps 100 Mhz) and hook up a brand new disk array and expecting the same performance as an 800 Mhz computer. Since you are running Oracle, there are hundreds of posts here in the ITRC Forums about improving performance, with the majority pointing to Oracle config as the most important factor. Getting faster I/O isn't nearly as effective as reducing the number of I/O's and that is application-specific (ie, Oracle config, SQL rewrite, new indexes, bigger SGA, etc).

[Nicolas Dumeige]

Hello,

Since I've posted this question, I've been reading "Modern Operating Systems" by Tanenbaum.

Now I realize a lot of answers were good and my way of looking at the problem comletely unsuited.

I've just wanted to thanks itrc forum members.

Nicolas