System performance

[Pepe Jimenez Muñoz]

Hello everybody.

I have a D320 with 384 Mb (not interleaved) and Informix. The machine is running the personel aplication of my company and have over 40 concurrent users.

The performance of this machine not is fine, and I want to do something before to upgrade the hardware if posible.

Here is the output of some command:

runq-sz %runocc swpq-sz %swpocc
sar -q --> Average 3,2 100 0,0 0

%usr %sys %wio %idle
sar -u --> Average 3 1 2 94

swpin/s bswin/s swpot/s bswot/s pswch/s
sar -w --> Average 0,00 0,0 0,00 0,0 917

Thanks.

[Andy Monks]

From the little info you've posted, you have a huge memory issue. (like not enough). Your cpu is idle, and according to the swpq-sz, you have tons swapped out.

Check how much paging is occurring with sar -w

[Adrian Voysey]

What you can also try looking at is vmstat or glance if installed. For example using vmstat (vmstat -n 5 5) 5 secs 5 times values of interest are :-

us +sy ( if you add these values together and they exceed 80% then , system is CPU bound (may need to get more processors, but looking at your sar -u system is not CPU bound.

Check the fr and sr values e.g. if fr is 5 and sr is 32 this means, for every 5 pages freed , 32 pages must be examined , a high value here means need more memory or something is hogging up memory,

The pi (pages in) and po (pages out) should be very low or 0 , if there are high values in here the system is paging , result in not enough memory. From your values looks like your system is swapping due to lack of memory.
The other values b , the number of processes blocked i.e. waiting for a system resource but none available, this should be a low value.

The other thing to check is messages file for any errors regarding shared memory problems or when user runs a particular part of application and it complains about shared memory, to find what is using your shared memory run the ipcs -m command , and check the id ( this should be a process id) and if you run ps -ef | grep id , may show you a process that is hogging the shared memeory and killing this off using ipcrm command
(see man pages for more info)

I hope this helps

[Dave Wherry]

One quick thing to check is the kernel parameter dbc_max_pct. It's default is 50 which means 50% of your memory will be allocated to buffer cache. Many times this is left at the default and for most systems it is too high. In your case it would mean 192MB is reserved for buffer cache. sar shows you are not waiting on IO so it is probably safe to lower this down to 10 - 15%.
That will give a good chunk of memory back to your user processes.
There will be several other things to look at. This is a quick place to start.

[Andy Monks]

dave, that's a maximum of 50%. Under memory pressure, like this machine obviously is under, it will be reduced. It will probably go down to dbc_min_pct which the default is 5%.

If Pepe, posts sar -w or vmstat info, we might be able to give him some more info. However, for sure, this machine could do with more memory.

[Albert E. Whale, CISSP]

Pepe,

Depending on what you are doing with the Server, I would Highly recommend at least 512 MB of Ram, or MORE!

Whenever you settle on the Ram Amount, don't forget to configure at least 1 1/2 times Swap Space.

Hope that helps!

[Philip Chan_1]

Hi Pepe,

The "pwsch/s" rate seem high, that I think many processes in your system are being blocked for some reasons, try finding out what processes are blocked then who is blocking them. I'm not familiar with Informix at all but make sure you don't have livelock problem with it.

I don't think memory is the problem based on the "sar -w" output. You just have a high context switch rate but not swap rate.

Rgds,
Philip

[Pepe Jimenez Muñoz]

Hi everybody:

first, thanks to all.

I?m looking all yours messages and the parameters twice, and:

dbc_max_pct = 20
dbc_min_pct = 5

I attach a document with the output of vmstat and swapinfo -tm, the results of this tell me that:

1- Perhaps I need more swap.
2- CPU is bound (sorry for the sar -u in the first message)

[Anthony deRito]

I don't see the problem here. Based on the latest information and the early information we are looking pretty good. Our page rate is very low and we have plenty of virtual memory available.

Even if you did have a high page rate, it is important to remember that paginf out is bad, paging in is normal. HP-UX is a virtual memory operating system and it will work just fine with paging in only the parts of a process that it needs to execute.

It seems as though there are a lot of system calls being made which are consuming a little more CPU time. There may be some blocking going on.

When you say the performance is not fine, is it response time or throughput issues? If it response time issues and your talking filesystem I/O, you may want to check to see if your doing async or sync filesystem access. As mentioned before, you may be blocking on sync calls which may take up a lot of time. If you have glance, use the process resource windows and check to see what resources your process is waiting for. Is is waiting for VM, I/O, sockets, priority...

If throughput is the issue, check out your sar -d parameters.

Hope this helps

Tony

[Philip Chan_1]

Hi Pepe,

Based on the your last vmstat output, your system definitely fall short on CPU power. But before investing the money on hardware, you should considering doing database tuning as you could. The performance gain by turning badly constructed SQL program into its optimized form are so drastic that you can never achieved the same result by upgrading the hardware.

Rgds,
Philip

[melvin mariney]

What version of Informix are you using and type of DBMS(IDS,SE,XPS,etc.)? It would be helpful if you posted the output from the following Informix commands if you are not on Informix SE or Online 5.x: 'onstat -p', 'onstat -g ioq', 'onstat -g rea', 'onstat -d', 'onstat -D', 'onstat -c'.

thanks

[Dave Wherry]

Andy is right about my previous post. That 50% is the maximum and is not pre-allocated. With the snapshots we see here, the advantage to setting dbc_max_pct lower is that transcient or occasional buffer demand, which we do not see, would not grab the memory from user space.
From what we see here it may not be extremely valuable, but probably worthwhile to do it anyway.

[Gabriel Hermida]

In Informix, run update statistics and verify the isolation mode and the lock mode for the tables (row, page).

[Joseph T. Wyckoff]

Forgive me, as my vocabulary is oriented to MS-SQL - but it aplies to some degree here...

Database tuning is a GOOD idea.

Have your DBA find out what kind of usage patterns your customers have - what kind of data they are typically after, and so forth.

You customers may be running substantially the same queries over and over again. Your DBA may be able to design or refine indexes (or train the customer to use better queries) so that they are less cpu, memory (or disk) intensive.

If on the other hand, you are doing primarily transaction oriented (enter once, and forget it) things - getting rid of unneeded indexes can help.

Let us say, users notice things slow during the day, while entering transactions - they will complain. If we drop UNNEEDED indexes, the INSERTS are faster, normal searches are as fast as they ever were, but some rare searches may be slower.

If we don't have enough, or the right indexes, we may see some speed increase by adding indexes.

In the evening let us say you have process which queries and does some summarizing, or backups or some such. Perhaps you can afford an afterhours task taking longer, in order to get better transaction performance during the day.

Boy another thing - make sure your data, logs, and so forth are spread over several physical drives (perhaps one filesystem is OK, as long as it is spread over several drives evenly) to reduce the average access times. Simply be moving logs to another drive, you may see some performance gain.

Well, obviously NONE of that is SIMPLE. But it might be cheap and effective. If Informix is the primary task of the server - get your DBA involved.

Otherwise - look for the main (most limiting) bottle neck and address it - perhaps it is the network?

Be careful, and good luck.