slow performance of the server

[Sunny Jaisinghani]

Hello all,

Recently we deployed an appln on PROD server after testing it on test server.
It is a web based timesheet appln which has a mysql database hosted on Linux server.

After we were satisfied with the testing of appln and database we migrated it to PROD. But the appln has been behaving weird.

The data fetching is very slow on the PROD server. we also tested the appln/DB on dev server. The performance was acceptable.

I have attached some statistics to compare all 3 servers.

After looking at many things i am not able to point to something for the slow response on PROD server. There are many differentiating points among the 3 machines; but which one is most responsible for slow response is still a mystery to me.

Can you guide me through this

Sunny

[Horia Chirculescu]

Hello,

From your sheet, I can understand that your prod is the oldest machine from all. You should upgrade at least your OS.

What is the output of

cat /proc/cpuinfo

for all those systems?

Horia.

[Sunny Jaisinghani]

cpuinfo attached.
Yes, PROD is the oldest.
My last resort would be to upgrade the OS, may also get a new server completely.

However, i want to try all possible things before that

[Horia Chirculescu]

Change the installed HDD (maybe also another SCSI controller ? - you have to check specs) with a better transfer rate in order to improve I/O latency, then upgrade the OS.

Horia.

[Sunny Jaisinghani]

Please note that all the servers have internal disks with capacity 75G, 160G, 250, etc.

[Horia Chirculescu]

I see now, all have U320 SCSI 10k HDD.

How about installed memory/swap size?

You should upgrade the OS.

Horia.

[Horia Chirculescu]

You could leave the application on the test server until you will be able to upgrade the server/get another one.

Horia.

[Sunny Jaisinghani]

PROD server has 6GB RAM and 4GB SWAP
I guess that is enough physical memory.

I had a doubt if i need to keep the physmem and swap equal or double but people recommended to keep it as it is as physical mem is enough so that swap would be hardly be used.

[Horia Chirculescu]

I would increase the swap in this case.

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1344830

Do you ever get core dumps with out of memory messages?

Does not hurt to make it 6G swap. If your HDD space is not a premium, you should consider it making swap bigger: 10-12G maybe?. Always use dedicated partitions for swap.

Horia.

[Sunny Jaisinghani]

yes... i would definitely increase the swap to 6GB atleast.

But increasing the swap won't solve the problem at hand. Because the appln is not memory intensive and i have sufficient phy mem to suffice the appl needs.

I have 2 options
1. Change the HDD and scsi controller which has better throughput
2. Update the OS


Are there more options before i change h/w or s/w

Can it be a problem with device drivers??

[Horia Chirculescu]

>Can it be a problem with device drivers??

This will be a problem solved with OS upgrade.
Installing/compiling new drivers on the same OS would require updating some of the system libraries and you will only loose time.

You can also try to install a newer kernel.

But again, my first choice would be to upgrade the OS.

Horia.

[Sunny Jaisinghani]

unfortunately i don't have a support contract with Red Hat.
So getting a new kernel would be difficult.

The current level is 2.4.21-9.ELsmp

Is there any other source from where i can get kernel update? which i can rely on

[Horia Chirculescu]

According to http://www.kernel.org/,
the latest 2.4 version is 2.4.37.9.

You could try a

yum update kernel

Alternatively, if you have installed the development tools you could install the source and compile it by yourself as you like. In this case you would keep only what you would need in your system.

Horia.

[MarkSeger · Accepted solution]

whenever I'm having performance issues my very first thing to do is run collectl either as a daemon on in another window and watch what's going on during your times of trouble to try and pin down which resource(s) you're being starved on. Is it disk? cpu? network? memory? With collectl you can watch all 4 while you're doing your query, though since by default (to limit the display to 80 columns) it only shows cpu, disk and network. To see all 4, just add memory to the display (if it will fit) by running the command:

collectl -s+m

Only after you determine WHICH resource is having issues can you take the next step of trying to fix the problem.

-mark

[Vitaly Karasik_1]

"HDD Data tx rate" - did you test this from your application or just by some copy command?

As masny others, I suggest you to upgrade your OS. You can either upgrade to RHEL/CENTOS 5 or upgrade RHEL3 to the latest update level using CENTOS repositories.
I don't think that hardware is a problem in this case.