linux performance problem

[Marta Gomez]

Hi everybody,

I've got a problem with a red hat linux system. It has got an oracle rac instance and an oracle development instance running on it.
Databases works fine, but when I do filesystems backup (indeed it happens too when i read files and scp to another system)cpu %iowait increases a lot, and oracle's database performance decreases a lot, whe have to stop backup to continue working ok with production database.
I've realize that it depends the files it reads during backup the database performance i have.
I've attached a file with tops before and during backup.

I started to think about modify memory kernel configuration, but now I think I should by memory.

Can anyone helps me about the way to solve the problem?

Thank you!!

[Gopi Sekar · Accepted solution]

what application you are using to take backup, looks like that application starts using too much memory (swap usage is increased to 500 - 700 MB)

once the system starts swapping, it has to do lot of io calls (for moving pages from memory to hard disk and vice versa). it will definetly increase your io calls.

i can think of two choices for this:

* Increase your RAM (by atleast another GB)
* Stop oracle (or the application which consumes more memory) which should release some vital memory to avoid swapping.

Hope this helps,
Gopi

[Ivan Ferreira]

Are you using ocfs?.

The ocfs performance is bad if you use standard linux tools to access it, it should not contain any files other than oracle's related files.

Check the ocfs documentation and verify that the things you are trying to do is supported and certified.

[Marta Gomez]

Thanks for so quick responses!!

I can't stop oracle database, it has to work 24h x day. Theses tops I send are stoping development instance, so i can't stop anything else.

I'm using ocfs for oracle datafiles, but i don't copy these files on backup, I only copy operating system files, oracle binaries and database export files.

Is usual to have 700 Mb of cached memory working only with oracle?
What means that load averages goes from 1.20 to 4.76 ?
I would be sure that the problem is memory problem before buy it.

Thanks!!

[Ivan Ferreira]

Iowait increases when the system stops wainting response from an I/O device. Check that you are using the apropiate block size for the tape device and the backup command. Using the right block size keeps the tape moving continuously. Or else, the tape device must rewind to position the tape for next write.

You can also use nice/renice to lower the priority of the backup job, so wont disrupt the other process.

[Gopi Sekar]

try to reduce the 700 MB cache for oracle to some thing around 500 MB and see whether it improves the performance during backup.

if so, then the more likely issue is swap and memory, upgrade your memory.

Hope this helps,
Gopi

[Stuart Browne]

What sort of IO subsystem do you have in this system?

Whilst caching 700MB of disk IO isn't un-heard of, on a very busy system, I can see a slow IO causing the caches to be relied upon a bit more heavily.

To get more accurate numbers, I'd suggest watching the contents of '/proc/meminfo', to see how much of that memory is dirty and inactive (will give you an idea of where the memory is, and why the swap is being chewed).

Gopi's initial suggestion of more memory isn't a bad one. You've got 2GB, you're consistantly having 1GB of swap utilized (eww! dirty!), so you've already got an immediate performance drop. Throw at least another 2GB (for comfort) at it, and it should last you a while longer.

It might also be an idea to review the tuning you did for the Oracle setup, to make sure you aren't a digit off in it's SHM values (i.e. make sure you didn't say Oracle could use 2GB of shared memory, when you've only *GOT* 2GB of memory!).

My thought is that you've also got some decent IO bottle necks which is why it's caching so much.

Anyway. To get a better idea of the IO bottle-necks, use either 'vmstat' or preferably 'sar' (sysstat package) to show when the peak times are, what sort of memory gets choped and when, and what the IO channels are doing when all of this memory gets chewed.

[Stuart Browne]

Blech.. I'm talking in circles now too, repeating myself..

It's after midnight.. My coach has turned back into a pumpkin, and my mind aparently is right behind it..

[Marta Gomez]

Thank you everybody!!

I think it can't be normal to have more than 1 Gb of used swap with only the oracle database working, so the plan is to put in 2 Gb of RAM and see what happens.

I'll do it next tuesday, so I'll tell you about it...

Thanks again!