Urgent performance issue

[Jun Yu]

Hello experts,

Server: DL760G2 with 4 xeon CPU, 6GB memory
Storage: local disk array+ FC SAN storage
OS: RHEL3U4
App: DB2UDB 8.2 fixpack 11
Swap: 2GB
FS:ext3 no LVM
Behavie:

1.
This is a DB trasaction server.Every night after the day time business closed, the administrator will manually(or by script) dump the data/log from DB2 to it's attached SAN storage by DB2's dump utility.And then verify those files by normal md5 tool.After that,the administrator or scripts will scp those files(about 30GB per night) to another backup server.

2.
This system runs pretty well at the day time, and only 500MB-1GB swap used even under very busy transactions at day time.

3.
After execute the night backup job i metioned above, the almost all the swap is ate by dump+check+scp operation. personally i do think that's also normal.

4.
Strange thing is the swap partition can't be released after the night backup job and even go through the whole night! So problem appears, the next business day morning , this DB system will accept lots of query/modify request from the clients but the swap is not clean , no enough swap for it at that moment, so the whole performance turn to ugly.


Shall we add the second swap to the system? As i metioned, during the normal day time business, swap utilizaton is low. Only after the night backup job(db dump,check,scp) the swap will be full and can't release before the nexe business day morning.

Regards,

JOe

[Vitaly Karasik_1]

> only 500MB-1GB swap used
in my opinion, it's not normal

Of course, adding more swap space will help a little, but I'd like to suggest the following:

1) upgrade RHEL to the latest available update (upd 7?)
2) read and follow all IBM recommendations regarding DB2 tuning and Linux for DB2 tuning. Probably you can say to DB2 to use less RAM for certain buffers/tasks.
3) *add more RAM*

[Jun Yu]

Thanks for response.

Just as i noticed , this performance issue only happen after the administrator do "night backup job"(db2 dump, scp file ). The system runs pretty good if we skip the daily backup.

The problem looks like that the linux can't release swap immediately after finished night backup job(dump+scp).That's cause the next morning the Database and whole system hasn't enough swap to deal with daily query/modification.

The physical memory is enough for such application because we have 7x24 basic performance monitoring. Even under very busy business hours between 9:30am -11:30am. The system runs pretty well,ofcoz we skip the last night's backup job first.

Regards,

[Steven E. Protter]

Shalom Joe,

The system will benefit greatly from more system memory and an appropriate amount of swap to acccommodate it.

The issue is the backup job is too much for the system. Storage of data on raid 5 can make this worse causing i/o wait issues on writes.

SEP

[Jun Yu]

Thanks Steven,

We've gathered performance information and these data indicate the system's hardware components(CPU and memory and it's fiber channel storage system) are still follow the estimation when we architect this system.

And i know the dump+scp nearly 30GB data will certainly eat the memory & swap resource, what i can't understand is such "one time per day" maintenance job cause the swap full and exactly "Keeping Full" even the backup job has finished.

Although the DL760G2(4XEON+6GB) is out of date, but it is still strong for daily transactions.

This is a mission critical application that we can't update/upgrade any parts(includes hardware & software) before we know exactly what is it really impact the next morning performance. Then we can have a upgrade/update plan for it.

Currently we are looking for any simple and direct procedures can help to troubleshoot the problem and temporary work on the problem.

i.e. Add another 2GB swap partition so the system so that we have more swap resource and don't afraid the daily backup job(dump+scp) will not eat all the swap for the next day's business?

Or simply run some command every night after the backup job to force release swap ?

Appreciate for any opinions.

JOe

[Vitaly Karasik_1]

Adding more swap (2*2GB partitions) should help.

>Strange thing is the swap partition can't >be released after the night backup job
It seems like there was a bug (feature?) in earlier 2.4 kernels
(see this thread for example http://www.ussg.iu.edu/hypermail/linux/kernel/0105.0/0050.html)

Why we see this in 2.4.20 kernels? I don't know...