Sybase backup server performance under HP-UX 11i

[Raul Aviles]

Hi, we have a RP8400 server with 8 processors, 8gb of ram. We are running HP-UX 11i. Out database is Sybase 12.0.
The server is normally 40% idle of processor usage but when we run the backup process thru Sybase Backup Server idle time goes to 0% and online application that uses Sybase have a really bad response time. How can we make that Sybase Backup Server don´t steal resources to online applications?
Actually we have assigned a higher nice value to the backup process, that means a lower priority than dataserver process and application process.

Thanks in advan

[Michael Schulte zur Sur]

Hi,

how many online engines and backup server processes have you got?

greetings,

Michael

[Bernhard Mueller]

Raul,

using nice is actually the way to go and keep the backupserver process out of the way as much as possible, you may wish to even use a "radical" nice value such as 19 if it impair crucial performance.

Now, for your understanding: Do not expect to always have an idle value of 30 or 20 percent by starting the backupserver using a nice value. Actually, the nice value comes into play *only* when the system is at 0% idle! Then nice says: "you first my friend"

If there are CPU resources available *why* shouldn't they be used? You paid enough money for them, so they should do some work, eh? ;-)

Acutally, when you test this (gzip is a good one for doing this!) you can see in "top" that for a given process such as zipping the kernel and unzipping, it you will always use about the same TIME (i.e. CPU TIME) however total execution duration (i.e. real time) will be much longer, but *only* when the system has other processes running requiring all avail CPU...

E.g. with 8 CPUs staring 9 gzips at the same time but with different nice values where all gzips will be compressing a copy of the same big file will show you what I mean.

Regards,
Bernhard

[Bernhard Mueller]

Now you tried it with nice and still are not satisfied?

Then you should consider using PRM (Process Ressource Manager) this helps you reserve resources (incl. memory) for critical processes.

http://docs.hp.com/hpux/onlinedocs/B8733-90009/B8733-90009.html

Regards,
Bernhard

[ALPER ONEY]

Hi,
Could you post the output of the following command?
cat $SYBASE/$SYBASE_ASE/install/RUN_

Regards.
ALPER ONEY
I.S.E TAKASBANK INC

[Raul Aviles]

Hi, everybody
First I have to tell you that we really have 6 processor working and 2 processor in ICOD mode (it means not used but can be used in the future on demand).
Next my answers:
Michael : 5 processors are assigned to Sybase and there area 5 engines. Every two hours we run 3 backup process.
Bernhard : We agree that is ok that we used 100% of all processor . That is wright. But the problem is that the response time of online request is bad when we run the backup process. As I told you the nice value of backup process is 30 and nice value of online application is 20, but it seems that backup process is stealing resources from online application. How can we control this?

Next I post the output of the following command?
cat $SYBASE/$SYBASE_ASE/install/RUN_

sybase>$cat RUN_SYBASE_back
#!/bin/sh
#
# Backup Server name: SYBASE_back
# Error log path: /sybase12/ASE-12_0/install/SYBASE_back.log
# Maximum number of network connections: 25
# Maximum number of server connections: 20
# Multibuf executable path: /sybase12/ASE-12_0/bin/sybmultbuf
#
/sybase12/ASE-12_0/bin/backupserver \
-SSYBASE_back \
-e/sybase12/ASE-12_0/install/SYBASE_back.log \
-N25 \
-C20 \
-M/sybase12/ASE-12_0/bin/sybmultbuf \
-m1024 \
sybase>$

Regards

[Bernhard Mueller]

Raul

as mentioned, with nice you can run it at 39 instead of 30, that is all you can do.

Since I understand this is am important production server, it may be actully worth spending 10k$ and use PRM. That should solve your problems.

Alternative thing to consider is activating the iCOD CPUS and add memory. Do you observe page-outs in vmstat? Maybe your system is swapping when the backupserver starts heavy processing, you can't get that under control with nice or anything else but more memory...

Regards,
Bernhard

[Michael Schulte zur Sur]

Hi,

please check, if the backup server using dump compression. You may use a compression, which is too high and have to explicitly assign a lower one. I found this at sybase.com

instead of
dump database to "compress::/work/compressed.dmp"
use
dump database to "compress::1::/work/

greetings,

Michael

[Raul Aviles]

Hi,
Michael : I have been checking information about backup server in sybase.com but i don´t find anything about the compress parameter. Would you please send me the article about the configuration of compress parameter

Thanks in advan

[Michael Schulte zur Sur]

Hi,

have a look at:
http://info.sybase.com/resolution/detail.stm?id_number=10840416
http://info.sybase.com/resolution/detail.stm?id_number=10878193
http://info.sybase.com/resolution/detail.stm?id_number=10416279
http://info.sybase.com/resolution/detail.stm?id_number=10947935

I hope, that one of these gives a clue to your problem,

Michael

[Bill Hassell]

One important 'feature' of a backup: most backup programs don't consume a lot of CPU cycles reading data and writing to a tape, but the amount of disk I/O is massive, and sequential. So regardless of how nice you make the backup, the number of disk I/O's per second won't really change and all of your online applications will be severely impacted as everything will be queued together. PRM will control I/O resources but basic HP-UX cannot. About the only thing you might try is tuning the disksort_seconds value. Start with 4 and perhaps down to 2. This will bump up the queing order for random I/O since the HP-UX disk I/O code gives preference to sequential I/O.

[Raul Aviles]

HI,
Bill : What meant to set to four or two the disksort_seconds parameters. I have been looking for information about this paramater and all I found is that "is the maximun wait for i/o".
But this parameters only affect sequential i/o?

Regards

[Bill Hassell]

There's a lengthy explanation in the patch for 11.0 (PHKL_21678) that describes the details. Unfortunately there isn't a lot of docs (spelled: disksort_seconds) about it except in the patch:

"This is a fairness problem with the disk sort algorithm. The disk sort algorithm is used to reduce the disk head retractions. With this algorithm, all I/O requests with the same priority are queued in non-descending order of disk block number before being processed if the queue is not empty. When requests come in faster than they can be processed, the queue becomes longer, the time needed to perform one scan (from smallest block number to largest block number of the disk) could be very long in the worst case scenarios. It is unfair for the request which came in early but has been continuously pushed back to the end of the queue because it has a large block number or it just missed the current scan. These kind of unlucky requests could line up in the queue for as long as the time needed for processing a whole scan (which could take a few minutes). This situation usually happens when a process tries to access a disk while another process is performing sequential accesses to the same disk.

Resolution:

To prevent this problem from happening, we have to take the time aspect into consideration in the sorting algorithm. We add a time stamp for each request when it is enqueued, which is used as the second sorting key for the queue (1st key: process priority; 2nd key: enqueued time; 3rd key: block number). The granularity of the time stamp value is controlled by a new tunable "disksort_seconds". If we set "disksort_seconds" to N (N>0), for all the requests with the same priority, we can guarantee that any given request will be processed earlier than those which come in N seconds later than this request. Within each N second period (requests have the same time stamp), all requests are sorted by non-descending block number order. By choosing the right "disksort_seconds" value, we can balance the maximum waiting time of requests and the efficiency of disk accesses. The tunable parameter can be set to 0, 1, 2, 4, 8, 16, 32, 64, 128 or 256 second(s). If "disksort_seconds" is 0 (default value), the time stamp is disabled, which means that time aspect is not taking effect."

[Larry O'Reilly]

Are you running dump database rather than dump transaction every two hours?

[Michael Schulte zur Sur]

Hi,

how far are you with your problem?

be happy,

Michael

[ALPER ONEY]

Hi,
I was just wondering why you use the parameter "-m" in your RUN file of backupserver. Do you want to improve dump or load performance? I have not needed to tune this parameter for my SYBASE system so far. I am not sure but it is worth trying to remove the last line which begins with -m parameter from your RUN file of backupserver in order to achieve your goal.
I am pasting a section of the ASE system administration guide at the end of the post.
Regards
ALPER ONEY
I.S.E TAKASBANK INC.
TEL:+902123152109

From ASE System Admin. Guide
------------------------------------
When you start Backup Server, you can use the -m parameter to improve the performance of the dump and load commands by configuring more shared memory for the Backup Server. The -m parameter specifies the maximum amount of shared memory used by the Backup Server. You must also configure your operating system to ensure that this amount of shared memory is available to the Backup Server. After a dump or load operation is completed, its shared memory segments are released.

Note: , Configuring more shared memory improves dump/load performance only if the performance limits of the hardware setup have not been reached. Increasing the value of -m may not result in improved performance when dumping to a slow tape device such as QIC, but it can improve performance significantly when dumping to a faster device such as DLT.
Syntax
The syntax is:

backupserver [-m nnn]
where nnn is the maximum amount of shared memory in megabytes that the Backup Server can use for all of its dump or load sessions.

Comments
This option sets the upper limit for shared memory usage. However, Backup Server may use less memory than specified if it detects that adding more memory will not improve performance.

Backup Server determines the amount of shared memory available for each stripe by dividing the -m value by the configured number of service threads (-P parameter).

The default value for -m is the number of service threads multiplied by 1MB. The default value for -P is 48, so the default maximum shared memory utilization is 48MB. However, Backup Server reaches this usage only if all the 48 service threads are active concurrently.

If you increase the maximum number of service threads, increase the -m value, also. If you increase the -P value but do not increase the -m value, the shared memory allocated per stripe can decrease to the point that the dump or load cannot be processed.

To determine how much to increase the -m value, use this formula:

(-m value in MB) * 1024/(-P value)

If the value obtained by this formula is less than 128KB, Backup Server will not boot.

The minimum value for -m is 6MB. The maximum value for -m depends on operating system limits on shared memory.

If you create a dump using a Backup Server with a high shared memory value, and attempt to load the dump using a Backup Server with a lower shared memory value, Backup Server uses only the available memory. This results in degradation of the load performance.

If the amount of shared memory available per stripe at load time is less than twice the block size used at dump time, Backup Server aborts the load with an error message.