topic Re: Disk Backup runs slow. in Operating System - HP-UX

Disk Backup runs slow.

Uday_S_Ankolekar — Mon, 06 May 2002 14:18:40 GMT

Here is the scenario,

Oracle 8i running on N-Class (HP-UX 11). around 300 Data files

We run HotBackup every night.
The perl script does following things.
Take each data file and copy into a different dirctory and then compress it and then go to next datafile ... This usually runs for 5 hours but this duration increases everyday (Today it is running for 15 hrs.!) But if we bounce the box it will run for 5 hours again and grows slowly day by day.
I want to know why this process keeps on increasing and why reboot fixes it.

Thanks

Uday

Re: Disk Backup runs slow.

Helen French — Mon, 06 May 2002 15:13:58 GMT

Hi Uday:

Just some thoughts: Do you have any error message/warning in syslog.log ? Anything regrading resource overflow ? Did you check the resource usage with glance ? Any kernel tuning required ? Any patches ? Anything wrong with the script ( some counter increase ?)

HTH,
Shiju

Re: Disk Backup runs slow.

harry d brown jr — Mon, 06 May 2002 15:16:11 GMT

Uday,

Memory leak? Check your swap area before starting and then afterwards. Hopefully you have glance, if so "watch the process".

live free or die
harry

Re: Disk Backup runs slow.

A. Clay Stephenson — Mon, 06 May 2002 15:18:54 GMT

Without much data to work with, this sounds like a classic memory leak problem. Have you noticed the Oracle background processes growing with time? I would start checking memory (and shared memory) with Glance.

Re: Disk Backup runs slow.

S.K. Chan — Mon, 06 May 2002 15:24:24 GMT

More like "process leaking memory" but do not rule out "kernel leaking memory" too. The system or user processes memory leak can be quickly determine. In glance/perfview, you'll see PROC_MEM_VIRT increasing over time. You will need need to look at process-level data too and that's in PROC_MEM_RES and APP_MEM_RES. Bottom line you got a lot of investigating to do here.

Re: Disk Backup runs slow.

Justo Exposito — Mon, 06 May 2002 15:24:54 GMT

Hi Uday,

Perhaps memory buffers, I reduce my memory buffers cache (dbc_pct) and the backups gone very slow then I increase it and gone fast.
Do you change this kernel param?

Regards,

Justo.

Re: Disk Backup runs slow.

Uday_S_Ankolekar — Mon, 06 May 2002 15:36:58 GMT

I collect almost all information during this backup process but I'm not able to make out what is causing this.

I've almost all recommended patches on this box.
Attached here is glimpses of system status during backup.

Re: Disk Backup runs slow.

S.K. Chan — Mon, 06 May 2002 15:51:56 GMT

Those are current snapshot and from a glance it looks fine. What you need to do is run some detection process over time to determine if there is any leaking ..
2 ways ..
=========
1) Use glance ..
# cd /opt/perf/examples/adviser
# glance -adviser_only -syntax proc_mem_leak -j300 > output.file
===> Run glance in adviser mode every 5 minutes with the use of the syntax in "proc_mem_leak".

2) This is from previous thread (thanks JRF)
# UNIX95= ps -e -o "user,vsz,pid,ppid,args" | sort -rnk2 | more
===> Look for growth in "vsz" over time.

Re: Disk Backup runs slow.

Thomas Schler_1 — Mon, 06 May 2002 15:52:43 GMT

Uday:

What about zombie processes? (I have a user who creates zombie processes with his perl scripts.) Maybe your backup scripts also leaves zombies that need resources for nothing.

Create a new account from where you only run the backup scripts, e.g. uid=dbbup, gid=dba. After finishing backups, have a look if there are some processes of this 'database backup user' still on the system (should not).

Re: Disk Backup runs slow.

Uday_S_Ankolekar — Mon, 06 May 2002 16:03:18 GMT

Chan,

My output also have the famous UNIX95 output for top ten process sorting on vsz.

I would like to run glance in advisory mode. ( Throw some for light on this .Pls.)

Thomas,

There are no Zombies. and once back up finished system runs normally. Users complain slow responce during backup ( This could be due to compress command)

Justo,

dbc_max_pct is 8 and min is 5.

Re: Disk Backup runs slow.

MANOJ SRIVASTAVA — Mon, 06 May 2002 16:24:27 GMT

Hi Uday

The box is up for 77 days , so I was thinking whether you are talking of bouncing the application/database or the box. Also you might considering increasing dbc_max . What tis teh value of Maxdsiz.

Manoj Srivastava

Re: Disk Backup runs slow.

S.K. Chan — Mon, 06 May 2002 16:27:37 GMT

Time name pid Data-VSS Shmem-VSS Other-VSS
10:13:05 java 2433 16356kb 2340kb 215856kb
10:13:05 X 1566 26140kb 7972kb 37524kb
10:14:05 java 2433 16356kb 2340kb 215856kb
10:14:05 X 1566 26140kb 7972kb 37524kb
10:15:05 java 2433 16356kb 2340kb 215856kb
10:15:05 X 1566 26140kb 7972kb 37524kb

Sample output running it at 1min interval on my machine. If you look at proc_mem_leak file it defined the threshold for VSS to 30MB by default, meaning it'll only list processes with more than 30MB of VSS. Obviously you can change this to capture a wider range of processes by editing proc_mem_leak and change "VSSthereshold". If you notice the above output the number doesn't change and realistically it should not change much, a significant growth over time will most likely be the suspect.

Re: Disk Backup runs slow.

Magdi KAMAL — Fri, 10 May 2002 06:30:47 GMT

Hi Uday,

Performance issue is the difficult points to resolve. So it could be any kind of bottlenecks : cpu, memory, disks and/or network.

1. Use the *time* command after a reboot and to measure the time variation for 2 days.

2. Is there any new applications or new releases on your system. Perhaps there is bad process consuming resources.

3. Check if any swap activities.
4. Are you using HP AutoRaid ? if yes, ensure that you are not on Raid 5. Some shutdown procedures are cleaning the system from huge log and temporary files. and after start up these files start growing ! Passing from Raid 0+1 to Raid 5 which is too slow.

5. If you have GlancePlus, Mesure Ware Agent and Perf View, this will allow you to know where you are wasting these plainty of time.

6. if not :

ipcs
iostat
vmstat
sar
netstat

and redirect them to logfiles and analyse them.

Good luck

Magdi