Disk Backup runs slow.

[Uday_S_Ankolekar]

Here is the scenario,

Oracle 8i running on N-Class (HP-UX 11). around 300 Data files

We run HotBackup every night.
The perl script does following things.
Take each data file and copy into a different dirctory and then compress it and then go to next datafile ... This usually runs for 5 hours but this duration increases everyday (Today it is running for 15 hrs.!) But if we bounce the box it will run for 5 hours again and grows slowly day by day.
I want to know why this process keeps on increasing and why reboot fixes it.

Thanks

Uday

[Helen French]

Hi Uday:

Just some thoughts: Do you have any error message/warning in syslog.log ? Anything regrading resource overflow ? Did you check the resource usage with glance ? Any kernel tuning required ? Any patches ? Anything wrong with the script ( some counter increase ?)

HTH,
Shiju

[harry d brown jr]

Uday,

Memory leak? Check your swap area before starting and then afterwards. Hopefully you have glance, if so "watch the process".

live free or die
harry

[A. Clay Stephenson]

Without much data to work with, this sounds like a classic memory leak problem. Have you noticed the Oracle background processes growing with time? I would start checking memory (and shared memory) with Glance.

[S.K. Chan]

More like "process leaking memory" but do not rule out "kernel leaking memory" too. The system or user processes memory leak can be quickly determine. In glance/perfview, you'll see PROC_MEM_VIRT increasing over time. You will need need to look at process-level data too and that's in PROC_MEM_RES and APP_MEM_RES. Bottom line you got a lot of investigating to do here.

[Justo Exposito]

Hi Uday,

Perhaps memory buffers, I reduce my memory buffers cache (dbc_pct) and the backups gone very slow then I increase it and gone fast.
Do you change this kernel param?

Regards,

Justo.

[Uday_S_Ankolekar]

I collect almost all information during this backup process but I'm not able to make out what is causing this.

I've almost all recommended patches on this box.
Attached here is glimpses of system status during backup.

[S.K. Chan]

Those are current snapshot and from a glance it looks fine. What you need to do is run some detection process over time to determine if there is any leaking ..
2 ways ..
=========
1) Use glance ..
# cd /opt/perf/examples/adviser
# glance -adviser_only -syntax proc_mem_leak -j300 > output.file
===> Run glance in adviser mode every 5 minutes with the use of the syntax in "proc_mem_leak".

2) This is from previous thread (thanks JRF)
# UNIX95= ps -e -o "user,vsz,pid,ppid,args" | sort -rnk2 | more
===> Look for growth in "vsz" over time.

[Thomas Schler_1]

Uday:

What about zombie processes? (I have a user who creates zombie processes with his perl scripts.) Maybe your backup scripts also leaves zombies that need resources for nothing.

Create a new account from where you only run the backup scripts, e.g. uid=dbbup, gid=dba. After finishing backups, have a look if there are some processes of this 'database backup user' still on the system (should not).

[Uday_S_Ankolekar]

Chan,

My output also have the famous UNIX95 output for top ten process sorting on vsz.

I would like to run glance in advisory mode. ( Throw some for light on this .Pls.)

Thomas,

There are no Zombies. and once back up finished system runs normally. Users complain slow responce during backup ( This could be due to compress command)

Justo,

dbc_max_pct is 8 and min is 5.

[MANOJ SRIVASTAVA]

Hi Uday

The box is up for 77 days , so I was thinking whether you are talking of bouncing the application/database or the box. Also you might considering increasing dbc_max . What tis teh value of Maxdsiz.


Manoj Srivastava

[S.K. Chan]

Time name pid Data-VSS Shmem-VSS Other-VSS
10:13:05 java 2433 16356kb 2340kb 215856kb
10:13:05 X 1566 26140kb 7972kb 37524kb
10:14:05 java 2433 16356kb 2340kb 215856kb
10:14:05 X 1566 26140kb 7972kb 37524kb
10:15:05 java 2433 16356kb 2340kb 215856kb
10:15:05 X 1566 26140kb 7972kb 37524kb

Sample output running it at 1min interval on my machine. If you look at proc_mem_leak file it defined the threshold for VSS to 30MB by default, meaning it'll only list processes with more than 30MB of VSS. Obviously you can change this to capture a wider range of processes by editing proc_mem_leak and change "VSSthereshold". If you notice the above output the number doesn't change and realistically it should not change much, a significant growth over time will most likely be the suspect.

[Magdi KAMAL]

Hi Uday,

Performance issue is the difficult points to resolve. So it could be any kind of bottlenecks : cpu, memory, disks and/or network.

1. Use the *time* command after a reboot and to measure the time variation for 2 days.

2. Is there any new applications or new releases on your system. Perhaps there is bad process consuming resources.

3. Check if any swap activities.
4. Are you using HP AutoRaid ? if yes, ensure that you are not on Raid 5. Some shutdown procedures are cleaning the system from huge log and temporary files. and after start up these files start growing ! Passing from Raid 0+1 to Raid 5 which is too slow.

5. If you have GlancePlus, Mesure Ware Agent and Perf View, this will allow you to know where you are wasting these plainty of time.

6. if not :

ipcs
iostat
vmstat
sar
netstat

and redirect them to logfiles and analyse them.

Good luck

Magdi