Oracle 8i running on N-Class (HP-UX 11). around 300 Data files
We run HotBackup every night. The perl script does following things. Take each data file and copy into a different dirctory and then compress it and then go to next datafile ... This usually runs for 5 hours but this duration increases everyday (Today it is running for 15 hrs.!) But if we bounce the box it will run for 5 hours again and grows slowly day by day. I want to know why this process keeps on increasing and why reboot fixes it.
Just some thoughts: Do you have any error message/warning in syslog.log ? Anything regrading resource overflow ? Did you check the resource usage with glance ? Any kernel tuning required ? Any patches ? Anything wrong with the script ( some counter increase ?)
Without much data to work with, this sounds like a classic memory leak problem. Have you noticed the Oracle background processes growing with time? I would start checking memory (and shared memory) with Glance.
More like "process leaking memory" but do not rule out "kernel leaking memory" too. The system or user processes memory leak can be quickly determine. In glance/perfview, you'll see PROC_MEM_VIRT increasing over time. You will need need to look at process-level data too and that's in PROC_MEM_RES and APP_MEM_RES. Bottom line you got a lot of investigating to do here.
Perhaps memory buffers, I reduce my memory buffers cache (dbc_pct) and the backups gone very slow then I increase it and gone fast. Do you change this kernel param?
Those are current snapshot and from a glance it looks fine. What you need to do is run some detection process over time to determine if there is any leaking .. 2 ways .. ========= 1) Use glance .. # cd /opt/perf/examples/adviser # glance -adviser_only -syntax proc_mem_leak -j300 > output.file ===> Run glance in adviser mode every 5 minutes with the use of the syntax in "proc_mem_leak".
2) This is from previous thread (thanks JRF) # UNIX95= ps -e -o "user,vsz,pid,ppid,args" | sort -rnk2 | more ===> Look for growth in "vsz" over time.
What about zombie processes? (I have a user who creates zombie processes with his perl scripts.) Maybe your backup scripts also leaves zombies that need resources for nothing.
Create a new account from where you only run the backup scripts, e.g. uid=dbbup, gid=dba. After finishing backups, have a look if there are some processes of this 'database backup user' still on the system (should not).
My output also have the famous UNIX95 output for top ten process sorting on vsz.
I would like to run glance in advisory mode. ( Throw some for light on this .Pls.)
Thomas,
There are no Zombies. and once back up finished system runs normally. Users complain slow responce during backup ( This could be due to compress command)
The box is up for 77 days , so I was thinking whether you are talking of bouncing the application/database or the box. Also you might considering increasing dbc_max . What tis teh value of Maxdsiz.
Time name pid Data-VSS Shmem-VSS Other-VSS 10:13:05 java 2433 16356kb 2340kb 215856kb 10:13:05 X 1566 26140kb 7972kb 37524kb 10:14:05 java 2433 16356kb 2340kb 215856kb 10:14:05 X 1566 26140kb 7972kb 37524kb 10:15:05 java 2433 16356kb 2340kb 215856kb 10:15:05 X 1566 26140kb 7972kb 37524kb
Sample output running it at 1min interval on my machine. If you look at proc_mem_leak file it defined the threshold for VSS to 30MB by default, meaning it'll only list processes with more than 30MB of VSS. Obviously you can change this to capture a wider range of processes by editing proc_mem_leak and change "VSSthereshold". If you notice the above output the number doesn't change and realistically it should not change much, a significant growth over time will most likely be the suspect.
Performance issue is the difficult points to resolve. So it could be any kind of bottlenecks : cpu, memory, disks and/or network.
1. Use the *time* command after a reboot and to measure the time variation for 2 days.
2. Is there any new applications or new releases on your system. Perhaps there is bad process consuming resources.
3. Check if any swap activities. 4. Are you using HP AutoRaid ? if yes, ensure that you are not on Raid 5. Some shutdown procedures are cleaning the system from huge log and temporary files. and after start up these files start growing ! Passing from Raid 0+1 to Raid 5 which is too slow.
5. If you have GlancePlus, Mesure Ware Agent and Perf View, this will allow you to know where you are wasting these plainty of time.
