Memory climbs to 100% during DB backup and stays there

[Steve Valvasori_1]

Configuration:
rp7420 w/11.i (4cpu/8GB)
Oracle 9i using BMC OBacktrack
Veritas Cluster Database Edition v3.5

We have a strange problem when we try to do a cold backup of our databases.
Normally the system runs ~60% memory utilization. When we start the cold backup, memory dips slightly, then ramps up to ~80% and stays there for a few minutes. If no problems occur, after the backup memory resumes it's normal 60%.
About 50% of the time, after a few minutes into the backup at 80% memory usage, it shoots up to 100% and stays there. This virtually hangs the server and we have to reboot it to solve the problem.
All the recommended patches are installed. We can reproduce the problem with the cluster up and down.

Any ideas?

Thanks,

Steve.

[Kent Ostby]

Well, one drastic measure would be to get the system hung and then TC the box and open a case with HP.

It's not the most elegant solution, but it would get you an answer.

HP has tools that can tell where the memory usage issues are.

I would assume that you've talked to whoever it is that supplies your backup software about patches there.

Hope this helps,

Oz

[Steve Valvasori_1]

I've gone the TC route, but the reponse from HP response center was:

"HP's protocol is to only analyze dumps that are generated by a system crash .
If the system hung and crashed on it's own , we'll be happy to take a look at it."

They want me to run a performance script when the problem occurs - but the server is hung and nothing will run correctly.

We have our DBA's looking into the obacktrack side, I want to make sure I've covered all the OS/Cluster possibilities.

Should the memory increase when a successfull backup is taking place? I would think since this is a cold backup, the database will shutdown - releasing memory?

[RAC_1]

So at the time of backup the database (which under cluster) is down. Right??

You start the backup with BMC oTrack. Right??
Now as backup starts, the memoru utlization goes up. Right?? When it is at about 80 % or so, do following to know what processes are using most of the memory.

glance -m
glance -g

UNIX95= ps -ef -o 'pid,ppid,vsz,args,ruser' | sort -nrk3

Check what are those processes. Is there any config file for BMC oTrack?? You should give it a look. Also how does disk utlization show when this whole thing is going on.

I would also suggest to put UNIX95 command in crontab, when you start backup and keep it running at regular inetrval (1 min.) and analyze it.

Anil

[Steve Valvasori_1]

Thanks for the idea's Anil - I've looked at glance during the high memory period and can't see any processes that are using all this memory.

We tested another database instance on the same server and we did not see this problem, so at least we are now pointing away from the OS or the Cluster as culprits.

I won't be able to test further until Monday so I'll update this then.

Steve.

[generic_1]

I think your backup app has a problem and needs a patch.

Also I am guessing your maxdsize is so high that it might be allowing it to exploit all your memory which is good expect when you have applications that keep taking more and more memory.
Take a look at swapinfo -t and also make sure your swap is big enough.

Hope this helps.

[Steve Valvasori_1]

Current kernel setting - maxdsiz_64bit 0x80000000
Swap space is okay (~45%), however once memory goes to 100%, vhand (swapper process) hits 100%+ CPU utilization, which I guess is to be expected.

[Emil Velez]

if you said you had glance maybe you have measurware (Openview Performance agent) on the system too. That tool is collecting 5 minute global information but also is collecting 1 minute process information on "interesting" processes. What that means is that if a process is using alot of memory, cpu or disk resources you will get data every minute about that process telling how much memory it is using, cpu useage and about 70 other metrics.

I would suggest running a extract of the process data in measureware to see what processes are running every minute and extract the global metrics to see what your global memory looks like.

Emil Velez

[Steve Valvasori_1]

Thank you all for your help on this. After testing on another DB instance on the same server went fine (as you would expect, memory down during the backup), we are now confident in asking the DBA's to look into this as it doesn't look like an OS or Cluster related issue.

Thanks again,

Steve.

[Steve Valvasori_1]

on to the DBA's then....