Simpler Navigation for Servers and Operating Systems - Please Update Your Bookmarks
Completed: a much simpler Servers and Operating Systems section of the Community. We combined many of the older boards, so you won't have to click through so many levels to get at the information you need. Check the consolidated boards here as many sub-forums are now single boards.
If you have bookmarked forums or discussion boards in Servers and Operating Systems, we suggest you check and update them as needed.
General
cancel
Showing results for 
Search instead for 
Did you mean: 

Memory climbs to 100% during DB backup and stays there

Steve Valvasori_1
Occasional Advisor

Memory climbs to 100% during DB backup and stays there

Configuration:
rp7420 w/11.i (4cpu/8GB)
Oracle 9i using BMC OBacktrack
Veritas Cluster Database Edition v3.5

We have a strange problem when we try to do a cold backup of our databases.
Normally the system runs ~60% memory utilization. When we start the cold backup, memory dips slightly, then ramps up to ~80% and stays there for a few minutes. If no problems occur, after the backup memory resumes it's normal 60%.
About 50% of the time, after a few minutes into the backup at 80% memory usage, it shoots up to 100% and stays there. This virtually hangs the server and we have to reboot it to solve the problem.
All the recommended patches are installed. We can reproduce the problem with the cluster up and down.

Any ideas?

Thanks,

Steve.
9 REPLIES
Kent Ostby
Honored Contributor

Re: Memory climbs to 100% during DB backup and stays there

Well, one drastic measure would be to get the system hung and then TC the box and open a case with HP.

It's not the most elegant solution, but it would get you an answer.

HP has tools that can tell where the memory usage issues are.

I would assume that you've talked to whoever it is that supplies your backup software about patches there.

Hope this helps,

Oz
"Well, actually, she is a rocket scientist" -- Steve Martin in "Roxanne"
Steve Valvasori_1
Occasional Advisor

Re: Memory climbs to 100% during DB backup and stays there

I've gone the TC route, but the reponse from HP response center was:

"HP's protocol is to only analyze dumps that are generated by a system crash .
If the system hung and crashed on it's own , we'll be happy to take a look at it."

They want me to run a performance script when the problem occurs - but the server is hung and nothing will run correctly.

We have our DBA's looking into the obacktrack side, I want to make sure I've covered all the OS/Cluster possibilities.

Should the memory increase when a successfull backup is taking place? I would think since this is a cold backup, the database will shutdown - releasing memory?
RAC_1
Honored Contributor

Re: Memory climbs to 100% during DB backup and stays there

So at the time of backup the database (which under cluster) is down. Right??

You start the backup with BMC oTrack. Right??
Now as backup starts, the memoru utlization goes up. Right?? When it is at about 80 % or so, do following to know what processes are using most of the memory.

glance -m
glance -g

UNIX95= ps -ef -o 'pid,ppid,vsz,args,ruser' | sort -nrk3

Check what are those processes. Is there any config file for BMC oTrack?? You should give it a look. Also how does disk utlization show when this whole thing is going on.

I would also suggest to put UNIX95 command in crontab, when you start backup and keep it running at regular inetrval (1 min.) and analyze it.

Anil
There is no substitute to HARDWORK
Steve Valvasori_1
Occasional Advisor

Re: Memory climbs to 100% during DB backup and stays there

Thanks for the idea's Anil - I've looked at glance during the high memory period and can't see any processes that are using all this memory.

We tested another database instance on the same server and we did not see this problem, so at least we are now pointing away from the OS or the Cluster as culprits.

I won't be able to test further until Monday so I'll update this then.

Steve.
generic_1
Respected Contributor

Re: Memory climbs to 100% during DB backup and stays there

I think your backup app has a problem and needs a patch.

Also I am guessing your maxdsize is so high that it might be allowing it to exploit all your memory which is good expect when you have applications that keep taking more and more memory.
Take a look at swapinfo -t and also make sure your swap is big enough.

Hope this helps.
Steve Valvasori_1
Occasional Advisor

Re: Memory climbs to 100% during DB backup and stays there

Current kernel setting - maxdsiz_64bit 0x80000000
Swap space is okay (~45%), however once memory goes to 100%, vhand (swapper process) hits 100%+ CPU utilization, which I guess is to be expected.
Emil Velez
Honored Contributor

Re: Memory climbs to 100% during DB backup and stays there

if you said you had glance maybe you have measurware (Openview Performance agent) on the system too. That tool is collecting 5 minute global information but also is collecting 1 minute process information on "interesting" processes. What that means is that if a process is using alot of memory, cpu or disk resources you will get data every minute about that process telling how much memory it is using, cpu useage and about 70 other metrics.

I would suggest running a extract of the process data in measureware to see what processes are running every minute and extract the global metrics to see what your global memory looks like.

Emil Velez
Steve Valvasori_1
Occasional Advisor

Re: Memory climbs to 100% during DB backup and stays there

Thank you all for your help on this. After testing on another DB instance on the same server went fine (as you would expect, memory down during the backup), we are now confident in asking the DBA's to look into this as it doesn't look like an OS or Cluster related issue.

Thanks again,

Steve.
Steve Valvasori_1
Occasional Advisor

Re: Memory climbs to 100% during DB backup and stays there

on to the DBA's then....