Not enough space

[Jeff_Traigle]

We had a problem with one of our database servers last week. I'm trying to do some postmortem investigation for the cause using Performance Manager. MWA is running on the database server.

The cause of the "net enough space" message appears to be related to memory and swap space. GBL_MEM_SYS_UTIL in particular doubled from 25.64% to 52.2% within 5 minutes. Data buffer cache is set at 5% and GBL_MEM_USER_UTIL was pretty steady at approximately 40%. GBL_SWAP_SPACE jumped from approximately 85% to 100%. The pageout and swapout rates remained 0, however.

The obvious question I have is, "What caused the system memory utilization to jump so drastically?"

So far I haven't been able to make any determination from the Process Details Drill Down Table. It reports on PROC_MEM_VIRT and PROC_MEM_RES, but these values didn't fluctuate very much. The only oddball process that popped up in the report for about 10 minutes at the time the problem occurred was a dd a DBA was apparently running as the oracle user. While that definitely hit the CPU and disk, I don't see that causing major memory problems. (If it could, I'm open to hearing the explanation.)

I haven't seen any metrics that indicate how much SYS vs. USER memory a process was using so I'm thinking this may remain a bit of a mystery, but I figured I would throw this out here to see if anyone might be able to point me in a direction to get the information that I'm not seeing that would shed some more light on the situation.

[Stephen Keane]

dd could affect your free memory badly depending on

(a) How much free memory you had in the first place

and

(b) What blocksize was being used in the dd command. It is common to use a relatively large lock size with dd to improve performance.

I wouldn't have thought that a dd by itself was the cause of your problem, more a straw/camel's back scenario.

[Steven E. Protter]

Collect some data and analyze it.

See attached script set.

Background, production, very helpful.

SEP
Contact form that mail's me offline:
http://www.isnamerica.com/contactsep.shtml

[Steven E. Protter]

duh.

Attachment is in THIS post.

SEP

[Jeff_Traigle]

Memory utilization was about 71% just prior to the spike. There's 16GB so that means roughly 4.5GB was free. There's 4GB of swap so at 85% utilization, there was only about 500MB. These are typical numbers on this box.

I talked to the DBA. He used a block size of 8192 on the dd command. He was copying a 2GB database file he mistakenly created on a cooked file system to a raw partition.