System crash "clnt_dg_create: out of memory"

[Luke Chuang]

Help. Mysterious problems crash a HP server.

My company has a HP Unix server that host an Oracle database. It was running OK until we upgraded its OS from 11.0 to OS 11.11 last month. We also upgraded Oracle from 8.1.6 to 9.2.0.1.

This server started to crash three weekends in a row. At first, I thought it simply ran out of memory. So I reduced Oracle instance size from 66% to 50% then to 33% of system memory. It still crashed (locked out) over the weekend. The error message is "clnt_dg_create: out of memory". Our system admin had to shut down and restart the server to make the server works again.

The problem is the server only crashed (locked out) over the weekend. My company is not a 24-7 shop. There is very little server activities over the weekend. This server is for internal uses only. Our system admin thinks this maybe something to do with the system backup process. (The way I backup the Oracle database is to create export dump files to VA7410 SAN. HP Omniback will backup SAN over the weekend.)

Server: HP9000 L2000 (1 CPU/ 1GB Memory)
OS: 11.11
Database: Oracle 9.2.0.1 64 bit (20GB total datafiles)
Storage: Built-in Autoraid and a VA7410 SAN (SAN is used to store Oracle export files).

Any suggestion and advice is welcome. Thanks in advances.

Luke Chuang

[Tore_1]

These kinds of problems are a bit to complicated for this forum. You need to place a support call at HP.

[Steven E. Protter]

Oracle is a hungry user of shared memory resources.

shmmax in the kernel should be set to about 1/4 of available memory which is physical memory plus swap.

swap needs to be set to twice physical memory on your machine. I admin the same machine, though mine have twice to four times teh memory.

If you mean by crash that the system reboots and the red fault light comes on.

Check for dumps in /var/adm/crash

If you find one, analyze them with q4 according to the document I attached.

Check /var/tombstones, the 98 and 99 files.

endif

Regards,

SEP

[Cheryl Griffin]

"clnt_dg_create: out of memory" indicates you have a memory leak. Check your patches, starting with Oracle recommended patches (in the release notes off the Oracle OTN site), then check for the latest RPC, NFS patch:
PHNE_28568 and it's dependencies.

[Michael Steele_2]

First occurrance at:

May 25 07:50:04 atlas syslog: clnt_dg_create: out of memory

Can you check your /var/adm/cron/log file for this time index.

Any cron going off at 07:50?

Below thread has same error for CIFS. Are you running SAMBA? NFS?

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xadb1afe90f1cd71190050090279cd0f9,00.html

This would indicate a network problem. Are your backups over the network?

I would suspect a network buffer overflow. Check this log:

netfmt -LN -f /var/adm/nettl.LOG* > file

vi and search for 'disaster', 'error', 'cable' as well as the time index.

Now the crashing:

Check the time in /etc/shutdownlog and paste in the panic string which should be the last line.

Also try to get a crash dump which may still be usable:

savecrash -rf /file

Let me know if you obtain a crash.

Finaly look into /var/tombstones/ts99 and note the GMT time under HPMC. See if the time matches.

[Jean-Louis Phelix]

Hi,

I think that you 'simply' have a memory leak. Perhaps one of your 11.00 software doesn't work correctly on 11.11. Try to follow memory usage for example using a top during next week-end and try to find out which process has a growing memory usage.

For example

nohup top -f /tmp/top.out -s 300 -d 300 &

will run top every 5 minutes 300 times.

Regards.

[Luke Chuang]

Thanks for your quick responses. Our one and only unix system admin is in his honey moon this week. I will forward your solutions to him and hopefully he can resolve the problem next week. Thanks again.

Luke Chuang
DBA

[Bruno Vidal]

Hi,
It appears to be a memory leak, so first you must ensure that you don't have any application eating everything (use sar/vmstat/glance). Now, if you are sure that it is related to your week-end backup, it can be a well known VxFs problem and inode. So first try to use kernel parameters vx_ninode.
Set like this:
vx_ninode = 2048 (but not too high, you don't have much memory). btw this parameter comes with PHKL_26761.

Cheers.