16gb memory for Linux AS2.1

[Ryan Lee]

I have a Linux AS2.1 Server for DL740 8CPU 16GB memory. the server hang when running about 50GB data(even use cp,rsync,tar).the error messages:
ENOMEM in journal_alloc_journal_head, retrying.
ENOMEM in journal_get_undo_access_Rsmp_767cdac6, retrying.
ENOMEM in get_transaction, retrying.
ENOMEM in journal_alloc_journal_head, retrying.

[Steven E. Protter]

Are you booting off the bigmem kernel?

This could be a factor.

Are there any messages in /var/log/messages?

Post them please.

SEP

[Jerome Henry]

It seems, on reading Redhat bugzilla list, that this is a bug in kernel itself... when it tries to allocate memory on large files, faster than RAM controller can output to disk, so the kernel keeps trying to allocate space, while ext3 tries itself to write system state on its file, which is impossible as there isn't already enough space for the file transfert !

Some developpers suggest to try to turn back to ext2, but some others seem to assess that it'll just hide the message but not solve the problem, the dolution being to implement a wait command in the kernel...

[Ryan Lee]

Thanks for your reply, I updated kernel version is 2.4.9-e.25. I try to test 8gb memory, not hang. and try to 14gb memory, the server hang again.also updated patch as below.
gcc-2.96.118
glibc-2.2.4-31
mm-1.1.3-8
*test tools*
http://people.redhat.com/dledford/memtest.html

[Ryan Lee]

I try to reverting to ext2, the linux still hang, but not shows any ENOMEM error messages.
When the linux hang before, I seen the Lowfree item very low.

[Stuart Browne]

Ming, you might find it useful to jump on this bandwagon:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=99025

Same errors you are having, only difference being they are using 32GB and not 16GB of memory.

You might want to throw your findings in there to see if it can help that along. I'm picking it's the same issue, given the details you've discussed with us thusfar.

[Ryan Lee]

I getting redhat recommendation to tuning vm of freepages/pagecache(1276 2552 3828
/2 20 40).
After tuning, the system no hang, but the memory usage and cache szie is lower that before. is it affect the performance??
how can I tuning right setting for my linux.

Following is sysrq-m output

Sep 16 13:53:19 phi kernel: SysRq : Show Memory
Sep 16 13:53:19 phi kernel: Mem-info:
Sep 16 13:53:19 phi kernel: Free pages: 12693268kB (12691744kB HighMem)
Sep 16 13:53:24 phi kernel: ( Active: 778081, inactive_dirty: 184,
inactive_clean: 45, free: 3173317 (1276 2552 3828) )
Sep 16 13:53:24 phi kernel: 1*4kB 9*8kB 7*16kB 2*32kB 2*64kB 1*128kB 0*256kB
0*512kB 0*1024kB 0*2048kB 0*4096kB = 508kB
Sep 16 13:53:24 phi kernel: active: 63, inactive_dirty: 0, inactive_clean: 0,
free: 127 (128 256 384)
Sep 16 13:53:25 phi kernel: 30*4kB 12*8kB 32*16kB 9*32kB 0*64kB 0*128kB 0*256kB
0*512kB 0*1024kB 0*2048kB 0*4096kB = 1016kB
Sep 16 13:53:25 phi kernel: active: 63, inactive_dirty: 184, inactive_clean: 0,
free: 254 (255 510 765)
Sep 16 13:53:27 phi kernel: 8488*4kB 99570*8kB 44661*16kB 16925*32kB 4688*64kB
1210*128kB 429*256kB 130*512kB 18*1024kB 1*2048kB 2430*4096kB = 12691744kB
Sep 16 13:53:28 phi kernel: active: 777955, inactive_dirty: 0, inactive_clean:
45, free: 3172936 (255 510 765)
Sep 16 13:53:28 phi kernel: Swap cache: add 0, delete 0, find 0/0
Sep 16 13:53:32 phi kernel: Page cache size: 778000
Sep 16 13:53:32 phi kernel: Buffer mem: 310
Sep 16 13:53:33 phi kernel: count_ramdisk_pages: pagemap_lru_lock locked
Sep 16 13:53:33 phi kernel: Ramdisk pages: 0
Sep 16 13:53:34 phi kernel: Free swap: 2044056kB
Sep 16 13:53:34 phi kernel: 4259839 pages of RAM
Sep 16 13:53:37 phi kernel: 3964911 pages of HIGHMEM
Sep 16 13:53:38 phi kernel: 137443 reserved pages
Sep 16 13:53:42 phi kernel: 818758 pages shared
Sep 16 13:53:49 phi kernel: 0 pages swap cached
Sep 16 13:53:51 phi kernel: 28 pages in page table cache
Sep 16 13:53:52 phi kernel: 148399 pages in slab cache
Sep 16 13:53:53 phi kernel: Buffer memory: 1240kB
Sep 16 13:53:53 phi kernel: CLEAN: 1041479 buffers, 2084061 kbyte, 272 used
(last=1041471), 0 locked, 0 protected, 0 dirty