cmahostd: page allocation failure. order:5, mode:0xd1

[Charles Castelain_1]

Hello,

I have installed a DL385 G1 with redhat4 update4 x86_64.

I have tested with both psp-7.52 and psp-7.60.

I receive in loop in /var/log/messages :

Oct 9 17:48:53 brachinfo001 kernel: cmahostd: page allocation failure. order:5, mode:0xd1
Oct 9 17:48:53 brachinfo001 kernel:
Oct 9 17:48:53 brachinfo001 kernel: Call Trace:{__alloc_pages+768} {__get_free_pages+11}
Oct 9 17:48:53 brachinfo001 kernel: {:cpqci:CpqCiDrvOsMallocContiguous+75}
Oct 9 17:48:53 brachinfo001 kernel: {:cpqci:CpqCiDrvCreateChannel+619}
Oct 9 17:48:53 brachinfo001 kernel: {:cpqci:cpqci_ioctl+1003} {__getblk+42}
Oct 9 17:48:53 brachinfo001 kernel: {:ext3:ext3_getblk+200} {:jbd:start_this_handle+964}
Oct 9 17:48:53 brachinfo001 kernel: {find_get_page+65} {wake_up_buffer+9}
Oct 9 17:48:53 brachinfo001 kernel: {:ext3:ext3_find_entry+1288} {__find_get_block+396}
Oct 9 17:48:53 brachinfo001 kernel: {sys_ioctl+853} {:cpqci:cpqci_ioctl32_handler+631}
Oct 9 17:48:53 brachinfo001 kernel: {compat_sys_ioctl+235} {cstar_do_call+27}
Oct 9 17:48:53 brachinfo001 kernel:
Oct 9 17:48:53 brachinfo001 kernel: Mem-info:
Oct 9 17:48:53 brachinfo001 kernel: Node 1 DMA per-cpu: empty
Oct 9 17:48:53 brachinfo001 kernel: Node 1 Normal per-cpu:
Oct 9 17:48:53 brachinfo001 kernel: cpu 0 hot: low 32, high 96, batch 16
Oct 9 17:48:53 brachinfo001 kernel: cpu 0 cold: low 0, high 32, batch 16
Oct 9 17:48:53 brachinfo001 kernel: cpu 1 hot: low 32, high 96, batch 16
Oct 9 17:48:53 brachinfo001 kernel: cpu 1 cold: low 0, high 32, batch 16
Oct 9 17:48:53 brachinfo001 kernel: cpu 2 hot: low 32, high 96, batch 16
Oct 9 17:48:53 brachinfo001 kernel: cpu 2 cold: low 0, high 32, batch 16
Oct 9 17:48:53 brachinfo001 kernel: cpu 3 hot: low 32, high 96, batch 16
Oct 9 17:48:53 brachinfo001 kernel: cpu 3 cold: low 0, high 32, batch 16
Oct 9 17:48:53 brachinfo001 kernel: Node 1 HighMem per-cpu: empty
Oct 9 17:48:53 brachinfo001 kernel: Node 0 DMA per-cpu:
Oct 9 17:48:53 brachinfo001 kernel: cpu 0 hot: low 2, high 6, batch 1
Oct 9 17:48:53 brachinfo001 kernel: cpu 0 cold: low 0, high 2, batch 1
Oct 9 17:48:53 brachinfo001 kernel: cpu 1 hot: low 2, high 6, batch 1
Oct 9 17:48:53 brachinfo001 kernel: cpu 1 cold: low 0, high 2, batch 1
Oct 9 17:48:53 brachinfo001 kernel: cpu 2 hot: low 2, high 6, batch 1
Oct 9 17:48:53 brachinfo001 kernel: cpu 2 cold: low 0, high 2, batch 1
Oct 9 17:48:53 brachinfo001 kernel: cpu 3 hot: low 2, high 6, batch 1
Oct 9 17:48:53 brachinfo001 kernel: cpu 3 cold: low 0, high 2, batch 1
Oct 9 17:48:53 brachinfo001 kernel: Node 0 Normal per-cpu:
Oct 9 17:48:53 brachinfo001 kernel: cpu 0 hot: low 32, high 96, batch 16
Oct 9 17:48:53 brachinfo001 kernel: cpu 0 cold: low 0, high 32, batch 16
Oct 9 17:48:53 brachinfo001 kernel: cpu 1 hot: low 32, high 96, batch 16
Oct 9 17:48:53 brachinfo001 kernel: cpu 1 cold: low 0, high 32, batch 16
Oct 9 17:48:53 brachinfo001 kernel: cpu 2 hot: low 32, high 96, batch 16
Oct 9 17:48:53 brachinfo001 kernel: cpu 2 cold: low 0, high 32, batch 16
Oct 9 17:48:53 brachinfo001 kernel: cpu 3 hot: low 32, high 96, batch 16
Oct 9 17:48:53 brachinfo001 kernel: cpu 3 cold: low 0, high 32, batch 16
Oct 9 17:48:53 brachinfo001 kernel: Node 0 HighMem per-cpu: empty
Oct 9 17:48:53 brachinfo001 kernel:
Oct 9 17:48:53 brachinfo001 kernel: Free pages: 7821176kB (0kB HighMem)
Oct 9 17:48:53 brachinfo001 kernel: Active:23913 inactive:11657 dirty:95 writeback:0 unstable:0 free:1955294 slab:3902 mapped:12558 pagetables:569
Oct 9 17:48:53 brachinfo001 kernel: Node 1 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Oct 9 17:48:53 brachinfo001 kernel: protections[]: 0 0 0
Oct 9 17:48:53 brachinfo001 kernel: Node 1 Normal free:3991720kB min:1444kB low:2888kB high:4332kB active:57468kB inactive:37084kB present:4194296kB pages_scanned:0 all_unreclaimable? no
Oct 9 17:48:53 brachinfo001 kernel: protections[]: 0 0 0
Oct 9 17:48:53 brachinfo001 kernel: Node 1 HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Oct 9 17:48:53 brachinfo001 kernel: protections[]: 0 0 0
Oct 9 17:48:53 brachinfo001 kernel: Node 0 DMA free:464kB min:4kB low:8kB high:12kB active:0kB inactive:0kB present:16384kB pages_scanned:461 all_unreclaimable? no
Oct 9 17:48:53 brachinfo001 kernel: protections[]: 0 0 0
Oct 9 17:48:53 brachinfo001 kernel: Node 0 Normal free:3828992kB min:1440kB low:2880kB high:4320kB active:38184kB inactive:9544kB present:4177916kB pages_scanned:0 all_unreclaimable? no
Oct 9 17:48:53 brachinfo001 kernel: protections[]: 0 0 0
Oct 9 17:48:53 brachinfo001 kernel: Node 0 HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Oct 9 17:48:53 brachinfo001 kernel: protections[]: 0 0 0
Oct 9 17:48:53 brachinfo001 kernel: Node 1 DMA: empty
Oct 9 17:48:53 brachinfo001 kernel: Node 1 Normal: 98*4kB 68*8kB 48*16kB 100*32kB 68*64kB 37*128kB 12*256kB 3*512kB 2*1024kB 1*2048kB 969*4096kB = 3991720kB
Oct 9 17:48:53 brachinfo001 kernel: Node 1 HighMem: empty
Oct 9 17:48:53 brachinfo001 kernel: Node 0 DMA: 6*4kB 3*8kB 6*16kB 4*32kB 3*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 464kB
Oct 9 17:48:53 brachinfo001 kernel: Node 0 Normal: 166*4kB 79*8kB 19*16kB 4*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 4*2048kB 932*4096kB = 3828992kB
Oct 9 17:48:53 brachinfo001 kernel: Node 0 HighMem: empty
Oct 9 17:48:53 brachinfo001 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Oct 9 17:48:53 brachinfo001 kernel: Free swap: 8385920kB
Oct 9 17:48:53 brachinfo001 kernel: 2097149 pages of RAM
Oct 9 17:48:53 brachinfo001 kernel: 95827 reserved pages
Oct 9 17:48:53 brachinfo001 kernel: 25119 pages shared
Oct 9 17:48:53 brachinfo001 kernel: 0 pages swap cached

What can I do to correct that?

Regards,

Charles Castelain

[Lassi Pölönen]

Exactly the same issue here wih also DL385 G1 and RHEL4 Update 4 x86_64. I have tested also with 7.70 nad no change.

That seems to be releated to load problems we've been experiencing. The server halts almost completely for a few seconds about every 10 - 15 seconds.

[G. Taylor]

I've seen this as well on a BL45p G1 w/PSP 7.70 & RHEL4u4.

[Ariel_4]

I had the same situation as well. In my case with some BL45p and BL25 with RHEL4 u4, but only the ones with high amount of memory showed those kernel errors, usually greater than or equal to 8GB RAM.

The problem was the FC HBA drivers. Once updated to most recent version (in my case 8.01.06.01) problem was solved. Actually that's the main correction that driver version does.

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=451433&prodTypeId=329290&prodSeriesId=1120361&swLang=8&taskId=135&swEnvOID=2026

Good luck!

Ariel