Warning in syslog. hpsa-host-scan

[tomekm]

Hi all!

Anybody can tell me what that mean? I don't have any idea.

Thanks a lot!

Nov 21 12:11:46 localhost kernel: hpsa-host-scan: page allocation failure. order:4, mode:0xd0
Nov 21 12:11:46 localhost kernel: Pid: 15355, comm: hpsa-host-scan Not tainted 2.6.32-358.23.2.el6.x86_64 #1
Nov 21 12:11:46 localhost kernel: Call Trace:
Nov 21 12:11:46 localhost kernel: [<ffffffff8112c287>] ? __alloc_pages_nodemask+0x757/0x8d0
Nov 21 12:11:46 localhost kernel: [<ffffffff81166dc2>] ? kmem_getpages+0x62/0x170
Nov 21 12:11:46 localhost kernel: [<ffffffff811679da>] ? fallback_alloc+0x1ba/0x270
Nov 21 12:11:46 localhost kernel: [<ffffffff8116742f>] ? cache_grow+0x2cf/0x320
Nov 21 12:11:46 localhost kernel: [<ffffffff81167759>] ? ____cache_alloc_node+0x99/0x160
Nov 21 12:11:46 localhost kernel: [<ffffffff81167fc7>] ? kmem_cache_alloc_trace+0x127/0x1b0
Nov 21 12:11:46 localhost kernel: [<ffffffffa012c819>] ? hpsa_scan_start+0x259/0x1670 [hpsa]
Nov 21 12:11:46 localhost kernel: [<ffffffff81081b5b>] ? try_to_del_timer_sync+0x7b/0xe0
Nov 21 12:11:46 localhost kernel: [<ffffffff81081be2>] ? del_timer_sync+0x22/0x30
Nov 21 12:11:46 localhost kernel: [<ffffffff8150efba>] ? schedule_timeout+0x19a/0x2e0
Nov 21 12:11:46 localhost kernel: [<ffffffff810811e0>] ? process_timeout+0x0/0x10
Nov 21 12:11:46 localhost kernel: [<ffffffff8150f11e>] ? schedule_timeout_uninterruptible+0x1e/0x20
Nov 21 12:11:46 localhost kernel: [<ffffffffa012dda0>] ? hpsa_kickoff_rescan+0x0/0x110 [hpsa]
Nov 21 12:11:46 localhost kernel: [<ffffffffa012de0b>] ? hpsa_kickoff_rescan+0x6b/0x110 [hpsa]
Nov 21 12:11:46 localhost kernel: [<ffffffffa012dda0>] ? hpsa_kickoff_rescan+0x0/0x110 [hpsa]
Nov 21 12:11:46 localhost kernel: [<ffffffff81096a36>] ? kthread+0x96/0xa0
Nov 21 12:11:46 localhost kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Nov 21 12:11:46 localhost kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Nov 21 12:11:46 localhost kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Nov 21 12:11:46 localhost kernel: Mem-Info:
Nov 21 12:11:46 localhost kernel: Node 0 DMA per-cpu:
Nov 21 12:11:46 localhost kernel: CPU    0: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    1: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    2: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    3: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    4: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    5: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    6: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    7: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    8: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    9: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   10: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   11: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   12: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   13: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   14: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   15: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   16: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   17: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   18: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   19: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   20: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   21: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   22: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   23: hi:    0, btch:   1 usd:   0
Nov 21 12:11:46 localhost kernel: Node 0 DMA32 per-cpu:
Nov 21 12:11:46 localhost kernel: CPU    0: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    1: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    2: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    3: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    4: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    5: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    6: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    7: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    8: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    9: hi:  186, btch:  31 usd:   1
Nov 21 12:11:46 localhost kernel: CPU   10: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   11: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   12: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   13: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   14: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   15: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   16: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   17: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   18: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   19: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   20: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   21: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   22: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   23: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: Node 0 Normal per-cpu:
Nov 21 12:11:46 localhost kernel: CPU    0: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    1: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    2: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    3: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    4: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    5: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    6: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    7: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    8: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    9: hi:  186, btch:  31 usd:   1
Nov 21 12:11:46 localhost kernel: CPU   10: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   11: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   12: hi:  186, btch:  31 usd: 186
Nov 21 12:11:46 localhost kernel: CPU   13: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   14: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   15: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   16: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   17: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   18: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   19: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   20: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   21: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   22: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   23: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: Node 1 Normal per-cpu:
Nov 21 12:11:46 localhost kernel: CPU    0: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    1: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    2: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    3: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    4: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    5: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    6: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    7: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    8: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU    9: hi:  186, btch:  31 usd: 156
Nov 21 12:11:46 localhost kernel: CPU   10: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   11: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   12: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   13: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   14: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   15: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   16: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   17: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   18: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   19: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   20: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   21: hi:  186, btch:  31 usd:   1
Nov 21 12:11:46 localhost kernel: CPU   22: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: CPU   23: hi:  186, btch:  31 usd:   0
Nov 21 12:11:46 localhost kernel: active_anon:69496 inactive_anon:97830 isolated_anon:0
Nov 21 12:11:46 localhost kernel: active_file:2818241 inactive_file:4889544 isolated_file:0
Nov 21 12:11:46 localhost kernel: unevictable:1199 dirty:447 writeback:0 unstable:0
Nov 21 12:11:46 localhost kernel: free:94102 slab_reclaimable:102603 slab_unreclaimable:35812
Nov 21 12:11:46 localhost kernel: mapped:11654 shmem:2090 pagetables:19651 bounce:0
Nov 21 12:11:46 localhost kernel: Node 0 DMA free:15732kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15332kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov 21 12:11:46 localhost kernel: lowmem_reserve[]: 0 2965 16095 16095
Nov 21 12:11:46 localhost kernel: Node 0 DMA32 free:76124kB min:8280kB low:10348kB high:12420kB active_anon:14556kB inactive_anon:42476kB active_file:906620kB inactive_file:1362336kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3037072kB mlocked:0kB dirty:48kB writeback:0kB mapped:1248kB shmem:24kB slab_reclaimable:219808kB slab_unreclaimable:14536kB kernel_stack:384kB pagetables:1428kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 21 12:11:46 localhost kernel: lowmem_reserve[]: 0 0 13130 13130
Nov 21 12:11:46 localhost kernel: Node 0 Normal free:51976kB min:36660kB low:45824kB high:54988kB active_anon:161320kB inactive_anon:193040kB active_file:4199552kB inactive_file:8563380kB unevictable:3104kB isolated(anon):0kB isolated(file):0kB present:13445120kB mlocked:3096kB dirty:1152kB writeback:0kB mapped:25808kB shmem:5440kB slab_reclaimable:176196kB slab_unreclaimable:83668kB kernel_stack:6408kB pagetables:42148kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov 21 12:11:46 localhost kernel: lowmem_reserve[]: 0 0 0 0
Nov 21 12:11:46 localhost kernel: Node 1 Normal free:233568kB min:45124kB low:56404kB high:67684kB active_anon:102108kB inactive_anon:155804kB active_file:6166792kB inactive_file:9632460kB unevictable:1692kB isolated(anon):0kB isolated(file):0kB present:16547836kB mlocked:1692kB dirty:588kB writeback:0kB mapped:19560kB shmem:2896kB slab_reclaimable:14408kB slab_unreclaimable:45044kB kernel_stack:2336kB pagetables:35028kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no
Nov 21 12:11:46 localhost kernel: lowmem_reserve[]: 0 0 0 0
Nov 21 12:11:46 localhost kernel: Node 0 DMA: 1*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15732kB
Nov 21 12:11:46 localhost kernel: Node 0 DMA32: 16579*4kB 192*8kB 56*16kB 231*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 76204kB
Nov 21 12:11:46 localhost kernel: Node 0 Normal: 12254*4kB 64*8kB 20*16kB 126*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 53880kB
Nov 21 12:11:46 localhost kernel: Node 1 Normal: 48936*4kB 3139*8kB 929*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 237768kB
Nov 21 12:11:46 localhost kernel: 7721460 total pagecache pages
Nov 21 12:11:46 localhost kernel: 10932 pages in swap cache
Nov 21 12:11:46 localhost kernel: Swap cache stats: add 53180, delete 42248, find 23821403/23823674
Nov 21 12:11:46 localhost kernel: Free swap  = 16377888kB
Nov 21 12:11:46 localhost kernel: Total swap = 16490488kB
Nov 21 12:11:46 localhost kernel: 8388606 pages RAM
Nov 21 12:11:46 localhost kernel: 178223 pages reserved
Nov 21 12:11:46 localhost kernel: 2104903 pages shared
Nov 21 12:11:46 localhost kernel: 6141865 pages non-shared
Nov 21 12:11:46 localhost kernel: hpsa 0000:02:00.0: out of memory

[Matti_Kurkela]

The hpsa driver encountered a situation where it could not allocate memory.

Even though your system seems to have quite a bit of free memory, it may be that the driver requested a contiguous chunk of memory in a particular range, and the kernel was not able to satisfy the request.

The first part of the message is a call trace, identifying exactly where the error was detected. The rest is memory management statistics, while the kernel attempts to find a way to satisfy the memory allocation request.

> Nov 21 12:11:46 localhost kernel: Node 0 DMA32: 16579*4kB 192*8kB 56*16kB 231*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 76204kB

For example, this indicates the available chunks of memory in NUMA node 0, in the DMA32 address range (= memory that can be used for DMA transfers with 32-bit PCI devices). Even though there is a total of about 76 MB free, all of it is in separate pieces of 64 kB or less. If the driver attempted to allocate a contiguous chunk of 128 kB or more in the DMA32 range, it would have failed.

Since there seems to be plenty of free memory, this would seem to be a memory fragmentation problem. If you have an application that is using a large number of small pieces of memory, stopping and restarting the application once in a while (e.g. weekly/daily) would allow the kernel a chance to coalesce the freed memory into larger chunks again. But that is just a workaround: a better solution would be to change the behaviour of the application. If the application has any memory-related settings that could be adjusted, now might be a good time to review them.

[tomekm]

Thank you for your answer.

Everything is clear but I still don't know what is it hpsa-host-scan I couldn't find any files called hpsa-host-scan or similar to this name. I suppose that hpsa-host-scan is a part of HP utillity but what hpsa-host-scan do and why I can't find it in my system.

BR
TM