System hang creating new logical drive

[Yanick Quirion]

Dear all,

I'm experiencing a problem when I want to create a new logical drive after adding new disk to my array.

First, I would like to tell you what is my environment. I'm running Linux RHEL 5 on a Proliant DL380-G5. To configure the RAID controller, I'm using the Array Configuration Utility version 8.0.14.0. I also installed the HP Server Management tools (hpasm) version 8.0.0-173.rhel5. To make sure you have all necessary information, please look at this report to see the exact version of agant I'm running:

HP cciss 3.6.18 HBA driver RPM v3.6.18-10
HP Array Configuration Utility v8.0-14
HP iLO Channel Interface Driver v8.0.0-144.rhel5
OpenIPMI +HP v8.0.0-113.rhel5
HP ProLiant Essentials Licensing Manager v1.0.1-1.rhel5
HP Command Line Array Configuration Utility v8.0-14
vArray Diagnostic Utility 8.0-14
hp System Health Application and Insight Management Agents Package v8.0.0-173.rhel5
hp Insight Diagnostics v8.0.0-210
HP Printer Drivers v1.6.7-4.1.el5_0.3
HP Linux Imaging and Printing Project v1.6.7-4.1.el5_0.3
hp High Performance ILO2 Mouse X Driver for Linux v1.1.1-41
hponcfg - An RILOE II/iLo online configuration utility v1.7.0-2
HP Systems Insight Manager Server software. vC.05.02.00.00-1
HP System Management Homepage v2.1.11-197
HP Version Control Agent v2.1.9-6
MPT Fusion drivers for 53C1030 and FC9XX Adapters v4.00.13.01-2
The NET-SNMP runtime libraries. v5.3.1-24.el5_2.1
A collection of SNMP protocol tools and libraries. v5.3.1-24.el5_2.1
HP Smart Array 6400 Controller v2.68
HP Smart Array P400 Controller v2.08
Integrated Lights Out Firmware v1.29
HP ProLiant System ROM v2006.12.28

This linux box is also running on 2.6.18-53.el5 kernel.

Each time I'm starting the Array Configuration Utility (ACU), I can use the free space to create a new LUN. When I click on "save" I got this into my system log /var/log/messages:

Jun 19 19:48:53 server cmaeventd[17194]: Logical drive 5 of Array Controller in slot 5, has changed from status Unconfigured to OK
Jun 19 19:48:54 server kernel: blocks= 1171858620 block_size= 512
Jun 19 19:48:55 server kernel: blocks= 1171858620 block_size= 512
Jun 19 19:48:55 server kernel: kobject_add failed for cciss!c1d4 with -EEXIST, don't try to register things with the same name in the same directory.
Jun 19 19:48:55 server kernel:
Jun 19 19:48:55 server kernel: Call Trace:
Jun 19 19:48:55 server kernel: [] kobject_add+0x16e/0x199 Jun 19 19:48:55 server kernel: [] exact_lock+0x0/0x14 Jun 19 19:48:55 server kernel: [] register_disk+0x43/0x199 Jun 19 19:48:55 server kernel: [] add_disk+0x34/0x3d Jun 19 19:48:55 server kernel: [] :cciss:rebuild_lun_table+0x48f/0x50f
Jun 19 19:48:55 server kernel: [] zone_statistics+0x3e/0x6d Jun 19 19:48:55 server kernel: [] :cciss:cciss_ioctl+0x3fa/0xc65 Jun 19 19:48:55 server kernel: [] snprintf+0x44/0x4c Jun 19 19:48:55 server kernel: [] flush_tlb_page+0xac/0xda Jun 19 19:48:55 server kernel: [] do_wp_page+0x246/0x67d Jun 19 19:48:55 server kernel: [] :cciss:do_ioctl+0x2a/0x39 Jun 19 19:48:55 server kernel: [] :cciss:cciss_compat_ioctl+0xaf/0x25f
Jun 19 19:48:55 server kernel: [] __up_read+0x19/0x7f Jun 19 19:48:55 server kernel: [] do_page_fault+0x4eb/0x81d Jun 19 19:48:55 server kernel: [] free_pages_and_swap_cache+0x73/0x8f
Jun 19 19:48:55 server kernel: [] compat_blkdev_ioctl+0x4c/0x5f Jun 19 19:48:55 server kernel: [] compat_sys_ioctl+0xc5/0x2b1 Jun 19 19:48:55 server kernel: [] sysenter_do_call+0x1b/0x67 Jun 19 19:48:55 server kernel:
Jun 19 19:48:55 server kernel: kobject_add failed for queue with -EEXIST, don't try to register things with the same name in the same directory.
Jun 19 19:48:55 server kernel:
Jun 19 19:48:55 server kernel: Call Trace:
Jun 19 19:48:55 server kernel: [] kobject_add+0x16e/0x199 Jun 19 19:48:55 server kernel: [] blk_register_queue+0x33/0x77 Jun 19 19:48:55 server kernel: [] :cciss:rebuild_lun_table+0x48f/0x50f
Jun 19 19:48:55 server kernel: [] zone_statistics+0x3e/0x6d Jun 19 19:48:55 server kernel: [] :cciss:cciss_ioctl+0x3fa/0xc65 Jun 19 19:48:55 server kernel: [] snprintf+0x44/0x4c Jun 19 19:48:55 server kernel: [] flush_tlb_page+0xac/0xda Jun 19 19:48:55 server kernel: [] do_wp_page+0x246/0x67d Jun 19 19:48:55 server kernel: [] :cciss:do_ioctl+0x2a/0x39 Jun 19 19:48:55 server kernel: [] :cciss:cciss_compat_ioctl+0xaf/0x25f


The part that's begins by "Call Trace:" will repeat indefinetly, and after couple of minute the system freeze, fdisk -l command doens't work and the ACU stops working. The only option I've got is to manually power down the server then restart it. This happend to me twice; the first time I thought a procerss was just stuck, but 3 months later (yesterday) I got the same issue.

Is there anyone who can help me on this? On my hand, I suspect a problem with the Linux kernel 2.6.18, but the latest version available from Redhat update is 2.6.18-92.1.1.el5 and I'm not sure this will fix the error.

I would like to thank all of you for taking the time to read that particular issue.

Best regards,
Yanick

[Blazhev_1]

Hi,

you firmware is very very old, particullary the firmware of the P400 is important, but it is good idea to update everything with the Firmware maintenance CD:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=1121516&prodNameId=3288134&swEnvOID=1005&swLang=8&mode=2&taskId=135&swItem=MTX-a3d64497af4d4cdeb699c2edd3

Update the controller driver, it adds support for newer kernels :

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=1121516&swItem=MTX-1d748fc62d64474bbc87235027&prodNameId=3288134&swEnvOID=4006&swLang=8&taskId=135&mode=5

or all drivers and software at once :

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=1121516&prodNameId=3288134&swEnvOID=4006&swLang=8&mode=2&taskId=135&swItem=MTX-3cd86daacb364795bb6a4eb7ca

If you don't have time for everything, the driver update is the most important, but under firmware fixes fdisk is mentioned too, so you should give it a try.

Regards

[Yanick Quirion]

Dear Blazhev,

Thank you very much for your answer. You are right, my firmware version are very old. I'm gonna update this when I get the chance. Also I've downloaded the controller driver and I will also updrage the at the same time.

For the PSP software, I already installed the version 8.00. If the drivers has not been updated at the same time, I don't understand why. The cciss driver I'm currently running seems to be 3.6.18 (HP cciss 3.6.18 HBA driver RPM 3.6.18-10). I will install the latest version which is 4.00.13.01-3.

Hope this will solve the issie. I will post you the results here.

Thanks
Yanick

[Blazhev_1]

Hi Yanick,

the latest version for the smart array conrollers is the one I posted. The 4.00.13.01-3 is for non-Smart array controllers is not the ciss one, it is for non-Smart array controllers like SC11Xe, Sc44Ge, u320 adapters and so on.

You need the 3.6.18-12 HP ProLiant Smart Array Controller (x86/AMD32) Driver for Red Hat Enterprise Linux 5 (x86).
Both drivers are different.

Regards