ioctl32 errors on RHEL 5.3

[John McNulty_2]

Does anyone know why I might be getting these errors written to dmesg and messages. Systems are DL580s with PSP 8.25 installed.

Having said that, the PSP supplied qla2xxx driver didn't install as (from the log) it doesn't support the latest Redhat kernel errata that the install picked up: 2.6.18-128.1.14.el5

Looking at the bus number of those sg devices they are on the SAN, are all LUN 0 and are "type 12", four on each bus. Would it be a good guess to say that they are EVA HSV ports?

I've grouped them together with the "attach" message for easier reading:

Jul 6 17:16:35 sdorac4b kernel: scsi 3:0:0:0: Attached scsi generic sg2 type 12
Jul 6 17:17:01 sdorac4b kernel: ioctl32(cmaeventd:9924): Unknown cmd fd(5) cmd(c054420b){00} arg(ffa87a90) on /dev/sg2
Jul 6 17:17:01 sdorac4b kernel: ioctl32(cmaeventd:9924): Unknown cmd fd(5) cmd(c058420b){00} arg(ffa87a90) on /dev/sg2
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmafcad:9943): Unknown cmd fd(5) cmd(c054420b){00} arg(ffdb0450) on /dev/sg2
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmafcad:9943): Unknown cmd fd(5) cmd(c058420b){00} arg(ffdb0450) on /dev/sg2
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmaidad:10231): Unknown cmd fd(6) cmd(c054420b){00} arg(ffc35ee0) on /dev/sg2
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmaidad:10231): Unknown cmd fd(6) cmd(c058420b){00} arg(ffc35ee0) on /dev/sg2

Jul 6 17:16:35 sdorac4b kernel: scsi 3:0:1:0: Attached scsi generic sg3 type 12
Jul 6 17:17:01 sdorac4b kernel: ioctl32(cmaeventd:9924): Unknown cmd fd(5) cmd(c054420b){00} arg(ffa87a90) on /dev/sg3
Jul 6 17:17:01 sdorac4b kernel: ioctl32(cmaeventd:9924): Unknown cmd fd(5) cmd(c058420b){00} arg(ffa87a90) on /dev/sg3
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmafcad:9943): Unknown cmd fd(5) cmd(c054420b){00} arg(ffdb0450) on /dev/sg3
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmafcad:9943): Unknown cmd fd(5) cmd(c058420b){00} arg(ffdb0450) on /dev/sg3
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmaidad:10231): Unknown cmd fd(6) cmd(c054420b){00} arg(ffc35ee0) on /dev/sg3
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmaidad:10231): Unknown cmd fd(6) cmd(c058420b){00} arg(ffc35ee0) on /dev/sg3

Jul 6 17:16:35 sdorac4b kernel: scsi 3:0:2:0: Attached scsi generic sg4 type 12
Jul 6 17:17:01 sdorac4b kernel: ioctl32(cmaeventd:9924): Unknown cmd fd(5) cmd(c054420b){00} arg(ffa87a90) on /dev/sg4
Jul 6 17:17:01 sdorac4b kernel: ioctl32(cmaeventd:9924): Unknown cmd fd(5) cmd(c058420b){00} arg(ffa87a90) on /dev/sg4
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmafcad:9943): Unknown cmd fd(5) cmd(c054420b){00} arg(ffdb0450) on /dev/sg4
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmafcad:9943): Unknown cmd fd(5) cmd(c058420b){00} arg(ffdb0450) on /dev/sg4
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmaidad:10231): Unknown cmd fd(6) cmd(c054420b){00} arg(ffc35ee0) on /dev/sg4
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmaidad:10231): Unknown cmd fd(6) cmd(c058420b){00} arg(ffc35ee0) on /dev/sg4

Jul 6 17:16:35 sdorac4b kernel: scsi 3:0:3:0: Attached scsi generic sg21 type 12
Jul 6 17:17:01 sdorac4b kernel: ioctl32(cmaeventd:9924): Unknown cmd fd(5) cmd(c054420b){00} arg(ffa87a90) on /dev/sg21
Jul 6 17:17:01 sdorac4b kernel: ioctl32(cmaeventd:9924): Unknown cmd fd(5) cmd(c058420b){00} arg(ffa87a90) on /dev/sg21
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmafcad:9943): Unknown cmd fd(5) cmd(c054420b){00} arg(ffdb0450) on /dev/sg21
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmafcad:9943): Unknown cmd fd(5) cmd(c058420b){00} arg(ffdb0450) on /dev/sg21
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmaidad:10231): Unknown cmd fd(6) cmd(c054420b){00} arg(ffc35ee0) on /dev/sg21
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmaidad:10231): Unknown cmd fd(6) cmd(c058420b){00} arg(ffc35ee0) on /dev/sg21

Jul 6 17:16:36 sdorac4b kernel: scsi 4:0:0:0: Attached scsi generic sg38 type 12
Jul 6 17:17:01 sdorac4b kernel: ioctl32(cmaeventd:9924): Unknown cmd fd(5) cmd(c054420b){00} arg(ffa87a90) on /dev/sg38
Jul 6 17:17:01 sdorac4b kernel: ioctl32(cmaeventd:9924): Unknown cmd fd(5) cmd(c058420b){00} arg(ffa87a90) on /dev/sg38
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmafcad:9943): Unknown cmd fd(5) cmd(c054420b){00} arg(ffdb0450) on /dev/sg38
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmafcad:9943): Unknown cmd fd(5) cmd(c058420b){00} arg(ffdb0450) on /dev/sg38
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmaidad:10231): Unknown cmd fd(6) cmd(c054420b){00} arg(ffc35ee0) on /dev/sg38
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmaidad:10231): Unknown cmd fd(6) cmd(c058420b){00} arg(ffc35ee0) on /dev/sg38

Jul 6 17:16:36 sdorac4b kernel: scsi 4:0:1:0: Attached scsi generic sg39 type 12
Jul 6 17:17:01 sdorac4b kernel: ioctl32(cmaeventd:9924): Unknown cmd fd(5) cmd(c054420b){00} arg(ffa87a90) on /dev/sg39
Jul 6 17:17:01 sdorac4b kernel: ioctl32(cmaeventd:9924): Unknown cmd fd(5) cmd(c058420b){00} arg(ffa87a90) on /dev/sg39
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmafcad:9943): Unknown cmd fd(5) cmd(c054420b){00} arg(ffdb0450) on /dev/sg39
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmafcad:9943): Unknown cmd fd(5) cmd(c058420b){00} arg(ffdb0450) on /dev/sg39
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmaidad:10231): Unknown cmd fd(6) cmd(c054420b){00} arg(ffc35ee0) on /dev/sg39
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmaidad:10231): Unknown cmd fd(6) cmd(c058420b){00} arg(ffc35ee0) on /dev/sg39

Jul 6 17:16:36 sdorac4b kernel: scsi 4:0:2:0: Attached scsi generic sg40 type 12
Jul 6 17:17:01 sdorac4b kernel: ioctl32(cmaeventd:9924): Unknown cmd fd(5) cmd(c054420b){00} arg(ffa87a90) on /dev/sg40
Jul 6 17:17:01 sdorac4b kernel: ioctl32(cmaeventd:9924): Unknown cmd fd(5) cmd(c058420b){00} arg(ffa87a90) on /dev/sg40
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmafcad:9943): Unknown cmd fd(5) cmd(c054420b){00} arg(ffdb0450) on /dev/sg40
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmafcad:9943): Unknown cmd fd(5) cmd(c058420b){00} arg(ffdb0450) on /dev/sg40
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmaidad:10231): Unknown cmd fd(6) cmd(c054420b){00} arg(ffc35ee0) on /dev/sg40
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmaidad:10231): Unknown cmd fd(6) cmd(c058420b){00} arg(ffc35ee0) on /dev/sg40

Jul 6 17:16:36 sdorac4b kernel: scsi 4:0:3:0: Attached scsi generic sg57 type 12
Jul 6 17:17:01 sdorac4b kernel: ioctl32(cmaeventd:9924): Unknown cmd fd(5) cmd(c054420b){00} arg(ffa87a90) on /dev/sg57
Jul 6 17:17:01 sdorac4b kernel: ioctl32(cmaeventd:9924): Unknown cmd fd(5) cmd(c058420b){00} arg(ffa87a90) on /dev/sg57
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmafcad:9943): Unknown cmd fd(5) cmd(c054420b){00} arg(ffdb0450) on /dev/sg57
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmafcad:9943): Unknown cmd fd(5) cmd(c058420b){00} arg(ffdb0450) on /dev/sg57
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmaidad:10231): Unknown cmd fd(6) cmd(c054420b){00} arg(ffc35ee0) on /dev/sg57
Jul 6 17:17:06 sdorac4b kernel: ioctl32(cmaidad:10231): Unknown cmd fd(6) cmd(c058420b){00} arg(ffc35ee0) on /dev/sg57

[Steven E. Protter]

Shalom,

Looks like a bad,missing or problematic scsi device.

Could be a tape drive, could be a fiber card. Something was there and detected and no longer is.

SEP

[Matti_Kurkela · Accepted solution]

The message comes from ioctl32, the kernel's ioctl routine for converting a 32-bit ioctl to 64-bit form. I assume you're running a 64-bit (x86_64) version of RHEL 5.3 then, right?

(Wait a minute... I hope you aren't running a 32-bit version of the PSP on a 64-bit OS? While that might be theoretically possible, it probably isn't a well-tested configuration. It might even be unsupported by HP, because a native 64-bit version exists.)

The error message is "Unknown cmd". It comes from the compat_ioctl_error() function in /fs/compat_ioctl.c.

This error means there is no translation mechanism (e.g. a compat_ioctl() function in the appropriate driver) to convert a 32-bit userspace ioctl to 64-bit form, perhaps because the current 64-bit driver does not implement this ioctl at all.

I would *not* call this an indication of hardware problems, because it seems the ioctl request is not reaching the hardware level at all!

Decoding the "cmd" field of the ioctl call is a bit tricky. If I've understood the documentation in the kernel source package correctly:

- the top bits are 11, so the ioctl is of type _IOWR, i.e. both reading and writing data.

- driver code is 0x42, or 'B'.

- function number is 0x0b, or 11 in decimal.

- the rest is the length of the data area pointed to by the arg parameter, and thus unimportant for determining what this ioctl is about.

I would have expected modern software to use SG_IO ioctls with /dev/sg* devices, but their driver code is 0x22, not 0x42. Furthermore, code 0x42 and function 0x0b is marked as unused in Documentation/ioctl/ioctl-number.txt file in Linux kernel documentation. Hmm...

This ioctl might be the way to access some extra functionality built into the PSP-supplied qla2xxx driver. Apparently this functionality does not exist in the "stock" driver.

(Ioctl numbers are a limited resource. I understand Linux kernel developers manage the adding of new ioctl functions rather strictly these days, and other, more scalable programming interfaces are preferred. This might be a part of the reason why the newer version of qla2xxx driver is not yet merged to the "vanilla" Linux kernel.)

----------

The fact that these are all LUN 0's may indeed have some special significance: they might be paths to the EVA monitoring/management device, which looks a bit like a LUN but is not a real disk. That would explain why the HP management daemons are attempting to access it.

Does the number of devices (=four on each HBA) match the number of different possible routes from each HBA to the EVA through your SAN fabric? If so, you're probably seeing a textbook case of multipathing: each possible path is detected as a separate /dev/sg* device.

MK

[John McNulty_2]

Hi MK.

Well I thought I'd grabbed the x86_64 PSP kit but now you're making me doubt myself ;-) I'll double check when I get into work later.

I thought they might be paths to the HSV management devices as all the disks being presented start from LUN 1 upwards and none of these devices are being picked up by multipathd. But there are only 2 paths per HBA to the EVA in this configuration. Having said that, there is a second EVA on the SAN which is intended to come into play at some point and the SAN guy might have zoned that in already. Will have to check that too.

I'll probably be downgrading the kernel to a version the PSP does support and reload the PSP today, as one of a number of steps I'm taking to troubleshoot a very nasty SAN problem affecting one of the fabrics connected to one of the HBAs. I don't think this will be the issue, but I need to eliminate anything that deviates from the ordinary, hence my wish to understand where these messages are coming from. If moving to the PSP supplied driver gets rid of these messages I'll report back and let you know.

Thanks, John

[John McNulty_2]

The kit is the 64bit version:

psp-8.25.rhel5.x86_64.en.tar.gz

John

[John McNulty_2]

Will have to leave checking this with the HP qla2xxx driver as an exercise for someone else, as my SAN problems turned out to be a temperamental multicore FC cable between two computer rooms. I'm up and running now and error free.

If these messages go away when the next PSP comes out then I'll report back.

John