serial8250: too much work for irq3

[monu_1]

Hi All,

we had problem started from "serial8250: too much work for irq3" followed by "kernel: ipmi_serial(ttyS1): Error from codec send_msg: -16" in messages log.

System has recovered after restart. Hoever, in similar like setup i noticed following things-

root@SSC-9# cat /proc/interrupts | grep serial



3: 0 20803440 IO-APIC-edge serial



4: 0 50019 IO-APIC-edge serial



root@SSC-9# cat /proc/interrupts | grep serial



3: 0 20803474 IO-APIC-edge serial



4: 0 50033 IO-APIC-edge serial



root@SSC-9# cat /proc/interrupts | grep serial



3: 0 20849554 IO-APIC-edge serial



4: 0 50047 IO-APIC-edge serial



root@SSC-9# cat /proc/interrupts | grep serial



3: 0 20850701 IO-APIC-edge serial



4: 0 50061 IO-APIC-edge serial



root@SSC-9# cat /proc/interrupts | grep serial



3: 0 20851019 IO-APIC-edge serial



4: 0 50075 IO-APIC-edge serial



root@SSC-9# cat /proc/interrupts | grep serial



3: 0 20851508 IO-APIC-edge serial



4: 0 50089 IO-APIC-edge serial



root@SSC-9# cat /proc/interrupts | grep serial



3: 0 20851874 IO-APIC-edge serial



4: 0 50103 IO-APIC-edge serial



root@SSC-9# cat /proc/interrupts | grep serial



3: 0 20852205 IO-APIC-edge serial



4: 0 50117 IO-APIC-edge serial
###############

The interrupt received for serial line 3 as above and it is increasing gradually very fast.

My questions are -

Is it expected?
Would this be any cause if reach to a certain limit?
Because of above are we seeing “serial8250: too much work for irq3”?
Who is sending lots of interrupts to IRQ3? How can we check this? At the same IRQ4 is stable, not seen much interrupts.

Your help in this regard is deeply appriciated.

BR,
MKS

[Matti_Kurkela · Accepted solution]

> Is it expected?

No. "Too much work at interrupt" indicates the service or function that uses that interrupt is overloaded and may not work well.

> Would this be any cause if reach to a certain limit?

Not really. It is just a counter. The problem is that something is signalling IRQ3 more often than the OS can service it, i.e. the problem is related to the _growth rate_ of that number.

> Who is sending lots of interrupts to IRQ3? How can we check this?

Your "cat /proc/interrupts" already indicates IRQ 3 is currently controlled by the "serial" driver.

From my general knowledge of PC hardware architecture, I know that IRQ3 is normally associated with serial port COM2, also known as ttyS1 in Linux.

> kernel: ipmi_serial(ttyS1): Error from codec send_msg: -16

This suggests your ttyS1 might actually be an IPMI serial-over-LAN device, which could be used as a remote console.

IPMI is a system management interface, implemented at the hardware/firmware level. Perhaps the IPMI controller is hung or otherwise malfunctioning?

You should check your hardware vendor's support pages, to see if there is a BIOS or other firmware update for your servers. If the firmware update information suggests the update fixes some IPMI-related bugs, it would probably be worth a try.

MK