BUG: soft lockup - CPU#3 stuck for 10s!

[kemo]

Hello

My server is hanging every while with the below error. please find server details below
--uname -mri
2.6.18-164.el5 x86_64 x86_64
--uname -a
Linux tpaenc01 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
--cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.4 (Tikanga)

this server has 1 CPU 4 core and it is used as Media Server for netbackup software, for encryption only. it is not heavily used as only 1 policy used for encryption.


=============== MESSAGES ====================
Apr 11 08:48:51 tpaenc01 kernel: BUG: soft lockup - CPU#3 stuck for 10s! [bptm:9894]
Apr 11 08:48:51 tpaenc01 kernel: CPU 3:
Apr 11 08:48:51 tpaenc01 kernel: Modules linked in: vmtape(PU) autofs4 hidp l2cap bluetooth lockd sunrpc dm_mirror dm_multipath scsi_dh video hwmon backlight sbs i2c_ec i2c_core
button battery asus_acpi acpi_memhotplug ac ipv6 xfrm_nalgo crypto_api parport_pc lp parport st sr_mod cdrom sg bnx2 hpilo serio_raw pcspkr dm_raid45 dm_message dm_region_hash
dm_log dm_mod dm_mem_cache qla2xxx scsi_transport_fc ata_piix libata shpchp cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Apr 11 08:48:51 tpaenc01 kernel: Pid: 9894, comm: bptm Tainted: P 2.6.18-164.el5 #1
Apr 11 08:48:51 tpaenc01 kernel: RIP: 0010:[] [] __smp_call_function+0x66/0x8b
Apr 11 08:48:51 tpaenc01 kernel: RSP: 0018:ffff810107177b08 EFLAGS: 00000297
Apr 11 08:48:51 tpaenc01 kernel: RAX: 0000000000000002 RBX: 0000000000000000 RCX: 0000000000000000
Apr 11 08:48:51 tpaenc01 kernel: RDX: 00000000000000ff RSI: 00000000000000ff RDI: 00000000000000c0
Apr 11 08:48:51 tpaenc01 kernel: RBP: 0000000000000000 R08: 0000000000000003 R09: 000000000000003d
Apr 11 08:48:51 tpaenc01 kernel: R10: ffff810107177a68 R11: 0000000000000046 R12: 0000000000000000
Apr 11 08:48:51 tpaenc01 kernel: R13: 0000000000000007 R14: 0000000000000292 R15: ffff81011a2c4c00
Apr 11 08:48:51 tpaenc01 kernel: FS: 00002b02b8756510(0000) GS:ffff81011ff206c0(0000) knlGS:0000000000000000
Apr 11 08:48:51 tpaenc01 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr 11 08:48:51 tpaenc01 kernel: CR2: 00002ad18dd99000 CR3: 000000010ee6d000 CR4: 00000000000006e0
Apr 11 08:48:51 tpaenc01 kernel:
Apr 11 08:48:51 tpaenc01 kernel: Call Trace:
Apr 11 08:48:51 tpaenc01 kernel: [] __smp_call_function+0x62/0x8b
Apr 11 08:48:51 tpaenc01 kernel: [] do_flush_tlb_all+0x0/0x6a
Apr 11 08:48:51 tpaenc01 kernel: [] smp_call_function+0x32/0x47
Apr 11 08:48:51 tpaenc01 kernel: [] _stext+0xc20/0x1000
Apr 11 08:48:51 tpaenc01 kernel: [] do_flush_tlb_all+0x0/0x6a
Apr 11 08:48:51 tpaenc01 kernel: [] on_each_cpu+0x10/0x22
Apr 11 08:48:51 tpaenc01 kernel: [] __remove_vm_area+0x2b/0x42
Apr 11 08:48:51 tpaenc01 kernel: [] remove_vm_area+0x18/0x25
Apr 11 08:48:51 tpaenc01 kernel: [] __vunmap+0x47/0xed
Apr 11 08:48:51 tpaenc01 kernel: [] :vmtape:lsys_free+0x3b/0x57
Apr 11 08:48:51 tpaenc01 kernel: [] :vmtape:_vm_free+0x21/0x41
Apr 11 08:48:51 tpaenc01 kernel: [] :vmtape:write_cb+0x52/0x67
Apr 11 08:48:51 tpaenc01 kernel: [] :vmtape:write_data+0x4f4/0x537
Apr 11 08:48:51 tpaenc01 kernel: [] :vmtape:_write+0xc17/0xc8a
Apr 11 08:48:51 tpaenc01 kernel: [] :vmtape:f_write+0x4d1/0x58b
Apr 11 08:48:51 tpaenc01 kernel: [] _stext+0xc20/0x1000
Apr 11 08:48:51 tpaenc01 kernel: [] :vmtape:vmt_write+0x215/0x3bc
Apr 11 08:48:51 tpaenc01 kernel: [] vfs_write+0xce/0x174
Apr 11 08:48:51 tpaenc01 kernel: [] sys_write+0x45/0x6e
Apr 11 08:48:51 tpaenc01 kernel: [] tracesys+0xd5/0xe0
Apr 11 08:48:51 tpaenc01 kernel:
Apr 11 10:37:47 tpaenc01 syslogd 1.4.1: restart.
Apr 11 10:37:47 tpaenc01 kernel: klogd 1.4.1, log source = /proc/kmsg started.
======================================================================

please advice!

thanks

[macosta · Accepted solution]

What vendor provides the vmtape kernel module? (I'm guessing Symantec, but I don't want to make any assumptions.)

Whoever that vendor is, contact them for support with at least this stack trace. They may ask for a vmcore/crash dump, but they'll tell you if they need one.

Their codepath is implicated in the trace, but even if they don't cause the issue, Redhat won't support you here because the kernel is tainted with by the vmtape module. Since it is not open source (Proprietary flag is set), you have to rely on the vendor that provided it to support your kernel.

[kemo]

thnx