host freeze with RH AS4, Error: kernel BUG at kernel/exit.c:840!

[alang]

I had a RedHat AS4 host that is running Oracle 9.2.0.7.
It has worked very well over 6 months (almost near one year), however, at few days ago it happened that entire system freeze.

The related logs within /var/log/messages are

Jul 19 13:45:03 h02 kernel: ------------[ cut here ]------------
Jul 19 13:45:03 h02 kernel: kernel BUG at kernel/exit.c:840!

anyone can give advice to us ?? Thanks in advance.

[skt_skt]

which is the update version?Anything else in messages before that, like file system full etc..

[alang]

The more information:
$ uname -r
2.6.9-5.ELsmp

$ cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant)

I guess that my system hasn't any update.

The following are logs before system freeze.
=================================
Jul 19 12:00:01 h02 crond(pam_unix)[27744]: session opened for user root by (uid=0)
Jul 19 12:00:01 h02 crond(pam_unix)[27745]: session opened for user root by (uid=0)
Jul 19 12:00:01 h02 crond(pam_unix)[27747]: session opened for user root by (uid=0)
Jul 19 12:00:01 h02 crond(pam_unix)[27744]: session closed for user root
Jul 19 12:00:01 h02 crond(pam_unix)[27745]: session closed for user root
Jul 19 12:00:03 h02 crond(pam_unix)[27747]: session closed for user root
Jul 19 12:01:01 h02 crond(pam_unix)[28309]: session opened for user root by (uid=0)
Jul 19 12:01:01 h02 crond(pam_unix)[28309]: session closed for user root
Jul 19 12:05:01 h02 crond(pam_unix)[28685]: session opened for user root by (uid=0)
Jul 19 12:05:02 h02 crond(pam_unix)[28685]: session closed for user root
Jul 19 12:10:01 h02 crond(pam_unix)[29616]: session opened for user root by (uid=0)
Jul 19 12:10:01 h02 crond(pam_unix)[29615]: session opened for user root by (uid=0)
Jul 19 12:10:01 h02 crond(pam_unix)[29617]: session opened for user oracle by (uid=0)
Jul 19 12:10:01 h02 crond(pam_unix)[29616]: session closed for user root
Jul 19 12:10:01 h02 crond(pam_unix)[29615]: session closed for user root
Jul 19 12:10:17 h02 crond(pam_unix)[29617]: session closed for user oracle
Jul 19 12:15:01 h02 crond(pam_unix)[30520]: session opened for user root by (uid=0)
Jul 19 12:15:01 h02 crond(pam_unix)[30521]: session opened for user oracle by (uid=0)
Jul 19 12:15:01 h02 crond(pam_unix)[30521]: session closed for user oracle
Jul 19 12:15:01 h02 crond(pam_unix)[30520]: session closed for user root
Jul 19 12:20:01 h02 crond(pam_unix)[31336]: session opened for user root by (uid=0)
Jul 19 12:20:01 h02 crond(pam_unix)[31338]: session opened for user root by (uid=0)
Jul 19 12:20:02 h02 crond(pam_unix)[31336]: session closed for user root
Jul 19 12:20:02 h02 crond(pam_unix)[31338]: session closed for user root
Jul 19 12:25:01 h02 crond(pam_unix)[32235]: session opened for user root by (uid=0)
Jul 19 12:25:01 h02 crond(pam_unix)[32235]: session closed for user root
Jul 19 12:30:01 h02 crond(pam_unix)[582]: session opened for user root by (uid=0)
Jul 19 12:30:01 h02 crond(pam_unix)[583]: session opened for user root by (uid=0)
Jul 19 12:30:01 h02 crond(pam_unix)[582]: session closed for user root
Jul 19 12:30:01 h02 crond(pam_unix)[583]: session closed for user root
Jul 19 12:35:01 h02 crond(pam_unix)[1510]: session opened for user root by (uid=0)
Jul 19 12:35:02 h02 crond(pam_unix)[1510]: session closed for user root
Jul 19 12:40:01 h02 crond(pam_unix)[2447]: session opened for user root by (uid=0)
Jul 19 12:40:01 h02 crond(pam_unix)[2448]: session opened for user root by (uid=0)
Jul 19 12:40:01 h02 crond(pam_unix)[2447]: session closed for user root
Jul 19 12:40:01 h02 crond(pam_unix)[2448]: session closed for user root
Jul 19 12:45:01 h02 crond(pam_unix)[3310]: session opened for user root by (uid=0)
Jul 19 12:45:01 h02 crond(pam_unix)[3310]: session closed for user root
Jul 19 12:50:01 h02 crond(pam_unix)[4135]: session opened for user root by (uid=0)
Jul 19 12:50:01 h02 crond(pam_unix)[4136]: session opened for user root by (uid=0)
Jul 19 12:50:01 h02 crond(pam_unix)[4135]: session closed for user root
Jul 19 12:50:02 h02 crond(pam_unix)[4136]: session closed for user root
Jul 19 12:55:01 h02 crond(pam_unix)[5055]: session opened for user root by (uid=0)
Jul 19 12:55:01 h02 crond(pam_unix)[5055]: session closed for user root
Jul 19 13:00:01 h02 crond(pam_unix)[5940]: session opened for user root by (uid=0)
Jul 19 13:00:01 h02 crond(pam_unix)[5941]: session opened for user root by (uid=0)
Jul 19 13:00:01 h02 crond(pam_unix)[5943]: session opened for user root by (uid=0)
Jul 19 13:00:01 h02 crond(pam_unix)[5944]: session opened for user oracle by (uid=0)
Jul 19 13:00:01 h02 crond(pam_unix)[5941]: session closed for user root
Jul 19 13:00:01 h02 crond(pam_unix)[5940]: session closed for user root
Jul 19 13:00:03 h02 crond(pam_unix)[5943]: session closed for user root
Jul 19 13:01:01 h02 crond(pam_unix)[6464]: session opened for user root by (uid=0)
Jul 19 13:01:02 h02 crond(pam_unix)[6464]: session closed for user root
Jul 19 13:05:02 h02 crond(pam_unix)[6857]: session opened for user root by (uid=0)
Jul 19 13:05:09 h02 crond(pam_unix)[6857]: session closed for user root
Jul 19 13:10:03 h02 crond(pam_unix)[7823]: session opened for user oracle by (uid=0)
Jul 19 13:10:03 h02 crond(pam_unix)[7822]: session opened for user root by (uid=0)
Jul 19 13:10:03 h02 crond(pam_unix)[7821]: session opened for user root by (uid=0)
Jul 19 13:10:04 h02 crond(pam_unix)[7821]: session closed for user root
Jul 19 13:10:08 h02 crond(pam_unix)[7822]: session closed for user root
Jul 19 13:10:27 h02 crond(pam_unix)[5944]: session closed for user oracle
Jul 19 13:10:36 h02 crond(pam_unix)[7823]: session closed for user oracle
Jul 19 13:15:02 h02 crond(pam_unix)[8677]: session opened for user root by (uid=0)
Jul 19 13:15:02 h02 crond(pam_unix)[8678]: session opened for user oracle by (uid=0)
Jul 19 13:15:03 h02 crond(pam_unix)[8678]: session closed for user oracle
Jul 19 13:15:03 h02 crond(pam_unix)[8677]: session closed for user root
Jul 19 13:20:01 h02 crond(pam_unix)[9516]: session opened for user root by (uid=0)
Jul 19 13:20:01 h02 crond(pam_unix)[9517]: session opened for user root by (uid=0)
Jul 19 13:20:02 h02 crond(pam_unix)[9516]: session closed for user root
Jul 19 13:20:03 h02 crond(pam_unix)[9517]: session closed for user root
Jul 19 13:25:01 h02 crond(pam_unix)[10415]: session opened for user root by (uid=0)
Jul 19 13:25:02 h02 crond(pam_unix)[10415]: session closed for user root
Jul 19 13:30:01 h02 crond(pam_unix)[11274]: session opened for user root by (uid=0)
Jul 19 13:30:01 h02 crond(pam_unix)[11275]: session opened for user root by (uid=0)
Jul 19 13:30:01 h02 crond(pam_unix)[11274]: session closed for user root
Jul 19 13:30:01 h02 crond(pam_unix)[11275]: session closed for user root
Jul 19 13:35:02 h02 crond(pam_unix)[12178]: session opened for user root by (uid=0)
Jul 19 13:35:03 h02 crond(pam_unix)[12178]: session closed for user root
Jul 19 13:40:01 h02 crond(pam_unix)[13100]: session opened for user root by (uid=0)
Jul 19 13:40:01 h02 crond(pam_unix)[13101]: session opened for user root by (uid=0)
Jul 19 13:40:01 h02 crond(pam_unix)[13100]: session closed for user root
Jul 19 13:40:02 h02 crond(pam_unix)[13101]: session closed for user root
Jul 19 13:45:01 h02 crond(pam_unix)[13951]: session opened for user root by (uid=0)
Jul 19 13:45:01 h02 crond(pam_unix)[13951]: session closed for user root
Jul 19 13:45:03 h02 kernel: ------------[ cut here ]------------
Jul 19 13:45:03 h02 kernel: kernel BUG at kernel/exit.c:840!

[Vitaly Karasik_1]

you should start from upgrading your RHEL to the latest available update (RHEL4 upd 4 for today, IIRC)

[Yipeng CHEN]

I experienced the same issue today on one of my test server, please share your fix.. Thank you very much for your sharing..

I had a RedHat AS4 host that is running Oracle 9.2.0.7.
It has worked very well over 6 months (almost near one year), however, at few days ago it happened that entire system freeze.

The related logs within /var/log/messages are

Jul 19 13:45:03 h02 kernel: ------------[ cut here ]------------
Jul 19 13:45:03 h02 kernel: kernel BUG at kernel/exit.c:840!

[skt_skt]

what is the update

#cat /etc/redhat-release

[Yipeng CHEN]

$ uname -r
2.6.9-5.ELsmp
[oracle@dbserv12 product]$ cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant Update 2)

Thanks, I understand that we need to update the Kernal, but want to make sure the Root cause of this Server's freezing...

[skt_skt]

i dont have the Root Cause details. I would recommend to log a case with RedHat if support is available.

[Yipeng CHEN]

Yes, Thank you. I would like to know how alang is fixing this same issue also, thus I will not spend too much time to investigate the details.

[alang]

the more detail about this BUG at
https://bugzilla.redhat.com/show_bug.cgi?id=150653

Finally, we patched the kernel level to 2.6.9-42.ELsmp (RHAS 4.4) to solve this issue.

Also, at the moment we have asked RedHat Support through hot-line '0080-186-1036' for that, we got the same answer from them.

Hope above is helpful to you.

[Yipeng CHEN]

Thank you very much for sharing this experience. I appreciatet it. Please help to confirm that I assumed that we do not need to recompile Oracle Binary after updating the Kernel?

Thank you again for sharing your experience.

[skt_skt]

no oracle binary compile is required. But when you are having the third party module like power path need to be reinstalled, lpfc driver (not built in the kernel) need to be recompiled.

Let me know if this is aplicable and you need any help

[skt_skt]

also remember that new kernel is always installed(-ivh) not upgraded(-Uvh). Also update the kernel-source/kernel-header/kernel-smp-devel which is applicable.

[Yipeng CHEN]

Thank you so much for this. Yes, we have power Path for SANs.
This is a great Forums!

[alang]

As you said you don't need to change anything regarding Oracle but the new kernel will not be available until complete the OS reboot.

BTW, we have another production server(as same as version) that also experienced the same condition at a couple of month ago. So it is strongly recommended that don't use this version(non-patched level) when implement the new system.

[Yipeng CHEN]

Thank you for this. Thus, I assumed that 2.6.9-42.ELsmp (RHAS 4.4) is the most stable version on AS4.
By the way, is your RHAS 4 running on 64 bit or 32 bit?
Ours is 32 bit for Oracle 9.2.0.6
and 64 bit for Oracle 10g R2.

[alang]

We have Oracle 9.2.0.6(32bit) running on RHAS4.4 and Oracle(same version,64bit) running on IBM-AIX.

[skt_skt]

there are 32bit and 64bit RHEL AS 4 available.. The kernel rpms are differnt for both versions. My understanding is oracle application is not related to the OS's bits

[Yipeng CHEN]

Thank you. Oracle software has 32-bit version running on 32 bit OS and 64-bit version runing on 64 bit OS. As far as I know, it is much more stable to run on 64 bit OS..