topic Re: Linux server reboot every two month in Operating System - Linux

Linux server reboot every two month

jack Hu_1 — Tue, 24 Mar 2009 00:58:40 GMT

Dear Sir:
I have HP Proliant BL460C server with Linux Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:32:02 EDT 2006 x86_64 x86_64 x86_64 GNU/Linux
I found that it will reboot every two month.
Another server HP D380G5 with the same linux version.
They all have the same problem that server will reboot every one or two month.
Is there any suggestion or any experience about these kind of event.
thanks for your support.
Jack

Re: Linux server reboot every two month

Ivan Ferreira — Tue, 24 Mar 2009 01:58:21 GMT

More information is needed, the date and time of reboot, and any messages registered on /var/log/messages on that date.

Also, identify if it was caused by ASM for example because of an evironmental problem.

Re: Linux server reboot every two month

Marco A. — Tue, 24 Mar 2009 02:04:28 GMT

Hi,

Don't you have any kind of cron job running performing that action? or any other job that could cause this due to a overflow, crash, etc ?

Regards,

Marco,

Re: Linux server reboot every two month

Taifur — Tue, 24 Mar 2009 05:05:27 GMT

HI Jack,

This issue may be related with any hardware parts such as RAM, CPU.Pls check reseating RAM,CPU and check shutdown.log, syslog.

Rgds//
Taifur

Re: Linux server reboot every two month

jack Hu_1 — Tue, 24 Mar 2009 07:38:21 GMT

Dear Sir:
The reboot time is not regular.And there is no cronjob for re-start server.I don't see any error message on the message file at the reboot time.I just wonder that why the same linux version of the two server will have the same problem of auto-rebooting.

Re: Linux server reboot every two month

avizen9 — Tue, 24 Mar 2009 08:46:42 GMT

is there any cronjob running for it
can you post crontab -l output, thanks

Re: Linux server reboot every two month

Huc_1 — Tue, 24 Mar 2009 09:57:00 GMT

I do not have your hardware software set up so could not tell you if this is also true for other system of the same "type/setup", and you may already have coved most of what I would look for but I would double check the following thinks on each system, try and narrow down your search and only record fact,before you jump to conclusions

at around crash time what do files in /var/log/* log on each system, make a compare list

Is there a pattern 'only on week days' never during week end ?... only on week end... never at night etc ..

also I would not limit myself to the system, (hardware and software) but would have a look at all surrounding environment, like perhaps the are there any other electrical appliance that would or could only run every 1,2 months and cause theses problems.

The above reply is vague, but we realy need more narrowed down informations before whe can get more specific...,

Keep at it, and report here, bets are we will be able to help.

Enjoy life.

Jean-Pierre Huc

Re: Linux server reboot every two month

Ivan Krastev — Tue, 24 Mar 2009 13:21:25 GMT

Do you have any PSP (Proliant Support Pack) or part of it installed? Some of the older version have problems and server hang/reboot.

regards,
ivan

Re: Linux server reboot every two month

Tim Towers — Tue, 24 Mar 2009 16:42:45 GMT

Firstly I would look in /var/log/messages to see if you can see shutdown messages which would indicate that the reboot was a controlled event.

If everything seems normal and the restart has no warning then you could install kdump to see what state the kernel was in when it rebooted.

We have systems running RHES4 on x86_64 and they do not reboot without good cause.

Re: Linux server reboot every two month

Nuwan Alwis — Wed, 25 Mar 2009 02:20:49 GMT

Hi Jack,
I have installed same OS on same Blade server and i havent came across such a situation. The only thing which i dont do is installing the PSP unless if it is really necessary. wel this could happening due to a application malfunction, faulty hardware such as HBA or NIC.
if you could send us the /var/log/messages
we can find a clue for the course of the reboot. (note us when the reboot occur and time)
also make sure about the cooling system of both servers.

Run a smart-start offline diagnostic with about 8-10 iterations excluding disks. this will give you a confirmation about your hardware.

Good Luck..!

Re: Linux server reboot every two month

macosta — Wed, 25 Mar 2009 02:25:48 GMT

Jack,

See if you have PSP installed. If so, please visit the following for details:
https://bugzilla.redhat.com/show_bug.cgi?id=243657

Essentially, the mainline kernel had a bug in it that leaked memory on x64 systems with >4G of memory when cciss_ioctl() was called (which is called by the hpasm services.)

It is fixed in RHEL4 update 4, and in kernels derived from 2.6.10 of the vanilla kernel.org source.

This may not be your issue, but I did notice this a while back, and it did affect systems of mine.

Re: Linux server reboot every two month

jack Hu_1 — Wed, 25 Mar 2009 04:38:21 GMT

Thanks for all your quickly answer my question.I attach the message file below at the reboot date.
About the HP PSP I know is version 8.1 .
I also check that another linux maybe will have the issue of PSP , 'cause that the PSP version is older that 8.1 . And it will reboot every one or two month. And the linux version is older that 2.6.9 .
And someone say maybe it's a H/W issue. That's true on one of my server(not the two server I ask ). After change M/B and re-install OS as 2.6.9-42 , the server is ok now.(never reboot again).

So, I will go to check PSP also the bug of memory issue.(I found that one server's memory usage will grow up until reboot server).
All the suggestions I will work with vendor to double check the root cause.

If someone found some error on the message file,please let me know.
thanks for all your help.
Jack

Re: Linux server reboot every two month

macosta — Wed, 25 Mar 2009 05:11:39 GMT

Jack, the -42 kernel release is the latest as of RHEL4 up4, so if that kernel does not display the issue, my previous comments may not be relevant. Again, this is only the case if you have an affected kernel and PSP installed and activated.

If you have support with HP, I'd suggest you contact them, where they could review your PSP, kernel, hardware, etc.

Re: Linux server reboot every two month

Nuwan Alwis — Wed, 25 Mar 2009 16:45:39 GMT

Hi Jack,
Can you send us any previous message logs because this only shows startup events.

Any way did you check the HBA cards and SFP cables?