topic Re: New Installation Red Hat in Operating System - Linux

New Installation Red Hat

Dave La Mar — Mon, 25 Feb 2008 17:31:24 GMT

Let me start by saying Linux is completely new to our shop and I confess to have little knowledge of administration. Yes, I realize training is a must.
What we have is a four node cluster, Oracle RAC.
What we see is periodice reboots of each node. These reboots occur at the same time on each node but on different days, at different times of the day.
The log files are not helpful in determining the issue. On Friday and Saturday night we saw reboots at 21:16 and 20:10 respectively. One would think this is a result of "something" cron'd, but I do not see any related cron entries for these times.
Our DBA group has opened an issue with Oracle, but I'm hoping someone out there has encountered this and can point me in the right direction.

Regards,

-dl

Re: New Installation Red Hat

Ivan Ferreira — Mon, 25 Feb 2008 18:06:30 GMT

>>> These reboots occur at the same time on each node but on different days, at different times of the day.

I don't understand this part correctly. But node reboots on Oracle RAC can be caused by Oracle OCR. You should find messages like "Rebooting for cluster integrity".

This may be an Oracle BUG, can be a problem with your nodes interconnects, timeout setting or access to the shared ocr/voting disk.

Re: New Installation Red Hat

Dave La Mar — Mon, 25 Feb 2008 18:13:57 GMT

Yes, Ivan, I agree it could be something to do with the voting disks. These are disks from our EVA 5000, but the difficulty is finding some information on the reason. The log files are not noting any eye catchers.
As for the reboots times - All four nodes seem to reboot at the same random time on any given day.

Thanks for any insight you may share.

-dl

Re: New Installation Red Hat

Dave La Mar — Mon, 25 Feb 2008 19:19:01 GMT

I should have noted, as well, that the hardware is DL380R05.

-dl

Re: New Installation Red Hat

skt_skt — Tue, 26 Feb 2008 02:40:46 GMT

what is the OracleRAC version?If nothing reported on the message file(ideal for an h/w related issue)then what does the alert logs for the instances reports? Do you see any strange messages on messages

Are you able to see the system statistics(like MEM/swap/CPU/load avaerag etc) around the problem window? was it stable earlier? or it is a brand new setup?

Re: New Installation Red Hat

Don Vanco - Linux Ninja — Tue, 26 Feb 2008 12:21:58 GMT

Dave - given that you're new to Linux, are you _sure_ you're looking at the right logs? Between the logs that the system has and the logs that Oracle adds I am amazed to see that you can find no compelling data.....

Also - have you looked at the logs on the EVA?

Given the apparent "time based" reboots - are there other processes that access the cluster at those times? There may be an external trigger....

Good luck -
DV

Re: New Installation Red Hat

Brian M Welch — Tue, 26 Feb 2008 17:38:35 GMT

I've been working on this issue with Dave, and I can attest to the lack of clues in our Linux log files. ( /var/log/messages, and the like.)

We've packaged all of the log files including Oracle Cluster logs, and it appears we've stumped the folks at Metalink as well.. Last I had heard, they were looking at the OCFS and RAW device files for clues.

I'd come across some articles about log file rotation and syslog.d issues in RHEL RAC clusters. Anyone have any insight on this?

Re: New Installation Red Hat

Rob Leadbeater — Tue, 26 Feb 2008 20:18:26 GMT

Hi Dave,

I would probably focus on the shared storage... If all of the nodes reboot at the same time, that suggests to me a problem on the EVA and/or the SAN switches...

That said, you haven't really given us much to go on in terms of the hardware/software configuration...

Cheers,

Rob

Re: New Installation Red Hat

Dave La Mar — Tue, 26 Feb 2008 21:13:31 GMT

Rob -
We just had HP Support look at the EVA logs, but nothing was found. Yes, there is a switch involved between the machines and the EVA switch.
Our plan was to attack the switch next.
Thanks for the input.

Regards,

-dl

Re: New Installation Red Hat

Rob Leadbeater — Tue, 26 Feb 2008 21:48:21 GMT

Hi Dave,

I was hinting that you might get more help, if you were to detail the setup more completely...

How are things connected together ?
What versions are you running ?

Cheers,

Rob

Re: New Installation Red Hat

Brian M Welch — Tue, 26 Feb 2008 22:07:28 GMT

We are running Red Hat Enterprise Linux ver. 4.5 using a 64 bit SMP kernel. [2.6.9-55.0.0.2.EL SMP] As Dave mentioned before, there's 4 ProLiant DL380R05 boxes in the cluster. The interconnect switch is a HP Procurve. We are using Qlogic HBA's which plug directly into the SAN Brocade.

Additionally, the Oracle version is 10.2.0.3.

Re: New Installation Red Hat

Robert_Boyd — Wed, 27 Feb 2008 15:09:41 GMT

Are you running the HP System Management tools on the members of the cluster? Is ASR enabled? There is a known problem that won't be fixed until SmartStart 8.0 is available. Check and see if you're getting anything in your ILO logs on any of these servers.

There is another article available that discusses this problem.

http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1201035290165+28353475&threadId=1135440

Robert

Re: New Installation Red Hat

Brian M Welch — Wed, 27 Feb 2008 19:42:33 GMT

Robert,

Thanks for your reply. We've looked in the ILO logs, and /var/log/messages, and haven't seen anything that would indicate that we are having issues with ASM.