Oracle RAC + reforming / changing issue

[Schelstraete Bart_1]

Hi,

We configured a MC/ServiceGuard cluster over here, using Veritas clustered filesystem and is used with an Oracle RAC instance.

This setup works fine, but we noticed the following issue:

When 1 machine is rebooting, the disks are 'reforming' or changing, and during that timeframe the other node can't access the disks.
And due to that, the Oracle can of course not write to the disks. The Oracle is not crashing due to this, but the transactions are 'queueing'.
Because we have a lot of diskgroups, this 'reforming' takes about 3 - 4 minutes.
= 3-4 minutes not possible to update/read the database.

Do somebody knows a way how I can solve this, or how to decrease that time?


tnx

[Steven E. Protter]

Shalom,

I think this is a configuration problem, but is probably normal based on how you configured the system.

1) If you have RAC configured correctly while the first system is booting the second system should go from passive to active state. It can not leading me to believe the the first system is licking the volume group (disk access).

2) There should be some evidence of this problem on the logs /var/adm/syslog/syslog.log and in the Oracle RAC logs.

3) Your configuration may be incomplete, however this is almost certainly an Oracle RAC configuration issue. The fact that its not crashing tells me that the system is behaving as the software expects it to. I suspect there is a problem in how the volume group is activated (vgchange).

Also, newer versions of RAC don't even need Serviceguard, so I'd say state your version, your OS and lets see if we have too much software installed.

SEP

[Schelstraete Bart_1]

Hi,

I want to remind you that we are using veritas, so not LVM.
And I also don't think that this is a RAC issue, because during those famous 3-4 minutes even a bdf is hanging on those discs.
So, if the OS can access those discs, it's normal that the Oracle also can't write to those discs.

HP-UX version: 11.23 (itanium)
Oracle 10.2
Serviceguard: 11.17 (+SGeRAC)
Veritas: latest version (4.1 I think)

[Peter Heinemann]

We've implemented the same in a two-node proof-of-concept; same versions of sgerac (and veritas); Oracle; HP-UX but not integrity.

We noticed one problem which might be part of what you're seeing, so just thought I'd pass it along.

We noticed a timing problem between Oracle shutdown (init.crs) and ServiceGuard shutdown (cmcluster). When running through the standard shutdown (e.g change to runlevel 2 or even a shutdown), the cluster shutdown executes before the Oracle shtudown has finished even though the Kxxx scripts are set in the appropriate order.

This causes Oracle CRS to crash the system that was being shut down, because the file systems are suddenly unavailable to Oracle.

The crash, and therefore cluster (and shared volume) reconfiguration will cause file systems to be hung, with attendant Oracle transactions queuing until a stable reconfiguration is complete.

This problem does NOT happen when we execute the shutdowns manually (i.e, init.crs stop first, then cmcluster stop).