topic Re: Second controller in Disk Enclosures

Second controller

Deborah Weatherspoon — Wed, 06 Feb 2008 19:20:05 GMT

Hi all,

We have two L3000 which should failover if there is a problem. However, yesterday we had a bad GBIC on one controller but both systems were down ??? They only replaced one GBIC and then everything was fine after changing the world wide name??? We have two controllers

Can anyone help me understand why didn't the systems failover? Is there anything I can do to prevent this next time ?

Thanks

Re: Second controller

TTr — Wed, 06 Feb 2008 20:29:22 GMT

I assume by failover you are using serviceguard and that the GBIC was on a storage controller and not LAN.

A bad GBIC does not warrant a service failover. In a critical environment as with your cluster, you would have set up dual path and load balancing for all your storage.

A single GBIC would NOT bring any of the servers down. Even the server with the bad GBIC should stay up and there should be NO failover. Something else is/went wrong or your server/storage setup is not right.

Re: Second controller

Torsten. — Thu, 07 Feb 2008 07:42:06 GMT

You should give more information about your hardware. Where is this GBIC? On the switch or on the storage array? What kind of array?
I guess you will find more information in your syslog and cluster log files.

Re: Second controller

Deborah Weatherspoon — Thu, 07 Feb 2008 18:49:40 GMT

We have a va7100. There is a fiber cable that is in the brocade switch and the controller on both systems. I must add that we have serviceguard and while vg00 mounted I had to manually activate the volume groups for serviceguard. This is something we never had to do.

Re: Second controller

Torsten. — Thu, 07 Feb 2008 18:54:25 GMT

I assume the bad GBIC was on the VA controller. Does the array have dual controllers?

Re: Second controller

Deborah Weatherspoon — Thu, 07 Feb 2008 19:04:35 GMT

Yes

Re: Second controller

TTr — Fri, 08 Feb 2008 02:12:27 GMT

> We have a va7100. There is a fiber cable
> that is in the brocade switch and
> the controller on both systems.

Is there one cable that from the switch to each of the L3000?

What about from the array to the switch? one or two cables?

What about the zones in the switch, how are they setup?

The VA7100 is active/passive with only one redundancy group (failover only not load balancing)

> I must add that we have serviceguard
> and while vg00 mounted I had to manually
> activate the volume groups for
> serviceguard. This is something we
> never had to do.

Are you booting each server from the SAN?

Re: Second controller

Torsten. — Fri, 08 Feb 2008 06:23:20 GMT

Please don't call the VA active/passive!

From the manual:

"There are two internal fibre channel loops, one from each controller. The Fibre
channel disks are dual ported; each fibre channel port is connected to a
different controller. The controllers are connected via an internal highperformance
bus, which allows the LUNs to be accessed through both
controllers, and for loop or disk failover communication."

(Users manual - page 40)

Re: Second controller

Torsten. — Fri, 08 Feb 2008 15:56:54 GMT

I'm still not sure about your configuration.
Can you post the results of:

# armdsp -a
# armtopology

Re: Second controller

TTr — Fri, 08 Feb 2008 16:07:23 GMT

I agrre with Torsten, I don't consider the VA7100 active/passive either :) but HP does. When I confronted them at one point they claimed one controller is active the other is passive!!!

I guess vendors manipulate the terminology to their advanage.

From that point on I started to clarify with each vendor what they mean by active/passive by verifying that "the array is active/passive on both controllers" or "each controller is active with the other one being passive".

Re: Second controller

Torsten. — Fri, 08 Feb 2008 16:17:21 GMT

You are right, terminology is a world of it's own and can create real trouble.

Anyway, the VA is a HP classic and just like all other (original) hp arrays active/active.

This means both controllers are able to access disks/LUNs at the same time.

Re: Second controller

Deborah Weatherspoon — Fri, 08 Feb 2008 19:05:01 GMT

Here is output:

Re: Second controller

Torsten. — Fri, 08 Feb 2008 19:48:09 GMT

Your array could take some maintenance.
The firmware is old (HP19 vs. HP22), the disk firmware is likely old, the battery charger fw is different (!), I guess commandview is old too.

But this is not the current problem (but the command output you posted is not like expected).

Can you post "ioscan -fn" and "vgdisplay -v" to complete the picture?

Re: Second controller

Deborah Weatherspoon — Fri, 08 Feb 2008 20:06:50 GMT

Here's the output

Thanks for your help.

Re: Second controller

Torsten. — Fri, 08 Feb 2008 20:22:29 GMT

What GBIC was bad?

Your vg03 does not have an alternate path!

You need to
vgextend /dev/vg03 /dev/dsk/c8t1d5
and sort the pathes.
(vgreduce the c10t1d5 and vgextend again to make it the alternate) - make sure to transfer this information to the other node too.

I assume switch port 18 is connected to controller 1 of the VA - this not clear to me yet.

This server can access the vg03 only via 1 path. If this fails, there is no access.

Check the other nodes "vgdisplay -v" output for "alternate links"!

Re: Second controller

Deborah Weatherspoon — Fri, 08 Feb 2008 20:39:22 GMT

Thanks Torsten.... I'll vgextend

Re: Second controller

Deborah Weatherspoon — Fri, 08 Feb 2008 20:40:33 GMT

I will have to vgextend... The members were great in answering my question....

Re: Second controller

Torsten. — Fri, 08 Feb 2008 21:47:14 GMT

You are welcome!

Just an additional note.
Check the output of
# strings /etc/lvmtab
You should see c8txdy and c10txdy for each vg (except vg00).
If not, adjust it.
I assume you are aware about the procedure how to transfer the LVM config from one cluster node to the others.