topic Re: MSA 1000, HA kit and SQL in MSA Storage

MSA 1000, HA kit and SQL

Debra H — Sat, 28 Jan 2006 23:02:04 GMT

We recently purchased a MSA 1000 San Kit with the HA kit. We have it configured to connect to 2 servers. One server manages the array and also manages a MySql database that takes hits all night long. The other server has several MS SQL databases. Whenever we initiate a backup through the MS SQL Enterprise Manager, we are writing to the same drive space on the array. Once the backup starts, we start to receive several errors from both the SAN array and the MS SQL server. The redundant controller beccomes active. 2 minutes later it swaps again. This continues until the MS SQL backup job is finished. What could be wrong?

Re: MSA 1000, HA kit and SQL

BR894765 — Sun, 29 Jan 2006 04:22:14 GMT

I have seen such situation with improper cabling, when fibre switches was connected to each other.

Also without special clusterware servers must write only to drive space that they "own".

Re: MSA 1000, HA kit and SQL

Debra H — Sun, 29 Jan 2006 10:11:37 GMT

I was told the cabling did not matter as long as each card was plugged into a switch and each controller was plugged into switch. Are there settings for which is primary?

Re: MSA 1000, HA kit and SQL

BR894765 — Sun, 29 Jan 2006 10:51:43 GMT

What errors you got from array? What firmware version do you use?

Re: MSA 1000, HA kit and SQL

Debra H — Sun, 29 Jan 2006 11:57:47 GMT

Firmware version on controller is 4.48.B342. We frequently receive the message below. It continuously goes from active to standby, active to standby. Eventually connections to drives fail. We can't backup our SQL db unless we reboot everything.

Description:
A 'Redundant Controller Active' trap signifies that the Storage Agent has detected that a backup array controller in a duplexed pair has switched over to the active role.

Details:
Controller Role 'Active'
Chassis Name 'SGM053801D'
Slot # 2
MEMORY 512
Model 3
Serial Number 'P56350HX3S90IQ

Re: MSA 1000, HA kit and SQL

Uwe Zessin — Sun, 29 Jan 2006 12:13:14 GMT

> I was told the cabling did not matter as long as each card was plugged into a switch and each controller was plugged into switch.

That is simply wrong. The 'ping-pong' strongly suggests a setup error.

The 'first' Fibre Channel adapter in the server as detected by the OS must connect to the first MSA controller and it must not have access to the second controller.

Re: MSA 1000, HA kit and SQL

Debra H — Sun, 29 Jan 2006 12:55:12 GMT

We do have it setup this way. Each server has 2 HBA cards. HBA card A connects to Switch A, HBA card B connects to Switch B and the Switch connects to the controller. Same thing for second server. I am thinking of removing the multipath and redundancy.

Re: MSA 1000, HA kit and SQL

BR894765 — Sun, 29 Jan 2006 13:06:49 GMT

HBA A from both servers must be connected to Switch A and this switch must be connected to one MSA controller, HBA B from both servers -- to switch B, switch B -- to another MSA controller. And do not put any link between switches.

Re: MSA 1000, HA kit and SQL

Debra H — Sun, 29 Jan 2006 13:35:54 GMT

What determines which HBA is A or B? Also, did 4.48 firmware address this issue?

Re: MSA 1000, HA kit and SQL

BR894765 — Sun, 29 Jan 2006 13:50:58 GMT

You can find cabling examples in Modular SAN Array 1000 User Guide.
http://h20000.www2.hp.com/bizsupport/TechSupport/DocumentIndex.jsp?contentType=SupportManual&locale=en_US&docIndexId=179911&taskId=101&prodTypeId=12169&prodSeriesId=377751

Re: MSA 1000, HA kit and SQL

Debra H — Sun, 29 Jan 2006 15:18:40 GMT

Ok. I will try to find it. The SAN kit came with a flyer. We installed everything at the saem time. The kit and the HA kit. Does it matter which ports on the switch the HBAs are plugged into?

Re: MSA 1000, HA kit and SQL

BR894765 — Tue, 31 Jan 2006 04:02:17 GMT

Debra, do you solved your problem?

Re: MSA 1000, HA kit and SQL

Debra H — Tue, 31 Jan 2006 08:19:31 GMT

No. My problem has not been solved. We have done the following:

1-23-06 - reseated controller cards. Moved HBAs to slots with same bus speed.
1-29-06 - 1. swapped controllers - didn't seem to make a difference.
2. hard set ports on the switches to 2 GB rather than autonegotiate.
3. Switched fiber cables connecting server A to switch A.
4. Swapped Server B fiber cables. Boot both servers to see which HBA connected first.
5. Removed any SanSurfer Express and SanSurfer Pro software.

This morning I came in and it flipped to the redundant controller last night. I am at my wits end with this.

Re: MSA 1000, HA kit and SQL

Debra H — Wed, 01 Feb 2006 08:18:06 GMT

I downgraded the MPIO driver per HP tech instructions from 1.5 to 1.1.
Now we are getting several more messages (at least 50)- see below. Both controllers continuously go from active to standby. This is getting ridiculous!!!! Would it make sense for me to undo the Multipath and all the redundancy?

Event Name: External Array Controller Status Change (16020) Event originator: synappdb Event Severity: Critical Event received: 31-Jan-2006, 21:42:34

Event description: External Array Controller Status Change. This trap signifies that the agent has detected a change in the status of a External Array Controller. The variable cpqFcaCntlrStatus indicates the current controller status. User Action: If the controller status is offline(4), access to the storage box has been lost. Check the storage box and all fibre channel connections for problems.

Chassis Name: SGM053801D
Controller Slot Number: 1
Controller Status: offline
Array Controller Model: msa1000
Controller Serial Number: P56350HX3S60H6
Accelerator Total Memory (MB): 512

Re: MSA 1000, HA kit and SQL

BR894765 — Wed, 01 Feb 2006 08:31:24 GMT

Can you draw your cabling diagram and attach it to post?

Re: MSA 1000, HA kit and SQL

Debra H — Wed, 01 Feb 2006 11:04:54 GMT

Sorry so late. We were trying to bring boths servers back up and donwgrade the MPIO drivers back to the originals off the CD - 1.5. Please excuse the cryptic diagram.

Sorry first document sent was too large.

Re: MSA 1000, HA kit and SQL

BR894765 — Sun, 05 Feb 2006 04:08:36 GMT

Debra, sorry for delay:(

Very strange, the cabling is correct.
What you mean by MPIO 1.5?
As I know, latest MSA MPIO is 1.1.
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=421495&prodTypeId=18964&prodSeriesId=421492&swLang=8&taskId=135&swEnvOID=1005

Looks like with hight load, drivers got the timeouts and MPIO tries to change path. Try to update Q200 drivers from MSA1000 SMB Support Software CD
http://h20000.www2.hp.com/bizsupport/TechSupport/DriverDownload.jsp?lang=en&cc=us&prodNameId=439551&taskId=135&prodTypeId=12169&prodSeriesId=440223&submit.y=5&submit.x=9〈=en&cc=us
May be it is possible to change timeouts for drivers but I'm not familiar with Q200...

Re: MSA 1000, HA kit and SQL

Debra H — Sun, 05 Feb 2006 07:00:21 GMT

We experimented with turning off the redundant switch which effectively disconnects the redundant HBA cards. We then connected both controllers to one switch. It has been running fine for the last 2 days. I am going to swap the switches to see if this also produces the same result. The swap the HBA cards. I am trying to figure out which piece of hardware is causing this problem. I did not remove the MPIO drivers because the cards were still attached.