30 I2C Read Failure EMC - Backup has been down for 3 days now.

[Mike Westmoreland]

This SAN's Array is what we use to back up our systems and it's been down for three days now. We're in deep and need to identify what the real problem is here. And what we need to do to fix it. I've tried reseating things and done some parts swapping.

MSA1000
Original msg - 03 Critical lock up detected. code=f215h.

We have a replacement MSA1000, array accelerator 256 mb cache module,the fiber channel io module and fiber cable, that was overnighted in.

Put the original array accelerator cache module into the replacement MSA1000.
Now when we boot up we get the msg 30 I2C Read Failure EMC.
What is the EMC?

I've seen a post saying the EMC may need to be replaced. http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1268465401125+28353475&threadId=1123664

I can use HyperTerminal to connect to the MSA1000, but I have a system msg that constantly scrolls down the screen. Pressing the return key gets me to a cml prompt but only for a moment.

How can I stop this scrolling display? So I can get to the cml prompt. cntrl c does not seem to do it.

I'm hoping this captured text will help someone to help us. Note the difference in firmware levels between the two MSA's.
---------------------------------------------
Captured text from HyperTerminal when the original MSA1000 is booting up.
----------------------------------------------
MSA1000 is booting up. Please wait ...\

********************************************************
MSA1000
Hewlett-Packard Corporation
Copyright 2006

Firmware version: 5.20 build 1500 (2007-01-24 16:03:28)
/Hardware version: 7

Controller 1 (right controller)
MSA1000 is booting up. Please wait ...
..Unrecoverable firmware error encountered. Error code: 0xf215
Faulting Service info:
Task: SCAN 10, status=READY, scheduled=619
priority=35, preempt=YES, time_slice=0
stack_base=0x003078C0, stack_size=32736, minimum_stack=32572

Stack trace:
PC=0x000DBDFC [SP=0x0030F750]
PC=0x000DDC84 [SP=0x0030F7B8]
PC=0x000DBA14 [SP=0x0030F818]
PC=0x000DE5C4 [SP=0x0030F850]
PC=0x00154E90 [SP=0x0030F860]
WARNING: Trace stopped at invalid PC=0x00000000
WARNING: Trace stopped at invalid SP=0x00000000

That's all we get, it goes no further.
----------------------------------------------

Captured text from HyperTerminal when the replacement MSA1000 with the original Array Accelerator Cache Module installed in it boots up.
----------------------------------------------
Copyright (C) 2002 Compaq Computer Corporation
All rights reserved

Firmware version: 2.38
Build number: 122
Board ID: e11
Voyager board
Board Revision: 7

RIS Size: 43008
Physical Requests: 1200, size 392
Logical Requests: 600, size 492
Init Stripe Locks: 128
Dynamic Pool Memory: 6301072

Searching for PCI devices...
Bus Dev Func ID Rev Description
--- --- ---- ---------- ---- -----------
0 0 0 0x00f91014 0x03 IBM Bridge (Quasar)
0 2 0 0xb0600e11 0x02 Compaq SCSI Device
0 3 0 0x00cf9005 0x01 Adaptec SCSI Device
0 3 1 0x00cf9005 0x01 Adaptec SCSI Device
0 4 0 0x00cf9005 0x01 Adaptec SCSI Device
0 4 1 0x00cf9005 0x01 Adaptec SCSI Device
0 15 0 0x1029103c 0x0b HP Fibre Channel Device

Setting all BARs (0xf0000000:0xf7ffffff)...
PCI Memory: ID[0x00cf9005].BAR[1]=0xf0000000:0xf0000fff (size=0x00001000)
PCI Memory: ID[0x00cf9005].BAR[1]=0xf0001000:0xf0001fff (size=0x00001000)
PCI Memory: ID[0x00cf9005].BAR[1]=0xf0002000:0xf0002fff (size=0x00001000)
PCI Memory: ID[0x00cf9005].BAR[1]=0xf0003000:0xf0003fff (size=0x00001000)
PCI Memory: ID[0x1029103c].BAR[3]=0xf0004000:0xf00041ff (size=0x00000200)

Initializing Quasar...
PCI Memory: ID[0x00f91014].BAR[1]=0x01000000:0x017fffff (size=0x00800000)
PASSED: Quasar Initialization
Initializing XOR Engine...
DIMM 0: Found 256 MB daughter board

Surge Memory Module Mapping:
Start address: 0xa0000000
Total size: 256 MB
FIFO address: 0xa0000400
FIFO size: 255 KB
Xfer pool address: 0xa0040000
Xfer pool size: 32512 KB
Persistent Mem addr: 0xa2000000
Persistent Mem size: 320 KB
Data cache address: 0xa2050000
Data cache size: 223 MB

Setting DIMM CAS latency 2
Beginning Surge self-test

Ignore the following message.
PCI Fatal Error! Bus=0 Dev=2 Func=0 Cmd=0x0c Reg=0x1e Err=0x8300


Surge Memory Module Mapping:
Start address: 0xa0000000
Total size: 256 MB
FIFO address: 0xa0000400
FIFO size: 255 KB
Xfer pool address: 0xa0040000
Xfer pool size: 32512 KB
Persistent Mem addr: 0xa2000000
Persistent Mem size: 320 KB
Data cache address: 0xa2050000
Data cache size: 223 MB

Parsing cache user def 0 nvram.

Checking to see if cache PIC is in locked out state.
Done.

Battery PIC information:

Time Stamp: 0x000000002aa54ad0

Cache Board 0:
PIC Revision: 39
Battery Status: 0xcc
Battery 1 Charging: 1
Battery 1 Good: 1
Battery 1 Open: 0
Battery 1 Short: 0
Battery 0 Charging: 1
Battery 0 Good: 1
Battery 0 Open: 0
Battery 0 Short: 0
Battery 0 Capacity: 100
Battery 1 Capacity: 100
Battery 0 Voltage: 6497 mV
Battery 1 Voltage: 6355 mV
PIC Control: 0x20

Cache Board 1: Not Present/Accessible.

Cache start at 0xa2054000 size = 0xdfac000 (234536960)
Running Surge cache area test.
Passed Surge cache area test.
Running Persistent Memory test.
Passed Persistent Memory test.
PASSED: Surge Initialization
Initializing SCSI Device 0...Done
Initializing SCSI Device 1...Done
Initializing SCSI Device 2...Done
Initializing SCSI Device 3...Done
Initializing Fibre Channel Device 0...Done
Tachlite Config Register 0x90000000
Tachlite Status Register 0xfe00017f
FC SelfTest.....OK
Chassis is Voyager EL.
GPIO present bits: 0x800000
Controller slot: 1

TraceBuffer address = 0x001167f8 with 4000 entries
Parsing backplane (chassis) nvram.
Node Name 500508b3009124e0
Port Name 500508b3009124e1
Parsing factory nvram.
Initializing Adapter: 0
Phase delay for bus 0 set to 35us
Initializing Adapter: 1
Phase delay for bus 1 set to 35us
Initializing Adapter: 2
Phase delay for bus 2 set to 35us
Initializing Adapter: 3
Phase delay for bus 3 set to 35us

Enabled INTS: 98000f8e
Initializing 4096 sectors for data bucket use
System Memory Pool Information...
Start Address: 0x1e7a70
Bytes in pool: 6301072 (0x602590)
Bytes available: 2115484
Min. Allocation: 128 bytes

Shared Memory Pool Information...
Start Address: 0x1000000
Bytes in pool: 8388608 (0x800000)
Bytes available: 4595568
Min. Allocation: 128 bytes

Leaving App. Init - active INTS: 800
Start parse
No secondary controller present, redundancy skipped.

Operating Mode
Mode: Active-Standby
Orig: Unknown
Role: Active

Bus 0 Reset
Bus 1 Reset
Bus 2 Reset
Bus 3 Reset
Scanning - Bus 0...initial state
Scanning - Bus 1...initial state
Scanning - Bus 2...initial state
Scanning - Bus 3...initial state
OSMDeleteTargetTSH - clearing Target TSH for bus 0 scsiId 3
OSMDeleteTargetTSH - clearing Target TSH for bus 0 scsiId 4
OSMDeleteTargetTSH - clearing Target TSH for bus 1 scsiId 3
OSMDeleteTargetTSH - clearing Target TSH for bus 1 scsiId 4
OSMDeleteTargetTSH - clearing Target TSH for bus 2 scsiId 3
OSMDeleteTargetTSH - clearing Target TSH for bus 2 scsiId 4
OSMDeleteTargetTSH - clearing Target TSH for bus 3 scsiId 3
OSMDeleteTargetTSH - clearing Target TSH for bus 3 scsiId 4
Drives Present (bus, id) are:

I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 3
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 2
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 1
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 0
I2C_Read: Device = EMU, I2C_Read_Chunk() failed.
EMU I2C rcm1!!
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 3
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 2
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 1
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 0
I2C_Read: Device = EMU, I2C_Read_Chunk() failed.
EMU I2C rcm1!!
Scanning - Bus 0...pre DV state
Scanning - Bus 1...pre DV state
Scanning - Bus 2...pre DV state
Scanning - Bus 3...pre DV state
Looking on for valid data on disk

No Valid persistent data found - using default configuration
Persistent memory buffer is enabled
No persistent Host data is available- doing first time initialization
Cache batteries found enabled at power-up.
Case 1.0: New Cache board with batteries enabled found.
Possible data loss.
Initializing 457824 sectors for data bucket use
Parsing fibre bay nvram.
fibre bay:
checksum offset = 45
no world wide name found.
no chassis name found.
ascii id = RA FBR PORT
FC 0 Memory at f0004000 1029103c
FC Init Complete
FC memory allocated: 0 local, 2980607 remote
FC IOP size: 176
FC Initialization Succeeded
NVRAM Contents:
backplane:
checksum offset = 109
iteration offset = 60
chassis name = SGA051102Y
world wide name = 500508B3009124E0
chassis serial num = SGA051102Y9L
backplane serial num = P74280AX3RG09L
factory:
board serial num = P56350DX3PS00B
Startup complete.

* MSA1000 Command Line Interface *
---------------------------------------------
Here's the scrolling display that I mentioned earlier that I can't seem to break out of.

The I2C bus can't seem to communicate with EMU.

---------------------------------------------

CLI> I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 3
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 2
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 1
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 0
I2C_Read: Device = EMU, I2C_Read_Chunk() failed.
ERROR: failed on do_i2c_emu_command().
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 3
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 2
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 1
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 0
I2C_Read: Device = EMU, I2C_Read_Chunk() failed.
ERROR: failed on do_i2c_emu_command().
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 3
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 2
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 1
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 0
I2C_Read: Device = EMU, I2C_Read_Chunk() failed.
ERROR: failed on do_i2c_emu_command().
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 3
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 2
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 1
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 0
I2C_Read: Device = EMU, I2C_Read_Chunk() failed.
ERROR: failed on do_i2c_emu_command().
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 3
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 2
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 1
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 0
I2C_Read: Device = EMU, I2C_Read_Chunk() failed.
ERROR: failed on do_i2c_emu_command().
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 3
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 2
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 1
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 0
I2C_Read: Device = EMU, I2C_Read_Chunk() failed.
ERROR: failed on do_i2c_emu_command().
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 3
I2C_Read_Chunk: Device = EMU, I2C_SetAddress() failed.
I2C_Read: Resetting I2C bus for device = EMU, retry = 2
---------------------------------------------

[Mike Westmoreland]

Did I ask the question the wrong way? I've not gotten a response.

I've since learned that the EMU is the environmental monitoring unit. And that it's attached to the scsi io module.

[Mike Westmoreland]

Poorly asked question.