Failed to see controller 1 HP 3PAR 7400

[djryan]

I have been fighting with this for days and I really need some help. We have replaced the controller 1 for our HP 3PAR 7400 in our lab and for some reason it is never seen with shownode or within the management studio. We bought the controller from HP however our service contract has expired and I don't beleive we can get any more support. This is what it looks like when I do a checkhealth command.

Thanks,

Dave Ryan

### Begin Summary Status

Component ----------------------------Description----------------------------- Qty
Cabling   Missing I/O module                                                     3
Cabling   Cable chains are unbalanced                                            2
Cage      Cages not on current firmware                                          3
Cage      Cages missing A loop                                                   3
cert      Certificates that have expired                                         1
File      sr_mnt not mounted                                                     1
LD        LDs in write through mode                                             78
LD        LDs without backup                                                    81
LD        Failed or degraded LDs                                                 6
LD        Preserved data storage space does not equal total node's Data memory   1
Network   Too few working admin network connections                              1
Node      Number of node environmental factors out of tolerance                  3
Node      Nodes that are not online                                              1
PD        Cages with unbalanced disks                                            6
PD        Magazines that are being serviced                                      2
PD        PDs that are degraded                                                 58
PD        Disks experiencing a high level of I/O per second                     12
Port      Unavailable host ports                                                 2
Task      Failed Tasks                                                           3
vlun      Hosts not connected to a port                                          4

[Paul28]

Hello djryan , 

This is largely because the node has not joined cluster or may have a hardware failure. 

Firstly, look at the LEDs on this node ; 

BLUE : its down / failed

Amber : component failure

Green solid : awaiting to join cluster. 

Note:  Remember to use the old node IDE / disk from the failed node ( if the node failure was not due to the IDE ) 

Check if the node has not come up due to node parameter settings issue we need to get in to the WHACK once serialled in and issue 'promedit' . Here we will need to set the values.  (Use CTRL + W  to get to WHACK prompt once connected serially to this node) 

Remember 7400 and 7400C parameters are different ; 

Look at the example for a 7400 below ;

SMI Whack> prom edit

Board Spin:       03

Size * 256 bytes: 04

Board Class:      920

Board Base:       200040

Board Rev:        91

Assembly Vendor:  FXN

Assembly Year:    2012

Assembly Week:    30

Assembly Day:     03

Assembly Serial:  00103530

Alternate Class:  ffff

Alternate Base:   ffff

Alternate Dash:   ff

System Serial:    <7 digit serial>         < ----- System Serial is set with  the command 'set perm sys_serial=<7 digit serial>'

Node ID:          01                                 < --------- set correct node ID  - node 1 = 01   node 0 = 00 ; likewise. 

Midplane Type:    1b                            < ---------- This is the correct value for 7400

Node Type:        40                              < ---------- This is the correct value for 7400

Next, at WHACK prompt you also need to enter the command to set the 7 digit serial number as below ; 

set perm sys_serial=<7 digit 3PAR serial> 

Once all of these are set type 'reset' at WHACK prompt and the node should reboot and join cluster.

[Torsten.]

You should connect to the serial port of the node and watch booting.

It would be a good idea to do a node rescue:

cli% startnoderescue node 1

from the running node 0.

[djryan]

Is there a trick getting into the console, I'm trying to use putty and don't seem to be having any luck?

[djryan]

57600 / 8 / None - No Flow Control 

What is the password to the console?

[Torsten.]

No password needed.
Connect to the console, then re-seat the node.
Capture the output.

Make sure to use the correct RJ45 to serial converter delivered with your 3PAR.

As said, try to issue the startnoderescue command from the other node.

[djryan]

OK I captured the output from a console however I cannot get into wack, I'm not sure why not CTRL-W is not working. I did a node rescue and this is the final output. The green light is still solid.

Why doesn't the console accept input?

Dave

[ 36.643059] pci 0000:07:00.0: no parent: BAR 0 [mem 0xc0108000-0xc0108fff 64bit]
[ 36.643062] pci 0000:07:00.0: no parent: BAR 2 [mem 0xc0100000-0xc0103fff 64bit]
[ 36.643064] pci 0000:07:00.0: no parent: BAR 4 [io 0x3100-0x31ff]
[ 36.643069] pirq_enable_irq: pci 0000:07:00.0 irq 25
[ 36.643075] __io_apic_set_pci_routing: irq 25 cfg->vector = 0x0
[ 36.643077] __assign_irq: reassignflag 0 irq 25 old_vector 0x0 move_in_progress 0 cfg->domain ffffffff mask ffffffff
[ 36.643080] get_IO_APIC_vector: irq = 25 vector = 34
[ 36.643082] IOAPIC found irq=25 vector=0x34 mask=0xffffffff domain=0xfff oldvec=0x0
[ 36.643093] pci 0000:07:00.0: PCI->APIC IRQ transform: INT A -> IRQ 25
[ 36.643117] arch_setup_msi_irqs_n: 0000:07:00.0 pcidev ffff88027c221800 irq 25 vector = 0x34 nvec = 0 ndesc = 1 node 0 vnum -1
[ 36.643120] __assign_irq: reassignflag 0 irq 25 old_vector 0x34 move_in_progress 0 cfg->domain fff mask fff
[ 36.643123] _create_irq_nr: irq_want 25 irq 25 reassignflag 0 nr_irqs_next 118 caller's vector 0x34 new vector 0x34 node 0
[ 36.643126] __assign_irq: reassignflag 0 irq 25 old_vector 0x34 move_in_progress 0 cfg->domain fff mask fff
[ 36.643128] msi_compose_msg: irq 25 dest 0xfff domain 0xfff
[ 36.643208] __assign_irq: reassignflag 0 irq 25 old_vector 0x34 move_in_progress 0 cfg->domain fff mask fff
[ 36.900299] __assign_irq: reassignflag 0 irq 24 old_vector 0x33 move_in_progress 0 cfg->domain fff mask fff
[ 36.919666] _create_irq_nr: irq_want 24 irq 24 reassignflag 0 nr_irqs_next 118 caller's vector 0x33 new vector 0x33 node 0
[ 36.941621] __assign_irq: reassignflag 0 irq 24 old_vector 0x33 move_in_progress 0 cfg->domain fff mask fff
[ 36.960987] msi_compose_msg: irq 24 dest 0xfff domain 0xfff
[ 36.972085] arch_setup_msi_irqs_n: 0000:06:00.0 pcidev ffff88027c3d7000 irq 24 vector = 0x33 nvec = 0 ndesc = 3 node 0 vnum -1
[ 36.994738] __assign_irq: reassignflag 1 irq 118 old_vector 0x0 move_in_progress 0 cfg->domain ffffffff mask fff
[ 37.014961] IOAPIC bios reserved irq=118 vector=0x32
[ 37.024820] IOAPIC found irq=118 vector=0x32 mask=0xfff domain=0xfff oldvec=0x0
[ 37.039349] _create_irq_nr: irq_want 24 irq 118 reassignflag 1 nr_irqs_next 119 caller's vector 0x33 new vector 0x32 node 0
[ 37.061479] __assign_irq: reassignflag 0 irq 118 old_vector 0x32 move_in_progress 0 cfg->domain fff mask fff
[ 37.081036] msi_compose_msg: irq 118 dest 0xfff domain 0xfff
[ 37.092322] __assign_irq: reassignflag 0 irq 24 old_vector 0x33 move_in_progress 0 cfg->domain fff mask fff
[ 37.111705] __assign_irq: reassignflag 0 irq 118 old_vector 0x32 move_in_progress 0 cfg->domain fff mask fff
[ 37.136968] Set xcbq_shutdown to 0 for node 2
[ 37.145702] Node ID mismatch. Expected message from node 2,but message was received from node 0 on port 3.
[ 37.547263] Port 3:0:1 - llss ffff88021ace0000 port ffffc90016fb1000 HBA Type 9205-8e Memory c0240000 IRQ 24 Bus #0 FW rev 17.11.0.0 [Initiator Mode]
[ 38.514309] Port 3:1:1 - emh ffff88021a9d0000 HBA Type f100 Port ffffc900121f8000 IRQ 25 Bus #2 Port 3:1:1 - FW rev 2.02.X.15 [Initiator Mode]
[ 38.543004] pci 0000:07:00.1: no parent: BAR 0 [mem 0xc0109000-0xc0109fff 64bit]
[ 38.557706] pci 0000:07:00.1: no parent: BAR 2 [mem 0xc0104000-0xc0107fff 64bit]
[ 38.572429] pci 0000:07:00.1: no parent: BAR 4 [io 0x3000-0x30ff]
[ 38.584716] pirq_enable_irq: pci 0000:07:00.1 irq 34
[ 38.594596] __io_apic_set_pci_routing: irq 34 cfg->vector = 0x0
[ 38.606359] __assign_irq: reassignflag 0 irq 34 old_vector 0x0 move_in_progress 0 cfg->domain ffffffff mask ffffffff
[ 38.627275] get_IO_APIC_vector: irq = 34 vector = 35
[ 38.637134] IOAPIC found irq=34 vector=0x35 mask=0xffffffff domain=0xfff oldvec=0x0
[ 38.652368] pci 0000:07:00.1: PCI->APIC IRQ transform: INT B -> IRQ 34
[ 38.665384] arch_setup_msi_irqs_n: 0000:07:00.1 pcidev ffff88027c221000 irq 34 vector = 0x35 nvec = 0 ndesc = 1 node 0 vnum -1
[ 38.688043] __assign_irq: reassignflag 0 irq 34 old_vector 0x35 move_in_progress 0 cfg->domain fff mask fff
[ 38.707408] _create_irq_nr: irq_want 34 irq 34 reassignflag 0 nr_irqs_next 119 caller's vector 0x35 new vector 0x35 node 0
[ 38.729365] __assign_irq: reassignflag 0 irq 34 old_vector 0x35 move_in_progress 0 cfg->domain fff mask fff
[ 38.748733] msi_compose_msg: irq 34 dest 0xfff domain 0xfff
[ 38.759885] __assign_irq: reassignflag 0 irq 34 old_vector 0x35 move_in_progress 0 cfg->domain fff mask fff
[ 40.671360] Port 3:1:2 - emh ffff880219858000 HBA Type f100 Port ffffc90012328000 IRQ 34 Bus #2 Port 3:1:2 - FW rev 2.02.X.15 [Initiator Mode]
[ 43.739965] CM Error: Link 3 to node 2 (step 9 DOWN): Link 3 to node 2 err_sts 0x1 err_id 0x20 (itr 0)
[ 43.739968]
[ 43.761498] Set xcbq_shutdown to 1 for node 2
[ 43.815918] Set xcbq_shutdown to 0 for node 2
[ 43.824657] Node ID mismatch. Expected message from node 2,but message was received from node 0 on port 3.
[ 44.366671] Port 3:0:2 - llss port ffffc90017016000 [Initiator Mode]
[ 44.380007] SCSI install done (7896ms taken)
[ 44.388930] BIOS indicates 12 CPUs expected
[ 44.404533] Started eagle_cm_patrol_thread pid 0x1ad6
[ 44.404536] this_node_total_cm_gb 8
[ 44.404540] patrol_delay 2688mS
[ 44.404555] Starting eagle_cm_patrol_thread....
[ 44.404559] Num Srcs 14 Src Size 16384
[ 44.444362] Starting scan mir log thread 0x1b4c
[ 44.444373] Setting recovery log scan delay to 292 ms
init: Loading tpd kernel module: done.
init: Setting up EOS interrupts
init: Loading tpd kernel patch modules: 135384 142145 140546 147603 149340
init: Configuring powerfail device /dev/pfail_storage: 801
init: done.
init: Initializing eanet0: done.
init: Rotating PM logs: done.
init: Rotating PR event logs: done.
init: Starting Process Manager: pm.
init: Rotating pr_sync_ide logs: done.
init: Starting PR Sync IDE: pr_sync_ide.
init: Starting libvirt management daemon: libvirtd2019-10-23 19:21:45.771+0000: 7507: info : libvirt version: 0.9.12
init: 2019-10-23 19:21:45.771+0000: 7507: warning : virGetHostname:2108 : getaddrinfo failed for '1635026-1': Name or service not known
init: .
init: Launching /etc/rc2.d/S30skip_fabric_ioctls script to work around ct_mgmt_get_port_info issue by setting skip_fabric_ioctls to 0x1
init: rsync daemon not enabled in /etc/default/rsync, not starting... ... (warning).
init: Starting periodic command scheduler: cron.
init: Running local boot scripts (/etc/rc.local).

3PAR(R) InForm(tm) OS 3.2.1.292 1635026-1 ttyS0
1635026-1 login:

[djryan]

Output after reseat. 

[ 31.518072] CP: 4way-CP 7400
[ 31.523790] harlink_ctl_root: 0 0 255 0 255 255 255
[ 31.534747] CM DIMM0: J0300 4096MB, interleaved 00:00000000 to 02:00000000, S/N d6873f11, ASIC 0
[ 31.552216] CM DIMM1: J0301 4096MB, interleaved 00:00000000 to 02:00000000, S/N d6873f12, ASIC 0
[ 31.569686] Info cluster_mem_setup_rsvd - total cm size 0x200000000,
[ 31.569687] device mem size 0x200000000, 0x0
[ 31.594520] CM: prefill memory from 0x1000000000(0x40000 blocks) and 0x1001600000(0x7fa8000 blocks) using xcbq[0]
[ 35.233814] Clear Cluster Memory in 3642ms
[ 35.591388] Info: total eagle memory: 0x200000000 B
[ 35.591389] Info: cache memory left : 0x1c8000000 B
[ 35.610773] Info: scratch buffer : 0x1000204000, 0xc000B
[ 35.610774] Info: control structures: 0x38000000 bytes (0x380MB)
[ 35.633780] Info: eagle_init_cmp: cmp_init with
[ 35.642783] ***ASIC 0***
[ 35.648501] mirsz: 0x30000000, non_mirr_sz: 0x1c8000000
[ 35.659576] tpci_pa: 0xc08000000, ea_pa: 0x1008000000
[ 35.670305] local_pa 0xc38000000, local_eapa 0x1038000000
[ 35.681744] CMP: Number of Cluster Memory Pages 516096
[ 35.751348] CMP: allocated: 173MB bytes at 0xffffc90077402000 for cmp structures
[ 35.766060] CMP: PCI Address Mirror: 0x00000c08000000 Local: 0x00000c38000000
[ 35.780588] CMP: Eagle Address Mirror: 0x00001008000000 Local: 0x00001038000000
[ 35.795118] CMP: Eagle Address Start: 0x00001008000000 End: 0x00001200000000
[ 35.809647] CMP: Page Count Mirror: 0x0000000000c000 Local: 0x00000000072000
[ 35.838356] CMP: PFN Start Mirror: 0000000004202496 Local: 0000000004251648
[ 36.109515] ea_gbpool_init 128 pools
[ 36.118819] Set xcbq_shutdown to 1 for node 0
[ 36.127485] Set xcbq_shutdown to 1 for node 1
[ 36.136144] Set xcbq_shutdown to 1 for node 2
[ 36.144807] *** eagle_setup_mir_record: HW mirror is enabled.
[ 36.168322] *** rm_init_config: initializing remote mirror configuration
[ 36.181949] LB rmm_lb_thread create SUCCESS
[ 36.197678] CRDIT: allocate 233472 wait cache entries
[ 36.213130] Info: global_tpd_init: cmp_start_threads...
[ 36.534853] Shutdown thread: 6237
[ 36.664383] BIOS page addr 0xffff88000005d000
[ 36.680665] PCI slot 0: LSI-SAS 9205-8e 1000.0087 06.00.0 1 24 FCAL
[ 36.695890] PCI slot 1: Emulex LPe12002 10df.f100 07.00.* 2 25 FCAL
[ 36.718200] IDE0: SanDisk PU1000064GBATSSD 131462402083 CS.51.00 62 GB
[ 36.733775] BIOS reports control cache is 2040 MB
[ 36.743127] BIOS indicates 12 CPUs expected
[ 36.768068] tpd_file_read error = -9
[ 36.775179] mfg_hp_readfile: no EEPROM update
[ 36.928945] Hardware Revision: 006
[ 36.935873] Part Identifier: HP_3PAR 7400
[ 36.945912] World Wide Name: 1000000000000000
[ 36.954569] Assembly Serial Number: PDHWNB5LM7C0C8
[ 36.964433] Assembly Part Number: QR483-63001
[ 36.973952] Saleable Serial Number: 4C14406657
[ 36.983125] Saleable Product Number: QR485A
[ 36.993162] Spare Part Number: 683246-001
[ 37.005794] Initialized serial link for EOS
[ 37.014122] __assign_irq: reassignflag 0 irq 23 old_vector 0x79 move_in_progress 0 cfg->domain fff mask fff
[ 37.033498] Internal Serial: install intr fpga-serial(3) on irq 23
[ 37.045781] Internal Serial: initialize UART ports
[ 37.055304] FPGA Serial Port 0 at pci 0xffffc900121e0400
[ 37.065871] FPGA Serial Port 1 at pci 0xffffc900121e0420
[ 37.076438] FPGA Serial Port 2 at pci 0xffffc900121e0440
[ 37.087013] Enable serial interrupts
[ 37.094142] Har power to 0xff: 1
[ 37.100569] before enabling error interrupt
[ 37.108962] After enabling error interrupt
[ 37.120165] init_tpd_main: starting SCSI initialization
[ 37.132972] QLIS: found 0 devices
[ 37.149476] File /opt/tpd/fw/hba/emfc/emfw-f100: Finished processing
[ 37.153160] pci 0000:06:00.0: no parent: BAR 0 [io 0x4000-0x40ff]
[ 37.153162] pci 0000:06:00.0: no parent: BAR 1 [mem 0xc0240000-0xc024ffff 64bit]
[ 37.153164] pci 0000:06:00.0: no parent: BAR 3 [mem 0xc0200000-0xc023ffff 64bit]
[ 37.153170] pirq_enable_irq: pci 0000:06:00.0 irq 24
[ 37.153176] __io_apic_set_pci_routing: irq 24 cfg->vector = 0x0
[ 37.153178] __assign_irq: reassignflag 0 irq 24 old_vector 0x0 move_in_progress 0 cfg->domain ffffffff mask ffffffff
[ 37.153180] get_IO_APIC_vector: irq = 24 vector = 33
[ 37.153182] IOAPIC found irq=24 vector=0x33 mask=0xffffffff domain=0xfff oldvec=0x0
[ 37.153194] pci 0000:06:00.0: PCI->APIC IRQ transform: INT A -> IRQ 24
[ 37.284326] arch_setup_msi_irqs_n: 0000:06:00.0 pcidev ffff88027c333000 irq 24 vector = 0x33 nvec = 0 ndesc = 3 node 0 vnum -1
[ 37.285655] pci 0000:07:00.0: no parent: BAR 0 [mem 0xc0108000-0xc0108fff 64bit]
[ 37.285657] pci 0000:07:00.0: no parent: BAR 2 [mem 0xc0100000-0xc0103fff 64bit]
[ 37.285659] pci 0000:07:00.0: no parent: BAR 4 [io 0x3100-0x31ff]
[ 37.285664] pirq_enable_irq: pci 0000:07:00.0 irq 25
[ 37.285670] __io_apic_set_pci_routing: irq 25 cfg->vector = 0x0
[ 37.285673] __assign_irq: reassignflag 0 irq 25 old_vector 0x0 move_in_progress 0 cfg->domain ffffffff mask ffffffff
[ 37.285675] get_IO_APIC_vector: irq = 25 vector = 34
[ 37.285677] IOAPIC found irq=25 vector=0x34 mask=0xffffffff domain=0xfff oldvec=0x0
[ 37.285689] pci 0000:07:00.0: PCI->APIC IRQ transform: INT A -> IRQ 25
[ 37.285713] arch_setup_msi_irqs_n: 0000:07:00.0 pcidev ffff88027c205800 irq 25 vector = 0x34 nvec = 0 ndesc = 1 node 0 vnum -1
[ 37.285716] __assign_irq: reassignflag 0 irq 25 old_vector 0x34 move_in_progress 0 cfg->domain fff mask fff
[ 37.285719] _create_irq_nr: irq_want 25 irq 25 reassignflag 0 nr_irqs_next 118 caller's vector 0x34 new vector 0x34 node 0
[ 37.285722] __assign_irq: reassignflag 0 irq 25 old_vector 0x34 move_in_progress 0 cfg->domain fff mask fff
[ 37.285725] msi_compose_msg: irq 25 dest 0xfff domain 0xfff
[ 37.285806] __assign_irq: reassignflag 0 irq 25 old_vector 0x34 move_in_progress 0 cfg->domain fff mask fff
[ 37.542957] __assign_irq: reassignflag 0 irq 24 old_vector 0x33 move_in_progress 0 cfg->domain fff mask fff
[ 37.562332] Set xcbq_shutdown to 0 for node 2
[ 37.571024] _create_irq_nr: irq_want 24 irq 24 reassignflag 0 nr_irqs_next 118 caller's vector 0x33 new vector 0x33 node 0
[ 37.571064] Node ID mismatch. Expected message from node 2,but message was received from node 0 on port 3.
[ 37.612189] __assign_irq: reassignflag 0 irq 24 old_vector 0x33 move_in_progress 0 cfg->domain fff mask fff
[ 37.631558] msi_compose_msg: irq 24 dest 0xfff domain 0xfff
[ 37.642639] arch_setup_msi_irqs_n: 0000:06:00.0 pcidev ffff88027c333000 irq 24 vector = 0x33 nvec = 0 ndesc = 3 node 0 vnum -1
[ 37.665291] __assign_irq: reassignflag 1 irq 118 old_vector 0x0 move_in_progress 0 cfg->domain ffffffff mask fff
[ 37.685515] IOAPIC bios reserved irq=118 vector=0x32
[ 37.695374] IOAPIC found irq=118 vector=0x32 mask=0xfff domain=0xfff oldvec=0x0
[ 37.709925] _create_irq_nr: irq_want 24 irq 118 reassignflag 1 nr_irqs_next 119 caller's vector 0x33 new vector 0x32 node 0
[ 37.732057] __assign_irq: reassignflag 0 irq 118 old_vector 0x32 move_in_progress 0 cfg->domain fff mask fff
[ 37.751609] msi_compose_msg: irq 118 dest 0xfff domain 0xfff
[ 37.762887] __assign_irq: reassignflag 0 irq 24 old_vector 0x33 move_in_progress 0 cfg->domain fff mask fff
[ 37.782271] __assign_irq: reassignflag 0 irq 118 old_vector 0x32 move_in_progress 0 cfg->domain fff mask fff
[ 38.217914] Port 3:0:1 - llss ffff88021a8f0000 port ffffc90016fb1000 HBA Type 9205-8e Memory c0240000 IRQ 24 Bus #0 FW rev 17.11.0.0 [Initiator Mode]
[ 39.167186] Port 3:1:1 - emh ffff88021a608000 HBA Type f100 Port ffffc900121f8000 IRQ 25 Bus #2 Port 3:1:1 - FW rev 2.02.X.15 [Initiator Mode]
[ 39.195859] pci 0000:07:00.1: no parent: BAR 0 [mem 0xc0109000-0xc0109fff 64bit]
[ 39.210584] pci 0000:07:00.1: no parent: BAR 2 [mem 0xc0104000-0xc0107fff 64bit]
[ 39.225284] pci 0000:07:00.1: no parent: BAR 4 [io 0x3000-0x30ff]
[ 39.237584] pirq_enable_irq: pci 0000:07:00.1 irq 34
[ 39.247457] __io_apic_set_pci_routing: irq 34 cfg->vector = 0x0
[ 39.259223] __assign_irq: reassignflag 0 irq 34 old_vector 0x0 move_in_progress 0 cfg->domain ffffffff mask ffffffff
[ 39.280137] get_IO_APIC_vector: irq = 34 vector = 35
[ 39.289995] IOAPIC found irq=34 vector=0x35 mask=0xffffffff domain=0xfff oldvec=0x0
[ 39.305247] pci 0000:07:00.1: PCI->APIC IRQ transform: INT B -> IRQ 34
[ 39.318247] arch_setup_msi_irqs_n: 0000:07:00.1 pcidev ffff88027c205000 irq 34 vector = 0x35 nvec = 0 ndesc = 1 node 0 vnum -1
[ 39.340907] __assign_irq: reassignflag 0 irq 34 old_vector 0x35 move_in_progress 0 cfg->domain fff mask fff
[ 39.360274] _create_irq_nr: irq_want 34 irq 34 reassignflag 0 nr_irqs_next 119 caller's vector 0x35 new vector 0x35 node 0
[ 39.382231] __assign_irq: reassignflag 0 irq 34 old_vector 0x35 move_in_progress 0 cfg->domain fff mask fff
[ 39.401597] msi_compose_msg: irq 34 dest 0xfff domain 0xfff
[ 39.412769] __assign_irq: reassignflag 0 irq 34 old_vector 0x35 move_in_progress 0 cfg->domain fff mask fff
[ 41.258017] Port 3:1:2 - emh ffff880219488000 HBA Type f100 Port ffffc90012328000 IRQ 34 Bus #2 Port 3:1:2 - FW rev 2.02.X.15 [Initiator Mode]
[ 44.094758] CM Error: Link 3 to node 2 (step 9 DOWN): Link 3 to node 2 err_sts 0x1 err_id 0x20 (itr 0)
[ 44.094761]
[ 44.116290] Set xcbq_shutdown to 1 for node 2
[ 44.170717] Set xcbq_shutdown to 0 for node 2
[ 44.179457] Node ID mismatch. Expected message from node 2,but message was received from node 0 on port 3.
[ 45.372175] Port 3:0:2 - llss port ffffc90017016000 [Initiator Mode]
[ 45.385513] SCSI install done (8259ms taken)
[ 45.394434] BIOS indicates 12 CPUs expected
[ 45.412183] Started eagle_cm_patrol_thread pid 0x1abc
[ 45.412186] this_node_total_cm_gb 8
[ 45.412190] patrol_delay 2688mS
[ 45.412207] Starting eagle_cm_patrol_thread....
[ 45.412210] Num Srcs 14 Src Size 16384
[ 45.452001] Starting scan mir log thread 0x1b2c
[ 45.452004] Setting recovery log scan delay to 292 ms
init: Loading tpd kernel module: done.
init: Setting up EOS interrupts
init: Loading tpd kernel patch modules: 135384 142145 140546 147603 149340
init: Configuring powerfail device /dev/pfail_storage: 801
init: done.
init: Initializing eanet0: done.
init: Rotating PM logs: done.
init: Rotating PR event logs: done.
init: Starting Process Manager: pm.
init: Rotating pr_sync_ide logs: done.
init: Starting PR Sync IDE: pr_sync_ide.
init: Starting libvirt management daemon: libvirtd2019-10-23 21:19:01.775+0000: 7479: info : libvirt version: 0.9.12
init: 2019-10-23 21:19:01.775+0000: 7479: warning : virGetHostname:2108 : getaddrinfo failed for '1635026-1': Name or service not known
init: .
init: Launching /etc/rc2.d/S30skip_fabric_ioctls script to work around ct_mgmt_get_port_info issue by setting skip_fabric_ioctls to 0x1
init: rsync daemon not enabled in /etc/default/rsync, not starting... ... (warning).
init: Starting periodic command scheduler: cron.
init: Running local boot scripts (/etc/rc.local).

3PAR(R) InForm(tm) OS 3.2.1.292 1635026-1 ttyS0
1635026-1 login:

[Torsten.]

The output looks incomplete.

A noderescue is scripting some lines to initialize the internal harddisk and set the serial number, then it is booting via tftp from the other node.

These lines are totally missing.

However, is the serial number in the login prompt correct?

[djryan]

If your referring to '2019-10-23 21:19:01.775+0000: 7479: warning : virGetHostname:2108 : getaddrinfo failed for '1635026-1': Name or service not known' yes it is right since this is supposed to be node 1 of a two node system and the serial number is correct.. Note I had to cut off some of the text because it doesn't allow more than 25000 characters which is why your missing some of the output but it was successful. 

What does this line mean?

Node ID mismatch. Expected message from node 2,but message was received from node 0 on port 3.

I can't tell what is configured in the eeprom because I can't seem to get into whack.