Failed to see controller 1 HP 3PAR 7400

[djryan]

I have been fighting with this for days and I really need some help. We have replaced the controller 1 for our HP 3PAR 7400 in our lab and for some reason it is never seen with shownode or within the management studio. We bought the controller from HP however our service contract has expired and I don't beleive we can get any more support. This is what it looks like when I do a checkhealth command.

Thanks,

Dave Ryan

### Begin Summary Status

Component ----------------------------Description----------------------------- Qty
Cabling   Missing I/O module                                                     3
Cabling   Cable chains are unbalanced                                            2
Cage      Cages not on current firmware                                          3
Cage      Cages missing A loop                                                   3
cert      Certificates that have expired                                         1
File      sr_mnt not mounted                                                     1
LD        LDs in write through mode                                             78
LD        LDs without backup                                                    81
LD        Failed or degraded LDs                                                 6
LD        Preserved data storage space does not equal total node's Data memory   1
Network   Too few working admin network connections                              1
Node      Number of node environmental factors out of tolerance                  3
Node      Nodes that are not online                                              1
PD        Cages with unbalanced disks                                            6
PD        Magazines that are being serviced                                      2
PD        PDs that are degraded                                                 58
PD        Disks experiencing a high level of I/O per second                     12
Port      Unavailable host ports                                                 2
Task      Failed Tasks                                                           3
vlun      Hosts not connected to a port                                          4

[Paul28]

Hello djryan , 

This is largely because the node has not joined cluster or may have a hardware failure. 

Firstly, look at the LEDs on this node ; 

BLUE : its down / failed

Amber : component failure

Green solid : awaiting to join cluster. 

Note:  Remember to use the old node IDE / disk from the failed node ( if the node failure was not due to the IDE ) 

Check if the node has not come up due to node parameter settings issue we need to get in to the WHACK once serialled in and issue 'promedit' . Here we will need to set the values.  (Use CTRL + W  to get to WHACK prompt once connected serially to this node) 

Remember 7400 and 7400C parameters are different ; 

Look at the example for a 7400 below ;

SMI Whack> prom edit

Board Spin:       03

Size * 256 bytes: 04

Board Class:      920

Board Base:       200040

Board Rev:        91

Assembly Vendor:  FXN

Assembly Year:    2012

Assembly Week:    30

Assembly Day:     03

Assembly Serial:  00103530

Alternate Class:  ffff

Alternate Base:   ffff

Alternate Dash:   ff

System Serial:    <7 digit serial>         < ----- System Serial is set with  the command 'set perm sys_serial=<7 digit serial>'

Node ID:          01                                 < --------- set correct node ID  - node 1 = 01   node 0 = 00 ; likewise. 

Midplane Type:    1b                            < ---------- This is the correct value for 7400

Node Type:        40                              < ---------- This is the correct value for 7400

Next, at WHACK prompt you also need to enter the command to set the 7 digit serial number as below ; 

set perm sys_serial=<7 digit 3PAR serial> 

Once all of these are set type 'reset' at WHACK prompt and the node should reboot and join cluster.

[Torsten.]

You should connect to the serial port of the node and watch booting.

It would be a good idea to do a node rescue:

cli% startnoderescue node 1

from the running node 0.

[djryan]

Is there a trick getting into the console, I'm trying to use putty and don't seem to be having any luck?

[djryan]

57600 / 8 / None - No Flow Control 

What is the password to the console?

[Torsten.]

No password needed.
Connect to the console, then re-seat the node.
Capture the output.

Make sure to use the correct RJ45 to serial converter delivered with your 3PAR.

As said, try to issue the startnoderescue command from the other node.

[djryan]

OK I captured the output from a console however I cannot get into wack, I'm not sure why not CTRL-W is not working. I did a node rescue and this is the final output. The green light is still solid.

Why doesn't the console accept input?

Dave

[ 36.643059] pci 0000:07:00.0: no parent: BAR 0 [mem 0xc0108000-0xc0108fff 64bit]
[ 36.643062] pci 0000:07:00.0: no parent: BAR 2 [mem 0xc0100000-0xc0103fff 64bit]
[ 36.643064] pci 0000:07:00.0: no parent: BAR 4 [io 0x3100-0x31ff]
[ 36.643069] pirq_enable_irq: pci 0000:07:00.0 irq 25
[ 36.643075] __io_apic_set_pci_routing: irq 25 cfg->vector = 0x0
[ 36.643077] __assign_irq: reassignflag 0 irq 25 old_vector 0x0 move_in_progress 0 cfg->domain ffffffff mask ffffffff
[ 36.643080] get_IO_APIC_vector: irq = 25 vector = 34
[ 36.643082] IOAPIC found irq=25 vector=0x34 mask=0xffffffff domain=0xfff oldvec=0x0
[ 36.643093] pci 0000:07:00.0: PCI->APIC IRQ transform: INT A -> IRQ 25
[ 36.643117] arch_setup_msi_irqs_n: 0000:07:00.0 pcidev ffff88027c221800 irq 25 vector = 0x34 nvec = 0 ndesc = 1 node 0 vnum -1
[ 36.643120] __assign_irq: reassignflag 0 irq 25 old_vector 0x34 move_in_progress 0 cfg->domain fff mask fff
[ 36.643123] _create_irq_nr: irq_want 25 irq 25 reassignflag 0 nr_irqs_next 118 caller's vector 0x34 new vector 0x34 node 0
[ 36.643126] __assign_irq: reassignflag 0 irq 25 old_vector 0x34 move_in_progress 0 cfg->domain fff mask fff
[ 36.643128] msi_compose_msg: irq 25 dest 0xfff domain 0xfff
[ 36.643208] __assign_irq: reassignflag 0 irq 25 old_vector 0x34 move_in_progress 0 cfg->domain fff mask fff
[ 36.900299] __assign_irq: reassignflag 0 irq 24 old_vector 0x33 move_in_progress 0 cfg->domain fff mask fff
[ 36.919666] _create_irq_nr: irq_want 24 irq 24 reassignflag 0 nr_irqs_next 118 caller's vector 0x33 new vector 0x33 node 0
[ 36.941621] __assign_irq: reassignflag 0 irq 24 old_vector 0x33 move_in_progress 0 cfg->domain fff mask fff
[ 36.960987] msi_compose_msg: irq 24 dest 0xfff domain 0xfff
[ 36.972085] arch_setup_msi_irqs_n: 0000:06:00.0 pcidev ffff88027c3d7000 irq 24 vector = 0x33 nvec = 0 ndesc = 3 node 0 vnum -1
[ 36.994738] __assign_irq: reassignflag 1 irq 118 old_vector 0x0 move_in_progress 0 cfg->domain ffffffff mask fff
[ 37.014961] IOAPIC bios reserved irq=118 vector=0x32
[ 37.024820] IOAPIC found irq=118 vector=0x32 mask=0xfff domain=0xfff oldvec=0x0
[ 37.039349] _create_irq_nr: irq_want 24 irq 118 reassignflag 1 nr_irqs_next 119 caller's vector 0x33 new vector 0x32 node 0
[ 37.061479] __assign_irq: reassignflag 0 irq 118 old_vector 0x32 move_in_progress 0 cfg->domain fff mask fff
[ 37.081036] msi_compose_msg: irq 118 dest 0xfff domain 0xfff
[ 37.092322] __assign_irq: reassignflag 0 irq 24 old_vector 0x33 move_in_progress 0 cfg->domain fff mask fff
[ 37.111705] __assign_irq: reassignflag 0 irq 118 old_vector 0x32 move_in_progress 0 cfg->domain fff mask fff
[ 37.136968] Set xcbq_shutdown to 0 for node 2
[ 37.145702] Node ID mismatch. Expected message from node 2,but message was received from node 0 on port 3.
[ 37.547263] Port 3:0:1 - llss ffff88021ace0000 port ffffc90016fb1000 HBA Type 9205-8e Memory c0240000 IRQ 24 Bus #0 FW rev 17.11.0.0 [Initiator Mode]
[ 38.514309] Port 3:1:1 - emh ffff88021a9d0000 HBA Type f100 Port ffffc900121f8000 IRQ 25 Bus #2 Port 3:1:1 - FW rev 2.02.X.15 [Initiator Mode]
[ 38.543004] pci 0000:07:00.1: no parent: BAR 0 [mem 0xc0109000-0xc0109fff 64bit]
[ 38.557706] pci 0000:07:00.1: no parent: BAR 2 [mem 0xc0104000-0xc0107fff 64bit]
[ 38.572429] pci 0000:07:00.1: no parent: BAR 4 [io 0x3000-0x30ff]
[ 38.584716] pirq_enable_irq: pci 0000:07:00.1 irq 34
[ 38.594596] __io_apic_set_pci_routing: irq 34 cfg->vector = 0x0
[ 38.606359] __assign_irq: reassignflag 0 irq 34 old_vector 0x0 move_in_progress 0 cfg->domain ffffffff mask ffffffff
[ 38.627275] get_IO_APIC_vector: irq = 34 vector = 35
[ 38.637134] IOAPIC found irq=34 vector=0x35 mask=0xffffffff domain=0xfff oldvec=0x0
[ 38.652368] pci 0000:07:00.1: PCI->APIC IRQ transform: INT B -> IRQ 34
[ 38.665384] arch_setup_msi_irqs_n: 0000:07:00.1 pcidev ffff88027c221000 irq 34 vector = 0x35 nvec = 0 ndesc = 1 node 0 vnum -1
[ 38.688043] __assign_irq: reassignflag 0 irq 34 old_vector 0x35 move_in_progress 0 cfg->domain fff mask fff
[ 38.707408] _create_irq_nr: irq_want 34 irq 34 reassignflag 0 nr_irqs_next 119 caller's vector 0x35 new vector 0x35 node 0
[ 38.729365] __assign_irq: reassignflag 0 irq 34 old_vector 0x35 move_in_progress 0 cfg->domain fff mask fff
[ 38.748733] msi_compose_msg: irq 34 dest 0xfff domain 0xfff
[ 38.759885] __assign_irq: reassignflag 0 irq 34 old_vector 0x35 move_in_progress 0 cfg->domain fff mask fff
[ 40.671360] Port 3:1:2 - emh ffff880219858000 HBA Type f100 Port ffffc90012328000 IRQ 34 Bus #2 Port 3:1:2 - FW rev 2.02.X.15 [Initiator Mode]
[ 43.739965] CM Error: Link 3 to node 2 (step 9 DOWN): Link 3 to node 2 err_sts 0x1 err_id 0x20 (itr 0)
[ 43.739968]
[ 43.761498] Set xcbq_shutdown to 1 for node 2
[ 43.815918] Set xcbq_shutdown to 0 for node 2
[ 43.824657] Node ID mismatch. Expected message from node 2,but message was received from node 0 on port 3.
[ 44.366671] Port 3:0:2 - llss port ffffc90017016000 [Initiator Mode]
[ 44.380007] SCSI install done (7896ms taken)
[ 44.388930] BIOS indicates 12 CPUs expected
[ 44.404533] Started eagle_cm_patrol_thread pid 0x1ad6
[ 44.404536] this_node_total_cm_gb 8
[ 44.404540] patrol_delay 2688mS
[ 44.404555] Starting eagle_cm_patrol_thread....
[ 44.404559] Num Srcs 14 Src Size 16384
[ 44.444362] Starting scan mir log thread 0x1b4c
[ 44.444373] Setting recovery log scan delay to 292 ms
init: Loading tpd kernel module: done.
init: Setting up EOS interrupts
init: Loading tpd kernel patch modules: 135384 142145 140546 147603 149340
init: Configuring powerfail device /dev/pfail_storage: 801
init: done.
init: Initializing eanet0: done.
init: Rotating PM logs: done.
init: Rotating PR event logs: done.
init: Starting Process Manager: pm.
init: Rotating pr_sync_ide logs: done.
init: Starting PR Sync IDE: pr_sync_ide.
init: Starting libvirt management daemon: libvirtd2019-10-23 19:21:45.771+0000: 7507: info : libvirt version: 0.9.12
init: 2019-10-23 19:21:45.771+0000: 7507: warning : virGetHostname:2108 : getaddrinfo failed for '1635026-1': Name or service not known
init: .
init: Launching /etc/rc2.d/S30skip_fabric_ioctls script to work around ct_mgmt_get_port_info issue by setting skip_fabric_ioctls to 0x1
init: rsync daemon not enabled in /etc/default/rsync, not starting... ... (warning).
init: Starting periodic command scheduler: cron.
init: Running local boot scripts (/etc/rc.local).

3PAR(R) InForm(tm) OS 3.2.1.292 1635026-1 ttyS0
1635026-1 login:

[djryan]

Output after reseat. 

[ 31.518072] CP: 4way-CP 7400
[ 31.523790] harlink_ctl_root: 0 0 255 0 255 255 255
[ 31.534747] CM DIMM0: J0300 4096MB, interleaved 00:00000000 to 02:00000000, S/N d6873f11, ASIC 0
[ 31.552216] CM DIMM1: J0301 4096MB, interleaved 00:00000000 to 02:00000000, S/N d6873f12, ASIC 0
[ 31.569686] Info cluster_mem_setup_rsvd - total cm size 0x200000000,
[ 31.569687] device mem size 0x200000000, 0x0
[ 31.594520] CM: prefill memory from 0x1000000000(0x40000 blocks) and 0x1001600000(0x7fa8000 blocks) using xcbq[0]
[ 35.233814] Clear Cluster Memory in 3642ms
[ 35.591388] Info: total eagle memory: 0x200000000 B
[ 35.591389] Info: cache memory left : 0x1c8000000 B
[ 35.610773] Info: scratch buffer : 0x1000204000, 0xc000B
[ 35.610774] Info: control structures: 0x38000000 bytes (0x380MB)
[ 35.633780] Info: eagle_init_cmp: cmp_init with
[ 35.642783] ***ASIC 0***
[ 35.648501] mirsz: 0x30000000, non_mirr_sz: 0x1c8000000
[ 35.659576] tpci_pa: 0xc08000000, ea_pa: 0x1008000000
[ 35.670305] local_pa 0xc38000000, local_eapa 0x1038000000
[ 35.681744] CMP: Number of Cluster Memory Pages 516096
[ 35.751348] CMP: allocated: 173MB bytes at 0xffffc90077402000 for cmp structures
[ 35.766060] CMP: PCI Address Mirror: 0x00000c08000000 Local: 0x00000c38000000
[ 35.780588] CMP: Eagle Address Mirror: 0x00001008000000 Local: 0x00001038000000
[ 35.795118] CMP: Eagle Address Start: 0x00001008000000 End: 0x00001200000000
[ 35.809647] CMP: Page Count Mirror: 0x0000000000c000 Local: 0x00000000072000
[ 35.838356] CMP: PFN Start Mirror: 0000000004202496 Local: 0000000004251648
[ 36.109515] ea_gbpool_init 128 pools
[ 36.118819] Set xcbq_shutdown to 1 for node 0
[ 36.127485] Set xcbq_shutdown to 1 for node 1
[ 36.136144] Set xcbq_shutdown to 1 for node 2
[ 36.144807] *** eagle_setup_mir_record: HW mirror is enabled.
[ 36.168322] *** rm_init_config: initializing remote mirror configuration
[ 36.181949] LB rmm_lb_thread create SUCCESS
[ 36.197678] CRDIT: allocate 233472 wait cache entries
[ 36.213130] Info: global_tpd_init: cmp_start_threads...
[ 36.534853] Shutdown thread: 6237
[ 36.664383] BIOS page addr 0xffff88000005d000
[ 36.680665] PCI slot 0: LSI-SAS 9205-8e 1000.0087 06.00.0 1 24 FCAL
[ 36.695890] PCI slot 1: Emulex LPe12002 10df.f100 07.00.* 2 25 FCAL
[ 36.718200] IDE0: SanDisk PU1000064GBATSSD 131462402083 CS.51.00 62 GB
[ 36.733775] BIOS reports control cache is 2040 MB
[ 36.743127] BIOS indicates 12 CPUs expected
[ 36.768068] tpd_file_read error = -9
[ 36.775179] mfg_hp_readfile: no EEPROM update
[ 36.928945] Hardware Revision: 006
[ 36.935873] Part Identifier: HP_3PAR 7400
[ 36.945912] World Wide Name: 1000000000000000
[ 36.954569] Assembly Serial Number: PDHWNB5LM7C0C8
[ 36.964433] Assembly Part Number: QR483-63001
[ 36.973952] Saleable Serial Number: 4C14406657
[ 36.983125] Saleable Product Number: QR485A
[ 36.993162] Spare Part Number: 683246-001
[ 37.005794] Initialized serial link for EOS
[ 37.014122] __assign_irq: reassignflag 0 irq 23 old_vector 0x79 move_in_progress 0 cfg->domain fff mask fff
[ 37.033498] Internal Serial: install intr fpga-serial(3) on irq 23
[ 37.045781] Internal Serial: initialize UART ports
[ 37.055304] FPGA Serial Port 0 at pci 0xffffc900121e0400
[ 37.065871] FPGA Serial Port 1 at pci 0xffffc900121e0420
[ 37.076438] FPGA Serial Port 2 at pci 0xffffc900121e0440
[ 37.087013] Enable serial interrupts
[ 37.094142] Har power to 0xff: 1
[ 37.100569] before enabling error interrupt
[ 37.108962] After enabling error interrupt
[ 37.120165] init_tpd_main: starting SCSI initialization
[ 37.132972] QLIS: found 0 devices
[ 37.149476] File /opt/tpd/fw/hba/emfc/emfw-f100: Finished processing
[ 37.153160] pci 0000:06:00.0: no parent: BAR 0 [io 0x4000-0x40ff]
[ 37.153162] pci 0000:06:00.0: no parent: BAR 1 [mem 0xc0240000-0xc024ffff 64bit]
[ 37.153164] pci 0000:06:00.0: no parent: BAR 3 [mem 0xc0200000-0xc023ffff 64bit]
[ 37.153170] pirq_enable_irq: pci 0000:06:00.0 irq 24
[ 37.153176] __io_apic_set_pci_routing: irq 24 cfg->vector = 0x0
[ 37.153178] __assign_irq: reassignflag 0 irq 24 old_vector 0x0 move_in_progress 0 cfg->domain ffffffff mask ffffffff
[ 37.153180] get_IO_APIC_vector: irq = 24 vector = 33
[ 37.153182] IOAPIC found irq=24 vector=0x33 mask=0xffffffff domain=0xfff oldvec=0x0
[ 37.153194] pci 0000:06:00.0: PCI->APIC IRQ transform: INT A -> IRQ 24
[ 37.284326] arch_setup_msi_irqs_n: 0000:06:00.0 pcidev ffff88027c333000 irq 24 vector = 0x33 nvec = 0 ndesc = 3 node 0 vnum -1
[ 37.285655] pci 0000:07:00.0: no parent: BAR 0 [mem 0xc0108000-0xc0108fff 64bit]
[ 37.285657] pci 0000:07:00.0: no parent: BAR 2 [mem 0xc0100000-0xc0103fff 64bit]
[ 37.285659] pci 0000:07:00.0: no parent: BAR 4 [io 0x3100-0x31ff]
[ 37.285664] pirq_enable_irq: pci 0000:07:00.0 irq 25
[ 37.285670] __io_apic_set_pci_routing: irq 25 cfg->vector = 0x0
[ 37.285673] __assign_irq: reassignflag 0 irq 25 old_vector 0x0 move_in_progress 0 cfg->domain ffffffff mask ffffffff
[ 37.285675] get_IO_APIC_vector: irq = 25 vector = 34
[ 37.285677] IOAPIC found irq=25 vector=0x34 mask=0xffffffff domain=0xfff oldvec=0x0
[ 37.285689] pci 0000:07:00.0: PCI->APIC IRQ transform: INT A -> IRQ 25
[ 37.285713] arch_setup_msi_irqs_n: 0000:07:00.0 pcidev ffff88027c205800 irq 25 vector = 0x34 nvec = 0 ndesc = 1 node 0 vnum -1
[ 37.285716] __assign_irq: reassignflag 0 irq 25 old_vector 0x34 move_in_progress 0 cfg->domain fff mask fff
[ 37.285719] _create_irq_nr: irq_want 25 irq 25 reassignflag 0 nr_irqs_next 118 caller's vector 0x34 new vector 0x34 node 0
[ 37.285722] __assign_irq: reassignflag 0 irq 25 old_vector 0x34 move_in_progress 0 cfg->domain fff mask fff
[ 37.285725] msi_compose_msg: irq 25 dest 0xfff domain 0xfff
[ 37.285806] __assign_irq: reassignflag 0 irq 25 old_vector 0x34 move_in_progress 0 cfg->domain fff mask fff
[ 37.542957] __assign_irq: reassignflag 0 irq 24 old_vector 0x33 move_in_progress 0 cfg->domain fff mask fff
[ 37.562332] Set xcbq_shutdown to 0 for node 2
[ 37.571024] _create_irq_nr: irq_want 24 irq 24 reassignflag 0 nr_irqs_next 118 caller's vector 0x33 new vector 0x33 node 0
[ 37.571064] Node ID mismatch. Expected message from node 2,but message was received from node 0 on port 3.
[ 37.612189] __assign_irq: reassignflag 0 irq 24 old_vector 0x33 move_in_progress 0 cfg->domain fff mask fff
[ 37.631558] msi_compose_msg: irq 24 dest 0xfff domain 0xfff
[ 37.642639] arch_setup_msi_irqs_n: 0000:06:00.0 pcidev ffff88027c333000 irq 24 vector = 0x33 nvec = 0 ndesc = 3 node 0 vnum -1
[ 37.665291] __assign_irq: reassignflag 1 irq 118 old_vector 0x0 move_in_progress 0 cfg->domain ffffffff mask fff
[ 37.685515] IOAPIC bios reserved irq=118 vector=0x32
[ 37.695374] IOAPIC found irq=118 vector=0x32 mask=0xfff domain=0xfff oldvec=0x0
[ 37.709925] _create_irq_nr: irq_want 24 irq 118 reassignflag 1 nr_irqs_next 119 caller's vector 0x33 new vector 0x32 node 0
[ 37.732057] __assign_irq: reassignflag 0 irq 118 old_vector 0x32 move_in_progress 0 cfg->domain fff mask fff
[ 37.751609] msi_compose_msg: irq 118 dest 0xfff domain 0xfff
[ 37.762887] __assign_irq: reassignflag 0 irq 24 old_vector 0x33 move_in_progress 0 cfg->domain fff mask fff
[ 37.782271] __assign_irq: reassignflag 0 irq 118 old_vector 0x32 move_in_progress 0 cfg->domain fff mask fff
[ 38.217914] Port 3:0:1 - llss ffff88021a8f0000 port ffffc90016fb1000 HBA Type 9205-8e Memory c0240000 IRQ 24 Bus #0 FW rev 17.11.0.0 [Initiator Mode]
[ 39.167186] Port 3:1:1 - emh ffff88021a608000 HBA Type f100 Port ffffc900121f8000 IRQ 25 Bus #2 Port 3:1:1 - FW rev 2.02.X.15 [Initiator Mode]
[ 39.195859] pci 0000:07:00.1: no parent: BAR 0 [mem 0xc0109000-0xc0109fff 64bit]
[ 39.210584] pci 0000:07:00.1: no parent: BAR 2 [mem 0xc0104000-0xc0107fff 64bit]
[ 39.225284] pci 0000:07:00.1: no parent: BAR 4 [io 0x3000-0x30ff]
[ 39.237584] pirq_enable_irq: pci 0000:07:00.1 irq 34
[ 39.247457] __io_apic_set_pci_routing: irq 34 cfg->vector = 0x0
[ 39.259223] __assign_irq: reassignflag 0 irq 34 old_vector 0x0 move_in_progress 0 cfg->domain ffffffff mask ffffffff
[ 39.280137] get_IO_APIC_vector: irq = 34 vector = 35
[ 39.289995] IOAPIC found irq=34 vector=0x35 mask=0xffffffff domain=0xfff oldvec=0x0
[ 39.305247] pci 0000:07:00.1: PCI->APIC IRQ transform: INT B -> IRQ 34
[ 39.318247] arch_setup_msi_irqs_n: 0000:07:00.1 pcidev ffff88027c205000 irq 34 vector = 0x35 nvec = 0 ndesc = 1 node 0 vnum -1
[ 39.340907] __assign_irq: reassignflag 0 irq 34 old_vector 0x35 move_in_progress 0 cfg->domain fff mask fff
[ 39.360274] _create_irq_nr: irq_want 34 irq 34 reassignflag 0 nr_irqs_next 119 caller's vector 0x35 new vector 0x35 node 0
[ 39.382231] __assign_irq: reassignflag 0 irq 34 old_vector 0x35 move_in_progress 0 cfg->domain fff mask fff
[ 39.401597] msi_compose_msg: irq 34 dest 0xfff domain 0xfff
[ 39.412769] __assign_irq: reassignflag 0 irq 34 old_vector 0x35 move_in_progress 0 cfg->domain fff mask fff
[ 41.258017] Port 3:1:2 - emh ffff880219488000 HBA Type f100 Port ffffc90012328000 IRQ 34 Bus #2 Port 3:1:2 - FW rev 2.02.X.15 [Initiator Mode]
[ 44.094758] CM Error: Link 3 to node 2 (step 9 DOWN): Link 3 to node 2 err_sts 0x1 err_id 0x20 (itr 0)
[ 44.094761]
[ 44.116290] Set xcbq_shutdown to 1 for node 2
[ 44.170717] Set xcbq_shutdown to 0 for node 2
[ 44.179457] Node ID mismatch. Expected message from node 2,but message was received from node 0 on port 3.
[ 45.372175] Port 3:0:2 - llss port ffffc90017016000 [Initiator Mode]
[ 45.385513] SCSI install done (8259ms taken)
[ 45.394434] BIOS indicates 12 CPUs expected
[ 45.412183] Started eagle_cm_patrol_thread pid 0x1abc
[ 45.412186] this_node_total_cm_gb 8
[ 45.412190] patrol_delay 2688mS
[ 45.412207] Starting eagle_cm_patrol_thread....
[ 45.412210] Num Srcs 14 Src Size 16384
[ 45.452001] Starting scan mir log thread 0x1b2c
[ 45.452004] Setting recovery log scan delay to 292 ms
init: Loading tpd kernel module: done.
init: Setting up EOS interrupts
init: Loading tpd kernel patch modules: 135384 142145 140546 147603 149340
init: Configuring powerfail device /dev/pfail_storage: 801
init: done.
init: Initializing eanet0: done.
init: Rotating PM logs: done.
init: Rotating PR event logs: done.
init: Starting Process Manager: pm.
init: Rotating pr_sync_ide logs: done.
init: Starting PR Sync IDE: pr_sync_ide.
init: Starting libvirt management daemon: libvirtd2019-10-23 21:19:01.775+0000: 7479: info : libvirt version: 0.9.12
init: 2019-10-23 21:19:01.775+0000: 7479: warning : virGetHostname:2108 : getaddrinfo failed for '1635026-1': Name or service not known
init: .
init: Launching /etc/rc2.d/S30skip_fabric_ioctls script to work around ct_mgmt_get_port_info issue by setting skip_fabric_ioctls to 0x1
init: rsync daemon not enabled in /etc/default/rsync, not starting... ... (warning).
init: Starting periodic command scheduler: cron.
init: Running local boot scripts (/etc/rc.local).

3PAR(R) InForm(tm) OS 3.2.1.292 1635026-1 ttyS0
1635026-1 login:

[Torsten.]

The output looks incomplete.

A noderescue is scripting some lines to initialize the internal harddisk and set the serial number, then it is booting via tftp from the other node.

These lines are totally missing.

However, is the serial number in the login prompt correct?

[djryan]

If your referring to '2019-10-23 21:19:01.775+0000: 7479: warning : virGetHostname:2108 : getaddrinfo failed for '1635026-1': Name or service not known' yes it is right since this is supposed to be node 1 of a two node system and the serial number is correct.. Note I had to cut off some of the text because it doesn't allow more than 25000 characters which is why your missing some of the output but it was successful. 

What does this line mean?

Node ID mismatch. Expected message from node 2,but message was received from node 0 on port 3.

I can't tell what is configured in the eeprom because I can't seem to get into whack. 

[Torsten.]

Once you run the startnoderescue command, you should see at the serial line:

+-----------------------------------------------------------------------------+
HP SPI Image 3.0.10. Release version. 13:40:19 Feb 03 2014.
+-----------------------------------------------------------------------------+
| CPU 1 x 1.80 GHz Sandy Bridge hexa core HT dual
| Control Cache Size 15.100 GB (cpu mem type 11) CL9
|               Pair0 DIMM0 (J0155): 16384 MB CL6/11
| Data Cache Size 4 GB CL4/6
|               Set0 DC DIMM 0.0.0 (J0300): 2048 MB CL4/6
|               Set0 DC DIMM 0.1.0 (J0301): 2048 MB CL4/6
| Slot ID 0 [2 Node HP 3PAR 7220 Centerplane]
| FPGA Eos v0.f
| SATA0 Disk SanDisk DX110128A5xnNMRI 151311401318 128 GB
| PCI Slot 0 LSI-SAS 9205-8e
| PCI Slot 1 Emulex LPe12002 VM80175440 2-port
| Board 920-200040.B6 FXN 2015/24/Tue 03131653 fc:15:b4:8d:f9:6a
| Cluster serial        <SERIAL>               fc:15:b4:8d:f9:6b
| Board reset reason     EXTERNAL_RESET
| Current Time          2015-09-18 17:50:23 (UTC)
+-----------------------------------------------------------------------------+
Booting from net...
TFTP "install" from 192.168.100.81....................................................................................................complete

Setting FSB WDT Boot Complete State.
Devs: 263
Linux version 2.6.18-prep-3pardata (ddejong@moes) (gcc version 4.4.5 (Debian 4.4.5-8) ) #8 SMP Tue Jul 10 15:21:46 PDT 2012
Command line: BOOT_IMAGE=install console=ttyS0,57600 acpi_hest=off acpi=ht genapic=phys cr=2 ipaddr=169.254.154.129 nm=255.255.0.0 rp=192.168.100.81::rescue hn=SERIAL_0

...

Whack>

Whack>set perm cnt_no_os_boot=0

Whack>set perm cnt_no_cluster=0

Whack>set perm cnt_no_shutdown=0

Whack>set perm cnt_os_panic=0

Whack>set perm cnt_same_fatal=0

Whack>set perm cnt_log_error=0

Whack>set perm sys_serial=<SERIAL>

Whack>net addr 169.254.154.129
My address is 169.254.154.129

Whack>net netmask 0.0.0.16
Network mask 0.0.0.16

Whack>net gateway 192.168.100.81
Gateway address 192.168.100.81

Whack>net server 192.168.100.81
Server address 192.168.100.81

Whack>boot net install cr=2 ipaddr=169.254.154.129 nm=255.255.0.0

Can you confirm?

[djryan]

I ran it again and this is the output up to the sync.

HP SPI Image 3.1.05. Release version. 12:21:30 Aug 19 2014.

Bootstrap CPU0
Intel(R) Xeon(R) CPU E5-2428L 0 @ 1.80GHz
Family 6, Model 45, Stepping 7, Features bfebfbff:3
SKU Info: Max Frequency Ratio 18, Max Core Count 6
CPU speed: 1800.00 MHz
Bus speed: 100.00 MHz
Bus to CPU ratio: 1/18

Board reset reason: EXTERNAL_RESET
Previous reason : EXTERNAL_RESET

--> Starting: [0] CPU Interrupt Vector Initialization
--> Starting: [1] CPU SDRAM Memory Scan
Disabled FSBC boot watchdog.

Warning: PCI scan has not completed. It is not safe to use most Whack
commands at this point. Please resume initialization by typing "go" now.
Whack>
Whack>set perm cnt_no_os_boot=0
Whack>set perm cnt_no_cluster=0
Whack>set perm cnt_no_shutdown=0
Whack>set perm cnt_os_panic=0
Whack>set perm cnt_same_fatal=0
Whack>set perm cnt_log_error=0
Whack>set perm sys_serial=1635026
Whack>net addr 169.254.222.196
My address is 169.254.222.196
Whack>net netmask 0.0.0.16
Network mask 0.0.0.16
Whack>net gateway 147.19.151.107
Gateway address 147.19.151.107
Whack>net server 147.19.151.107
Server address 147.19.151.107
Whack>boot net install cr=2 ipaddr=169.254.222.196 nm=255.255.0.0 rp=147.19.151.107::rescue hn=1635026_1
--> Starting: [2] PCI Bus Initialization
--------- Starting PCI scan #2:

00.00.00 SandyBridge DMI2:
Vendor Intel, Device 0x3c00, Rev 0x07
Host/PCI bridge - Intel Device
00.01.00 SandyBridge PCIe Root Port 1a:
Vendor Intel, Device 0x3c02, Rev 0x07
PCI/PCI bridge, bus 01-07
01.00.00 PEX8648 #0:
Vendor PLX, Device 0x8648, Rev 0xbb
PCI/PCI bridge, bus 02-07
02.01.00 PEX8648 #0 DS1 (slot 2):
Vendor PLX, Device 0x8648, Rev 0xbb
PCI/PCI bridge, bus 03
02.04.00 PEX8648 #0 DS4:
Vendor PLX, Device 0x8648, Rev 0xbb
PCI/PCI bridge, bus 04
04.00.00 Harrier LPC2:
Vendor 3PAR, Device 0x0003, Rev 0x00
Large Memory Window 1 at 30:00000000
Disk cache controller
02.05.00 PEX8648 #0 DS5:
Vendor PLX, Device 0x8648, Rev 0xbb
PCI/PCI bridge, bus 05
05.00.00 Harrier LPC0:
Vendor 3PAR, Device 0x0003, Rev 0x00
Small Memory Window 1 at a0000000
Large Memory Window 1 at 10:00000000
Medium Memory Window 1 at 0c:00000000
Disk cache controller
02.08.00 PEX8648 #0 DS8:
Vendor PLX, Device 0x8648, Rev 0xbb
PCI/PCI bridge, bus 06
06.00.00 Slot 0:
Vendor LSI, Device 0x0087, Rev 0x01
SAS mass storage controller
02.09.00 PEX8648 #0 DS9:
Vendor PLX, Device 0x8648, Rev 0xbb
PCI/PCI bridge, bus 07
07.00.00 Slot 1:
Vendor Emulex, Device 0xf100, Rev 0x03
Fibre Channel controller
07.00.01 Slot 1:
Vendor Emulex, Device 0xf100, Rev 0x03
Fibre Channel controller
00.01.01 SandyBridge PCIe Root Port 1b:
Vendor Intel, Device 0x3c03, Rev 0x07
PCI/PCI bridge, bus 08
00.03.00 SandyBridge PCIe Root Port 3a:
Vendor Intel, Device 0x3c08, Rev 0x07
PCI/PCI bridge, bus 09
00.03.01 SandyBridge PCIe Root Port 3b:
Vendor Intel, Device 0x3c09, Rev 0x07
PCI/PCI bridge, bus 0a
00.03.02 SandyBridge PCIe Root Port 3c:
Vendor Intel, Device 0x3c0a, Rev 0x07
PCI/PCI bridge, bus 0b
00.03.03 SandyBridge PCIe Root Port 3d:
Vendor Intel, Device 0x3c0b, Rev 0x07
PCI/PCI bridge, bus 0c
00.04.00 SandyBridge DMA Engine CH0:
Vendor Intel, Device 0x3c20, Rev 0x07
Base system controller
00.04.01 SandyBridge DMA Engine CH1:
Vendor Intel, Device 0x3c21, Rev 0x07
Base system controller
00.04.02 SandyBridge DMA Engine CH2:
Vendor Intel, Device 0x3c22, Rev 0x07
Base system controller
00.04.03 SandyBridge DMA Engine CH3:
Vendor Intel, Device 0x3c23, Rev 0x07
Base system controller
00.04.04 SandyBridge DMA Engine CH4:
Vendor Intel, Device 0x3c24, Rev 0x07
Base system controller
00.04.05 SandyBridge DMA Engine CH5:
Vendor Intel, Device 0x3c25, Rev 0x07
Base system controller
00.04.06 SandyBridge DMA Engine CH6:
Vendor Intel, Device 0x3c26, Rev 0x07
Base system controller
00.04.07 SandyBridge DMA Engine CH7:
Vendor Intel, Device 0x3c27, Rev 0x07
Base system controller
00.05.00 SandyBridge Memory Map/VTd:
Vendor Intel, Device 0x3c28, Rev 0x07
Base system controller
00.05.02 SandyBridge RAS:
Vendor Intel, Device 0x3c2a, Rev 0x07
Base system controller
00.05.04 SandyBridge IOAPIC:
Vendor Intel, Device 0x3c2c, Rev 0x07
Base system controller
00.11.00 PCH Virtual Root Port:
Vendor Intel, Device 0x1d3e, Rev 0x06
PCI/PCI bridge, bus 0d
00.1c.00 PCH PCI Express Port 1:
Vendor Intel, Device 0x1d10, Rev 0xb6
PCI/PCI bridge, bus 0e
00.1c.06 PCH PCI Express Port 7:
Vendor Intel, Device 0x1d1c, Rev 0xb6
PCI/PCI bridge, bus 0f
0f.00.00 Intel 82574 #0:
Vendor Intel, Device 0x10d3, Rev 0x00
Ethernet network controller
00.1c.07 PCH PCI Express Port 8:
Vendor Intel, Device 0x244e, Rev 0xb6
PCI/PCI bridge, bus 10
10.00.00 Intel 82574 #1:
Vendor Intel, Device 0x10d3, Rev 0x00
Ethernet network controller
00.1e.00 PCH DMI to PCI Bridge:
Vendor Intel, Device 0x1d25, Rev 0xa6
PCI/PCI bridge, bus 11
00.1f.00 PCH LPC:
Vendor Intel, Device 0x1d41, Rev 0x06
PCI/ISA bridge
00.1f.02 PCH SATA:
Vendor Intel, Device 0x1d02, Rev 0x06
SATA mass storage controller
00.1f.03 PCH SMBus:
Vendor Intel, Device 0x1d22, Rev 0x06
System Management Bus controller

PCI address allocations:
16-bit I/O space: [ 1000 .. 503f]
32-bit prefetchable memory: [ (none) ]
32-bit non-prefetchable memory: [ 90000000 .. c07fffff]
64-bit prefetchable memory: [10:00000000 .. 3f:ffffffff]
36-bit prefetchable memory: [08:00000000 .. 0f:ffffffff]

PCI bus scan and allocation completed in 2 passes.

PCIe 1.0.0: Serial EEPROM contents verified.
--> Starting: [3] Serial Port Initialization
--> Starting: [4] Remote Console Initialization
--> Starting: [5] Random Number Generator Init
--> Skipping: [6] CPU SDRAM Memory Test
--> Starting: [7] Process NVRAM values
Prior reset 2019-10-23 21:15:13
Last reset 2019-10-24 13:26:53
This reset 2019-10-24 13:30:54
Time now 2019-10-24 13:31:34
--> Starting: [9] CPU APIC Initialization
CPU 0 [0] Microcode check...
Intel(R) Xeon(R) CPU E5-2428L 0 @ 1.80GHz
Family 6, Model 45, Stepping 7, Features bfebfbff:3

SKU Info: Max Frequency Ratio 18, Max Core Count 6
CPU speed: 1800.00 MHz
Bus speed: 100.00 MHz
Bus to CPU ratio: 1/18
CPU 0 [0] Done

CPU 3 [3] starting up.
Intel(R) Xeon(R) CPU E5-2428L 0 @ 1.80GHz
Family 6, Model 45, Stepping 7, Features bfebfbff:3

SKU Info: Max Frequency Ratio 18, Max Core Count 6
CPU 3 [3] shutting down.

CPU 5 [5] starting up.
Intel(R) Xeon(R) CPU E5-2428L 0 @ 1.80GHz
Family 6, Model 45, Stepping 7, Features bfebfbff:3

SKU Info: Max Frequency Ratio 18, Max Core Count 6
CPU 5 [5] shutting down.

CPU 6 [6] starting up.
Intel(R) Xeon(R) CPU E5-2428L 0 @ 1.80GHz
Family 6, Model 45, Stepping 7, Features bfebfbff:3

SKU Info: Max Frequency Ratio 18, Max Core Count 6
CPU 6 [6] shutting down.

CPU 8 [8] starting up.
Intel(R) Xeon(R) CPU E5-2428L 0 @ 1.80GHz
Family 6, Model 45, Stepping 7, Features bfebfbff:3

SKU Info: Max Frequency Ratio 18, Max Core Count 6
CPU 8 [8] shutting down.

CPU b [b] starting up.
Intel(R) Xeon(R) CPU E5-2428L 0 @ 1.80GHz
Family 6, Model 45, Stepping 7, Features bfebfbff:3

SKU Info: Max Frequency Ratio 18, Max Core Count 6
CPU b [b] shutting down.

CPU 2 [2] starting up.
Intel(R) Xeon(R) CPU E5-2428L 0 @ 1.80GHz
Family 6, Model 45, Stepping 7, Features bfebfbff:3

SKU Info: Max Frequency Ratio 18, Max Core Count 6
CPU 2 [2] shutting down.

CPU a [a] starting up.
Intel(R) Xeon(R) CPU E5-2428L 0 @ 1.80GHz
Family 6, Model 45, Stepping 7, Features bfebfbff:3

SKU Info: Max Frequency Ratio 18, Max Core Count 6
CPU a [a] shutting down.

CPU 9 [9] starting up.
Intel(R) Xeon(R) CPU E5-2428L 0 @ 1.80GHz
Family 6, Model 45, Stepping 7, Features bfebfbff:3

SKU Info: Max Frequency Ratio 18, Max Core Count 6
CPU 9 [9] shutting down.

CPU 1 [1] starting up.
Intel(R) Xeon(R) CPU E5-2428L 0 @ 1.80GHz
Family 6, Model 45, Stepping 7, Features bfebfbff:3

SKU Info: Max Frequency Ratio 18, Max Core Count 6
CPU 1 [1] shutting down.

CPU 7 [7] starting up.
Intel(R) Xeon(R) CPU E5-2428L 0 @ 1.80GHz
Family 6, Model 45, Stepping 7, Features bfebfbff:3

SKU Info: Max Frequency Ratio 18, Max Core Count 6
CPU 7 [7] shutting down.

CPU 4 [4] starting up.
Intel(R) Xeon(R) CPU E5-2428L 0 @ 1.80GHz
Family 6, Model 45, Stepping 7, Features bfebfbff:3

SKU Info: Max Frequency Ratio 18, Max Core Count 6
CPU 4 [4] shutting down.

..............................

ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata1.00: ATA-9: PU1000064GBATSSD, CS.51.00, max UDMA/133
ata1.00: 125045424 sectors, multi 1: LBA48 NCQ (depth 31/32)
ata1.00: configured for UDMA/133
Vendor: ATA Model: PU1000064GBATSSD Rev: CS.5
Type: Direct-Access ANSI SCSI revision: 05
SCSI device sda: 125045424 512-byte hdwr sectors (64023 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back
SCSI device sda: 125045424 512-byte hdwr sectors (64023 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back
sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 >
sd 0:0:0:0: Attached scsi disk sda
sd 0:0:0:0: Attached scsi generic sg0 type 0
LKCD: Crash dump driver initialized.
LKCD: block device driver registered
LKCD: dump gzip safety buffer: ffff81027d60a000, 4096
oprofile: using NMI interrupt.
Netfilter messages via NETLINK v0.30.
ip_conntrack version 2.4 (8192 buckets, 65536 max) - 280 bytes per conntrack
ctnetlink v0.90: registering with nfnetlink.
ip_tables: (C) 2000-2006 Netfilter Core Team
arp_tables: (C) 2002 David S. Miller
TCP bic registered
NET: Registered protocol family 1
NET: Registered protocol family 10
IPv6 over IPv4 tunneling driver
ip6_tables: (C) 2000-2006 Netfilter Core Team
NET: Registered protocol family 17
Freeing unused kernel memory: 4352k freed
Write protecting the kernel read-only data: 610k
=============================================================================

Netimagic(tm) Software Installer v
(C) 1991-2012 3PAR Inc. and others. All rights reserved.

=============================================================================
>> Configuring eth0: 169.254.222.196/255.255.0.0 gw
ADDRCONF(NETDEV_UP): eth0: link is not ready
>> Retrieving installation profile from http://147.19.151.107/iderescue/profile-DEFAULT
e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
mknod: /dev/zero: File exists
Creating partition table.
Disk is EOS sized.
1+0 records in
1+0 records out
SCSI device sda: 125045424 512-byte hdwr sectors (64023 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back
sda: unknown partition table
SCSI device sda: 125045424 512-byte hdwr sectors (64023 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back
sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 >
Building filesystems.
kjournald starting. Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting. Commit interval 5 seconds
EXT3 FS on sda3, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting. Commit interval 5 seconds
EXT3 FS on sda7, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Syncing 147.19.151.107::rescue to /mnt.

Dave

[djryan]

Sorry I might have cut off some of the output

+-----------------------------------------------------------------------------+
HP SPI Image 3.1.05. Release version. 12:21:30 Aug 19 2014.
+-----------------------------------------------------------------------------+
| CPU 1 x 1.80 GHz Sandy Bridge hexa core HT dual
| Control Cache Size 7.100 GB (cpu mem type 11) CL9
| Pair0 DIMM0 (J0155): 8192 MB CL5/11
| Data Cache Size 8 GB CL4/6
| Set0 DC DIMM 0.0.0 (J0300): 4096 MB CL4/6
| Set0 DC DIMM 0.1.0 (J0301): 4096 MB CL4/6
| Slot ID 1 [4 Node HP 3PAR 7400 Centerplane]
| FPGA Eos v0.f
| SATA0 Disk SanDisk PU1000064GBATSSD 131462402083 64 GB
| PCI Slot 0 LSI-SAS 9205-8e
| PCI Slot 1 Emulex LPe12002 VM80175440 2-port
| Board 920-200040.B5 FXN 2014/35/Thu 00264540 a4:5d:36:2e:2d:6a
| Cluster serial 1635026 a4:5d:36:2e:2d:6b
| Board reset reason EXTERNAL_RESET
| Current Time 2019-10-24 13:52:48 (UTC-4)
+-----------------------------------------------------------------------------+
Booting from net...
TFTP "install" from 147.19.151.107....................................................................................................complete

Setting FSB WDT Boot Complete State.
Devs: 263
Linux version 2.6.18-prep-3pardata (ddejong@moes) (gcc version 4.4.5 (Debian 4.4.5-8) ) #8 SMP Tue Jul 10 15:21:46 PDT 2012
Command line: BOOT_IMAGE=install console=ttyS0,57600 acpi_hest=off acpi=ht genapic=phys cr=2 ipaddr=169.254.17.183 nm=255.255.0.0 rp=147.19.151.107::rescue hn=1635026_1

....

Warning: PCI scan has not completed. It is not safe to use most Whack
commands at this point. Please resume initialization by typing "go" now.
Whack>
Whack>set perm cnt_no_os_boot=0
Whack>set perm cnt_no_cluster=0
Whack>set perm cnt_no_shutdown=0
Whack>set perm cnt_os_panic=0
Whack>set perm cnt_same_fatal=0
Whack>set perm cnt_log_error=0
Whack>set perm sys_serial=1635026
Whack>net addr 169.254.17.183
My address is 169.254.17.183
Whack>net netmask 0.0.0.16
Network mask 0.0.0.16
Whack>net gateway 147.19.151.107
Gateway address 147.19.151.107
Whack>net server 147.19.151.107
Server address 147.19.151.107
Whack>boot net install cr=2 ipaddr=169.254.17.183 nm=255.255.0.0 rp=147.19.151.107::rescue hn=1635026_1

Dave

[Torsten.]

This looks good.

You should see a task also (showtask).

How about task details? (showtask <task# -d)

Can you logon now to node 1?

[djryan]

No it fails, and the green light is solid. It can't join the cluster, that is why I created this ticket.

2019-10-24 10:51:43 PDT Created task.
2019-10-24 10:51:43 PDT Updated Running node rescue for node 1 as 0:1454
2019-10-24 10:51:51 PDT Updated Using IP 169.254.17.183
2019-10-24 10:51:51 PDT Updated Informing system manager to not autoreset node 1.
2019-10-24 10:51:51 PDT Updated Resetting node 1.
2019-10-24 10:51:51 PDT Updated Attempting to contact node 1 via NEMOE.
2019-10-24 10:52:20 PDT Updated Setting boot parameters.
2019-10-24 10:52:40 PDT Updated Waiting for node 1 to boot the node rescue kernel.
2019-10-24 10:52:56 PDT Updated Kernel on node 1 has started. Waiting for node to retrieve install details.
2019-10-24 10:53:15 PDT Updated Node 1 has retrieved the install details. Waiting for file sync to begin.
2019-10-24 10:53:32 PDT Updated File sync has begun. Estimated time to complete this step is 5 minutes on a lightly loaded system.
2019-10-24 11:00:38 PDT Updated Remote node has completed file sync, and will reboot.
2019-10-24 11:00:38 PDT Updated Waiting for node to rejoin cluster.
2019-10-24 11:30:38 PDT Error The node has not rejoined the cluster after 30 minutes. The rescue was not successful.
2019-10-24 11:30:38 PDT Failed Could not complete task.

[djryan]

This is why the previous HP engineer said I need to get into WHACK which I haven't been able to do in order to do:

WHACK>prom edit

[Torsten.]

What exactly should you edit?

[Torsten.]

post deleted, this was the wrong thread, sorry.

[djryan]

I was told to edit the following as seen on page 1 of my ticket.

Remember 7400 and 7400C parameters are different ; 

Look at the example for a 7400 below ;

SMI Whack> prom edit

Board Spin:       03

Size * 256 bytes: 04

Board Class:      920

Board Base:       200040

Board Rev:        91

Assembly Vendor:  FXN

Assembly Year:    2012

Assembly Week:    30

Assembly Day:     03

Assembly Serial:  00103530

Alternate Class:  ffff

Alternate Base:   ffff

Alternate Dash:   ff

System Serial:    <7 digit serial>         < ----- System Serial is set with  the command 'set perm sys_serial=<7 digit serial>'

Node ID:          01                                 < --------- set correct node ID  - node 1 = 01   node 0 = 00 ; likewise. 

Midplane Type:    1b                            < ---------- This is the correct value for 7400

Node Type:        40                              < ---------- This is the correct value for 7400

[djryan]

I really have no idea what you are talking about. We had a controller go out and we ordered a new one from HP. We recieved it and it has never worked from day one. This is the reason for this ticket. I have no idea what your talking about. We have 2 fiber connections and 2 SAS connections and the ethernet connnections to it. I don't know what a VC module is.

If you don't have VC modules, there is no counterpart for the NICs, hence your problem.

[Paul28]

Hello  djryan , 

To connect serially, you need the right settings.

On the serial connect software tool, set the below ; 

1. Baud = 57600
2. Data bits = 8
3. Parity = None
4. Stop bits = 1
5. Flow control = Xon/Xoff

If we can get a prompt at the serial ( but whack does not run )  type 'reset' and the node will reboot 

While rebooting at the  point when you reach 'GRUB'  (Welcome to GRUB)  use the CTRL + W again which will get us to the WHACK. 

Also, I like to know how you are trying to connect to serial  - what cable do you use and color ?  

Note : An orange / Red could be a crossover cable. which may not help. 

[djryan]

I am using a cisco DB9 to RJ45 console cable because no one can find the original one. If this is incorrect can you please provide me with the correct part number for this cable. 

Thanks,

Dave

[djryan]

I found the adapter that came with the unit and we still cannot get it to accept input. We have partno 180-0055-01 adapter with a standard ethernet cable and it still does not respond to input. I tried pressing CTRL-w at the GRUB menu but it ignores me and I try to type at the prompt but it doesn't accept input.

init: rsync daemon not enabled in /etc/default/rsync, not starting... ... (warning).
init: Starting periodic command scheduler: cron.
init: Running local boot scripts (/etc/rc.local).

3PAR(R) InForm(tm) OS 3.2.1.292 1635026-1 ttyS0
1635026-1 login:

[Paul28]

Looks like that part works if its not a faulty one. However, Part Number: 818725-001 is the HPE recommended part.