How to replace card TIA simplivity 380.

[hbangvip]

I am replacing the TIA card. The card light is green but OVC still does not recognize the card. Can anyone please guide me about this error?

[Joshua610]

How to troubleshoot before replacing TIA card?

Answer

This article aims to ensure that all HPE SimpliVity support engineers are following the correct procedure to determine whether a TIA card is truly faulty or not. High numbers of replacements are seen for TIA cards, many of which turn out to be healthy on testing once returned. Following the procedure outlined below will ensure that we are properly testing our TIA cards before replacing.

Where a TIA card is to be replaced, engineers are requested to attach the TIA replacement checklist, as well as a support bundle from the node in question to the case for subsequent analysis by our engineering team to help us gather further learnings from these failures.

TIA errors may be seen during deployment, and post deployment.

TIA Errors During Deployment

During deployment it is possible that TIA related errors may also be seen. In such scenarios reseating the TIA card, and draining the flea power and re-attempting the deployment is recommended.

If issues are still seen post deployment, and when access to the OVC via an ssh session is possible, follow the procedure as outlined below.

TIA Errors Post Deployment

1) Verify Accelerator Card status. 

Versions 3.5.2 and later: 

#modprobe tiadriver 

#cat /proc/IA0/card_status  

Versions 3.5.1 and earlier: 

#insmod /var/tmp/build/drivers/tiadriver.ko 

#cat /proc/IA0/card_status 

Take note of: 

a.    Nios State 

b.    State Restore Status 

 Nios_Heartbeat: Alive 

 Nios_Active: Active 

 Nios_State: Fatal Error 

 Nios_PC: 0x00000000 

 Card_Status: Success 

 Post_Status: Success 

 Reset_Type: Cold_Reset 

Number_NVRAM_1MB_Blocks: 7856 

 ECC_Correctable_Count: 14 

 SEU_Count: 0 

 Power_Cycle_Count: 0 

 SS_Enabled_Count: 0 

 Execute_SS_Count: 0 

 ARM_Correctable_ECC_Count: 0 

 State_Restore_Status: State_Restore_No_Image 

2) Verify heartbeat was lost in syslog: 

Verify that the “Lost NIOS heartbeat” exists in the syslog. This message should be 

dated at the same time as the “SimpliVity OmniCube Accelerator unresponsive” 

alarm in the VCenter window. An example using grep is shown below. (on the 

SVA)   

#grep "IA: Warning: IA0: Lost NIOS heartbeat" /var/log/syslog 

3) Verify NVRAM Saved To Disk: 

Verify that the stack has saved the NVRAM to disk. This can be done by verifying 

that the following two lines are in the SVTFS log and are time stamped for the 

current time. 

(note – remember that for Senna (3.5.1 and earlier) releases, the SVTFS log timestamps are usually 

4 hours ahead of the syslog) The two commands are shown below. 

a.      #grep "Persisting complete NVRAM image to disk" /var/svtfs/0/log/svtfs.log 

b.     #grep "NVRAM image saved to hard disk" /var/svtfs/0/log/svtfs.log  

NOTE:  You may need to look back through older/all svtfs logs to confirm.

Output similar to following should be seen: 

2020-01-28T17:59:23.152Z INFO 0x7f1a60890880 [:] [nvrammanager] nvram_manager.cpp:2137 Persisting complete NVRAM image to disk. Size 8237613056 8237613056 

2020-01-28T17:59:30.797Z INFO 0x7f1a60890880 [:] [nvrammanager] nvram_manager.cpp:2186 NVRAM image saved to hard disk.  

4) Confirmed NVRAM persisted successfully: 

If NVRAM has been successfully saved to disk, proceed to reset to manufacture state and reboot the TIA card and OVC in an attempt to recover the card.

NOTE:

Resetting to manufacture state of the card should ONLY be done, where NVRAM has been confirmed to have flushed, and where card_status = No_Image

NOTE: A nostart file may also be present, so confirm this and remove it firstly. 

Check both locations to confirm:

NOTE: SVTFS must be in a stopped state when running the following steps:

sudo su 

source /var/tmp/build/bin/appsetup 

Confirm existence of nostart file and remove.

Check in: /var/svtfs/0 AND /var/svtfs/svt-hal/0

To remove:

rm /var/svtfs/0/nostart

Load the TIA driver:

(3.5.1 or earlier) 

insmod /var/tmp/build/drivers/tiadriver.ko 

(3.5.2 or later) 

modprobe tiadriver 

Clear NVRAM back to manufacture state, arm for PCIe reboot, and execute an OVC reboot :

tiacli --initializeOmniCubeAcceleratorToManufactureState 

tiacli --enableOmniCubeAcceleratorpciereboot 

Reboot 

Once the OVC reboots, verify that the stack comes up successfully by checking the status of svtfs. 

tail –f /var/svtfs/o/log/svtfs.log (SVTFS should be in a running state with a PID as below) 

svtfs (0) start/running, process 2694 

Once stack is successfully up, do a final check on card status: 

Versions 3.5.2 and later: 

#modprobe tiadriver 

#cat /proc/IA0/card_status  

Versions 3.5.1 and earlier: 

#insmod /var/tmp/build/drivers/tiadriver.ko 

#cat /proc/IA0/card_status 

5) Reconfirm card status

If card status is healthy, and svtfs comes up successfully, then no further action needed, and card can remain. 

If svtfs fails to come up, and card_status still shows errors, replace card.

6) If NVRAM Hadn’t Saved To Disk 

 If it has been confirmed that NVRAM has not been persisted, and card status shows “State_Restore_No_Image” an attempt to flush NVRAM to onboard FLASH on the TIA can be made by running the following two commands and repeat steps to confirm successfully persisted. 

#tiacli –enablestatesave 

#tiacli –executestatesave 

Once done, repeat troubleshooting steps to confirm card_status and NVRAM status.

If NVRAM had not flushed correctly, then the node will need to be redeployed.

NOTE: TIA may still actually be healthy. Node should be redeployed firstly, and TIA card health tested again before replacing the TIA card.

[hbangvip]

Dear Joshua610,

Thank you very much. I followed the steps you sent and the card worked.

Once again I thank you!

[Joshua610]

Thank you for that - take care & stay Blessed my friend. 

[Thaufique_Mod · Accepted solution]

Hello @hbangvip,

Let us know if you were able to resolve the issue.

If you are satisfied with the answers then kindly click the "Accept As Solution" button for the most helpful response so that it is beneficial to all community members.

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".