Hi @CR85 !
Since you have rebooted Chassis 2, did you see any errors just after reboot of Chassis 2 explaining why IRF can't form up? Check with "display version" if both chassis are running the same s/w version, since you mention upgrade it's very possible one of your chassis to remain on the older version and that's why IRF can't form. Logbuffer messages from both Chassis before and after reboot of Chassis 2 should reveal why IRF process can't form the stack. Also, what is the state of IRF ports on both Chassis - up, down? What is the state of BAGG1 from Chassis 1's and 2's perspective?
Hi @CR85 !
Thank you for the information provided!
Could you post output from the following commands:
Chassis1:
=========
display system-working-mode
display ecmp mode
display interface FortyGigE1/0/0/1
display transceiver diagnosis interface FortyGigE1/0/0/1
display transceiver alarm interface FortyGigE1/0/0/1
Chassis2:
=========
display system-working-mode
display ecmp mode
display interface FortyGigE2/0/0/1
display transceiver diagnosis interface FortyGigE2/0/0/1
display transceiver alarm interface FortyGigE2/0/0/1
just a note while waiting for the output - at this time Chassis 2 is in recovery state and all its ports except FortyGigE2/0/0/1. MAD is in faulty state, because XGE2/1/0/21 is down - disabled by the MAD (that we are going to verify when I get the output of requested commands). Until you restore the IRF link, e.g. restore physical link between FortyGigE1/0/0/1 and FortyGigE2/0/0/1, the Chassis 2 will stay isolated - Master role, but all interfaces except FortyGigE2/0/0/1 are down to minimize network impact of split-brain.
That is why I requested the output from display transceiver command to see what is going on with physical link between FortyGigE1/0/0/1 and FortyGigE2/0/0/1. If we will see no alarms and normal Tx and Rx signal levels, but ports down, try to re-plug the transceiver on Chassis 1 and check if interface will go up. Monitor for log messages on both chassis - if they sence that the IRF physical port goes up, they should try to form IRF and if that fail there must be an error message with the reason.
Yes, it's a DAC cable, that's why there is no monitoring values. I am wondering why the link stays down, because I think this is the root cause - IRF link stays down->IRF is not formed->split-brain->MAD disabled ports on Chassis 2.
Try to re-plug the transceiver on Chassis 1 and 2 and check if link will go up. Monitor for log messages on both chassis - if they sence that the IRF physical port goes up, they should try to form IRF and if that fail there must be an error message with the reason. Do you have a spare DAC to try, just in case?
Hi Christoph!
I have requested "display diag" outputs in PM, let's see, maybe there will be some additional info that could point us to the root cause.
Hi @CR85 !
I have checked the files you uploaded. Actually the most interesting file is the "after_cabletest" one - only it contains traces of the event when you tried optical transceiver instead of DAC and I see how STM process (main driver of IRF stack) on Chassis 2 detected IRF packets from Chassis 1:
%@176442%Nov 23 13:42:40:428 2020 sw79 IFNET/3/PHY_UPDOWN: Physical state on the interface FortyGigE2/0/0/1 changed to up.
%@176443%Nov 23 13:42:40:430 2020 sw79 IFNET/5/LINK_UPDOWN: Line protocol state on the interface FortyGigE2/0/0/1 changed to up.
%@176444%Nov 23 13:42:42:981 2020 sw79 STM/6/STM_LINK_UP: IRF port 1 came up.
Then Chassis 2 restarted in order to converge:
%@176446%Nov 23 13:45:37:886 2020 HPE SYSLOG/6/SYSLOG_RESTART: System restarted --
However after the restart I see no message that Fourty2/0/0/1 went UP, seems like after the restart it's still DOWN and that's why MAD kicks in:
%@176813%Nov 23 13:57:27:129 2020 sw79 DEV/1/MAD_DETECT: Multi-active devices detected, please fix it.
Unfortunately neither of diags contains traces if DACs made any attempt to come up. I am starting to assume the issue with P07 patch...
Could you do me a favor and generate new "display diags" from Chassis 1 and 2, because neither of both contains messages from that period of time?
Also, each MPU on both Chassis has two directories - 'logfile' and 'diagfile'. Could you retrieve the contents of those directories, ZIP them into different archives and name the archives to reflect the source of those files, for example:
chassis1#slot10#flash:/logfile/*.*
chassis1#slot10#flash:/diagfile/*.*
pack to ZIP archive named 'chassis1slot10.zip'
chassis1#slot11#flash:/logfile/*.*
chassis1#slot11#flash:/diagfile/*.*
pack to ZIP archive named 'chassis1slot11.zip'
etc... or make one archive with directory structure inside that will let me know which file from which filesystem comes. I am trying to find that error message of STM process whith the reason why it didn't form the stack during the test with optical transceivers.