Update to my previous post. I ended up getting this fixed myself.
We did eventually hear back from a tier 3 storage support engineer, but not directly and he had only indirectly viewed our config files, after over a week not a single person at HP was willing to share a screen or anything to look at the CMC directly. I have to assume if they are the experts this should have been something they should have been able to see as an issue pretty early on if they had just looked at it. Their recommended solution was to downgrade the CNA firmware back to a 2015 firmware, I didn't end up doing that.
It turns out the packet loss notices had nothing to do with packets being dropped in the network, it is either a timing or buffering issue, something that changed between 12.6 to 12.7. As noted before I have 10gbe on the iscsi side and 1gbe on the management side, somewhat in the middle of those is the "lefthand os interface" which is where the nodes talk to each other. What I did not previously know is that the lefthand os interface is also what is using for balancing, syncing, etc. I originally had this interface for each node on the management side, having not wanted to clutter up the iscsi network if possible and also because this was required for our prior cluster that had a FOM because the FOM has no iscsi connectivity and is really only there as a manager and quorom witness.
After changing the lefthand os interface to the 10gbe bond on each node (so it is sharing the iscsi bond now) I have received zero packet loss notifications. Not sure why this was never an issue on 12.6 or if this is what HPE considers best practice since they offer little detail about the 3 interfaces and how they should be setup, but I've had no issues in the last 24 hours with this configuration. Hope this helps someone else.
