HP2848 port asymmetry? MPI all-to-all communication

[Carsten Kutzner]

Hi,
I wonder whether the HP2848 switch has any asymmetry build-in (backplane?), or if any asymmetry could be induced accidentally (port buffers?) by the way in which it is used. I am doing MPI all-to-all communication on 32 ports and the performance differs depending on whether the MPI ranks are ordered with respect to the switch's ports or not.

In an ordered fashion, meaning that the process with MPI rank 0 uses port 0, rank 1 port 1 and so on, the performance is a lot worse for messages larger than approx. 2 k compared to the case when the ranks are shuffled (e.g. rank 0 on port 13, rank 1 on port 24, ...)
This behaviour is seen when using 32 ports, not for 16, 8, 4 ports. I tested with 3 different MPI implementations, so I think it is most likely due to the switch.

Performance differences are seen in the following way: For a given message size, a typical all-to-all call takes 0.03 seconds if the ranks are shuffled, but 0.2 seconds if the ranks are ordered. Doing the test several times with ranks that are slighly shuffled yields something like 0.03s, 0.20s, 0.21s, 0.03s, ... either fast or slow.

Does anyone have an idea what is the reason for this behaviour??

Thanks,
Carsten

[Manfred Arndt]

Hi Carsten,

The performance degradation is probably due to packet loss. This switch has a dynamic memory buffer pool shared across all ports. With 32 ports active, you probably are exceeding the buffers assigned to the default priority queues.

Try enabling 'qos-passthrough-mode'. This is designed to enhance the performance of line-rate traffic transfers. This feature is intended where QoS is not of major importance, but where lossless data transfers are key.

See 10-25 in the following guide:
ftp://ftp.hp.com/pub/networking/software/Mgmt-Oct2005-59906023-Chap10.pdf

Cheers

[Carsten Kutzner]

Hi Manfred,

thanks a lot for your reply. I enabled qos-passthrough-mode in the switch and it helped a bit, that means now the critical message size where packet drops occur is about twice as big. The fact that shuffling the input ports solved the problem caught my interest and I took a look into the switch.

It seems like four 12-port BCM5690 are connected to an 8-port 5670 with 10 Gbps link each. Probably here is the bottleneck: 12 ports generate more than 10 Gbps on these links during my test program, so the switch has to drop packets internally if they cannot be stored somewhere. When I only use 9 of each 12 ports (leaving the ports 10-12, 22-24,... empty) there is no packet loss. (Using 10 ports out of 12 still shows drops, don't know why).

So my conclusion is: When only 32 ports are needed, don't use ports 1-32 but rather 1-9, 13-21,25-33,37-45 to get maximum performance.
Or generally distribute the non-used ports evenly across the switch. Am I right here?

Thanks,
Carsten

[Manfred Arndt]

Hi Carsten,

Yes, you are correct. It definately sounds like the 12 ports are generating more than 10 Gbps for a sustained duration in your test configuration. Distributing the ports as you suggest will elimitate the 10 Gbps bottleneck to the 5670.

In "real-world" environments, most networks will not be subjected to this type of traffic. For high performance environments, you might want to consider our recently introduced 3500 switch that does not suffer from this limitation.

Cheers