Meaning of output from scacp

[MarkOfAus]

Volker,

The "SHOW PORT/ADDR=81F93830/CHAN/VC" shows
buffer sizes as:

EWA
Current: 1426
Remote: 1426
Local: 1426
Negotiated: 1426

EWB
Current: 564
Remote: 8120
Local: 8120
Negotiated: 8120


In this "cluster", there is a one-to-one connection, so ewa <-> ewa and ewb <-> ewb.
The output is a little hard to decipher, given there is listed eia<->ewb, ewb<->eia, eia<->ewb, ewb<->ewb, ewb<->eia, eia<->eia, ewa<->ewa.

Regards,
Mark.

[MarkOfAus]

Volker,

"interestingly, the EWB counter for Jumbo transmits is 44, whereas the same counter for EWA is 0. I doubt that EWA is actually sending or receiving Jumbo frames."

I think that is correct. Why, well I am not sure. I figure that jumbo frames should be handled ok by a connection that is basically a cross-over connection, albeit over fibre. So, ewa should be sending/receiving jumbo frames. Ewb, on the other hand, has switches inbetween it, each of which is not configured for jumbo frames (vms can send them, the switch will kill them before hitting the stack).


"What's the setting of NISCS_MAX_PKTSZ ?"

It is 8192.
SYSGEN> SHOW NISCS_MAX_PKTSZ
Parameter Name Current Default Min. Max. Unit Dynamic
-------------- ------- ------- ------- ------- ---- -------
NISCS_MAX_PKTSZ 8192 8192 576 9180 Bytes

[MarkOfAus]

Wim,

" Note for the SCA part in the enclosure :

EWA
Maximum buffer size 1498

EWB
Maximum buffer size 8998
"

Ok, but what does this mean? Are they adjustable?


" I know nothing about Jumbo's but
EWB also has a low number of collisions.

It tried to use jumbo's but failed ?"

Yes, presuming our network people told me the correct information, the switches on which ewb is communicating are not jumbo enabled.


"Is the stuff between the cluster nodes jumbo ready ?"

Not for ewb, but I assumed for ewa, which is communicating through a fibre link, so essentially is a direct connection to the other server.


Regards,
Mark

[Volker Halle]

Mark,

maybe your network is not like you think it is...

To find out, which LAN interface in this cluster is able to talk with which other LAN interface on other nodes in this cluster, try this:

$ SET TERM/WIDTH=132
$ MC SCACP SHOW CHANNEL

This will show you all possible SCA communication channels, that exist in the local node. Look at those channels with Channel State 'Open'.

A channel is formed between a local LAN adapter (running SCA protocol) and all other LAN adapters in the cluster, which receive/send the SCA Hello multicast message.

If the channel state is 'Clsd', there once was a network path, but it's not available anymore (in such a case, I also see Buffer Size 564 at the channel level). If there has been network re-configuration activity during the lifetime of this node, you can expect to see various closed channels, but you also can see the CH open and close times.

Before any further activity, make sure that there are at least 2 working channels between any two nodes in your cluster. This would prevent problems, if anything happens to any of your LANs.

Note that the Maximum buffer size for SCA in the SDA output matches the SCACP SHOW LAN buffer sizes:

EIA2 60-07 (SCA) Maximum buffer size 1498
EWA2 60-07 (SCA) Maximum buffer size 1498
EWB10 60-07 (SCA) Maximum buffer size 8998

This is at least consistent.

It also looks like SCA on the LAN adapter EWB has been stopped and started in the running system. Normally SCA would be the first protocol started during boot.

Volker.

[Richard Stockdale]

The LAN drivers always have jumbo frames enabled, which are 9018 byte max for Broadcom devices, 8192 bytes for Intel Gigabit devices.

The LANCP SET DEV/JUMBO or the LAN_FLAGS settings only changes what VCI applications are told is the max buffer size. And this is enforced on transmit by the LAN drivers.

You can do an SDA SHOW LAN/VCI to see the current state of the VCI applications and the buffer size.

If you change LAN_FLAGS or do a LANCP SET DEV/[NO]JUMBO, applications won't pick up the change unless you stop and restart them on the device. So you have to SCACP STOP LAN devname then START LAN devname to change PEDRIVER settings on a device.

The best way to see usage of jumbo frames is LANCP SHOW DEV/INTERNAL_COUNTERS so you can see what jumbo transmits and receives have been done.

Note that if the number of jumbo transmits is small, like 44 as described, it is probably because PEDRIVER is doing size probing and not getting a response on the size probe.

The 'buffer size' in SCACP is correct. Note that it is limited by NISCS_MAX_PKTSIZE so even if the LAN device supports 9018 byte packets, PEDRIVER only goes up to NISCS_MAX_PKTSIZE less some header size.

Also, LANCP SHOW DEV/CHAR shows the device buffer size reported for QIO users. This was never changed when jumbo frames were implemented to avoid affecting existing QIO applications. QIO applications can send/receive jumbo frames regardless of the setting of LAN_FLAGS or SET DEV/JUMBO.

The LAN_FLAGS and SET DEV/JUMBO exists for VCI applications. Some like PEDRIVER can dynamically figure out the max packet size usable. TCP/IP can not, hence the existence of this jumbo flag setting.

- Dick

[MarkOfAus]

Volker,

"maybe your network is not like you think it is..."

And I am starting to think you are correct... :-)

"To find out, which LAN interface in this cluster is able to talk with which other LAN interface on other nodes in this cluster, try this:

$ SET TERM/WIDTH=132
$ MC SCACP SHOW CHANNEL

This will show you all possible SCA communication channels, that exist in the local node. Look at those channels with Channel State 'Open'."

See attached output.
There was a bit of plugging/unplugging when I moved one of the servers to a remote location, perhaps this explains the eia<->ewb , though I doubt it.


"Note that the Maximum buffer size for SCA in the SDA output matches the SCACP SHOW LAN buffer sizes:

EIA2 60-07 (SCA) Maximum buffer size 1498
EWA2 60-07 (SCA) Maximum buffer size 1498
EWB10 60-07 (SCA) Maximum buffer size 8998

This is at least consistent.
"

But again it leads to the question of why? Why is ewb greater than ewa when they are the same speed?

"It also looks like SCA on the LAN adapter EWB has been stopped and started in the running system. Normally SCA would be the first protocol started during boot."

I can't get anything past you. :-)

I stopped ewb (scacp stop lan ewb) to test throughput on ewa. eia was the backup (10/100) during this test.

Regards
Mark

[Richard Stockdale]

>>But again it leads to the question of why? Why is ewb greater than ewa when they are the same speed?

Speed is of no consequence other than whether the LAN device is capable of supporting jumbo packets. The size is determined by the results of PEDRIVER size probing and, of course, whether the LAN device was enabled for jumbo frames when PEDRIVER started on the device.

- Dick

[MarkOfAus]

Dick,

"You can do an SDA SHOW LAN/VCI to see the current state of the VCI applications and the buffer size."

See attached sho lan/vci output. I couldn't find any reference to buffers.

"Note that if the number of jumbo transmits is small, like 44 as described, it is probably because PEDRIVER is doing size probing and not getting a response on the size probe."

This was on the device connected to the lan via switches incapable (read: cisco) of handling jumbo frames. So, as you say, the 44 was probably it probing until I disabled jumbo frames (and then stopped the lan device & restarted it).

Note, that with jumbo frames enabled on ewb AND the switches it was connected to at both ends incapable of jumbo frames, it still became the prefered path during a copy operation even though its priority was lower than ewa. Hence this discussion.


"The 'buffer size' in SCACP is correct. Note that it is limited by NISCS_MAX_PKTSIZE so even if the LAN device supports 9018 byte packets, PEDRIVER only goes up to NISCS_MAX_PKTSIZE less some header size.
"

So does this mean it is better to increase NISCS_MAX_PKTSIZE to, say, 10000?

Regards,
Mark.

[MarkOfAus]

Dick,

" >>But again it leads to the question of why? Why is ewb greater than ewa when they are the same speed?

Speed is of no consequence other than whether the LAN device is capable of supporting jumbo packets. The size is determined by the results of PEDRIVER size probing and, of course, whether the LAN device was enabled for jumbo frames when PEDRIVER started on the device.
"

Ok, so it looks like ewa is not using jumbo frames but neither is ewb (I disabled them).

Why is not ewa negotiating jumbo frames? ewa is connected remotely to the other server via media converters and a fibre connection. Do I need a capable switch in-between? Seems odd if so.

More info.
Remote ewa is DEGXA-TB, local ewa is DEGXA-TB.
Remote ewa is 5703 LOM, local ewb is DEGXA-TB.


Regards
Mark

[Richard Stockdale]

>>See attached sho lan/vci output. I couldn't find any reference to buffers.

The field is "Max xmt size".

>>Note, that with jumbo frames enabled on ewb AND the switches it was connected to at both ends incapable of jumbo frames, it still became the prefered path during a copy operation even though its priority was lower than ewa. Hence this discussion.

It is hard to tell without being there. But priority wouldn't necessarily override lossiness of a channel when forming an ECS.

>>So does this mean it is better to increase NISCS_MAX_PKTSIZE to, say, 10000?

It would be if NISCS_MAX_PKTSIZE were not limited to 8192, or at least, I thought it was. I just checked a V73-2 system and it was limited to 9180. I'm not sure it it is limited to the max lookaside list size or really limited to 8192 bytes.

- Dick