Meaning of output from scacp

[MarkOfAus]

Hi all,

Here is the output of "scacp show lan" on 7.3-2 machine, clustered:
Device Errors + Mgt Buffer MgtMax Line Total Current
Device Type Events Status Priority Size BufSiz Speed Pkts(S+R) LAN Address
------ ---- ------ ------ -------- ---- ------ ----- --------- -----------
LCL 0 Run Online Local Restart 0 1426 0 N/A 215858 00-00-00-00-00-00
EIA 82559 72 Run Online Restart -3 1426 0 100 102775206 AA-00-04-00-05-04
EWA DEGXA 11 Run Online Restart 10 1426 0 1000 301676325 00-17-A4-3D-DF-7F
EWB DEGXA 7 Run Online Restart 5 8120 0 1000 147026381 00-17-A4-3D-CF-0A


(If you can't read from this horrible formatting, the "buffer size" for the EWA device is 1426, for the EWB device it is 8128)

What I would like to know is what is the "Buffer Size" column?

Why do the two ewa and ewb have different "buffer sizes" when they are the same cards?

Do I trust VMS has got this "buffer size" correct?

A lancp sho dev/char ewa and ewb show that they have the same minimum and maximum receive buffers.

The only clue, perhaps, is that ewa is a fibre-connected (nic to fibre-converter) whereas ewb is connected into the lan and therefore using a switch?

EWA has jumbo frames enabled, EWB did, but they have been disabled since a reboot.

Any information would be helpful.

Regards
Mark

[Volker Halle · Accepted solution]

Mark,

the 'buffer size' is most likely the useable buffer size for the SCA protocol. Are you sure about JUMBO frames and EWA/EWB ? Could it be the other way around ?

What does ANAL/SYS and SDA> SHOW LAN/FULL give a Receive buffer size for these LAN interfaces ?

Volker.

[Robert Gezelter]

Mark,

I would tend to agree with Volker that EWB has jumbo frame support enabled.

The original Ethernet specification has a lower limit of 1500 bytes/frame (see the actual specification, for the precise contents).

Gigabit Ethernet permits so-called "Jumbo frames", up to 9000 bytes in size.

- Bob Gezelter, http://www.rlgsc.com

[MarkOfAus]

Volker,

I attached the voluminous output of sda show lan, but have also included a bit of data in this reply (the part that shows the buffers).

"the 'buffer size' is most likely the useable buffer size for the SCA protocol. Are you sure about JUMBO frames and EWA/EWB ? Could it be the other way around ?"

Both devices were jumbo, now only ewa is:

Output from lancp (driver messages portion):
EWA
---
18-FEB-2008 20:53:28.79 Link up: 1000 mbit, full duplex, flow control disabled
18-FEB-2008 20:53:25.56 Link down
18-FEB-2008 13:12:07.47 Link up: 1000 mbit, full duplex, flow control disabled
18-FEB-2008 13:12:04.23 Link down
18-FEB-2008 13:12:04.23 Jumbo frames enabled
18-FEB-2008 13:10:16.17 Link up: 1000 mbit, full duplex, flow control disabled
18-FEB-2008 13:10:12.86 Device type is BCM5703C (UTP) Rev A0 (11000000)
18-FEB-2008 13:10:12.85 DEGXA-TB located in 64-bit, 33-mhz PCI slot
18-FEB-2008 13:10:12.85 Auto-negotiation mode set by console (EGA0_MODE)
S

And EWB:
--------

18-FEB-2008 20:57:56.38 Link up: 1000 mbit, full duplex, flow control (recv)
18-FEB-2008 20:57:53.25 Link down
18-FEB-2008 20:57:53.25 Jumbo frames disabled
18-FEB-2008 15:39:49.46 Link up: 1000 mbit, full duplex, flow control (recv)
18-FEB-2008 15:39:49.14 Link down
18-FEB-2008 15:39:45.62 Link up: 1000 mbit, full duplex, flow control (recv)
18-FEB-2008 15:26:02.39 Possible duplex mode mismatch condition detected
18-FEB-2008 14:25:26.44 Possible duplex mode mismatch condition detected
18-FEB-2008 13:25:06.84 Possible duplex mode mismatch condition detected
18-FEB-2008 13:25:01.69 Link up: 100 mbit, half duplex, flow control disabled
18-FEB-2008 13:24:49.83 Link down
18-FEB-2008 13:18:40.03 Link up: 1000 mbit, full duplex, flow control (recv)
18-FEB-2008 13:15:51.47 Link down
18-FEB-2008 13:12:07.38 Link up: 1000 mbit, full duplex, flow control (recv)
18-FEB-2008 13:12:04.23 Link down
18-FEB-2008 13:12:04.23 Jumbo frames enabled
18-FEB-2008 13:10:16.07 Link up: 1000 mbit, full duplex, flow control (recv)
18-FEB-2008 13:10:12.90 Device type is BCM5703C (UTP) Rev A0 (11000000)
18-FEB-2008 13:10:12.89 DEGXA-TB located in 64-bit, 33-mhz PCI slot
18-FEB-2008 13:10:12.89 Auto-negotiation mode set by console (EGB0_MODE)



"What does ANAL/SYS and SDA> SHOW LAN/FULL give a Receive buffer size for these LAN interfaces ?"

-- EWA Device Information (cont) 20-FEB-2008 08:38:48 --

Put rcv ptr/index 00000000 Get rcv ptr/index 000003E1
Put xmt ptr/index 000000D9 Get xmt ptr/index 000000D9
Put cmd ptr/index 00000121 Get cmd ptr/index 00000000
Put uns ptr/index 00000008 Get uns ptr/index 00000000
Put smt ptr/index 00000000 Get smt ptr/index 00000000
RBufs owned by dev 328 Rcv packet limit 128
XEnts owned by dev 0 XEnts owned by host 512
CEnts owned by dev 0 Transmit timer 0
UEnts owned by dev 0 Control timer 0
SEnts owned by dev 0 Periodic SYSID timer 621
Current rcv buffers 328 Ring unavail timer 0
Rqst MAX rcv buffers 256 USB timer 144
Rqst MIN rcv buffers 128 Receive alignment 0
Curr MAX rcv buffers 264 Receive buffer size 1518
Curr MIN rcv buffers 264 Min 1st chain segment 0
FILL rcv buffers 264 Min transmit length 0
ADD rcv buffers 328 Dev xmt header size 0


-- EWB Device Information (cont) 20-FEB-2008 08:40:23 --

Put rcv ptr/index 00000000 Get rcv ptr/index 00000144
Put xmt ptr/index 000001E9 Get xmt ptr/index 000001E9
Put cmd ptr/index 00000083 Get cmd ptr/index 00000000
Put uns ptr/index 00000008 Get uns ptr/index 00000000
Put smt ptr/index 00000000 Get smt ptr/index 00000000
RBufs owned by dev 327 Rcv packet limit 128
XEnts owned by dev 0 XEnts owned by host 512
CEnts owned by dev 0 Transmit timer 0
UEnts owned by dev 0 Control timer 0
SEnts owned by dev 0 Periodic SYSID timer 675
Current rcv buffers 327 Ring unavail timer 0
Rqst MAX rcv buffers 256 USB timer 48
Rqst MIN rcv buffers 128 Receive alignment 0
Curr MAX rcv buffers 264 Receive buffer size 1518
Curr MIN rcv buffers 264 Min 1st chain segment 0
FILL rcv buffers 264 Min transmit length 0
ADD rcv buffers 328 Dev xmt header size 0



Regards,
Mark

[MarkOfAus]

Robert,

"I would tend to agree with Volker that EWB has jumbo frame support enabled."

And so would I! :-)
The problem is when both were set to jumbo, the buffer sizes were as they are now. When one is changed, the buffer sizes remain the same. Ergo, it would seem buffer sizes have nothing to do with jumbo frames?


"The original Ethernet specification has a lower limit of 1500 bytes/frame (see the actual specification, for the precise contents).

Gigabit Ethernet permits so-called "Jumbo frames", up to 9000 bytes in size."

That is why, when I removed jumbo frames from the lan connected device because of switch incompatibility, I expected to notice the buffer change.


Regards,
Mark.

[MarkOfAus]

I apologise for late replies, but ITRC has been playing silly buggers for the last 24 hours; at least in my "end of town".

[Volker Halle]

Mark,

as the system has not been rebooted after disabling Jumbo frames on EWB, I would expect that SCACP> SHOW LAN still reflects the setting seen at boottime.

SCACP> SHOW CHANNEL should show you the current buffer size being used and that should be the same value as for EWA.

SCA will always try to use the biggest possible buffer size on the channel between any 2 nodes and it will also reduce the size, if the equipment on the LAN path does not support larger buffer sizes.

You can see the current, remote, local and negotiated buffer sizes with SDA:

$ ANAL/SYS
SDA> SHOW PORT
...
SDA> SHOW PORT/ADDR=pe_pdt/CHAN/VC

Look for Active Channel and 'Buffer Size' on the righthand side of the display.

Volker.

[Volker Halle]

Mark,

interestingly, the EWB counter for Jumbo transmits is 44, whereas the same counter for EWA is 0. I doubt that EWA is actually sending or receiving Jumbo frames.

What's the setting of NISCS_MAX_PKTSZ ?

Volker.

[Wim Van den Wyngaert]

Note for the SCA part in the enclosure :

EWA
Maximum buffer size 1498

EWB
Maximum buffer size 8998

Wim

[Wim Van den Wyngaert]

I know nothing about Jumbo's but
EWB also has a low number of collisions.

It tried to use jumbo's but failed ?

Is the stuff between the cluster nodes jumbo ready ?

Wim

[MarkOfAus]

Volker,

The "SHOW PORT/ADDR=81F93830/CHAN/VC" shows
buffer sizes as:

EWA
Current: 1426
Remote: 1426
Local: 1426
Negotiated: 1426

EWB
Current: 564
Remote: 8120
Local: 8120
Negotiated: 8120


In this "cluster", there is a one-to-one connection, so ewa <-> ewa and ewb <-> ewb.
The output is a little hard to decipher, given there is listed eia<->ewb, ewb<->eia, eia<->ewb, ewb<->ewb, ewb<->eia, eia<->eia, ewa<->ewa.

Regards,
Mark.

[MarkOfAus]

Volker,

"interestingly, the EWB counter for Jumbo transmits is 44, whereas the same counter for EWA is 0. I doubt that EWA is actually sending or receiving Jumbo frames."

I think that is correct. Why, well I am not sure. I figure that jumbo frames should be handled ok by a connection that is basically a cross-over connection, albeit over fibre. So, ewa should be sending/receiving jumbo frames. Ewb, on the other hand, has switches inbetween it, each of which is not configured for jumbo frames (vms can send them, the switch will kill them before hitting the stack).


"What's the setting of NISCS_MAX_PKTSZ ?"

It is 8192.
SYSGEN> SHOW NISCS_MAX_PKTSZ
Parameter Name Current Default Min. Max. Unit Dynamic
-------------- ------- ------- ------- ------- ---- -------
NISCS_MAX_PKTSZ 8192 8192 576 9180 Bytes

[MarkOfAus]

Wim,

" Note for the SCA part in the enclosure :

EWA
Maximum buffer size 1498

EWB
Maximum buffer size 8998
"

Ok, but what does this mean? Are they adjustable?


" I know nothing about Jumbo's but
EWB also has a low number of collisions.

It tried to use jumbo's but failed ?"

Yes, presuming our network people told me the correct information, the switches on which ewb is communicating are not jumbo enabled.


"Is the stuff between the cluster nodes jumbo ready ?"

Not for ewb, but I assumed for ewa, which is communicating through a fibre link, so essentially is a direct connection to the other server.


Regards,
Mark

[Volker Halle]

Mark,

maybe your network is not like you think it is...

To find out, which LAN interface in this cluster is able to talk with which other LAN interface on other nodes in this cluster, try this:

$ SET TERM/WIDTH=132
$ MC SCACP SHOW CHANNEL

This will show you all possible SCA communication channels, that exist in the local node. Look at those channels with Channel State 'Open'.

A channel is formed between a local LAN adapter (running SCA protocol) and all other LAN adapters in the cluster, which receive/send the SCA Hello multicast message.

If the channel state is 'Clsd', there once was a network path, but it's not available anymore (in such a case, I also see Buffer Size 564 at the channel level). If there has been network re-configuration activity during the lifetime of this node, you can expect to see various closed channels, but you also can see the CH open and close times.

Before any further activity, make sure that there are at least 2 working channels between any two nodes in your cluster. This would prevent problems, if anything happens to any of your LANs.

Note that the Maximum buffer size for SCA in the SDA output matches the SCACP SHOW LAN buffer sizes:

EIA2 60-07 (SCA) Maximum buffer size 1498
EWA2 60-07 (SCA) Maximum buffer size 1498
EWB10 60-07 (SCA) Maximum buffer size 8998

This is at least consistent.

It also looks like SCA on the LAN adapter EWB has been stopped and started in the running system. Normally SCA would be the first protocol started during boot.

Volker.

[Richard Stockdale]

The LAN drivers always have jumbo frames enabled, which are 9018 byte max for Broadcom devices, 8192 bytes for Intel Gigabit devices.

The LANCP SET DEV/JUMBO or the LAN_FLAGS settings only changes what VCI applications are told is the max buffer size. And this is enforced on transmit by the LAN drivers.

You can do an SDA SHOW LAN/VCI to see the current state of the VCI applications and the buffer size.

If you change LAN_FLAGS or do a LANCP SET DEV/[NO]JUMBO, applications won't pick up the change unless you stop and restart them on the device. So you have to SCACP STOP LAN devname then START LAN devname to change PEDRIVER settings on a device.

The best way to see usage of jumbo frames is LANCP SHOW DEV/INTERNAL_COUNTERS so you can see what jumbo transmits and receives have been done.

Note that if the number of jumbo transmits is small, like 44 as described, it is probably because PEDRIVER is doing size probing and not getting a response on the size probe.

The 'buffer size' in SCACP is correct. Note that it is limited by NISCS_MAX_PKTSIZE so even if the LAN device supports 9018 byte packets, PEDRIVER only goes up to NISCS_MAX_PKTSIZE less some header size.

Also, LANCP SHOW DEV/CHAR shows the device buffer size reported for QIO users. This was never changed when jumbo frames were implemented to avoid affecting existing QIO applications. QIO applications can send/receive jumbo frames regardless of the setting of LAN_FLAGS or SET DEV/JUMBO.

The LAN_FLAGS and SET DEV/JUMBO exists for VCI applications. Some like PEDRIVER can dynamically figure out the max packet size usable. TCP/IP can not, hence the existence of this jumbo flag setting.

- Dick

[MarkOfAus]

Volker,

"maybe your network is not like you think it is..."

And I am starting to think you are correct... :-)

"To find out, which LAN interface in this cluster is able to talk with which other LAN interface on other nodes in this cluster, try this:

$ SET TERM/WIDTH=132
$ MC SCACP SHOW CHANNEL

This will show you all possible SCA communication channels, that exist in the local node. Look at those channels with Channel State 'Open'."

See attached output.
There was a bit of plugging/unplugging when I moved one of the servers to a remote location, perhaps this explains the eia<->ewb , though I doubt it.


"Note that the Maximum buffer size for SCA in the SDA output matches the SCACP SHOW LAN buffer sizes:

EIA2 60-07 (SCA) Maximum buffer size 1498
EWA2 60-07 (SCA) Maximum buffer size 1498
EWB10 60-07 (SCA) Maximum buffer size 8998

This is at least consistent.
"

But again it leads to the question of why? Why is ewb greater than ewa when they are the same speed?

"It also looks like SCA on the LAN adapter EWB has been stopped and started in the running system. Normally SCA would be the first protocol started during boot."

I can't get anything past you. :-)

I stopped ewb (scacp stop lan ewb) to test throughput on ewa. eia was the backup (10/100) during this test.

Regards
Mark

[Richard Stockdale]

>>But again it leads to the question of why? Why is ewb greater than ewa when they are the same speed?

Speed is of no consequence other than whether the LAN device is capable of supporting jumbo packets. The size is determined by the results of PEDRIVER size probing and, of course, whether the LAN device was enabled for jumbo frames when PEDRIVER started on the device.

- Dick

[MarkOfAus]

Dick,

"You can do an SDA SHOW LAN/VCI to see the current state of the VCI applications and the buffer size."

See attached sho lan/vci output. I couldn't find any reference to buffers.

"Note that if the number of jumbo transmits is small, like 44 as described, it is probably because PEDRIVER is doing size probing and not getting a response on the size probe."

This was on the device connected to the lan via switches incapable (read: cisco) of handling jumbo frames. So, as you say, the 44 was probably it probing until I disabled jumbo frames (and then stopped the lan device & restarted it).

Note, that with jumbo frames enabled on ewb AND the switches it was connected to at both ends incapable of jumbo frames, it still became the prefered path during a copy operation even though its priority was lower than ewa. Hence this discussion.


"The 'buffer size' in SCACP is correct. Note that it is limited by NISCS_MAX_PKTSIZE so even if the LAN device supports 9018 byte packets, PEDRIVER only goes up to NISCS_MAX_PKTSIZE less some header size.
"

So does this mean it is better to increase NISCS_MAX_PKTSIZE to, say, 10000?

Regards,
Mark.

[MarkOfAus]

Dick,

" >>But again it leads to the question of why? Why is ewb greater than ewa when they are the same speed?

Speed is of no consequence other than whether the LAN device is capable of supporting jumbo packets. The size is determined by the results of PEDRIVER size probing and, of course, whether the LAN device was enabled for jumbo frames when PEDRIVER started on the device.
"

Ok, so it looks like ewa is not using jumbo frames but neither is ewb (I disabled them).

Why is not ewa negotiating jumbo frames? ewa is connected remotely to the other server via media converters and a fibre connection. Do I need a capable switch in-between? Seems odd if so.

More info.
Remote ewa is DEGXA-TB, local ewa is DEGXA-TB.
Remote ewa is 5703 LOM, local ewb is DEGXA-TB.


Regards
Mark

[Richard Stockdale]

>>See attached sho lan/vci output. I couldn't find any reference to buffers.

The field is "Max xmt size".

>>Note, that with jumbo frames enabled on ewb AND the switches it was connected to at both ends incapable of jumbo frames, it still became the prefered path during a copy operation even though its priority was lower than ewa. Hence this discussion.

It is hard to tell without being there. But priority wouldn't necessarily override lossiness of a channel when forming an ECS.

>>So does this mean it is better to increase NISCS_MAX_PKTSIZE to, say, 10000?

It would be if NISCS_MAX_PKTSIZE were not limited to 8192, or at least, I thought it was. I just checked a V73-2 system and it was limited to 9180. I'm not sure it it is limited to the max lookaside list size or really limited to 8192 bytes.

- Dick

[MarkOfAus]

Dick,
">>See attached sho lan/vci output. I couldn't find any reference to buffers.

The field is "Max xmt size"."

Got it, thanks.


">>Note, that with jumbo frames enabled on ewb AND the switches it was connected to at both ends incapable of jumbo frames, it still became the prefered path during a copy operation even though its priority was lower than ewa. Hence this discussion.

It is hard to tell without being there. But priority wouldn't necessarily override lossiness of a channel when forming an ECS."

So, I am at the point where the hardware seems to be at fault as, apparently, VMS has done its utmost to "tune" the connection.

Regards
Mark

[MarkOfAus]

Is there any way I can diagnose why ewa is not negotiating jumbo frames with the remote server? lancp sho dev ewa/intern shows that the frames are enabled, but doesn't give any clues as to why it isn't using them.

[Richard Stockdale]

>>Why is not ewa negotiating jumbo frames? ewa is connected remotely to the other server via media converters and a fibre connection. Do I need a capable switch in-between? Seems odd if so.

The switch doesn't matter, as far as whether PEDRIVER does size probing. It only matters that PEDRIVER sees the local and remote channels with a max buffer size larger than it is currently using. Then it will try to probe for a larger size since there is a possibility of increasing to a larger packet size.

>>More info.
>>Remote ewa is DEGXA-TB, local ewa is DEGXA-TB.
>>Remote ewa is 5703 LOM, local ewb is DEGXA-TB.

These devices are equivalent, all Broadcom BCM5703 Gigabit.

>>So, I am at the point where the hardware seems to be at fault as, apparently, VMS has done its utmost to "tune" the connection.

It is a bit difficult to trace what has gone on and figure out a solution off the cuff.

I'd suggest getting both sides set up the way they should be with jumbo frames enabled as you want and then go from there.

>>Is there any way I can diagnose why ewa is not negotiating jumbo frames with the remote server? lancp sho dev ewa/intern shows that the frames are enabled, but doesn't give any clues as to why it isn't using them.

LANCP SHOW DEV EWA/INT shows they are enabled. Then verify that PEDRIVER thinks they are enabled with SCACP SHOW LAN and SHOW CH/ALL and SHOW VC/ALL. Then LANCP SHOW DEV/INT would show jumbo transmits and receives. And verify the switch has jumbo frames enabled as well.

Escalating the problem to support might also help.

Regards,

- Dick

[MarkOfAus]

Dick,

"I'd suggest getting both sides set up the way they should be with jumbo frames enabled as you want and then go from there."

Ah, this was the key point. I had been concentrating on the machine that was shut down and moved, stopping and starting the lan on it via scacp.

Once I went to the other machine and stopped its ewa and restarted, it immediately picked up the new buffer size of 8120 and began sending/receiving jumbo packets.

Regards,
Mark.

[MarkOfAus]

The buffer size correlates to the frame size of the packets on the connection. 8120 indicates jumbo frames.

Jumbo frames only get enabled when both sides of the connection agree and the way to achieve this, if they get "confused", is to use scacp to stop and start that lan device after you have enabled jumbo packets via lancp or lan_flags.

Everyone gets 10 points because you are all very helpful and all answers lead to the correct solution.

Once again, thank you all for your time and efforts: Volker, Wim, Bob and Dick! You're champions.

Regards
Mark