Second bcm0 negotiate can disrupt S14settime/ntpdate.

[Chris Ellam]

We are seeing a problem with our configuration where the S14settime ntpdate request can fail due to the NTP Servers not being reachable. It appears that between S00inet and S14settime that Tru64 is not satisfied with the initial bcm0 negotiations and makes a further request and this interrupts the network connectivity and depending on the time taken to re-negotiate the ntpdate -b request can fail.

The SRM is V7.3-2, the OS is Tru64 V5.1B-4.

The interfaces are the DEGXA-SX (Fibre)on the ES47 and the DS25 on-board copper gigabit interface. The console variable ega0_mode can be set to either auto or 1000mbs_full_duplex and has no effect. The Cisco switch port is set to 1000mbs_full_duplex (and according to Cisco auto-negotiation if requested will be answered)
This is a Catalyst 6500 the modules are WS-X6724-SFP and WS-X6748-GE-TX.

I see on the console log that bcm0 negotiates twice, see below.

For clarity I must state that the ifconfig_n settings in rc.config has the variable has speed set to 2000.

We have also tried adding
lan_config -i bcm0 -a 0 -s 1000 -x 1 in /etc/inet.local.

However, with the above settings it appears that a second negotiation takes place between S00inet and S14settime.

Here is a boot log extract.

bcm0: Link down
bcm0 at pci2 slot 5
bcm0: DEGXA-TA Gigabit Ethernet (LOM), hardware address: 00-02-A5-C6-0D-E4
bcm0: Driver Rev = 1.0.29, Chip Rev = BCM5703_A2, Phy = Broadcom BCM5703 Integrated Copper
pci1 (primary bus:1) at nexus
pci3 (primary bus:3) at nexus
Created FRU table binary error log packet
kernel console: ace0
bcm0: Link up via auto-negotiation (1000 Mbps, full duplex, tx=0 rx=0 flow control)
dli: configured
NetRAIN configured.

Configuring network
hostname: vfgcwp01
ee0: auto negotiation disabled
ee0: 100 Mbps Full duplex
Loading LMF licenses
Combine OSF-BASE ALS-IL-2003NOV21-32 with OSF-BASE ALS-IL-2003NOV19-292
System error logger started
Binary error logger started
add host kauacws: gateway krhrouter
add host katrsws: gateway krhrouter
add net default: gateway 10.199.10.240
Setting kernel timezone variable
Setting the current time and date with ntpdate
bcm0: Link up via auto-negotiation (1000 Mbps, full duplex, tx=0 rx=0 flow control)

[Rob Leadbeater]

Hi Chris,

In my experience, with the Broadcom based cards it's always best to set both ends of the link to Auto Negotiate.

I'm not 100% sure if it will solve your problem, as I seem to recall similar issues on some ES47s I used to manage, however I'd start by setting the Cisco switch ports to Auto/Auto, rather than have them fixed...

Hope this helps,

Regards,

Rob

[Ivan Ferreira]

Maybe you can also move your ntpdate script so it starts a little later:

cd /sbin/rc3.d
mv S45xntpd S55xntpd

[Chris Ellam]

Thanks Rob,
Our testing has proved that it doesn't really matter what you set the SRM mode to. The issue here is that Tru64 requests a second negotiation and this one can take some time.

Thanks Ivan,
the possibility of moving the ntp scripts is being looked at. However, we wish to ensure that we do not introduce other problems.

We are looking for the root cause of our problem, we have full networking capabilities but sometime at a re-boot the ntpdate request fails and the system runs a while with an incorrect date.

We have a support request open and have had various responses which over looked that the second auto-negotiation took place and was logged.

We have asked that it be elevated to get a response from Tru64 engineering to tell us what the code is really doing.

I was hoping that a Tru64 Network engineer would be looking at the forum and could give us a qualified answer.

[Rob Leadbeater]

Hi Chris,

My point was that you shouldn't be fixing the Cisco switch port.

As the Alpha is set to auto negotiate, as shown by the extract from your boot log, then you should set the switch port to auto negotiate as well.

The second auto negotiate request you're seeing from the unix level is possibly coming about because the Cisco rejected the first...

Cheers,

Rob

P.S. Don't forget to assign points...

[Chris Ellam]

Our problem appears to have been associated with using the speed 2000 in the ifconfig definition, this appears to have forced the second negotiation that sometimes caused ntpdate to fail. So our solution is to remove the speed setting. In addition we added a second modified script like settime that occurs later in the start sequence added a delay and also made a ping to the ntp servers and logged the results.

We do not want negotiation to occur after boot time as link flaps delay our real-time communications when a negotiation occurs. But it appears that there is nothing you can do in Tru64 to avoid this. We must conclude that lan_config doesn't turn off auto-negotiation in the driver as documented and classify this as being a bug.

We would have liked the Tru6 bcm driver support to have explained our finding and concerns but this has not happened.

Many thanks to all those that replied.