Mac Address access in VLAN on Procurve switches

[pimpong]

Hi ,

I have a network created mainly with HP Procurve switches (4000, 8000, 5300xl) and a couple of cisco catalyst 2950.

I have some 15 VLANs created , and on one of them (no 13 !!) I experience the following:

Client computers can grab their IPs from the DHCP server (which also has a subinterface in the VLAN) and after a while they stop communicating with the network. No specific order for that to happen.

If I disable their LAN card and reenable it they get the IP address back from the DHCP server but they still do not communicate with the network.

They cannot even ping the DHCP server.

If I change their MAC Address with one of the MAC Addresses of the clients that have access to the network , they start functioning again. If I put back their original MAC Addresses they go quiet again.

The same is available if in the same port I plug a client that has access at that time. It continues to have access to the network.

I have done some packet dumping and I have learned that packets from the (non communicating) clients leave the switch and arrive at the destination , but answering packets leaving the destination to not reach back the clients.

I do not have STP , Flow-Control , port-security or mac-lockout enabled.

lockout-mac shows "Number of locked out MAC addresses = 0

I only disabled LACP on this very switch (5304xl) on whitch I run the debugging now.

Another problem stopping me from figguring out what's happening is that "logging IP-ADDRESS" shows absolutely no result on the syslog server.

Any other VLAN is working fine on the network , and if I put the clients (ports) in different VLANs they start communicating with the network.

Any ideas on this would really be helpful.

Thanks in advance.

[Matt Hobbs]

It is strange that it is only affecting the one VLAN. Is this VLAN cabled any differently to the other VLANs?

You need to be careful with the 4000/8000M's as they have a single mac-address table. They can only have one uplink going to the rest of your network.

http://www.hp.com/rnd/support/faqs/8000_4000_2424.htm#question27

When the problem occurs, check the arp table on your router, and also check the mac-address tables. Try a clear arp on your router to see if that temporarily helps.

When you say they cannot communicate with the network, is it only affecting communications going to other VLANs, or does it also affect traffic between two hosts in the same VLAN?

If you still can't find what is causing the problem, if you could attach your network map and some switch configurations it may help somewhere here.

[Mohieddin Kharnoub]

Hi

For better understanding do what Matt suggested of network map and configuration.

I still need to know few things :

1- Where you route between Vlans ?
2- Do you have any Access Control List enabled ?
3- Do you have any security device, Firewall, IDP , Procurve Access control server .....?
4- Is the DHCP scope of this vlan has a correct Router ID (Gateway points to the main router) ?

Your (non communicating) clients can get an DHCP ip, but can't ping the DHCP server, i think its a security issue.

Again Matt's suggestion about checking the arp table on the router, and the mac-address tables and clearing the arp on the router may help.

Good Luck !!!

[pimpong]

Thank you for your answers.

I checked and from all the 4000/8000 only one has an uplink from a 5300 and it forwards another uplink to a cisco 2950.
But it is not part of a loop and does not have the troubeling VLAN.

When the problem occurs the arp table on the router and the mac-address table on the switch are the same as when everything works. They contain addresses of both workstations that have connectivity and of those that have not.
Clearing each of the arp table on the router and the mac-addresses on the switch doesn't do any good.


Very good point Matt : only communications outside the VLAN are cut. Workstations can communicate with other workstations in the same VLAN at all times.

The VLANs are routed by a Linux router.

There are no ACLs enabled.
No Access Control Server, no IDS , only a firewall on the linux router , but when the firewall is set on ACCEPT the problem doesn't go away.

The GATEWAY set by DHCP is the correct one.
Even if I set the IP manually nothing good happens.

I thought that the non communicating clients can get DHCP addresses because they are generated differently from TCP packets whitch are not broadcasted in the network.


I attached a drawing of my testing topology.
I also copied some logs at the end of this post.

If I run Ethereal on a non communicating host all I "hear" are broadcast packets of any kind - DHCP , DNS , arp requests ... but no packets destined to the non communicating host.

I also tried setting a monitoring port on the switch and started Ethereal on a laptop plugged in that port ... In the mean time I pinged between a non communicating host and an host vrom another VLAN. The monitornig showed all kinds of packets except packets destined to the non communicating host.

Theese are the packets that get lost somewhere before they hit the final switch pecause I can "see" them on the generating host, on the router ... but not at the destination and not on the switch the non communicating host is connected to.

LOGS

192.168.13.233 - No Connectivity
192.168.13.230 - Full Connectivity

On 192.168.13.233:

ping 192.168.13.1
request timed out
request timed out
request timed out
...

Sniffing On the router 192.168.13.1(ns.crush.lan) while pinging the router from 192.168.13.233:

tcpdump -v -i eth4.13

10:55:09.991755 IP (tos 0x0, ttl 128, id 1481, offset 0, flags [none], length: 60) 192.168.13.233 > ns.crush.lan: icmp 40: echo request seq 42752

10:55:09.991824 IP (tos 0x0, ttl 64, id 1344, offset 0, flags [none], length: 60) ns.crush.lan > 192.168.13.233: icmp 40: echo reply seq 42752


Conclusion :

ICMP requests leaving 192.168.13.233 reach 192.168.13.1 but reply ICMP packets leaving 192.168.13.1 don't reach 192.168.13.233


Both non and communicating hosts can ping eachother.

On 192.168.13.233:

Ping 192.168.13.230
Reply from 192.168.13.230: bytes-32 time<1ms TTL=64
Reply from 192.168.13.230: bytes-32 time<1ms TTL=64

On 192.168.13.230:

Ping 192.168.13.233
Reply from 192.168.13.233: bytes-32 time<1ms TTL=64
Reply from 192.168.13.233: bytes-32 time<1ms TTL=64



Thanks again !

[Mohieddin Kharnoub]

Hi

You are saying that "The VLANs are routed by a Linux router" , have you tried to enable routing between Vlans on the 5304 switch ?
If you can attach the 5304 switch's configuration that will help better understanding for your configuration.

- Usually in such cases, routing done on the 5300, and a default static route will be added on the 5300 points to your linux router for internet access.

Good Luck !!!

[pimpong]

Do I have to disable meshing on all switches in order to enable Routing between VLANs ?
Because I cand find any routing option in the menus.

[Matt Hobbs]

You can't run routing and meshing at the same time.

When you have a non-communicating host you can see the ICMP packets reaching the router and being sent back from the router itself. Can you verify if those packets are actually being seen on the wire itself?

(What I am saying is although the router says it is sending the packet, do you know for sure that it is coming out?)

Set up a mirror-port to monitor the port the router is connected to. Or put a hub in between the router and the switch and connect your sniffer to the hub.

What are the IP address ranges you are using throughout the network and their subnet masks?

What if you simply change VLAN ID's thoughout the network... ? That's a long shot but may be worth trying in the troubleshooting process.