slow/terible inter-vlan performance

[Gflex78]

Hi.

We have one 5900 setup fith difrent vlans.

When we do network speed tests between two server on  default vlan 1 we have great speed around 9 GB sec

When we do test to betwen two servers on difrent vlans for example from vlan 2 to vlan one we recive speed of around 200 MB sec

We are running version 7.1.045

1.#
2. version 7.1.045, Release 2422P01
3.#
4. sysname 5900-01
5.#
6. clock timezone Amsterdam add 01:00:00
7. clock protocol ntp
8.#
9. telnet server enable
10.#
11. irf mac-address persistent timer
12. irf auto-update enable
13. undo irf link-delay
14. irf member 1 priority 1
15. irf mode normal
16.#
17. dns server x.x.x.x
18.#
19. lldp global enable
20.#
21. loopback-detection interval-time 20
22.#
23. system-working-mode standard
24. fan prefer-direction slot 1 port-to-power
25. password-recovery enable
26.#
27.vlan 1
28.#
29.vlan 2
30.#
31.vlan 20
32. name VLAN20
33.#
34.vlan 30
35. name VLAN30
36.#
37.vlan 40
38. name VLAN40
39.#
40.vlan 50
41. name VLAN50
42.#
43.vlan 51
44. name VLAN51
45.#
46.vlan 60
47. name VLAN60
48.#
49.vlan 190
50. name VLAN190
51.#
52.vlan 210
53. name VLAN210
54.#
55. stp instance 0 priority 12288
56. stp bpdu-protection
57. stp global enable
58.#
59.interface NULL0
60.#
61.interface Vlan-interface1
62. ip address x.x.x.x x.x.x.x
63.#
64.interface Vlan-interface2
65. ip address dhcp-alloc
66.#
67.interface Vlan-interface20
68. ip address dhcp-alloc
69.#
70.interface Vlan-interface210
71. ip address dhcp-alloc
72.#
73.interface FortyGigE1/0/49
74. port link-mode bridge
75. port link-type trunk
76. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
77.#
78.interface FortyGigE1/0/50
79. port link-mode bridge
80. port link-type trunk
81. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
82.#
83.interface FortyGigE1/0/51
84. port link-mode bridge
85. port link-type trunk
86. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
87.#
88.interface FortyGigE1/0/52
89. port link-mode bridge
90. port link-type trunk
91. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
92.#
93.interface M-GigabitEthernet0/0/0
94. ip address x.x.x.x x.x.x.x
95.#
96.interface Ten-GigabitEthernet1/0/1
97. port link-mode bridge
98. port link-type trunk
99. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
100.#
101.interface Ten-GigabitEthernet1/0/2
102. port link-mode bridge
103. port link-type trunk
104. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
105. stp port bpdu-protection enable
106.#
107.interface Ten-GigabitEthernet1/0/3
108. port link-mode bridge
109. port link-type trunk
110. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
111. stp port bpdu-protection enable
112.#
113.interface Ten-GigabitEthernet1/0/4
114. port link-mode bridge
115. port link-type trunk
116. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
117. stp port bpdu-protection enable
118.#
119.interface Ten-GigabitEthernet1/0/5
120. port link-mode bridge
121. port link-type trunk
122. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
123. stp port bpdu-protection enable
124.#
125.interface Ten-GigabitEthernet1/0/6
126. port link-mode bridge
127. port link-type trunk
128. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
129. stp port bpdu-protection enable
130.#
131.interface Ten-GigabitEthernet1/0/7
132. port link-mode bridge
133. port link-type trunk
134. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
135. stp port bpdu-protection enable
136.#
137.interface Ten-GigabitEthernet1/0/8
138. port link-mode bridge
139. port link-type trunk
140. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
141. stp port bpdu-protection enable
142.#
143.interface Ten-GigabitEthernet1/0/9
144. port link-mode bridge
145. port link-type trunk
146. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
147. stp port bpdu-protection enable
148.#
149.interface Ten-GigabitEthernet1/0/10
150. port link-mode bridge
151. port link-type trunk
152. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
153. stp port bpdu-protection enable
154.#
155.interface Ten-GigabitEthernet1/0/11
156. port link-mode bridge
157. port link-type trunk
158. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
159. stp port bpdu-protection enable
160.#
161.interface Ten-GigabitEthernet1/0/12
162. port link-mode bridge
163. port link-type trunk
164. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
165. stp port bpdu-protection enable
166.#
167.interface Ten-GigabitEthernet1/0/13
168. port link-mode bridge
169. port link-type trunk
170. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
171. stp port bpdu-protection enable
172.#
173.interface Ten-GigabitEthernet1/0/14
174. port link-mode bridge
175. port link-type trunk
176. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
177. stp port bpdu-protection enable
178.#
179.interface Ten-GigabitEthernet1/0/15
180. port link-mode bridge
181. port link-type trunk
182. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
183. stp port bpdu-protection enable
184.#
185.interface Ten-GigabitEthernet1/0/16
186. port link-mode bridge
187. port link-type trunk
188. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
189. stp port bpdu-protection enable
190.#
191.interface Ten-GigabitEthernet1/0/17
192. port link-mode bridge
193. port link-type trunk
194. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
195. stp port bpdu-protection enable
196.#
197.interface Ten-GigabitEthernet1/0/18
198. port link-mode bridge
199. port link-type trunk
200. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
201. stp port bpdu-protection enable
202.#
203.interface Ten-GigabitEthernet1/0/19
204. port link-mode bridge
205. port link-type trunk
206. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
207. stp port bpdu-protection enable
208.#
209.interface Ten-GigabitEthernet1/0/20
210. port link-mode bridge
211. port link-type trunk
212. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
213. stp port bpdu-protection enable
214.#
215.interface Ten-GigabitEthernet1/0/21
216. port link-mode bridge
217. port link-type trunk
218. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
219. stp port bpdu-protection enable
220.#
221.interface Ten-GigabitEthernet1/0/22
222. port link-mode bridge
223. port link-type trunk
224. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
225. stp port bpdu-protection enable
226.#
227.interface Ten-GigabitEthernet1/0/23
228. port link-mode bridge
229. port link-type trunk
230. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
231. stp port bpdu-protection enable
232.#
233.interface Ten-GigabitEthernet1/0/24
234. port link-mode bridge
235. port link-type trunk
236. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
237. stp port bpdu-protection enable
238.#
239.interface Ten-GigabitEthernet1/0/25
240. port link-mode bridge
241. port link-type trunk
242. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
243. stp port bpdu-protection enable
244.#
245.interface Ten-GigabitEthernet1/0/26
246. port link-mode bridge
247. port link-type trunk
248. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
249. stp port bpdu-protection enable
250.#
251.interface Ten-GigabitEthernet1/0/27
252. port link-mode bridge
253. port link-type trunk
254. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
255. stp port bpdu-protection enable
256.#
257.interface Ten-GigabitEthernet1/0/28
258. port link-mode bridge
259. port link-type trunk
260. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
261. stp port bpdu-protection enable
262.#
263.interface Ten-GigabitEthernet1/0/29
264. port link-mode bridge
265. port link-type trunk
266. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
267. stp port bpdu-protection enable
268.#
269.interface Ten-GigabitEthernet1/0/30
270. port link-mode bridge
271. port link-type trunk
272. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
273. stp port bpdu-protection enable
274.#
275.interface Ten-GigabitEthernet1/0/31
276. port link-mode bridge
277. port link-type trunk
278. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
279. stp port bpdu-protection enable
280.#
281.interface Ten-GigabitEthernet1/0/32
282. port link-mode bridge
283. port link-type trunk
284. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
285. stp port bpdu-protection enable
286.#
287.interface Ten-GigabitEthernet1/0/33
288. port link-mode bridge
289. port link-type trunk
290. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
291. stp port bpdu-protection enable
292.#
293.interface Ten-GigabitEthernet1/0/34
294. port link-mode bridge
295. port link-type trunk
296. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
297. stp port bpdu-protection enable
298.#
299.interface Ten-GigabitEthernet1/0/35
300. port link-mode bridge
301. port link-type trunk
302. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
303. stp port bpdu-protection enable
304.#
305.interface Ten-GigabitEthernet1/0/36
306. port link-mode bridge
307. port link-type trunk
308. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
309. stp port bpdu-protection enable
310.#
311.interface Ten-GigabitEthernet1/0/37
312. port link-mode bridge
313. port link-type trunk
314. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
315. stp port bpdu-protection enable
316.#
317.interface Ten-GigabitEthernet1/0/38
318. port link-mode bridge
319. port link-type trunk
320. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
321. stp port bpdu-protection enable
322.#
323.interface Ten-GigabitEthernet1/0/39
324. port link-mode bridge
325. port link-type trunk
326. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
327. stp port bpdu-protection enable
328.#
329.interface Ten-GigabitEthernet1/0/40
330. port link-mode bridge
331. port link-type trunk
332. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
333. stp port bpdu-protection enable
334.#
335.interface Ten-GigabitEthernet1/0/41
336. port link-mode bridge
337. port link-type trunk
338. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
339. stp port bpdu-protection enable
340.#
341.interface Ten-GigabitEthernet1/0/42
342. port link-mode bridge
343. port link-type trunk
344. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
345. stp port bpdu-protection enable
346.#
347.interface Ten-GigabitEthernet1/0/43
348. port link-mode bridge
349. port link-type trunk
350. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
351. stp port bpdu-protection enable
352.#
353.interface Ten-GigabitEthernet1/0/44
354. port link-mode bridge
355. port link-type trunk
356. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
357. stp port bpdu-protection enable
358.#
359.interface Ten-GigabitEthernet1/0/45
360. port link-mode bridge
361. port link-type trunk
362. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
363. stp port bpdu-protection enable
364.#
365.interface Ten-GigabitEthernet1/0/46
366. port link-mode bridge
367. port link-type trunk
368. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
369. stp port bpdu-protection enable
370.#
371.interface Ten-GigabitEthernet1/0/47
372. port link-mode bridge
373. port link-type trunk
374. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210
375. stp port bpdu-protection enable
376.#
377.interface Ten-GigabitEthernet1/0/48
378. port link-mode bridge
379. port link-type trunk
380. port trunk permit vlan 1 to 2 20 30 40 50 to 51 60 190 210

[parnassus]

forty and so ports set to Port Type = Trunk? no Access Ports on the entire Switch?

[Gflex78]

yes there are no Access Ports on the switch becose its dedicated to hosting hyper-v server that have virtual machines on difrent subbnets.

Do you think that changing some port to a access port will increase the oerfomance?

[peyrache]

hello

could you post Display diagnostic output please

thanks

JY

[sdide]

Hi,

I cant solve the problem without more info, but i just wanted to state that we have a very similar setup, all trunks with loads of VLANs to vmware servers, only difference is our 5900 doesn't do any routing. All out routing is done on different platforms. But that should not matter. We get around 9Gbit/s regardsless if its inter-vlan or on the same vlan.  We have a mix of 5700s and 5900s around 24 units in total) they all have trunks only with many vlans on all ports. They all perform well. But - its all layer 2.

Please do a

display ip routing-table

and post the result - if the routing bewteen vlan 1 and 2 is done locally, its odd.

Regards

[Vince-Whirlwind]

Routing shouldn't be a problem as this is a wire-speed packet-forwarding switch.
Routing should not add any more latency than switching.

Maybe doublecheck how you can check whether wirespeed has been switched off. Or if there are some performance limitations (too many IP addresses, etc...).

Is inter-VLAN routing done on the switch, definitely not on the vSwitch?

Does your traceroute do anything funny?

Does the vFirewall do anything different if the packets are going inter-VLAN?

Does the server OS FW do anythiung differently if the pakctes are going inter-VLAN?