Reduce number of queues?

[RicN]

In a switch with, say, eight queues on the interface ports and you are not using QoS at all or perhaps just use two QoS priorities (normal and high), would it be to any benefit to reduce the number of queues on the switch?

If I understand it correctly the switch allocates some amount of port buffer memory to each queue, and I am wondering if a large number of unused queues would actually decrease the switch forwarding capacity?

[Mohieddin Kharnoub]

Hi

Great way of thinking,

however,

I'm not sure if your question is same to :
Would be the Switches with 4 Queues faster than the ones with 8 ? :)

For my understanding, if the QoS been used, then its great to have 8 Queues so we can have 8 Levels even we are not using all of them.

And i can't find any reason for the allocated buffer to be a factor in reducing the Switching/Forwarding speed, since all been considered in the design.

Anyway, the Lower Priority Queues should go after the Higher priority one.

Good Luck !!!

[RicN]

Thank you for a good reply!

What I was after was something like this:

Assume that each port has a total of e.g. 100kb memory available to use as a buffer. If this 100kb is divided by eight queues then we would have a lower amount of buffer memory available per queue, than if we would have used only four queues.

If my network only uses for example 802.1p priorities 0 and 3 then six of the interface queues would always be empty. And that would lead to a greater risk to the buffers on the two active queues would be full and frames dropped.

If this is how it is working of course! :) That is why I am asking.

[Mohieddin Kharnoub]

Hi

I guess you have an important point of view here.

Basically, switches varies between each other in many Key features and Specifications, and Buffer is a MAJOR one.

I've read before in Juniper CoS how they size buffer fullness and the percentage of queue buffer fullness of high packets after which high packets are dropped.

Good Luck !!!

[Jeff Carrell]

"if my network only uses for example 802.1p priorities 0 and 3 then six of the interface queues would always be empty. And that would lead to a greater risk to the buffers on the two active queues would be full and frames dropped."


it doesn't work that way, when the switch starts its processing cycle, it looks (ahead) at the queues and starts processing based on what's in the queue...if there is nothing in the 1st queue, it moves to the next and so on...so if there is data in q3, then the switch sees that and starts process q3 using q7,q6,q5,q4 times and its q3 time, then q0 (as its next), then q2,q1 times are used still processing q0 (normal pri)...and then the process starts over again...

this is called weighted round robin (WRR) queuing...very efficient use of time...no queue gets all the cycle of time and every queue gets a percentage of the processing cycle...

the result, the switch is never "idle" between queues, it is always processing queues...


hth...jeff

[André Beck]

Hi RicN,

"capacity" is the word things revolve around here. Indeed there are traffic flows where the available buffer space in an egress queue can't be large enough, that's usually throughput-oriented bulk traffic. On the other hand, queues (especially priority queues) for realtime traffic *have* to be short, as it makes no sense to hold up packets in them forever and neither to starve other queues.

So I expect a switch vendor to supply detailed information as to how the hardware in question is implemented regarding its QoS features, how the memory is split between ingress and egress queues, how the dequeueing at the egress is scheduled, how the triggers for WRED, ECN and tail drop are wired (as well as how 802.3 PAUSE kicks in) and what is to be tuned for which traffic class. I haven't yet read up on what HP provides here (didn't have a real QoS-sensitive deployment with ProCurves so far). What I'd expect is something akin to the Cisco 3560/3750 class docs, where you can essentially tune your heart out after having a tough time understanding the hardware Qos model (and they like to supply auto-macros and whole design guides that apply and describe the best practices).

There's no simple answer without knowing the details of a switch implementation and the default configuration. When there's really fixed chunks of memory wasted by unused egress queues and maximizing throughput as well as sustaining microbursts/overfeeds longer are the main goals, then "reducing the number of queues" could well help (but I'd actually expect to be able to assign the memory to queues on demand, sizing unused ones to zero in the process, and sizing the used queues so they best serve their respective traffic classes).

HTH,
Andre.

[Matt Hobbs]

On certain switches with smaller buffers, there can be a benefit in consolidating the queues if there are frequent Gig to 100 transfers. (Such as the 2800). If the small queues get filled, packets get dropped and transfer rates suffer.

[RicN]

Thank you all for your answers!


Jeff:

>this is called weighted round robin (WRR) queuing...
>very efficient use of time...no queue gets all the
>cycle of time and every queue gets a percentage of
>the processing cycle...

What you are describing is of course correct, but what I was wondering was not the actual scheduling of the queues, but the the memory allocations to them.

Andre, very good answer! And it would indeed be very interesting to know how Procurve has implemented the things you have mentioned. I wonder if the Procurve documentation give that much technical details?

Andre:

>but I'd actually expect to be able to assign the
>memory to queues on demand, sizing unused ones to
>zero in the process, and sizing the used queues
>so they best serve their respective traffic classes

I suppose that would be the key to it all. If that is how it is actually implemented then you would not have to worry about the not-used queues wasted memory resources.


Do anyone know by the way how much memory a port has available to buffers on a, say, 5400?

[André Beck]

Re RicN,

>>but I'd actually expect to be able to assign the
>>memory to queues on demand, sizing unused ones to
>>zero in the process, and sizing the used queues
>>so they best serve their respective traffic classes
>
>I suppose that would be the key to it all.
>If that is how it is actually implemented
>then you would not have to worry about the
>not-used queues wasted memory resources.

The yl/zl class is doing this to some extend via "GMB" (guaranteed minimum bandwidth) configuration (see "Port Traffic Controls" in the management guide). This should be read together with the 70 pages of QoS chapter in the advanced traffic management guide. For this platform, it's probably best to first configure the number of queues your QoS design really needs and then size them using GMB. Apparently it redistributes available memory to the queues as needed, even down to zeroing a queue (but I'm not sure, it might just define the WRR dequeueing schedule).

Now, just for the fun of it, compare to this chapter:

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750e_3560e/software/release/12.2_46_se/configuration/guide/swqos.html

Especially the part starting at "Queueing and scheduling overview"...

HTH,
Andre.

[Manfred Arndt]

Hi RicN,

Yes, for high performance deployments with link speed mismatches (e.g. Gig-to-100Mb) or oversubscribed links there is a benefit in reducing the number of queues if supported on the switch. This can help reduce packet loss due to microburst congestion caused by TCP data applications on the default priority queue.

On certain HP ProCurve switches you can choose 2, 4, or 8 queue modes to optimize how the memory buffer are assigned per queue. For example, if you are only using 3 priorities (say voice, video and data), then best performance will be achieved in the 4 queue mode.

To set the number of queues to 4 on the HP 5400/6200/3500 switch series use:

# qos queue-config 4-queues

Here is the link to the 5400 reviewers guide that describes the per-port buffer sizes (pg 30):

http://www.hp.com/rnd/pdfs/ProCurve_Switch_5400zl_Series_Reviewers_Guide.pdf

Cheers,
Manfred

[Manfred Arndt]

Hi RicN,

Sorry, almost forgot to respond to your second question...

Changing the number of queues does not affect the switch forwarding capacity.

What it does is change the amount of memory that is assigned to the various queues.

Deeper buffers can help smooth out short TCP data bursts that exceed the available egress link capacity.

Cheers,
Manfred

[RicN]

Hello Manfred and thank you for your reply!

This was the information I originaly wanted. If I understand you correctly then the high number of queues on for example the 5400 could in fact result in somewhat lower performance, i.e. more potential drops due to each queue is smaller?

Is it correct that the buffer sizes are fixed and not dynamicly relocated? For example, if queue 7 buffer is empty and never used, and all other queues are full - then the memory for queue 7 would not be used?

Can I use the bandwidth-min command to change the queue-sizes or does that command does something else?

Grateful for any more information, since this is very interesting.

[Manfred Arndt]

Hi RicN,

Correct, the buffer sizes are fixed and not dynamicaly relocated on the HP ProCurve 8200/6200/5400/3500 switch series. In your example, the memory in queue 7 is not used and held in reserve in case priority 7 traffic were to arrive when the system is congested.

This is important to insure that the switch performs well when forwarding traffic on multiple ports and/or queues simultaneously to prevent starvation that can cause applications to fail or time out.

The bandwidth-min command does not affect the queue sizes. What it does is adjust the queue servicing algorithm to insure that lower priority packets are not starved when there is lots of higher priority traffic.

If you disable the bandwidth-min command on an interface, you turn the queue servicing into strict priority.

# no interface bandwidth-min output

Bandwidth min is enabled by default, so that the switch does a weighted fair queueing.


Hope that helps,
Manfred

[RicN]

Thank you again for your answers!

Could it be said the a good general recommendation should be to tune the number of queue to the avarage number of priority values used on the network?

Or would it generaly be best to leave it alone?

As I read from the paper you supplied earlier it seems like the queue sizes would not really change except for the normal queue? That is, if I am using all eight queues then each of the not-normal queue would have around 70 kB of memory (for example queue for prio 7), but if I lower the amount of queues to four then the queue for prio-value 6 and 7 seems to have also only 70 kb (all memory "won" is taken to the normal queue). I assume that this could mean that if I later use both prio-values 6 and 7 then the queue for both of them would be of larger risk to be filled?

[Manfred Arndt]

Reducing the number of queues is a good recommendation for many/most customers.

The 4 queue model works well if you have or are thinking of using VoIP (and/or real-time video).

The 2 queue model works best for high-performance applications that require no QoS.

The high priority queues are intended for real-time latency or jitter sensitive applications, like VoIP and video conferencing. High priority queues are not recommended for use with bursty TCP protocols or video on demand.

Most real time applications send a low rate of smoothly distributed packets. For these, 70KB is much more than required and rarely at risk of being filled (even when supporting a large number of concurrent real-time sessions).

[RicN]

Thanks for your answers Manfred! I see from your profile that you work for HP and it is very nice to be able to get this kind of help on the forums!

I have some more questions in this matter:

1. If I am having, say, four queues and 802.1p priority level 6 and 7 is sharing the same queue - will there be any difference to the handling of frames INSIDE that queue, that is - will the prio 7 frames get any better treatment than frames with prio 6? Or does everything that ends up in the same queue use the same first in - first out ordinary sequence?

2. The buffer memory that makes up the actual queues on the switch, is that something that is physicaly bound to the port or is a part of the main RAM of the switch?

3. It was interesting to see the memory allocation sizes for the 5400-family, and I am wondering if this is different between different switch models? Does the older/cheaper models have less buffer memory attached to the ports?

4. Does it make any difference to what criterion I bind to the QOS value when speaking of performance (VLAN, incoming port, IP address, tcp/udp ports)? What I mean is, it should (?) be easier for the switch if I map a certain 802.1p priority to an entire VLAN, than if I say that traffic with a certain TCP destination port should have some priority.

Do I lose something if I make use of QOS markings depending of layer 4 information, which I guess, makes the switch forced to inspect each frame much deeper?

[Manfred Arndt]

1. If multiple 802.1p priorities are sharing the same queue - then everything that ends up in the same queue use the same first in - first out sequence.

2. The queue buffer memory is dedicated "high-speed" memory bound to the ports. This memory is significantly more expensive than the main CPU RAM, as it must keep up with the fabric packet rates (in excess of 10GbE).

3. The amount of buffer memory can vary dramatically between different switches and competitor models.

While most switches can keep up at full line rates with IXIA packet blaster tests where all ports are set at the same speed. Many will perform less than ideal under load in real-world situations with bursty TCP traffic (due to link-speed mismatch)...resulting in packet loss which can impact the aggregate network performance.

The 5400 Intelligent Edge family has significantly deeper buffer memory than most other models.

4. The 5400 family can keep up with full line rate traffic, independant of the number of packet classification and/or ACL rules you define. So go ahead and define QoS and security rules that meet your needs and don't worry about the impact on performance.

[RicN]

Many thanks again for the answers!

I think most of my questions regarding QoS has now solved!

>The 5400 family can keep up with full line
>rate traffic, independant of the number of
>packet classification and/or ACL rules you
>define.

That is of course very impressive by the 5400-family! Does it differ between different Procurve switches at what subjects you can base 802.1p at? That is, can a more ordinary edge switch (2800 series for example) also classify at layer 4 and do it with good performance?