Troubleshooting QoS

Owned by richard.edmonstone
Last updated: Dec 02, 2019 by Scott Leishman
19 min read

Problem

Your QoS config isn't working the way you expect it to. Perhaps you can't seem to commit it or traffic isn't being handled as you'd expect.

Solution

Check your config

First, check that your config is valid. All QoS policies need a default profile to determine how traffic that doesn't match a particular class or subport should be handled. Similarly, each class and subport will also need a profile associated with it.

QoS policies now require an extra name node when configuring them, so if you see the following error try adding name before your policy's name:

vyatta@vm-8-1# set policy qos foo shaper profile default Configuration path: policy qos [foo] is not valid [edit] vyatta@vm-8-1# set policy qos name foo shaper profile default [edit] vyatta@vm-8-1#

 

If you are attempting to match based on priority, or remarking packets, ensure you have used the correct configuration as they are quite similar. For matching you need:

set policy qos name q shaper class 1 match on_priority dscp cs1

For remarking, you need to specify mark as well:

set policy qos name q shaper class 1 match remarking mark dscp cs1

If you are having issues with bandwidth, keep in mind the default behavior for all config nodes that can be configured with a bandwidth setting is to consume 100% of the parent node's available bandwidth. If you have multiple subports or classes and haven't explicitly assigned the bandwidth they are allowed to consume, then the first of these selected by the scheduler could use all of the available bandwidth.

Traffic classes need particular attention as they are processed in strict priority order. If maximum bandwidths are not associated with a traffic class, then lower priority traffic classes will be starved if there is sufficient traffic in the higher priority traffic classes to use up all available bandwidth.

Finally, when constructing matches, keep in mind that QoS only matches the outer headers visible. For features like tunnels or QinQ, it is only the outer values that can be matched on, not the internal values so the classification has to take this into account.

An example valid config is provided below for comparison:

Diagnosing issues

If you believe your config is good but are still having issues, there are several tools you can use to triage the problem. If you do use these, it's a good idea to attach the output to any tickets that might get raised.

Confirm you have attached your policy to an interface

It's possible to configure a policy without binding it to an output interface. In this situation, the policy doesn't actually do anything as it doesn't know which traffic it needs to be applied to. A simple way to determine if your policy has been bound is to search for the policy name in the interfaces dataplane config:

Show the queues associated with the policy

There is an op-mode show queuing command which will display the state of the traffic classes associated with all existing policies. It shows how much traffic has been sent to each traffic class, and how many packets have been dropped.

Give it an interface name and it will provide more details, showing the state of the queues and separating the stats into each class in the policy. Here we can see that all of the traffic is being sent to queue 0, in traffic class 0, in class 1:

These commands can help confirm that your policy is configured correctly and that traffic is being sent to the correct queue.

Show how much traffic has matched a class

There is a similar command show queuing class which shows how much traffic has been matched against the classes configured:

Providing the interface name allows only the classes associated with that interface to be shown, e.g. show queuing dp0p192p1 class.

This can help determine if the correct class is being matched.

These show commands are all derived from a vplane command that dumps the qos config and operational state. While this is unlikely to be as useful as the op-mode commands, it may be useful to provide this information when raising bugs.

The easiest way to access this is to provide the qos show command to the vplsh shell:

Observing received traffic rates

It may be useful to have some idea of how much traffic is being received by the router. The best way to do this is to use the monitor interface op-mode command which will display an interactive traffic monitoring utility.

Checking packet contents

If you need to check packet properties, such as the marking applied, it is possible to capture received packets with the traffic subcommand of monitor interface. If you are sending traffic at a high rate, it may be best to capture this output to a file, or use grep to search for the particular information you're interested in.

This output is generated by the tshark utility which it is also possible to run directly on an interface.

Useful Tools

There are several useful tools for generating traffic for testing QoS:

  • Spirent TestCentre can generate line rate traffic and is highly configurable, but quite complex and primarily accessed through a windows app.

  • ping can generate Mbits of traffic with -f flag which may be enough to test many configs. The -i flag sets the interval between packets which can help control the transmitted bandwidth and the -Q flag allows the TOS IP header field to be set (multiply DSCP values by 4 to calculate the corresponding TOS value).

  • iperf is a client/server network throughput analysis tool. It requires a server sink to be established at the receiving end of a link (ideally the other end of the link the QoS policy you are testing is on) and a client to connect to it through the router under test. It's very easy to configure the transmitted bandwidth, the DSCP values (-S) and provide throughput bandwidth calculations (-i).  It can't be used to send multiple streams simultaneously though with different settings.

  • Ostinato is GUI tool which can be used to send streams of traffic to a router, and designed to act a bit like wireshark in reverse.

 

How to enable QoS debug logging in the dataplane

Here is how to enable QoS debug logging in the dataplane:

  1. Enable debugging for the dataplane syslog facility with:

    vyatta@vyatta:~$ configure

    vyatta@vyatta# set system syslog global facility dataplane level debug

    vyatta@vyatta# commit

  2. Use vplsh to set the QoS debug bit in the dataplane's dp_debug flags variable:

    vyatta@vyatta:~$ /opt/vyatta/bin/vplsh-l -c "debug qos"

  3. Use vplsh to check which dataplane debug flags are set:

    vyatta@vyatta:~$ /opt/vyatta/bin/vplsh -l -c "debug"

    Debug 0x8013 init link nl_interface qos

    vyatta@vyattaT:~$

    The debug flags set by default are: init, link and netlink interface, it the example above the QoS debug flag (0x8000) is set.

  4. You can also use vplsh to clear the QoS debug flag like so (note the additional '-' in front of qos):

    vyatta@UUT:~$ /opt/vyatta/bin/vplsh -l -c "debug -qos"

    vyatta@UUT:~$ /opt/vyatta/bin/vplsh -l -c "debug"

    Debug 0x13 init link nl_interface

    vyatta@UUT:~$

  5. The QoS debug messages appear in /var/log/dataplane/vplane.log, or can be dumped using vplsh like so:

    vyatta@vyatta:~$ /opt/vyatta/bin/vplsh -l -c "log"

    EAL: TSC frequency is ~2693752 KHz

    EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !

    DATAPLANE: Dataplane version 2.2.0 - Copyright (C) 2012-2016 Brocade Communication Systems, Inc.

    DATAPLANE: 3 ports available (enabled mask 0x7)

    DATAPLANE: 1 core(s) available

    DATAPLANE: dp0s4 Link enabled

    DATAPLANE: dp0s4 Link up at 10000 Mbps, full duplex

    DATAPLANE: dp0s4: mcast HW Rx filters: Operation not supported

    DATAPLANE: dp0s3 Link enabled

    DATAPLANE: dp0s3 Link up at 10000 Mbps, full duplex

    DATAPLANE: dp0s3: mcast HW Rx filters: Operation not supported

    DATAPLANE: dp0s5 Link enabled

    DATAPLANE: dp0s5 Link up at 10000 Mbps, full duplex

    DATAPLANE: dp0s5: mcast HW Rx filters: Operation not supported

    QOS: PCP map not enabled, enabling

    QOS: DSCP map not enabled, enabling

    QOS: PCP map not enabled, enabling

    QOS: DSCP map not enabled, enabling

    QOS: DSCP map not enabled, enabling

    QOS: PCP map not enabled, enabling