Cisco WAN QoS Troubleshooting Lab — Fix Voice Jitter, Packet Loss, and Slow Applications

Category: Network Troubleshooting | Cisco QoS | Enterprise WAN
Level: Intermediate to Advanced
Technology: Cisco IOS XE, LLQ, CBWFQ, Traffic Shaping, Policing
Lab Type: Real-World Enterprise Troubleshooting Simulation

Tutorial Overview

A network can be fully operational while business applications are performing poorly.

Users may experience choppy voice calls, freezing video meetings, and slow application response times even when router interfaces are up, routing is stable, and basic connectivity tests succeed.

In this tutorial, we will reproduce and troubleshoot a common enterprise WAN problem: a branch office generating more traffic than its contracted WAN bandwidth can support.

We will investigate the problem using Cisco CLI commands, examine the service provider's traffic policing behavior, implement Hierarchical Quality of Service (HQoS), and verify whether voice and business-critical traffic receive the intended forwarding treatment.

This is a simulated incident designed to reproduce realistic troubleshooting conditions. All device addresses, configurations, and sample outputs are for training.

Learning Objectives

After completing this tutorial, you will be able to:

1. Identify WAN congestion and its impact on applications.

2. Differentiate physical interface speed from contracted WAN bandwidth.

3. Investigate latency, jitter, throughput, and packet loss.

4. Verify Cisco interface statistics and QoS policies.

5. Configure DSCP marking and application classification.

6. Configure LLQ and CBWFQ traffic queues.

7. Implement parent shaping with child queuing.

8. Simulate service-provider traffic policing.

9. Troubleshoot QoS classification, priority drops, and bandwidth limitations.

10. Verify application performance before and after implementing QoS.

1. Real-World Problem Statement

The Network Operations Center receives an incident report:

Users at Branch-1 are experiencing poor VoIP quality, Microsoft Teams freezing, and slow access to Data Center applications. The issue becomes severe whenever the backup server transfers large amounts of data.

Initial monitoring reports:

• WAN interfaces are up.

• Routing is stable.

• Data Center destinations are reachable.

• No physical interface errors have been detected.

• WAN utilization increases significantly during backup activity.

The branch router has a 1 Gbps physical interface, but the service provider supplies only a 20 Mbps MPLS WAN connection.

Application Traffic Requirements

The traffic demand exceeds the available WAN capacity by 8 Mbps.

Troubleshooting Question: How can the network preserve voice and critical application performance when the available bandwidth is insufficient for all traffic?

2. Lab Network Topology

The lab uses three Cisco routers and four simulated application clients.

IP Addressing Plan

Lab Assumption: The Branch LAN uses a simple Layer 2 switch connecting all four test clients to R1. In a production deployment, voice, users, servers, and management equipment should normally be separated into appropriate VLANs and security zones.

3. Initial Device Configuration

Before investigating QoS, configure the IP addressing and basic routing.

3.1 Branch Router R1

enable
configure terminal

hostname R1

interface GigabitEthernet0/0/1
 description BRANCH-LAN
 ip address 192.168.10.1 255.255.255.0
 no shutdown
!
interface GigabitEthernet0/0/0
 description WAN-TO-ISP
 ip address 172.16.12.1 255.255.255.252
 no shutdown
!
ip route 10.20.20.0 255.255.255.0 172.16.12.2
!
end

3.2 Service Provider Router ISP-RTR

enable
configure terminal

hostname ISP-RTR

interface GigabitEthernet0/0/0
 description CONNECTION-TO-BRANCH-R1
 ip address 172.16.12.2 255.255.255.252
 no shutdown
!
interface GigabitEthernet0/0/1
 description CONNECTION-TO-DC-R2
 ip address 172.16.23.1 255.255.255.252
 no shutdown
!
ip route 192.168.10.0 255.255.255.0 172.16.12.1
ip route 10.20.20.0 255.255.255.0 172.16.23.2
!
end

3.3 Data Center Router R2

enable
configure terminal

hostname R2

interface GigabitEthernet0/0/0
 description WAN-TO-ISP
 ip address 172.16.23.2 255.255.255.252
 no shutdown
!
interface GigabitEthernet0/0/1
 description DATA-CENTER-LAN
 ip address 10.20.20.1 255.255.255.0
 no shutdown
!
ip route 192.168.10.0 255.255.255.0 172.16.23.1
!
end

3.4 Verify Connectivity

On R1:

show ip interface brief
show ip route
ping 172.16.12.2
ping 10.20.20.1

On R2:

show ip route
ping 192.168.10.1

What to Verify

• All relevant interfaces are up/up.

• Static routes are installed.

• R1 can reach R2.

• The Data Center server has the correct default gateway.

• The branch clients use 192.168.10.1 as their default gateway.

Do not proceed to QoS troubleshooting until basic Layer 3 reachability works correctly.

4. Simulate the Service Provider's 20 Mbps Limit

To reproduce a rate-limited WAN, configure a policer on the simulated ISP router.

This policy represents a service provider enforcing the contracted 20 Mbps bandwidth.

ISP-RTR Configuration

configure terminal

policy-map ISP-20M-POLICER
 class class-default
  police cir 20000000 conform-action transmit exceed-action drop
!
interface GigabitEthernet0/0/0
 service-policy input ISP-20M-POLICER
!
end

How the Policer Works

The provider-facing ingress policer evaluates traffic received from Branch R1.

• Traffic conforming to the configured rate and burst allowance is transmitted.

• Nonconforming traffic is dropped.

• The router records applicable policing statistics.

The policer uses a token-bucket mechanism, so actual drop behavior depends on the configured or default burst allowance. It does not simply drop exactly 8 Mbps at every instant.

Lab Note: The policing syntax and ingress-policy support must be validated against the simulated platform. Do not apply this artificial service-provider policer to an actual production circuit.

Verify Provider Policing

show policy-map interface GigabitEthernet0/0/0

Check the configured CIR and the conform/exceed statistics.

5. Generate Application Traffic

Use four test clients in the Branch LAN and an iperf3 server in the Data Center.

The objective is to generate approximately 28 Mbps of offered traffic.

Step 1 — Start Data Center Test Servers

On 10.20.20.10, start four iperf3 server instances on separate ports:

iperf3 -s -p 5201 &
iperf3 -s -p 5202 &
iperf3 -s -p 5203 &
iperf3 -s -p 5204 &

Step 2 — Generate Voice-Class Test Traffic

On client 192.168.10.11:

iperf3 -c 10.20.20.10 -p 5201 -u -b 2M -t 90

Step 3 — Generate Video-Class Test Traffic

On client 192.168.10.12:

iperf3 -c 10.20.20.10 -p 5202 -u -b 6M -t 90

Step 4 — Generate Business-Class Test Traffic

On client 192.168.10.13:

iperf3 -c 10.20.20.10 -p 5203 -u -b 5M -t 90

Step 5 — Generate Backup-Class Test Traffic

On client 192.168.10.14:

iperf3 -c 10.20.20.10 -p 5204 -u -b 15M -t 90

Run these tests concurrently to create the intended aggregate load.

Important: These are synthetic UDP traffic streams used to represent application demand. They are not actual VoIP, video, or enterprise application sessions. UDP packet loss and jitter measured by the test should not be treated as identical to a real application's user experience.

Expected Observation Before QoS

With the 20 Mbps provider policer active and 28 Mbps of offered traffic, the ISP router may report increasing exceed-drop counters.

The exact number of drops depends on the traffic generators, policing burst allowance, packet sizes, and duration.

6. Troubleshoot the Issue Before Applying QoS

Step 1 — Check Interface Status

On R1:

show interfaces GigabitEthernet0/0/0

What to Verify

• Interface is up/up.

• Physical speed is correct.

• Input and output errors are absent or understood.

• Traffic load is significant.

The interface can operate at 1 Gbps even though the provider enforces a 20 Mbps service limit.

Step 2 — Check for an Existing QoS Policy

show policy-map interface GigabitEthernet0/0/0

Before remediation, the branch WAN egress interface should have no shaping/queuing service policy for this lab.

Step 3 — Check Provider Drops

On ISP-RTR:

show policy-map interface GigabitEthernet0/0/0

What to Verify

• The 20 Mbps policer is active.

• Conforming traffic is being transmitted.

• Exceed drops increase when offered traffic exceeds the provider profile.

Step 4 — Measure Latency

On R1:

ping 10.20.20.10 repeat 20

Compare RTT results with and without the bulk traffic test.

Ping is useful as a supporting measurement, but cannot by itself establish the exact source of congestion.

Step 5 — Check Test Application Results

Review the iperf3 results for the four streams.

Record:

• Sender throughput.

• Receiver throughput.

• UDP jitter.

• Lost datagrams.

• Packet-loss percentage.

Initial Troubleshooting Conclusion

If interface health is good, the ISP policer reports exceed drops, and application performance deteriorates during heavy traffic, the evidence supports a WAN capacity and traffic-management problem.

The next step is to classify traffic and implement local shaping with application-aware scheduling.

7. Configure DSCP Marking on Branch R1

Before creating the WAN queuing policy, the router must identify the different application traffic classes.

For this controlled lab, the traffic generators are identified by their source IP addresses.

Step 1 — Create Classification ACLs

configure terminal

ip access-list extended ACL-VOICE-SOURCE
 permit ip host 192.168.10.11 any
!
ip access-list extended ACL-VIDEO-SOURCE
 permit ip host 192.168.10.12 any
!
ip access-list extended ACL-BUSINESS-SOURCE
 permit ip host 192.168.10.13 any
!
ip access-list extended ACL-BACKUP-SOURCE
 permit ip host 192.168.10.14 any
!

Production Design Note: Matching all traffic from a host is convenient for this lab but is generally insufficient for production application classification. A real IP phone can generate signaling, management, and bearer traffic with different QoS requirements. Production policies should validate the actual traffic type and trust boundary.

Step 2 — Create Class Maps

class-map match-any CM-VOICE-SOURCE
 match access-group name ACL-VOICE-SOURCE
!
class-map match-any CM-VIDEO-SOURCE
 match access-group name ACL-VIDEO-SOURCE
!
class-map match-any CM-BUSINESS-SOURCE
 match access-group name ACL-BUSINESS-SOURCE
!
class-map match-any CM-BACKUP-SOURCE
 match access-group name ACL-BACKUP-SOURCE
!

Step 3 — Configure the Marking Policy

policy-map PM-MARK-LAN
 class CM-VOICE-SOURCE
  set dscp ef
 !
 class CM-VIDEO-SOURCE
  set dscp af41
 !
 class CM-BUSINESS-SOURCE
  set dscp af21
 !
 class CM-BACKUP-SOURCE
  set dscp cs1
!

Step 4 — Apply Marking on LAN Ingress

interface GigabitEthernet0/0/1
 service-policy input PM-MARK-LAN
!
end

Marking Result

Verify Marking

show policy-map interface GigabitEthernet0/0/1

What to Verify

• Each test source matches its intended marking class.

• Class counters increment as the corresponding test runs.

• The configured set dscp action is correct.

Where available, verify the resulting packet markings using captures or platform-specific QoS counters rather than relying only on class-match counts.

8. Configure Hierarchical QoS on Branch R1

Now configure a parent shaping policy and a child queuing policy.

Step 1 — Create DSCP-Based Queuing Classes

configure terminal

class-map match-any CM-VOICE
 match dscp ef
!
class-map match-any CM-VIDEO
 match dscp af41
!
class-map match-any CM-BUSINESS
 match dscp af21
!

Step 2 — Create the Child Queuing Policy

policy-map PM-WAN-CHILD
 class CM-VOICE
  priority 2000
 !
 class CM-VIDEO
  bandwidth 6000
 !
 class CM-BUSINESS
  bandwidth 5000
 !
 class class-default
!

Understanding the Child Policy

Voice

The priority 2000 command provides LLQ priority treatment with a configured 2 Mbps rate.

Video

The bandwidth 6000 command provides a minimum 6 Mbps bandwidth allocation during congestion.

Business Applications

The bandwidth 5000 command provides a minimum 5 Mbps bandwidth allocation during congestion.

Default Traffic

The remaining traffic uses the default class according to the applicable scheduling implementation and available capacity.

The minimum bandwidth allocations are not hard maximum traffic limits.

Step 3 — Create the Parent Shaping Policy

policy-map PM-WAN-PARENT
 class class-default
  shape average 20000000
  service-policy PM-WAN-CHILD
!

This configures the parent shaper for an average rate of 20 Mbps.

Step 4 — Apply the Parent Policy

interface GigabitEthernet0/0/0
 description WAN-TO-ISP
 service-policy output PM-WAN-PARENT
!
end

The final design is:

This is a conceptual hierarchy. The actual child scheduler services all eligible queues rather than forwarding all default traffic only after the other classes have completely finished.

Platform Note: The commands represent a common IOS XE routing-platform MQC model. Validate hardware features, admission limits, and accepted policy syntax for your actual platform and release.

9. Verify the Hierarchical QoS Configuration

Step 1 — Verify Class Maps

show class-map

Confirm the application classes and DSCP match criteria.

Step 2 — Verify QoS Policies

show policy-map

Confirm:

• Voice priority rate: 2 Mbps.

• Video minimum bandwidth: 6 Mbps.

• Business minimum bandwidth: 5 Mbps.

• Parent shaping rate: 20 Mbps.

• Correct parent-child association.

Step 3 — Verify Active Interface Policy

show policy-map interface GigabitEthernet0/0/0

Illustrative Output

GigabitEthernet0/0/0

 Service-policy output: PM-WAN-PARENT

 Class-map: class-default
   Shape (average) cir 20000000

   Service-policy: PM-WAN-CHILD

   Class-map: CM-VOICE
     Match: dscp ef
     Priority: 2000 kbps
     Total drops: 0

   Class-map: CM-VIDEO
     Match: dscp af41
     Bandwidth: 6000 kbps

   Class-map: CM-BUSINESS
     Match: dscp af21
     Bandwidth: 5000 kbps

   Class-map: class-default
     Default scheduling

The above is abbreviated teaching output, not a captured device transcript.

What Needs to Be Verified?

Parent Policy

• Output shaping policy is attached correctly.

• CIR is configured as intended.

• The child policy appears beneath the parent.

Voice Class

• EF-marked packets match.

• Priority scheduling is active.

• Priority drops do not increase unexpectedly.

Video Class

• AF41 traffic matches.

• Minimum bandwidth allocation is correct.

• Packet loss and queue depth are acceptable.

Business Class

• AF21 traffic matches.

• The class receives service under congestion.

• Queue-drop counters are reviewed.

Default Class

• Bulk traffic is not incorrectly entering the voice or video class.

• Remaining traffic receives available transmission capacity.

10. Retest the Network After QoS Implementation

Run the same four concurrent iperf3 tests used before the QoS policy was applied.

This is important because a valid before-and-after comparison requires similar traffic conditions.

Metrics to Compare

These are expected trends, not guaranteed numerical results.

Verification on Branch R1

show policy-map interface GigabitEthernet0/0/0
show policy-map interface GigabitEthernet0/0/1
show interfaces GigabitEthernet0/0/0

Verification on ISP-RTR

show policy-map interface GigabitEthernet0/0/0

Important Observation

Shaping at exactly 20 Mbps does not guarantee zero provider-side drops.

The provider may account for additional overhead or use a different token-bucket burst allowance.

If the policer continues dropping packets, investigate the provider's accounting method and burst profile. A carefully tested shaping rate slightly below the contracted rate may be necessary.

Do not reduce the rate arbitrarily without checking its impact on critical applications.

11. Troubleshooting Common QoS Problems

Problem 1 — Voice Is Still Breaking

Symptom: Voice packet loss continues even after HQoS is configured.

Check:

show policy-map interface GigabitEthernet0/0/0
show policy-map interface GigabitEthernet0/0/1

Possible Causes

• Voice packets are not marked EF.

• Priority class does not match.

• Priority allocation is too small.

• Unexpected traffic is entering the priority queue.

• Drops occur on the return path or another network device.

Corrective Action: Validate classification first, then compare priority rate, offered load, and drop statistics.

Problem 2 — Provider Drops Continue

Symptom: The parent shaper is configured, but ISP policing drops remain high.

Possible Causes

• Incorrect shaping rate.

• Incompatible burst allowances.

• Encapsulation overhead.

• Incorrect policy direction.

• Different provider measurement intervals.

Corrective Action: Compare provider and router counters under the same test conditions, then adjust shaping parameters within the agreed service profile.

Problem 3 — Child Counters Remain Zero

Symptom: Parent shaping is visible, but child traffic-class counters do not increase.

Possible Causes

• DSCP markings are incorrect.

• Traffic does not pass through the intended policy.

• Child policy is not attached correctly.

• Platform counter limitations.

• Unsupported hierarchical feature combination.

Corrective Action: Check the ingress marking counters, then verify the complete egress policy hierarchy.

Problem 4 — High Latency Under Load

Symptom: Shaping works, but applications still experience high response times.

Possible Causes

• Sustained buffering.

• Excessive queue depth.

• Incorrect class assignment.

• Heavy congestion.

• Application or endpoint limitations.

Corrective Action: Examine offered load, queueing delay, drops, and bandwidth allocations. Avoid increasing queue limits without evaluating latency.

Problem 5 — Backup Throughput Is Lower

Symptom: Backup transfers take longer after QoS implementation.

Explanation

This can be expected.

When critical applications are active, backup traffic receives a smaller share of available bandwidth.

QoS intentionally favors business-critical traffic over bulk data transfers.

A longer backup completion time is not automatically a QoS failure.

12. Advanced Troubleshooting — What If QoS Is Not the Root Cause?

Not every network performance issue is caused by congestion.

If QoS is correctly configured and the problem remains, investigate:

Physical Interface Problems

Check CRC errors, discards, duplex, and link stability.

Routing Problems

Check forwarding paths, route changes, and asymmetric routing.

Application Problems

Check server CPU, memory, application processing, and storage performance.

WAN Provider Problems

Check transport latency, provider packet loss, and congestion inside the provider network.

Return-Path QoS

A policy applied to Branch R1's outgoing WAN interface does not automatically control Data Center-to-Branch traffic.

MTU and Encapsulation

Check whether encapsulation overhead, fragmentation, or MTU mismatches are affecting traffic.

Recommended Troubleshooting Sequence

13. Production Change and Rollback Considerations

Before applying QoS to a production WAN:

1. Save a backup of the current configuration.

2. Record baseline performance and QoS counters.

3. Confirm the physical interface and contracted CIR.

4. Check existing service policies.

5. Verify available bandwidth and QoS hardware resources.

6. Schedule the configuration change through the appropriate change process.

7. Prepare rollback commands.

8. Monitor application performance during and after implementation.

Example Rollback

If the new QoS policy causes an unexpected impact:

configure terminal

interface GigabitEthernet0/0/0
 no service-policy output PM-WAN-PARENT
!
interface GigabitEthernet0/0/1
 no service-policy input PM-MARK-LAN
!
end

This removes the two lab policies from their interfaces.

In production, restore the original policy attachments and configuration from the approved pre-change backup rather than assuming that removing a policy always restores the original behavior.

14. Hands-On Lab Exercises

Exercise 1 — Identify Provider Policing

Generate 28 Mbps of traffic without branch shaping.

Verify whether the simulated ISP policer reports drops.

Expected Learning: Understand why physical interface speed and contracted service bandwidth are different.

Exercise 2 — Configure Voice Priority

Apply the child queuing policy and verify that EF-marked traffic matches the voice class.

Expected Learning: Understand LLQ scheduling and priority traffic classification.

Exercise 3 — Test Incorrect DSCP Marking

Change the test voice traffic marking from EF to CS0.

Observe how traffic classification and queuing change.

Expected Learning: Understand why correct packet marking is essential.

Exercise 4 — Reduce the Shaping Rate

In a controlled lab, temporarily change the parent shaping rate from 20 Mbps to 15 Mbps.

Compare queueing behavior, traffic throughput, and loss.

Expected Learning: Understand how shaping rate affects available application bandwidth and queue depth.

Exercise 5 — Increase Backup Traffic

Increase synthetic backup traffic while maintaining voice and video traffic.

Observe priority-class behavior, default-class counters, and packet loss.

Expected Learning: Understand how QoS protects selected traffic during overload.

15. Tutorial Summary

In this tutorial, we reproduced and investigated a common enterprise WAN congestion problem.

The branch generated approximately 28 Mbps of traffic over a service limited to 20 Mbps, causing the simulated provider policer to drop nonconforming traffic.

We then configured:

• DSCP marking for application traffic.

• LLQ for voice.

• CBWFQ for video and business applications.

• Parent shaping at the WAN service rate.

• Child queueing for application-aware congestion management.

Finally, we verified the QoS hierarchy, traffic-class counters, provider policing behavior, and synthetic application performance.

Final Takeaway

A successful ping does not mean the network is performing correctly.

Effective troubleshooting requires understanding actual traffic demand, WAN capacity, latency, jitter, packet loss, QoS classification, queue scheduling, and service-provider behavior.

A properly engineered QoS policy does not eliminate congestion. It ensures that critical traffic receives appropriate treatment when network resources are limited.

Continue Learning with Network Guide Academy

Build practical Cisco networking skills through real-world troubleshooting scenarios, detailed lab guides, complete device configurations, and verification exercises.

Explore CCNA, CCNP, enterprise networking, routing, switching, security, and network automation.

Website: https://networkguideacademy.com/

Learn Networking. Build Skills. Troubleshoot with Confidence.

Technical References

• Cisco — QoS Hierarchical Scheduling

• Cisco — Understanding Bandwidth and Priority Commands

• Cisco — QoS Packet Policing