rss logo

QoS on GNU/Linux with tc command

Tux logo

Intro

On GNU/Linux systems, quality of service can be set and managed using the tc command. According to wikipedia (source: https://en.wikipedia.org/), tc (for traffic control) is the user-space utility program used to configure the Linux kernel packet scheduler. In this article, we'll look at a few examples to show how we can use it.

As we will see, GNU/Linux traffic control isn't that simple… Let's see here how it works.

Main Commands

Limit download (ingress) rate

Note: Qdiscs on ingress traffic only provide policing without shaping. To shape ingress traffic, the IFB (Intermediate Functional Block) device must be used.

As you can see on the image below, the upload/egress traffic goes directly from eth0 to the internet, while the download/ingress traffic goes to eth0 through the ifb0 interface to be traffic shapped before returning to eth0.

GNU/Linux | ifb interface for qos explained

In order to create our ifb interface we need to load module and set interface up.

  • Load ifb module for one interface (numifbs=1):
root@host:~# modprobe ifb numifbs=1
  • Enable ifb0 interface:
root@host:~# ip link set dev ifb0 up
  • Create ingress:
root@host:~# tc qdisc add dev eth0 ingress handle ffff:
  • Forward all ingress traffic from eth0 to the ifb0 interface:
root@host:~# tc filter add dev eth0 parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0
  • And, for example, create ingress rules to limit all download traffic at 1Mb/s on eth0 (ifb0) interface:
root@host:~# tc qdisc add dev ifb0 root handle 2: htb default 1 root@host:~# tc class add dev ifb0 parent 2: classid 2:1 htb rate 1024kbit

Show real time rules

  • Install watch command:
root@host:~# apt install watch
  • For ingress (download) traffic:
root@host:~# watch -n 1 tc -s class show dev ifb0
  • For egress (upload) traffic:
root@host:~# watch -n 1 tc -s class show dev eth0

Reinit rules

If we want to reset/disable our QoS rules we have to type the following commands.

  • For ingress, ifb0 interface:
root@host:~# ip link set dev ifb0 down root@host:~# tc qdisc del dev eth0 root root@host:~# tc qdisc del dev ifb0 root
  • For egress, eth0 interface:
root@host:~# tc qdisc del dev eth0 ingress root@host:~# tc qdisc del dev ifb0 ingress

Practical example of QoS on a NAT router

Let's say we have a 1Gb/s Internet connexion and we want to share Bandwidth on theses three networks : Servers, Users and WiFi. To be efficient the QoS needs to be applied on approximatively 90% of total Bandwidth (900Mb/s in this case).

Bandwidth allocation

  • Here how we want to share:
    • Servers:
      • Download: max 900Mb/s, min 400Mb/s
      • Upload: max 900Mb/s, min 700Mb/s
    • Users:
      • Download: max 500Mb/s, min 200Mb/s
      • Upload: max 200Mb/s, min 100Mb/s
    • WiFi:
      • Download: max 200Mb/s, min 100Mb/s
      • Upload: max 100Mb/s, min 80Mb/s

Architecture

GNU/Linux | qos architecture
QoS architecture.

Script

  • Full script to apply QoS rules:
#!/bin/sh # load modules modprobe ifb numifbs=1 modprobe sch_fq_codel modprobe act_mirred #modprobe br_netfilter #in order to netfilter aware about bridge traffic modprobe act_connmark #https://unix.stackexchange.com/questions/155751/setting-traffic-class-on-return-packets #QoS reset for IFB in ifb0 ; do ip link set dev $IFB down done for IF in eth0 ifb0 ; do tc qdisc del dev $IF root 2> /dev/null > /dev/null tc qdisc del dev $IF ingress 2> /dev/null > /dev/null done for IFB in ifb0 ; do ip link set dev $IFB up done #if we want to disable qos run the script with stop argument if [ "$1" = "stop" ]; then echo "stop" iptables -F -t mangle iptables -X -t mangle exit 0 fi ############################### #UPLOAD (egress traffic) RULES# ############################### #RULES tc qdisc add dev eth0 root handle 1: htb default 10 tc class add dev eth0 parent 1: classid 1:1 htb rate 900mbit #here we set our max Bandwidth ##Servers (192.168.1.0 network) and default (if no matching case) tc class add dev eth0 parent 1:1 classid 1:10 htb rate 700mbit ceil 900mbit prio 1 ##Users (192.168.2.0 network) tc class add dev eth0 parent 1:1 classid 1:20 htb rate 100mbit ceil 200mbit prio 2 ##WiFi (192.168.3.0 network) tc class add dev eth0 parent 1:1 classid 1:30 htb rate 100mbit ceil 80mbit prio 3 #FILTERS ##Servers (192.168.1.0 network); handle 1 means mark 1 (see iptables rules) tc filter add dev eth0 parent 1:0 protocol ip prio 1 handle 1 fw classid 1:10 ##Users (192.168.2.0 network); handle 2 means mark 2 (see iptables rules) tc filter add dev eth0 parent 1:0 protocol ip prio 2 handle 2 fw classid 1:20 ##WiFi (192.168.3.0 network); handle 3 means mark 3 (see iptables rules) tc filter add dev eth0 parent 1:0 protocol ip prio 3 handle 3 fw classid 1:30 ## Martin Devera, author of HTB, then recommends SFQ for beneath these classes : tc qdisc add dev eth0 parent 1:10 handle 110: sfq perturb 10 tc qdisc add dev eth0 parent 1:20 handle 120: sfq perturb 10 tc qdisc add dev eth0 parent 1:30 handle 130: sfq perturb 10 ################################## #DOWNLOAD (ingress traffic) RULES# ################################## #RULES tc qdisc add dev ifb0 root handle 2: htb default 10 tc class add dev ifb0 parent 2: classid 2:1 htb rate 900mbit #here we set our max Bandwidth ##Servers (192.168.1.0 network) and default (if no matching case) tc class add dev ifb0 parent 2:1 classid 2:10 htb rate 400mbit ceil 900mbit prio 1 ##Users (192.168.2.0 network) tc class add dev ifb0 parent 2:1 classid 2:20 htb rate 200mbit ceil 500mbit prio 2 ##WiFi (192.168.3.0 network) tc class add dev ifb0 parent 2:1 classid 2:30 htb rate 100mbit ceil 200mbit prio 3 #FILTERS ##Servers (192.168.1.0 network); handle 1 means mark 1 (see iptables rules) tc filter add dev ifb0 parent 2:0 protocol ip prio 1 handle 1 fw flowid 2:10 ##Users (192.168.2.0 network); handle 2 means mark 2 (see iptables rules) tc filter add dev ifb0 parent 2:0 protocol ip prio 2 handle 2 fw flowid 2:20 ##WiFi (192.168.3.0 network); handle 3 means mark 3 (see iptables rules) tc filter add dev ifb0 parent 2:0 protocol ip prio 3 handle 4 fw flowid 2:30 #Create ingress on external interface tc qdisc add dev eth0 ingress handle ffff: #Forward all ingress traffic to the IFB device #tc filter add dev eth0 parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0 tc filter add dev eth0 parent ffff: protocol ip u32 match u32 0 0 action connmark action mirred egress redirect dev ifb0 flowid ffff:1 ########## #IPTABLES# ########## #Flush mangle rules iptables -F -t mangle iptables -X -t mangle iptables -t mangle -N QOS iptables -t mangle -A FORWARD -o eth0 -j QOS iptables -t mangle -A OUTPUT -o eth0 -j QOS iptables -t mangle -A QOS -j CONNMARK --restore-mark iptables -t mangle -A QOS -m mark ! --mark 0 -j ACCEPT iptables -t mangle -A QOS -s 192.168.1.0/24 -m mark --mark 0 -j MARK --set-mark 1 iptables -t mangle -A QOS -s 192.168.2.0/24 -m mark --mark 0 -j MARK --set-mark 2 iptables -t mangle -A QOS -s 192.168.3.0/24 -m mark --mark 0 -j MARK --set-mark 3 iptables -t mangle -A QOS -p icmp -j MARK --set-mark 0x1 iptables -t mangle -A QOS -p tcp -m tcp --sport 22 -j MARK --set-mark 0x1 #iptables -t mangle -I QOS -p tcp -m tcp --tcp-flags SYN,RST,ACK SYN -j MARK --set-mark 0x1 iptables -t mangle -A QOS -j CONNMARK --save-mark #(optionnal) set mark to give priority to ssh, icmp and packets initiating a tcp connection (SYN flags activated). see : https://inetdoc.net/guides/lartc/lartc.cookbook.fullnat.intro.html iptables -t mangle -A PREROUTING -p icmp -j MARK --set-mark 0x1 iptables -t mangle -A PREROUTING -p tcp -m tcp --sport 22 -j MARK --set-mark 0x1 iptables -t mangle -I PREROUTING -p tcp -m tcp --tcp-flags SYN,RST,ACK SYN -j MARK --set-mark 0x1
  • We can check that our frames are correctly marked with the conntrack tool:
root@host:~# apt-get install conntrack root@host:~# conntrack -L | grep 'mark=1'
  • If we don't like iptables we can mark traffic with this nftables.conf file:
flush ruleset table ip mangle { chain QOS_UPLOAD { type filter hook output priority -150; ip saddr { 192.168.1.0/24 } counter meta mark set 1 ip saddr { 192.168.2.0/24 } counter meta mark set 2 ip saddr { 192.168.3.0/24 } counter meta mark set 3 } chain QOS_DOWNLOAD { type filter hook forward priority -150; ip daddr { 192.168.1.0/24 } counter meta mark set 1 ip daddr { 192.168.1.0/24 } counter ct mark set mark ip daddr { 192.168.2.0/24 } counter meta mark set 2 ip daddr { 192.168.2.0/24 } counter ct mark set mark ip daddr { 192.168.3.0/24 } counter meta mark set 3 ip daddr { 192.168.3.0/24 } counter ct mark set mark #ip daddr 192.168.3.0/24 counter meta mark set ct mark } }

References

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Contact :

contact mail address