Bug in queue tree decision logic

markdutton · September 10, 2011, 1:12am

Hi All

I have found a bug in the queue tree decision logic which is making it impossible for me to use the router as I need to.

The scenario is I have 2 WAN interfaces and I have 2 queue trees attached to these interfaces. I have various mangle rules to create connection and packet marks to fee the queue trees.

One of the interfaces is the default route for Internet traffic and the other is fed via policy routes.

All of this works, so long as the routing decisions come from traffic originating inside the router.

However, if I have traffic originating from outside the router, that connects to my “non-default” interface, even though the router creates an implicit routing table to ensure the traffic egresses that same connection, the traffic is processed by the wrong queue tree, which as you can imagine is disastrous.

I want to keep my routes as they are. Most traffic goes out via the GP ADSL interface. The reason I have these incoming connections on the other interface is that they are for remote users coming in on VPN. This is a high quality 5Mb/S symmetrical interface. They are connecting to servers that would otherwise egress through the GP ADSL interface.

The theory is sound. Everything works, except the router seems to be using the static routing table to determine which queue tree to traverse, rather than being associated directly to the appropriate Interface.

Anyone see this and figured a workaround? I would like to see it treated as a bug, but I posted here as I find the forum quicker than support most times.

fewi · September 10, 2011, 1:30am

Can you post the output of “/interface print detail”, “/ip address print detail”, “/ip route print detail”, “/ip firewall export”, and “/queue export”, as well as give more detail on what queue you’re expecting traffic in, what queue you’re seeing traffic in instead, and how you’re determining that?

markdutton · September 10, 2011, 2:33am

Following are all logs except firewall export, which is huge and I would rather not disclose it anyway.

The issue is the queues should be attached to the egress interface and they are not.

I know this because when I connect externally to my 4-Amcom-BDSL interface via https (443), and drag a file down to my external computer, I can see the 4-Amcom-BDSL interface egressing at 500k and I can see the 5-Amnet queue tree showing 500k through the http-upload queue.

If I then add a static route to my external IP address (where I am connecting from), the queue tree for the 4-Amcom-BDSL jumps to life and I get 4490k through the http-upload queue

This is easy to setup and test.

[admin@MikroTik] > /interface print detail
Flags: D - dynamic, X - disabled, R - running, S - slave
0 R name="1-Local" type="ether" mtu=1500 l2mtu=1524
1 ;;; For when router is not reachable. Offers DHCP.
name="2-Console" type="ether" mtu=1500 l2mtu=1524
2 R name="3-Production" type="ether" mtu=1500 l2mtu=1524
3 R name="4-Amcom-BDSL" type="ether" mtu=1524 l2mtu=1524
4 R name="5-Amnet" type="ether" mtu=1492 l2mtu=1524
5 R name="Workshop" type="vlan" mtu=1500 l2mtu=1520
6 R name="Guest" type="vlan" mtu=1500 l2mtu=1520
7 R name="VOIP" type="vlan" mtu=1500 l2mtu=1520
8 R name="Development" type="vlan" mtu=1500 l2mtu=1520

[admin@MikroTik] > ip address print detail
Flags: X - disabled, I - invalid, D - dynamic
0 ;;; default configuration
address=192.168.88.1/24 network=192.168.88.0 interface=2-Console
actual-interface=2-Console
1 address=192.168.44.9/24 network=192.168.44.0 interface=3-Production
actual-interface=3-Production
2 address=192.168.46.9/24 network=192.168.46.0 interface=Workshop
actual-interface=Workshop
3 address=192.168.50.9/24 network=192.168.50.0 interface=VOIP
actual-interface=VOIP
4 address=192.168.56.9/24 network=192.168.56.0 interface=Development
actual-interface=Development
5 address=192.168.47.9/24 network=192.168.47.0 interface=Guest
actual-interface=Guest
6 address=203.xxx.xxx.242/30 network=203.xxx.xxx.240 interface=5-Amnet
actual-interface=5-Amnet
7 address=172.31.110.3/24 network=172.31.110.0 interface=4-Amcom-BDSL
actual-interface=4-Amcom-BDSL
8 ;;; Extra IP addresses
address=203.xxx.xxx.169/29 network=203.xxx.xxx.168 interface=5-Amnet
actual-interface=5-Amnet
9 address=203.xxx.xxx.171/29 network=203.xxx.xxx.168 interface=5-Amnet
actual-interface=5-Amnet
10 address=203.xxx.xxx.172/29 network=203.xxx.xxx.168 interface=5-Amnet
actual-interface=5-Amnet
11 address=203.xxx.xxx.173/29 network=203.xxx.xxx.168 interface=5-Amnet
actual-interface=5-Amnet
12 address=203.xxx.xxx.174/29 network=203.xxx.xxx.168 interface=5-Amnet
actual-interface=5-Amnet
13 ;;; Managed Services
address=203.xxx.xxx.170/29 network=203.xxx.xxx.168 interface=5-Amnet
actual-interface=5-Amnet
14 address=116.xxx.xxx.110/30 network=116.xxx.xxx.108 interface=4-Amcom-BDSL
actual-interface=4-Amcom-BDSL

[admin@MikroTik] > ip route print detail
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
0 A S dst-address=0.0.0.0/0 gateway=116.xxx.xxx.109
gateway-status=116.xxx.xxx.109 reachable 4-Amcom-BDSL distance=100
scope=10 target-scope=10 routing-mark=VC
1 A S dst-address=0.0.0.0/0 gateway=203.xxx.xxx.241
gateway-status=203.xxx.xxx.241 reachable 5-Amnet check-gateway=arp
distance=15 scope=10 target-scope=10
2 ADC dst-address=116.xxx.xxx.108/30 pref-src=116.xxx.xxx.110
gateway=4-Amcom-BDSL gateway-status=4-Amcom-BDSL reachable distance=0
scope=10
3 ADC dst-address=172.31.110.0/24 pref-src=172.31.110.3 gateway=4-Amcom-BDSL
gateway-status=4-Amcom-BDSL reachable distance=0 scope=10
4 ADC dst-address=192.168.44.0/24 pref-src=192.168.44.9 gateway=3-Production
gateway-status=3-Production reachable distance=0 scope=10
5 ADC dst-address=192.168.46.0/24 pref-src=192.168.46.9 gateway=Workshop
gateway-status=Workshop reachable distance=0 scope=10
6 ADC dst-address=192.168.47.0/24 pref-src=192.168.47.9 gateway=Guest
gateway-status=Guest reachable distance=0 scope=10
7 ADC dst-address=192.168.50.0/24 pref-src=192.168.50.9 gateway=VOIP
gateway-status=VOIP reachable distance=0 scope=10
8 ADC dst-address=192.168.56.0/24 pref-src=192.168.56.9 gateway=Development
gateway-status=Development reachable distance=0 scope=10
9 ADC dst-address=192.168.88.0/24 pref-src=192.168.88.1 gateway=2-Console
gateway-status=2-Console unreachable distance=0 scope=200
10 ADC dst-address=203.xxx.xxx.240/30 pref-src=203.xxx.xxx.242 gateway=5-Amnet
gateway-status=5-Amnet reachable distance=0 scope=10
11 ADC dst-address=203.xxx.xxx.168/29 pref-src=203.xxx.xxx.169 gateway=5-Amne>
gateway-status=5-Amnet reachable distance=0 scope=10

[admin@MikroTik] > queue export

sep/10/2011 10:25:31 by RouterOS 5.5

software id = 288S-LAAF

/queue tree
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=700k
max-limit=700k name=ADSL-Out parent=5-Amnet priority=8
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0
max-limit=4500k name=MBE-out parent=4-Amcom-BDSL priority=8
/queue type
set default kind=pfifo name=default pfifo-limit=50
set ethernet-default kind=pfifo name=ethernet-default pfifo-limit=50
set wireless-default kind=sfq name=wireless-default sfq-allot=1514
sfq-perturb=5
set synchronous-default kind=red name=synchronous-default red-avg-packet=1000
red-burst=20 red-limit=60 red-max-threshold=50 red-min-threshold=10
set hotspot-default kind=sfq name=hotspot-default sfq-allot=1514 sfq-perturb=
5
add kind=sfq name=VOIP sfq-allot=200 sfq-perturb=5
set default-small kind=pfifo name=default-small pfifo-limit=10
/queue tree
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=256k
max-limit=690k name=VOIP packet-mark=VOIP parent=ADSL-Out priority=1
queue=VOIP
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=64k
max-limit=256k name=Slow-Internet packet-mark=Slow-Traffic parent=
ADSL-Out priority=8 queue=default
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=64k
max-limit=690k name=Default packet-mark=no-mark parent=ADSL-Out priority=
6 queue=default
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=64k
max-limit=500k name=HTTP-Upload packet-mark=http-upload parent=ADSL-Out
priority=6 queue=default
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=188k
max-limit=512k name=VC packet-mark=VC parent=ADSL-Out priority=2 queue=
default
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=64k
max-limit=500k name=Fast-Internet packet-mark=fast-traffic parent=
ADSL-Out priority=3 queue=default
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=256k
max-limit=690k name=vpn-passthrough packet-mark=vpn-passthrough parent=
ADSL-Out priority=7 queue=synchronous-default
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=400k
max-limit=400k name=MBE-VOIP packet-mark=VOIP parent=MBE-out priority=1
queue=VOIP
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=400k
max-limit=4490k name=MBE-Default packet-mark=no-mark parent=MBE-out
priority=6 queue=default
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=1M
max-limit=4490k name=MBE-Fast-Internet packet-mark=fast-traffic parent=
MBE-out priority=3 queue=default
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=400k
max-limit=4490k name=MBE-HTTP-Upload packet-mark=http-upload parent=
MBE-out priority=6 queue=default
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=256k
max-limit=2048k name=MBE-Slow-Internet packet-mark=Slow-Traffic parent=
MBE-out priority=8 queue=default
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=1500k
max-limit=4490k name=MBE-VC packet-mark=VC parent=MBE-out priority=2
queue=default
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=400k
max-limit=4490k name=MBE-vpn-passthrough packet-mark=vpn-passthrough
parent=MBE-out priority=7 queue=default
/queue interface
set 1-Local queue=ethernet-default
set 2-Console queue=ethernet-default
set 3-Production queue=ethernet-default
set 4-Amcom-BDSL queue=ethernet-default
set 5-Amnet queue=ethernet-default

markdutton · September 10, 2011, 3:14am

OK. Sorry guys. I have led you astray. The router is very busy, so sometimes it can lead me to make a false assumption.

I just did a packet trace on the interfaces. It turns out that the router is NOT using the wrong queues. It is using the wrong interface.

However, it is outbounding traffic on the 5-Amnet interface with a SOURCE address of the 4-Amcom-BDSL interface. So it is SPOOFING the address that the initial request came in on. Effectively I have a round robin scenario where my external computer in going to one interface and the router is returning on another. Because it is spoofing, everything appears to work.

I am not sure that this is best practise behaviour, but as it happens, I need to figure out how to stop it happening. Is there a way to get the router to create an “on the fly” routing table that is created when an initial connection hits any external interface, thus allowing the return traffic to go through the same interface. The Snapgear router does this by default and this is the behaviour I would like to mimic.

Regards

Mark

fewi · September 10, 2011, 3:44am

For any packet the router will consult the routing table, regardless of how the packet entered the router. If you want a return route that the routing table usually wouldn’t pick you usually have to use routing marks. In your specific scenario you’d be marking connections that come in via a specific interface, and then in mangle apply routing marks to all packets that are part of that connection, and then have a route for that routing mark punting those packets back out the interface you want them to leave the router through.

The second paragraph in the PCC wiki briefly explains the concept: http://wiki.mikrotik.com/wiki/Manual:PCC#Policy_routing

It seems like you’re not doing that, looking at the routing table you posted (and by the post you made - I started typing this up before that and it’s now a heavily edited version, I hope I edited everything to make sense). Here’s roughly what you’d need. I’m setting passthrough to ‘yes’ assuming you’re using some sort of packet marks for queueing purposes. If you don’t have other mangle rules you can set it to ‘no’ to save on processing time. If you need further help you’ll have to at least post your mangle rule set so get help to fit this in.

/ip firewall address-list
add list=rfc1918 address=10.0.0.0/8
add list=rfc1918 address=172.16.0.0/12
add list=rfc1918 address=192.168.0.0/16
/ip firewall mangle
add chain=prerouting in-interface=4-Amcom-BDSL connection-state=new action=mark-connection new-connection-mark=to-4-Amcom-BDSL
add chain=prerouting in-interface=5-Amnet connection-state=new action=mark-connection new-connection-mark=to-5-Amnet
add chain=prerouting connection-mark=to-4-Amcom-BDSL src-address-list=rfc1918 action=mark-routing new-routing-mark=to-4-Amcom-BDSL passthrough=no
add chain=prerouting connection-mark=to-5-Amnetsrc-address-list=rfc1918 action=mark-routing new-routing-mark=to-5-Amnet passthrough=no
/ip route 
add dst-address=0.0.0.0/0 routing-mark=to-4-Amcom-BDSL gateway=116.x.x.109
add dst-address=0.0.0.0/0 routing-mark=to-5-Amnet  gateway=203.x.x.241

Those should have a lower distance than your existing default routes, which apparently they will according to your current routing table. The mangle rules have a good shot at just working at the top of the rule set before any other rules you have, but if they don’t post what you got to get help fitting them in.

Technically you only need to do this for the circuit with the higher distance (the BDSL one), but this won’t hurt as such and will keep things working if you ever flip the distances around to prefer the BDSL circuit (maybe while the primary circuit is impacted).

Incidentally, whatever router you have that just ‘does this’ is doing something along these lines behind the scenes. RouterOS is very naked - it can do all of that, but you explicitly have to tell it to. On the other hand you have full control over everything.

If you compare this to the PCC example it applies the marks to the return packets in rules via in-interface. That’s cool because there’s only one in that example. You have lots, so it’s easier to mark it by private IP space as a source rather than duplicate the rules for every possible LAN interface. It’s just more efficient, same effect.

markdutton · September 10, 2011, 6:04am

Thanks Fewi

I was on the same tack as you stated in your post. I have been able to get the outbound interface working fine using routing marks and a custom routing table. The issue now for me is I already have a huge mangle table for my packet marking rules.

To keep things tidy and to get maximum performance, I already have a stack of packet marking rules at the top of the routing table. These are filtered by routing mark. Below these are a stack of routing marking rules that are using DSCP codepoints, IP addresses and ports as filters. By putting these below as each packet flows through the router, it only needs to iterate up to about 10 packet marking rules based on a single parameter, being connection mark. It looks like this (Lines 0 to 3 removed for clarity)

[admin@MikroTik] > ip firewall mangle print detail
Flags: X - disabled, I - invalid, D - dynamic
3 ;;; Mark VOIP
chain=forward action=mark-packet new-packet-mark=VOIP passthrough=no
connection-mark=VOIP

4 ;;; Mark VC
chain=forward action=mark-packet new-packet-mark=VC passthrough=no
connection-mark=VC

5 ;;; Fast traffic
chain=forward action=mark-packet new-packet-mark=fast-traffic
passthrough=no connection-mark=fast-traffic

6 chain=output action=mark-packet new-packet-mark=fast-traffic
passthrough=no connection-mark=fast-traffic

7 ;;; slow traffic
chain=forward action=mark-packet new-packet-mark=Slow-Traffic
passthrough=no connection-mark=slow-traffic

8 ;;; http-upload
chain=forward action=mark-packet new-packet-mark=http-upload
passthrough=no connection-mark=http-upload

9 ;;; vpn-passthrough
chain=forward action=mark-packet new-packet-mark=vpn-passthrough
passthrough=no connection-mark=vpn-passthrough

10 ;;; Mark codepoint 46 as VOIP connection
chain=forward action=mark-connection new-connection-mark=VOIP
passthrough=no src-address=192.168.44.0/24 dscp=46

11 ;;; Zultys out 1 as VOIP
chain=forward action=mark-connection new-connection-mark=VOIP
passthrough=no protocol=udp src-address=192.168.44.12

12 ;;; Zultys out 2 as VOIP
chain=forward action=mark-connection new-connection-mark=VOIP
passthrough=no protocol=udp src-address=192.168.44.13

13 ;;; Classify ICMP as VOIP
chain=forward action=mark-connection new-connection-mark=VOIP
passthrough=no protocol=icmp

14 ;;; Mark VC
chain=forward action=mark-connection new-connection-mark=VC
passthrough=no protocol=udp src-address=192.168.44.32

15 ;;; DSCP 45
chain=forward action=mark-connection new-connection-mark=VC
passthrough=no dscp=45

16 chain=forward action=mark-packet new-packet-mark=http-upload
passthrough=no connection-mark=http-upload

17 ;;; Mark n-able as slow queue
chain=forward action=mark-connection new-connection-mark=slow-traffic
passthrough=yes src-address=192.168.44.5

18 ;;; Mark mail as slow queue
chain=forward action=mark-connection new-connection-mark=slow-traffic
passthrough=yes protocol=tcp dst-port=25

19 chain=forward action=mark-packet new-packet-mark=Slow-Traffic
passthrough=no connection-mark=slow-traffic

20 ;;; Mark HTTP outbound as http-upload
chain=forward action=mark-connection new-connection-mark=http-upload
passthrough=yes protocol=tcp src-port=80,443

21 ;;; Mark RDP and others as fast traffic
chain=forward action=mark-connection new-connection-mark=fast-traffic
passthrough=yes protocol=tcp src-port=3389,22,8291,7100-7150,7505

22 chain=forward action=mark-packet new-packet-mark=fast-traffic
passthrough=no connection-mark=fast-traffic

23 ;;; Fast output for router management
chain=output action=mark-connection new-connection-mark=fast-traffic
passthrough=yes protocol=tcp src-port=8291

24 chain=output action=mark-packet new-packet-mark=fast-traffic
passthrough=no connection-mark=fast-traffic

So I thought I could mark incoming new connections that come in on the specified interface with a routing mark. As I wanted the routing mark to be processed at least once on the way out, I used a connection mark on the originating packet. This would allow the packet to turn around and on the way out it would create a routing mark. This rule would be set to passthrough so the mangle table could create new connection marks for my QoS packet marking setup. I was hoping a single routing mark on the first packet would be enough and that the router would cache the routing table for all subsequent packets. The first rules looked like this. NOTE I have an input rule and a preroute rule so I could catch traffic to and through the router.

[MikroTik] > ip firewall mangle print detail
Flags: X - disabled, I - invalid, D - dynamic
0 ;;; Adhoc inbound connection on BDSL interface
chain=input action=mark-connection new-connection-mark=adhoc-bdsl
passthrough=no connection-state=new in-interface=4-Amcom-BDSL

1 chain=prerouting action=mark-connection new-connection-mark=adhoc-bdsl
passthrough=no connection-state=new in-interface=4-Amcom-BDSL

2 chain=prerouting action=mark-routing new-routing-mark=adhoc-bdsl
passthrough=yes connection-mark=adhoc-bdsl

Unfortunately it seems that the router does not cache routes for a single connection. It seems I have to use the connection marking method permanently for the adhoc-routing mechanism, so I will need to use a far less efficient method for my packet marking. I will have to change all my connection marking rules based on my QoS parameters to packet marking rules and the router will have to do much more work now.

That is, unless there is a way to get the router to cache its routes on a “per connection” basis.

Do you think this is possible?

Mark

markdutton · September 10, 2011, 6:13am

BTW Fewi

I like your idea of using RFC1918 as the source address list so only outbound packets would fire the routing mark.

I just put a new rule in my adhoc-bdsl routing table that duplicated the main table rule so the rule would fire in both directions and the inbound route was catered for. It would be interesting to know which uses more CPU. Processing more parameters to determine if a rule should fire, or processing the marking step twice as often.