VoIP with load balancing PCC

I have set my routerboard to use 2 wan on load balance with PCC technique.

/ip firewall mangle
add action=accept chain=prerouting disabled=no dst-address=10.0.1.0/24 comment="Accept da WAN1"
add action=accept chain=prerouting disabled=no dst-address=10.0.2.0/24 comment="Accept da WAN2"

#Identify which WAN interface the traffic came in
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no in-interface=WAN1 new-connection-mark=WAN1 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no in-interface=WAN2 new-connection-mark=WAN2 passthrough=yes

#PCC
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-type=!local in-interface=bridge new-connection-mark=WAN1 passthrough=yes per-connection-classifier=both-addresses:2/0 comment="PCC stream WAN1"
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-type=!local in-interface=bridge new-connection-mark=WAN2 passthrough=yes per-connection-classifier=both-addresses:2/1 comment="PCC stream WAN2"

#Connection routing mark prerouting chain
add action=mark-routing chain=prerouting connection-mark=WAN1 disabled=no in-interface=bridge new-routing-mark=WAN1-mark passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2 disabled=no in-interface=bridge new-routing-mark=WAN2-mark passthrough=yes 

#Connection routing mark output chain
add action=mark-routing chain=output connection-mark=WAN1 disabled=no new-routing-mark=WAN1-mark passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2 disabled=no new-routing-mark=WAN2-mark passthrough=yes

P.s. I think that the first 2 rules “accept” don’t really serve.

Now I’ve added in the LAN a VoIP PBX to test it.
Obviously, the traffic to and from the VoIP (IP 192.168.1.100) don’t have to be processed by PCC and the PBX had to be forced to use only one wan.
Which rule I can add to force VoIP PBX to use only one WAN?

Assign an “unused” mark so it’s not processed by other mangles and uses default routing.
/ip firewall mangle
add action=mark-connection connection-mark=no-mark chain=prerouting src-address= comment=“Mark pbx” new-connection-mark=PBX

Interesting…I did not think about using other marks.
But which is the default route ? Wan 1 has a gateway (ISP1) and Wan 2 has another gateway (ISP2)

I thought to force the routing adding this rules in mangle:

/ip firewall mangle
#Mark Voip connection, force use WAN2
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-type=!local in-interface=bridge src-address=192.168.1.100/24 new-connection-mark=Voip_WAN2 passthrough=yes comment="Voip connection mark WAN2"

#Voip routing mark, prerouting
add action=mark-routing chain=prerouting connection-mark=Voip_WAN2 disabled=no in-interface=bridge new-routing-mark=WAN2-mark passthrough=yes comment="Voip new routing mark WAN2, in bridge"

#Voip routing mark, output
add action=mark-routing chain=output connection-mark=Voip_WAN2 disabled=no new-routing-mark=WAN2-mark passthrough=yes comment="Voip new routing mark WAN2, chain output"

Can it work?
I also have to reserve some bandwidth, about 200 kb/s, on upload. How can I do it ?

if you really want to do marking, reuse existing marks:

add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge src-address=192.168.1.100 new-connection-mark=WAN2 passthrough=yes comment=“Voip connection mark WAN2”

Rest of original script can be reused

Yes, you are right. I’ve added another connection mark but there is no need to do it. Thanks :wink:

WAN2 is 10M down/1M up.
Now, how can i reserve some bandwidth for VoIP upload ?

add a queue tree on wan link, and define queues matching packet marks.
https://wiki.mikrotik.com/wiki/Manual:Queue#Queue_Tree
https://wiki.mikrotik.com/wiki/Manual:HTB

I haven’t rule to mark packet in mangle.
Can i use queue tree on WAN 2 without marking the packet ?

No, because the packet-mark chooses the right queue within the tree.

Can I use simple queue without marking packets?

To use queue tree I have to mark all packet to and from my PBX IP.
Next, set a queue tree global on wan 2. Which type of queue?
Next a queue with “limit at” for marked packets.
Did I understand right ?

You can use “no-mark” for bulk! So what you can do, is mangle voip selectively and throw rest of unmarked packets in “bulk” queue.

What is needed:

  • in postrouting, mark packets with PBX connection mark → for that you’ll need a separate connection mark
  • setup htb on wan interface (not global) with max-limit < upload limit
  • add PBX queue, child to htb, with “pbx” packet mark and reserved bandwidth, prio x
  • add bulk queue, child to htb, with “no-mark” and no limits set, with prio y (y>x)

(lower prio number → higher priority)

With your suggestion i have to add this on mangle.
Marking connection of Voip:

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-type=!local in-interface=bridge src-address=192.168.1.246 new-connection-mark=Voip_WAN2 passthrough=yes comment="Voip connection mark WAN2"

add action=mark-routing chain=prerouting connection-mark=Voip_WAN2 disabled=no in-interface=bridge new-routing-mark=WAN2-mark passthrough=yes comment="Voip routing mark WAN2_mark chain prerouting"

add action=mark-routing chain=output connection-mark=Voip_WAN2 disabled=no new-routing-mark=WAN2-mark passthrough=yes comment="Voip routing mark WAN2_mark chain output"

Marking packets from Voip PBX to WAN2:

/ip firewall mangle
#Voip packet marking
add action=mark-packet chain=postrouting connection-mark=Voip_WAN2 new-packet-mark=fromVoip

Adding queue tree

/queue tree
add name=uploadWAN2 parent=WAN2 max-limit=1024k
add name=Voip_upload_WAN2 parent=uploadWAN2 max-limit=1024k limit-at=200k priority=1 packet-mark=fromVoip
add name=otherClientWAN2 parent=uploadWAN2 max-limit=1024k priority=8 packet-mark=no-mark

Can it work this way ?
Packet with no mark in the queue will be only packets to WAN2 with no mark or all packets (wan1 and wan2) with no mark ?

it should work: verify counters on queues that they are actually used.
Only packets with no-mark for wan2, as queue is attached to wan2.

Make sure max-limit is not too high: there should be no buffering on isp modem => no added latency.

I’ve added the dst-nat on port 5060, 5061 to the pbx an the range of port used for udp 10000-12000.
It seems start working when i added this mangle rule that are matching some traffic.

/ip firewall mangle
add chain=forward in-interface=WAN1 action=mark-connection new-connection-mark=WAN1
add chain=forward in-interface=WAN2 action=mark-connection new-connection-mark=WAN2

But I think there is no need… connections has to be already marked when entering from internet or going to internet.
Forward is for traffic that passing through the router.
Input and output are chain to use for traffic that is entering or going to internt.
Maybe I misunderstood ?

Normally you mark each connection when handling its initial packet, so that you could handle all packets belonging to that connection the same way. Isolated rules posted above do not show the context so they may be correct if rules preceding them don’t let other-than-initial packets of each connection to reach these two, or wrong if they do.

I assume your phones on LAN register to the PBX behind WAN; if so, the signalling (SIP) connections are always initiated from LAN. However, the first RTP packet of each call may come from a different direction, so connection marking for in-interface=WAN2 may also be important.

I’m not sure this is the wisest route to go here. See, the PBX is going to set it’s RTP IP to one of those WAN IP addresses. That’s how it works. So if you have two WAN IPs and make a call or receive a call, the PBX needs to know what IP to assign for RTP to receive the media back. At any point during that call you could end up changing that IP. A Re-INVITE, holding, parking, etc, etc.. This could cause audio issues if the other side doesn’t get updated properly or routed properly.

There is also the fact this is a Request/Reply transaction so there is no guarantee that the request makes it to the destination and the reply makes it back to the source. SIP/VoIP, whatever you want to call it, already has a low threshold for latency, delays, jitter, etc.. I personally (and considering Voice/Telephony is what I do and I use MTs for that need) would never do this. There’s no need to split the traffic like this for 80Kbps of traffic. Seriously, it’s 80Kbps of traffic. That means you can support a half dozen calls with just about 600Kbps of traffic.

So yeah when you think about it what is the point of load balancing less than 1Mbps of traffic for voice when you’ll barely touch that. You would need to hit almost 13 concurrent active calls for you to peak at 1Mbps of traffic just for SIP. While bandwidth is an important factor in the quality of the call, it’s not about having a huge pipe because you have 6 phones it’s about the quality of the bandwidth. As I said, low tolerance for latency, delay and jitter which for normal TCP/web traffic isn’t a thing but you hit 10-15 ms of delay/jitter/latency your call is losing serious quality. Get up to almost 30 it’s basically garbage.

Yeah, I just don’t see the need to load balance such a low amount of traffic and split it up among multiple backbone pipes for a single call. You’re introducing numerous factors that will impact the call and overall, how to troubleshoot the call. Is the call being choppy because of? Is it because it’s delayed in reforming the packets on your side, it is delayed because one of the routes is having an issue and it might not even be an issue locally you can see but between them and their peers? If you’re sending that audio over 2-3 connections and you start having issues, you now have 2-3 connections you have to figure out is the cause of the issue (or not).

I have only 1 line. I measured the traffic during a call, it’s a maximum of 100Kb.
Phones are not in the lan of the mikrotik router. Are on another lan managed directly from PBX (pbx has it’s own poe router/switch inside).

VoIP connections is excluded from PCC using the rules that force traffic through the WAN2 interface, so it will use only WAN2.
I’ve assumed also that inocoming call can only use WAN2 to reach my PBX. The VoIP server only know one IP. The IP you use to set up the connection from PBX to the VoIP server.

Forward rules are the latest rules in mangle.
The fill mangle rules list:

/ip firewall mangle
add action=accept chain=prerouting disabled=no dst-address=10.0.1.0/24 comment="Accept da WAN1"
add action=accept chain=prerouting disabled=no dst-address=10.0.2.0/24 comment="Accept da WAN2"

#Identify which WAN interface the traffic came in
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no in-interface=WAN1 new-connection-mark=WAN1 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no in-interface=WAN2 new-connection-mark=WAN2 passthrough=yes

#Stream pcc
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-type=!local in-interface=bridge src-address=192.168.1.100 new-connection-mark=Voip_WAN2 passthrough=yes comment="Voip connection mark WAN2"
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-type=!local in-interface=bridge new-connection-mark=WAN1 passthrough=yes per-connection-classifier=both-addresses:2/0 comment="PCC stream WAN1"
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-type=!local in-interface=bridge new-connection-mark=WAN2 passthrough=yes per-connection-classifier=both-addresses:2/1 comment="PCC stream WAN2"

#Connection routing mark prerouting chain
add action=mark-routing chain=prerouting connection-mark=Voip_WAN2 disabled=no in-interface=bridge new-routing-mark=WAN2-mark passthrough=yes comment="Voip routing mark WAN2_mark chain prerouting"
add action=mark-routing chain=prerouting connection-mark=WAN1 disabled=no in-interface=bridge new-routing-mark=WAN1-mark passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2 disabled=no in-interface=bridge new-routing-mark=WAN2-mark passthrough=yes 

#Connection routing mark output chain
add action=mark-routing chain=output connection-mark=Voip_WAN2 disabled=no new-routing-mark=WAN2-mark passthrough=yes comment="Voip routing mark WAN2_mark chain output"
add action=mark-routing chain=output connection-mark=WAN1 disabled=no new-routing-mark=WAN1-mark passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2 disabled=no new-routing-mark=WAN2-mark passthrough=yes

add chain=forward in-interface=WAN1 action=mark-connection new-connection-mark=WAN1
add chain=forward in-interface=WAN2 action=mark-connection new-connection-mark=WAN2

add action=mark-packet chain=postrouting connection-mark=Voip_WAN2 new-packet-mark=fromVoip

I’ve checked the queue tree in a test setup.

The Voip queue takes all bandwidth up to its maximum limit even if in the other queue there is upload activity. I thought queue would split in half the bandwidth, after satisfying the minimum bandwidth imposed by limit-at.
How bandwidth division works in a queue?

Documentation is not clear on that point: https://wiki.mikrotik.com/wiki/Manual:HTB.
One example has such situation, but the effect/goal is not elaborated.

Then again is that a valid situation for you? I would think not: voip should have higher prio, and it’s volume will be much smaller than rest in most cases.

This is exactly what you need. The VoIP must get all it wants because otherwise all the calls will be unusable. But it may be that you’ve marked some other than VoIP traffic by mistake. Roughly, each active voice call needs about 100 kbit/s per direction. Video calls are a different story. Signalling traffic (control of the calls) occupies a negligible bandwidth but needs to have at least the same priority like the voice streams (RTP) as you need to be able to terminate a call even if the bandwidth is stuffed with the voice streams.


Each queue within the tree gets a guaranteed minimum bandwidth (but it is your responsibility to assign it in such a way that it is physically possible, and you may guarantee zero) and has a maximum allowed bandwidth which it can get if other queues do not make use of their assignments.

So in practical terms, the VoIP queue must have all the bandwidth it may possibly need guaranteed, and the sum of eventual guaranteed bandwidths of all queues must not exceed the physical bandwidth of the link.

The situation i tested is the worst case: 1 Voip call during an upload of a recorded message in the voice mail. The voice quality is fine for me. I will set a higher priority on VoIP queue.
Simply I expected that the bandwidth, in such case, was split about in two. But during the test the Voip queue took all the bandwidth. On the normal queue, the upload was interrupted due to timeout. There were no bandwidth for the “normal client” queue.