VoIP with load balancing PCC

Technetium · Mon May 06, 2019 9:20 pm

I have set my routerboard to use 2 wan on load balance with PCC technique.

/ip firewall mangle
add action=accept chain=prerouting disabled=no dst-address=10.0.1.0/24 comment="Accept da WAN1"
add action=accept chain=prerouting disabled=no dst-address=10.0.2.0/24 comment="Accept da WAN2"

#Identify which WAN interface the traffic came in
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no in-interface=WAN1 new-connection-mark=WAN1 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no in-interface=WAN2 new-connection-mark=WAN2 passthrough=yes

#PCC
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-type=!local in-interface=bridge new-connection-mark=WAN1 passthrough=yes per-connection-classifier=both-addresses:2/0 comment="PCC stream WAN1"
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-type=!local in-interface=bridge new-connection-mark=WAN2 passthrough=yes per-connection-classifier=both-addresses:2/1 comment="PCC stream WAN2"

#Connection routing mark prerouting chain
add action=mark-routing chain=prerouting connection-mark=WAN1 disabled=no in-interface=bridge new-routing-mark=WAN1-mark passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2 disabled=no in-interface=bridge new-routing-mark=WAN2-mark passthrough=yes 

#Connection routing mark output chain
add action=mark-routing chain=output connection-mark=WAN1 disabled=no new-routing-mark=WAN1-mark passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2 disabled=no new-routing-mark=WAN2-mark passthrough=yes

P.s. I think that the first 2 rules "accept" don't really serve.

Now I've added in the LAN a VoIP PBX to test it.
Obviously, the traffic to and from the VoIP (IP 192.168.1.100) don't have to be processed by PCC and the PBX had to be forced to use only one wan.
Which rule I can add to force VoIP PBX to use only one WAN?

sebastia · Mon May 06, 2019 10:14 pm

Assign an "unused" mark so it's not processed by other mangles and uses default routing.
/ip firewall mangle
add action=mark-connection connection-mark=no-mark chain=prerouting src-address=<pbx box> comment="Mark pbx" new-connection-mark=PBX

Technetium · Mon May 06, 2019 10:50 pm

Interesting...I did not think about using other marks.
But which is the default route ? Wan 1 has a gateway (ISP1) and Wan 2 has another gateway (ISP2)

I thought to force the routing adding this rules in mangle:

/ip firewall mangle
#Mark Voip connection, force use WAN2
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-type=!local in-interface=bridge src-address=192.168.1.100/24 new-connection-mark=Voip_WAN2 passthrough=yes comment="Voip connection mark WAN2"

#Voip routing mark, prerouting
add action=mark-routing chain=prerouting connection-mark=Voip_WAN2 disabled=no in-interface=bridge new-routing-mark=WAN2-mark passthrough=yes comment="Voip new routing mark WAN2, in bridge"

#Voip routing mark, output
add action=mark-routing chain=output connection-mark=Voip_WAN2 disabled=no new-routing-mark=WAN2-mark passthrough=yes comment="Voip new routing mark WAN2, chain output"

Can it work?
I also have to reserve some bandwidth, about 200 kb/s, on upload. How can I do it ?

sebastia · Mon May 06, 2019 11:12 pm

if you really want to do marking, reuse existing marks:

add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge src-address=192.168.1.100 new-connection-mark=WAN2 passthrough=yes comment="Voip connection mark WAN2"

Rest of original script can be reused

Technetium · Mon May 06, 2019 11:47 pm

Yes, you are right. I've added another connection mark but there is no need to do it. Thanks

WAN2 is 10M down/1M up.
Now, how can i reserve some bandwidth for VoIP upload ?

sebastia · Mon May 06, 2019 11:56 pm

add a queue tree on wan link, and define queues matching packet marks.
https://wiki.mikrotik.com/wiki/Manual:Queue#Queue_Tree
https://wiki.mikrotik.com/wiki/Manual:HTB

Technetium · Tue May 07, 2019 12:20 am

I haven't rule to mark packet in mangle.
Can i use queue tree on WAN 2 without marking the packet ?

sindy · Tue May 07, 2019 6:47 am

Can i use queue tree on WAN 2 without marking the packet ?

No, because the packet-mark chooses the right queue within the tree.

Technetium · Tue May 07, 2019 9:45 am

Can I use simple queue without marking packets?

To use queue tree I have to mark all packet to and from my PBX IP.
Next, set a queue tree global on wan 2. Which type of queue?
Next a queue with "limit at" for marked packets.
Did I understand right ?

sebastia · Tue May 07, 2019 9:52 am

You can use "no-mark" for bulk! So what you can do, is mangle voip selectively and throw rest of unmarked packets in "bulk" queue.

What is needed:
* in postrouting, mark packets with PBX connection mark -> for that you'll need a separate connection mark
* setup htb on wan interface (not global) with max-limit < upload limit
* add PBX queue, child to htb, with "pbx" packet mark and reserved bandwidth, prio x
* add bulk queue, child to htb, with "no-mark" and no limits set, with prio y (y>x)

(lower prio number -> higher priority)

Technetium · Tue May 07, 2019 12:31 pm

With your suggestion i have to add this on mangle.
Marking connection of Voip:

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-type=!local in-interface=bridge src-address=192.168.1.246 new-connection-mark=Voip_WAN2 passthrough=yes comment="Voip connection mark WAN2"

add action=mark-routing chain=prerouting connection-mark=Voip_WAN2 disabled=no in-interface=bridge new-routing-mark=WAN2-mark passthrough=yes comment="Voip routing mark WAN2_mark chain prerouting"

add action=mark-routing chain=output connection-mark=Voip_WAN2 disabled=no new-routing-mark=WAN2-mark passthrough=yes comment="Voip routing mark WAN2_mark chain output"

Marking packets from Voip PBX to WAN2:

/ip firewall mangle
#Voip packet marking
add action=mark-packet chain=postrouting connection-mark=Voip_WAN2 new-packet-mark=fromVoip

Adding queue tree

/queue tree
add name=uploadWAN2 parent=WAN2 max-limit=1024k
add name=Voip_upload_WAN2 parent=uploadWAN2 max-limit=1024k limit-at=200k priority=1 packet-mark=fromVoip
add name=otherClientWAN2 parent=uploadWAN2 max-limit=1024k priority=8 packet-mark=no-mark

Can it work this way ?
Packet with no mark in the queue will be only packets to WAN2 with no mark or all packets (wan1 and wan2) with no mark ?

sebastia · Tue May 07, 2019 2:09 pm

it should work: verify counters on queues that they are actually used.
Only packets with no-mark for wan2, as queue is attached to wan2.

Make sure max-limit is not too high: there should be no buffering on isp modem => no added latency.

Technetium · Tue May 07, 2019 3:59 pm

I've added the dst-nat on port 5060, 5061 to the pbx an the range of port used for udp 10000-12000.
It seems start working when i added this mangle rule that are matching some traffic.

/ip firewall mangle
add chain=forward in-interface=WAN1 action=mark-connection new-connection-mark=WAN1
add chain=forward in-interface=WAN2 action=mark-connection new-connection-mark=WAN2

But I think there is no need... connections has to be already marked when entering from internet or going to internet.
Forward is for traffic that passing through the router.
Input and output are chain to use for traffic that is entering or going to internt.
Maybe I misunderstood ?

sindy · Tue May 07, 2019 4:15 pm

Normally you mark each connection when handling its initial packet, so that you could handle all packets belonging to that connection the same way. Isolated rules posted above do not show the context so they may be correct if rules preceding them don't let other-than-initial packets of each connection to reach these two, or wrong if they do.

I assume your phones on LAN register to the PBX behind WAN; if so, the signalling (SIP) connections are always initiated from LAN. However, the first RTP packet of each call may come from a different direction, so connection marking for in-interface=WAN2 may also be important.

Samot · Tue May 07, 2019 5:18 pm

I'm not sure this is the wisest route to go here. See, the PBX is going to set it's RTP IP to one of those WAN IP addresses. That's how it works. So if you have two WAN IPs and make a call or receive a call, the PBX needs to know what IP to assign for RTP to receive the media back. At any point during that call you could end up changing that IP. A Re-INVITE, holding, parking, etc, etc.. This could cause audio issues if the other side doesn't get updated properly or routed properly.

There is also the fact this is a Request/Reply transaction so there is no guarantee that the request makes it to the destination and the reply makes it back to the source. SIP/VoIP, whatever you want to call it, already has a low threshold for latency, delays, jitter, etc.. I personally (and considering Voice/Telephony is what I do and I use MTs for that need) would never do this. There's no need to split the traffic like this for 80Kbps of traffic. Seriously, it's 80Kbps of traffic. That means you can support a half dozen calls with just about 600Kbps of traffic.

So yeah when you think about it what is the point of load balancing less than 1Mbps of traffic for voice when you'll barely touch that. You would need to hit almost 13 concurrent active calls for you to peak at 1Mbps of traffic just for SIP. While bandwidth is an important factor in the quality of the call, it's not about having a huge pipe because you have 6 phones it's about the quality of the bandwidth. As I said, low tolerance for latency, delay and jitter which for normal TCP/web traffic isn't a thing but you hit 10-15 ms of delay/jitter/latency your call is losing serious quality. Get up to almost 30 it's basically garbage.

Yeah, I just don't see the need to load balance such a low amount of traffic and split it up among multiple backbone pipes for a single call. You're introducing numerous factors that will impact the call and overall, how to troubleshoot the call. Is the call being choppy because of? Is it because it's delayed in reforming the packets on your side, it is delayed because one of the routes is having an issue and it might not even be an issue locally you can see but between them and their peers? If you're sending that audio over 2-3 connections and you start having issues, you now have 2-3 connections you have to figure out is the cause of the issue (or not).

Technetium · Tue May 07, 2019 8:05 pm

I have only 1 line. I measured the traffic during a call, it's a maximum of 100Kb.
Phones are not in the lan of the mikrotik router. Are on another lan managed directly from PBX (pbx has it's own poe router/switch inside).

VoIP connections is excluded from PCC using the rules that force traffic through the WAN2 interface, so it will use only WAN2.
I've assumed also that inocoming call can only use WAN2 to reach my PBX. The VoIP server only know one IP. The IP you use to set up the connection from PBX to the VoIP server.

Forward rules are the latest rules in mangle.
The fill mangle rules list:

/ip firewall mangle
add action=accept chain=prerouting disabled=no dst-address=10.0.1.0/24 comment="Accept da WAN1"
add action=accept chain=prerouting disabled=no dst-address=10.0.2.0/24 comment="Accept da WAN2"

#Identify which WAN interface the traffic came in
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no in-interface=WAN1 new-connection-mark=WAN1 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no in-interface=WAN2 new-connection-mark=WAN2 passthrough=yes

#Stream pcc
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-type=!local in-interface=bridge src-address=192.168.1.100 new-connection-mark=Voip_WAN2 passthrough=yes comment="Voip connection mark WAN2"
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-type=!local in-interface=bridge new-connection-mark=WAN1 passthrough=yes per-connection-classifier=both-addresses:2/0 comment="PCC stream WAN1"
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no dst-address-type=!local in-interface=bridge new-connection-mark=WAN2 passthrough=yes per-connection-classifier=both-addresses:2/1 comment="PCC stream WAN2"

#Connection routing mark prerouting chain
add action=mark-routing chain=prerouting connection-mark=Voip_WAN2 disabled=no in-interface=bridge new-routing-mark=WAN2-mark passthrough=yes comment="Voip routing mark WAN2_mark chain prerouting"
add action=mark-routing chain=prerouting connection-mark=WAN1 disabled=no in-interface=bridge new-routing-mark=WAN1-mark passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2 disabled=no in-interface=bridge new-routing-mark=WAN2-mark passthrough=yes 

#Connection routing mark output chain
add action=mark-routing chain=output connection-mark=Voip_WAN2 disabled=no new-routing-mark=WAN2-mark passthrough=yes comment="Voip routing mark WAN2_mark chain output"
add action=mark-routing chain=output connection-mark=WAN1 disabled=no new-routing-mark=WAN1-mark passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2 disabled=no new-routing-mark=WAN2-mark passthrough=yes

add chain=forward in-interface=WAN1 action=mark-connection new-connection-mark=WAN1
add chain=forward in-interface=WAN2 action=mark-connection new-connection-mark=WAN2

add action=mark-packet chain=postrouting connection-mark=Voip_WAN2 new-packet-mark=fromVoip

Technetium · Wed May 08, 2019 10:38 pm

I've checked the queue tree in a test setup.

The Voip queue takes all bandwidth up to its maximum limit even if in the other queue there is upload activity. I thought queue would split in half the bandwidth, after satisfying the minimum bandwidth imposed by limit-at.
How bandwidth division works in a queue?

sebastia · Wed May 08, 2019 11:45 pm

Documentation is not clear on that point: https://wiki.mikrotik.com/wiki/Manual:HTB.
One example has such situation, but the effect/goal is not elaborated.

Then again is that a valid situation for you? I would think not: voip should have higher prio, and it's volume will be much smaller than rest in most cases.

sindy · Wed May 08, 2019 11:51 pm

The Voip queue takes all bandwidth up to its maximum limit even if in the other queue there is upload activity.

This is exactly what you need. The VoIP must get all it wants because otherwise all the calls will be unusable. But it may be that you've marked some other than VoIP traffic by mistake. Roughly, each active voice call needs about 100 kbit/s per direction. Video calls are a different story. Signalling traffic (control of the calls) occupies a negligible bandwidth but needs to have at least the same priority like the voice streams (RTP) as you need to be able to terminate a call even if the bandwidth is stuffed with the voice streams.

How bandwidth division works in a queue?

Each queue within the tree gets a guaranteed minimum bandwidth (but it is your responsibility to assign it in such a way that it is physically possible, and you may guarantee zero) and has a maximum allowed bandwidth which it can get if other queues do not make use of their assignments.

So in practical terms, the VoIP queue must have all the bandwidth it may possibly need guaranteed, and the sum of eventual guaranteed bandwidths of all queues must not exceed the physical bandwidth of the link.

Technetium · Thu May 09, 2019 11:02 am

Documentation is not clear on that point: https://wiki.mikrotik.com/wiki/Manual:HTB.
One example has such situation, but the effect/goal is not elaborated.

Then again is that a valid situation for you? I would think not: voip should have higher prio, and it's volume will be much smaller than rest in most cases.

The situation i tested is the worst case: 1 Voip call during an upload of a recorded message in the voice mail. The voice quality is fine for me. I will set a higher priority on VoIP queue.
Simply I expected that the bandwidth, in such case, was split about in two. But during the test the Voip queue took all the bandwidth. On the normal queue, the upload was interrupted due to timeout. There were no bandwidth for the "normal client" queue.

Technetium · Thu May 30, 2019 12:24 am

The VoIP PBX isn't working very well.
It is working but sometimes an incoming call is mute. No audio is received.

Maybe because the PBX port are dst-natted?
Port 5060, 5061 and a range from 8000-10000 are dst- natted to the IP of the PBX.

The packet involved in a dst-nat are processed in input or output chain ? or are in the forward chain?
Now I have this suspect because the counter of the VoIP packet marking rule seems not counting all traffic.

sindy · Sat Jun 01, 2019 10:30 am

The NAT processing is even more complex. The rules in srcnat and dstnat chains are checked only for initial packet of each new "connection" (TCP session, UDP stream, ICMP request/response sequences, ...) and the result of the rule is stored in the connection's data, so all subsequent packets are handled accordingly depending on they direction. So the actual treatment is done in the connection tracking module. See the Routing Diagram and Packet Flow Chains, the rest of the page is also worth having a look.

Try /ip firewall connection print detail where srcnat or dstnat.

Please post the export of the Mikrotik's configuration following the hint in my automatic signature, and a drawing of the network topology - position of the phones, PBX and WAN (a photo of a handmade one is enough).

Technetium · Mon Jun 03, 2019 1:26 pm

Here the export:

/interface bridge
add admin-mac=XXXXX auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether1 ] name=WAN1
set [ find default-name=ether2 ] name=WAN2
/interface ethernet switch port
set 0 default-vlan-id=0
set 1 default-vlan-id=0
set 2 default-vlan-id=0
set 3 default-vlan-id=0
set 4 default-vlan-id=0
set 5 default-vlan-id=0
set 6 default-vlan-id=0
set 7 default-vlan-id=0
set 8 default-vlan-id=0
set 9 default-vlan-id=0
set 10 default-vlan-id=0
set 11 default-vlan-id=0
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
add name=dhcp_pool1 ranges=192.168.1.2-192.168.1.254
add name=dhcp_vlan20 ranges=192.168.178.2-192.168.178.254
/ip dhcp-server
add add-arp=yes address-pool=dhcp_pool1 disabled=no interface=bridge name=\
    dhcp_server_local
/queue tree
add max-limit=3700k name=uploadWAN2 parent=WAN2
add limit-at=1M max-limit=3700k name=Voip_upload_WAN2 packet-mark=fromVoip \
    parent=uploadWAN2 priority=3
add bucket-size=0.2 limit-at=1M max-limit=3700k name=otherClientWAN2 \
    packet-mark=no-mark parent=uploadWAN2
/interface bridge port
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=ether6
add bridge=bridge comment=defconf interface=ether7
add bridge=bridge comment=defconf interface=ether8
add bridge=bridge comment=defconf interface=ether10
add bridge=bridge comment=defconf interface=sfp-sfpplus1
add bridge=bridge interface=ether9
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=WAN1 list=WAN
add interface=WAN2 list=WAN
/ip address
add address=192.168.1.1/24 comment=defconf interface=bridge network=\
    192.168.1.0
/ip arp
add address=192.168.1.246 interface=bridge mac-address=XXXX
/ip dhcp-client
add comment=defconf dhcp-options=hostname,clientid disabled=no interface=WAN1
add dhcp-options=hostname disabled=no interface=WAN2
/ip dhcp-server network
add address=192.168.1.0/24 comment=defconf dns-server=\
    8.8.8.8,8.8.4.4,10.0.1.1,10.0.2.1 gateway=192.168.1.1 netmask=24
add address=192.168.178.0/24 dns-server=8.8.8.8,8.8.4.4 gateway=192.168.178.1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4
/ip dns static
add address=192.168.88.1 name=router.lan
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related disabled=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=accept chain=prerouting comment="Accept da WAN1" dst-address=\
    10.0.1.0/24
add action=accept chain=prerouting comment="Accept da WAN2" dst-address=\
    10.0.2.0/24
add action=mark-connection chain=prerouting connection-mark=no-mark \
    in-interface=WAN1 new-connection-mark=WAN1 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark \
    in-interface=WAN2 new-connection-mark=WAN2 passthrough=yes
add action=mark-connection chain=prerouting comment=\
    "Voip connection mark WAN2" connection-mark=no-mark dst-address-type=\
    !local in-interface=bridge new-connection-mark=Voip_WAN2 passthrough=yes \
    src-address=192.168.1.246
add action=mark-connection chain=prerouting comment="PCC stream WAN1" \
    connection-mark=no-mark dst-address-type=!local in-interface=bridge \
    new-connection-mark=WAN1 passthrough=yes per-connection-classifier=\
    both-addresses:2/0
add action=mark-connection chain=prerouting comment="PCC stream WAN2" \
    connection-mark=no-mark dst-address-type=!local in-interface=bridge \
    new-connection-mark=WAN2 passthrough=yes per-connection-classifier=\
    both-addresses:2/1
add action=mark-routing chain=prerouting comment=\
    "Voip routing mark WAN2_mark chain prerouting" connection-mark=Voip_WAN2 \
    in-interface=bridge new-routing-mark=WAN2-mark passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN1 in-interface=\
    bridge new-routing-mark=WAN1-mark passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2 in-interface=\
    bridge new-routing-mark=WAN2-mark passthrough=yes
add action=mark-routing chain=output comment=\
    "Voip routing mark WAN2_mark chain output" connection-mark=Voip_WAN2 \
    new-routing-mark=WAN2-mark passthrough=yes
add action=mark-routing chain=output connection-mark=WAN1 new-routing-mark=\
    WAN1-mark passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2 new-routing-mark=\
    WAN2-mark passthrough=yes
add action=mark-packet chain=postrouting comment="Voip PBX packet mark" \
    connection-mark=Voip_WAN2 new-packet-mark=fromVoip passthrough=yes
add action=mark-connection chain=forward in-interface=WAN1 \
    new-connection-mark=WAN1
add action=mark-connection chain=forward in-interface=WAN2 \
    new-connection-mark=WAN2
add action=mark-connection chain=forward comment=\
    "Voip connection mark WAN2 chain forward" in-interface=bridge \
    new-connection-mark=WAN2 src-address=192.168.1.246
/ip firewall nat
add action=dst-nat chain=dstnat comment="PBX 443 su WAN1" dst-port=443 \
    in-interface=WAN1 protocol=tcp to-addresses=192.168.1.246 to-ports=443
add action=dst-nat chain=dstnat comment="PBX 443 su WAN2" dst-port=443 \
    in-interface=WAN2 protocol=tcp to-addresses=192.168.1.246 to-ports=443
add action=dst-nat chain=dstnat comment="Server  su WAN1 tcp" \
    dst-port=25184 in-interface=WAN1 protocol=tcp to-addresses=192.168.1.253 \
    to-ports=443
add action=dst-nat chain=dstnat comment="Server  su WAN1 udp" \
    dst-port=25184 in-interface=WAN1 protocol=udp src-port="" to-addresses=\
    192.168.1.253 to-ports=443
add action=dst-nat chain=dstnat comment="Server  su WAN2  tcp" \
    dst-port=25184 in-interface=WAN2 protocol=tcp src-port="" to-addresses=\
    192.168.1.253 to-ports=443
add action=dst-nat chain=dstnat comment="Server  su WAN2 udp" \
    dst-port=25184 in-interface=WAN2 protocol=udp src-port="" to-addresses=\
    192.168.1.253 to-ports=443
add action=dst-nat chain=dstnat comment="PBX 5060 TCP" dst-address=\
    10.0.1.10 dst-port=5060 protocol=tcp to-addresses=192.168.1.246 to-ports=\
    5060
add action=dst-nat chain=dstnat comment="PBX 5060 TCP" dst-address=\
    10.0.2.10 dst-port=5060 protocol=tcp to-addresses=192.168.1.246 to-ports=\
    5060
add action=dst-nat chain=dstnat comment="PBX 5060 UDP" dst-address=\
    10.0.1.10 dst-port=5060 protocol=udp to-addresses=192.168.1.246 to-ports=\
    5060
add action=dst-nat chain=dstnat comment="PBX 5060 UDP" dst-address=\
    10.0.2.10 dst-port=5060 protocol=udp to-addresses=192.168.1.246 to-ports=\
    5060
add action=dst-nat chain=dstnat comment="PBX 5060 TCP" dst-address=\
    10.0.1.10 dst-port=5061 protocol=tcp to-addresses=192.168.1.246 to-ports=\
    5061
add action=dst-nat chain=dstnat comment="PBX 5060 TCP" dst-address=\
    10.0.2.10 dst-port=5061 protocol=tcp to-addresses=192.168.1.246 to-ports=\
    5061
add action=dst-nat chain=dstnat comment="PBX 5060 UDP" dst-address=\
    10.0.1.10 dst-port=5061 protocol=udp to-addresses=192.168.1.246 to-ports=\
    5061
add action=dst-nat chain=dstnat comment="PBX 5060 UDP" dst-address=\
    10.0.2.10 dst-port=5061 protocol=udp to-addresses=192.168.1.246 to-ports=\
    5061
add action=dst-nat chain=dstnat comment="PBX Range 10000-15000 UDP" \
    dst-address=10.0.1.10 dst-port=10000-15000 protocol=udp to-addresses=\
    192.168.1.246 to-ports=10000-15000
add action=dst-nat chain=dstnat comment="PBX Range 10000-15000 UDP" \
    dst-address=10.0.2.10 dst-port=10000-15000 protocol=udp to-addresses=\
    192.168.1.246 to-ports=10000-15000
add action=masquerade chain=srcnat out-interface=WAN1
add action=masquerade chain=srcnat out-interface=WAN2
/ip route
add check-gateway=ping comment="WAN 2 marked route" distance=3 gateway=\
    10.0.2.1 routing-mark=WAN2-mark
add check-gateway=ping comment="WAN 1 marked route" distance=1 gateway=\
    10.0.1.1 routing-mark=WAN1-mark
add comment="Defult route WAN 2" distance=1 gateway=10.0.2.1
add comment="Default route WAN 1" distance=1 gateway=10.0.1.1
/system clock
set time-zone-name=Europe/Rome
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool sniffer
set filter-ip-address=192.168.1.253/32

The network topology is in the image below.

sindy · Mon Jun 03, 2019 5:55 pm

Phones are not in the lan of the mikrotik router. Are on another lan managed directly from PBX (pbx has it's own poe router/switch inside).

Does the PBX do anything with the RTP streams? I mean, when the remote party is behind your WAN uplink, does the PBX send the phone's IP address in the c= line of the SDP or its own one? What is the remote party, a phone in the internet connecting to your PBX or your VoIP provider's exchange?

There are a lot of things which can go wrong, especially when you use Mikrotik's SIP ALG functionality in firewall and there is anything else than just individual phones which send SIP and RTP from the same IP address on the LAN side. So if the PBX hides the existence of the phones on its mikrotik-facing interface and sends/expects everything on its own IP address, it looks as a single phone to the SIP ALG so the ALG works properly; if it doesn't (it is enough that it uses one IP address for SIP and another one for RTP), the ALG creates a mess.

Technetium · Mon Jun 03, 2019 7:14 pm

On the other side, external there is the server of the VoIP provider. The PBX is internal and manages all the phones network.

Generally receiving a call, on the other side there may be any device (a PSTN phone, a mobile phone.. a Voip device..)

The problem is that sometimes a call result in a mute call (one way audio, and the audio is not received), sometimes answering a phone call the call dropped.
Sometimes the PBX seems unable to reach the VoIP server of the provider.

But the rule in mangle seems to be correct to route every PBX connection to WAN 2.

sindy · Mon Jun 03, 2019 10:03 pm

On the other side, external there is the server of the VoIP provider. The PBX is internal and manages all the phones network.

Which doesn't answer my question. Do you know whether the RTP from the internal phone towards the VoIP provider reaches Mikrotik from the IP of the phone or from the IP of the PBX?

Generally receiving a call, on the other side there may be any device (a PSTN phone, a mobile phone.. a Voip device..)

If you mean that your PBX should be able to talk to remote VoIP devices directly, not via VoIP provider's gear which can deal with NAT at your end, there is no way how it could work unless your PBX would have a public IP.

The problem is that sometimes a call result in a mute call (one way audio, and the audio is not received)

This could be related to use of phone's own IP as RTP source (no RTP tunneling through the PBX) as said above.

sometimes answering a phone call the call dropped.
Sometimes the PBX seems unable to reach the VoIP server of the provider.

These two can be related to

occasional malfunctioning policy routing (packets taking the wrong WAN),
malfunctioning QoS if the upload traffic through the "VoIP WAN" is so heavy that SIP packets which by mistake don't get the priority treatment can't make it through from the PBX to the VoIP provider,
the incoming (download) WAN traffic on the "VoIP WAN" being so heavy that the VoIP provider's responses don't fit because you do not or can not throttle forwarding of that other download traffic to your LAN clients. The ISPs en route between the VoIP provider and your device most likely ignore any DSCP marking eventually set by the VoIP provider, so you have to take your download bandwidth, subtract from it the bandwidth you need for VoIP, subtract from it the bandwidth other UDP streams occupy (if any), and mark any TCP traffic coming in through that WAN and heading to LAN devices for throttling it by a queue to the bandwidth which remains after those subtractions.

Technetium · Mon Jun 03, 2019 10:43 pm

Which doesn't answer my question. Do you know whether the RTP from the internal phone towards the VoIP provider reaches Mikrotik from the IP of the phone or from the IP of the PBX?

RTP streams are directed and originate (outgoing stream) to the PBX IP. The PBX has it's own router (and switch) inside that directly manage a "subnet" only for phones.

These two can be related to
occasional malfunctioning policy routing (packets taking the wrong WAN),
malfunctioning QoS if the upload traffic through the "VoIP WAN" is so heavy that SIP packets which by mistake don't get the priority treatment can't make it through from the PBX to the VoIP provider

It's what i'm asking. Can be that with my config (posted above) connection from and to the PBX are routed on the wrong WAN ?
Why a packets can take the wrong WAN ?

Observing the queue tree counter, no packets from the PBX IP where dropped.

sindy · Tue Jun 04, 2019 12:17 am

Which doesn't answer my question. Do you know whether the RTP from the internal phone towards the VoIP provider reaches Mikrotik from the IP of the phone or from the IP of the PBX?
RTP streams are directed and originate (outgoing stream) to the PBX IP. The PBX has it's own router (and switch) inside that directly manage a "subnet" only for phones.

"routing" and "managed NAT" is not the same thing. I don't understand what you wrote regarding "RTP streams are directed and originate (outgoing stream) to the PBX IP".

The PBX tells the phone to send RTP to itself (the PBX) or tells it to send it to the IP of the VoIP provider? Also, does it tell the VoIP provider to send RTP to itself (the PBX) or to the IP of the phone?

These two can be related to
It's what i'm asking. Can be that with my config (posted above) connection from and to the PBX are routed on the wrong WAN ?
Why a packets can take the wrong WAN ?

Observing the queue tree counter, no packets from the PBX IP where dropped.

Your mangle rules in chain=forward are useless as they duplicate the rules in prerouting, but should not be harmful. Also the rule "Voip routing mark WAN2_mark chain output" seems useless to me as chain output only handles packets sent by Mikrotik itself, not transiting through it.

To get a connection mark Voip_WAN2, the packet must come from the IP of the PBX. That is likely be true for all signalling, but you still haven't clearly stated whether it is also true for RTP sent by the phones. Unless the PBX mangles the RTP as explained above, the RTP comes from other-than-PBX IP to Mikrotik, so the Mikrotik PCCs it instead of forcing it via WAN2.

What might happen, as the SIP ALG is active and so the SDPs sent to the provider are "corrected" to ask it to send RTP to the public IP address of the WAN, is that the first packet of the RTP stream between the PBX and the VoIP provider comes from the provider side and therefore the connection gets marked as WAN2, not Voip_WAN2. The same can happen for signalling (SIP) if the PBX does not register with the VoIP provider but just sends calls to it and receives calls from it as your dst-nat rules suggest. Without the registration messages, I think the tracked connections in the firewall exist for 3 minutes after the last message like any other UDP connection and then disappear, so if there is signalling silence for some time and then a SIP message comes in from the provider side, the connection is marked with connection-mark WAN2 and if further calls are going in and out before the 3 minutes expire, it stays like that.

Also, you don't shape the incoming traffic as I've explained in the previous post. So you only set some priority to VoIP for upload but don't reserve bandwidth for it on download.

Do 10.0.1.10 and 10.0.2.10 obfuscate actual public IPs or do you really have private IPs on your WANs?

Technetium · Tue Jun 04, 2019 10:17 am

The WAN interface of the mikrotik router are behind the ISP router because its doesen't support full bridge mode so on the WAN interface i have the private IP.

The RTP stream received to the mikrotik router have the PBX ip because the PBX is a router that manage the phone network. So the PBX translate all stream through its interface.
The phones are connected directly to the PBX internal switch and it have the subnet ip (192.168.10.XXX).

The connection mark "Voip_WAN2" is used only to mark the packet in postrouting. Packet mark is used in the upload queue to set a high priority for he VoIP PBX traffic.

The same can happen for signalling (SIP) if the PBX does not register with the VoIP provider but just sends calls to it and receives calls from it as your dst-nat rules suggest.

Without registering to the VoIP provider no external call can be made or received.

sindy · Tue Jun 04, 2019 11:23 am

The WAN interface of the mikrotik router are behind the ISP router because its doesen't support full bridge mode so on the WAN interface i have the private IP.

So you have another set of port forwarding rules on the ISP router. This can cause surprises depending on how the ISP's router handles NAT. But you haven't answered the other question, whether the PBX registers with the VoIP provider's exchange or whether it is configured as a trunk at the VoIP provider, i.e. the VoIP provider sends its INVITEs to your public WAN IP.

You can use /ip firewall connection print detail where dst-address~":5060" or src-address~":5060" to check the status of the SIP connection - what source address it uses on the WAN (it is called reply-dst-address there) and what connection-mark it has. If the PBX is not registering itself with the VoIP provider's exchange, the connection will have a remaining lifetime of up to 3m and thus it may even disappear and be re-created as I wrote earlier. So in that case it may get re-created by an INVITE coming form outside and the connection-mark may be incorrect; in such case you'll have to add another action=mark-connection rule checking for the source address of the packet to be the one of the VoIP exchange and if so, assigning the Voip-WAN2 value of the connection-mark.

After a successful registration, the connection lifetime is set to 60m (regardless the actual expiration time of the registration) and it usually never disappears because re-registration takes place sooner.

The RTP stream received to the mikrotik router have the PBX ip.

I hope you are sure, not guessing.

1. the PBX is a router that manage the phone network.
2. the PBX translate all stream through its interface.

These two statements do not necessarily mean the same, and I'm still not sure we get each other right as none of us is a native English speaker.

A router can be a mere router - a packet received on one interface is forwarded out via another interface, and nothing is changed in the packet's L3 and above (only the L2 changes).
A router can do a "plain NAT" - in this case, source and/or destination address may change as a packet is forwarded, but the information about the change is not used in any related communication.
A router can have a SIP ALG in it (like Mikrotik), where the information about the (planned in advance) change of address and port of a UDP (RTP) stream passing through the NAT part is used to update the SDPs in SIP messages, so that the recipient of such SDP doesn't need to learn from where the RTP actually arrives because it gets, in the modified SDP, the address and port at which the SIP ALG router expects it.

A PBX can do a mere routing, but it is not the same as doing "media proxy" or "RTP forwarding", which is what I've described in the post before - instead of just delivering the SDPs unchanged between the two participants of the call, it replaces the connection-address in each SDP as it forwards it, indicating one of its own IPs as a destination, and forwards the RTP packets between the two sockets while changing their source and destination addresses.

I'd suggest you to use tool sniffer quick to see whether RTP is arriving and leaving during those silent calls, and sniffing into file will let you see the contents of the SDPs using Wireshark.

VoIP with load balancing PCC

VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Re: VoIP with load balancing PCC

Who is online