Overall it is working fine, I can measure 2Gbit/s down and 1 GBit/s which is the offering of my FTTH connection.
Problem:
Recently I realized that this setup blocks incoming FaceTime calls. More precisely FaceTime calls are ringing but when I accept, it hangs and just displays “connecting…”.
I figured out that the problem is solved when I disable one of the WAN connections. So it is likely caused by PCC on dual WANs.
I would appreciate any direction like
a.) did I missed any NAT config?
b.) is FaceTime basically not working in dual WAN setup so I should pin all FaceTime traffic to a dedicated WAN connection out of the two WANs?
Please find my full config below:
# aug/08/2020 19:34:40 by RouterOS 6.45.9
# software id = <removed>
#
# model = RB4011iGS+
# serial number = <removed>
/interface bridge
add admin-mac=<removed> auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether6 ] disabled=yes
set [ find default-name=ether7 ] disabled=yes
set [ find default-name=ether8 ] disabled=yes
/interface pppoe-client
add disabled=no interface=ether9 max-mtu=1480 name=pppoe-out1 password=<removed> service-name=ISP1 use-peer-dns=yes user=<removed>
add disabled=no interface=ether10 max-mtu=1480 name=pppoe-out2 password=<removed> service-name=ISP2 use-peer-dns=yes user=<removed>
/interface ethernet switch port
set 0 default-vlan-id=0
set 1 default-vlan-id=0
set 2 default-vlan-id=0
set 3 default-vlan-id=0
set 4 default-vlan-id=0
set 5 default-vlan-id=0
set 6 default-vlan-id=0
set 7 default-vlan-id=0
set 8 default-vlan-id=0
set 9 default-vlan-id=0
set 10 default-vlan-id=0
set 11 default-vlan-id=0
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=default-dhcp disabled=no interface=bridge lease-time=1d name=defconf
/ppp profile
set *0 on-down="/ip firewall mangle remove [find comment=\$interface];" on-up="/ip firewall mangle add action=accept chain=prerouting disabled=no in-interface=bridge dst-address=\$\"local-address\" comment=\$interface;"
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=sfp-sfpplus1
/ip neighbor discovery-settings
set discover-interface-list=none
/interface detect-internet
set detect-interface-list=WAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=pppoe-out1 list=WAN
add interface=pppoe-out2 list=WAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=192.168.88.0
/ip dhcp-server lease
add address=192.168.88.238 client-id=1:28:80:88:34:f0:a2 mac-address=<removed> server=defconf
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf gateway=192.168.88.1
/ip dns static
add address=192.168.88.238 name=<removed>
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" disabled=yes protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-connection chain=input connection-mark=no-mark in-interface=pppoe-out1 new-connection-mark=ISP1_conn passthrough=no
add action=mark-connection chain=input connection-mark=no-mark in-interface=pppoe-out2 new-connection-mark=ISP2_conn passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=ISP1_conn passthrough=yes per-connection-classifier=src-address-and-port:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=ISP2_conn passthrough=yes per-connection-classifier=src-address-and-port:2/1
add action=mark-routing chain=prerouting connection-mark=ISP1_conn in-interface=bridge new-routing-mark=to_ISP1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=ISP2_conn in-interface=bridge new-routing-mark=to_ISP2 passthrough=no
add action=mark-routing chain=output connection-mark=ISP1_conn new-routing-mark=to_ISP1 passthrough=no
add action=mark-routing chain=output connection-mark=ISP2_conn new-routing-mark=to_ISP2 passthrough=no
add action=accept chain=prerouting comment=*e dst-address=<removed> in-interface=bridge
add action=accept chain=prerouting comment=*f dst-address=<removed> in-interface=bridge
/ip firewall nat
add action=masquerade chain=srcnat out-interface=pppoe-out1
add action=masquerade chain=srcnat out-interface=pppoe-out2
/ip route
add check-gateway=ping distance=1 gateway=pppoe-out1 routing-mark=to_ISP1
add check-gateway=ping distance=1 gateway=pppoe-out2 routing-mark=to_ISP2
add check-gateway=ping distance=1 gateway=pppoe-out1
add check-gateway=ping distance=2 gateway=pppoe-out2
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www-ssl disabled=no
set api disabled=yes
set api-ssl disabled=yes
/ip ssh
set strong-crypto=yes
/ip upnp
set enabled=yes
/system clock
set time-zone-name=Europe/Budapest
/system package update
set channel=long-term
/tool bandwidth-server
set enabled=no
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=none
/tool mac-server ping
set enabled=no
As facetime seems to be SIP + RTP but with some proprietary features, I can imagine that it is unhappy if the RTP is coming from another public IP than the SIP, which easily happens if the PCC rule sends the RTP packets out via another WAN than the one it has chosen for the SIP packets. So unless you can reliably identify facetime packets and force all of them (both SIP and RTP) via the same WAN, you’ll have to reserve an IP address for each iDevice and make Mikrotik route all its traffic via a single WAN.
Anyone has an idea how could I detect FaceTime packets? Maybe some L7 firewall rules?
If it is not possible, then I am currently considering switching back the ISP device from bridge mode into router mode and let the MikroTik router work as DHCP client picking two DHCP addresses from the ISP router. This way I would have only one public IP address of the ISP device but it might introduce some double NAT issues because both the MikroTik router as well as the ISP router would do NAT.
I skimmed through your config and it seems you have two ISPs, both PPPoE clients. I have the same setup, two ISPs, both PPPoE clients.
This is what I used to fix broken HTTPS among other things. And yes sir, aggregated bandwidth will work with this config, I get doubled bandwidth especially in game downloads like Battle.net, Steam etc which I suspect is thanks to their implementation of Multipath TCP.
I have thoroughly tested this config for a month.
Do not forget to take care of:
MTU/MRU values (My ISP2 agreed to implement RFC4638 for me personally)
Port forwarding (I use UPnP as I’m confident in my skills of keeping my systems clean)
My ISP 1 gives public IPv4/IPv6
ISP 2 gives CGNAT IPv4
So stuff like that needs to be accounted for.
Please note I used !dst-address-list as I discovered without it, it causes packet loss on LAN traffic.
/ip firewall mangle
###To ensure LAN traffic is not affected###
add action=accept chain=prerouting in-interface=bridge dst-address-list=not_in_internet
###To ensure what comes in from one interface gets routed through same interface###
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out1 new-connection-mark=ISP1_conn
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out2 new-connection-mark=ISP2_conn
###ForHTTPS we use "both addresses" to avoid breaking it###
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-port=80,443 in-interface=bridge new-connection-mark=ISP1_conn per-connection-classifier=both-addresses:2/0 protocol=tcp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-port=80,443 in-interface=bridge new-connection-mark=ISP2_conn per-connection-classifier=both-addresses:2/1 protocol=tcp
###For other connections, to get full speeds we use "both addresses and ports"###
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet in-interface=bridge new-connection-mark=ISP1_conn per-connection-classifier=both-addresses-and-ports:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet in-interface=bridge new-connection-mark=ISP2_conn per-connection-classifier=both-addresses-and-ports:2/1
###Routing Mark###
add chain=prerouting dst-address-list=!not_in_internet connection-mark=ISP1_conn in-interface=bridge action=mark-routing new-routing-mark=to_ISP1
add chain=prerouting dst-address-list=!not_in_internet connection-mark=ISP2_conn in-interface=bridge action=mark-routing new-routing-mark=to_ISP2
add chain=output dst-address-list=!not_in_internet connection-mark=ISP1_conn action=mark-routing new-routing-mark=to_ISP1
add chain=output dst-address-list=!not_in_internet connection-mark=ISP2_conn action=mark-routing new-routing-mark=to_ISP2
/ip route
add dst-address=0.0.0.0/0 gateway=pppoe-out1 routing-mark=to_ISP1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=pppoe-out2 routing-mark=to_ISP2 check-gateway=ping
For a SIP call, there are a few things to be aware of. The initial SIP management stream, usually in the UDP 5060-5062 range, is like a control channel. The peers authenticate and manage signaling over this path. When there is a call, RTP is typically used. The peers negotiate the RTP peer IPs and ports to be used for each leg of the call.
There are 2 independent call legs --one in each direction (audio/video from you to the remote peer, and audio/video from the remote peer to you). If your SIP peer negotiates the public IP of one path (often determined using STUN), then your router sends that RTP session out the other path, the source packets for the RTP stream are not originating from the negotiated IP, so the remote peer drops/rejects the traffic.
Mikrotik has a feature that looks into SIP traffic help RTP packets negotiate correctly. I’m not sure if it works in your scenario, but you might check to see if it is enabled. It is SIP under the /ip firewall service-ports submenu.
I did simple ping tests to LAN IPs from LAN clients. 1 packet loss per 5000 packets. Weird thing is, it does not show up in the stats of the interfaces. So I don’t know what’s going on.
My script is used with two ISPs, ISP 1 = free public IPv4/IPv6, ISP 2 = CGNAT IPv4.
FaceTime works, other VoIP apps work, gaming works. So try it on your setup and the chain must be prerouting to capture everything.
This is normally the risky part with protocols that have multiple connections and expect them to be from the same source address.
To be more safe, remove at least the “ports” part from that, and probably also select only source address.
(of course when you do the latter, it means that all traffic from a single internal system will always be via the same ISP and external IP, that could be a disadvantage when you have only a small number of systems)
Yes, but that would not only affect http/https (in fact it does not affect those protocols in themselves, but it will affect applications).
It will also affect ftp, sip etc that really have multiple connections within the same protocol.
To avoid any problems I only use src address for the PCC in the large network at work. With 300-1000 users it does not really matter.
When it is a home network it of course will be different, but still I would use only “both addresses” in the PCC and see how it works out.
Name an application that gets broken with “both addresses”.
I’ve tested banking apps/sites like Paypal, local ones, gaming, streaming, FTP from file sharing sites, Google Drive etc.
The majority of popular sites/apps seems to make use of Multi-Path TCP which is where “both addresses” will give you the benefit of aggregated bandwidth.
I wrote “To avoid any problems”. I cannot name an application, but I can envision how there could be problems, and I can be certain there won’t be problems when using only source address.
With 300-1000 phones and laptops being used on the network with all kinds of applications I don’t know about, and no possibility to test them all, I am just keeping on the safe side.
Look up Multi-Path TCP, many applications make use of multiple “WAN” IPs. iOS natively supports, Samsung has been implementing it at the OS level for a decade for instance with their “Dual Wi-Fi/LTE” feature.
What you’re referring to is theoretical at best. No one has ever shown any hard evidence for “broken connections due to multiple source IPs”.
How will using both-addresses end up with “multiple source IPs” in the first place?!
The hash will always be the same for a given pair of addresses - so the resulting WAN will be the same.
Nah, you got it wrong. What the other guy tried telling is, when an app/service/site does multiple connections to multiple destination IPs. Hence the hash will never be the same for each pair of “source IP” and “destination IP” where the latter varies.