Community discussions

MikroTik App
  4. RouterOS
  5. General

Weird MTU problem on LTE failover connection

Post Reply
 
ishanjain
just joined
Topic Author
Posts: 4
Joined: Tue Sep 29, 2020 8:40 am

Weird MTU problem on LTE failover connection

Thu Feb 18, 2021 10:55 am

My router is Mikrotik RB450GX4 and I am running 6.48.1.

Here is what my Network looks like,

LTE Modem -> Switch -> [VLAN #2] -> lte-vlan interface
Wifi AP ----------^ -> [VLAN #1] -> 2f-switch interface

Switch tags the traffic with VLAN Tags.

The LTE modem is, Huawei 5G CPE Pro and I am using Airtel LTE.

I have confirmed with `ping -M do -c 1330 <ip>` and from the LTE interface stats on my phone that Airtel is using 1358 as the MTU value.

I have configured 1500 L3 MTU on lte-vlan interface & 1594(4 bytes for vlan tag) L2 MTU.

The issue I am seeing is, HTTP Requests to some sites like Google work fine but requests to any other sites get stuck at `SYN_SENT` stage in the TCP handshake. ICMP is working perfectly fine.

I have already checked nearly a dozen threads in the forum where people have had similar issues and I asked on Mikrotik IRC, Everything seems to be pointing to a MTU issue.

Someone on IRC suggested I try clamping the MTU to MSS and I also found that exact suggestion in pretty much all the threads here so I added a rule like,
Code: Select all
/ip firewall mangle add chain=forward action=change-mss new-mss=clamp-to-pmtu passthrough=yes tcp-flags=syn protocol=tcp out-interface=lte-vlan log=no log-prefix="change-mss"
but I haven't _really_ had success with it. Traffic is still getting stuck at SYN_SENT stage. I am not sure what else to try to debug/fix this problem and I have not found any other suggestions either so, Any help here is much appreciated.
Top
 
ishanjain
just joined
Topic Author
Posts: 4
Joined: Tue Sep 29, 2020 8:40 am

Re: Weird MTU problem on LTE failover connection

Fri Feb 19, 2021 2:02 pm

BUMP
(This post was approved quite late and has already shifted down in the list a lot :/)
Top
 
sindy
Forum Guru
Forum Guru
Posts: 6875
Joined: Mon Dec 04, 2017 9:19 pm

Re: Weird MTU problem on LTE failover connection

Fri Feb 19, 2021 11:20 pm

Being stuck at syn_sent doesn't seem like an MTU thing to me. The initial "three way handshake" consists of packets which have zero payload size, and syn_sent means that even the the SYN+ACK response from the server, which simply cannot be lost due to insufficient MTU because it consists of just the IP and TCP headers without any payload, has not arrived.

So I'd use sniffing to see how the initial exchange in connections to these servers looks like when established via the main WAN, in order to make sure that the servers don't behave in some unusual way, and then sniff on the LTE interface to see whether the SYN+ACK comes or not.

And I'd rather suspect some NAT issue, causing the mobile ISP to drop the requests. Post the config, see my automatic signature below regarding anonymisation.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
Top
 
ishanjain
just joined
Topic Author
Posts: 4
Joined: Tue Sep 29, 2020 8:40 am

Re: Weird MTU problem on LTE failover connection

Sun Feb 21, 2021 8:38 am

Here is my complete RouterOS configuration.
(I removed config related to Queue Trees and mangle rules related to the queue tree)
Code: Select all
# feb/21/2021 11:26:55 by RouterOS 6.48.1
# software id = HH6E-47K7
#
# model = RB450Gx4
# serial number = B8D00B9BCFD4
/interface bridge
add admin-mac=C4:AD:34:9A:95:76 auto-mac=no comment="Default Bridge" name=\
    bridge
/interface ethernet
set [ find default-name=ether1 ] comment="BSNL ONT" mac-address=\
    D8:07:B6:B4:7E:9F mtu=1520
set [ find default-name=ether2 ] comment="2F Switch"
set [ find default-name=ether3 ] comment="1F TV"
set [ find default-name=ether4 ] comment="Raspberry Pi"
set [ find default-name=ether5 ] comment="1F Wireless AP" poe-out=off
/interface pppoe-client
add comment="BSNL WAN" disabled=no interface=ether1 keepalive-timeout=5 name=\
    pppoe-bsnl user=<user-id>@ftth.bsnl.in
/interface vlan
add comment="2F Wireless AP + Switch" interface=ether2 name=lan-2f-vlan \
    vlan-id=1
add comment="LTE Failover 2F Switch" interface=ether2 mtu=1280 name=lte-vlan \
    vlan-id=2
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add comment=ONT name=ExternalMainLAN
add comment="LTE Modem" name=ExternalFailoverLAN
/ip pool
add name=lan-dhcp ranges=192.168.2.15-192.168.2.254
/ip dhcp-server
add address-pool=lan-dhcp disabled=no interface=bridge lease-time=30m name=\
    defconf
/ppp profile
set *0 only-one=yes use-mpls=no use-upnp=no
/queue type
set 0 kind=sfq sfq-perturb=10
set 9 kind=sfq
/snmp community
set [ find default=yes ] addresses=192.168.2.0/24
/interface bridge port
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf frame-types=admit-only-vlan-tagged \
    interface=lan-2f-vlan
/ip firewall connection tracking
set icmp-timeout=30s loose-tcp-tracking=no tcp-close-wait-timeout=1m \
    tcp-fin-wait-timeout=2m tcp-last-ack-timeout=30s \
    tcp-syn-received-timeout=1m tcp-syn-sent-timeout=2m \
    tcp-time-wait-timeout=2m udp-stream-timeout=2m udp-timeout=30s
/ip neighbor discovery-settings
set discover-interface-list=LAN
/ip settings
set accept-source-route=yes
/interface detect-internet
set detect-interface-list=WAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=pppoe-bsnl list=WAN
add comment=ONT interface=ether1 list=ExternalMainLAN
add interface=lte-vlan list=WAN
add comment="LTE Modem" interface=lte-vlan list=ExternalFailoverLAN
/ip address
add address=192.168.2.1/24 comment=defconf interface=bridge network=\
    192.168.2.0
add address=192.168.1.2/24 comment="Static IP for ONT" interface=ether1 \
    network=192.168.1.0
add address=192.168.8.101 comment="Static IP for LTE Modem" disabled=yes \
    interface=lte-vlan network=192.168.8.0
/ip dhcp-client
add add-default-route=no disabled=no interface=lte-vlan use-peer-dns=no \
    use-peer-ntp=no
/ip dhcp-server config
set store-leases-disk=never
/ip dhcp-server lease
add address=192.168.2.5 client-id=1:a4:c4:94:5f:18:74 mac-address=\
    A4:C4:94:5F:18:74 server=defconf
add address=192.168.2.3 client-id=\
    ff:26:38:ff:d9:0:2:0:0:ab:11:f6:c2:2e:cf:62:a3:12:8c mac-address=\
    B8:27:EB:08:8E:F2 server=defconf
add address=192.168.2.8 client-id=1:a4:b1:c1:90:59:bb mac-address=\
    A4:B1:C1:90:59:BB server=defconf
add address=192.168.2.224 client-id=1:c:e0:dc:d2:31:2a mac-address=\
    0C:E0:DC:D2:31:2A server=defconf
add address=192.168.2.228 client-id=1:80:ce:b9:a1:e9:67 mac-address=\
    80:CE:B9:A1:E9:67 server=defconf
add address=192.168.2.218 client-id=1:0:1f:3c:de:7d:da mac-address=\
    00:1F:3C:DE:7D:DA server=defconf
/ip dhcp-server network
add address=192.168.2.0/24 comment=defconf dns-server=192.168.2.3 gateway=\
    192.168.2.1 netmask=24
/ip dns
set allow-remote-requests=yes servers=192.168.2.3
/ip dns static
add address=192.168.2.1 comment=defconf name=router.lan
/ip firewall address-list
add address=0.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=172.16.0.0/12 comment=RFC6890 list=not_in_internet
add address=192.168.0.0/16 comment=RFC6890 list=not_in_internet
add address=10.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=169.254.0.0/16 comment=RFC6890 list=not_in_internet
add address=127.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=224.0.0.0/4 comment=Multicast list=not_in_internet
add address=198.18.0.0/15 comment=RFC6890 list=not_in_internet
add address=192.0.0.0/24 comment=RFC6890 list=not_in_internet
add address=192.0.2.0/24 comment=RFC6890 list=not_in_internet
add address=198.51.100.0/24 comment=RFC6890 list=not_in_internet
add address=203.0.113.0/24 comment=RFC6890 list=not_in_internet
add address=100.64.0.0/10 comment=RFC6890 list=not_in_internet
add address=240.0.0.0/4 comment=RFC6890 list=not_in_internet
add address=192.88.99.0/24 comment="6to4 relay Anycast [RFC 3068]" list=\
    not_in_internet
add address=1.1.1.1 comment=DNS list=Cloudflare
add address=1.0.0.1 comment=DNS list=Cloudflare
add address=255.255.255.255 comment=RFC6890 list=not_in_internet
add list=ddos-attackers
add list=ddos-targets
add address=192.168.2.2-192.168.2.254 comment=LAN list=allowed_to_router
add address=192.168.3.0/24 comment="Guest Network" list=guest
add address=192.168.2.6-192.168.2.254 comment="Private Network" list=\
    private_network
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked log-prefix="accept es,re,un"
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN log-prefix="not from lan"
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related disabled=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
add action=accept chain=input comment="Accept access to router" \
    src-address-list=allowed_to_router
add action=drop chain=forward comment="Drop tries to reach non-public addresse\
    s from LAN while still allowing access to Personal VPN Subnet and LTE Mode\
    m" dst-address=!192.168.8.1 dst-address-list=not_in_internet \
    in-interface-list=LAN log-prefix=wan-non-public out-interface-list=WAN
add action=drop chain=forward comment=\
    "Drop incoming from internet which is not public IP" in-interface-list=\
    WAN src-address-list=not_in_internet
add action=accept chain=icmp comment="echo reply" icmp-options=0:0 protocol=\
    icmp
add action=accept chain=icmp comment="net unreachable" icmp-options=3:0 \
    protocol=icmp
add action=accept chain=icmp comment="host unreachable" icmp-options=3:1 \
    protocol=icmp
add action=accept chain=icmp comment=\
    "host unreachable fragmentation required" icmp-options=3:4 protocol=icmp
add action=accept chain=icmp comment="allow echo request" icmp-options=8:0 \
    protocol=icmp
add action=accept chain=icmp comment="allow time exceed" icmp-options=11:0 \
    protocol=icmp
add action=accept chain=icmp comment="allow parameter bad" icmp-options=12:0 \
    protocol=icmp
add action=drop chain=icmp comment="deny all other types"
add action=jump chain=forward comment="Jump to DDOS Detection" \
    connection-state=new in-interface-list=WAN jump-target=detect-ddos
add action=return chain=detect-ddos dst-limit=32,32,src-and-dst-addresses/10s
add action=return chain=detect-ddos dst-limit=32,32,src-and-dst-addresses/10s \
    protocol=tcp tcp-flags=syn,ack
add action=add-dst-to-address-list address-list=ddos-targets \
    address-list-timeout=10m chain=detect-ddos
add action=add-src-to-address-list address-list=ddos-attackers \
    address-list-timeout=10m chain=detect-ddos
/ip firewall mangle
add action=change-mss chain=forward comment="Clamp MSS to PMTU on LTE VLAN" \
    disabled=yes log-prefix=change-mss new-mss=clamp-to-pmtu out-interface=\
    lte-vlan passthrough=yes protocol=tcp tcp-flags=syn
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN
add action=redirect chain=dstnat comment="DNS Hijack" dst-address=\
    !192.168.2.3 dst-port=53 protocol=udp src-address=!192.168.2.3 to-ports=\
    53
add action=redirect chain=dstnat comment="DNS Hijack" dst-address=\
    !192.168.2.3 dst-port=53 protocol=tcp src-address=!192.168.2.3 to-ports=\
    53
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    out-interface-list=ExternalMainLAN
/ip route
add check-gateway=ping comment="Main WAN Route" distance=1 gateway=pppoe-bsnl
add check-gateway=ping comment="LTE Failover Route" distance=2 gateway=\
    lte-vlan
add check-gateway=ping distance=1 dst-address=192.168.8.1/32 gateway=lte-vlan
/ip service
set telnet disabled=yes
set ftp disabled=yes
/ip ssh
set strong-crypto=yes
/ip upnp
set enabled=yes
/ip upnp interfaces
add interface=bridge type=internal
/ppp aaa
set accounting=no
/snmp
set enabled=yes
/system clock
set time-zone-name=Asia/Kolkata
/system identity
set name="Ishan's Mikrotik"
/system logging
add topics=pppoe,debug,packet
add disabled=yes topics=dhcp
add topics=firewall,packet,debug
/system note
set show-at-login=no
/system ntp client
set enabled=yes server-dns-names=time.cloudflare.com
/system package update
set channel=development
/system routerboard settings
set auto-upgrade=yes cpu-frequency=auto
/tool bandwidth-server
set enabled=no
/tool graphing
set store-every=24hours
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool sniffer
set file-name=mtu.issue filter-interface=lte-vlan memory-limit=10000KiB

The LTE Modem is Huawei 5G CPE Pro running in Bridge Mode. It's not really possible for me to inspect traffic on that device because the software is locked down.

I have tried to inspect traffic using packet-sniffer on lte-vlan interface and it there is no response from a remote server after sending `SYN`.
hung.connection.png

And this is only happening with some servers. For example, HTTP requests to `icanhazip.com` work just fine but traffic to anything behind cloudflare(i.e. Anything that is using Cloudflare's DNS proxy) or Mikrotik Forum or *most* other sites is getting stuck.
You do not have the required permissions to view the files attached to this post.
Top
 
ishanjain
just joined
Topic Author
Posts: 4
Joined: Tue Sep 29, 2020 8:40 am

Re: Weird MTU problem on LTE failover connection

Sun Feb 21, 2021 8:57 am

Hey, Thank you so much for responding to this thread!

I just resetted the 5G CPE Pro and now it seems to be working fine. I should've saved logs to submit to Huawei but forgot to do that. :|||
Top
Post Reply

Who is online

Users browsing this forum: Bing [Bot], nickrod50 and 159 guests
  4. RouterOS
  5. General