Hello everyone.

I expose my problem to you.
I have a network configured using EoIp tunnels, so we have Master tunnels on a mikrotik machine that manages tunnels and where there are also fiber VLANs.
Now, each tunnel on the master is uniquely connected to its slave tunnel on the router of the reference base station.
Then tunnels on the master and tunnels on the slave (rb base station)
On the base station I have AP cambium and mikrotik with various connected clients.

The problem is this.
If I start the download of a file on the browser (for example the 660 mega kali linux iso) after 5/6 minutes the download crashes, at a certain point it stops and goes down to 0 and stops there, until giving network error. This happens even if I try to update my Mac or iPhone iOS. practically the download goes wrong.
this does NOT happen if I use torrents.
This problem is reflected on the whole network both mikrotik and cambium clients, they surf, but if they try to make a download after X minutes it stops but pinging google in that moment the network is online.
I re-checked the MTU of the entire network and it is set to 1500 on the master and slave eoip tunnels and on the ethernet links and APs. the pppoe mikrotik sessions automatically take MTU 1480 while cambium 1492. I did a lot of testing decreasing the MTU value in the eoip tunnel but without success, I also tried to raise it.
Nothing, the download crashes.
I was able to find out that, leaving 1500 on the EoIP tunnels and manually putting 1466 on the pppoe mikrotik session the download works, if I get up to 1467 it goes wrong.
on cambium instead same base station if I put 1466 gives me error.

I tried to set 1466 on the tunnels but without success.

It's definitely an MTU problem but I can't figure out how to fix it.
Some advice?


jac.

Hello,

It seems to be a TCP/MSS problem, take a look at this :

https://wiki.mikrotik.com/wiki/Manual:I ... all/Mangle

I don't think I can put this rule on the tunnel machine.
The PPP sessions run a third CISCO machine. Which only works to authenticate sessions.
I should definitely work on the tunnels, but I don't understand how.
I have to work upstream of the configuration, not on individual customers.

To me it is just unclear how a MTU issue should cause a problem as late as after minutes of download. Normally it pops up immediately when a https session is being set up as already the TLS negotiation maxes out the reported MTU in both directions so if there is a MTU misconfiguration somewhere along the path, the https sessions don't even fully start.

To check whether it really is an MTU issue or not, you can use /ip firewall mangle add chain=prerouting protocol=tcp tcp-flags=syn action=change-mss new-mss=1300 at the PPPoE server for all the TCP connections in both directions. (new-mss=clamp-to-pmtu would not serve the purpose, you have to set a numeric value).

if I try to put it as you say, he tells me

tcp mss clamping not possible in prerouting and input chains

Just let me know if you solved the problem!

if I try to put it as you say, he tells me

tcp mss clamping not possible in prerouting and input chains

OK, sorry, then use chain=postrouting instead.

Nothing to do, I downloaded 280 mb of the 700mb test file and it crashed.

Nothing to do, I downloaded 280 mb of the 700mb test file and it crashed.
I remain of the idea however that I must intervene in some way on the configuration of the EOIP tunnels.

Can you draw the overall scheme? And, if you test using two clients using the same chain of EoIP tunnels, and let each one download the same file and start the download on the second one let's say 20 seconds after you've started it on the first one, does the download stop at both at the same time or also 20 seconds apart? Will the number of bytes transferred be those 280 MB summary volume for both or each will be able to transfer its own 280 MB?

This is the network configuration.

Router mikrotik (CCR1072) where the fiber VLANs arrive and where all EOIP tunnels are configured.
Each master tunnel is directly connected to its slave tunnel configured directly on mikrotik routerboard. The routers of arrival are sometimes ccrs sometimes not, they are all mikrotiks anyway.
Under the routerboards there are AP cambium and AP mikrotik and the clients are connected to these AP . The master and slave tunnels are configured with MTU at 1500.
I did the test you told me, the first file downloaded 550 MB to 770, the second file sent 20 seconds later was blocked before downloading 435MB of 770.

This is the network configuration.

Router mikrotik (CCR1072) where the fiber VLANs arrive and where all EOIP tunnels are configured.
Each master tunnel is directly connected to its slave tunnel configured directly on mikrotik routerboard. The routers of arrival are sometimes ccrs sometimes not, they are all mikrotiks anyway.
Under the routerboards there are AP cambium and AP mikrotik and the clients are connected to these AP . The master and slave tunnels are configured with MTU at 1500.
I did the test you told me, the first file downloaded 550 MB to 770, the second file sent 20 seconds later was blocked before downloading 435MB of 770.

There is nothing new in the network configuration you gave, except that I've finally understood that there is no "master tunnel" and "slave tunnel" but actually "master end of a tunnel" and "slave end of a tunnel". But you haven't stated what is the transport between the machines on which the tunnels end, i.e. nothing about the own MTU of that transport. You haven't stated whether all the EoIP interfaces at the central CCR1072 are connected to the same bridge and there is a single PPPoE server attached to that bridge (as I suspect), or whether one PPPoE server is attached directly to each of the EoIP interfaces (which might ease the load on the CCR1072's CPU a bit).

So maybe follow the hint in my automatic signature for the central router and one of the base station ones.

BTW, while the download is ongoing, what does /tool profile show on the CCR1072? The thing with EoIP is that it tells the upper layers whatever MTU you set on it but of course if the original Ethernet frame uses that MTU, it has to be split into two for transmission over a physical link as it grows by the IP and GRE and sometimes Ethernet or 802.11 headers, so in most cases you end up with two transport frames per each payload one, so whereas the bitrate grows just by a few percent, the packet rate doubles, and the packet rate is what causes the CPU load.

The behaviour you describe cannot be related to MTU directly, something has to overflow somewhere. You can use packet sniffer to see by your own eyes that during the download, the maximum MTU is being used all the time, it is how TCP works - the maximum packet size doesn't change once the session has been established because for the reason above, TCP splices the data stream into as few as possible pieces as large as possible.

Good morning.

To my knowledge there is an upstream radius server that authenticates ppp sessions.
I don't know if it's possible to set MTU on that.
As for other machines between the two tunnels, they confirm that there is nothing.


I did the tests with packet sniffer and opened wireshark for package analysis.
I saw that leaving MTU 1500 on the tunnels and defaulting on the mikrotik (1480), I have on wireshark:

packet length-> 1494
TCP SIZE -> 1440
with these two values ​​the download goes into error


if you change the ppp session instead, from 1480 to 1466 without modifying the tunnel, on wireshark I have:

packet length -> 1480
TCP SIZE -> 1426
with these values ​​the download is successful.


I tried to change the MTU on the tunnels. I realized that THE VALUES of wireshark DO NOT CHANGE but remain as the first test (1494-1440 even if I changed the value of the tunnel from 1500 to 1480) and goes into error.

So all in all you were right and it is an MTU related issue, but it is a weirdly intermittent one as otherwise your downloads would have to break much sooner - as said, when TCP is used to send a file, it tries to slice it into pieces maxing out the smallest available MTU on the path. There could be an exception to this rule if the receiving side would constantly report a small receive window, i.e. the remaining size of the receiving buffer in the TCP stack which can be filled by the payload of the incoming packets. So if the receiving application using that TCP socket is slow to fetch the data from the buffer, the buffer may be smaller than the MSS all the time, so if the sender would decide to send smaller packets rather than to wait for the window to become larger, the available MTU would not be maxed out most of the time, and once it would, the download would fail because the discovered MTU wasn't correct. But for this to happen, several things have to go wrong - namely, some element on the path must be letting through frames larger than the actual MTU somewhere further on the path, but smaller frames must be going for long enough for this to remain unnoticed.

Top to bottom (L4 to L1):

• the SYN and SYN+ACK packets inform the server and client, respectively, about the MSS they can accept, calculating it based on their local MTU. However, TCP packets are sent with "don't fragment" flag set, which means that any router on the path which finds the packet too big to fit to outbound interface's MTU drops it and sends back an ICMP notification saying that it would have to be fragmented and what is the maximum size which will fit. The sender than sends the same data again using a shorter packet according to this information. So during first few attempts, the packet finally reaches the destination and gets acknowledged, and the sender keeps its MSS in sending direction on this reduced value. This process may not happen immediately as the payload of first few packets of a session may be far less than the actual MSS of the path, and it may also happen repeatedly during and ongoing session if there are multiple paths through the network, each with a different MTU.
• the PPPoE gets an information about MTU supported by the Ethernet layer from the EoIP tunnel or from the bridge to which it is connected; the smallest of the MTUs of all ports connected to the bridge is reported as bridge MTU where you use the bridge itself as an interface. You can see it in bridge parameters. The PPPoE itself adjusts the MTU of the underlying L2 interface by subtracting its own 8 bytes of headers, making it 1492 bytes if the L2 interface below has 1500.
• if you set the EoIP MTU to a higher value than the MTU of the physical path minus the size of (ethernet (+ vlan) + ip + gre) headers together, the EoIP algorithm will send IP packets larger than 1500 bytes so the IP stack below will split each packet carrying an EoIP-encapsulated PPPoE packet carrying a TCP packet into two fragments, one which fits to the MTU of its underlying physical interface and a much shorter one carrying what didn't fit to the first one with additional 20 bytes of the IP header. This way, it makes the physical MTU invisible to the PPPoE, but causes the rate of transport IP packets to almost double the rate of the PPPoE packets (almost because not all packets max out the PPPoE's MTU).

PPPoE doesn't have any mechanism to detect the actual MTU of the path. It blindly relies on the MTU reported by its underlying inerface and if a packet doesn't fit, it cannot do anything about it because the upper layer (IP) doesn't track the interface MTU constantly, so starting to report a different MTU would have no effect as nobody would ask again.

So the first thing I keep wondering about is how is it possible that it works most of the time - watching the window size reported in the ACK packets sent by the download recipient should confirm or deny my theory that most of the time the receiving buffer is almost full so the recipient asks for smaller-than-MSS packets.

Another possible explanation is that something is wrong with how the TCP stack fragments or re-assembles the GRE packets, so when the window size becomes lower than usual and the EoIP needs to deliver the frame using two packets and the difference is just one byte in either direction, the splitting or re-assembly fails while for bigger differences it works properly. To find this out, you would have to capture on the physical interface and at the EoIP one simultaneously, and ping using various packet sizes to find out the critical size which causes the issue:/ping x.x.x.x do-not-fragment size=y, using values between, say, 1300 and 1500 one by one, maybe using a script:

local restbl {"0"=>3} ;for i from=1300 to=1520 step=1 do={set ($restbl->$i) [ping re.mo.te.ip count=1 do-not-fragment size=$i]} ; foreach size,rslt in=$restbl do={if ($rslt<3 && $rslt>-1) do={put ($size.": ".$rslt)}}

My problem is that I haven't found any value which would fail in my lab setup, until reaching the path MTU the GRE transport packets go without fragmentation, and then second fragments are added, and the reassembly works fine too:

157 time=6.102 num=315 direction=tx src-mac=CC:2D:E0:C9:B7:E6 dst-mac=64:D1:54:87:38:54 interface=ether1 src-address=192.168.5.1
dst-address=192.168.5.100 protocol=ip ip-protocol=gre size=1512 cpu=1 fp=no ip-packet-size=1498 ip-header-size=20 dscp=0
identification=37708 fragment-offset=0 ttl=255

158 time=6.108 num=317 direction=tx src-mac=CC:2D:E0:C9:B7:E6 dst-mac=64:D1:54:87:38:54 interface=ether1 src-address=192.168.5.1
dst-address=192.168.5.100 protocol=ip ip-protocol=gre size=1513 cpu=1 fp=no ip-packet-size=1499 ip-header-size=20 dscp=0
identification=37964 fragment-offset=0 ttl=255

159 time=6.114 num=319 direction=tx src-mac=CC:2D:E0:C9:B7:E6 dst-mac=64:D1:54:87:38:54 interface=ether1 src-address=192.168.5.1
dst-address=192.168.5.100 protocol=ip ip-protocol=gre size=1514 cpu=1 fp=no ip-packet-size=1500 ip-header-size=20 dscp=0
identification=38220 fragment-offset=0 ttl=255

160 time=6.119 num=321 direction=tx src-mac=CC:2D:E0:C9:B7:E6 dst-mac=64:D1:54:87:38:54 interface=ether1 src-address=192.168.5.1
dst-address=192.168.5.100 protocol=ip ip-protocol=gre size=1514 cpu=1 fp=no ip-packet-size=1500 ip-header-size=20 dscp=0
identification=38476 fragment-offset=0 ttl=255

161 time=6.119 num=322 direction=tx src-mac=CC:2D:E0:C9:B7:E6 dst-mac=64:D1:54:87:38:54 interface=ether1 src-address=192.168.5.1
dst-address=192.168.5.100 protocol=ip ip-protocol=gre size=35 cpu=1 fp=no ip-packet-size=21 ip-header-size=20 dscp=0
identification=38476 fragment-offset=1480 ttl=255

162 time=6.126 num=325 direction=tx src-mac=CC:2D:E0:C9:B7:E6 dst-mac=64:D1:54:87:38:54 interface=ether1 src-address=192.168.5.1
dst-address=192.168.5.100 protocol=ip ip-protocol=gre size=1514 cpu=1 fp=no ip-packet-size=1500 ip-header-size=20 dscp=0
identification=38732 fragment-offset=0 ttl=255

163 time=6.126 num=326 direction=tx src-mac=CC:2D:E0:C9:B7:E6 dst-mac=64:D1:54:87:38:54 interface=ether1 src-address=192.168.5.1
dst-address=192.168.5.100 protocol=ip ip-protocol=gre size=36 cpu=1 fp=no ip-packet-size=22 ip-header-size=20 dscp=0
identification=38732 fragment-offset=1480 ttl=255


If you end up with the same result like me, I'd recommend to ping for a long time (minutes) with the maximum packet size which fits without fragmentation and the do-not-fragment option; if you can see time intervals where such packets do pass and other time intervals where they don't, there must be something on the path that changes the MTU value.

Thank you for your help.
Unfortunately I'm stuck, I've done all the possible tests and I can't solve.

I don't understand why then, lowering the MTU on mikrotik works and on cambium no, this thing is meaningless

country?

the beautiful ITALY!

Too far to just drive by...

ehehehehe (y)