A few weeks ago, a very strange problem started on my network.
Only for pppoe connected clients, (Telegram and facebook messenger) stop working. They work like 5 minutes and then stop again.
I tried the following:
1- Changing the MTU
2- netinstall my mikrotik device (1036).
3- Changin masquerade to src-nat.
4- Setting the pppoe server on bridge instead of the port (I put that port in the bridge).
5- Setting the pppoe server ip as DNS in the pppoe profile.
6- Changing bridge from rstp to none.
7- disabling and enabling ip firewall of the bridge.
8- choosing only (pap) for pppoe server.
I almost tried everything possible to fix this problem and I couldn’t till this moment.
The hotspot users work perfect and everything is working..
ICMP carries many kinds of information, it is not only used for ping. In this particular case, if a packet arrives to a piece of equipment for which it is too large to handle, the equipment normally fragments it. But if such packet is marked with “don’t fragment”, the equipment instead sends back an ICMP packet saying that the packet could not have been processed and what is its MTU. So if this happens but someone in between the two points is blocking ICMP completely, the “not to be fragmented” packet never reaches the destination and the sender never learns why.
Why this problem does not occur with hotspot users, but only with pppoe connected clients?!!
The PPPoE encapsulation occupies part of the packet so the MTU of the encapsulated packets is accordingly smaller than the MTU on the underlying physical interface. There was some issue with Mikrotik’s handling of this the way described above in 6.41, but it should have been fixed in 6.41.1:
*) ppp - fixed change-mss functionality in some specific traffic (introduced in v6.41)
or even use a lower value until it starts working (because I am not sure that if the actual MTU discovery yields a lower value, the one provided by that mangle rule will be overridden).
Also, don’t forget to put that mangle rule to the top of the list.
As I’ve started writing the answer I’ve realized I don’t understand exactly your topology. Does the Mikrotik in question act as a PPPoE server to which other PPPoE clients connect, or as a PPPoE client on the internet uplink? If it is a PPPoE server for other clients, using what kind of interface it is connected to Internet, plain Ethernet or PPPoE (or something else)?
2011 is working as bridge, and the clients are connected after it.
After 2011 there are wireless access points —> stations —> clients
Some of the clients are TP-Link routers others are Ubnt Nanosation devices. And the end-users are connected to those routers using their mobile phones and computers.
go to /tools packet-sniffer at both Mikrotiks and configure sniffing into a file at the 1036’s uplink (internet-facing) interface and at 2011’s interface facing towards the 1036. In both cases, filter by the IP address of a test client (assuming that the 1036 doesn’t do NAT on its internet-facing interface, if it does, it will be much harder to match the flows between the two files). Apply the settings (file name & interface & IP address to be sniffed & file size somewhere like 4000 kB), then press Start.
establish a connection from the test client to the service which is affected by the issue and sniff until the connection experiences problems.
stop the sniffing, download the captured files and analyse them using Wireshark
But in this configuration, I can imagine only the following scenarios:
there is an MTU bottleneck further in the network, the ICMP “fragmentation needed” arrives to the 1036 (because hotspot clients are all happy) but it does not make it to the PPPoE clients because you’ve yourself restricted ICMP in firewall rules of the 1036 (as you say the 2011 is only a bridge). You would identify this case by seeing the icmp “fragmentation needed” in the capture from the 1036 but not in the capture from the 2011
the bottleneck is the PPPoE between the 1036 and the client. in such case:
if the too large packets don’t get through from the client to the service, the PPPoE implementation or settings at the client machines would have to be broken,
if the too large packets don’t get through from the service to the client, the 1036’s PPPoE server should be sending the icmp “fragmentation needed” packets towards the service (i.e. towards internet) but it either doesn’t or you’ve yourself restricted ICMP in firewall rules of the 1036. You would identify this case by seeing too large packets to come from the service towards the client in the capture from the 1036 but not seeing them in the capture from the 2011.
You may also place here the output of “/ip firewall export” on the 1036 to see whether any ICMP restriction exists there - it can be an implicit one if you haven’t explicitly set an exception for ICMP from a “drop the rest” rule in the “forward” chain.
Hm… your overall topology is much more complex with all those actual clients connected to WAPs which themselves are PPPoE clients of the 1036, and with 10 different src-nats at the 1036 and probably other NATs at the WAPs.
So you’ll have to carefully choose the proper IP addresses for filtering while sniffing, and the 4000 kB may not be enough if traffic from many individual clients is NATed to a single common IP address.
I can see there are no firewall filter rules at all in the 1036 configuration, so if you are filtering the ICMP yourself, it happens somewhere else in the network.
It starts with understanding what the issue is, how it fits in your network topology, and then debug what is really going on.
A network with PPPoE (and no RFC4638) will never be completely trouble-free.
It will be worse when there are operators that think that ICMP is dangerous and has to be blocked, and it will be better when “crude fixes” like the mangle rule are applied in the correct place.
