Community discussions

MikroTik App
 
elechero
newbie
Topic Author
Posts: 26
Joined: Sat Jun 06, 2015 3:40 am

Some services are not working with pppoe connection

Mon Feb 12, 2018 8:14 pm

Hi,

A few weeks ago, a very strange problem started on my network.

Only for pppoe connected clients, (Telegram and facebook messenger) stop working. They work like 5 minutes and then stop again.

I tried the following:
1- Changing the MTU
2- netinstall my mikrotik device (1036).
3- Changin masquerade to src-nat.
4- Setting the pppoe server on bridge instead of the port (I put that port in the bridge).
5- Setting the pppoe server ip as DNS in the pppoe profile.
6- Changing bridge from rstp to none.
7- disabling and enabling ip firewall of the bridge.
8- choosing only (pap) for pppoe server.

I almost tried everything possible to fix this problem and I couldn't till this moment.

The hotspot users work perfect and everything is working..

Thanks in advance.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10221
Joined: Mon Jun 08, 2015 12:09 pm

Re: Some services are not working with pppoe connection

Mon Feb 12, 2018 8:59 pm

Try adding this in the ip firewall:
/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu passthrough=yes \
    protocol=tcp tcp-flags=syn
Also make sure you are not blocking all ICMP traffic.
(however, when someone further upstream does that, you will have problems too)
 
elechero
newbie
Topic Author
Posts: 26
Joined: Sat Jun 06, 2015 3:40 am

Re: Some services are not working with pppoe connection

Mon Feb 12, 2018 9:14 pm

Thank you very much for your reply.

I tried the (change-mss in mangle) as you suggested but, unfortunately, it did not fixed the problem.
(however, when someone further upstream does that, you will have problems too)
Can you explain this point more for me, please, because I think this maybe is the problem.
 
elechero
newbie
Topic Author
Posts: 26
Joined: Sat Jun 06, 2015 3:40 am

Re: Some services are not working with pppoe connection

Mon Feb 12, 2018 9:45 pm

Why this problem does not occur with hotspot users, but only with pppoe connected clients?!!
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Some services are not working with pppoe connection

Wed Feb 14, 2018 12:56 am

(however, when someone further upstream does that, you will have problems too)
Can you explain this point more for me, please, because I think this maybe is the problem.
ICMP carries many kinds of information, it is not only used for ping. In this particular case, if a packet arrives to a piece of equipment for which it is too large to handle, the equipment normally fragments it. But if such packet is marked with "don't fragment", the equipment instead sends back an ICMP packet saying that the packet could not have been processed and what is its MTU. So if this happens but someone in between the two points is blocking ICMP completely, the "not to be fragmented" packet never reaches the destination and the sender never learns why.
Why this problem does not occur with hotspot users, but only with pppoe connected clients?!!
The PPPoE encapsulation occupies part of the packet so the MTU of the encapsulated packets is accordingly smaller than the MTU on the underlying physical interface. There was some issue with Mikrotik's handling of this the way described above in 6.41, but it should have been fixed in 6.41.1:
*) ppp - fixed change-mss functionality in some specific traffic (introduced in v6.41)
.
What version are you running?

Instead of
/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu passthrough=yes protocol=tcp tcp-flags=syn
you may try
/ip firewall mangle
add action=change-mss chain=forward tcp-mss=1452 passthrough=yes protocol=tcp tcp-flags=syn
or even use a lower value until it starts working (because I am not sure that if the actual MTU discovery yields a lower value, the one provided by that mangle rule will be overridden).

Also, don't forget to put that mangle rule to the top of the list.
 
elechero
newbie
Topic Author
Posts: 26
Joined: Sat Jun 06, 2015 3:40 am

Re: Some services are not working with pppoe connection

Wed Feb 14, 2018 12:11 pm

Thank you very much sindy for your reply.

My mikrotik device is working on 6.41 while I am writing this post. But I faced this problem when it was working under 6.37.

I will try to upgrade it now to the last release which is 6.41.2 and try to use the mangle role as you suggested.

Thank you again for your help.
 
elechero
newbie
Topic Author
Posts: 26
Joined: Sat Jun 06, 2015 3:40 am

Re: Some services are not working with pppoe connection

Wed Feb 14, 2018 1:11 pm

Unfortunately, I upgraded to 6.41.2 and tried many values of the mangle role, but I did not help. It is so strange. What should I do?!
Last edited by elechero on Wed Feb 14, 2018 3:55 pm, edited 1 time in total.
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Some services are not working with pppoe connection

Wed Feb 14, 2018 1:31 pm

As I've started writing the answer I've realized I don't understand exactly your topology. Does the Mikrotik in question act as a PPPoE server to which other PPPoE clients connect, or as a PPPoE client on the internet uplink? If it is a PPPoE server for other clients, using what kind of interface it is connected to Internet, plain Ethernet or PPPoE (or something else)?
 
elechero
newbie
Topic Author
Posts: 26
Joined: Sat Jun 06, 2015 3:40 am

Re: Some services are not working with pppoe connection

Wed Feb 14, 2018 2:58 pm

As I've started writing the answer I've realized I don't understand exactly your topology. Does the Mikrotik in question act as a PPPoE server to which other PPPoE clients connect, or as a PPPoE client on the internet uplink? If it is a PPPoE server for other clients, using what kind of interface it is connected to Internet, plain Ethernet or PPPoE (or something else)?
Thanks for your reply.

It is connected to internet using static ip.
It is acting as PPPOE server and Hotspot server.
 
elechero
newbie
Topic Author
Posts: 26
Joined: Sat Jun 06, 2015 3:40 am

Re: Some services are not working with pppoe connection

Wed Feb 14, 2018 3:09 pm

Here is a picture:
mikrotik-pic.jpg
You do not have the required permissions to view the files attached to this post.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10221
Joined: Mon Jun 08, 2015 12:09 pm

Re: Some services are not working with pppoe connection

Wed Feb 14, 2018 3:25 pm

And where is/are your PPPoE client? In the 2011 or in users equipment connected to the 2011?
 
elechero
newbie
Topic Author
Posts: 26
Joined: Sat Jun 06, 2015 3:40 am

Re: Some services are not working with pppoe connection

Wed Feb 14, 2018 3:40 pm

And where is/are your PPPoE client? In the 2011 or in users equipment connected to the 2011?
Thank you pe1chl,

2011 is working as bridge, and the clients are connected after it.

After 2011 there are wireless access points ---> stations ---> clients

Some of the clients are TP-Link routers others are Ubnt Nanosation devices. And the end-users are connected to those routers using their mobile phones and computers.
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Some services are not working with pppoe connection

Wed Feb 14, 2018 3:40 pm

Okay. In that case, I'd recommend you to
  • disable the change-mss rule
  • go to /tools packet-sniffer at both Mikrotiks and configure sniffing into a file at the 1036's uplink (internet-facing) interface and at 2011's interface facing towards the 1036. In both cases, filter by the IP address of a test client (assuming that the 1036 doesn't do NAT on its internet-facing interface, if it does, it will be much harder to match the flows between the two files). Apply the settings (file name & interface & IP address to be sniffed & file size somewhere like 4000 kB), then press Start.
  • establish a connection from the test client to the service which is affected by the issue and sniff until the connection experiences problems.
  • stop the sniffing, download the captured files and analyse them using Wireshark
But in this configuration, I can imagine only the following scenarios:
  • there is an MTU bottleneck further in the network, the ICMP "fragmentation needed" arrives to the 1036 (because hotspot clients are all happy) but it does not make it to the PPPoE clients because you've yourself restricted ICMP in firewall rules of the 1036 (as you say the 2011 is only a bridge). You would identify this case by seeing the icmp "fragmentation needed" in the capture from the 1036 but not in the capture from the 2011
  • the bottleneck is the PPPoE between the 1036 and the client. in such case:
    • if the too large packets don't get through from the client to the service, the PPPoE implementation or settings at the client machines would have to be broken,
    • if the too large packets don't get through from the service to the client, the 1036's PPPoE server should be sending the icmp "fragmentation needed" packets towards the service (i.e. towards internet) but it either doesn't or you've yourself restricted ICMP in firewall rules of the 1036. You would identify this case by seeing too large packets to come from the service towards the client in the capture from the 1036 but not seeing them in the capture from the 2011.
You may also place here the output of "/ip firewall export" on the 1036 to see whether any ICMP restriction exists there - it can be an implicit one if you haven't explicitly set an exception for ICMP from a "drop the rest" rule in the "forward" chain.
 
p3rad0x
Long time Member
Long time Member
Posts: 637
Joined: Fri Sep 18, 2015 5:42 pm
Location: South Africa
Contact:

Re: Some services are not working with pppoe connection

Wed Feb 14, 2018 3:42 pm

Just a quick question how many clients are you running on that network at the moment?
 
elechero
newbie
Topic Author
Posts: 26
Joined: Sat Jun 06, 2015 3:40 am

Re: Some services are not working with pppoe connection

Wed Feb 14, 2018 3:49 pm

Okay. In that case, I'd recommend you to
  • disable the change-mss rule
  • go to /tools packet-sniffer at both Mikrotiks and configure sniffing into a file at the 1036's uplink (internet-facing) interface and at 2011's interface facing towards the 1036. In both cases, filter by the IP address of a test client (assuming that the 1036 doesn't do NAT on its internet-facing interface, if it does, it will be much harder to match the flows between the two files). Apply the settings (file name & interface & IP address to be sniffed & file size somewhere like 4000 kB), then press Start.
  • establish a connection from the test client to the service which is affected by the issue and sniff until the connection experiences problems.
  • stop the sniffing, download the captured files and analyse them using Wireshark
But in this configuration, I can imagine only the following scenarios:
  • there is an MTU bottleneck further in the network, the ICMP "fragmentation needed" arrives to the 1036 (because hotspot clients are all happy) but it does not make it to the PPPoE clients because you've yourself restricted ICMP in firewall rules of the 1036 (as you say the 2011 is only a bridge). You would identify this case by seeing the icmp "fragmentation needed" in the capture from the 1036 but not in the capture from the 2011
  • the bottleneck is the PPPoE between the 1036 and the client. in such case:
    • if the too large packets don't get through from the client to the service, the PPPoE implementation or settings at the client machines would have to be broken,
    • if the too large packets don't get through from the service to the client, the 1036's PPPoE server should be sending the icmp "fragmentation needed" packets towards the service (i.e. towards internet) but it either doesn't or you've yourself restricted ICMP in firewall rules of the 1036. You would identify this case by seeing too large packets to come from the service towards the client in the capture from the 1036 but not seeing them in the capture from the 2011.
You may also place here the output of "/ip firewall export" on the 1036 to see whether any ICMP restriction exists there - it can be an implicit one if you haven't explicitly set an exception for ICMP from a "drop the rest" rule in the "forward" chain.
Thank you very much for your answers.
I will try to do what you have suggested.
Here is my firewall exported output:
# feb/14/2018 16:47:15 by RouterOS 6.41.2
# software id = NCJG-Y5DS
#
# model = CCR1036-12G-4S
# serial number = 529A04732C4C
/ip firewall filter
add action=passthrough chain=unused-hs-chain comment="place hotspot rules here" \
    disabled=yes
/ip firewall mangle
add action=change-mss chain=forward disabled=yes new-mss=0 passthrough=yes \
    protocol=tcp tcp-flags=syn tcp-mss=1452-1452
add action=change-mss chain=forward disabled=yes new-mss=clamp-to-pmtu \
    passthrough=yes protocol=tcp tcp-flags=syn
/ip firewall nat
add action=passthrough chain=unused-hs-chain comment="place hotspot rules here" \
    disabled=yes
add action=masquerade chain=srcnat comment="masquerade hotspot network" \
    src-address=10.0.0.0/16
add action=masquerade chain=srcnat comment="masquerade hotspot network" \
    src-address=10.15.0.0/16
add action=masquerade chain=srcnat comment=PPTP-BR src-address=10.20.0.0/16
add action=masquerade chain=srcnat comment=LAPTOP src-address=192.168.178.0/24
add action=masquerade chain=srcnat comment="masquerade hotspot network" \
    src-address=10.16.0.0/16
add action=masquerade chain=srcnat comment="masquerade hotspot network" \
    src-address=10.100.0.0/16
add action=masquerade chain=srcnat comment="for SHAHBA" src-address=\
    172.30.0.0/24
 
elechero
newbie
Topic Author
Posts: 26
Joined: Sat Jun 06, 2015 3:40 am

Re: Some services are not working with pppoe connection

Wed Feb 14, 2018 3:52 pm

Just a quick question how many clients are you running on that network at the moment?
More than 2000 hotspot clients, and about 50 pppoe clients.
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Some services are not working with pppoe connection

Wed Feb 14, 2018 3:57 pm

Hm... your overall topology is much more complex with all those actual clients connected to WAPs which themselves are PPPoE clients of the 1036, and with 10 different src-nats at the 1036 and probably other NATs at the WAPs.

So you'll have to carefully choose the proper IP addresses for filtering while sniffing, and the 4000 kB may not be enough if traffic from many individual clients is NATed to a single common IP address.

I can see there are no firewall filter rules at all in the 1036 configuration, so if you are filtering the ICMP yourself, it happens somewhere else in the network.
 
elechero
newbie
Topic Author
Posts: 26
Joined: Sat Jun 06, 2015 3:40 am

Re: Some services are not working with pppoe connection

Wed Feb 14, 2018 4:15 pm

Hm... your overall topology is much more complex with all those actual clients connected to WAPs which themselves are PPPoE clients of the 1036, and with 10 different src-nats at the 1036 and probably other NATs at the WAPs.

So you'll have to carefully choose the proper IP addresses for filtering while sniffing, and the 4000 kB may not be enough if traffic from many individual clients is NATed to a single common IP address.

I can see there are no firewall filter rules at all in the 1036 configuration, so if you are filtering the ICMP yourself, it happens somewhere else in the network.
Thank you for your help, I will try to do it. I believe I am net filtering the ICMP but I will double check everything.
 
kabiraftab
just joined
Posts: 13
Joined: Fri Jan 25, 2013 5:20 pm

Re: Some services are not working with pppoe connection

Tue Oct 13, 2020 7:56 pm

Sorry for bringing an old post up, I am facing the same issue. There seems to be no solutions presented here, anyone solved it?
 
pe1chl
Forum Guru
Forum Guru
Posts: 10221
Joined: Mon Jun 08, 2015 12:09 pm

Re: Some services are not working with pppoe connection

Tue Oct 13, 2020 8:02 pm

It starts with understanding what the issue is, how it fits in your network topology, and then debug what is really going on.
A network with PPPoE (and no RFC4638) will never be completely trouble-free.
It will be worse when there are operators that think that ICMP is dangerous and has to be blocked, and it will be better when "crude fixes" like the mangle rule are applied in the correct place.

Who is online

Users browsing this forum: GoogleOther [Bot], yonutm and 87 guests