Community discussions

 
nostromog
Member Candidate
Member Candidate
Topic Author
Posts: 161
Joined: Wed Jul 18, 2018 3:39 pm

Problem with 6to4 inside PPPoE

Mon Oct 08, 2018 7:28 pm

I have had a long nightmare trying to connect my machine to Hurricane Electric tunnelbroker. Now it is no longer a nightmare, at least I know the problem, even if I have not yet found a solution.

Context:
* My provider, Telefonica/Movistar, dominant operator in Spain, is well known for its neglect to IPv6, which led Spain from top positions in IPv6 development/deployment to the bottom of world in deployment scale
* They don't offer native IPv6, and their ONT/router combo does not support 6to4 tunnels
* When I "unbundled" their VLANs so that the data one went into my MikroTik hAP ac as PPPoE, I started trying HE tunnels:
* MikroTik connects well, and ICMP/UDP packets flows well
* When I try to send TCP it fails. I was thinking (misled) that it were MTU/MSS issues, but they were not
* I decided to try a new strategy: to put a mirror ether port in the WAN side of the ONT, and another in the MikroTik (LAN) side, and wireshark it ;)

What I found, is a strange traffic pattern:
* Outside of the Mikrotik the connection goes well upstream (packet seen in a normal TCP), but there is a lot of spurious retransmissions (from our side) as if the other side
responses were not arriving to my testing laptop.
* Inside of the Mikrotik those "normal" responses from upstream are not seen beyond the first two (SYN, SYN+ACK) packets.

The interesting finding is that all packets my laptop sends have, when seen from the WAN side, incorrect PPPoE payload length. A typical one would say in
wireshark
3381 (Incorrect, should be 173)
. Those packets seem to arrive to destination, as answers arrive well to the router...
Inside, in the LAN side, received packets that enter the MikroTik have the same incorrect PPPoE payload length.

My hypothesis is that their HGU (Home Gateway Unit, they call it) is corrupting the PPPoE packets, possibly when trying to do QoS or mark connection or flow.
This only happens after it sees a SYN+ACK, and affects the payload calculation in both directions. Apparently they PPPoE concentrators ignore the payload field, so everything works for them.

Now my question: can I have the RouterOS machinery ignore broken payload fields, and allow the Spanish market to use MikroTik for fibre IP tunnels in the main operator?

BTW, thanks for the hAP lite I got in the Valencia MUM, it was extremely helpful to finish this debugging process.
 
User avatar
Anumrak
Forum Guru
Forum Guru
Posts: 1051
Joined: Fri Jul 28, 2017 2:53 pm

Re: Problem with 6to4 inside PPPoE

Tue Oct 09, 2018 9:45 am

Hey. Interesting situation. Can I see your PPPoE client config without sens. data and 6to4 tunnel config?
 
nostromog
Member Candidate
Member Candidate
Topic Author
Posts: 161
Joined: Wed Jul 18, 2018 3:39 pm

Re: Problem with 6to4 inside PPPoE

Tue Oct 09, 2018 2:05 pm

There it is, I edited the tunnel endpoints and I'm not posting the addresses/routes or serial numbers.

BTW, /interface 6to4 export hide-sensitive does NOT hide the "ipsec-secret" attribute that I have in
a different, ipsec protected, 6to4 tunnel that works perfectly. :)

As I said, ICMP, UDP and TCP packets that don't involve outgoing ACK or non zero length flow well
through the HGU. In the packet dumps from each side one can see that the malformed packets
are symmetrical (inside are the incoming packets only, outside are the outgoing packets only).
I have packet captures, but I'm not sure about how to anonimize without destroying (I mean correcting)
the broken PPPoE payload lenght...


# oct/09/2018 10:14:51 by RouterOS 6.43.2
# model = RouterBOARD 962UiGS-5HacT2HnT
# XXX -> my current public address,
# YYY -> tunnel broker
# I use a ppp profile to run a script that updates DDNS,
# updates the hairpin NAT rules and the 6to4 tunnels endpoints
/ppp profile
   add change-tcp-mss=yes name=movistar on-up=new-ip use-compression=no
# I have tested repeatedly max-mtu=auto, 1492, 1488 (as there is a VLAN
# outside of the HGU that eats another 4 bytes). After auto
# it stabilises to 1480 usually, but the value makes no difference.
# The pppoe parameters are the public standard for the provider.
/interface pppoe-client
    add add-default-route=yes disabled=no interface=ether1 \
    max-mtu=1488 name=pppoe-out1 password=adslppp profile=movistar \
    service-name=telefonica use-peer-dns=yes user=adslppp@telefonicanet.pa
# This is again standard. I have tested several dscp values (inherit...)
# keepalive needs sometimes to be fiddled with for the link to get established.
# Re: mtu I have tried 1280, auto and several values. No difference.
# I have also tested with both values of the dont-fragment flag.
# I usually enable sit2 for brief testing, and then disable again, because all
# ipv6 enabled world (deb repos, google, other CDNs, forum.mikrotik.com...)
# stops working.
/interface 6to4
   add comment="Hurricane Electric IPv6 Tunnel Broker" \
   disabled=yes dont-fragment=inherit dscp=0 !keepalive \
   local-address=XXX.XXX.XXX.XXX name=sit2 remote-address=YYY.YYY.YYY.YYY
 
User avatar
Anumrak
Forum Guru
Forum Guru
Posts: 1051
Joined: Fri Jul 28, 2017 2:53 pm

Re: Problem with 6to4 inside PPPoE

Tue Oct 09, 2018 3:13 pm

Why you using ethernet interface for pppoe traffic, when your transport is ISP vlan? If you meant that in your ISP infra exists vlan, you don't need worry about it, cause ISP had to pop up his l2 mtu on all his switches.
 
nostromog
Member Candidate
Member Candidate
Topic Author
Posts: 161
Joined: Wed Jul 18, 2018 3:39 pm

Re: Problem with 6to4 inside PPPoE

Tue Oct 09, 2018 4:58 pm

Why you using ethernet interface for pppoe traffic, when your transport is ISP vlan? If you meant that in your ISP infra exists vlan, you don't need worry about it, cause ISP had to pop up his l2 mtu on all his switches.
VLANs are only visible in the "outer" side, when I mirror the fibre into one of the ethernet ports of the HGU. I was not sure if my provider could take higher l2 mtu, so I stayed in the safe side. But I have tried auto, 1500 (upping my L2 MTU), 1492, 1488, 1480 (which is the one that gets selected when I say "auto"). All with the same results. It is NOT a MTU issue, though I lost a lot of time thinking it was.

In theory the HGU should not fiddle with the payload when it strips the VLAN payload, but obviously it does. The configuration I have is supposed to process the voice (mandatory) and TV (I didn't contract it) VLANs, but leave the data VLAN delivered (without the VLAN label) to its ether1. This is the where my MikroTik is connecting to.

Maybe there is a bug in its VLAN code, maybe in the QoS or firewall code. Maybe the code that computes the payload does not initialize to zero the variable. The length field contains apparently semi-random values, like 31234 or 4528, almost always much bigger than the real value.

My ISP HGU is forced into me because it does not exist a legal way to recover the keys and reprogram a different ONT to connect to the fibre. I'd love to be able to switch to a SFP+ ONT, it would save me a lot of electric power and space, but I can't at least for the moment. I highly doubt that it is legal to impose a service-providing ONT upon all the users, so I guess the European Union will end up forcing them to unbundle it to everyone via legal complaints by its competition.
 
User avatar
Anumrak
Forum Guru
Forum Guru
Posts: 1051
Joined: Fri Jul 28, 2017 2:53 pm

Re: Problem with 6to4 inside PPPoE

Tue Oct 09, 2018 5:11 pm

Why you using ethernet interface for pppoe traffic, when your transport is ISP vlan? If you meant that in your ISP infra exists vlan, you don't need worry about it, cause ISP had to pop up his l2 mtu on all his switches.
VLANs are only visible in the "outer" side, when I mirror the fibre into one of the ethernet ports of the HGU. I was not sure if my provider could take higher l2 mtu, so I stayed in the safe side. But I have tried auto, 1500 (upping my L2 MTU), 1492, 1488, 1480 (which is the one that gets selected when I say "auto"). All with the same results. It is NOT a MTU issue, though I lost a lot of time thinking it was.

In theory the HGU should not fiddle with the payload when it strips the VLAN payload, but obviously it does. The configuration I have is supposed to process the voice (mandatory) and TV (I didn't contract it) VLANs, but leave the data VLAN delivered (without the VLAN label) to its ether1. This is the where my MikroTik is connecting to.

Maybe there is a bug in its VLAN code, maybe in the QoS or firewall code. Maybe the code that computes the payload does not initialize to zero the variable. The length field contains apparently semi-random values, like 31234 or 4528, almost always much bigger than the real value.

My ISP HGU is forced into me because it does not exist a legal way to recover the keys and reprogram a different ONT to connect to the fibre. I'd love to be able to switch to a SFP+ ONT, it would save me a lot of electric power and space, but I can't at least for the moment. I highly doubt that it is legal to impose a service-providing ONT upon all the users, so I guess the European Union will end up forcing them to unbundle it to everyone via legal complaints by its competition.
You could try another ISP and within his pipe start up your 6to4.
 
User avatar
xvo
Long time Member
Long time Member
Posts: 599
Joined: Sat Mar 03, 2018 1:12 am
Location: Moscow, Russia

Re: Problem with 6to4 inside PPPoE

Tue Oct 09, 2018 6:27 pm

But I have tried auto, 1500 (upping my L2 MTU), 1492, 1488, 1480 (which is the one that gets selected when I say "auto").
PPPoE default is 1492, 6to4 substracts 20 (that is why “auto” is 1480=1500-20), so you should at least try 1472. And specify it on both ends - yours and in HE settings as well.
If it still doesn’t work, then you can rule out MTU :)
 
nostromog
Member Candidate
Member Candidate
Topic Author
Posts: 161
Joined: Wed Jul 18, 2018 3:39 pm

Re: Problem with 6to4 inside PPPoE

Wed Oct 10, 2018 1:42 am

But I have tried auto, 1500 (upping my L2 MTU), 1492, 1488, 1480 (which is the one that gets selected when I say "auto").
PPPoE default is 1492, 6to4 substracts 20 (that is why “auto” is 1480=1500-20), so you should at least try 1472. And specify it on both ends - yours and in HE settings as well.
If it still doesn’t work, then you can rule out MTU :)
Those numbers were for the (outer) PPPoE tunnel. It is the PPPoE link that shows 1480 when left to auto, don't ask me why. It still might be a MikroTik bug, mistaking two stacked tunnels overheads, cause everything seems to work ok when forced to 1492.... The 6to4 link shows 1460 when left to auto. Everything works well for ICMP or UDP. It is the TCP bug that kills me, see above. In summary the problem is that the ONT+HGU, in theory as a transparent bridge, is corrupting the PPPoE length header field and both the MikroTik and my linux laptop using (kernel) pppoe drop those packets.

I'm yet to check if I can substitute their ONT, or they will replace the HGU, or upgrade its firmware, for one that does not have this annoying bug. A tech support issue is open, I'm waiting for an answer.
 
User avatar
xvo
Long time Member
Long time Member
Posts: 599
Joined: Sat Mar 03, 2018 1:12 am
Location: Moscow, Russia

Re: Problem with 6to4 inside PPPoE

Wed Oct 10, 2018 2:22 am

So what MTU do you have on the 6to4 after all?
And in the HE cabinet?
 
User avatar
Anumrak
Forum Guru
Forum Guru
Posts: 1051
Joined: Fri Jul 28, 2017 2:53 pm

Re: Problem with 6to4 inside PPPoE

Wed Oct 10, 2018 8:37 am

But I have tried auto, 1500 (upping my L2 MTU), 1492, 1488, 1480 (which is the one that gets selected when I say "auto").
PPPoE default is 1492, 6to4 substracts 20 (that is why “auto” is 1480=1500-20), so you should at least try 1472. And specify it on both ends - yours and in HE settings as well.
If it still doesn’t work, then you can rule out MTU :)
Those numbers were for the (outer) PPPoE tunnel. It is the PPPoE link that shows 1480 when left to auto, don't ask me why. It still might be a MikroTik bug, mistaking two stacked tunnels overheads, cause everything seems to work ok when forced to 1492.... The 6to4 link shows 1460 when left to auto. Everything works well for ICMP or UDP. It is the TCP bug that kills me, see above. In summary the problem is that the ONT+HGU, in theory as a transparent bridge, is corrupting the PPPoE length header field and both the MikroTik and my linux laptop using (kernel) pppoe drop those packets.

I'm yet to check if I can substitute their ONT, or they will replace the HGU, or upgrade its firmware, for one that does not have this annoying bug. A tech support issue is open, I'm waiting for an answer.
TCP heavier than UDP by 12 bytes, maybe your ISP have less mtu than that, with ppp and ipv6 headers. By default 6to4 tunnel mtu is 1280, so max tcp payload will be 1260. Complete data with all headers and one vlan will be 1366 bytes. In your case with 1460 mtu in 6to4 tunnel = 1546 which also fine, but depends on your ISP switches config. BUT 1460 MTU in your 6to4 tunnel is bad idea, because your ipv6 + ipv4 payload is already much higher than your pppoe tunnel capacity(1460+40+20>1480-1492). Most flows of UDP segmetns will be fine, because they have no handshakes, but with TCP full payload could be troubles.
 
nostromog
Member Candidate
Member Candidate
Topic Author
Posts: 161
Joined: Wed Jul 18, 2018 3:39 pm

Re: Problem with 6to4 inside PPPoE

Wed Oct 10, 2018 11:10 am

    So what MTU do you have on the 6to4 after all?
    And in the HE cabinet?
    Yesterday night I made the final test: I patched rp-pppoe code so
    that it would accept packets with the wrong length at header field
    and run PPPoE + HE 6to4 tunnel in my laptop.

    $ git clone git@github.com:Distrotech/rp-pppoe.git && cd rp-pppoe
    $ # some editing until...
    $ git diff
    diff --git a/src/pppoe.c b/src/pppoe.c
    index a513785..608234a 100644
    --- a/src/pppoe.c
    +++ b/src/pppoe.c
    @@ -175,7 +175,7 @@ sessionDiscoveryPacket(PPPoEConnection *conn)
         if (ntohs(packet.length) + HDR_SIZE > len) {
            syslog(LOG_ERR, "Bogus PPPoE length field (%u)",
                   (unsigned int) ntohs(packet.length));
    -       return;
    +       //return; // SGP: testing Broken HGU
         }
     
         if (packet.code != CODE_PADT) {
    @@ -751,7 +751,7 @@ asyncReadFromEth(PPPoEConnection *conn, int sock, int clampMss)
         if (ntohs(packet.length) + HDR_SIZE > len) {
            syslog(LOG_ERR, "Bogus PPPoE length field (%u)",
                   (unsigned int) ntohs(packet.length));
    -       return;
    +       //return; // SGP: testing Broken HGU
         }
     #ifdef DEBUGGING_ENABLED
         if (conn->debugFile) {
    @@ -802,7 +802,8 @@ asyncReadFromEth(PPPoEConnection *conn, int sock, int clampMss)
         if (plen + HDR_SIZE > len) {
            syslog(LOG_ERR, "Bogus length field in session packet %d (%d)",
                   (int) plen, (int) len);
    -       return;
    +       //return; // SGP: testing Broken HGU
    +       plen = len - HDR_SIZE;
         }
     
         /* Clamp MSS */
    @@ -880,7 +881,7 @@ syncReadFromEth(PPPoEConnection *conn, int sock, int clampMss)
         if (ntohs(packet.length) + HDR_SIZE > len) {
            syslog(LOG_ERR, "Bogus PPPoE length field (%u)",
                   (unsigned int) ntohs(packet.length));
    -       return;
    +       //return; // SGP: testing Broken HGU
         }
     #ifdef DEBUGGING_ENABLED
         if (conn->debugFile) {
    $ make
    # unplug the cable going from HGU to MikroTik from router and plug it into my laptop
    $ sudo pppd pty './pppoe -I enp0s31f6'  user "adslppp@telefonicanet.pa" noauth lcp-echo-interval 20 noipdefault defaultroute usepeerdns mtu 1492
    $ sudo ifup my-he-ipv6
    
    And everything runs. In addition I get thousands of errors in my log, such as
    $ grep Bogus /var/log/syslog
    (...)
    Oct  9 22:56:05 chiron pppoe[32329]: Bogus length field in session packet 31486 (114)
    Oct  9 22:56:05 chiron pppoe[32329]: Bogus PPPoE length field (31487)
    Oct  9 22:56:05 chiron pppoe[32329]: Bogus length field in session packet 31487 (555)
    Oct  9 22:56:05 chiron pppoe[32329]: Bogus PPPoE length field (31490)
    Oct  9 22:56:05 chiron pppoe[32329]: Bogus length field in session packet 31490 (126)
    Oct  9 22:56:05 chiron pppoe[32329]: Bogus PPPoE length field (31492)
    Oct  9 22:56:05 chiron pppoe[32329]: Bogus length field in session packet 31492 (555)
    Oct  9 22:56:06 chiron pppoe[32329]: Bogus PPPoE length field (31530)
    (...)
    

    After I checked that everything was working I started playing by hand with the MTU. Everything works with
    • outer (pppoe-out1) MTU of 1492
    • inner (sit2) MTU of 1472
    Obviously even if I left the HE one at 1480 (I tested with both 1472 and 1480 set up in the tunnelbroker).

    Definitely my problem was NOT a MTU one, but a failing hardware/software one.
     
    User avatar
    Anumrak
    Forum Guru
    Forum Guru
    Posts: 1051
    Joined: Fri Jul 28, 2017 2:53 pm

    Re: Problem with 6to4 inside PPPoE

    Wed Oct 10, 2018 1:41 pm

    Why you don't want to make HE tunnel mtu lower than pppoe tunnel mtu?
     
    nostromog
    Member Candidate
    Member Candidate
    Topic Author
    Posts: 161
    Joined: Wed Jul 18, 2018 3:39 pm

    Re: Problem with 6to4 inside PPPoE

    Wed Oct 10, 2018 3:48 pm

    Why you don't want to make HE tunnel mtu lower than pppoe tunnel mtu?
    Where have you got the idea that I don't want?

    When PPPoE tunnel MTU is 1492, 6to4 tunnel MTU is 1472, 20 bytes smaller
    when PPPoE tunnel MTU is 1480 (what MikroTik negotiates), 6to4 tunnel MTU is 1460... 20 bytes smaller again
    and so on.
     
    User avatar
    Anumrak
    Forum Guru
    Forum Guru
    Posts: 1051
    Joined: Fri Jul 28, 2017 2:53 pm

    Re: Problem with 6to4 inside PPPoE

    Wed Oct 10, 2018 4:24 pm

    Why you don't want to make HE tunnel mtu lower than pppoe tunnel mtu?
    Where have you got the idea that I don't want?

    When PPPoE tunnel MTU is 1492, 6to4 tunnel MTU is 1472, 20 bytes smaller
    when PPPoE tunnel MTU is 1480 (what MikroTik negotiates), 6to4 tunnel MTU is 1460... 20 bytes smaller again
    and so on.
    IPv6 header weight is 40 bytes without options. And IPv4 is 20 bytes. So TCP(20) + IPv6(40) + IPv4(20). This is all we got before PPPoE interface. It is 80 bytes overhead without options and payload in IPv6 header. Set PPPoE interface 1480 and try to ping ipv4 address with d flag. Max payload in ICMP will be 1452(1480-20(IP)-8(ICMP).
     
    nostromog
    Member Candidate
    Member Candidate
    Topic Author
    Posts: 161
    Joined: Wed Jul 18, 2018 3:39 pm

    Re: Problem with 6to4 inside PPPoE  [SOLVED]

    Wed Oct 10, 2018 10:27 pm

    Solved!

    I no longer need workarounds, and can confirm that for me HE tunnels work allright:

    after a firmware upgrade of my HGU from _n43 to _n53 now myHE tunnel works like a charm!
     
    User avatar
    Anumrak
    Forum Guru
    Forum Guru
    Posts: 1051
    Joined: Fri Jul 28, 2017 2:53 pm

    Re: Problem with 6to4 inside PPPoE

    Thu Oct 11, 2018 8:59 am

    Solved!

    I no longer need workarounds, and can confirm that for me HE tunnels work allright:

    after a firmware upgrade of my HGU from _n43 to _n53 now myHE tunnel works like a charm!
    Hurray! :)

    Who is online

    Users browsing this forum: No registered users and 58 guests