It looks like it doesn’t. I tried setting them both to 1500 but in the status it still shows MTU at 1492. The original configuration of the ONT that the ISP provided me before I put it in bridge mode had an MTU of 1480.
It looks like I’ll have to run a packet trace as you suggest. I hope to do so in the upcoming days, and I’ll post any developments here.
Thanks very much for your time!
OK, I think I figured it out thanks to @mducharme and these two posts:
I just added that clamp-to-pmtu mangle rule to both the IPv4 and IPv6 firewalls, and it magically fixed everything. I also set the MTU and MRU in the PPP connection settings to 1492 for good measure, but it appears to work with the default (1480) as well.
So why might this have been a problem? Is it something janky with my ISP? Or a Mikrotik bug? Or possibly the fiber ONT is also manipulating MSS / MTU / MRU values even though it’s in bridge mode?
Many thanks to @mducharme for pointing me in the right direction, I never would have though to look into MTU issues for IPv6.
If clamp-to-pmtu solves the problem this probably means there is something in the network path that is dropping ICMPv6 messages. This is pretty bad and you should try and figure out where this is happening and fix it if possible.
As R1CH says, something is blocking ICMPv6 so that path MTU discovery is not working. The MikroTik should not block it by default unless you created rules for that. Even though you said before that you did not, go into your IPv6->Firewall and double check that you have rules allowing all lCMPv6 on both input and forward chains, from everywhere to everywhere without restrictions.
Another possibility is that if you have another device on your internal network that is incorrectly configured to send Router Advertisements and has IPv6 forwarding enabled, your computer may be using that device as a default gateway, and it is in turn sending them to the MikroTik. If it is blocking ICMPv6, then you would have these Path MTU discovery issues. Could be a wireless access point or anything similar that allows you to configure it as a router.
Does everything look normal here?
The defconf: rfc4890 drop hop-limit=1 rule isn’t possibly causing problems?
As far as other devices advertising themselves as routers, it’s a very small network, I’m sure there’s nothing like that on the network.
It looks fine, but it is hard to tell because you keep sending screenshots instead of config dumps, and not all fields are shown on the screen by default. It is better to post config dumps instead of screen snapshots, they have more info and take up less space.
Please go to the device via command line, make your terminal window large so that the lines don’t get abbreviated, and type “/ipv6 firewall export” and press enter, copy and paste the results.
/ipv6 firewall export
# aug/24/2018 17:23:25 by RouterOS 6.42.7
# software id = KZAB-HII5
#
# model = 751U-2HnD
# serial number = 2FF2013816F7
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
add address=::224.0.0.0/100 comment="defconf: other" list=bad_ipv6
add address=::127.0.0.0/104 comment="defconf: other" list=bad_ipv6
add address=::/104 comment="defconf: other" list=bad_ipv6
add address=::255.0.0.0/104 comment="defconf: other" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=33434-33534 protocol=udp
add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/16
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=input comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=forward comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
Yes that is correct. That shouldn’t cause problems.
If there is traffic hitting the ICMPv6 hop limit 1 rule then you can try disabling it to see if it helps. Although, I am starting to suspect that your ISP may have a bad configuration and is blocking ICMPv6 somewhere they should not be, perhaps on their PPPoE Concentrator. They may not have enough customers on IPv6 yet with a setup like yours to notice this issue.
Also do a packet capture, in the MikroTik go to Tools->Packet capture, under General, give it a filename with .pcap extension, go to filter tab, set IP protocol to 58 (this is ICMPv6 protocol number), direction any, filter operation or, click apply. Disable the mangle rule you made and click start. Then go to a website that wouldn’t load before making the mangle rule. Then click stop, download the file from the router’s files folder, then then you can send and attach that (it can also be opened in wireshark). Then re-enable your mangle rule so your service works again.
I have pretty much the same problem, in my case ipv6 is a 6to4 tunnel inside a pppoe interface. Could the problem be coming from some “inherit” in do-not-fragment that makes that the ipv4 tunnel drops the ipv6 big packet, and thus the ipv6 stack never sees the error? (wild guess)
If using HE tunnelbroker over PPPoE you need to lower the MTU on the tunnelbroker side, the default on their end is 1480 which is too big if you have PPPoE overhead. If your PPPoE is 1480, decrease that setting to 1460, and then it should be OK. It is done through their web interface under advanced settings tab for your tunnel.
Sorry for the delay. I am attaching the packet capture as you described. I ran it through TraceWrangler to anonymize it, and now it shows some packets as malformed, in the original capture they’re all valid. I don’t know if TraceWrangler eliminated any other important information. If it is did, let me know and I can PM you the original.
Thanks a lot!
IPv6-20180827_anon.pcapng.txt (2.23 KB)
The capture did not see a “Packet too big” message, it should be there in Wireshark: https://packetpushers.net/ipv6-and-the-importance-of-the-icmpv6-packet-too-big-message/
If you disabled the mangle rule that you had created to work around the problem before taking that capture, chances are good that your ISP is dropping the message, that would be a major configuration issue on their end. You should try to get them to fix that.
Yep, the capture was with the Mangle / MSS workaround disabled. So it’s probably my ISP. The problem is getting them to understand, they generally assume their users are protozoically dumb and will just ask me to update my antivirus software and confirm that I can still logon to Facebook…
The only thing I still wonder about is why IPv6 appeared to work “out of the box” with the ONT that the ISP provided me with its standard configuration running in router mode. I didn’t use it too much that way before switching it to bridge mode and connecting the Mikrotik, but it definitely seemed to work without problems.
Thanks a lot for helping me to diagnose this.
mducharme
If using HE tunnelbroker over PPPoE you need to lower the MTU on the tunnelbroker side, the default on their end is 1480 which is too big if you have PPPoE overhead. If your PPPoE is 1480, decrease that setting to 1460, and then it should be OK. It is done through their web interface under advanced settings tab for your tunnel.
Does not make any difference, I tried all the combinations of values. Additionally I restricted all MTUs (of both the 6to4 and their side) to 1280 as they instruct to do, or left me/them as 1480, 1472, 1460, 1452…
Always the same behaviour. I’m a bit lost. I tried the mangle rules, but no change… The occasional session gets through, but most die. When I do IPv6 packet level tracing of http connections I see the handshake with no problems, and then my client sending the “GET / HTTP/1.1” up to a dozen times; sometimes an answer comes back, but most times it is just RST,FIN after 5 minutes. It is not easy to trace the 6to4 tunnel itself or the pppoe, I think their ONT can be set up to mirror the traffic in one of the ethernet ports, I might try this when I’m back home next week.
sb56637
The only thing I still wonder about is why IPv6 appeared to work “out of the box” with the ONT that the ISP provided me with its standard configuration running in router mode. I didn’t use it too much that way before switching it to bridge mode and connecting the Mikrotik, but it definitely seemed to work without problems.
Are we speaking Movistar FTTH here? (this is my case).
sb56637
Are we speaking Movistar FTTH here? (this is my case).
No, it’s a different ISP.
ICMPv6 is allowed on your MikroTik on both forward and input chain? Your MikroTik is acting as the PPPoE client? If the MikroTik is acting as the PPPoE client, you should see the negotiated MTU and MRU and should be able to calculate from that the correct MTU for the hurricane electric side of the tunnel by subtracting 20 more. Your side of the HE tunnel can be 1280 because usually you don’t need as much for upload.
Yes:
[admin@Mikrotik] > :put [/system resource get uptime ]
4d03:24:04
[admin@Mikrotik] > /ipv6 firewall filter print stats where protocol~"icmpv6"
Flags: X - disabled, I - invalid, D - dynamic
# CHAIN ACTION BYTES PACKETS
0 ;;; defconf: accept ICMPv6
input accept 112 2
1 ;;; defconf: accept ICMPv6
forward accept 824 9
I have the tunnel disabled most of the time, as things stop working as the clients pick up the ipv6 addresses… [I only up it for debugging, but editing after I pasted the above, now after some more experiments the forward rule accepted 131 packets just with some more icmp/tcp experiments.]
Yes, my router is the client. I have experimented with those values, but everything is consistent and ipv4 works like a charm. From linux I can ping ipv6.tunnelbroker.net with up to 1360 bytes, ipv6.google.com up to 1232 bytes, more than this (no upper limit) is a blackhole for both:
$ ping -c 1 -s 1360 ipv6.tunnelbroker.net
PING ipv6.tunnelbroker.net(tunnelbroker.net (2001:470:0:63::2)) 1360 data bytes
1368 bytes from tunnelbroker.net (2001:470:0:63::2): icmp_seq=1 ttl=56 time=301 ms
--- ipv6.tunnelbroker.net ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 301.014/301.014/301.014/0.000 ms
$ ping -c 1 -s 1361 ipv6.tunnelbroker.net
PING ipv6.tunnelbroker.net(tunnelbroker.net (2001:470:0:63::2)) 1361 data bytes
--- ipv6.tunnelbroker.net ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
$ ping -c 1 -s 1232 ipv6.google.com
PING ipv6.google.com(dh-in-x8b.1e100.net (2a00:1450:400b:c03::8b)) 1232 data bytes
72 bytes from dh-in-x8b.1e100.net (2a00:1450:400b:c03::8b): icmp_seq=1 ttl=47 (truncated)
--- ipv6.google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 143.711/143.711/143.711/0.000 ms
$ ping -c 1 -s 1233 ipv6.google.com
PING ipv6.google.com(dub08s01-in-x0e.1e100.net (2a00:1450:400b:801::200e)) 1233 data bytes
--- ipv6.google.com ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
Have you tried a different tunnel server on the HE side? I wonder if maybe they have a previously unnoticed configuration issue with one of their tunnelbroker servers that might explain your issue. Trying a different tunnelbroker server would allow you to identify if that is the case. This really sounds like it might be an HE problem.
I tried the Frankfurt and London endpoints, which are the closest to home. I also got puzzled and tend to think that this is their problem, but I have no clear ways to further diagnose. My country (Spain) is in a very sorry situation as the ex-monopoly and main provider (Telefonica/Movistar) refuses to route IPv6, even now that google is reporting almost 25% of their traffic is IPv6, so it allows us little space for keeping up to date with technology in practice. 
They have a forum, and their top engineers go on that forum. If you explain that your router is configured to allow all ICMPv6 and you have lowered the MTU on the tunnel (since you are on PPPoE) and yet you are unable to use IPv6 because of missing packet-too-big messages, I’m sure they will look into what is happening.
From the sounds of it, the HE tunnelbroker server itself is not sending back the packet-too-big based on your configured MTU for the tunnel, which is resulting in your issue. It must be sending packet-too-big based on the default tunnel MTU of 1480 instead of your selected MTU. This would result in a problem only for customers with an HE IPv6 tunnel with < 1500 IPv4 MTU, which is probably a minority of HE customers, so they might not have noticed such an issue.