Problems with IPv6 Connectivity with a /64 Prefix Delegation

NotPaul · July 19, 2022, 9:43pm

Hey folks,

I’m in a situation where I’m not able to achieve IPv6 connectivity with my Mikrotik-RB750Gr3,

My ISP is delegating /64 prefixes (i know, I know)
I can get the prefix delegated via dhcpv6.
I can assign the IP pool to my LAN
Devices on my LAN can get assigned IPv6 addresses.
Any IPv6 packets I generate from the router or from the LAN show up as TX packets on ether1, my WAN (per packet sniffer)
No response packets from ever show up in response to public outbound packets. Response packets DO show up from the local-link address of the next hop router
Local-link and non-local-link packets are identically constructed with identical SRC/DST MAC addrs, only difference being the destination IPv6 address.
I’ve disabled every firewall rule, and added blanket accept all rules on input/output/forward
My ISP claims they can’t see any IPv6 packets coming from my device, despite me seeing them in sniffer.
This problem persists across RouterOS v6.48.6 (long-term) and RouterOS v7.3.1 (stable).

Now here’s the kicker:

If I plug in my laptop or desktop directly to my upstream (in place of the router), everything works fine.
My ISP indicates they dump routers vs end devices into different vlans so this isn’t that weird. But the fact that they claim they can’t see any outbound ipv6 traffic from my hEX is really weird.

At this point I’m reasonably confident this is an issue with my ISP, but I wanted further insight from you all on if you’ve ever seen a bug like this before. I’m especially interested if you think the /64 PD is a contributing factor.

Config dumps follow.

[admin@hex] > /ipv6 dhcp-client/print detail 
Flags: D - dynamic; X - disabled, I - invalid 
 0    interface=ether1 status=bound duid="0xREMOVED dhcp-server-v6=fe80::AAAA:BBBB:fe4f:52e8 request=prefix add-default-route=yes default-route-distance=1 use-peer-dns=yes dhcp-options="" pool-name="dhcpv6" 
      pool-prefix-length=64 prefix-hint=::/0 dhcp-options="" prefix=AAAA:BBBB:CCCC:DDDD::/64, 2d22h16m12s

[admin@hex] > /ipv6 address/print detail 
Flags: X - disabled, I - invalid, D - dynamic; G - global, L - link-local 
 0 DL address=fe80::DDDD:EEEE:fe05:3a05/64 from-pool="" interface=bridge actual-interface=bridge eui-64=no advertise=no no-dad=no 
 1 DL address=fe80::DDDD:EEEE:fe05:3a04/64 from-pool="" interface=ether1 actual-interface=ether1 eui-64=no advertise=no no-dad=no 
 2  G address=AAAA:BBBB:CCCC:DDDD::/64 from-pool=dhcpv6 interface=bridge actual-interface=bridge eui-64=no advertise=yes no-dad=no

[admin@hex] > /ipv6 pool/print detail 
Flags: D - dynamic 
 0 D id=4 name="dhcpv6" prefix=AAAA:BBBB:CCCC:DDDD::/64 prefix-length=64 expires-after=2d22h8m7s

Here’s what ping looks like on the device. Ping from the lan has the exact same behavior (with a full IP). Also note if I assign the entire PD to the router’s ether1, it also has this same failed behavior.

[admin@hex] > ping src-address=AAAA:BBBB:CCCC:DDDD:: interface=ether1 address=2607:f8b0:4002:81a::200e 
  SEQ HOST                                     SIZE TTL TIME       STATUS                                                                                                                                                                                        
    0 2607:f8b0:4002:81a::200e                                     timeout                                                                                                                                                                                       
    1 2607:f8b0:4002:81a::200e                                     timeout                                                                                                                                                                                       
    2 2607:f8b0:4002:81a::200e                                     timeout                                                                                                                                                                                      
    3 2607:f8b0:4002:81a::200e                                     timeout                                                                                                                                                                                      
    sent=4 received=0 packet-loss=100%

And the packet dump:

 9 time=5.472 num=10 direction=tx src-mac=18:FD:74:01:01:01 dst-mac=DD:EE:FF:4F:52:E8 interface=ether1 src-address=AAAA:BBBB:CCCC:DDDD:: dst-address=2607:f8b0:4002:81a::200e protocol=ipv6 ip-protocol=icmpv6 
   size=70 cpu=2 ip-packet-size=16 ttl=64

Now compare to the same ping but to the local link (again from the public range).

[admin@hex] > ping src-address=AAAA:BBBB:CCCC:DDDD:: interface=ether1 address=fe80::DEDE:DEDE:fe4f:52e8
  SEQ HOST                                     SIZE TTL TIME       STATUS                                                                                                                                                                                        
    0 fe80::DEDE:DEDE:fe4f:52e8                  56  64 1ms493us   echo reply                                                                                                                                                                                    
    1 fe80::DEDE:DEDE:fe4f:52e8                  56  64 1ms405us   echo reply                                                                                                                                                                                    
    2 fe80::DEDE:DEDE:fe4f:52e8                  56  64 1ms451us   echo reply                                                                                                                                                                                    
    3 fe80::DEDE:DEDE:fe4f:52e8                  56  64 1ms203us   echo reply                                                                                                                                                                                    
    4 fe80::DEDE:DEDE:fe4f:52e8                  56  64 1ms369us   echo reply                                                                                                                                                                                    
    sent=5 received=5 packet-loss=0% min-rtt=1ms203us avg-rtt=1ms384us max-rtt=1ms493us

–

132 time=139.164 num=133 direction=tx src-mac=18:FD:74:01:01:01 dst-mac=DD:EE:FF:4F:52:E8 interface=ether1 src-address=AAAA:BBBB:CCCC:DDDD:: dst-address=fe80::DEDE:DEDE:fe4f:52e8 protocol=ipv6 
   ip-protocol=icmpv6 size=70 cpu=1 ip-packet-size=16 ttl=64

133 time=139.165 num=134 direction=rx src-mac=DD:EE:FF:4F:52:E8 dst-mac=18:FD:74:01:01:01 interface=ether1 src-address=fe80::DEDE:DEDE:fe4f:52e8 dst-address=AAAA:BBBB:CCCC:DDDD:: protocol=ipv6 
   ip-protocol=icmpv6 size=70 cpu=1 ip-packet-size=16 ttl=64

[admin@hex] > /ipv6 firewall/filter/print detail 
Flags: X - disabled, I - invalid; D - dynamic 
 0    chain=forward action=accept protocol=icmpv6 out-interface=ether1 log=no log-prefix="" 

 1    chain=input action=accept protocol=icmpv6 in-interface=ether1 log=no log-prefix="" 

 2    chain=output action=accept protocol=icmpv6 out-interface=ether1 log=no log-prefix="" 

[Rest removed for brevity]

Sob · July 19, 2022, 11:11pm

It can be problem with gateway. Unlike IPv4 DHCP, DHCPv6 does not provide default gateway. It may seem a bit weird at first, but the standard simply doesn’t provide such option at all. The add-default-route=yes you see in DHCPv6 client is non-standard hack that uses address of DHCPv6 server as gateway. It works fine if DHCPv6 server and gateway are same machine, which is often true. But not always and then if fails. Correct way how to get gateway is from RA, which RouterOS doesn’t do by default.

So try this:

in DHCPv6 client set add-default-route=no
set /ipv6 settings set accept-router-advertisements=yes (in v6 it works instantly, v7 requires reboot)

NotPaul · July 20, 2022, 12:02am

Hello Sob,

Thank you! That moved the needle but still didn’t resolve the problem. I can now ping from the router to the open Internet, but devices on the LAN, while able to obtain IPv6 addresses, still can’t access the Internet.

Of note is the IP assigned to the ether1 interface is not from the same /64. The pool handed out via DHCPv6 is AA:BB:CC:6b0::/64, but the IP automatically assigned when I turned on RA in the settings is AA:BB:CC:6af:DD:EE:fe05:3a04/64.

6af vs 6b0.

When I run ping from the router it shows up as from AA:BB:CC:6af:DD:EE:fe05:3a04/64.

My naive read here is that their upstream is OK taking packets from AA:BB:CC:6af:DD:EE:fe05:3a04/64 but not the delegated range of AA:BB:CC:6b0::/64. Which I don’t understand.

[admin@hex] > /ipv6 dhcp-client/print detail 
Flags: D - dynamic; X - disabled, I - invalid 
 0    interface=ether1 status=bound duid="0xREMOVED" dhcp-server-v6=fe80::AAAA:BBBB:fe4f:52e8 request=prefix add-default-route=no use-peer-dns=yes dhcp-options="" pool-name="DHCPv6" pool-prefix-length=64 prefix-hint=::/0 dhcp-options="" 
      prefix=AA:BB:CC:6b0::/64, 2d23h58m30s

[admin@hex] > /ipv6 address/print detail 
Flags: X - disabled, I - invalid, D - dynamic; G - global, L - link-local 

 0  G address=AA:BB:CC:6b0::/64 from-pool=DHCPv6 interface=bridge actual-interface=bridge eui-64=no advertise=yes no-dad=no 

 1 DL address=fe80::DD:EE:fe05:3a05/64 from-pool="" interface=bridge actual-interface=bridge eui-64=no advertise=no no-dad=no 

 2 DL address=fe80::DD:EE:fe05:3a04/64 from-pool="" interface=ether1 actual-interface=ether1 eui-64=no advertise=no no-dad=no 

 3 DG address=AA:BB:CC:6af:DD:EE:fe05:3a04/64 from-pool="" interface=ether1 actual-interface=ether1 eui-64=no advertise=no no-dad=no

[admin@hex] > /ipv6 route/print detail 
Flags: D - dynamic; X - disabled, I - inactive, A - active; c - connect, s - static, r - rip, b - bgp, o - ospf, d - dhcp, v - vpn, m - modem, y - copy; H - hw-offloaded; + - ecmp 
   DAc   dst-address=AA:BB:CC:6af::/64 routing-table=main gateway=ether1 immediate-gw=ether1 distance=0 scope=10 

   DAc   dst-address=AA:BB:CC:6b0::/64 routing-table=main gateway=bridge immediate-gw=bridge distance=0 scope=10 

   D d   dst-address=AA:BB:CC:6b0::/64 routing-table=main gateway="" blackhole immediate-gw="" distance=1 scope=30 target-scope=10 

   DAc   dst-address=fe80::%ether1/64 routing-table=main gateway=ether1 immediate-gw=ether1 distance=0 scope=10 

   DAc   dst-address=fe80::%bridge/64 routing-table=main gateway=bridge immediate-gw=bridge distance=0 scope=10

tdw · July 20, 2022, 1:43am

On receiving a Router Advertisment containing prefix information (they all should) and the autonomous address-configuration flag set (it typically is) the WAN interface will automatically be assigned a GUA formed from the prefix and EUI-64 address generated from the interface MAC address, in just the same way non-router endpoints such as PCs would.

Note this RA prefix is not the same as that obtained through DHCPv6 prefix delegation. The RA prefix supplies the WAN subnet, prefix delegation supplies the routed LAN subnet(s).

I suspect that from the odd snippets of output in earlier posts the LAN (i.e. the bridge interface) host part of the address is all zeros, it should not be as this is a reserved address. You can use ::1/64 from the pool, or similar, if you want a specific fixed address, otherwise use the EUI64 option to generate an address automatically from the interface MAC address.

NotPaul · July 20, 2022, 2:04am

Hello twd,

That is informative and makes sense, thank you.

In the prior configuration the host component of the v6 address was indeed all 0s. In the new configuration with router advertisements enabled that’s no longer the case. As I indicated I now have connectivity from the router, but not form the LAN.

In all cases, however, the MAC address of the gateway remains the same (DD:EE:FF:4F:52:E8). This is both before and after accepting router advertisements. Just for some reasons icmp packets with a source ipv6 address of the WAN subnet now obtained via RA (AA:BB:CC:6af:DD:EE:fe05:3a04/64) elicit responses, but identical packets (same src/dst mac, etc) with a source ipv6 address from the prefix delegation, do not.

Given that some packets destined for the open Internet sent to DD:EE:FF:4F:52:E8 succeed and some packets sent to DD:EE:FF:4F:52:E8 appear to be dropped, based on source IP, has me confused.
-Paul

tdw · July 20, 2022, 2:27am

I was referring to the IPv6 address being added to the LAN (interface=bridge) from the pool, not the WAN (interface=ether1) address.

If ICMP works but other traffic doesn’t it suggests a firewall issue. Post the output of /ipv6 export after redacting any public IP addresses, etc.

NotPaul · July 20, 2022, 2:37am

Hello twd,

ICMPv6 traffic originating from the Mikrotik itself now works (ping on device). I haven’t tried to generate other kinds of traffic from the router.

No traffic, including ICMPv6 traffic, originating from the LAN is able to solicit a response. Thus I don’t think this is a firewall issue. I see the LAN originating outbound ICMPv6 packets on the packet sniffer on ether1, tx. They are identical to the packets from the router that elicit a response, except with a different source ipv6 address.

Export as requested. These are the default v6 firewall rules. plus 3 icmp6 allow rules I added to the top for counter reasons.

[admin@hex] > /ipv6 export 
# jul/19/2022 22:33:06 by RouterOS 7.3.1
# software id = D7KF-P1C1
#
# model = RB750Gr3
# serial number =
/ipv6 address
add address=::1 from-pool=v6 interface=bridge
/ipv6 dhcp-client
add interface=ether1 pool-name=v6 request=prefix
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
add address=::224.0.0.0/100 comment="defconf: other" list=bad_ipv6
add address=::127.0.0.0/104 comment="defconf: other" list=bad_ipv6
add address=::/104 comment="defconf: other" list=bad_ipv6
add address=::255.0.0.0/104 comment="defconf: other" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=forward out-interface=ether1 protocol=icmpv6
add action=accept chain=input in-interface=ether1 protocol=icmpv6
add action=accept chain=output out-interface=ether1 protocol=icmpv6
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=33434-33534 protocol=udp
add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=input comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=forward comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
/ipv6 settings
set accept-redirects=no accept-router-advertisements=yes max-neighbor-entries=8192

Note I changed assignment from the pool to ::1 from ::0, as you sketched. No change in behavior from the LAN.

NotPaul · July 20, 2022, 2:41am

I should add: The counters for all the firewall deny rules remain at 0 during these tests.

tdw · July 20, 2022, 2:55am

Nothing immediately obvious. If you use any online IPv6 web tools to ping AA:BB:CC:6b0::1 do you see anything with the packet sniffer on ether1 RX?

NotPaul · July 20, 2022, 3:10am

Using one of the online ping tools:

AA:BB:CC:6af:DD:EE:fe05:3a04 works
AA:BB:CC:6b0::1 yields “Destination unreachable: No route” on the tool, nothing on the sniffer. But seems to (coincidentally?) trigger ICMPv6 activity between my dhcpv6 local link addr and my ether1 local link addr.
AA:BB:CC:6b0:AB:BC:CD:DE which is the IP assigned to a device on my lan, yields the same as the previous bullet (no route, icmp activity between local links)

I’ll add: The no route bit is a new development, I believe since I began accepting router advertisements.

tdw · July 20, 2022, 11:31pm

It is odd that the sniffer is not picking up the ICMP activity. If you use an online traceroute tool, instead of ping, how far does the trace reach?

NotPaul · July 21, 2022, 1:21am

Hello tdw,

Thanks for the idea. Wow, it dies 6 hops from my device at the very edge of my ISP. This looks like an ISP mess. I’ll circle back to them. Thank you.
-Paul

NotPaul · August 13, 2022, 4:32pm

To close the loop here for those googling in the future:

The problem was on the ISP side. The ISP, which also uses Mikrotik routers, reported that a change occurred in v6.48.6 that caused their v6 prefix delegation routing configuration to no longer function. They moved to passthrough routing and the problem resolved.

So it was a Mikrotik problem, just not my Mikrotik problem.

Thank you sob and tdw.
-Paul