Multiple IPv6 prefixes when using macvlans

gfunkdave · September 25, 2024, 9:17pm

I have AT&T fiber internet and a RB5009 running 7.16. They require you to use their gateway, which they provision with a /60. They reserve the first 8 prefixes for themselves and will allow you to request the remaining 8 prefixes. If you request anything but a /64, it doesn’t matter - the gateway will only provide a single /64 at a time, one per MAC address.

I have two VLANs: guest and LAN. I have configured two MACVLAN interfaces on ether8, my WAN port. I have set each of them to correspond to one VLAN. The IPv6 address pool for each VLAN gets a single prefix. But devices that connect to either VLAN get IPv6 addresses in two separate /64 subnets: one from the correct address pool and one that does not appear anywhere else. Can anyone help me understand what’s happening?

/interface macvlan
add interface=ether8 mac-address=F6:66:7A:0C:77:E0 mode=private name=\
    macvlan-guest
add interface=ether8 mac-address=2A:E8:A2:02:7A:03 mode=private name=\
    macvlan-lan

/ipv6 address
add address=::1 from-pool=lan-ipv6 interface=vlan-lan
add address=::1 from-pool=guest-ipv6 interface=vlan-guest
/ipv6 dhcp-client
add add-default-route=yes interface=macvlan-lan pool-name=lan-ipv6 request=\
    prefix use-interface-duid=yes use-peer-dns=no
add add-default-route=yes interface=macvlan-guest pool-name=guest-ipv6 \
    request=prefix use-interface-duid=yes use-peer-dns=no
    
/ipv6 nd
set [ find default=yes ] advertise-dns=no interface=vlan-lan ra-interval=\
    10s-30s ra-lifetime=5m
add advertise-dns=no interface=vlan-guest ra-interval=10s-30s ra-lifetime=5m
/ipv6 nd prefix default
set preferred-lifetime=30m valid-lifetime=12h
/ipv6 settings
set accept-router-advertisements=no

This results in:

[david@RoutyMcRouterson] > /ipv6/address print
Flags: D - DYNAMIC; G - GLOBAL, L - LINK-LOCAL
Columns: ADDRESS, FROM-POOL, INTERFACE, ADVERTISE
 #    ADDRESS                       FROM-POOL   INTERFACE      ADVERTISE
 0  G fddc::100/64                              wireguard1     no
 1  G 2600:1700:7c50:379a::1/64     lan-ipv6    vlan-lan       yes
 2  G 2600:1700:7c50:3799::1/64     guest-ipv6  vlan-guest     yes
 3 DL fe80::bec1:da6a:de90:d3aa/64              wireguard1     no
 4 D  ::1/128                                   lo             no
 5 DL fe80::4aa9:8aff:fed0:92e3/64              vlan-lan       no
 6 DL fe80::4aa9:8aff:fed0:92e3/64              bridge1        no
 7 DL fe80::4aa9:8aff:fed0:92e3/64              vlan-guest     no
 8 DL fe80::4aa9:8aff:fed0:92e9/64              ether8         no
 9 DL fe80::28e8:a2ff:fe02:7a03/64              macvlan-lan    no
10 DL fe80::f466:7aff:fe0c:77e0/64              macvlan-guest  no

Devices attached to the LAN get addresses from 2600:1700:7c50:379a::/64 and from 2600:1700:7c50:379c::/64. I have no idea where the 379c addresses are coming from.

CGGXANNX · September 26, 2024, 3:55am

Can you check on the client devices whether the addresses with the unexpected prefixes are marked as “deprecated” and no longer “preferred”. On Windows you run ipconfig /all to see this information next to the IP address. On Linux use ip addr, deprecated addresses are listed with “deprecated” on the same line.

If that’s the case, then what you observe is normal. When you made change to the macvlan, dhcpv6 client, bridge and vlan interfaces or the /ipv6 address entries, new prefixes might be acquired and then advertised on the vlan-lan and vlan-guest interfaces. On the clients however, the old prefixes do not immediately disappear. Instead, they become “deprecated” and stay listed until the “valid lifetime” of the prefix expires. If you look at your settings:

/ipv6 nd prefix default
set preferred-lifetime=30m valid-lifetime=12h

You currently have the default value for “valid-lifetime” set to 12 hours. Which means the addresses with the old prefixes will stay in deprecated states for up to 12 hours before they disappear. If that bothers you, you can reduce the default value for valid-lifetime (it should be greater or equals to preferred-lifetime). On my routers I have both values set to only 10 minutes without encountering any problems.

gfunkdave · September 26, 2024, 3:52pm

THanks, I’ve changed the valid-lifetime and preferred-lifetime to 10 mins. And I see that the “c” address shows as deprecated. I’m wondering why it would still be around, long after 12 hours have passed since my initial config changes.

david@zoidberg:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether d8:bb:c1:50:3b:dc brd ff:ff:ff:ff:ff:ff
    inet 192.168.4.5/24 brd 192.168.4.255 scope global dynamic noprefixroute enp3s0
       valid_lft 43184sec preferred_lft 43184sec
    inet6 2600:1700:7c50:379c:ea18:90b9:7649:4e33/64 scope global deprecated dynamic mngtmpaddr noprefixroute
       valid_lft 43195sec preferred_lft 0sec
    inet6 2600:1700:7c50:379a:e144:55d5:eec6:a0cf/64 scope global temporary dynamic
       valid_lft 7185sec preferred_lft 595sec
    inet6 2600:1700:7c50:379a:6ca4:242e:d894:63dd/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 7185sec preferred_lft 595sec
    inet6 fe80::d224:407e:ce38:93ec/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

Edit: here’s something odd. I came back to that same Linux box just now and the valid-lifetime of that weird “c” address keeps getting updated to 43200 seconds (12 hours). What would cause that? All the other IPv6 addresses seem to have gotten the message about the valid-lifetime only being 10 minutes.

CGGXANNX · September 26, 2024, 5:40pm

That’s strange. That means the deprecated prefix keeps being advertised (as deprecated). Can you check in the IPv6 → ND → Prefixes table to see whether that 9c prefix is listed there. If yes you can probably delete the entry.

gfunkdave · September 26, 2024, 8:40pm

It isn’t in there. I wonder if it could be bleeding through from the AT&T gateway upstream of the router somehow? But I have /ipv6/settings/accept-router-advertisements=no so I’m not sure how that could be.

[david@RoutyMcRouterson] > /ipv6 nd prefix print
Flags: X - disabled, I - invalid; D - dynamic
 0  D prefix=2600:1700:7c50:379a::/64 6to4-interface=none interface=vlan-lan
      on-link=yes autonomous=yes valid-lifetime=10m preferred-lifetime=10m

 1  D prefix=2600:1700:7c50:3799::/64 6to4-interface=none interface=vlan-guest
      on-link=yes autonomous=yes valid-lifetime=10m preferred-lifetime=10m

And yet, the AT&T gateway doesn’t show that prefix at all either:

gfunkdave · September 26, 2024, 9:18pm

Update: the router appears to be advertising old prefixes for some reason.

I disabled IPv6 on the router and on the AT&T gateway for about 15 minutes, then re-enabled it. While it was disabled I ran tcpdump on an Ubuntu box on my LAN. As expected there were no RAs received. After re-enabling, the gateway picked up two new prefixes, which are reflected in the router:

[david@RoutyMcRouterson] > /ipv6/nd/prefix/print
Flags: X - disabled, I - invalid; D - dynamic
 0  D prefix=2600:1700:7c50:379f::/64 6to4-interface=none interface=vlan-lan
      on-link=yes autonomous=yes valid-lifetime=10m preferred-lifetime=10m

 1  D prefix=2600:1700:7c50:379e::/64 6to4-interface=none interface=vlan-guest
      on-link=yes autonomous=yes valid-lifetime=10m preferred-lifetime=10m
      
[david@RoutyMcRouterson] > ipv6 address print
Flags: D - DYNAMIC; G - GLOBAL, L - LINK-LOCAL
Columns: ADDRESS, FROM-POOL, INTERFACE, ADVERTISE
#    ADDRESS                       FROM-POOL   INTERFACE      ADVERTISE
0  G fddc::100/64                              wireguard1     no
1  G 2600:1700:7c50:379f::1/64     lan-ipv6    vlan-lan       yes
2  G 2600:1700:7c50:379e::1/64     guest-ipv6  vlan-guest     yes
3 D  ::1/128                                   lo             no
4 DL fe80::28e8:a2ff:fe02:7a03/64              macvlan-lan    no
5 DL fe80::4aa9:8aff:fed0:92e9/64              ether8         no
6 DL fe80::4aa9:8aff:fed0:92e3/64              bridge1        no
7 DL fe80::4aa9:8aff:fed0:92e3/64              vlan-lan       no
8 DL fe80::4aa9:8aff:fed0:92e3/64              vlan-guest     no
9 DL fe80::f466:7aff:fe0c:77e0/64              macvlan-guest  no

But as you can see, the box is still getting RAs for the 379a prefix but not the 379c prefix anymore:

16:09:42.552875 IP6 (class 0xc0, flowlabel 0x68cd7, hlim 255, next-header ICMPv6 (58) payload length: 88) fe80::4aa9:8aff:fed0:92e3 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 88
        hop limit 0, Flags [none], pref medium, router lifetime 300s, reachable time 0ms, retrans timer 0ms
          source link-address option (1), length 8 (1): 48:a9:8a:d0:92:e3
            0x0000:  48a9 8ad0 92e3
          prefix info option (3), length 32 (4): 2600:1700:7c50:379f::/64, Flags [onlink, auto], valid time 600s, pref. time 600s
            0x0000:  40c0 0000 0258 0000 0258 0000 0000 2600
            0x0010:  1700 7c50 379f 0000 0000 0000 0000
          prefix info option (3), length 32 (4): 2600:1700:7c50:379a::/64, Flags [onlink, auto], valid time 600s, pref. time 0s
            0x0000:  40c0 0000 0258 0000 0000 0000 0000 2600
            0x0010:  1700 7c50 379a 0000 0000 0000 0000

Since it isn’t receiving RAs for 379c anymore, that address’s valid lifetime on that same Ubuntu box is reliably ticking down and not being renewed anymore…but the old 379a one is in its place.

david@zoidberg:~$ ip addr

2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether d8:bb:c1:50:3b:dc brd ff:ff:ff:ff:ff:ff
    inet 192.168.4.5/24 brd 192.168.4.255 scope global dynamic noprefixroute enp3s0
       valid_lft 23073sec preferred_lft 23073sec
    inet6 2600:1700:7c50:379c:ea18:90b9:7649:4e33/64 scope global deprecated dynamic mngtmpaddr noprefixroute
       valid_lft 41287sec preferred_lft 0sec
    inet6 2600:1700:7c50:379f:38da:1878:2e3a:3c8/64 scope global temporary dynamic
       valid_lft 584sec preferred_lft 584sec
    inet6 2600:1700:7c50:379f:bbc7:db89:bf5b:7ff/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 584sec preferred_lft 584sec
    inet6 2600:1700:7c50:379a:6ca4:242e:d894:63dd/64 scope global deprecated dynamic mngtmpaddr noprefixroute
       valid_lft 584sec preferred_lft 0sec
    inet6 fe80::d224:407e:ce38:93ec/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

This is a bug in the Mikrotik IPv6 stack, right?

CGGXANNX · September 27, 2024, 4:16am

Yeah. The phantom prefixes look like bugs to me. Are they still there if you reboot the router (just the router, not the modem or client devices)?

On the clients, deprecated addresses are normally “harmless”. The OS and the apps won’t use them for new connections anymore, and also don’t advertise that they have them. While they are still listed (as deprecated) on the devices, the devices will still accept incoming packets with these addresses in the destination field. So that an ongoing connection can still be maintained in case the remote party hasn’t noticed that the address has changed.

gfunkdave · September 27, 2024, 2:47pm

I’ll try rebooting the router later today, but in the meantime I’ve discovered something else. I haven’t changed anything since yesterday. The only thing I’m aware of that has changed is the 12 hour timeout has elapsed. Now, the router is no longer advertising the old prefix. Great! But instead it is advertising the prefix from the other vlan as deprecated.

david@zoidberg:~$ ip addr
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether d8:bb:c1:50:3b:dc brd ff:ff:ff:ff:ff:ff
    inet 192.168.4.5/24 brd 192.168.4.255 scope global dynamic noprefixroute enp3s0
       valid_lft 25736sec preferred_lft 25736sec
    inet6 2600:1700:7c50:379f:bbc7:db89:bf5b:7ff/64 scope global deprecated dynamic mngtmpaddr noprefixroute
       valid_lft 594sec preferred_lft 0sec
    inet6 2600:1700:7c50:379e:6547:8ef8:3cb6:fbbe/64 scope global temporary dynamic
       valid_lft 594sec preferred_lft 594sec
    inet6 2600:1700:7c50:379e:45d3:6f05:d840:4c94/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 594sec preferred_lft 594sec
    inet6 fe80::d224:407e:ce38:93ec/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

gfunkdave · September 27, 2024, 7:41pm

After rebooting the router, it seems to be working. I also changed an aspect of the configuration in the meantime. I removed one of the macvlans and assigned a DHCP6 client directly to the ether8 interface.

mpkossen · October 24, 2024, 11:44am

Thanks very much for sharing this. VRRP was giving me a headache and this appears to be working much better.

gfunkdave · October 29, 2024, 6:29pm

Update to this. Mikrotik replied to my bug report ticket. Apparently the router continuing to advertise old prefixes as deprecated is the expected behavior:

When a router advertises a prefix, it announces its validity time and the client uses this prefix for such period. If an advertising device changers prefix, it announces a new prefix and clients starts to use it, but it does not forget the old prefix. That is why router advertises also old prefixes with lifetime set to 0. End devices receive new update of a prefix which was used before and determines that lifetime for it is 0 now. This means that the prefix is deprecated and should not be used any more. This is completely normal and expected behaviour. If an end device receives a prefix with lifetime set to 0, but it still uses this prefix, then the problem most likely is on the end device, not on the router.

Old prefixes are stored in RAM and RAM is cleared on a reboot.