I’ve come across a weird problem with my setup at a DC on a CHR (ROS: 7.10.1). We’ve been assigned a /48 prefix, say, 2001:5555:6::/48. I can use 2001:5555:6::1 as the DC-provided gateway, so I added 2001:5555:6::2 to my uplink VLAN interface. I also added a default route to 2000::/3 via 2001:5555:6::1. I can access/ping public IPv6 addresses from the router, and can ping the 2001:5555:6::2 address from external hosts on the internet. The 2001:5555:6:2::1/64 address is added to a VLAN, and it’s advertising the prefix.
VLAN 10 is the uplink interface and VLAN 2 is an internal one for VMs behind the router. The VM interfaces are added to a VLAN like an access port (untagged traffic). IPv4 is working without a hitch.
Now, the problem. Internal hosts on VLAN 2 get an IP, can ping 2001:5555:6::2 (public facing address on VLAN 10) or 2001:5555:6:2::1 (internal router address on VLAN 2), but they can’t even ping 2001:5555:6::1 (the ISP-provided GW). When I start a ping from an internal host (say, 2001:5555:6:2::10), the v6 ICMP Echo Request packets arrive at the remote host, it sends back the reply. The reply is received on the router on VLAN 10 (public side), but then the traffic stops and they are not sent back to VLAN 2 where the request originated. So all seems to appear fine and dandy until the MT router decides not to forward the returning packets.
Pinging an internal public v6 address from the internet (like 2001:5555:6:2::10) doesn’t work, either. Pinging 2001:5555:6::2 or 2001:5555:6::1 does work. They are not forwarded traffic, though.
This seems very simple, so I’m baffled at why it doesn’t work. I remember someone mentioning on the forum that “simple” (as in, non-VLAN) interfaces work in a similar scenario. Actually that’s my own experience as well. But now it might not be relevant. Any help is appreciated.
I suspect that setting frame-types to admit-only-vlan-tagged on the bridge is not sufficient alone and you also have to set ingress-filtering to yes if you want VLAN 2 to pass tagged through the bridge port of the virtual switch and the /interface/vlan interface to process it at the bridge interface of the router. And there is a similar inconsistency on ether1 - under /interface/bridge/vlan you say that ether1 is a tagged member of VLAN 2, but at the same time ingress-filtering is set to no, no frame-types are specified, and pvid is set to 2 on the row for ether1 in /interface/bridge/port, which sets access mode for VLAN 2 on ether1 for ingress.
So how do you actually want VLAN 2 to run through ether1, tagged or tagless (trunk or access mode)?
Yeah, sorry about the snippet - the defaults actually turn on a few things that aren’t apparent from a simple (compact) export that only shows settings that differ from the defaults. Here’s the verbose export. Ingress filtering is turned on. VLANs work without issues and in the expected way for IPv4 and other purposes. The IPv6 firewall is empty for now, btw.
Ah, sorry - TIL that whilst in RouterOS 6, no was the default for ingress-filtering, in 7.15.3 it is yes - not sure when exactly this change took place. As I saw pvid=2 on both the default port of the bridge and the row that makes ether1 a bridge port, I did not even think about such a change.
Sometimes VLAN interfaces do behave weird. E.g. if the IPsec transport packets land on a VLAN interface (also a ROS 6 case, I haven’t had such a setup yet on a ROS 7 device), IPsec does not hand over the DHCPINFORM packet received from the Windows to the DHCP server (so the Windows do not get the routing table they ask for).
So as a diagnostic step, and maybe a temporary workaround, try making your vlan 2 interface a single member port of a dedicated bridge with protocol-mode set to none and vlan-filtering set to no and move all the IP and IPv6 related configuration items from the vlan 2 interface to this auxiliary bridge.
Out of interest how is the provider supplying the /48? If they are just presenting a /48 directly rather than routing it via a transit subnet they are one of the providers who really don’t know what they are doing with IPv6 - see https://www.ripe.net/publications/docs/ripe-690/ section 4.1 for discussions on the preferable, acceptable and unacceptable methods.
A bodge to fix a direct /48 presentation would be to use use NDproxy, except Mikrotik do not implement it.
Actually, I’ll need to ask them about that. What they provided is a single IPv6 address, like 2001:5555:6::1/48 and that’s all. I can use this address as default route for v6, and any address I put on the router public-facing interface within that /48, is able to run traffic. Outbound traffic is sent out from VLAN clients as well, but the return packets can’t “jump back” to the originating VLAN and packets originating from the internet are not forwarded to the internal public addresses.
Could I run ndproxy in a separate server/VM? That might be a temporary solution, if it works.
According to the description in the OP, folks at your ISP use the method mentioned in par. 4.1.4 of the RIPE document, i.e. they sacrifice a single /64 from the /48 they gave you as a link subnet. Could be worse actually, you still have another 65535 /64s to use in your network. I also got from your OP that you can see the ping responses to actually arrive to your Tik, so at the ISP side, everything is OK with the routing of your /48 (assuming that you have obfuscated your /48 only in some places, because you actually use two distinct /48 prefixes in the OP) and the “only” issue is that the Tik does not forward the responses to the host in VLAN 2 resp. in the 2001:5555:6:2::/64 subnet. Is that understanding correct?
Ugh, sorry. I meant to write a single /48, I fixed the OP now. The substitute network is 2001:5555:6::/48, I don’t have access to multiple /48 prefixes there.
EDIT: yes, the assumptions are correct. Also, inbound connections can’t access internal IPv6 addresses, either.
I don’t know whether it is a bad news or a good one, but I have replicated your setup as closely as I could on a CHR running 7.15.3, and it just works. In my case, ether1 is an access port in VLAN 410, and ether2 is an access port in VLAN 402, /interface/vlan for both vlan IDs are attached to the bridge, and global addresses from distinct subnets are attached to them. A device from an appropriate subnet is connected to each of the two Ethernet interfaces of the CHR, the device connected to ether1 being a main site router, and the device connected to ether2 can happily ping an address in the internet that way:
To make sure I completely understand your setup, could you show the complete, IP-obfuscated export, so I can replicate it on a test CHR?
It might be something that has been fixed in newer versions. I was able to replicate my issue on a switch (CRS326, ROS 7.10.0) that’s before the CHR on the ISP side and provides the trunk on ether1. I know I need to upgrade, but it’s a bit complicated to allocate downtime.
I see. It’s actually quite different, because you configured the ether1, ether2 interfaces as access ports, whereas for me ether1 exists as a trunk. In my virtual setup (Proxmox based) the host and the CHR only carry tagged traffic. Managing inter-VLAN traffic ang general, basic firewalling is the responsibility of the CHR. Individual client/project VMs are attached interfaces in access mode on specific VLANs.
I paste my test config on the test CHR (nothing removed, IPs obfuscated). I made it from scratch but used the same naming scheme. It fails the same way, i. e. IPv6 routing fails like I described earlier. However, your test setup uses access ports on VLANs, and it seems to work. I have a /56 at another DC and there I use an older config with a very similar setup to yours, using only access ports in the virtual ROS (VLAN->interface assignments are done at the HV level via multiple Linux bridges). That one works flawlessly. Between that setup and this the only important change is that here the VLAN interfaces are created on the CHR bridge and there are no ports configured as access ports in the HV. This must be a ROS bug.
# 2024-08-25 18:27:47 by RouterOS 7.15.3
# software id =
#
/interface bridge
add frame-types=admit-only-vlan-tagged name=br1-adm protocol-mode=none pvid=2 \
vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no
set [ find default-name=ether2 ] disable-running-check=no disabled=yes
/interface vlan
add interface=br1-adm name=vlan2-adm vlan-id=2
add interface=br1-adm name=vlan10-wan vlan-id=10
/interface bridge port
add bridge=br1-adm frame-types=admit-only-vlan-tagged interface=ether1 pvid=2
/ip neighbor discovery-settings
set discover-interface-list=!all
/interface bridge vlan
add bridge=br1-adm tagged=ether1,br1-adm vlan-ids=2,10
/ip address
add address=99.99.106.214/28 interface=vlan10-wan network=99.99.106.208
add address=10.192.2.90/24 interface=vlan2-adm network=10.192.2.0
/ip dhcp-client
# DHCP client can not run on slave or passthrough interface!
add interface=ether1
/ip dns
set servers=8.8.8.8,8.8.4.4
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=99.99.106.209 \
routing-table=main scope=30 suppress-hw-offload=no target-scope=10
/ipv6 route
add check-gateway=ping disabled=no dst-address=2000::/3 gateway=\
2001:5555:6::1 routing-table=main suppress-hw-offload=no
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh port=50022
set api disabled=yes
set winbox port=58291
set api-ssl disabled=yes
/ipv6 address
add address=2001:5555:6::90 advertise=no interface=vlan10-wan
add address=2001:5555:6:2::1 interface=vlan2-adm
/system identity
set name=router2test.foobar.org
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=1.pool.ntp.org
add address=2.pool.ntp.org
Indeed, and I have admitted that openly straight away However, I can’t imagine how this difference should be related, given that you can ping the router itself from both sides (VLAN 10 and VLAN 2), so the IPv6 packets clearly do pass through the trunk port to/from the bridge freely in both directions via both VLANs. The issue only occurs when the router has to receive the packet on one VLAN interface and forward it via the other one.
I could see from your export that it is in fact the only difference, though, so it looks like a really convoluted bug as it needs a specific combination of seemingly unrelated settings to show up.
So I took the last step and modified my bridge setup, and changed the ether1 mode from access port to the WAN VLAN to trunk one on the Proxmox side accordingly (I just had to change the ID of the WAN VLAN to the actual one, which is 3 in my network):
Thanks for the help and for digging into it. I guess I’ll try to send a supout from the test VM to support and see what the MT pros say. This arrangement should just work.
I still suspect on their side of the link they are doing the equivalent of
/ipv6 address
add address=2001:5555:6::1/48 interface=etherX
rather than
/ipv6 address
add address=2001:5555:6::1/64 interface=etherX
/ipv6 route
add dst-address=2001:5555:6::/48 gateway=2001:5555:6::xxxx
as they would have to provide or ask for the target address 2001:5555:6::xxxx where they are routing the other subnets to.
I also got from the OP that they can see the ping responses to actually arrive to their Tik
It would be worth the OP confirming that they are ICMP Echo Response and not ICMP Neighbour Solicitation packets, just seeing ICMPv6 packets arrive is insufficient detail.
Only now I understood what you had in mind in your previous post. Yes, indeed, since @kobuki explicitly mentioned “v6 ICMP Echo Request” to arrive to the remote host, I somehow did not expect that (s)he hasn’t checked that what has arrived was indeed an “echo response”, not just “some” ICMPv6.
Such an explanation sounds definitely more logical to me than a bug that requires 10 specific confitions to be met simultaneously in order to show up.
First, let me clarify one thing. I mistakenly said that ICMP replies are coming back on the router when pinging from an internal address - they don’t. MT’s Torch is showing some confusing info. Getting a dump and analyzing that showed this clearly. The remote host sends (or tries to send) them, but they’re not forwarded to the router. The same happens with starting the ping on the remote side, Echo Requests are not coming in. However, I can see neighbor solicitation messages and the solicited-node multicast address from the ISP router that go unanswered.
I noticed another weirdness. I temporarily added an address from the internal prefix to the public VLAN interface. It’s pingable, per usual. Then I moved it to a host behind the internal VLAN interface (VLAN 2). It kept working, and is repeatable. I reset the client’s IPv6 stack and also that of the router and it kept working. It probably means that a neighbor entry got added to the upstream router and until that expires, the configured address works. I verified that I can indeed access the internal host (can ping, ssh, etc). In this case, neighbor solicitation messages get answered properly when the address is added to the public VLAN interface. This is less of a surprise as the router is always accessible when I add ny address from the /48 on the public interface.
So it seems to me that ROS indeed has a bug. BTW: “a bug that requires 10 specific conditions to be met simultaneously in order to show up” – it made me smirk but it’s just inter-vlan IPv6 routing, nothing special. You can see the exact setup in my latest export that I previously attached. About 5-6 lines of commands that are important.
Another bit: I asked the ISP to change how the /48 is routed. They changed it so they provide a /127 address, one for them and one for us. I configured one on our router, they configured the other on theirs. The address is independent of the /48 that it routes. The old method (using an address from the first /64 and routing through PREFIX::1/64) still works. It didn’t fix anything, sadly.
I’m ready to send a bug report and a supout, unless you maybe have any more idea to try…
In @kobuki’s configuration there’s an error in LAN interface setting: IPv6 addresses are set without prefix length, they should have /64 included. Without it, it’s taken to be /128 (just like in IPv4 it’s assumed to be /32).
?
However, I can’t imagine how this difference should be related, given that you can ping the router itself from both sides (VLAN 10 and VLAN 2), so the IPv6 packets clearly do pass through the trunk port to/from the bridge freely in both directions via both VLANs. The issue only occurs when the router has to receive the packet on one VLAN interface and forward it via the other one.