[SOLVED] IPv6 policy routing with multiple routing tables - which gateway to use?

RouterOS General
1

Hello,

I have 2 WANs with their respective /56s and would like to distribute each to its respective VLAN, so that clients on VLAN1 would get an IP from ISP1 and clients on VLAN2 would get an IP from ISP2. This is not provider-independent space (and the ISPs wouldn’t support that anyway) so I need to make sure that the traffic actually gets routed to the proper ISP.

I am currently doing it as follows:

  • mangle rule to tag relevant traffic with a routing mark
  • routing table corresponding to said routing mark
  • default route in each routing table with a gateway of “%” for the corresponding ISP - this bit doesn’t work

The problem I see is that it does not appear to work unless I also have a default route in the main table, but then I am only able to have one default route (having them with the same “distance” results in seemingly-undefined behavior where the last added route “wins”). I have tried setting “fe80::1%” as the gateway but that does not appear to work either.

It seems like I would be able to make it work if I had a gateway IP address for each ISP, then I could reference said gateway in each ISP’s respective routing tables, and have a specific route for each gateway in the main table. However, in IPv6, it doesn’t look like I have a gateway address?


Description:

fr-wan: WAN interface for ISP1
uk-wan-l2tp: WAN interface for ISP2

ISP1/fr-wan prefix: fd7a:95ad:8f85:9900::/56
ISP2/uk-wan-l2tp prefix: fde7:3a9c:93e5:a6e0::/60
(first 3 groups censored by sample ULA address block for privacy, but these are otherwise publicly-routable IP addresses; censoring is consistent within the entire file)

fr-lan & fr-untrusted-lan: VLAN which should go out via ISP1
uk-lan & uk-untrusted-lan: VLAN which should go out via ISP2


Relevant config:

# WAN interfaces
/interface pppoe-client add comment="upstream ftth" name=fr-wan
/interface l2tp-client add comment="upstream l2tp" name=uk-wan-l2tp

# LAN interfaces
/interface vlan add comment="french lan" interface=bridge1 name=fr-lan vlan-id=100
/interface vlan add comment="french untrusted lan" interface=bridge1 name=fr-untrusted-lan vlan-id=110
/interface vlan add comment="uk lan" interface=bridge1 name=uk-lan vlan-id=200
/interface vlan add comment="uk untrusted lan" interface=bridge1 name=uk-untrusted-lan vlan-id=210

# interface lists, used in firewall rules below
/interface list add comment="trusted lans" include=LAN name="trusted lans"
/interface list add comment="untrusted lans" name=untrusted-lans
/interface list add comment="to go out via UK" name=out-via-uk
/interface list add comment="to go out via fr" name=out-via-fr
/interface list add comment="allow wan access" include=out-via-fr,out-via-uk name="allow wan access"
/interface list add comment="allow basic network services - DHCP, DNS, ICMP, etc" include="trusted lans,untrusted-lans" name=allow-basic-network-services

# interface list members
/interface list member add interface=fr-wan list=WAN
/interface list member add interface=uk-wan-l2tp list=WAN
/interface list member add interface=fr-lan list="trusted lans"
/interface list member add interface=uk-lan list="trusted lans"
/interface list member add interface=fr-lan list=out-via-fr
/interface list member add interface=fr-untrusted-lan list=out-via-fr
/interface list member add interface=uk-lan list=out-via-uk
/interface list member add interface=uk-untrusted-lan list=out-via-uk
/interface list member add interface=fr-untrusted-lan list=untrusted-lans
/interface list member add interface=uk-untrusted-lan list=untrusted-lans

# routing tables
/routing table add disabled=no fib name=out-via-fr
/routing table add disabled=no fib name=out-via-uk

# IPv6 misc config - ND, etc
/ip neighbor discovery-settings set discover-interface-list=LAN
/ip settings set rp-filter=loose
/ipv6 settings set accept-router-advertisements=no

# IPv6 addresses
/ipv6 address add address=fd7a:95ad:8f85:9904:: comment="fr lan" interface=fr-lan
/ipv6 address add address=fd7a:95ad:8f85:9905:: comment="fr untrusted lan" interface=fr-untrusted-lan
/ipv6 address add address=fde7:3a9c:93e5:a6e0:: advertise=no comment="uk internet-facing" interface=uk-wan-l2tp
/ipv6 address add address=fde7:3a9c:93e5:a6e2:: comment="uk untrusted lan" interface=uk-untrusted-lan
/ipv6 address add address=fd7a:95ad:8f85:9900:: advertise=no comment="fr internet facing" interface=fr-wan
/ipv6 address add address=fde7:3a9c:93e5:a6e1:: comment="uk lan" interface=uk-lan

# IPv6 routing
/ipv6 route add disabled=no distance=1 dst-address=::/0 gateway=fe80::1%fr-wan routing-table=out-via-fr suppress-hw-offload=no
/ipv6 route add disabled=no distance=1 dst-address=::/0 gateway=fe80::1%uk-wan-l2tp routing-table=out-via-uk suppress-hw-offload=no
/ipv6 route add comment="system fr" disabled=no distance=1 dst-address=::/0 gateway=fe80::1%fr-wan routing-table=main suppress-hw-offload=no
/ipv6 route add comment="system uk" disabled=no distance=1 dst-address=::/0 gateway=fe80::1%uk-wan-l2tp@main routing-table=main suppress-hw-offload=no

# IPv6 firewall - filter
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
/ipv6 firewall filter add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept UDP traceroute" port=33434-33534 protocol=udp
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/10
/ipv6 firewall filter add action=accept chain=input comment="accept dhcpv6" dst-port=547 in-interface-list=allow-basic-network-services protocol=udp src-port=546
/ipv6 firewall filter add action=accept chain=input comment="accept DNS" dst-port=53 in-interface-list=allow-basic-network-services protocol=udp
/ipv6 firewall filter add action=accept chain=input comment="accept DNS TCP" dst-port=53 in-interface-list=allow-basic-network-services protocol=tcp
/ipv6 firewall filter add action=accept chain=input comment="allow wg" dst-port=13231 protocol=udp
/ipv6 firewall filter add action=accept chain=input comment="accept http, https, ssh from wan" dst-port=2222,80,443 in-interface-list=WAN protocol=tcp
/ipv6 firewall filter add action=drop chain=input comment="defconf: drop everything else"
/ipv6 firewall filter add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
/ipv6 firewall filter add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
/ipv6 firewall filter add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
/ipv6 firewall filter add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
/ipv6 firewall filter add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" hop-limit=equal:1 protocol=icmpv6
/ipv6 firewall filter add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=icmpv6
/ipv6 firewall filter add action=accept chain=forward comment="defconf: accept HIP" protocol=139
/ipv6 firewall filter add action=accept chain=forward comment="allow from LAN to WAN" in-interface-list="allow wan access" out-interface-list=WAN
/ipv6 firewall filter add action=accept chain=forward comment="allow between trusted LANs" in-interface-list="trusted lans" out-interface-list="trusted lans"
/ipv6 firewall filter add action=drop chain=forward comment="defconf: drop everything else"

# IPv6 firewall - mangle
/ipv6 firewall mangle add action=mark-connection chain=input in-interface=uk-wan-l2tp new-connection-mark=received-from-uk passthrough=no
/ipv6 firewall mangle add action=mark-connection chain=input in-interface=fr-wan new-connection-mark=received-from-fr passthrough=no

# IMPORTANT - mark outbound connections with their respective routing mark - this is where the problem might be?
/ipv6 firewall mangle add action=mark-routing chain=prerouting dst-address-list=!our-addresses dst-address-type=!local in-interface-list=out-via-fr new-routing-mark=out-via-fr passthrough=yes src-address-list=our-addresses
/ipv6 firewall mangle add action=mark-routing chain=prerouting dst-address-list=!our-addresses dst-address-type=!local in-interface-list=out-via-uk new-routing-mark=out-via-uk passthrough=yes src-address-list=our-addresses

/ipv6 firewall mangle add action=mark-routing chain=output connection-mark=received-from-fr new-routing-mark=out-via-fr passthrough=no
/ipv6 firewall mangle add action=mark-routing chain=output connection-mark=received-from-uk new-routing-mark=out-via-uk passthrough=no

# IPv6 ND - advertise the subnet-specific DNS server address
/ipv6 nd add dns=fd7a:95ad:8f85:9904:: hop-limit=64 interface=fr-lan
/ipv6 nd add dns=fd7a:95ad:8f85:9905:: hop-limit=64 interface=fr-untrusted-lan
/ipv6 nd add dns=fde7:3a9c:93e5:a6e1:: hop-limit=64 interface=uk-lan
/ipv6 nd add dns=fde7:3a9c:93e5:a6e2:: hop-limit=64 interface=uk-untrusted-lan

I would appreciate if an expert could either go over the config and tell me where I’m going wrong, or alternatively outline how to do this from scratch.

Thanks!

2

But it isn’t required to further complicate things by having “local” addresses on the interfaces, at least not when your /56 networks are static.
I have it working with just some simple mangle routes that set the outgoing route depending on source address (for 2 different providers).

3

Create two VRFs (one for FR, the other for UK) and put the relevant interfaces into each one. You can then have a default route per VRF.

4

Edit: thanks all, I got this running with VRFs and manual inter-VRF routes.

Would still be keen to see a working config of IPv6 policy routing just to understand where my previous attempt went wrong?

Also, got a minor problem - it seems like that from a VRF it’s not possible to reach any of the services on the router such as DNS, HTTP, etc (even on a non-VRF interface’s IP) - inter-VRF routing is set up and machines on both sides (the VRF and the “main” routing table) can reach each other, yet from the VRF I am unable to reach any of the router’s own hosted services even on IPs of interfaces not part of the VRF. Is this a known issue or should I investigate further?

5

It works for me without any VRF by just assigning IPv6 ranges from the range given by the provider to local interfaces, and using the source address in the policy rules to do the routing.
I.e. use a mangle rule for traffic from a local range not to another local range (using address lists) and setting the appropriate routing mark, and have two routing tables each with the appropriate default rule.
So the routing is entirely determined by the source address.

You can have multiple ranges on the same interface and then the client will choose which source address to use and hence which outgoing provider will be used.
In practice I see clients choose the source address that is “nearest” to the destination address to use, so when you want to use this for load balancing it helps when your assigned ranges are a bit apart from eachother (we have a 2001 and a 2a02 range).

6

Would you be able to share what your routing tables look like? I think in my previous configuration I had multiple routing tables and mangle rules configured correctly, the problem is that I couldn’t find a way to break out of the provider-specific routing table without having to go through the main table’s default route (of which I can have only one).

7

My 2 provider-specific route tables contain only a default route via the gateway to that provider.
That means you need to be careful to only put the routing-mark on traffic that is actually going to exit from your router!
So make an address list with all local ranges (I have two /48 ranges but I also put f800::/5 in that) and only mark traffic where the source is in that range and the destination is NOT in that range. I also have dst-address-type=!local in the mangle rule for good measure.

8

What gateway addresses did you use, and how did you obtain them?

In my situation with a PPPoE and L2TP interface, I don’t have explicit gateway addresses - both gateways are “%” which are dependent on a default ::/0 route in the main table (of which I can only have one).

9

Ok the case where I use two providers just has static addresses over ethernet, both providers are on fiber. No PPPoE.
But at home I have PPPoE and the IPv6 default gateway is just a ::/0 route to the PPPoE interface name, no gateway address.
I am not aware of restrictions what one could have in the seconday routing tables, but maybe there are some.

10

Generally services can only be accessed from a single VRF. Usually that’s the main VRF but for a lot of services you can specify which VRF you want them to respond within. What you can’t do is have them respond in multiple VRFs.