How to configure IPv6 Load Balancing? (PCC, multiple ISPs/GUAs)

Omar007 · Mon Jun 13, 2022 11:28 pm

To match the configuration used for IPv4 to load balance over multiple ISPs, I'm looking to implement the same on the IPv6 side of things (dual stack network). In short, the plan for IPv6 is basically the same as with IPv4:
- Have a routing table for each ISP
- Add a default route in said table for the ISP in question
- Mark incoming connections for each ISP with a respective connection mark (mangle prerouting)
- Mark outgoing connections using PCC with a respective connection mark (mangle prerouting & mangle output)
- Mark routing for the respective connection marks to their respective routing tables
- Transform traffic between external and internal networks

This latter step is for traffic to flow properly and making the outside transparent to the inside network. This means that in IPv4 we have a masquerade rule in the srcnat chain handling the translation of outgoing traffic and, as needed, dst-nat rules in the dstnat chain for port-forwarding (/32 publics to many internal addresses). For IPv6 this means using prefix translation to convert an ULA to the respective GUA for the given ISP and viceversa (/48 publics to /48 internal).
Where this whole thing seems to go wrong for IPv6 is on this very last part. The original idea was to use the mangle dnpt and snpt rules but these require the connections to be untracked and are as such not an option as this would prevent the use of the firewall. That means the next option is using netmap rules in the dstnat (prerouting equivalent for the mangle mark rules) and output chains in the NAT table.
Sadly, this latter option also does not seem to function properly and at this point I'm not sure if I'm just fucking something up or this is just broken in RouterOS atm (I'm on v7.3.1). The rules are currently as follows:

/ipv6 firewall nat
# NPT from ULA to GUA (isp1)
add action=netmap chain=dstnat connection-mark=isp1 src-address=priv:ater:ange::/48 to-address=publ:icra:nge1::/48
add action=netmap chain=output connection-mark=isp1 src-address=priv:ater:ange::/48 to-address=publ:icra:nge1::/48
# NPT from GUA (isp1) to ULA
add action=netmap chain=dstnat dst-address=publ:icra:nge1::/48 to-address=priv:ater:ange::/48
add action=netmap chain=output dst-address=publ:icra:nge1::/48 to-address=priv:ater:ange::/48
# NPT from ULA to GUA (isp2)
add action=netmap chain=dstnat connection-mark=isp2 src-address=priv:ater:ange::/48 to-address=publ:icra:nge2::/48
add action=netmap chain=output connection-mark=isp2 src-address=priv:ater:ange::/48 to-address=publ:icra:nge2::/48
# NPT from GUA (isp2) to ULA
add action=netmap chain=dstnat dst-address=publ:icra:nge2::/48 to-address=priv:ater:ange::/48
add action=netmap chain=output dst-address=publ:icra:nge2::/48 to-address=priv:ater:ange::/48
# NPT from ULA to GUA (isp3)
add action=netmap chain=dstnat connection-mark=isp3 src-address=priv:ater:ange::/48 to-address=publ:icra:nge3::/48
add action=netmap chain=output connection-mark=isp3 src-address=priv:ater:ange::/48 to-address=publ:icra:nge3::/48
# NPT from GUA (isp3) to ULA
add action=netmap chain=dstnat dst-address=publ:icra:nge3::/48 to-address=priv:ater:ange::/48
add action=netmap chain=output dst-address=publ:icra:nge3::/48 to-address=priv:ater:ange::/48

I'd be very happy to hear any suggestions on what is going on here and how this same thing (PCC Load Balancing) can be achieved with IPv6 rules in RouterOS.

Tue Jun 14, 2022 3:52 am

very interesting topic to make suitable dual-stack PCC Load balancers

i have not deployed the first of this but it will be needed that's for sure

DarkNate · Thu Jun 16, 2022 12:08 am

It works the same as IPv4 with PCC/Nth, however, you cannot use GUAs on the LAN interfaces, you will need to use Provider independent address space (not ULA because they are broken in dual stack networks) and perform NPTv6 (stateless NAT66 – For now, not supported in ROS v7) to perform the load balancing between multiple ISPs.

Here's a visual representation:
https://en.wikipedia.org/wiki/IPv6-to-I ... homing.png

Omar007 · Thu Jun 16, 2022 3:04 am

@chechito: it is and as far as I'm aware I've covered everything and on the theoretical side nothing is wrong. Sadly in practice, the IPv4 side works, the IPv6 side doesn't. I have not yet been able to figure out why. It really does look like RouterOS is broken here and everything is actually correct...

@DarkNate: I am aware it works the same way, that is exactly how I've set it up as explained in the OP. Fwiw, you can use GUAs on LAN interfaces just fine but you will not be able to use those addresses for these connection flows then of course but if you're not interested in this type of setup, it's a perfectly fine way to configure things.
Anyway, since I do want to configure multi-home, all my internal interfaces use ULAs. And if you're going this route, your WAN interfaces don't even need a GUA assigned either. This whole setup can be done without any of the GUAs assigned or referenced anywhere (other than in your translation step ofc

)
I'm not entirely sure what you mean with "you will need to use Provider independent address space (not ULA because they are broken in dual stack networks)". This type of thing is done so you don't need to get a RIR assigned address space or do any BGP exchanges etc. The example image you're supplying is also explicitly using an ULA range, which is exactly what the goal is of this use-case in the first place.

---

Anyway, tl/dr, it really looks like this part of the stack is just plain broken in RouterOS atm. I've tried to log what it is actually doing and it does not seem to actually do the translation correctly. It looks like it just flattens the whole source prefix to the all zeroes address for the target prefix instead of mapping the prefix addresses 1-to-1 as it should.
I saw the 7.4beta2 release and tried a quick update to see if that had any effect since it mentioned some changes in the IPv6 stack but the whole release just causes the device to boot-loop so it looks like at this point in time things are just broken on this side and I have no idea if this is fixed in the upcoming version...

Not sure what the best course of action is now tbh. I think I'll probably have to mail about this to support or something?

kevinds · Thu Jun 16, 2022 3:50 am

instead of mapping the prefix addresses 1-to-1 as it should.

Not sure what the best course of action is now tbh. I think I'll probably have to mail about this to support or something?

This is currently broken in RouterOS, nothing you can do about it.. Open a ticket to say "Me too", but they are aware.. IPv6 seems to be a really slow moving part of RouterOS.. There are a number of IPv6 bugs, some of them crippling.

Omar007 · Thu Jun 16, 2022 3:38 pm

Is there an overview available somewhere of known issues? Maybe I've just missed it but if not, it sure would've been nice to know this...
Or even just a list of core things that work in IPv4 but not yet in IPv6? I do know they had one of these i.r.t. hardware offloading but, afaik, not on core functionality.

un9edsda · Sat Jun 18, 2022 3:51 am

@DarkNate: ... I'm not entirely sure what you mean with "you will need to use Provider independent address space (not ULA because they are broken in dual stack networks)".

It might be a reference that by default in line with RFC6724 Default Address Selection for Internet Protocol Version 6 (IPv6) section 2.1. Policy Table ULA address won't be used in case of a dual stack setup https://datatracker.ietf.org/doc/html/r ... ection-2.1

However this issue can be handled on GNU/Linux and Microsoft windows systems.
In case of current GNU/Linux systems all it takes is adding the following two lines to /etc/gai.conf

label fd00::/8 1
precedence fd00::/8 41

I got the idea from the following comment on Reddit https://www.reddit.com/r/ipv6/comments/ ... &context=3

In case of recent Microsoft Windows operating systems it takes only one line in Windows PowerShell run with administrator privilege:

netsh interface ipv6 add prefixpolicy fd00::/8 41 1

as explained in this Super User Q&A https://superuser.com/questions/1469774 ... 78#1469778

Being said that there are two things to keep in mind in case of using NPTv6 and ULA (without GUA on the LAN side) on recent RouterOS (7.3.1 or 7.4beta2). The first one is that

/ipv6 firewall filter
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid

rule included in the Building Advanced Firewall section of the RouterOS documentation https://help.mikrotik.com/docs/display/ ... d+Firewall has to be disabled in the firewall otherwise for example

ping6 -c 5 google.com

would time out.

The second one is that if the

/ipv6 firewall filter
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN

rule (which is included both in the default configuration and the Building Advanced Firewall example) is kept enabled than the IPv6 connectivity test on http://test-ipv6.com/ site will fail.

Anyway, tl/dr, it really looks like this part of the stack is just plain broken in RouterOS atm. I've tried to log what it is actually doing and it does not seem to actually do the translation correctly. It looks like it just flattens the whole source prefix to the all zeroes address for the target prefix instead of mapping the prefix addresses 1-to-1 as it should.

The undocumented NPTv6 feature do work in recent RouterOS versions (at least for single home setup) using the pair of sample firewall rules provided by @Sob in Feature Request: IPv6 NAT66 Support in post #71:

/ipv6 firewall mangle
add chain=postrouting action=snpt src-address=fd00:1234:5678:9a00::/56 src-prefix=fd00:1234:5678:9a00::/56 dst-prefix=publ:icpr:efix:b700::/56
add chain=prerouting action=dnpt dst-address=publ:icpr:efix:b700::/56 src-prefix=publ:icpr:efix:b700::/56 dst-prefix=fd00:1234:5678:9a00::/56

However please read his next (#72) post in that thread for his caveat against it. viewtopic.php?p=934680#p934676
Actually those are the results of RFC6296 section 2.6 Checksum-Neutral Mapping https://datatracker.ietf.org/doc/html/r ... ection-2.6

Omar007 · Sat Jun 18, 2022 3:41 pm

It might be a reference that by default in line with RFC6724 Default Address Selection for Internet Protocol Version 6 (IPv6) section 2.1. Policy Table ULA address won't be used in case of a dual stack setup https://datatracker.ietf.org/doc/html/r ... ection-2.1

Hmm that would still be the wrong conclusion then though because ULAs will work just fine. You're correct in that you need to handle additional stuff if you don't want it to fall back to IPv4 if you try to reach GUAs though. As ULAs would hit fc00::/7, they get labelled with w/e that one is on your system (default 6 on my system iirc) and your destination would be labelled for ::/0 (default 1). A mismatch here means that systems would indeed end up preferring IPv4 over IPv6 if you don't force either but that is irrespective of being able to use ULAs in your IPv6 network or not (applications you force to use IPv6 won't even notice).

However this issue can be handled on GNU/Linux and Microsoft windows systems.
In case of current GNU/Linux systems all it takes is adding the following two lines to /etc/gai.conf
Code: Select all
label fd00::/8 1
precedence fd00::/8 41
I got the idea from the following comment on Reddit https://www.reddit.com/r/ipv6/comments/ ... &context=3

In case of recent Microsoft Windows operating systems it takes only one line in Windows PowerShell run with administrator privilege:
Code: Select all
netsh interface ipv6 add prefixpolicy fd00::/8 41 1
as explained in this Super User Q&A https://superuser.com/questions/1469774 ... 78#1469778

So yea this stuff is relevant to make IPv6 preferred over IPv4 by making sure your source and destinations are labelled the same (both 1 in this case by making the ULAs marked 1 and ::/0 was already 1).

Being said that there are two things to keep in mind in case of using NPTv6 and ULA (without GUA on the LAN side) on recent RouterOS (7.3.1 or 7.4beta2). The first one is that
Code: Select all
/ipv6 firewall filter
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
rule included in the Building Advanced Firewall section of the RouterOS documentation https://help.mikrotik.com/docs/display/ ... d+Firewall has to be disabled in the firewall otherwise for example
Code: Select all
ping6 -c 5 google.com
would time out.

The second one is that if the
Code: Select all
/ipv6 firewall filter
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
rule (which is included both in the default configuration and the Building Advanced Firewall example) is kept enabled than the IPv6 connectivity test on http://test-ipv6.com/ site will fail.

Luckily the firewall is the easy part and completely irrelevant for the problem currently at hand. Yes, if you're using the default firewall and not paying attention to the effect/requirements of your use-cases this may be something to think about. However, at the same time, it's not even relevant if we continue to the next part as dnpt and snpt completely bypass/prevent the use of firewall stuff in the first place as it needs to be untracked.

The undocumented NPTv6 feature do work in recent RouterOS versions (at least for single home setup) using the pair of sample firewall rules provided by @Sob in Feature Request: IPv6 NAT66 Support in post #71:
However please read his next (#72) post in that thread for his caveat against it. viewtopic.php?p=934680#p934676
Actually those are the results of RFC6296 section 2.6 Checksum-Neutral Mapping https://datatracker.ietf.org/doc/html/r ... ection-2.6

The checksum part is completely moot as I can't use this in the first place atm. Not using snpt/dnpt because it completely prevents the use of the firewall (which is why I needed to switch to netmap in the first place) and not using netmap as it's broken in specifically RouterOS apparently (it should be working fine at the kernel/Linux level anyway). Still broken in 7.4beta4 as well :'(
If you know of a way for snpt/dnpt to work with firewalling I'd be happy to hear that but otherwise from my understanding the only thing in existence to do this is to use netmap. Or have a dedicated/separate device in your network for NPT or something...

un9edsda · Fri Feb 17, 2023 6:28 am

Actually those are the results of RFC6296 section 2.6 Checksum-Neutral Mapping https://datatracker.ietf.org/doc/html/r ... ection-2.6
The checksum part is completely moot as I can't use this in the first place atm. Not using snpt/dnpt because it completely prevents the use of the firewall (which is why I needed to switch to netmap in the first place) and not using netmap as it's broken in specifically RouterOS apparently (it should be working fine at the kernel/Linux level anyway). Still broken in 7.4beta4 as well :'(
If you know of a way for snpt/dnpt to work with firewalling I'd be happy to hear that but otherwise from my understanding the only thing in existence to do this is to use netmap. Or have a dedicated/separate device in your network for NPT or something...

My current understanding of the matter is that since NPTv6 translation is stateless therefore one may use the stateless firewall (raw) instead of the stateful nat and filter. However I do know that achieving the same level of security usually is way trickier with a stateless firewall than with a stateful. On the other hand the documentation usually still misses IPv6 case studies (such as this).

How to configure IPv6 Load Balancing? (PCC, multiple ISPs/GUAs)

How to configure IPv6 Load Balancing? (PCC, multiple ISPs/GUAs)

Re: How to configure IPv6 Load Balancing? (PCC, multiple ISPs/GUAs)

Re: How to configure IPv6 Load Balancing? (PCC, multiple ISPs/GUAs)

Re: How to configure IPv6 Load Balancing? (PCC, multiple ISPs/GUAs)

Re: How to configure IPv6 Load Balancing? (PCC, multiple ISPs/GUAs)

Re: How to configure IPv6 Load Balancing? (PCC, multiple ISPs/GUAs)

Re: How to configure IPv6 Load Balancing? (PCC, multiple ISPs/GUAs)

Re: How to configure IPv6 Load Balancing? (PCC, multiple ISPs/GUAs)

Re: How to configure IPv6 Load Balancing? (PCC, multiple ISPs/GUAs)

Who is online