So in the MikroTik wiki, they used action=src nat as an example, whereas, on various MUM presentations, they used action=netmap.
Note: We are NOT doing or interested in deterministic NAT.
So basically this what we want:
NAT multiple subnet slices out of the 100.64.0.0/8 to public/25
And accordingly, allow incoming traffic destined towards public/25 + destined for only ports 1024-65535 to be dst-natted to the various subnet slices out of the 100.64.0.0/8 to allow customers to take advantage of port randomisation and get port forwarding to work correctly for P2P traffic etc.
What are the chances 100.64.0.0/8 customers would all use port 1024 for instance for their Bittorent clients, right? Zero.
This is an imperfect solution compared to IPv6, but we would like to give customers at least a better if not perfect P2P networking experience while IPv6 is being rolled out.
So this is what we’ve tried along with IPSec passthrough attribute:
#src-address-list=local, local is address list containing multiple CGNAT subnets like 100.64.0.0/24, 100.64.0.256/24 etc#
/ip firewall nat
add action=netmap chain=srcnat comment="Netmap for outbound TCP" ipsec-policy=out,none protocol=tcp src-address-list=local to-addresses=public/25 to-ports=1-65535
add action=netmap chain=srcnat comment="Netmap for outbound UDP" ipsec-policy=out,none protocol=udp src-address-list=local to-addresses=public/25 to-ports=1-65535
add action=netmap chain=srcnat comment="Netmap for outbound non TCP/UDP" ipsec-policy=out,none src-address-list=local to-addresses=public/25
#Example we only want to allow accessible for port frowarding for 100.64.8.0/21 instead of everything inside src-address-list=local#
add action=dst-nat chain=dstnat comment="For inbound port forwarding TCP" dst-address=public/25 dst-port=1024-65535 in-interface-list=WAN protocol=tcp to-addresses=100.64.8.0/21 to-ports=1024-65535
add action=dst-nat chain=dstnat comment="For inbound port forwarding UDP" dst-address=public/25 dst-port=1024-65535 in-interface-list=WAN protocol=udp to-addresses=100.64.8.0/21 to-ports=1024-65535
So the above rules, sort of works… On the customer end, we were able to seed torrent traffic without any issues but the ports are still “closed” for the public /25 mapped to the customer at the time of testing, which we checked with the Port Checker: https://portchecker.co/
Is there a proper way of doing CGNAT to allow this to work correctly? I feel something is wrong with the rules themselves.
A different network operator was able to open up ports from the public for their CGNATted customers using MikroTik, we are not sure how they did it.
one thing:
100.64.0.0/8 do “not exist”
the right is
100.64.0.0/10
from
100.64.0.0 to 100.127.255.255
public/25 ???
what is this?
one single address like 2.3.4.5/25
or one pool of public address like 2.3.4.0/25 → from 2.3.4.0 to 2.3.4.127 ???
destined for only ports 1024-65535 to be dst-natted
You open by hand port-to-client?
UPnP work only on client-side
The port used for auto NAT go from 32768 to 65534, port from 0 to 32767, and 65535 are not used automatically for auto NAT
You have only 32767 port per public IP usable for auto NAT, assuming one client use near 500 ports, you can have near 64 users for public IP before the NAT is full…
if you have 128 IP (0 and 127 are IP), with 64 users for IP, the hypotetic max number of users without reach end-of-ports are 8192.
Using P2P can increase the ports per users from 500 to 5000 and max number of users per IP can go from 64 to 6 …
we would like to give customers at least a better if not perfect P2P networking experience
Buy one IPv4 adrress for each customer and more, for use it as dynamic pool of Public IP addresses.
The addresses are NOT out of stock, they are simply for sale and are expensive.
I still don’t get why you need any dst-nat rules at all. To my understanding, the only thing the two peers in any p2p network need is that all the NATs between each peer and the internet do not change the source port (unless it cannot be kept because some other client is connecting from the same source port to the same destination address and port). When this is true for both, they both connect to some rendez-vous point and exchange information about the ports they use at their side and the public IP from which they access internet (detected with help of the rendez-vous point). Once each peer has received this information, both start sending from their local port to the remote address and port of the other peer, thus creating a pinhole (tracked connection in linux/RouterOS vernacular) by means of src-nat. Since UDP is stateless, by sending a packet from c.c.c.c:C to s.s.s.s:S, the pinhole opens also for packets sent from s.s.s.s:S to public.ip.assigned.to.client:C, and once such packet arrives, it gets “un-src-nated” and delivered to c.c.c.c:C. So the packets sent by the first peer to start are not reaching the second one until the second one sends its first packet; since then, both firewalls let the packets through.
For connections between a peer inside your 10.64.0.0/10 and a peer out there in the internet - yes. For connections between two of your peers, it will be more interesting.
In order that it worked, you need to src-nat also the connections your clients initiate towards the addresses in your public /25 range, but that’s not enough. If two clients initiate such connection, RouterOS creates an independent tracked connection for each of them, and it cannot match them together and merge them. So you need the following arrangement (roughly, adapt it to your actual configuration):
The addresses from the /25 subnet you use for src-nat must not be assigned to any interface of the router itself, otherwise it won’t work because packets for any address on the router itself are not routed, unless they get dst-nated at arrival.
The route must have both ends of the IPIP tunnel as gateways because if you set rp-filter to strict (which you may want to do as you are an ISP so dropping forged packets saves you some problems). The action=drop routing rules are there to prevent the packets from looping through the hairpin tunnel until the second tracked connection gets created.
To make the action=src-nat rule act also on the hairpinned traffic, which is necessary for the whole idea to work, you must make both hairpin-1 and hairpin-2 members of interface list WAN (which they indeed are from the point of view of the network topology).
The question is whether you can remove that just like that, as it depends on how that /25 is configured between you and your upstream provider:
if the gateway IP the ISP gave you is from another subnet than this /25 one, the ISP routes that /25 to you via your own IP in that other subnet and there is no need to have the addresses from the /25 up on your router itself
if the gateway IP they gave you is from that /25, your router must respond to ARP requests for all addresses from that /25 you want to use; to do that without setting them up on the router itself, you must create ARP records of type publish on the WAN interface (don’t specify any MAC address, it will save 00:00:00:00:00:00 and the one of the WAN interface will be actually sent) and add individual routes to these /32 addresses that shadow the route to the whole /25 that is added dynamically as you add the own address in the /25.
Assuming your own IP in the /25 is .2 and the gateway provided by the ISP is.1, you’ll need something like /ip arp
add address=the.public.subnet.3 interface=your-wan-interface-name publish=yes
…
add address=the.public.subnet.126 interface=your-wan-interface-name publish=yes
/ip route
add dst-address=the.public.subnet.3 gateway=hairpin-1,hairpin-2
…
add dst-address=the.public.subnet.126 gateway=hairpin-1,hairpin-2
So we used the single src NAT, and it worked fine, but for some reason, we can’t run a speedtest.net test on our test server hosted on the same network with a different IP address of course on the edge router. The Access router is connected to this edge router, where we use Hoptspot+PPPoE for some users. We have public/25 in IP>Address>Interface=LAN (switch which is used for distribution) for src NAT for both of these users.
Then we have another public/25 which is paid static IP again for both PPPoE users and Hotspot. The moment we disabled it from IP>Address, the Hotspot users with paid static IPs couldn’t use the internet anymore, but the PPPoE users could just fine.
Nice to learn at this stage that there are actually two routers, so the whole exercise with the hairpin IPIP tunnel could probably be omitted as the access router could as well send the packets to your own egde router which would happily send them back. I’ve proposed the IPIP tunnel in order that it didn’t depend on what the upstream ISP would do with these packets. But OK, the hairpin IPIP tunnel saves some bandwidth on the cable, curious what traffic statistics will you see on it
Please update the drawing with the position of the speedtest server and the linking of the two /25 subnets to the interfaces of the routers, and put it as an attachment here rather to linking to an external site. There’s the Attachments tab still below the [Save draft] [Previev] [Submit] buttons.
As I’ve understood, the hotspot functionality is running at the edge router, but the src-nat rules are already at the access routess, is that correct?
At which router have you “disabled it (the other public /25 I assume) from IP>Address”, at the edge one or at the access one?
Are the configurations of the two “access routers” identical (except particular subnets) or they differ?
The speedtest server is a problem for every type of user (PPPoE with public IP, hotspot with public IP, CGNAT one) or only for some?
I’m afraid that with this level of complexity, I’ll need configuration exports from all three machines, obfuscated in a way that the relationship between subnets and routes is not lost, and with an indication of the configuration row whose change disables internet access for the hotspot users.
How does the core router know to send traffic for 6.0.0.0/25 and 6.0.0.128/25 to the access routers? BGP is disabled at access router 1 and not configured at all at access router 2, and there are no static routes to 6.0.0.x/y at the core router.
as the hotspot handling is done at the access routers, not at the core as I’ve understood initially, I’m of little use here, because I never had a reason to understand the hotspot magic and thus I don’t know what firewall rules the hotspot functionality dynamically creates. So you’ll have to use /tool sniffer at both the access router and at the core router to find out what happens when the hotspot client is trying to access a public IP outside your network and what happens when it is trying to access the internal speedtest server.
