So, I have an interesting setup that I want to try out. A bit of a challenge. Just thought I'd see what people thought, and if there is a problem with my logic here. This is actually a simplified version of what I'm trying to accomplish, but illustrates the issue.

Assume these are all public IPs, no masquerade needed / wanted.

Internet ----->(port8) Router ----> port1 (1.1.1.0/24) (Port 80,8080, 10000-11000)
|----> port2 (1.1.1.0/24) (All other ports)

(just to be clear, port1,port2, and port8 are all ports on the same Router. So the router has three connections, one to the internet, one to each of 2 seperate LANs. Those two LANs have no connections to each other.).

Basically the router is needing to split traffic based on port. But what is complicated is both places it is splitting the traffic to have the same IP addresses.

Here are some examples.

Connected to port1 is a web server. Connected to port2 is a SQL server.
Both machines have the same IP address, 1.1.1.20 They can not see each other, which is fine and how we want it. They both only think they are the only machine with the IP address of 1.1.1.20

When a request from the internet resolves to 1.1.1.20:80, it should get routed to port1. And the response from port1 back to the machine that made the request should be from port1 out to the internet.

When a request from the internet resolves to 1.1.1.20:112 (or any other port not routed to port1), it should get routed to port2. And the response from port2 back to the machine that made the request should be from port2 out to the internet.

If machine 1.1.1.20 (port1) makes a request out to some site on the internet, the traffic should get routed through port1. Likewise if machine 1.1.1.20 (port2) makes a request out to the internet, the traffic should get routed through port2.

So here is the general thoughts I have for rules / how to achieve this (not using actual mikrotik commands, but just what I'm doing):

Use 3 routing tables (probably redundant, could be just 2, but whatever).
main
port1Table
port2Table

Routes:
main
0.0.0.0/0 ---> port8
1.1.1.0/24 ---> port2 (because most traffic goes here)

port1Table
0.0.0.0/0 ---> port8
1.1.1.0/24 ---> port1

port2Table
0.0.0.0/0 ---> port8
1.1.1.0/24 ---> port2

Create AddressList LANIPs=1.1.1.0/24

Mangle Rules:
prerouting -> in interface=port1 Set-connection-mark=p1 passthrough=yes(?)
prerouting -> in interface=port2 Set-connection-mark=p2 passthrough=yes(?)
prerouting -> has-connection-mark=p1 Set-routing=port1Table passthrough=yes(?)
prerouting -> has-connection-mark=p2 Set-routing=port2Table passthrough=yes(?)

prerouting -> in interface=port8 tcp=80,8080,10000-11000 dst-addresslist=LANIPs Set-connection-mark=p1 passthrough=no(?)
prerouting -> in interface=port8 Set-connection-mark=p2 dst-addresslist=LANIPs passthrough=no(?) <--- To catch all other connections

Doesn't seem like it should be that complicated.

The issue is that the connection-marks don't seem to be really doing anything.

The computer 1 connected via port 1 will make a request to the internet (say 8.8.8.8:53). The computer on the internet responds trying to connect to the epherical port given by computer1 (like 59234 or something weird). I can see the response
Src: 8.8.8.8:53 Dst:1.1.1.20:59234
being routed to port2Table. Even though the request was started from port1

Any ideas / help would be appreciated.

You're making it much harder than it needs to be. Now I submit to you a dilemma about your proposed configuration:
What happens to ICMP - how would you be able to, say, ping both servers to make sure they were available on the network?

I suspect that your router may not accept that the same IP address can be in the ARP cache on two different interfaces.

Going to great leaps and bounds to use the public IP directly on each server isn't worth the effort. Just use NAT and map the external ports to whatever internal IP you want them to go to, and allow for hairpin NAT if you need to reach the public IP from the private LAN side.

If you're just doing this for the sake of mad science - if you fancy yourself a Dr. Moreau of the packets - then one thing I see wrong is that you should have pass-thorugh=no on your mark-routing rules, and move the inbound connection-marking rules for the various ports above the route-marking rules. (leave the port-based connection marking rules set to pass-through=yes). I still think the ARP thing is probably going to be your biggest problem.

I suspect there's a goal behind this design - perhaps it can be achieved in a more straightforward way that doesn't break all kinds of rules of networking.

There is actually a point to this. The reason I can't NAT it is that each port is not a /24. It is more like a /15 of public IPs. I can't really NAT using 100,000 source IPs..

And this is actually live and in production. I have a custom project where a company is basically building a subscription service to blacklist hackers / ip scanners. So instead of a firewall that just drops the connections, it forwards them to another server that also has those IP addresses assigned to it, so it can test the requests / log them / do whatever needs to be done, some of which is making requests out to the internet back to the machine that made the request.

But I actually do like mad science as well. So I do want to find out why this isn't working. I'll look over that suggestion, and see if it makes a difference.

Thanks!

Well, you could do a selective NAT based on the blacklist - or simply have a policy route to next-hop=IP of filter box/sandbox which redirects all destination IPs to self and then replies on whatever service is desired with ye olde sandbox.....

Or you could put a router in front of the sandbox and do dstnat on the sandbox router. So the real WAN router just has a policy that does a route-mark if the source IP of a packet coming in port8 is in the blacklist, then forward to the sandbox router IP as the "default GW" - only route-mark packets coming in port 8 - this way, the standard default GW still works on the replies.

Well, you could do a selective NAT based on the blacklist - or simply have a policy route to next-hop=IP of filter box/sandbox which redirects all destination IPs to self and then replies on whatever service is desired with ye olde sandbox.....

Or you could put a router in front of the sandbox and do dstnat on the sandbox router. So the real WAN router just has a policy that does a route-mark if the source IP of a packet coming in port8 is in the blacklist, then forward to the sandbox router IP as the "default GW" - only route-mark packets coming in port 8 - this way, the standard default GW still works on the replies.

So there is no blacklist currently. All internet addresses need to have this behavior. It will build a blacklist over time. So all traffic needs to be split this way. So I think that disables the selective NAT idea, unless I am misunderstanding it.

Maybe I should clarify one more point, as well.

The computer 1 connected via port 1 will make a request to the internet (say 8.8.8.8:53). The computer on the internet responds trying to connect to the epherical port given by computer1 (like 59234 or something weird). I can see the response
Src: 8.8.8.8:53 Dst:1.1.1.20:59234
being routed to port2Table. Even though the request was started from port1

It is VERY important that 1.1.1.20 and 8.8.8.8 are communicating. There will be one machine on 1.1.1.20, one on 1.1.1.19, etc. if 8.8.8.8 makes a request to 1.1.1.20:80, then the machine at 1.1.1.20 has to answer it. And the response going back needs to be from 1.1.1.20. So when 8.8.8.8 makes a call to 1.1.1.20:80 and 1.1.1.19:80, I can't have masquerade going on causing 1.1.1.20 and 1.1.1.19's IP addresses to be masqueraded (hidden and combined) to like 1.1.1.1 or whatever the ip address on the router's port1 is.

I'm not 100% sure what you mean about the next-hop=ip of filter. There are two different next-hops, depending on which port is being requested.

using the third option, of another router, with dstnat, I think that wouldn't allow all the outgoing requests from port1 to use their real IP addresses, it would all be masqueraded via the additional router, which would be an issue as mentioned above.

I'm not a routing expert. But I love to solve problems. I'll look more into your suggestions and see how the router behaves. It's currently processing over 2k packets / second, so there is a lot of data to look through. But it also means I can see how changes behave pretty quickly.

Thanks for your help. If you felt like offering some specific mikrotik config ideas to help I'd definitely look over them as well.

Oh, the client has to have the public IPs assigned to his servers. It reads the server configurations and makes choices based off of what IP address the server has. I just read up on hairpin NAT, pretty cool, I hadn't seen that term before, but both of those are unfortunately not an option in this case.

Ok, I moved the rules as you specified. I didn't know that it would make a difference. But it does appear that more traffic is going out the correct interfaces. By tomorrow I should know for sure if those changes did it.

Just to follow up.

I was able to successfully set this up.

There are still a little bit of an issue with multi-homed. But works fine for one input to the outside world and 2 ports splitting the traffic out to.

The key things were:
1) Had to make sure the Mark Connection rules had Connection-Mark: no-mark set.
2) Had to make sure all the rules had Passthrough disabled.
3) Put in a couple of Mark-Connection rules in the forward chain for Out interfaces matching up, to keep them in the correct routing tables. Might be some additional overkill, but I wanted to make sure the routing was going exactly where I wanted it to.

As a side note I also set up some ICMP rules to force pings to go to one or the other set of machines, depending on what I felt like at the time.

Thanks again for your help!

I have to admit this is one of the stranger things I've seen go by on here in a while.
Just be sure the rest of your crew knows how to deal with this strange configuration.

Yeah, I'm a one man ISP. So I'm the only one that deals with the routers. So it's all good. Thanks again for the in sites.

Would you be kind to illustrate network map with routers connected and their subnets please, it looks really interesting that i would like to try in a lab.


Thanks