Dual wan question routing only no loadbalancing

RouterOS General
1

Please bare with me this takes a while to explain, and thank you for your valuable time looking at this.

This is what I have that works right now:

1 cisco router with a T3 internet connection. We’ll call it the 200 router.

The 200 router has public IP’s 200.200.x.0/24, x.96.56.0/24, x.97.248.0/24, and x.97.129.0 routed to it by my ISP.

In the 200 routers routing table I have these routes:


x.96.56.0/30 to 200.200.X.69
x.96.56.128/25 to 200.200.x.69
x.97.129.0/24 to 200.200.x.69


The 200 router has an IP address of 200.200.X.3 on it’s ethernet port. 200.200.X.1 and .2 are caching DNS servers.

It is plugged into a switch along with the DNS servers. We’ll call it the 200 switch.

Also plugged into the 200 switch is a dual zeon Mikrotik router with two ethernet interfaces. We’ll call it the Mikrotik router

The ethernet port 1 on the Mikrotik router connected to the 200 switch has 200.200.X.69 assigned to it. It also has proxy arp activated on this interface

The ethernet port 2 has x.97.248.1/24, x.96.56.1/30, and 172.16.20.1/24 assigned to it.

On the Mikrotik router ethernet port 2 is plugged into a wireless layer 2 bridge out to various towers.

On these towers I have Mikrotik access points with pppoe servers on them assigning the 172 subnet and the x.97.248.0 subnet.

Since the 200.200.x.69 interface has proxy arp turned on and is plugged into the same switch as the 200.200.x.3 interface of the 200 router I can assign private IPs where I need them as well as the public series on any tower.

This also means that for the private 172 subnet I am masquerading out the 200.200.x.69 interface.

This also means I have a default route 0.0.0.0 to 200.200.x.3 to get to the internet on the 200 router right now.


On the other side of a long wireless P2P there rests a cisco router with x.96.56.2/30 on the ethernet interface on the opposite site being connected (through the layer 2 P2P) to the x.96.56.1/30 interface of the Mikrotik router. We’ll call it the CiscoP2P router

Inside the Mikrotik’s routing table I have these routes:


x.96.56.128/25 to x.96.56.2/30
x.97.129.0/24 to x.96.56.2/30


This is so the CiscoP2P router can dhcp these addresses out on a further network.


All of this works fine.


Now we are getting a second T3 with another cisco router with it’s own set of IPs routed only to it. This new routed by the ISP subnet will be x.99.81.0/22. We’ll call it the 99 router.

The 99 router will have x.99.81.1/22 assigned to it’s ethernet port.

This ethernet port will be connected to the 200 switch along with the 200 router, dns servers, and the Mikrotik router.

I need to route these IPs from the new 99 router through the mikrotik router to the CiscoP2P router with x.96.56.2/30 as it’s IP.


How should I do this?



My first impression was that since the x.99.81.1/22 interface of the 99 router would be in the same switch as the 200.200.x.69 interface of the Mikrotik then I could assign x.99.81.2/22 to the 200 interface of the Mikrotik then enter
this route in the Mikrotik:

x.99.81.0/22 to x.96.56.2


This should send it to the CiscoP2P router right?

But since I have the default route set to 0.0.0.0 to 200.200.x.3 in the Mikrotik maybe this will not work?


I though I might have to setup mangle based routing for the x.99.81.0/22 subnet, but I’m only using two interfaces on the Mikrotik and one of them is physically connected (by a switch) to the 99 routers x.99.81.1/22 ethernet port so I

though I woudln’t have to do this.

The new T3 isn’t installed yet, but I’ve installed the 99 router and setup the Mikrotik just as I outlined above and I can ping both the 200 router and the 99 router from the Mikrotik.

Does this mean that the routing is working for the x.99.81.0/22 subnet?

2

Don’t use proxy ARP for the new networks. You may be stuck with it for the existing networks but that was a mistake to begin with.

To get the new router into production remove x.99.81.1/22 from its LAN interface and assign 10.1.0.1/30 instead. Assign 10.1.0.2/30 on the Mikrotik router interface that connects to the new Cisco router.

On the Cisco router add a static route for x.99.81.0/24 via 10.1.0.2/30:

ip route x.99.81.0 255.255.255.0 10.1.0.2

Now the Cisco router knows to route traffic for that network via the Mikrotik router.
On the Mikrotik router, add a route for the same network via the Cisco P2P router:

/ip route add dst-address=x.99.81.0/24 gateway=x.96.56.2

Now the Mikrotik router knows to route traffic for that network via the Cisco P2P router.

So now the new Cisco 99 and Mikrotik routers know how to get to the network behind the Cisco P2P router. Great. That router will know how to send traffic back because of the default routes you have in place. Now you need to add policy routing to make sure that traffic sourced from x.99.81.0/24 goes back via the new Cisco router, but only for networks that aren’t somewhere else connected to the Mikrotik router. First make an address list of local networks:

/ip firewall address-list
add list=local-networks address=x.97.248.0/24
add list=local-networks address=x.96.56.0/24
add list=local-networks address=200.200.x.0/24
add list=local-networks address=x.97.129.0
add list=local-networks address=172.16.20.0/24

Then add mangle rules that set a routing mark for packets entering via the interface that faces the Cisco P2P router - I’m assuming it’s called ether2. All traffic entering via that interface that doesn’t go to local networks gets a routing mark of ‘99’. This could be optimized if the router is taxed too hard by this.

/ip firewall mangle 
add chain=prerouting in-interface=ether2 src-address=x.99.81.0/24 dst-address-list=!local-networks action=mark-routing new-routing-mark=99

Then add a route for that mark back out via the /30 to the new Cisco router:

/ip route 
add routing-mark=99 dst-address=0.0.0.0/0 gateway=10.1.0.1

There are still many more things wrong with this network that could be improved to make it less messy.

You should, if you can, subnet the existing public IPs and stop using proxy ARP altogether. Use /30 links between the routers, and properly route networks. This may be difficult given that you have DNS servers on those IPs, but it is possible.

You should also stop overlaying multiple IP addresses on different subnets on the same physical broadcast domains (stop using multiple networks on one interface). Either add interfaces to routers, or get VLAN capable switches and use VLANs and VLAN interfaces.

Ideally you’d also stop using static routes and would run a routing protocol like OSPF between all routers. There are only four routers, so it wouldn’t be that hard to do.

You may want to contact a consultant and get some bids on helping you clean up your network.

3

To fewi:

Wow! Thank you so much for your response. :smiley:

Much Karma your way.

It makes complete sense to me.

I think I’ll have to set the mangle rule to only match the ..99.81.0/22 subnet instead of not matching all the others because the 172

subnet is already in one address list I have for the webproxy cache running on the Mikrotik and the .97.248.0/24 subnet is not in an address list so it bypasses the webproxy cache.

This should still accomplish the same thing correct?


Man, thank you very much for the advice on fixing up this beast.

I’m slowly in the process of doing exactly that with the Vlan’s you described.

I would have started that way but this network is actually running along side an existing crappy and not designed by me totally bridged network.

So when I inherited it from a lazy person that had to find employment elsewhere :slight_smile: I had to make this new network still work inside the old bridged one (legacy customers) hence not really any routing to and from the towers.

I also could not get enough IPs from our service provider to do what I wanted hence the proxy arp for public IPs. Normally the customer gets a private unless they need a public then it gets assigned through pppoe.

This network is also wwaayy bigger than what I have typed up here. I only put the parts pertinent to my question because it would’ve been way to long and no one would read it. lol.

I do have another question though:
You have the mangle rule for the routing mark in the prerouting chain. I already have the prerouting chain working for my QOS priority setup so I’m assuming this rule still works exactly the same in the forward chain?

Thank you again for your valuable time and advice with this

4

You would the local-networks address list to contain all networks within your autonomous system - even if those networks are already listed in other address lists. Multiple address lists can refer to the same networks without problems. The idea behind that address list is that source based policy routing is going to take precedence over other routes. So the traffic for local networks would still get sent to the WAN router just because the source is in that network you’re trying to source route - even though there’s a better route. Worst case this could lead to routing loops - where the Mikrotik router sends a packet to the new Cisco 99 router, and that Cisco router sends the packet back to the Mikrotik because of a route it has in place. Even worse, it might send it out to the second ISP, which sends it to the first ISP, which sends it to you, back it goes to the Mikrotik router, out to the Cisco 99 router, and so on.
Usually default routes work great because if you have more specific routes, they just match due to a longer prefix match - so the default route isn’t used. With routing marks you’re saying “whatever else you think is right, ignore that - use the route I’m telling you to use”. Therefore you have to make sure that you don’t tell the router to make the wrong routing decisions. In this case the simplest workaround is to have an address list with ALL local routes a packet could be going to, and to only set a routing mark if a packet isn’t going to any destination on that address list.
At least I think that’s how the router would behave. Try it out without the address list, if it doesn’t work, add it in.

You would want the routing marks to be set before a routing decision is made - so you do want to do that in prerouting. The simplest approach would be to add the routing mark rule to the very top of the rule set, and add passthrough=yes. That way the routing mark gets set and then everything else gets processed just as it normally would, and the rule would not interfere with whatever processing you’re already doing in prerouting.

5

To fewi

Again, you are right. Makes perfect sense to me now.

You are a true asset to these forums.

I didn’t realize that IPs can be in more than one address list at the same time. I will definitely take this approach

Yes! I completely forgot about passthrough=yes. That will be exactly what I need in this situation.

Thanks, again your help has been great.

6

Hey FEWI.

I don’t know if you will see this or not but I’ve got the connection up now and I’ve tried the routes you gave me here and I can’t get it to work.

When I’m trying traceroutes out on the net to an IP on the X.99.81.0 subnet it gets to the 200 router then fails and I can’t figure out why.

If there is anything you might be able to help with I would appreciate it.

Thanks,
Allen