Routing issues with multiple ISPs

taduikis · Thu Sep 16, 2010 2:05 am

Hi to all,

Recently I've moved my main router from highly configured Linux system to RouterOS. I've managed to achieve almost everything I need, but few things remain unsolved.
I currently have access to two ISPs through two different interfaces in my system. On one of them public IP addresses are assigned manually (there are 3 of them) and another interface gets its IP from an ISPs' DHCP server. I was about to explain what do I do with these public IPs, but it might not be necessary at the moment.. Simply speaking my intention is to shape traffic and let it out through these different ISPs according to my rules.

My current routing table looks like this:

Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 
 0 A S  dst-address=0.0.0.0/0 gateway=88.88.66.1 gateway-status=88.88.66.1 reachable ether2 distance=1 scope=30 target-scope=10 routing-mark=1 

 1 A S  dst-address=88.88.66.0/24 pref-src=88.88.66.70 gateway=ether2 gateway-status=ether2 reachable distance=1 scope=30 target-scope=10 routing-mark=1 

 2 A S  dst-address=0.0.0.0/0 gateway=88.88.66.1 gateway-status=88.88.66.1 reachable ether2 distance=1 scope=30 target-scope=10 routing-mark=2 

 3 A S  dst-address=88.88.66.0/24 pref-src=88.88.66.82 gateway=ether2 gateway-status=ether2 reachable distance=1 scope=30 target-scope=10 routing-mark=2 

 4 A S  dst-address=0.0.0.0/0 gateway=77.77.253.1 gateway-status=77.77.253.1 reachable ether2 distance=1 scope=30 target-scope=10 routing-mark=3 

 5 A S  dst-address=77.77.253.0/24 pref-src=77.77.253.154 gateway=ether2 gateway-status=ether2 reachable distance=1 scope=30 target-scope=10 routing-mark=3 

 6 ADS  dst-address=0.0.0.0/0 gateway=66.66.79.254 gateway-status=66.66.79.254 reachable ether3 distance=2 scope=30 target-scope=10 

 7 ADC  dst-address=88.88.66.0/24 pref-src=88.88.66.82 gateway=ether2 gateway-status=ether2 reachable distance=0 scope=10 

 8 ADC  dst-address=66.66.64.0/20 pref-src=66.66.68.252 gateway=ether3 gateway-status=ether3 reachable distance=0 scope=10 

 9 ADC  dst-address=77.77.253.0/24 pref-src=77.77.253.154 gateway=ether2 gateway-status=ether2 reachable distance=0 scope=10 

10 ADC  dst-address=172.10.10.0/29 pref-src=172.10.10.1 gateway=ether1 gateway-status=ether1 reachable distance=0 scope=10 

11 ADC  dst-address=192.168.0.0/21 pref-src=192.168.0.1 gateway=ether1 gateway-status=ether1 reachable distance=0 scope=10 

12 ADC  dst-address=192.168.100.0/24 pref-src=192.168.100.1 gateway=ether1 gateway-status=ether1 reachable distance=0 scope=10

And routing rule table looks like this:

 0   src-address=88.88.66.70/32 routing-mark=1 interface=ether2 action=lookup table=1 
 1   src-address=88.88.66.82/32 routing-mark=2 interface=ether2 action=lookup table=2 
 2   src-address=77.77.253.154/32 routing-mark=3 interface=ether2 action=lookup table=3

The shaping and masquerading part works fine. The problem is that no IP of ether2 can send back traffic to it's source. I mean that if I ping the public address of ether2 from the outside, it doesn't reply, while interface torch does see incoming ICMP requests on that interface and it also shows correct dst-addr. This of course applies to any other incoming traffic. I think I'm missing something obvious here, because in the Linux based router setup I've added some rules or routes to localhost. Unfortunately I wasn't able to reproduce this on RouterOS, and I don't have that Linux iproute config anymore.

This might require additional explaining, so I'm willing to provide any details necessary..

Feklar · Thu Sep 16, 2010 11:04 pm

Use use these rules in the mangle section of the firewall.

/ip firewall mangle
add action=mark-connection chain=input comment="Reply to pings going from ISP1 side with ISP1 connection" disabled=no in-interface=ether1 new-connection-mark=\
    input1_connection passthrough=yes
add action=mark-connection chain=input comment="Reply to pings going from ISP2 side with ISP2 connection" disabled=no in-interface=ether2 new-connection-mark=\
    input2_connection passthrough=yes
add action=mark-routing chain=output comment="" connection-mark=input1_connection disabled=no new-routing-mark=to_outside1 passthrough=no
add action=mark-routing chain=output comment="" connection-mark=input2_connection disabled=no new-routing-mark=to_outside2 passthrough=no

Use these rules if you want to do port forwarding and want to access those forwards from each interface.

add action=mark-connection chain=forward comment="Port Forwarding Routing" connection-state=new disabled=no in-interface=ether1 new-connection-mark=outside1_connection \
    passthrough=no
add action=mark-connection chain=forward comment="" connection-state=new disabled=no in-interface=ether2 new-connection-mark=outside2_connection passthrough=no
add action=mark-routing chain=prerouting comment="" connection-mark=outside1_connection disabled=no new-routing-mark=to_outside1 passthrough=no
add action=mark-routing chain=prerouting comment="" connection-mark=outside2_connection disabled=no new-routing-mark=to_outside2 passthrough=no

Obviously modify it to fit the number of interfaces, and your routing and connections marks that you want to use.

taduikis · Thu Sep 16, 2010 11:53 pm

So I have to solve my problem using the mangle table?
I didn't set anything on mangle table in my original Linux setup apart from rules that specify which public IP address internal subnets get for certain ports. These entries are successfully set on RouterOS and do work as they should.
Can't I just do something with routing rules, tables and 127.0.0.0?
And not only ping i would like to work, but things like winbox, www service also.

I have found my previous iproute setup in linux, it was as follows:

ifconfig eth0 88.88.66.70 netmask 255.255.255.0 up
ip addr add 88.88.66.82/24 brd 88.88.66.255 dev eth0

ip route add 88.88.66.0/24 dev eth0 src 88.88.66.70 table 1
ip route add default via 88.88.66.1 table 1

ip route add 88.88.66.0/24 dev eth0 src 88.88.66.82 table 3
ip route add default via 88.88.66.1 table 3

ip route add 88.88.66.0/24 dev eth0 src 88.88.66.70
ip route add 88.88.66.0/24 dev eth0 src 88.88.66.82

ip rule add from 88.88.66.70 table 1
ip rule add from 88.88.66.82 table 3

ip route add 88.88.66.0/24 dev eth0 table 1
ip route add 88.88.66.0/24 dev eth0 table 3

ip route add 192.168.0.0/20 dev eth1 table 1
ip route add 192.168.0.0/20 dev eth1 table 3
ip route add 127.0.0.0/8 dev lo table 1
ip route add 127.0.0.0/8 dev lo table 3

ip route add default dev eth2

ip rule add fwmark 1 table 1
ip rule add fwmark 3 table 3

And iptables was:

port_list1="53,80,110,113,443,2106,3724,6667,27015,27016,27017,43594,43595"
port_list2="6112,7777,5555,27900,29900,29910,27901,55123,55124,55215,16567,29901,29920,28910,34562"

iptables -t mangle -A PREROUTING -s 192.168.0.0/24 -p tcp -m multiport --dports $port_list1 -d 0/0 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -s 192.168.0.0/24 -p udp -m multiport --dports $port_list1 -d 0/0 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -s 192.168.0.0/24 -p tcp -m multiport --dports $port_list2 -d 0/0 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -s 192.168.0.0/24 -p udp -m multiport --dports $port_list2 -d 0/0 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -s 192.168.2.0/24 -p tcp -m multiport --dports $port_list1 -d 0/0 -j MARK --set-mark 3
iptables -t mangle -A PREROUTING -s 192.168.2.0/24 -p udp -m multiport --dports $port_list1 -d 0/0 -j MARK --set-mark 3
iptables -t mangle -A PREROUTING -s 192.168.3.0/24 -p tcp -m multiport --dports $port_list1 -d 0/0 -j MARK --set-mark 3
iptables -t mangle -A PREROUTING -s 192.168.3.0/24 -p udp -m multiport --dports $port_list1 -d 0/0 -j MARK --set-mark 3
iptables -t mangle -A PREROUTING -s 192.168.4.0/24 -p tcp -m multiport --dports $port_list1 -d 0/0 -j MARK --set-mark 3
iptables -t mangle -A PREROUTING -s 192.168.4.0/24 -p udp -m multiport --dports $port_list1 -d 0/0 -j MARK --set-mark 3
iptables -t mangle -A PREROUTING -s 192.168.5.0/24 -p tcp -m multiport --dports $port_list1 -d 0/0 -j MARK --set-mark 3
iptables -t mangle -A PREROUTING -s 192.168.5.0/24 -p udp -m multiport --dports $port_list1 -d 0/0 -j MARK --set-mark 3

iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -o eth0 -j SNAT --to-source 88.88.66.70
iptables -t nat -A POSTROUTING -s 192.168.0.0/21 -o eth0 -j SNAT --to-source 88.88.66.82

iptables -t nat -A POSTROUTING -s 192.168.0.0/21 -o eth2 -j MASQUERADE

eth0 - ISP1 (2 manually assigned public IP's)
eth1 - Local network (192.168.0.0/21)
eth2 - ISP2 (DHCP assigned dynamic IP)

Such setup worked perfectly, I was able to ping my router from outside, connect to it through SSH on any interface or public IP, could set port forwards however I wanted. Unfortunately I'm not able to reproduce similar results in RouterOS

Feklar · Fri Sep 17, 2010 12:27 am

You will also need to set up routes with the appropriate routing mark for it to work. I'm not sure how Linux handles those kinds of things internally, it could be that they already have that logic built into it on a higher level.

What the mangle rules are doing is saying any connections that come in on this interface, mark it so I know what of my routes to respond back with. What is currently happening according to your post is the connections come in, and then the MikroTik goes to respond, but it's using the main routing table, so it's choosing the route with the lowest weight to respond on. This works fine if you are using that interface to connect on, if you're using the secondary one, then it's responding to you form somewhere you won't recognize. These rules should fix that for you.

taduikis · Fri Sep 17, 2010 12:27 am

I've noticed that I cannot do anything with ISP1 (the one with 3 public IPs) from inside the router. If I try to ping something while using my public IP on ether2 as src-addr in ping settings, it doesn't get a reply. If I set interface to ether2 - it works. This definitely has something to do with routing tables.

And what me concerns at the moment, is how to do this on RouterOS:

ip route add 127.0.0.0/8 dev lo table 1
ip route add 127.0.0.0/8 dev lo table 3

Feklar · Fri Sep 17, 2010 12:35 am

Please post the results of these commands. We can probably help you form there. It seems to me you are bringing in how you think of setting up a Linux box and trying to apply it to RouterOS. While a lot of things are the same, with RouterOS being based on the Linux kernel, they are closed source and have their own way of doing some of these things.

/ip firewall export
/ip address export
/ip route export

taduikis · Fri Sep 17, 2010 1:14 am

Well, your proposed way does in fact work almost as expected. I could leave it that way, but who knows, maybe something better can be done

I am aware that in RouterOS some things are done differently and it's worth to say more conveniently in most cases. But with this particular case I don't see that convenience I had before (not a biggie if there are some other ways of course).

Previously I could set port forwards as easy as this:

iptables -t nat -A PREROUTING -p tcp -i eth0 -d 88.88.66.70 --dport 8291 -j DNAT --to 192.168.1.2:8291
iptables -t nat -A PREROUTING -p tcp -i eth2 --dport 33555 -j DNAT --to 192.168.0.21:33555

As you can see no marking was needed. Port 8291 on ISP1 would forward to 1.2, port 33555 on ISP2 would be forwarded to 0.21. Just simple as that.

And I agree with you about not knowing how everything is internally handled in these operating systems.. That was the reason why I posted here with a request for help to get a different scheme of achieving the same thing.

Besides I'm still blind from inside the routeros when trying to escape with any of ISP1 IPs. For example:

> ping 8.8.8.8 src-address=88.88.66.70
8.8.8.8 ping timeout

While in my Linux router, I could simply type "ping 8.8.8.8 -I 88.88.66.70" and get a reply. 8.8.8.8 is Google's public DNS by the way.

Here are the data you requested and some more.

Address table:

/ip address
add address=77.77.253.154/24 broadcast=77.77.253.255 comment="" disabled=no \
    interface=ether2 network=77.77.253.0
add address=192.168.0.1/21 broadcast=192.168.7.255 comment="" disabled=no \
    interface=ether1 network=192.168.0.0
add address=172.10.10.1/29 broadcast=172.10.10.7 comment="" disabled=no \
    interface=ether1 network=172.10.10.0
add address=192.168.100.1/24 broadcast=192.168.100.255 comment="" disabled=no \
    interface=ether1 network=192.168.100.0
add address=88.88.66.82/24 broadcast=88.88.66.255 comment="" disabled=no \
    interface=ether2 network=88.88.66.0
add address=88.88.66.70/24 broadcast=88.88.66.255 comment="" disabled=no \
    interface=ether2 network=88.88.66.0

Routing table:

/ip route
add comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    88.88.66.1 routing-mark=1 scope=30 target-scope=10
add comment="" disabled=no distance=1 dst-address=88.88.66.0/24 gateway=\
    ether2 pref-src=88.88.66.70 routing-mark=1 scope=30 target-scope=10
add comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    88.88.66.1 routing-mark=2 scope=30 target-scope=10
add comment="" disabled=no distance=1 dst-address=88.88.66.0/24 gateway=\
    ether2 pref-src=88.88.66.82 routing-mark=2 scope=30 target-scope=10
add comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    77.77.253.1 routing-mark=3 scope=30 target-scope=10
add comment="" disabled=no distance=1 dst-address=77.77.253.0/24 gateway=\
    ether2 pref-src=77.77.253.154 routing-mark=3 scope=30 target-scope=10
/ip route rule
add action=lookup comment="" disabled=no interface=ether2 routing-mark=1 \
    src-address=88.88.66.70/32 table=1
add action=lookup comment="" disabled=no interface=ether2 routing-mark=2 \
    src-address=88.88.66.82/32 table=2
add action=lookup comment="" disabled=no interface=ether2 routing-mark=3 \
    src-address=77.77.253.154/32 table=3

Firewall filtering rules:

/ip firewall filter
add action=accept chain=forward comment="Accept forward for client pool" \
    disabled=no dst-address=0.0.0.0/0 in-interface=ether1 src-address=\
    192.168.0.0/21
add action=accept chain=input comment=\
    "Accept established inbound connections" connection-state=\
    established disabled=no in-interface=ether2
add action=accept chain=input comment=\
    "Accept related inbound connections" connection-state=related \
    disabled=no in-interface=ether2
add action=accept chain=input comment=\
    "Accept inbound 8291 port for winbox control" disabled=no \
    dst-address=0.0.0.0/0 dst-port=8291 in-interface=ether2 protocol=tcp \
    src-address=0.0.0.0/0
add action=accept chain=input comment="Accept SSH connections" \
    disabled=no dst-address=0.0.0.0/0 dst-port=22 in-interface=ether2 \
    protocol=tcp src-address=0.0.0.0/0
add action=accept chain=input comment="Accept HTTP connections" \
    disabled=yes dst-address=0.0.0.0/0 dst-port=80 in-interface=ether2 \
    protocol=tcp src-address=0.0.0.0/0
add action=accept chain=input comment=\
    "Accept unflooded ICMP ping requests" disabled=no in-interface=\
    ether2 limit=1,5 protocol=icmp
add action=drop chain=input comment="Drop everything unaccepted inbound" \
    disabled=no dst-address=0.0.0.0/0 in-interface=ether2 src-address=\
    0.0.0.0/0
add action=accept chain=input comment=\
    "Accept established inbound connections" connection-state=\
    established disabled=no in-interface=ether3
add action=accept chain=input comment=\
    "Accept related inbound connections" connection-state=related \
    disabled=no in-interface=ether3
add action=accept chain=input comment=\
    "Accept inbound 8291 port for winbox control" disabled=no \
    dst-address=0.0.0.0/0 dst-port=8291 in-interface=ether3 protocol=tcp \
    src-address=0.0.0.0/0
add action=accept chain=input comment="Accept SSH connections" \
    disabled=no dst-address=0.0.0.0/0 dst-port=22 in-interface=ether3 \
    protocol=tcp src-address=0.0.0.0/0
add action=accept chain=input comment="Accept HTTP connections" \
    disabled=yes dst-address=0.0.0.0/0 dst-port=80 in-interface=ether3 \
    protocol=tcp src-address=0.0.0.0/0
add action=accept chain=input comment=\
    "Accept unflooded ICMP ping requests" disabled=no in-interface=\
    ether3 limit=1,5 protocol=icmp
add action=drop chain=input comment=\
    "Drop everything unaccepted inbound" disabled=no dst-address=\
    0.0.0.0/0 in-interface=ether3 src-address=0.0.0.0/0

Firewall mangle table:

/ip firewall mangle
add action=mark-routing chain=prerouting comment="" disabled=no dst-address=\
    0.0.0.0/0 dst-port=53,80,110,113,443,6667,6668,7000 new-routing-mark=1 \
    passthrough=yes protocol=tcp src-address=192.168.0.0/24
add action=mark-routing chain=prerouting comment="" disabled=no dst-address=\
    0.0.0.0/0 dst-port=53,80,110,113,443,6667,6668,7000 new-routing-mark=1 \
    passthrough=yes protocol=udp src-address=192.168.0.0/24
add action=mark-routing chain=prerouting comment="" disabled=no dst-address=\
    0.0.0.0/0 dst-port=53,80,110,113,443,6667,6668,7000 new-routing-mark=2 \
    passthrough=yes protocol=tcp src-address=192.168.2.0/24
add action=mark-routing chain=prerouting comment="" disabled=no dst-address=\
    0.0.0.0/0 dst-port=53,80,110,113,443,6667,6668,7000 new-routing-mark=2 \
    passthrough=yes protocol=udp src-address=192.168.2.0/24
add action=mark-routing chain=prerouting comment="" disabled=no dst-address=\
    0.0.0.0/0 dst-port=53,80,110,113,443,6667,6668,7000 new-routing-mark=2 \
    passthrough=yes protocol=tcp src-address=192.168.3.0/24
add action=mark-routing chain=prerouting comment="" disabled=no dst-address=\
    0.0.0.0/0 dst-port=53,80,110,113,443,6667,6668,7000 new-routing-mark=2 \
    passthrough=yes protocol=udp src-address=192.168.3.0/24
add action=mark-routing chain=prerouting comment="" disabled=no dst-address=\
    0.0.0.0/0 dst-port=53,80,110,113,443,6667,6668,7000 new-routing-mark=3 \
    passthrough=yes protocol=tcp src-address=192.168.4.0/24
add action=mark-routing chain=prerouting comment="" disabled=no dst-address=\
    0.0.0.0/0 dst-port=53,80,110,113,443,6667,6668,7000 new-routing-mark=3 \
    passthrough=yes protocol=udp src-address=192.168.4.0/24
add action=mark-routing chain=prerouting comment="" disabled=no dst-address=\
    0.0.0.0/0 dst-port=53,80,110,113,443,6667,6668,7000 new-routing-mark=3 \
    passthrough=yes protocol=tcp src-address=192.168.5.0/24
add action=mark-routing chain=prerouting comment="" disabled=no dst-address=\
    0.0.0.0/0 dst-port=53,80,110,113,443,6667,6668,7000 new-routing-mark=3 \
    passthrough=yes protocol=udp src-address=192.168.5.0/24

Firewall NAT table:

/ip firewall nat
add action=src-nat chain=srcnat comment=\
    "IP 88.88.66.70 for subnet 192.168.0.0/24" disabled=no dst-address=\
    0.0.0.0/0 out-interface=ether2 src-address=192.168.0.0/24 to-addresses=\
    88.88.66.70
add action=src-nat chain=srcnat comment=\
    "IP 88.88.66.82 for subnet 192.168.2.0/24" disabled=no dst-address=\
    0.0.0.0/0 out-interface=ether2 src-address=192.168.2.0/24 to-addresses=\
    88.88.66.82
add action=src-nat chain=srcnat comment=\
    "IP 88.88.66.82 for subnet 192.168.3.0/24" disabled=no dst-address=\
    0.0.0.0/0 out-interface=ether2 src-address=192.168.3.0/24 to-addresses=\
    88.88.66.82
add action=src-nat chain=srcnat comment=\
    "IP 77.77.253.154 for subnet 192.168.4.0/24" disabled=no dst-address=\
    0.0.0.0/0 out-interface=ether2 src-address=192.168.4.0/24 to-addresses=\
    77.77.253.154
add action=src-nat chain=srcnat comment=\
    "IP 77.77.253.154 for subnet 192.168.5.0/24" disabled=no dst-address=\
    0.0.0.0/0 out-interface=ether2 src-address=192.168.5.0/24 to-addresses=\
    77.77.253.154

Feklar · Fri Sep 17, 2010 7:50 pm

1.) So you have all of your public connections running to one interface correct? Is there any reason why you have it set up like this, it should still work but it might make things a little messy. Are you running this on an x86?
2.) You don't have anything in the main routing table, while you don't need anything as long as all your packets get routing marked, if any aren't marked they have no way to get out to the internet, and if one of your other routes should fail it will have nothing to fall back on for internet access either. You basically specify the same routes without any routing marks and weight them appropriately. This will also affect your 172.10.10.0/29 subnet, I don't see a NAT rule for that subnet, and also if you want to route traffic out from that subnet it will have no routing table to use.
3.) Try running the ping commands with specifying the routing table it should use to send the pings out of. This will tell you if the router is able to send traffic out successfully out of each of the routes or not.

ping 8.8.8.8 interval=1 routing-table=1 count=6
ping 8.8.8.8 interval=1 routing-table=2 count=6
ping 8.8.8.8 interval=1 routing-table=3 count=6

4.) For the rules I gave you earlier you are going to have to specify a dst-address instead of the in-interface with your current set up. Otherwise it will catch too many connections for it to work right.
5.) Instead of the action=src-nat in your NAT rules, you can instead try using action=masquerade. Both should work fine, just masquerade seems to work better for me in most situations. You are already marking stuff for routing, so the router will still know what route to use to send packets out of.

taduikis · Fri Sep 17, 2010 8:47 pm

Let's go in your defined order here:
1) Yes and no. ether2 has ISP1 connected to it, 3 manually configured IPs are bound to it (you can see that in addr list above). ISP2 is connected to ether3, it gets its IP via DHCP from ISP2. There are no other way of configuring them, are there? Currently everything is running on x86, but I'm considering moving it to RB1000/RB1100.
2) Well as I said, I've ported my config from my previous setup in Linux router. I tried to replicate the same setup here, which I almost succeeded to do. I shall explain everything in detail below, this should explain everything and provide answers to your questions. One more thing I should mention, that DHCP client set to listen on ether3 is configured to 'add default route' and default route distance is set to 2.
3) I did run ping with routing-table defined before, just didn't mention here that everything works fine. Replies come in, and by their slightly higher rtt times I can confirm that packets go thru ISP1 (it's a DSL link).
4) Yep. I've figured out that much.
5) I'm not sure, but I think on Linux machine that didn't work correctly. In RouterOS this could be different though.

And as for what I'm trying to do, you probably could have guessed by that mangle table I gave you in previous post. So the story is: I have access to two ISPs, one of them provides me with few static IP addresses (ISP1, connected to ether2), another (ISP2) give IP via DHCP and it's connected to ether3. Since ISP2 connection is alot faster it's used for everything other than DNS, HTTP, HTTPS, IRC, POP3, etc.. So mangling ant NAT tables distribute these ISP1 public addresses for HTTP(...) access to local subnets accordingly and ISP2 gives access to anything other than that. 192.168.100.0/24 is a pool for unidentified clients, addresses of this pool are given to DHCP clients not listed in my static lease table, they are also given a DNS address 192.168.100.1 which points everything to my external web server 172.10.10.2 (NAT rules for it are present, I just deleted them before posting). In addition it redirects every request to that same server. This is needed to provide some directions of what to do when client connect some new device to the network and tries to access any webpage. 172 subnet is required as a different subnet for dst-nat rules to work for a complete redirection of all http traffic, I also use it to inform some people that forgets to pay for services.

I don't claim any copyrights to this setup, but as far as I'm concerned I drew it myself. For that particular reason I'm sure it's not perfect, so any suggestions to do something differently are welcome

Besides, fewi from this forum already changed few things, and made my life easier

Feklar · Fri Sep 17, 2010 10:33 pm

Ok I was just confused as I saw two different subnets defined on Ether2, and didn't see anything on Ether3. I just generally like to try and keep each internet connection on their own interface, hence me asking if it was on an x86, basically to see how many Ethernet ports you had. So now my understanding is that you have two internet connections.

And it looks like you may be close to what you are looking to do. My next question is, is there a reason why you would like to src-nat each subnet out a different IP address?

There are probably going to be a couple of ways to do this, but if I was looking to do a similar setup, I think my configuration would look a little bit more like this. I'm not 100% certain that this would work, but it should at least be close. Having two different subnets from one ISP might make things a bit more complicated however. But having the gateway defined as the interface, and using scr-nat instead of masquerade might take care of that. I've just never had to specify an interface for a gateway before to know how well it works.

/ip firewall mangle
add action=mark-connection chain=prerouting disabled=no dst-port=53,80,110,113,443,6667,6668,7000 new-connection-mark=to_outside1 passthrough=yes protocol=tcp src-address=!192.168.100.0/24 comment="Mark connections to go out of ISP 1"
add action=mark-connection chain=prerouting disabled=no dst-port=53,80,110,113,443,6667,6668,7000 new-connection-mark=to_outside1 passthrough=yes protocol=udp src-address=!192.168.100.0/24
add action=mark-connection chain=prerouting comment="Mark everything else for ISP2" new-connection-mark=to_outside2 passthrough=yes src-address=!192.168.100.0/24
add action=mark-routing chain=prerouting connection-mark=to_outside1 new-routing-mark=to_outside1
add action=mark-routing chain=prerouting connection-mark=to_outside2 new-routing-mark=to_outside2
/ip route
add distance=5 dst-address=0.0.0.0/0 gateway=88.88.66.1
add distance=1 dst-address=0.0.0.0/0 gateway=88.88.66.1 routing-mark=to_outside1
add distance=2 (replicate this for the DHCP route, in theory that should always stay the same, it's just your IP address that can change.)
/ip firewall nat
add chain=dstnat action=dst-nat src-address=192.168.100.0/24 to-address=172.10.10.2
add chain=srcnat action=masquerade out-interface=ether2
add chain=srcnat action=masquerade out-interface=ether3

taduikis · Sat Sep 18, 2010 7:44 pm

Well yeah, that part with IPs in different subnets from ISP1 does make it tricky. Unfortunately I was given no other option, when I ordered few additional IPs they couldn't give ones that would be in the same subnet as two existing ones I had for ages.

The reason I want to distribute these certain public IPs for different subnets is to provide some distinction and separation. For example we have here lots of news sites where visitors can comment articles and real wars are happening there between commentators, so sometimes they get banned from writing comments or even visiting whole news portals. Some torrent trackers also limit accessing them too much from the same IP. So in the end, each different living block gets it's own IP, it doesn't completely solve the problem, but reduces it little, that's the best I can do as far as I'm concerned.

So to be more specific, everything looks like this.
Destination ports 53,80,110,113,443,6667,6668,7000 are let out in such a manner:
192.168.0.0/24 => 88.88.66.70
192.168.2.0/24 => 88.88.66.82
192.168.3.0/24 => 77.77.253.154
192.168.4.0/24 => 77.77.253.154
192.168.5.0/24 => 77.77.253.154

Every dst-port that isn't in that list (or subnet) is routed through default gateway, which is ISP2. That means all heavy traffic (like torrents and various other p2ps, ftp, etc..) doesn't interfere with browsing www.

192.168.100.0/24 is not used much and doesn't need to have internet access to it. I have explained its purpose in my previous post.
172.10.10.0/30 does not need internet also, it's just a separate subnet where I keep my web server. This could of course be connected to a separate ethernet nic, but unfortunately I have no space for it left. This could be changed if I move my router to RB1100 hw, but currently everything works fine as it is.

But what concerns me, that all this routing table mess shouldn't be related with connmarking or in fact anything in the RouterOS firewall section at all. In my Linux config, I could have easily flushed everything in iptables and it still worked. I mean I would get replies when pinging 8.8.8.8 with src-address set to one of these public IPs of ISP1..
Something in routing table might be missing, or some magic with route distances has to be done..

raymonvdm · Mon Sep 27, 2010 9:32 pm

Hi,

I also trying to estaiblish a dual wan router using mikrotik, I`m using a RB192 for 3 years now and am trying to add a secondary cable interface using the example in this post

Here are my settings:


 7   ;;; Mark connections to go out of Ziggo
     chain=prerouting action=log protocol=tcp dst-port=119 log-prefix="" (to see of packets ar forwarded to the right interface)

 8   ;;; Mark connections to go out of Ziggo
     chain=prerouting action=mark-connection new-connection-mark=to_ziggo passthrough=yes protocol=tcp dst-port=119 

 9   chain=prerouting action=mark-connection new-connection-mark=to_ziggo passthrough=yes protocol=udp dst-port=119 

10   ;;; Mark everything else for Solcon
     chain=prerouting action=mark-connection new-connection-mark=to_solcon passthrough=yes 

11   chain=prerouting action=mark-routing new-routing-mark=to_ziggo passthrough=yes connection-mark=to_ziggo 

12 X chain=prerouting action=mark-routing new-routing-mark=to_solcon passthrough=yes connection-mark=to_solcon

Routing table


add distance=1 dst-address=0.0.0.0/0 gateway=xx.xx.52.1 routing-mark=to_solcon disabled=no
add distance=2 dst-address=0.0.0.0/0 gateway=xx.xx.174.1 routing-mark=to_ziggo disabled=no
add distance=5 dst-address=0.0.0.0/0 gateway=xx.xx.52.1 disabled=no

But when i enable

12 X chain=prerouting action=mark-routing new-routing-mark=to_solcon passthrough=yes connection-mark=to_solcon

The packets seems to be forwarded the correct interfaces but i cannot connect to the outside world

What do i need to do wiht my nat rules, do the also know wich routing-mark to use ? Because i can`t seem to get it working

fewi · Mon Sep 27, 2010 9:40 pm

NAT usually shouldn't have anything with routing marks, you usually just have to make sure you have a rule in the srcnat chain that masquerades traffic for out-interface=INTERFACENAME.

Post the output of "/ip firewall nat export" if you're unsure.

raymonvdm · Mon Sep 27, 2010 10:00 pm

add action=src-nat chain=srcnat comment="MiniITX outbound SRCNAT to xx.xx.52.5" disabled=no dst-address-list=!NO-SRC-NAT src-address=192.168.110.200 to-addresses=xx.xxx.52.5
add action=src-nat chain=srcnat comment="MCE outbound SRCNAT naar xx.xxxx.52.2" disabled=no dst-address-list=!NO-SRC-NAT src-address=192.168.110.30 to-addresses=xx.xxx.52.2
add action=src-nat chain=srcnat comment="XBOX360 outbound SRCNAT naar xx.xxx52.4" disabled=no dst-address-list=!NO-SRC-NAT src-address=192.168.110.40 to-addresses=xx.xxx.52.4

add action=src-nat chain=srcnat comment="Ziggo Outgoing" disabled=yes dst-address-list=!NO-SRC-NAT src-address=192.168.110.0/24 to-addresses=xx.xxx.175.32
add action=src-nat chain=srcnat comment="Solcon DSL" disabled=no dst-address-list=!NO-SRC-NAT src-address=192.168.110.0/24 to-addresses=xx.xxx.52.3

These are the rules that i use, i have multiple ip adresses on the DSL connection and a Single ip on the Ziggo connection

fewi · Mon Sep 27, 2010 10:23 pm

You need to add the out-interface qualifiers. Otherwise the first rule matches both circuits and you NAT to the wrong IP address and the ISP drops it.

raymonvdm · Mon Sep 27, 2010 10:30 pm

But on what rules do i need to specify the out interface. Do i need to do this on all of them ?

fewi · Mon Sep 27, 2010 10:36 pm

add action=src-nat chain=srcnat comment="Ziggo Outgoing" disabled=yes dst-address-list=!NO-SRC-NAT src-address=192.168.110.0/24 to-addresses=xx.xxx.175.32
add action=src-nat chain=srcnat comment="Solcon DSL" disabled=no dst-address-list=!NO-SRC-NAT src-address=192.168.110.0/24 to-addresses=xx.xxx.52.3

On those. Add out-interface=INTERFACENAME on both, substituting the names of the ISP interfaces. Your config so far doesn't show them or I'd substitute them and post the resulting rules.

Without that, the traffic from 192.168.110.0/24 matches that first Ziggo rule. When traffic goes out the Solcon interface traffic gets NAT'd to xx.xxx.175.32 and Solcon drops it.

raymonvdm · Tue Sep 28, 2010 12:12 am

Code: Select all
add action=src-nat chain=srcnat comment="Ziggo Outgoing" disabled=yes dst-address-list=!NO-SRC-NAT src-address=192.168.110.0/24 to-addresses=xx.xxx.175.32
add action=src-nat chain=srcnat comment="Solcon DSL" disabled=no dst-address-list=!NO-SRC-NAT src-address=192.168.110.0/24 to-addresses=xx.xxx.52.3
On those. Add out-interface=INTERFACENAME on both, substituting the names of the ISP interfaces. Your config so far doesn't show them or I'd substitute them and post the resulting rules.

Without that, the traffic from 192.168.110.0/24 matches that first Ziggo rule. When traffic goes out the Solcon interface traffic gets NAT'd to xx.xxx.175.32 and Solcon drops it.

Do you mean that the interface name should be equal to the name of the mangle rule because it is still not working when i enable the last mangle rule. However when i disable this rule the masq is working

12 X chain=prerouting action=mark-routing new-routing-mark=to_solcon passthrough=yes connection-mark=to_solcon

These are my interfaces


set 1 arp=enabled auto-negotiation=yes bandwidth=unlimited/unlimited comment="" disabled=no full-duplex=yes l2mtu=1518 mac-address=00:0C:42:19:7B:86 master-port=none mtu=1500 \
    name="Eth2 - Solcon DSL" speed=100Mbps
set 4 arp=enabled auto-negotiation=yes bandwidth=unlimited/unlimited comment="Ziggo WAN" disabled=no full-duplex=yes l2mtu=1518 mac-address=00:0C:42:19:7B:89 master-port=none \
    mtu=1500 name="Eth5 - Ziggo WAN" speed=100Mbps

fewi · Tue Sep 28, 2010 12:31 am

Re-enable the mangle rule and edit these:

add action=src-nat chain=srcnat comment="Ziggo Outgoing" disabled=yes dst-address-list=!NO-SRC-NAT src-address=192.168.110.0/24 to-addresses=xx.xxx.175.32
add action=src-nat chain=srcnat comment="Solcon DSL" disabled=no dst-address-list=!NO-SRC-NAT src-address=192.168.110.0/24 to-addresses=xx.xxx.52.3

to look like this

add action=src-nat chain=srcnat comment="Ziggo Outgoing" disabled=yes dst-address-list=!NO-SRC-NAT src-address=192.168.110.0/24 to-addresses=xx.xxx.175.32 out-interface="Eth5 - Ziggo WAN" 
add action=src-nat chain=srcnat comment="Solcon DSL" disabled=no dst-address-list=!NO-SRC-NAT src-address=192.168.110.0/24 to-addresses=xx.xxx.52.3 out-interface="Eth2 - Solcon DSL"

That should do it

raymonvdm · Tue Sep 28, 2010 1:09 am

I don`t know what i am doing wrong because it still isn`t working. Even when i disable the passthough option of the mangle rules it still is not working.

I may have found another issue, when i ping from the outside world to the Ziggo ip the traffic is comming in on interface "Eth5 - Ziggo WAN" but the return traffic is sent out using "Eth2 - Solcon DSL"
so i must create something to let incomming traffic go out the same interface.

raymonvdm · Fri Oct 01, 2010 11:25 pm

I stil cannot get it to work

Traffic incomming on solcon interface is comming in and going out on the solcon inteface (This Works)
Traffic incomming on ziggo interface is comming in and going out on the ziggo interface (This Works)

But when i want to redirect al port 80 trafic to the ziggo interface is does not work, the traffic seems to be marked to the right connection mark and routing mark, but is sent out the wrong interface

Feklar · Sat Oct 02, 2010 12:34 am

I stil cannot get it to work

Traffic incomming on solcon interface is comming in and going out on the solcon inteface (This Works)
Traffic incomming on ziggo interface is comming in and going out on the ziggo interface (This Works)

But when i want to redirect al port 80 trafic to the ziggo interface is does not work, the traffic seems to be marked to the right connection mark and routing mark, but is sent out the wrong interface

Try running torch on the Ziggo interface and see what the IP addresses are. Does the IP match with one of the IP addresses assigned to that interface?

raymonvdm · Sun Oct 03, 2010 2:08 am

I already tried to find out what went wrong using torch. But i did see the traffic going out interface Ziggo, But i also did see the connections go out inteface Solcon

But now it seems to be working if i enable and disable the following rules. For the connections to be successfully marked i need to specify the ports on both mangle rules. One where i send them to Ziggo. And one where i need to specifily them not to go to solcon. I also need to disable the solcon routing mangle. But when i disable the connection mangle for solcon the internet is lost again.

Something strange is happening here. Maybe it`s a RouterOS bug ?

add action=mark-connection chain=prerouting comment="Mark connections to go out of Ziggo" disabled=no dst-address-list="!Private IPs" dst-port=80 new-connection-mark=2ziggo passthrough=yes protocol=tcp
add action=log chain=prerouting comment="" disabled=yes dst-address-list="!Private IPs" dst-port=80 log-prefix=ZIGGO protocol=tcp
add action=mark-connection chain=prerouting comment="" disabled=yes dst-address-list="!Private IPs" dst-port=80 new-connection-mark=2ziggo passthrough=no protocol=udp
add action=log chain=prerouting comment="Mark Routing Ziggo" connection-mark=conn_mark_ziggo disabled=yes log-prefix=ziggorouting
add action=log chain=prerouting comment="" connection-mark=2ziggo disabled=yes log-prefix=2ziggo
add action=mark-routing chain=prerouting comment="" connection-mark=2ziggo disabled=no new-routing-mark=2ziggo passthrough=yes
add action=log chain=prerouting comment="Mark connections to go out of Solcon" disabled=yes dst-address-list="!Private IPs" log-prefix=SOLCON
add action=mark-connection chain=prerouting comment="" disabled=yes dst-address-list="!Private IPs" new-connection-mark=2solcon passthrough=yes
add action=log chain=prerouting comment="" disabled=yes dst-address-list="!Private IPs" dst-port=!80 log-prefix=solcon protocol=tcp
add action=mark-connection chain=prerouting comment="" disabled=no dst-address-list="!Private IPs" dst-port=!80 new-connection-mark=2solcon passthrough=yes protocol=tcp
add action=mark-connection chain=prerouting comment="" disabled=no dst-address-list="!Private IPs" dst-port=!80 new-connection-mark=2solcon passthrough=yes protocol=udp
add action=log chain=prerouting comment="Mark Routing Solcon" connection-mark=conn_mark_solcon disabled=yes log-prefix=solconrouting
add action=mark-routing chain=prerouting comment="" connection-mark=2solcon disabled=yes new-routing-mark=2solcon passthrough=yes

For now it is working but the speed of the Ziggo connection is very slow. I`m figering out if this is a Ziggo issue or a Routerboard issue.

Feklar · Sun Oct 03, 2010 8:45 am

What I was refering to specifically is if you look a the IP listed under dst. address in Torch, it will tell you what IP your HTTP requests are getting NAT'ed out of. Using that you would be able to see if the wrong Mangle, or NAT rule is catching the HTTP connection, and narrow down what to look at.

raymonvdm · Sun Oct 03, 2010 1:38 pm

First i will be trying to contact diswire to get me a RB450G because i seem to stumble againt cpu speed issues on the RB192

However it is working for know i`m very unhappy with the speed. Als when i replace my Solcon connection with the Ziggo connection the RB192 seems to hit the roof

Routing issues with multiple ISPs

Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Re: Routing issues with multiple ISPs

Who is online