(Solved) 3 wan interfaces

RouterOS General
1

Hello,

I need to use Mikrotik to route traffic from ISP to 2 firewalls. So i have eth1 connected to the ISP gateway. eth2 is connected to fw1 and eth3 to fw2. All have public ip addresses.
From outside the network i can ping the ISP gtw ip and the Mikrotik eth1.
I can ping eth2 but cannot ping fw1. I receive a TTL expired in transit if i ping eth3 and fw2.

This is an IP schema similar to what i have:

ISP -4.4.8.73

ETH1 - 4.4.8.74
gtw - 4.4.8.73

eth2 - 4.4.8.205
fw1 - 4.4.8.206
gtw - 4.4.8.205

eth3 - 4.4.9.81
fw2 - 4.4.9.82
gtw - 4.4.9.81


/interface ethernet
set [ find default-name=ether1 ] comment=“WAN1 ISP”
set [ find default-name=ether2 ] comment=“WAN2 FGT”
set [ find default-name=ether3 ] comment=“WAN3 SW”
/ip neighbor discovery
set ether1 comment=“WAN1 ISP”
set ether2 comment=“WAN2 FGT”
set ether3 comment=“WAN3 SW”
/ip address
add address=4.4.8.74/30 interface=ether1 network=4.4.8.72
add address=4.4.8.205/30 interface=ether2 network=4.4.8.204
add address=4.4.9.81/28 interface=ether3 network=4.4.9.80
/ip dns
set servers=8.8.8.8,8.8.4.4,209.244.0.3,209.244.0.4
/ip route
add distance=1 gateway=4.4.8.73
/system routerboard settings
set protected-routerboot=disabled

Do i need to add anything else to the configuration? What’s wrong?

Thanks

2 SUBNETS + 2 LANS + VPN in one GW
2

Probably you have destroyed your example by changing the IP numbers, as normally this would work.

3

I just removed the first number of an octet, i can assure you it’s very acurate…

4

TTL expired in transit generally implies that two devices on a link have mismatched routing table information.

Suppose R1 and R2 are routers connected to each other using a /30 ptp network.
R1 = 10.0.0.1/30 and R2 = 10.0.0.2/30
R1 has a static route 192.0.2.0/24 gateway=10.0.0.2
R2’s default GW is 10.0.0.1
Suppose further that R2 does not have any interface configured with any 192.0.2.x address, and has no routes for 192.0.2.0/24

If a packet arrives at R1 with a destination IP of 192.0.2.192, it will forward this packet to 10.0.0.2 (R2)
R2 will receive the packet, and the routing table will only have one matching entry: the default GW.
R2 will forward the packet to R1.
R1 will forward it back to R2, which will forward it back to R1.
The packet will thus bounce back and forth until the TTL reaches zero, and whichever router has the packet when it reaches zero will discard the packet and send the ICMP “TTL Expired” message to the packet’s source IP.

5

You are right. If i ping the Mikrotik interface that is connected to the ISP i’m getting TTL expired. Is the Mikrotik configuration ok? Do i need to change something?

Thanks

6

The information in your post is a bit unclear to me - your problem states that you have 3 WANs, but your sample configurations make it sound as if you have one WAN, and then two firewalls attached to your router - as if they’re clients…

If your configuration is truly multi-wan, then you have some work to do in the mangle table in order to implement policy routing.
The reason the pings are most likely failing (if the pings are coming in WAN2, for instance) is that the reply is probably being sent out WAN1, and the ISP is doing anti-spoofing (a good thing) on that connection, thus dropping the reply packet as it’s coming “from” an IP address that is not connected to this interface.

Configure policy routing as follows:

**Step1: **Set up policy-based default GW routes.

[size=100]/ip route add dst=0.0.0.0/0 gateway=4.4.8.73 routing-mark=ISP1[/size]
[size=100]/ip route add dst=0.0.0.0/0 gateway=4.4.8.205 routing-mark=ISP2[/size]
[size=100]/ip route add dst=0.0.0.0/0 gateway=4.4.9.81 routing-mark=ISP3[/size]

These routes will be used for any packet with a matching routing mark applied to them.

Step2: Exempt any of your locally-connected IP ranges from policy routing using route rules

/ip route rule add dst=192.168.0.0/16 action=lookup-only-in-table table=main

Step3: Use connection marking to make IP connections “sticky” to whatever interface they should be routed to…

For firewall efficiency, make a custom chain “classify” that marks new connections so they stick to a particular ISP

/ip firewall mangle
add chain=prerouting connection-mark=no-mark action=jump jump-target=classify

Mark inbound connections from the various ISP connections.

/ip firewall mangle
add chain=classify in-interface=ether1 action=mark-connection new-connection-mark=wan1
add chain=classify in-interface=ether2 action=mark-connection new-connection-mark=wan2
add chain=classify in-interface=ether3 action=mark-connection new-connection-mark=wan3
add chain=classify connection-mark=!no-mark action=return comment="exit chain if connection has been marked"

Mark outbound connections from the LAN based on some balancing criteria, such as PCC, and return from the chain when finished

/ip firewall mangle
add chain=classify per-connection-classifier=src-address-and-port:3/0 action=mark-connection new-connection-mark=wan1
add chain=classify connection-mark=no-mark per-connection-classifier=src-address-and-port:3/1 action=mark-connection new-connection-mark=wan2
add chain=classify connection-mark=no-mark per-connection-classifier=src-address-and-port:3/2 action=mark-connection new-connection-mark=wan3
add chain=classify action=return

Step 4: apply routing marks to packets in prerouting and output chains

/ip firewall mangle

add chain=output connection-mark=wan1 action=mark-routing new-routing-mark=ISP1
add chain=output connection-mark=wan2 action=mark-routing new-routing-mark=ISP2
add chain=output connection-mark=wan3 action=mark-routing new-routing-mark=ISP3
add chain=prerouting connection-mark=wan1 action=mark-routing new-routing-mark=ISP1
add chain=prerouting connection-mark=wan2 action=mark-routing new-routing-mark=ISP2
add chain=prerouting connection-mark=wan3 action=mark-routing new-routing-mark=ISP3

Step5: Use masquerade as src-nat for packets leaving each WAN interface:
/ip firewall nat
add chain=srcnat action=masquerade out-interface=ether1
add chain=srcnat action=masquerade out-interface=ether2
add chain=srcnat action=masquerade out-interface=ether3

Step6: Ensure that your firewall filter rules protect the router on all 3 WAN interfaces, etc.

/ip firewall filter
add chain=input connection-state=established,related action=accept
add chain=input protocol=icmp action=accept
add chain=input in-interface=ether4 action=accept
add chain=input action=drop

Note that in-interface=ether4 assumes that ether4 is your LAN. If you have multiple LAN interfaces, then add duplicates of this rule for each LAN IP interface you have in your configuration.
Your forward chain should be similar in structure to the input chain:

Pseudo-code:
fast-track established,related
allow established,related (for protocols that don’t work with fasttrack)
allow out-interface=ether1
allow out-interface=ether2
allow out-interace=ether3
allow in-interface=LAN
drop

7

Thanks for your extended reply. Let me be more explicit. The ISP has assigned us 2 ip ranges. They are coming on the same interface . I need to split this by sending 1 ip range to a firewall and the other one to a 2nd firewall. I don’t have any LAN connected to the Mikrotik. Usually if i add the ip addresses and the correct gateway a router should route all the packets without any additional setup. I’m guessing that for Mikrotik i need additional configuration. Please correct me if I’m wrong.

Thanks

8

Okay - your configuration can be very very very simple:
no firewall forward chain rules
no NAT rules
no mangle rules
Just put IP addresses on 3 interfaces, and make sure there are no bridges configured, and that the ethernet interfaces are set with master-port=none
Set the default GW for the Mikrotik itself (0.0.0.0/0 gateway=isp IP address on eth1)
Done
You can protect the Mikrotik itself using the input chain of the firewall filter - basically allow established,related connections, and icmp. AFter that, drop all with in-interface=ether1
You don’t need to add any routes for the client FW interfaces because when you put the IP addresses on those interfaces, they are automatically added to the routing table as connected routes.

9

Thanks for your reply. This is exactly what i did. I reset the router and removed the default config. Then i added the interfaces ip and the default gateway. But is not working as expected. This is the /export

jan/03/1970 01:14:18 by RouterOS 6.35.4

software id = I2DY-SMLT

/interface ethernet
set [ find default-name=ether1 ] comment="WAN1 ISP"
set [ find default-name=ether2 ] comment="WAN2 FGT"
set [ find default-name=ether3 ] comment="WAN3 SW"
/ip neighbor discovery
set ether1 comment="WAN1 ISP"
set ether2 comment="WAN2 FGT"
set ether3 comment="WAN3 SW"
/ip address
add address=4.4.8.74/30 interface=ether1 network=4.4.8.72
add address=4.4.8.205/30 interface=ether2 network=4.4.8.204
add address=4.4.9.81/28 interface=ether3 network=4.4.9.80
/ip dns
set servers=8.8.8.8,8.8.4.4,209.244.0.3,209.244.0.4
/ip route
add distance=1 gateway=4.4.8.73

Could this be related to the ISP config? I will ask them to double check.
Thanks

10

Yes, the ISP could be routing it to you incorrectly.

Try doing a traceroute to your IP addresses from the outside to see where things break down. If your router’s WAN address ever appears (4.4.8.74) then the ISP has done their job with routing. It’s also possible that they are filtering some ports.

Of course the firewalls need to be correctly configured. You can also test whether your ISP is to blame directly by adding a 4th interface ether4 = 192.168.1.1/24 and configuring a test PC as 192.168.1.2 / 255.255.255.0 / gateway=192.168.1.1, and connecting it to ether4
Then try to ping/test/etc to your firewalls from this test laptop. If this also fails, then you’ve got problems in your configuration somewhere - almost certainly not the Mikrotik, as it’s pretty much being used as the TCP/IP equivalent of a “dumb switch” and there’s almost nothing to break in the configuration there. I’d check the firewalls and their policies.

EDIT: Note - you won’t be able to surf the Internet from this test PC without also adding a masquerade rule to the Mikrotik:
/ip firewall nat add chain=srcnat src-address=192.168.1.0/24 out-interface=ether1 action=masquerade

11

Thank you for your help. The issue was on the ISP side. I wasted my time trying to fix something that was not broken. Sorry for wasted your time too…