Dual WAN VPN SEerver

SomeYoungGuy · Thu Jul 20, 2017 2:29 pm

I'm busy setting something up, that should be quite easy, but something seems to be failing.
I want to set up the router as pretty much a VPN server only, so i don't need any other traffic flowing over it other than whats comes in over the VPN, and i want it setup in a multiple WAN senarios, so that i can have a VPN over ISPA, B, C, etc.
My VPNS are not connecting fully, I am getting to the "SYN" status only. If i enable a default gateway, then connections work fine, but I don't want this as i only want it that ISPA traffic flows over ISPA's network.

My understanding is that the Mangle Rules would do this for you, but it does not seem to work.

/ip address
add address=XXX.XXX.XXX.XXX/27 comment="ISP-A" interface=ether1 network=XXX.XXX.XXX.XXX
add address=XXX.XXX.XXX.XXX/27 comment="ISP-B" interface=ether2 network=XXX.XXX.XXX.XXX

/ip firewall mangle
add action=mark-routing chain=prerouting in-interface=ether1 log=yes new-routing-mark=ISP-A-Route-Out
add action=mark-routing chain=prerouting in-interface=ether2 log=yes new-routing-mark=ISP-B-Route-Out

/ip route
add distance=1 gateway=XXX.XXX.XXX.XXX routing-mark=ISP-A-Route-Out
add distance=1 gateway=XXX.XXX.XXX.XXX routing-mark=ISP-B-Route-Out
add disabled=yes distance=99 gateway=XXX.XXX.XXX.XXX  #Only if i enable this does the VPN and basically any connection to the router work

I have two live IP addresses bound to the two nics, so its an absolute in-and-out arrangement - where traffic in on eth1 must basically go out of eth1. I can use connection marking, but it seems pointless since the prerouting Chain can identify packs based on its In-Interface. My logs show it hitting the prerouting, so im guessing it has marked the packets, but it doesnt return any packets unless i enable the default route without any route marking filter.

Any assistance would help guys,

P.S. Pinging each via Each ISP works correctly

idlemind · Thu Jul 20, 2017 5:46 pm

How about packets that need to egress the router? How do they know which interface to use and without a routing able entry life is a little tough.

One option would be to get a prefix (IPv4 or IPv6) and use BGP to do some rough prefix based load-balancing. This can induce Asymmetry but would at least help control which pipe is used when egressing the router in a much cleaner fashion.

To continue down your path of 2 ISP connections w/a /27 each... Try marking the connections on ingress and using the connection markings to perform the policy routing (routing-marks). Depending on your situation (if you have NAT going on as well and the destination servers are really behind the MikroTik you may need to switch from input to forward on the connection-marking rules.

/ip firewall mangle remove [ find where action=mark-routing and new-routing-mark=ISP-A-Route-Out ]
/ip firewall mangle remove [ find where action=mark-routing and new-routing-mark=ISP-B-Route-Out ]

/ip firewall mangle add action=mark-connection chain=input in-interface=ether1 log=yes connection-mark=no-mark new-connection-mark=ISP-A-Route-Out
/ip firewall mangle add action=mark-connection chain=input in-interface=ether2 log=yes connection-mark=no-mark new-connection-mark=ISP-B-Route-Out

/ip firewall mangle add action=mark-routing chain=prerouting log=yes connection-mark=ISP-A-Route-Out new-routing-mark=ISP-A-Route-Out
/ip firewall mangle add action=mark-routing chain=prerouting log=yes connection-mark=ISP-B-Route-Out new-routing-mark=ISP-B-Route-Out

That should get you there. I'm not in a position to test the syntax so be gentle

SomeYoungGuy · Fri Jul 21, 2017 4:53 pm

Yip, that's pretty much what i tried... The SYN packet is sent, and it cant/doesn't get a SYNACK. Anything going in, doesn't go out, unless i have a default route out that doesn't have a Routing mark requirement.
Remember the use case here is that direct VPN connections to this actual router will be made only. Before i solve the flow of the packets out of the VPN, and forwarded on... i want the VPNs to connect directly to he router. The VPN service is the one on the actual Router.

I can ping the router and get replies from the correct eth
I cannot connect to the the web port 8080, or the Winbox Port, or VPN... unless i allow a default gateway without a route mark requirement. But then of course it set the outbound path based on the gateway statically.

Is it possible that services that are provided directly by the Router like web, winbox, VPN etc, strip the routing mark? Maybe then i need to mark the packet flow "from" the service?

If a log at each stage, according to this diagram:
https://wiki.mikrotik.com/images/thumb/ ... px-Pfd.png

I get a log up to the "Input Filter", then no further logging. Where in this diagram does the Router provided Services sit?

idlemind · Fri Jul 21, 2017 5:08 pm

So you changed your rules to look like what I put in (connection tracking on ingress vs routing-mark) and it didn't work? Your initial post only shows the routing-mark (policy routing) policies.

MikroTik posters have confirmed that on-board services do not respect VRFs so it's possible on-board services won't respect policy routing. That remains to be seen though.

EDIT: I just tested this, both the connection-mark and policy-routing mangle get hit by the internal service but it doesn't seem to get a match in the routing-table. I suspect internal services do not respect routing-marks during route-lookup. Maybe MikroTik can post to clear it up more officially.

SomeYoungGuy · Fri Jul 21, 2017 10:51 pm

EDIT: I just tested this, both the connection-mark and policy-routing mangle get hit by the internal service but it doesn't seem to get a match in the routing-table. I suspect internal services do not respect routing-marks during route-lookup. Maybe MikroTik can post to clear it up more officially.

Hi idlemind, Yea, that's pretty much what i found.

I'm not sure though how to manipulate the packets to make them go out the correct way. So let's say it meaning less to mark them on the way on, but what if we could catch and mark them on the way out?

idlemind · Fri Jul 21, 2017 10:56 pm

I think your snag is in the route-lookup section and specific to traffic originated on the box. Oddly enough we had a similar situation with Cisco iWAN. I could look up the details but it ended up that we routed the traffic 1 hop in and then back out so the router saw it like it would any other traffic. Some hacky work around might be needed here too.

andywright · Sat Jul 22, 2017 12:00 am

I'm busy setting something up, that should be quite easy, but something seems to be failing.
I want to set up the router as pretty much a VPN server only, so i don't need any other traffic flowing over it other than whats comes in over the VPN, and i want it setup in a multiple WAN senarios, so that i can have a VPN over ISPA, B, C, etc.
My VPNS are not connecting fully, I am getting to the "SYN" status only. If i enable a default gateway, then connections work fine, but I don't want this as i only want it that ISPA traffic flows over ISPA's network.

My understanding is that the Mangle Rules would do this for you, but it does not seem to work.
Code: Select all
/ip address
add address=XXX.XXX.XXX.XXX/27 comment="ISP-A" interface=ether1 network=XXX.XXX.XXX.XXX
add address=XXX.XXX.XXX.XXX/27 comment="ISP-B" interface=ether2 network=XXX.XXX.XXX.XXX

/ip firewall mangle
add action=mark-routing chain=prerouting in-interface=ether1 log=yes new-routing-mark=ISP-A-Route-Out
add action=mark-routing chain=prerouting in-interface=ether2 log=yes new-routing-mark=ISP-B-Route-Out

/ip route
add distance=1 gateway=XXX.XXX.XXX.XXX routing-mark=ISP-A-Route-Out
add distance=1 gateway=XXX.XXX.XXX.XXX routing-mark=ISP-B-Route-Out
add disabled=yes distance=99 gateway=XXX.XXX.XXX.XXX  #Only if i enable this does the VPN and basically any connection to the router work
I have two live IP addresses bound to the two nics, so its an absolute in-and-out arrangement - where traffic in on eth1 must basically go out of eth1. I can use connection marking, but it seems pointless since the prerouting Chain can identify packs based on its In-Interface. My logs show it hitting the prerouting, so im guessing it has marked the packets, but it doesnt return any packets unless i enable the default route without any route marking filter.

Any assistance would help guys,

P.S. Pinging each via Each ISP works correctly

I'll throw this in here if only for discussion as I doubt it's the correct/best way of doing this.

Anyway, I've been setting up something similar - two WAN connections, one is a 1Gig fibre and the other a 3G router as a fallback connection. I've got this to the point where I can access the router (eg. Webfig) over either connection. I'm using Mangle rules on Input and Output to mark incoming packets and to set outgoing routing based on those marks. Here's where I am at the moment...

/ip address
add address=156.xxx.xxx.246/24 interface=ether1-WAN network=156.xxx.xxx.0
add address=192.168.88.1/24 interface=ether2-TestLAN network=192.168.88.0
add address=192.168.100.148/24 interface=ether5-192.168.100 network=192.168.100.0
add address=192.168.1.2/24 interface=ether4-3G network=192.168.1.0

/ip firewall mangle
add action=mark-connection chain=input connection-mark=no-mark connection-state=new in-interface=ether1-WAN new-connection-mark=WAN passthrough=no
add action=mark-routing chain=output connection-mark=WAN new-routing-mark=WAN passthrough=no
add action=mark-connection chain=input connection-mark=no-mark connection-state=new in-interface=ether4-3G new-connection-mark=3G passthrough=no
add action=mark-routing chain=output connection-mark=3G new-routing-mark=3G passthrough=no

/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1-WAN
add action=masquerade chain=srcnat out-interface=ether4-3G

/ip route
add distance=1 gateway=<3G Router Gateway> routing-mark=3G
add distance=1 gateway=<WAN Gateway> routing-mark=WAN
add distance=1 gateway=<WAN Gateway>
add distance=10 gateway=<3G Router Gateway>

idlemind · Sat Jul 22, 2017 1:23 am

I'd be curious which interface is actually being used for the egress traffic. Are you able to do a packet capture (possibly torch) and see the traffic actually egressing 3g when that IP is accessed while the primary ISP is still up (to comply with bcp38)?

andywright · Sat Jul 22, 2017 2:10 am

I'd be curious which interface is actually being used for the egress traffic. Are you able to do a packet capture (possibly torch) and see the traffic actually egressing 3g when that IP is accessed while the primary ISP is still up (to comply with bcp38)?

The correct interface is definitely being used - I see traffic in both directions with packet sniffer. I can access Webfig over the interface connected to the 3G router from an iPhone on a 4G connection; the router's 'other' internet connection is the default route. I've also just got ospf-gre-ipsec running over both connections to another mikrotik box. That setup creates 2 gre-over-ipsec tunnels - one over the main fibre internet connection and the other over 3G, then runs ospf over those to give redundant vpn connectivity between the 2 routers.

SomeYoungGuy · Sat Jul 22, 2017 11:12 am

I think your snag is in the route-lookup section and specific to traffic originated on the box. Oddly enough we had a similar situation with Cisco iWAN. I could look up the details but it ended up that we routed the traffic 1 hop in and then back out so the router saw it like it would any other traffic. Some hacky work around might be needed here too.

I'd be very interested to know how I can do this?

I had thought about doing something with dst-nat, like to nat the incoming connection to another port, say eth8 (on 192.168.1.2) - i sure tried that, but think i ended up with similar results.

BTW, in the end what im looking for is a way to bond the two VPN's, so that I can have an "ultra robust" point-to-point VPN, that is fault tolerant (i.e. if one ISP goes down, it will not effect the overall connection). All this WITHOUT BGP (has to be).

Dual WAN VPN SEerver

Dual WAN VPN SEerver

Re: Dual WAN VPN SEerver

Re: Dual WAN VPN SEerver

Re: Dual WAN VPN SEerver

Re: Dual WAN VPN SEerver

Re: Dual WAN VPN SEerver

Re: Dual WAN VPN SEerver

Re: Dual WAN VPN SEerver

Re: Dual WAN VPN SEerver

Re: Dual WAN VPN SEerver

Who is online