I am reasonably new to RB and ROS having moved from using custom Linux and BSD distros on embedded PC hardware. I have a lot of experience with networks but reasonably new to RB.

I have a situation on test where I have RB150 with 3 DSL lines. Each DSL line has a /29 with RB assigned one address and the DSL router assigned another. I have each DSL line on ether2,3 and 4 and ether1 is LAN.

I have setup masquerade and by setting default gateway I can masq through the connection that is set as default gateway.

However I noticed that when I try to connect to the RB using its IP address on the DSL connection I can only connect to the connection that contains the default gateway.

On past systems I expected the router core to transmit reply packets using the gateway for the subnet in which the IP address it received the packet from. In effect, routing the packet to the gateway in the subnet in which the router source address lines. This doesnt seem to be the case on RoS. I noticed that there is no obvious way to define the gateway for each subnet.

A quick check with torch and a few log rules showed that the packet would arrive on an interface for the relevant DSL connection e.g. ether4 but the reply would try to leave by the default gateway and interface for the defaulty gateway e.g. ether2 but using a source address for ether4. A further check on ARP table showed the RB was sending the packet to the gateway router on ether2 with a source address for ether4. Source address rejection on the DSL router (either client or head end) rejected the packet.

The next step I tried was to implement source address routing. I added a mangle rule to add a route mark for each packet based on source address. I then added routing rules to route for dest 0.0.0.0/0 conditional on the router mark to the gateway for the relevant src address. This all seemed quite sensible.

However the system still tried to send the packet on the default gateway. I thought the mark was not being applied. A few more log commands showed the packet was being tagged with a routing mark.

At this stage I cannot get the system to work. I have been through the documentation many times, have defaulted the configuration and rebuilt it several times, scanned the forums and tried everything. I cant believe someting as fundamental and simple as this can be so difficult. I am at a loss to see what I have missed and would appreciate any comments. I have noticed a few posts on the forum that suggest there may be an issue with route marking on the packets leaving the router itself.

For those interested, my plan is to implement a bonding service. I have full access to my own routers in a data centre with excellent connectivity. I also have access to the back of the head-end routers on our DSL service. I plan to place the RB between the DSL head-end routers and the transit routers. I have a large IP range in place so I could offer bonded public IP address ranges. I can thus offer a RB bonding service or a DSL bonding service. At this stage I have a very successful service running on my embedded BSD platform but I have packet flow limitations and would like to move to RB. A longer term plan would then be to offer this as a service that RB users could use and we could use for client solutions.

To make all of this work I plan 2 or more EoIP tunnels and bonding. But this means i need to make sure the packets leave the router on the correct DSL interface at the tail-end and ensure that tunnel1 leaves on DSL1, tunnel2 on DSL2 etc. At this stage I cannot get this to happen.

If anyone can offer any suggestions or comments I would be very appreciative! Thanks in advance for all who read/comment.

What you are trying to do is possible. You have to connection mark, then packet mark, then route mark incoming packets on each incoming interface. This allows you to then setup a route table for each and make sure the packets coming in one DSL will go out the same interface.

Post some rules / mangle chains / routing tables and we'll help you get it right.

Sam

Thanks. From what I have read on forums I knew it must be possible. I had been testing with ICMP which is not good as it does not work with connection tracking. My next test was to connection track GRE then try that. A bit of work with torch on a working single EoIP test tunnel had shown that EoIP seems to use only GRE.

I will tidy up the test routers to eliminate everything except the EoIP tests then will post the interface, route and mangle dumps.

Thanks for the offer of help - much appreciated!

I have the same sort of thing running on WRAP boards and Linux. I use GRE tunnels, teql and some scripts to do the monitoring. However RB is much more elegant hardware and software and I am very keen to move to RB - especially to support option for hosted head-end RB for bonding as a service.

I have prepared some information on the setup and the tests I have completed. Information and dumps attached below. I will leave the configuration in this state and will apply only changes related to this thread so we have a controlled environment.

Thanks again for all the help. It is very welcome and much appreciated! As this seems common I am willing to document this extensively if we can get it sorted out so that it may help others in a similar situation.

Intended scenario
=================

--- WAN1 ---\ /---- WAN1 ---
left --- rb --- WAN2 ----- Internet ----- WAN2 --- rb --- right
--- WAN3 ---/ \---- WAN3 ---

setup bonded tunnels to create

left --- rb ---- BONDED TUNNEL SET ---- rb --- right

Strategy
======
Bonded Tunnel Set will be created using 3 EoIP tunnels
Each EoIP tunnel will use separate WAN pair at left and right

Problems
======
Unable to establish multiple tunnels between endpoints simultaneously
Unable to direct EoIP tunnel to use a specific WAN pair

Notes
====

IP Addresses are added to EoIP tunnels only for testing. In final configuration these will be removed and a single /30 will be assigned to the bonding interface to which the EoIP tunnels will be enslaved. This was not attempted initially until the tunnel issues were resolved.

Test Situation
=========

acsrb03 - right

3 WAN connections on separate physical interfaces

ether1 LAN 10.1.1.3/24
ether2 Zen1 62.3.96.195/27 gw 62.3.96.222
ether3 Zen2 217.155.236.210/29 gw 217.155.236.214
ether4 Bytel1 80.76.193.235/29 gw 80.76.193.238
ether5 NC

acsrb04 - left

3 WAN connections on ether2 via shared switch ports

ether1 LAN 192.168.0.246/24
ether2 Bytel1 80.76.200.130/26 gw 80.76.200.192
ether2 Zen1 217.155.138.147/29 gw 217.155.138.150
ether2 Zen2 82.69.183.210/29 gw 82.69.183.214
ether3 NC
ether4 NC
ether5 NC

IPIP Tunnel Test
===========

Interfaces:
Router Name Local Remote
Left ipip1 82.69.193.210 217.155.236.210
Left ipip2 80.76.200.130 80.76.193.235
Right ipip1 217.155.236.210 82.69.183.210
Right ipip2 80.76.193.235 80.76.200.130
Addresses:
Router Interface Address
left ipip1 192.168.99.38/30
left ipip2 192.168.99.42/30
right ipip1 192.168.99.37/30
right ipip2 192.168.99.41/30

Tests:
------
1. Right can ping 192.168.99.38 if default route is set to 217.155.236.210. Cannot ping 192.168.99.42. Left can ping 192.168.99.37 if default route set to 82.69.183.210. Cannot ping 192.168.99.41
2. Right can ping 192.168.99.42 if default route set to 80.76.193.238. Cannot ping 192.168.99.38. Left can ping 192.168.99.41 if default route set to 80.76.200.190. Cannot ping 192.168.99.42
3. No pings transfer if default route on right set to 217.155.236.210 and on left to 80.76.200.190.
4. Torch and firewall log of IPIP packets show traffic egresses on default gateway. Router selects interface and ARPs for gateway for default route.

Conclusion
----------
Tunnel packets are not being exhcnaged between endpoints unless default gateways direct traffic to connection specifieid in tunnel remote address at far end.
Two tunnels using separate WAN connections has not been achieved.
Router does not select egresss interface and router based on source address of tunnel packets.

EOIP Tunnel Test
===========
Interfaces:
Router Name MAC Id Remote Address
Right eoip1 00:00:5e:80:00:10 1 80.76.200.130
Right eoip2 00:00:5e:80:00:20 2 82.69.183.210
Left eoip1 00:00:5e:80:00:11 1 80.76.193.235
Left eoip2 00:00:5e:80:00:21 2 217.155.236.210
Addresses:
Router Interface Address
left eoip1 192.168.99.34/30
left eoip2 192.168.99.30/30
right eoip1 192.168.99.33/30
right eoip2 192.168.99.29/30

Tests and conclusion
--------------------
Tests for IPIP tunnel were repeated and results replicated for EoIP tunnels. The conclusion regarding default gateway was reinforced.

Resolution
=======
IP packets need to be routed according to the source address to ensure that packets egress on the interface containing the subnet for the source address and are addressed to the default gateway for the relevant source address.

Queries
=====

EoIP tunnels need to be bound to use a specific WAN interface. There appears to be no way to ensure a specific EoIP tunnel uses a specific WAN interface. IPIP tunnels use the concept of a local address which suggets an IPIP interface can be bound to a WAN interface. How can an EoIP tunnel pair be forced to use a speciic WAN pair to create situation as follows:

EoIP1 uses WAN1 at left and right
EoIP2 uses WAN2 at left and right
EOIP3 uses WAN3 at left and right
Bond1 enslaves EOIP1, EOIP2 and EOIP3 at left and right

Solution Attempt 1
============

Packets need to be routed according to source IP address. Route marking could be used anfd following was deployed:

1. Insert a mangle rule in firewall on output chain (tag packets egressing router) and match source address. Assign a new routing mark tagging packet for relevant interface
2. Insert a route rule to route packet according to routing mark and route to relevant gateway and interface based on routing mark.
3. Insert two log rules into magle to log packets from source address on WAN with PKT ZEN2 and packets with routing mark ZEN2 with ROUTE MARK ZEN2

Results
-------
The log shows the packet being logged and the route marked packet being logged. This suggests the route mark is effective. Torch and log on the egress interface shows the packet still leaves by the default gateway and interface.

Conclusion
----------
The packet originated by the router appears to be tagged and marked but the route rule appears to ignore the route rule and instead uses the default gateway.

Router Configuration Dumps
==========================

Left
----

/interface ethernet export
set ether1 name="ether1" mtu=1500 mac-address=00:0C:42:12:24:FE arp=enabled \
disable-running-check=yes auto-negotiation=yes full-duplex=yes \
cable-settings=default speed=100Mbps comment="" disabled=no
set ether2 name="ether2" mtu=1500 mac-address=00:0C:42:12:24:FF arp=enabled \
disable-running-check=yes auto-negotiation=yes full-duplex=yes \
cable-settings=default speed=100Mbps comment="" disabled=no
set ether3 name="ether3" mtu=1500 mac-address=00:0C:42:12:25:00 arp=enabled \
disable-running-check=yes auto-negotiation=yes full-duplex=yes \
cable-settings=default speed=100Mbps comment="" disabled=yes
set ether4 name="ether4" mtu=1500 mac-address=00:0C:42:12:25:01 arp=enabled \
disable-running-check=yes auto-negotiation=yes full-duplex=yes \
cable-settings=default speed=100Mbps comment="" disabled=yes
set ether5 name="ether5" mtu=1500 mac-address=00:0C:42:12:25:02 arp=enabled \
disable-running-check=yes auto-negotiation=yes full-duplex=yes \
cable-settings=default speed=100Mbps comment="" disabled=yes
/interface eoip export
add name="eoip1" mtu=1500 mac-address=00:00:5E:80:00:11 arp=enabled \
remote-address=80.76.193.235 tunnel-id=1 comment="" disabled=no
add name="eoip2" mtu=1500 mac-address=00:00:5E:80:00:21 arp=enabled \
remote-address=217.155.236.210 tunnel-id=2 comment="" disabled=no
/interface ipip export
add name="ipip1" mtu=1480 local-address=82.69.183.210 \
remote-address=217.155.236.210 comment="" disabled=no
add name="ipip2" mtu=1480 local-address=80.76.200.130 \
remote-address=80.76.193.235 comment="" disabled=no

/ ip address export
add address=192.168.0.246/24 network=192.168.0.0 broadcast=192.168.0.255 \
interface=ether1 comment="" disabled=no
add address=217.155.138.147/29 network=217.155.138.144 \
broadcast=217.155.138.151 interface=ether2 comment="" disabled=no
add address=80.76.200.130/26 network=80.76.200.128 broadcast=80.76.200.191 \
interface=ether2 comment="" disabled=no
add address=82.69.183.210/29 network=82.69.183.208 broadcast=82.69.183.215 \
interface=ether2 comment="" disabled=no
add address=192.168.99.38/30 network=192.168.99.36 broadcast=192.168.99.39 \
interface=ipip1 comment="" disabled=no
add address=192.168.99.42/30 network=192.168.99.40 broadcast=192.168.99.43 \
interface=ipip2 comment="" disabled=no
add address=192.168.99.34/30 network=192.168.99.32 broadcast=192.168.99.35 \
interface=eoip1 comment="" disabled=no
add address=192.168.99.30/30 network=192.168.99.28 broadcast=192.168.99.31 \
interface=eoip2 comment="" disabled=no

/ip route export
add dst-address=10.1.1.0/24 gateway=192.168.99.41 scope=255 target-scope=10 \
comment="" disabled=no
add dst-address=0.0.0.0/0 gateway=80.76.200.190 scope=255 target-scope=10 \
comment="" disabled=no

Right
=====

/interface ethernet export
set ether1 name="ether1" mtu=1500 mac-address=00:0C:42:12:25:03 arp=enabled \
disable-running-check=yes auto-negotiation=yes full-duplex=yes \
cable-settings=default speed=100Mbps comment="" disabled=no
set ether2 name="ether2" mtu=1500 mac-address=00:0C:42:12:25:04 arp=enabled \
disable-running-check=yes auto-negotiation=yes full-duplex=yes \
cable-settings=default speed=100Mbps comment="" disabled=no
set ether3 name="ether3" mtu=1500 mac-address=00:0C:42:12:25:05 arp=enabled \
disable-running-check=yes auto-negotiation=yes full-duplex=yes \
cable-settings=default speed=100Mbps comment="" disabled=no
set ether4 name="ether4" mtu=1500 mac-address=00:0C:42:12:25:06 arp=enabled \
disable-running-check=yes auto-negotiation=yes full-duplex=yes \
cable-settings=default speed=100Mbps comment="" disabled=no
set ether5 name="ether5" mtu=1500 mac-address=00:0C:42:12:25:07 arp=enabled \
disable-running-check=yes auto-negotiation=yes full-duplex=yes \
cable-settings=default speed=100Mbps comment="" disabled=no

/ interface eoip export
add name="eoip1" mtu=1500 mac-address=00:00:5E:80:00:10 arp=enabled \
remote-address=80.76.200.130 tunnel-id=1 comment="" disabled=no
add name="eoip2" mtu=1500 mac-address=00:00:5E:80:00:20 arp=enabled \
remote-address=82.69.183.210 tunnel-id=2 comment="" disabled=no

/ interface ipip
add name="ipip1" mtu=1480 local-address=217.155.236.210 \
remote-address=82.69.183.210 comment="" disabled=no
add name="ipip2" mtu=1480 local-address=80.76.193.235 \
remote-address=80.76.200.130 comment="" disabled=no

/ ip address export
add address=10.1.1.3/24 network=10.1.1.0 broadcast=10.1.1.255 interface=ether1 \
comment="" disabled=no
add address=62.3.96.195/27 network=62.3.96.192 broadcast=62.3.96.223 \
interface=ether2 comment="" disabled=no
add address=217.155.236.210/29 network=217.155.236.208 \
broadcast=217.155.236.215 interface=ether3 comment="" disabled=no
add address=80.76.193.235/29 network=80.76.193.232 broadcast=80.76.193.239 \
interface=ether4 comment="" disabled=no
add address=192.168.99.37/30 network=192.168.99.36 broadcast=192.168.99.39 \
interface=ipip1 comment="" disabled=no
add address=192.168.99.41/30 network=192.168.99.40 broadcast=192.168.99.43 \
interface=ipip2 comment="" disabled=no
add address=192.168.99.33/30 network=192.168.99.32 broadcast=192.168.99.35 \
interface=eoip1 comment="" disabled=no
add address=192.168.99.29/30 network=192.168.99.28 broadcast=192.168.99.31 \
interface=eoip2 comment="" disabled=no

/ ip route
add dst-address=192.168.0.0/24 gateway=192.168.99.42 scope=255 target-scope=10 \
comment="" disabled=no
add dst-address=0.0.0.0/0 gateway=217.155.236.214 scope=255 target-scope=10 \
routing-mark=zen2 comment="" disabled=no
add dst-address=0.0.0.0/0 gateway=62.3.96.222 scope=255 target-scope=10 \
routing-mark=zen1 comment="" disabled=no
add dst-address=0.0.0.0/0 gateway=80.76.193.238 scope=255 target-scope=10 \
routing-mark=bytel1 comment="" disabled=no
add dst-address=0.0.0.0/0 gateway=80.76.193.238 scope=255 target-scope=10 \
comment="" disabled=no
/ ip route rule
add routing-mark=bytel1 action=lookup table=bytel1 comment="" disabled=no
add routing-mark=zen2 action=lookup table=zen2 comment="" disabled=no

/ ip firewall mangle
add chain=output src-address=80.76.193.235 dst-address=!82.69.183.209 \
action=log log-prefix="PREROUTE" comment="" disabled=yes
add chain=output src-address=80.76.193.235 action=mark-routing \
new-routing-mark=bytel1 passthrough=yes comment="" disabled=no
add chain=output packet-mark=bytel1 action=mark-routing \
new-routing-mark=bytel1 passthrough=yes comment="" disabled=yes
add chain=output src-address=62.3.96.195 action=mark-routing \
new-routing-mark=zen1 passthrough=yes comment="" disabled=no
add chain=output src-address=217.155.236.210 action=log log-prefix="PKT ZEN2" \
comment="" disabled=no
add chain=output src-address=217.155.236.210 action=mark-routing \
new-routing-mark=zen2 passthrough=yes comment="" disabled=no
add chain=output routing-mark=zen2 action=log log-prefix="ZEN2 ROUTE MARK" \
comment="" disabled=no

Let me step thru a few things that might help.

Use L2TP tunnels as the base for the rest of the setup. This is standard udp/1701 and easy to traverse thru firewalls.

3 wan router (cable) -> 1 wan router (data center)

I'm going to say 'cable' for the above router just for ease of explaining. Pretend that router has 3 cable internet connections. The 'data center' side will have 1 wan connection.

The data center router needs X ip addresses - the same number of tunnels you plan to bond. I'll use 3 in this example.

1. Assign data center router 3 _new_ ip addresses that are routeable. For this example we'll use fake ips 172.16.30.1, 172.16.30.2, and 172.16.30.3. Assign all 3 of these to a loopback adapter if you choose. I always setup an empty bridge interface and call it loopback.

2. Add dst-nat rules for destination-ip=172.16.30.1, protocol=udp, port 1701, action 'redirect'.

3. Add dst-nat rules for destination-ip=172.16.30.2, protocol=udp, port 1701, action 'redirect'.

4. Add dst-nat rules for destination-ip=172.16.30.3, protocol=udp, port 1701, action 'redirect'.

Items 2 - 4 establish an entry in the connection table and now will reply with the correct src-ip. Without this crucial step you could try to source route l2tp but it seems to still flip flop because of udp timeouts. This piece here solved most of our pain. Crucial piece.

5. The cable router now has 3 different ip addresses it can make connections to. Cable wan 1 goes to .1, wan 2 goes to .2, wan 3 goes to .3.

6. Enter static routes on the cable router for each of the 3 remote l2tp tunnel endpoints. . . use whatever gateway IP you want each tunnel to go out. 172.16.30.1 -> cable gateway 1, 172.168.30.2 -> cable gateway 2, etc. . .

8. Create 3 l2tp tunnels all going to the data center box across different wan connections. Each of your 2 l2tp-clients will have a different endpoint ip address. Test connectivity at this point first, don't bother adding eoip until you know 3 tunnels are stable. Assign /30 networks to each l2tp tunnel. For instance, 10.0.0.1 & 10.0.0.2, 10.0.0.5 & 10.0.0.6, 10.0.0.9 & 10.0.0.10. Make sure to use correct /30s.

9. Layer EoIP tunnels on top of l2tp using the /30 ips you assign to each l2tp tunnel. Assign each tunnel it's own ID and make sure you don't use duplicate macs. You now can put GRE thru the tunnels without worrying about someone killing GRE somewhere in the middle.

10. Now create a bonding interface and assign an IP on top of that to route things thru.

Sam

Thanks for the reply.

The trick with the dst-nat is smart! The redirect to map the traffic to a local interface is a nice work around to get the tunnel endpoint onto separate IP addresses. I havent fully analysed the packet flow to see how and where this works!

The l2tp tunnel is a good idea and I agree the UDP is much easier to firewall pass. I have in the past used the trick of UDP "pings" to hold a connection track in the NAT router - sort of NAT knocking. The single UDP port also does work well for running a tunnel through a NAT connection - a nice feature! I have long wanted to write a script in a router that would curl a login to a wifi hotspot login page then tunnel a connection through. A project for another day!

I will assign the rb at data centre 3 IP addresses and give this a try. I assume when I assign this to the transit interface the rb will arp for all 3 ip addresses.

Have you tried this on a 3 line end point to end point private link? I dont tend to do this as often as I once did as I prefer using a head-end router to relay traffic through and thus create a star topology rather than mesh - it makes tail to tail routing easier when site number increases.

Only issue is it seems wasteful for IP addresses - especially as tunnel count rises. At least multiple tail end connections can share same 3 IP addresses at head-end. I am also thinking that as I control the head-end and tail-end I could route private addresses over DSL network and thus reduce public IP usage.

BTW I have studied the bonding EoIP tunnel setup in manual. The doc shows a 2 ISP link with 2 ISP at each site. I simply cannot see how this works or could work. I dont see how the connection is locked to the relevant WAN address.

Thanks for all the help on this; it is very welcome and much appreciated! I will come back with the results as soon as I have tested it and let you know how it works out. Will also document it in due course.

Using multiple ip at the head-end and static routes at the tail end worked fine. Traffic was routed on relevant connection and it all worked well.

I havent yet tried the l2tp option but have it running with EoIP over IPIP and also with EoIP directly. Bonding with 4 lines working well and traffic tests look good.

I will try the l2tp and the dst-nat redirect trick and let you know how it gets on. The need for l2tp may not be relevant in most cases as we have control over DSL connections "end to end" and dont need to encrypt. I am wondering if the impact for l2tp and encryption will reduce performance or scale on head-end.

I am planning to use one head-end server for multiple tail-end connections. DO you have any thoughts on the scale and number of tunnels that can be maintained? I will firewall at tail-end and optionally upstream from the bonding server to reduce complexity and performance impact on bonding server. I am wondering if I should opt for a higher speed head-end with more memory e.g. xeon with 2gb and server NIC with processing capability.

Finally, on the Linux system I use I tend to use equalise at the head and teql at the tail and use some traffic control to make sure the buffers in the DSL routers remain empty and queueing is managed at tail-end so we can implement QoS such as VoIP priority. I havent yet tried queue on RB but am planning to restrict upload on each DSL to 95% of maximum and overall bonded tunnel set at around 85% to 90% to allow for tunnel wrap overhead.

Will report back on results.

Tucker -
I was following this thread w/interest...I had hoped in the end you would show a dump of your configuration so the rest of us could use the 'wheel' instead of re-inventing it, (the 'wheel'). :-)

Any way - would you post your config - minus any real IPs for security's sake - for the rest of us to analyze. I have used EOIP, EoIP over IPIP and L2pt/gre tunnels a few times and can never seem to get certain things to work consistantly - perhaps what you have is the answer for us all.

Thanks for your time.

Thom

Sorry for delayed response.

I would be happy to post my config but I have never yet got it to the point where I am totally happy.

At present I still have the multiple gateway problem. Basically I cannot find a way to reliably make sure a packet egresses by the interface on which it ingresses. So if I have 3 WAN connections with WAN1 as default gateway traffic entering the router via WAN2 or WAN3 will egress via WAN1. Obviously for load balance this is not ideal. I have got around this by assigning multiple IPs to each end of the connection and routing by destination address which works well.

However I think there must be an easier way. I have been contemplating using a connection mark on pre-routing to mark the inbound connection then hopefully using this to track the packet and use a routing mark in output chain to make sure the packet egresses on the correct interface. I have not yet tested this.

This does leave one problem. There seems no obvious way to assign an EoIP tunnel to egress via a local interface and the choice will be based on arbitrary default gateway. This then leaves me wondering which way the traffic will egress.

I then thought I could get around this using ipip as it does have concept of defining the source and destination addresses and thus hopefully allowing me to assign the traffic to an interface. However I cannot see any way to get the ROS to adhere to a routing mark for a locally generated packet and again all traffic leaves by the default gateway.

I cannot help but think it would be much easier if ROS allowed the concept tof source based routing as Linux supports. Of course my lack of solving this may be due to less than expert knowledge of the inner workings of ROS.

Any ideas?

mark connections coming in secondary WANS... then mark those packets, then mark routing:

add chain=prerouting action=mark-connection new-connection-mark=in-pip-conn \
passthrough=yes in-interface=l2tp-pip comment="" disabled=yes

add chain=prerouting action=mark-packet new-packet-mark=in-pip-packet \
passthrough=yes connection-mark=in-pip-conn comment="" disabled=yes

add chain=prerouting action=mark-routing new-routing-mark=out-pip \
passthrough=yes packet-mark=in-pip-packet comment="" disabled=yes

then use a routing table called out-pip (just what I used for PortableIP tunnel I have) and you should be good. Create this same set for the other interface if you have more than 2 wans.

Sam

That seems to make sense. Just so I am sure, are you suggesting that the connections are marked in the pre-routing stage according to the interface they originate from. Then route the connections or packets in routing stage according to the mark they have had assigned during pre-routing. I can see how that would work well and I could then add routes that route packets to the preferred WAN connection according to the route mark.

I have had my head around the routing and marking of packets that come into the router from external but I have struggled to see how RB allows selection of packets based upon the source interface for purposes of routing. If I understand the connection marking you are suggesting that in pre-routing the in-interface will be associated with the originator of the tunnel and this can be used to select which WAN to route the traffic on to.

I have a recollection of trying to route by source before and finding out that RB did not offer source based routing only destination based routing. Looking at the router flow I would guess that at the pre-route stage we know the interface the packet or connection originated form but do not yet know nor can select the destination. By marking the packet/connection at this stage we can pass the information on source from the pre-routing to t he routing stage and thus implement source based routing. Does this work on RB?

I am more familiar with Linux and I have been used to the interface offering a hint or explicit selection of gateway based upon source IP address. This connection/packet marking seems to offer similar function and if it works I prefer the idea of having complete control over the routing.

This seems to have cropped up a few times. If I can get a conclusive answer and solution to this I will document the settings and provide the information for everyones benefit.

Thanks
TJ

TJ / Sam -

This is my config for the moment. I have a couple of "non-production' servers behind it. So far everything works right, of course the key point here is non-production.

There is a corresponding bonding router at the other end with similiar setup - the other half of the equation if you will. The only major difference is that it has 5 IP addresses (actually 7 including the bonding address and it's gateway to the Internet) on a single NIC. It does have the same type of routing/rules as this config - just the other end.

I also point this bonded router to my main router which for the most part handles all the nat'ing, firewalling, etc. I don't know if this helps or hurts you TJ - it works for me...

I plan to put it in production in a few weeks if I don't find anything wrong with it....

Thom


CONFIG************************

[XXXX@Rapidwifi BII] interface> print
Flags: X - disabled, D - dynamic, R - running
# NAME TYPE RX-RATE TX-RATE MTU
0 R Local_lan ether 0 0 1500
1 R ether1 ether 0 0 1500
2 R ether2 ether 0 0 1500
3 R ether3 ether 0 0 1500
4 R ether4 ether 0 0 1500
5 R ether5 ether 0 0 1500
6 X ether6 ether 0 0 1500
7 X ether7 ether 0 0 1500
8 X ether8 ether 0 0 1500
9 R eoip-tunnel6 eoip-tunnel 0 0 1500
10 R eoip-tunnel7 eoip-tunnel 0 0 1500
11 R bonding2 bonding 0 0 1500
12 R eoip-tunnel8 eoip-tunnel 0 0 1500
13 R eoip-tunnel9 eoip-tunnel 0 0 1500
14 R eoip-tunnel10 eoip-tunnel 0 0 1500




[XXXX@Rapidwifi BII] ip address> print
Flags: X - disabled, I - invalid, D - dynamic
# ADDRESS NETWORK BROADCAST INTERFACE
0 66.114.148.14/30 66.114.148.12 66.114.148.15 ether1
1 66.114.148.18/30 66.114.148.16 66.114.148.19 ether2
2 66.114.148.26/30 66.114.148.24 66.114.148.27 ether4
3 66.114.148.22/30 66.114.148.20 66.114.148.23 ether3
4 66.114.148.30/30 66.114.148.28 66.114.148.31 ether5
5 ;;; To main router
172.25.2.1/29 172.25.2.0 172.25.2.7 Local_lan



[XXXX@Rapidwifi BII] ip firewall nat> print
Flags: X - disabled, I - invalid, D - dynamic
0 chain=dstnat action=redirect to-ports=0-65535 dst-address=66.114.148.14 dst-port=1701 protocol=udp

1 chain=dstnat action=redirect to-ports=0-65335 dst-address=66.114.148.18 dst-port=1701 protocol=udp

2 chain=dstnat action=redirect to-ports=0-65335 dst-address=66.114.148.22 dst-port=1701 protocol=udp

3 chain=dstnat action=redirect to-ports=0-65335 dst-address=66.114.148.26 dst-port=1701 protocol=udp

4 chain=dstnat action=redirect to-ports=0-65335 dst-address=66.114.148.30 dst-port=1701 protocol=udp

5 chain=dstnat action=dst-nat to-addresses=172.25.1.2 to-ports=0-65535 dst-address=207.115.82.153

6 chain=srcnat action=src-nat to-addresses=207.115.82.153 to-ports=0-65535 out-interface=bonding2

7 chain=dstnat action=redirect to-ports=0-65335 dst-address=172.25.1.2 dst-port=1701 protocol=udp

8 chain=dstnat action=redirect to-ports=0-65335 dst-address=172.25.2.1 dst-port=1701 protocol=udp



[XXXX@Rapidwifi BII] ip firewall mangle> print
Flags: X - disabled, I - invalid, D - dynamic
0 chain=prerouting action=mark-connection new-connection-mark=in-wan1 passthrough=yes in-interface=ether1

1 chain=prerouting action=mark-packet new-packet-mark=in-wan1 passthrough=yes connection-mark=in-wan1

2 chain=prerouting action=mark-routing new-routing-mark=out-wan1 passthrough=yes packet-mark=in-wan1

3 chain=prerouting action=mark-connection new-connection-mark=in-wan2 passthrough=yes in-interface=ether2

4 chain=prerouting action=mark-packet new-packet-mark=in-wan2 passthrough=yes connection-mark=in-wan2

5 chain=prerouting action=mark-routing new-routing-mark=out-wan2 passthrough=yes packet-mark=in-wan2

6 chain=prerouting action=mark-connection new-connection-mark=in-wan3 passthrough=yes in-interface=ether3

7 chain=prerouting action=mark-packet new-packet-mark=in-wan3 passthrough=yes connection-mark=in-wan3

8 chain=prerouting action=mark-routing new-routing-mark=out-wan3 passthrough=yes packet-mark=in-wan3

9 chain=prerouting action=mark-connection new-connection-mark=in-wan4 passthrough=yes in-interface=ether4

10 chain=prerouting action=mark-packet new-packet-mark=in-wan4 passthrough=yes connection-mark=in-wan4

11 chain=prerouting action=mark-routing new-routing-mark=out-wan4 passthrough=yes packet-mark=in-wan4

12 chain=prerouting action=mark-connection new-connection-mark=in-wan5 passthrough=yes in-interface=ether5

13 chain=prerouting action=mark-packet new-packet-mark=in-wan5 passthrough=yes connection-mark=in-wan5

14 chain=prerouting action=mark-routing new-routing-mark=out-wan5 passthrough=yes packet-mark=in-wan5




[XXXX@Rapidwifi BII] ip route> print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf
# DST-ADDRESS PREF-SRC G GATEWAY DISTANCE INTERFACE
0 A S 66.114.148.2/32 66.114.148.14 r 66.114.148.13 ether1
1 A S 66.114.148.3/32 66.114.148.18 r 66.114.148.17 ether2
2 A S 66.114.148.4/32 66.114.148.22 r 66.114.148.21 ether3
3 A S 66.114.148.5/32 66.114.148.26 r 66.114.148.25 ether4
4 A S 66.114.148.6/32 66.114.148.30 r 66.114.148.29 ether5
5 ADC 66.114.148.12/30 66.114.148.14 ether1
6 ADC 66.114.148.16/30 66.114.148.18 ether2
7 ADC 66.114.148.20/30 66.114.148.22 ether3
8 ADC 66.114.148.24/30 66.114.148.26 ether4
9 ADC 66.114.148.28/30 66.114.148.30 ether5
10 ADC 172.25.1.0/30 172.25.1.2 bonding2
11 ADC 172.25.2.0/29 172.25.2.1 Local_lan
12 X S 207.115.65.160/27 r 172.25.2.2 Local_lan
13 X S 207.115.80.0/27 r 172.25.2.2 Local_lan
14 A S 207.115.82.152/29 r 172.25.2.2 Local_lan
15 A S 0.0.0.0/0 r 172.25.1.1 bonding2
16 A S 0.0.0.0/0 66.114.148.14 r 66.114.148.13 ether1
17 A S 0.0.0.0/0 66.114.148.18 r 66.114.148.17 ether2
18 A S 0.0.0.0/0 66.114.148.22 r 66.114.148.21 ether3
19 A S 0.0.0.0/0 66.114.148.26 r 66.114.148.25 ether4
20 A S 0.0.0.0/0 66.114.148.30 r 66.114.148.29 ether5


[XXXX@Rapidwifi BII] ip route rule> print
Flags: X - disabled, I - inactive
0 routing-mark=out-wan1 interface=ether1 action=lookup table=out-wan1

1 routing-mark=out-wan2 interface=ether2 action=lookup table=out-wan2

2 routing-mark=out-wan3 interface=ether3 action=lookup table=out-wan3

3 routing-mark=out-wan4 interface=ether4 action=lookup table=out-wan4

4 routing-mark=out-wan5 interface=ether5 action=lookup table=out-wan5

5 src-address=0.0.0.0/0 dst-address=0.0.0.0/0 action=lookup table=main

Thanks for the config info. This is very similar to what I am doing and works very well. The only issue is that it becomes very wasteful on IP addresses at the head-end. It also means the routes become quite difficult to manage - especially if as I have you have multiple remote routers connected to a single head-end router. I am still trying to find a way to make this happen.

It would be great if we could select egress interface based upon tunnel number I have been playing with a config in Linux that uses OpenVPN between endpoints and I run each tunnel in the bonded set on a different UDP port. That means a simple route based upon UDP port is all that is required and it works great. However I dont see any VPN on RB that can be carried over UDP or TCP with selectable ports.

When I get a final config I am happy with I will document this ...

what.......

It would be great if we could select egress interface based upon tunnel number

I've asked for this many times in the past... being able to bind a tunnel endpoint to an IP is really needed here. IPSec has a src address that you can tell it what to use, but none of the other tunnel types.