Yours and mine gateways.

RouterOS Beginner Basics
1

We are a non-profit community-owned wireless co-operative boldly bringing broadband where anyone with any sense would fear to tread.

We’re having to expand our bandwidth to meet demand and are installing a second adsl modem at our gateway in a distant city - and have a problem that defeats my notes from wireless networking 101.

We have an RB433 with two wireless interfaces (wlan1 and wlan2) and three ethernet interfaces. Ether1 is connected to modem 1, ether 2 to modem 2 and ether three to a Radius server.

We need to route all wlan1’s default (ie Internet traffic) to modem 1 and all wlan2’s default traffic to modem 2 but still enable intranet traffic between wlans 1 and 2, and route RADIUS packets etc from both wlans to ether 3.

It isn’t possible to create tunnel interfaces on the modems LAN sides but I can’t see how within the RB433 and RouterOS (V3.10) the two default data streams can be kept apart.

I’m sure it can be done (Mangle?) but the how of it is way outside my expertise.

Anyone willing to help?

2

tombrdfrd66 -

Sure - but basically what you are asking for is policy based routing - it’s in the wiki…

If the wiki doesn’t do it for you then paste your config up here and any other relative ‘data’ and we’ll see what we can do…

By the way - how did that link across the water turn out?


R/

Thom

3

'Fraid the wiki didn’t do it for me, but then nothing much does these days. Being over 40 sux. I can see the potential is there, but can’t see how the wiki examples apply to my case.

It should be relatively simple. Everything 0.0.0.0/0 from community A’s AP (10.1.0.2/24) arriving at their wlan at the gateway (wlan 1, 10.1.0.1/24) gets routed through the RB433 out of ether 1 (172.16.1.2/24) to ‘their’ modem 172.16.1.1/24 while everything 0.0.0.0/0 from community B’s AP arriving at their wlan at the gateway (wlan 2, 10.0.0.1/24) gets routed through the RB433 out of ether 2 (172.16.0.2/24) to their modem at 172.16.0.1/24. Routing between the networks is also required (10.1.0.0/16 <> 10.0.0.0/16) and a route to the RADIUS server (172.16.3.1/24) on ether 3 (172.16.3.2/24) from both nets is the only other requirement.

The IP routing table goes into a blue funk if you give it two default routes.

It’s still all in the air - I just haven’t had a chance to get on to it. Wife wants the house painted and there’s a log-shed to fill. Running a wireless network is just a night-time hobby at the moment!

4

you need something like described here:
http://wiki.mikrotik.com/wiki/Load_Balancing_over_Multiple_Gateways

mark connections from those locations you want to control, and then add routes based on those marks.

5

Thanks Normis, that looks to be along the lines of what we want tho’ I’ll need to study the Wiki entry closely to actually make sense of it, and will probably have to take my socks off so that I can count on my toes as well.

Nice - and sensible - as it would be load balancing in any meaningful sense has been ruled out by the decision of the communities concerned to each run their own ISP connection and plan over their own modem. Where is Joe Stalin when you need him!

6

OK, supplemental question required.

The router through which we wish to keep the two Internet connections apart (Group A to modem A, Group B to modem B) could also be the pppoe server interface for both groups. We can have a Group A pppoe service and a Group B pppoe service, but does the routing-mark attached through the inbound interface (wlan1 = Group A, wlan 2 = Group B) survive passage through the pppoe interface to dictate the packet’s route out of the router (to modem A or modem B)?

7

OK, did that and it works - but it works too well. EVERYTHING from Group A is being routed to Group A’s gateway and EVERYTHING from Group B to Group B’s modem. Intranet traffic from Group A to Group B and vice versa is also being routed to the gateways and presumably out into the Internet - not just the ‘default’ stuff.

How do I ONLY route the 0.0.0.0/0 addresses to the separate gateways while routing 10.0.0.0/16, 10.1.0.0/16 and 172.16.n.0/24 traffic locally, in the usual way? Do I have to write a routing rule for every exception?

And what is a ‘route tag’? It pops in the OS (v3.10) but not a mention of it in the manual, or the Wiki that I can find. Is there an explanation anywhere?

8

tombrdfrd66 -

Post your route and route rule tables - let’s see what you have…

/ip route print
/ip route rule print


/R

Thom

9

tombrdfrd66 -

On second thought…

Post your mangle rules for routing.

Maybe this will help you so try it first and if it doesn’t then post everything asked for…

In your mangle rules for routing - where you are marking your routes…

In the magle rule itself, I surmise from the article that you are marking everything for ‘A’ group or ‘B’ group without exception. So what you need to do is create a list of local addresses - in /ip firewall, the address list tab. Here create a list, call it ‘local-addresses’, add your local IPs that you do NOT want routed out to the Internet when one of these is the dst-addr in a request from another local address.

Next, go to you route mangle rules that you created earlier, open them, go to the advanced tab I believe, there you’ll see a box marked dst-address, put the label ‘local-addresses’ in the there. Next to that box is a very small box, click on it - there you will see a ’ ! ’ apprear - this means ‘NOT’. So what we did is if the src-addr matches ‘A’ or ‘B’ and the dst-addr is NOT ‘local-addresses’ then a routing mark will be applied to the connection/packet/whatever. If the dst-addr IS ‘local-addresses’ then the packet/connection will NOT get marked with a routing mark…

OK?

R/
Thom

10

Thanks for that, Thom.

I don’t want to post my routing tables etc. except as a last resort - I know only too well they’re an awful mess and I’m ashamed of them, but tho’ ad hoc and ramshackle - they work.

Your surmise is likely correct. As each ‘group’ arrives on its own interface at the gateway router (Group A = wlan1/pppoe,A // Group B = wlan2/pppoe,B) I set route-marking to pick up everything arriving at the respective wlan, and thought that as the Group mark only appeared on the routing table linked to default routes everything NOT default would be routed without regard to the routing mark. But it seems (tho’ I might be wrong - I still haven’t plumbed the depths of this matter) that a routing mark overrides the rest of the table - ie

a packet arriving at wlan1 addressed to 172.16.1.1 gets a routing mark ‘group A’. The routing table is then consulted and although there is a ‘main’ route to 172.16.1.0/24 this is ignored because it isn’t a ‘Group A’ route. As there isn’t a ‘Group A’ route defined for 172.etc it therefore defaults to 0.0.0.0/0 and sends it off to the ISP.

This isn’t what I would have expected, but seems to be what is happening.

I suspect the solution is to bind the mangle/route-mark to everything arriving on the pppoe interface instead, as this is (supposed to be) Internet bound anyway with only internal traffic using the wlan route. Unfortunately the only way to find out if I’m right is to try it out, but I only have a functioning network to experiment on, and if I break something I plunge the Trust’s subscribers into digital darkness!

11

tombrdfrd66 -

Well not to disagree with you but a more extensible answer to your issue is the one I sugeested above where by you mark the connections as before but use the ’ ! ’ (not) dst-addr list function to make the final decision on whether to route locally (via the main table) or send the request off to the Internet via routing tables ‘A’ and ‘B’…

This has the advantage of not plumeting your network in to the dismal digital abyss if you guess wrong…

It is also easily extenable by simply adding ‘local’ addresses to your ‘local-addresses’ list. You wouldn’t have to change the mangle rules at all. (Except to maybe add a new one as the network grows but local addresses will always be local addresses)…

R/

Thom

12

Thanks for that, Thom. And for your patience.

As what I’m about to do has the potential to create a Gordian Knot of mangled routing which could take hours to unpick and put together again I’d like to make sure I get this right before I start. (And if it’s really bad I’ll have the natives storming my barricades with lit faggots demanding their porn back!)

In the attached pic. the Linksys router at the Gateway is actually an RB433 and the ‘RADIUS’ server is actually an RB532 running User-Manager, which also does DNS and NTP for both Groups and can provide a standby link to either modem.

My thinking is to set up routes at Group A’s modem sending anything for 10.2.0.0/16 or 172.16.0.0/16 over wlan and 0.0.0.0/0 over PPPoE. Ditto Group B save that it would be 10.1.0.0/16 that gets routed over the wlan. At the Gateway router everything arriving by PPPoE is route-marked to the Group’s modem and there is no non-routemarked default gateway.

Your suggestion does away with the need for the PPPoE and the ‘filtering’ at each Group’s AP which is not to be sniffed at, but under our Trust internal traffic within the community is free and unlimited save for priority for Internet traffic, and using the PPPoE link on the backhaul to record the Group’s total Internet-only use will provide a useful check on the ISP’s accounting!

Nevertheless if you can see any hitches, flaws or problems I’d love to know of them before they bite me on the bum.

Have you any idea what a Route-tag is, seeing as Mikrotik aren’t telling anyone?
gateway.jpg

13

tombrdfrd66 -

Well the basics of your plan sound well founded…

Since I don’t know what you PPoE setup is like I can not imagine what you are going to need for mangle rules…you pretty much have everyone ‘captured’ at the PPoE gateway to start with…

The only questions I would have is are the dsl modems capable of masquerading multiple subnets as they go to the Internet? Are they capable of actually routing requests back to their respective gateways in to your system when the request is ‘local’ vice an Internet request?

Not sure what you mean by route-tag? I believe that they are using something like that in conjunction VLAN ID and there was some mention of it in OSPF or BGP along with MPLS and something else they just added to aid in creating ‘mesh’ networks (I can’t remember now, and I could be way off base as well…). Haven’t used it yet, probably will in the future…I try to keep our network out on the ‘bloody’ cutting edge to keep up with the customers…

R/

Thom

14

Hi Guys.

I can’t get it to work.

Here are the relevant parts of the tables. A few names have been changed to protect the innocent.

Group A’s AP. (RB532 with RouterOS v3.10)

[admin@GroupA] > ip address print
Flags: X - disabled, I - invalid, D - dynamic

ADDRESS NETWORK BROADCAST INTERFACE

0 10.1.70.1/24 10.1.70.0 10.1.70.255 wlan3
1 10.10.0.1/24 10.10.0.0 10.10.0.255 ether2
2 10.110.10.1/24 10.110.10.0 10.110.10.255 wlan2
3 10.1.0.94/28 10.1.0.80 10.1.0.95 wlan1 (wlan = local)
4 D 10.100.0.94/32 10.100.0.81 0.0.0.0 Gateway (pppoe = internet)

(Note: Internet-bound traffic is accounted via RADIUS and PPPoE, local traffic is not.)
admin@GroupA] > ip route print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit

DST-ADDRESS PREF-SRC GATEWAY-STATE GATEWAY DISTANCE INTERFACE

0 ADS 0.0.0.0/0 reachable 10.100.0.81 1
1 X S 0.0.0.0/0 10.0.0.81 1

(Note: Other entries in table route local traffic to groupA over wlan. Only default and thus Internet traffic is routed over the pppoe link - in theory.)

[admin@GroupA] > tool traceroute n.n.n.n
ADDRESS STATUS
1 10.100.0.81 3ms 1ms 2ms
2 172.16.1.1 7ms 19ms 2ms
3 ip-n.n.n.n 72ms 61ms 61ms

(Traceroute to Internet address ignores policy routing on Gateway router (to Modem B) and goes via Modem A.)

[admin@GroupA] > tool traceroute 172.16.2.1
ADDRESS STATUS
1 10.100.0.81 4ms 2ms 2ms
2 172.16.2.1 13ms 2ms 2ms

(Yet Modem B is reachable.)



Gateway router (RB433 with RouterOS v3.10)

[admin@Gateway] > ip address print
Flags: X - disabled, I - invalid, D - dynamic

ADDRESS NETWORK BROADCAST INTERFACE

0 172.16.1.4/24 172.16.1.0 172.16.1.255 ether1 (Group B’s modem)
1 172.16.2.2/24 172.16.2.0 172.16.2.255 ether3 (Group A’s modem)
2 10.1.0.81/28 10.1.0.80 10.1.0.95 Group A
3 10.0.0.49/28 10.0.0.48 10.0.0.63 Group B
4 172.16.3.4/24 172.16.3.0 172.16.3.255 ether2
5 D 10.100.0.81/32 10.100.0.94 0.0.0.0
6 D 10.150.0.1/32 10.150.0.50 0.0.0.0

[admin@Gateway] > ip firewall mangle print
Flags: X - disabled, I - invalid, D - dynamic
0 X chain=prerouting action=mark-routing new-routing-mark=GroupB
passthrough=yes in-interface=

1 chain=prerouting action=mark-routing new-routing-mark=GroupA
passthrough=yes in-interface=

[admin@Gateway] /ip route rule> print
Flags: X - disabled, I - inactive
0 src-address=10.100.0.94/32 dst-address=0.0.0.0/0 routing-mark=GroupB
interface=ether3 action=lookup table=GroupB

1 X dst-address=0.0.0.0/0 routing-mark=GroupA interface=ether1 action=lookup
table=GroupA

[admin@Gateway] /ip route> print
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit

DST-ADDRESS PREF-SRC GATEWAY-STATE GATEWAY DISTANCE INTERFACE

0 A S 0.0.0.0/0 reachable 172.16.1.1 1 ether1
1 A S 0.0.0.0/0 reachable 172.16.2.1 1 ether3

(Note: Route 1 via 172.16.2.1. has listed Routing Mark 'GroupA" but console IP route print doesn’t appear to carry Routing Marks???)
If I switch in similar policy routing for Group A, giving Route 0 on Gateway a GroupA routing mark, Group B loses its gateway altogether and can’t get through Gateway.

As far as I can see this exactly follows the example given in the manual. Can anyone see/speculate why it isn’t working?

15

That’s what I would have thought. In fact if there is a way of bridging the PPPoE interfaces to the respective modems it would seem to answer all my problems but I’m told from on High that it doesn’t work like that.

No, they’re not - hence I have to masquerade the ethernet interface attached to the modem.

‘Route Tag’ is an option on the ‘attributes’ tab of the ‘New Route’ window of V3. in Winbox. Sounds as though it might be useful, but like many of the Mikrotik Bells and Whistles it’s probably only available to those in the know and who can swan off to MUMs in strange, exotic and expensive places.

16

Well for anyone following this thread it seems that the problem was that you can’t use policy routing with a pppoe interface. If the pppoe link goes down the mangle configuration is lost and so the route-mark doesn’t attach to packets arriving on the pppoe interface when it comes back up.

However configuring mangle to attach routing marks by source address when the source address is the pppoe-allocated address does seem to work.