Have we done something stupid? large subnet problem...

RouterOS General
1

Hi All,

Firstly - we’re wondering if we’ve overlooked something obvious…

Our problem is a large-ish subnet… 10.0.96.0/21.
Wireless users in one area are assigned 10.0.100.0/24 by Radius PPPoE, and users in another are assigned 10.0.101.0/24.

Gateway & DNS are on 10.0.100.254, Radius is on 10.0.100.1.

Now 101.0/21 users can initially contact Radius, DNS & gateway, but after a while (hours, days maybe) they can no longer access the 100.0/24 portion of the subnet.

Mikrotik towers are in the same range as users in areas, all bridged with one IP on the bridge of each MT - DNS & gateway on all ranges in subnet as above. (/21). There’s no problem between MT routers on different ranges in the subnet, and anything connected directly to the bridge - just PPPoE addresses…

We’ve had to move all 101.0/24 users to the 100.0/24 range until we figure out the solution…

Is it something dumb?

:unamused:

2

Hi guys,

just wondering if anyone can point out our stupid mistake, or suggest an alternate configuration?

Thanking you in anticipation…

3

Its very few information.
E.g. first IP-subnets are mentioned, then things about bridging…

How are the subnets interconnected?
What does “no longer access” mean. In networks there are layers and protocols and tools to test layer related connectivity like torch, ping, traceroute, etc.

Large bridged networks tend to have problems with loops, therefore STP is a subject, large IP-networks should use routing-protocols like OSPF.

4

Thank you for your speedy reply!

Right to clarify - the network topolgy is a star configuration, so there are no possible loops… no branches have an alternate path to follow at this stage.

Right - now to try and give more info:

The entire network (routers) are assigned IP addresses on the bridge (all backbone interfaces and WDS on wireless are assigned to the bridge, AP’s are not) in the range of 10.0.96.1 - 10.0.103.254, with a subnet mask of /21.

We have two main areas, where routers are assigned 10.0.101.xxx and 10.0.100.xxx

We use RADIUS authentication over PPPoE to connect clients through the AP’s, which have no IP assigned to them (not bridged).

The gateway address is 10.0.100.254, and the RADIUS address is 10.0.100.1

If we assign addresses on the RADIUS manager in the 10.0.100.xxx range, all is well, but if we assign 10.0.101.xxx ip’s then they can connect initally, but after disconnecting and trying again, they either cannot access the RADIUS, or they can, but have no Internet connectivity.

Now, if I connect via PPPoE, being assigned a 10.0.100.xxx IP, then I sometimes can access the 10.0.101.xxx routers, and after some time (short time) I can no longer access these routers, and vice-versa.

Someimes, if I clear the ARP table on the router I am connecting to, then I can access these IP’s, but doing this does not seem to be consistant.

Examples of the config:

/ interface bridge
add name=“br0” arp=proxy-arp disabled=no
/ interface bridge port
add interface=ether1 bridge=br0
add interface=wlan1 bridge=br0
/
/ ip address
add address=10.0.101.53/21
/
/ ip dns
set primary-dns=10.0.100.254 secondary-dns=0.0.0.0 allow-remote-requests=yes
/
/ ip route
add dst-address=0.0.0.0/0 gateway=10.0.100.254
disabled=no
/
/ ppp profile
set default local-address=10.0.101.53 remote-address=0.0.0.0
use-compression=default use-vj-compression=default use-encryption=yes
only-one=no change-tcp-mss=yes dns-server=10.0.100.254
/
/ radius
add service=ppp,wireless address=10.0.100.1
secret=“secret” disabled=no
/

Thanks for your time!

5

So a bridged network design is used, where the routers have only management IP-addresses and WDS for bridging the wireless backbone?
Where are the PPPoE-Servers located, on each particular access-point?
And is there a central router, connecting the star together, being also def. gateway with IP-address in the /21-subnet?

If i’am wrong, could you draw one path from client to internet with every hop?

if connectivity is lost, is it an IP- or a MAC-problem, i.e. ping is not possible or ARP?

Why is proxy-arp enabled in the config above?

6

more information for you:

Yes, the network design is as you say - Backbone wireless links run in ‘bridge’ mode with dynamic WDS, and Dynamic WDS default bridge set to br0.
Points in the star have further router nodes along the way, and some split further again.

set wlan1 name=“wlan1-link” mode=bridge ssid=“link-pp” frequency=5320 band=5ghz
periodic-calibration=enabled wds-mode=dynamic wds-default-bridge=br0 disabled=no
set wlan2 name=“wlan2-omni” mode=ap-bridge ssid=“omni” frequency=2412 band=2.4ghz-b periodic-calibration=enabled disabled=no default-forwarding=no

PPPoE servers are on access points at each router site:

/ interface pppoe-server server
add service-name=“service” interface=wlan2 max-mtu=1480 max-mru=1480
authentication=pap,chap,mschap1,mschap2 one-session-per-host=no disabled=no
/
/ ppp aaa
set use-radius=yes accounting=yes
/

yes, there is a central router which acts as the main firewall and traffic QoS shaper, and behind that a DNS & gateway server - both on the /21 subnet.

“If i’am wrong, could you draw one path from client to internet with every hop?”
Not sure what you mean? If you are asking if there is only one possible path fom client to internet - then yes, there is only one possible path.

“if connectivity is lost, is it an IP- or a MAC-problem, i.e. ping is not possible or ARP”
IP - ping is not possible from the PPPoE client. All is fine between routers.

“Why is proxy-arp enabled in the config above?”
The reason for using proxy-arp is firstly as suggesed in the manual:

Proxy-ARP feature
“Use addresses from different networks on different interfaces, or enable proxy-arp on ether 1 or ether 2”

“This behaviour can be usefull, for example, if you want to assign dial-in (ppp, pppoe, pptp) clients
IP addresses from the same address space as used on the connected LAN.”

And secondly things don’t seem to work without it…

Hope this helps, and once again - thanks for your time!

7

Ok, so basically you have a wireless backbone with a star topology. Each edge node connecting
to the wireless backbone acts as a PPPoE server towards its clients, but your IP addressing structure
has no subdivisions at all. Not only is your wireless backbone all inside one big IP network,
but even the client-facing interfaces of the edge nodes, which act as PPPoE servers, are handing
out IP addresses that also fall into the same all-encompassing IP network. That’s why you need proxy-arp,
because once a PPPoE client connects and is assigned an IP address suddenly that address springs into
existance on the “wrong” side of the edge node.

Is that how your network is constructed? If so, to be honest, I’d scrap the whole thing and start from zero,
because that design would IMHO suck really hard. But that’s only my opinion.

In fact, from the IP addressing point of view, and with all that proxy-arp going on all over the place, your
network behaves so very Layer-2-like that you might just as well go for it and drop all the PPPoE servers on all the
edge nodes, transparently bridge the client facing interface towards the WDS / backhaul, and then just use
one big PPPoE server at the border of your network. That would be a little bit cleaner, though still IMHO not
a very good design for a larger network.

Now, for your original problem, I’d recommend that you carfully double-check all netmasks on all devices,
especially on the interfaces that are configured for proxy-arp. A misconfigured netmask in combination with
proxy-arp can lead to irritating behaviour that can be “interesting” to debug.


–Tom

8

hmmm… suddenly things seem a little clearer… I have double-checked netmasks previously, as that’s what I thought was causing it - found an error, checked again with no change - but I will go and check again…

So you’re saying we should be handing out IP’s in a different range on PPPoE and remove the proxy-arp?

Is that how your network is constructed? If so, to be honest, I’d scrap the whole thing and start from zero,
because that design would IMHO suck really hard. But that’s only my opinion.

I’m interested in your opinion - how would you change this configuration? At the moment things are working quite well, but if there’s room for improvement, I’m all ears!

9

I would suggest also a redesign.

One thing is using one central PPPoE-Server directly connected to the internet-gateway. Then customer-IP-network could be completely separated from the transport/backbone IP-network.
Also i found it easyier to use a routed backbone than a bridged one, but this needs using EoIP-tunnels instead of WDS.

Of course it should be possible to go on using the bridged network, but its more complex and more difficult in troubleshooting, distributed EoIP-servers may be an advantage.

10

Yes, that would be better.

I agree with mag and suggest you switch to a routed (Layer 3) backbone connecting all edge nodes.
Then you’d assign IP address ranges (subnets) for the PPPoE servers of each edge node, with each
and every edge node having its own, disjunct IP subnet from which addresses are handed out for
connecting PPPoE clients.
If you have edge nodes to which further edge nodes are connected, i.e. a chain of edge nodes that hangs
off the backbone, then choose the PPPoE / client address pools of all nodes in one chain to be blockwise
continous, then you’ll be able to aggregate the address block towards the backbone into one larger prefix.


–Tom

11

Thanks a lot for your suggestions guys - Fortunately we’re not too far down the road to make changes in the routing method…

I’ll be back with results of our current problem, and will discuss the re-design with my esteemed colleague and mentor… see if this will fit in with his big picture.

Thanks again - I’ll be back in a day or two…

12

I discussed this very briefly with my colleague - and it seems he’s not yet convinced by my description…

Found this in a Nortel equipment fact sheet:

Why is Nortel using a Layer 3 approach as opposed to a Layer 2 bridging approach?
Nortel decided to use the layer 3 approach for WMN for the following reasons:
• Nortel wanted to build a highly resilient, self healing, auto configuring solution – L3 protocols are better suited to this type of network
Layer 2 solutions involve flooding onto the network (just the way that bridges operate), as well as re-computation of the Spanning Tree Protocol which increases the overhead in the network and increases the recovery time when failures occur.

So I’m guessing it be for these reasons you advocate layer-3?

I’ll be back soon…

ps: we haven’t yet got to the bottom of our subnet problem… we are looking hard for the problem!

13

Yes, what you quoted from Nortel are some of the typical reasons why layer 3 topologies are usually better suited for more complex or growing networks. Also, I find routed networks with clearly seperated address ranges easier to debug - you can always do a traceroute etc. towards a particular address and you’ll always clearly know which way the packets travel through the network.

But from what we’ve discussed earlier you design problems are only partially layer 2 vs. layer 3; it’s more in the way your addressing is set up and the requirement for proxy-arp and potential for confusion that results from your design…


–Tom