I’m coming along with my Road Warrior setup for the hAP. I need help with what I hope will be the last step, which is such a common idea that I’m surprised I haven’t found it already documented somewhere.
Currently, the hAP has:
WAN access with NAT (standard configuration)
Local subnet (standard configuration)
EoIP tunnel to office on a separate private subnet
I have access ports set up so I can connect to the local subnet, and static routes set up between the 2 subnets, which gives me routed access to the WAN (with NAT) and routed access to the office, this is all working fine.
I also have access ports that are directly bridged to the EoIP tunnel. This gives me broadcast/multicast domain access to the office, which is the major goal of the whole setup. However, as you might guess, it means that while connected this way, WAN access is through the office router. Traffic from a web site goes to the office then out the office over the EoIP link to the remote router. Of course this is to be expected, as the Office DHCP server is handing out the office router as the gateway, but it is wasteful of time and bandwidth.
What’s the best way to divert WAN traffic off the EoIP link? The best I can come up with from my limited knowledge would be some sort of IP-based NAT filter on the bridge, but even then I’m not sure how it would work even ignoring the double-NAT issue.
An easy but not totally reliable hack would be to just setup a DHCP server with a small pool on the EoIP device. It will almost certainly answer first. Just an IP to each EoIP bridge and make the gateway for that DHCP server match that IP.
A more complicated solution might involve NAT. I’m thinking you’d have to use bridge rules though.
I am still far from understanding the whole packet flow diagram, so I do not know what is possible or most efficient. Some ideas I had for solutions that I do not know how to implement to test:
Filter/modify/mangle packets to change the MAC address of the office gateway when it goes through the EoIP tunnel. Not so great because this would block access to the office router itself, which would hinder troubleshooting. I guess I could filter on MAC address/IP address combo to get around that, right?
Filter/modify/mangle packets to change the DCHP responses to change the gateway IP address in the response when it goes through the EoIP tunnel
Set up ICMP redirect responses from the office gateway for packets that come in over the tunnel, but I have read ICMP Redirects are Bad.
As I am working through this configuration and getting experience with the current set up, I am starting to see that there are subtle problems no matter which way we go. For example, servers that use GeoIP or have our Office public IP in a whitelist would need to be routed through the EoIP tunnel in any case if we want the road team to have the same experience as being in the office.
Also, I’ve given up on the idea that we can make this work with a single-radio device like mAP because it is too complicated to set up the radios on the road (really going for as close to plug-and-play as possible here), so as long as we have 2 radios, we can have separate APs, one routed to the office and one tunneled, and let people switch to whichever one they need for a device/task. That makes for a rather simple set of instructions: use the routed AP if possible, and if something does not work, try the tunneled AP. This will give us options from which we will gain experience.
Still, I would like to have the third option for the remote team, which is Office LAN on tunnel, WAN routed locally (what I asked for in my original post). So please keep the ideas coming.
I think you have two options, for me the most practical is the first one:
Remove your reliance on the same broadcast domain by implementing DNS based service discovery.
Use bridge nat to change the dst-mac-address of packets egressing the EoIP tunnel based on destination IP matching, hopefully your main network is an easily identifiable subnet. Something like /interface bridge nat add out-interface=eoip1 dst-address=!10.12.0.0/12 to-dst-mac-address=ab:cd:ef:gh:ij:kl
It’s not just Bonjour, it’s that and Dropbox and Chromecast and if I can get the EoIP to work I can eliminate a lot of “why doesn’t this work over VPN” questions.
Use records like: _googlecast._tcp.
I don’t have a chromecast to test it with personally but that would be a start. This would allow you to switch to a routed solution that would handle your other needs far easier.
Thank you for the idea. I have not heard of anyone being successful with that kind of setup. Are you using it?
I see 2 problems with it. First, it completely defeats the purpose of zero configuration by requiring that I find and register DNS records for all the zeroconf devices, which is hard because unlike other manual configuration options, zeroconf devices do not always have manual configuration capabilities, which is why they use zeroconf! Second, it does not completely solve our problems, because mDNS is not the only non-routable protocol running on our local net. (Based on all the issues I’ve seen with mDNS proxying, I’m not even sure it will solve all our mDNS issues. Devices that advertise services on mDNS expect clients to be in the same subnet and broadcast domain.)
It seems to me easier to statically assign IP and Gateway to the road warrior devices based on the WiFi configuration than trying to manually assign DNS records for all the zeroconf devices. Since the configurations are tied to a specific WiFi network (the one provided by our traveling router) I could set up and test all the configurations on the traveling devices while the are in the office. Going the extra step, I could put two APs on the router just to allow devices to switch configurations if for some reason they needed that.
Alternately I could set up a pair of non-caching HTTP proxies on the remote router and just redirect HTTP traffic at layer 3, which will hopefully cover most of the speed issues. (I haven’t checked if anyone streams video over other protocols like RTSP, but I suspect that things that break would break anyway due to the inherent double-NAT in the remote routed network: client → hAP (NAT)-> public WiFi (NAT) → internet.) I wonder if the CPU on the hAP is up to the task of hosting 2 proxies.
This was one of my original thoughts but I did not know how to configure it or if the hAP could even do it. And so far I have not found a way to make this happen. I have the subnets set up to make this easy, but the rule you propose generates the error “outgoing interface matching not possible in input and prerouting chains” despite the fact that I’m putting it in the bridge dstnat chain (by which I mean the error message is bad, not the behavior). Thinking about it, it would not make sense anyway, because the packet would still go out the EoIP interface, just now with a bad MAC address.
Next I tried replacing “out-interface” with “dst-mac” and specifying the MAC of the office router, specifying to-dst-mac-address to be the remote router EoIP bridge. That did not work* and I’m not sure why. If someone can explain this, please do. I pored over the Packet Flow diagram for quite a while trying to figure it out. My best guess is that because the destination MAC address is still on the bridge, the bridging decision is not affected by the change in MAC address and the packet is still not routed. It just gets forwarded out he EoIP interface as before.
Just for the sake of investigating, I tried specifying to-dst-mac-address to be the remote router LAN bridge. That did not work*, which makes sense because there is no layer 2 link between the office subnet and the remote subnet and if there were, well, they would not be subnets then.
*Note: by “that did not work” I mean that packets from the remote side of the EoIP link, from clients with IP addresses on the office subnet, addressed to public IP addresses, did not make it to their destination. To test, I tried sending UDP packets to a listener on a publicly accessible server and the packets were never received. Of course, something more sophisticated, like making a TCP connection also did not work.
Maybe by matching in-interface in the bridge nat rule to the WLAN port that is added to the bridge (for wireless clients) or better yet a “!eoip” for in-interface. Maybe the packet will get messed with in the correct placement.
I configured DNS-SD myself on my LAN. I did it with a printer personally. I don’t have a chromecast to verify their functionality. When I was researching printing there were notes about the devices being able to register themselves in DNS. The recommended approach was to add a zone for these types of devices and add that special zone to search domains in DHCP. This of course all required the service records to be created either by the device or at least by DHCP which required the necessary permissions to be set on BIND. None of these DNS solutions work with the default resolver in RouterOS.
If someone can explain this, please do. 
I pored over the Packet Flow diagram for quite a while trying to figure it out. My best guess is that because the destination MAC address is still on the bridge, the bridging decision is not affected by the change in MAC address and the packet is still not routed. It just gets forwarded out he EoIP interface as before.