Community discussions

MikroTik App
 
User avatar
bcbigb
just joined
Topic Author
Posts: 20
Joined: Sat Dec 28, 2013 5:48 pm

Interface selection for internal/router-generated traffic -- A solution and request for feedback.

Wed Aug 04, 2021 11:18 am

Hi guys,

NOTE: DNS FWD example at the code breaks in the middle.
NOTE: Quick solution for router-generated traffic issues in the code break at the bottom.

In order to populate this with keywords to help others find this solution, let me give the background: I started this journey after happily seeing the new FWD "record" (rule, really) in DNS in the 6.47 update, wanting to use it to only selectively/conditionally forward queries for our internal network (meaning "company.local" domain) from our remote office to our main office (which has the AD/DNS server) on the other end of an IPSEC tunnel. This was partly so I could avoid the trouble of scripting DNS server changes for either the entire router or just the DHCP service due to the debugging/maintenance/headache associated with them when they failed or hit some odd exception. The IPSEC tunnel is set up almost at defaults and is very likely not relevant here as it passes traffic properly that is directed over it, so I'll skip posting that config for right now.

---

Starting on 6.47.10 LT, I couldn't get the FWD records to work to save my life, despite me being proficient with regexes and double-checking them in a tester, plus combing through it over and over in case I was missing something. I was not tripped up by the manual's terminal-specific example using the extra leading slashes to escape the special characters in the regex, including the slashes regex uses to escape special characters of its own. I was doubly sure I was getting it right as queries that matched the regex entries, regardless of the source of the query (RouterOS or workstation behind it), were not showing up in the DNS cache, meaning they were almost certainly being intercepted properly but failing further down the pipeline I decided to update to 6.48.3 ST branch to see if this changelog entry from 6.48[.0] was relevant: "dns - fixed listening for DNS queries when only dynamic static entries exist (introduced in v6.47)". It didn't seem to be applicable as I still had the default "router.lan // 192.168.88.1" static entry in there and had tested with a static entry of my own previously. The firmware update had no effect on this issue, as expected.

FWD rule for subdomains:
/ip dns static add forward-to=192.168.2.8 regexp=".*\\.company\\.local" ttl=10m type=FWD

FWD rule for root domain:
/ip dns static add forward-to=192.168.2.8 regexp="^company\\.local" ttl=10m type=FWD

I just checked and TTL appears to have no effect, even though it's a required field for a FWD rule. The TTL in the DNS cache is, as is expected and correct, whatever the target DNS server sends back for that specific DNS record. Maybe it's a fallback in case it doesn't receive a TTL in a record? I've heard of that happening in corner-cases and is probably useful to ensure the cache doesn't clog up over time, but... who knows.

Also, I know you could combine them into a single rule making the period optional before the company.local, but hey, I had other problems to solve at the time :-).



Knowing there were inconsistencies when using MT's internal tools (self-generated traffic), particularly ones that had no source interface or source IP selection option, I had been partly testing using a workstation behind the remote office's router, running basic directed and undirected DNS queries via nslookup. The queries directed at the AD/DNS server were working as expected, but not the undirected ones (which implicitly went to the local router I'm trying to use FWD rules on). Suspecting that the traffic wasn't making it out the IPSEC tunnel to the other side (where the AD/DNS server is), I was going to load up Wireshark on the server to confirm definitively if the DNS queries were making it over the tunnel (which almost certainly would have confirmed it, in retrospect), but first I went down the rabbit hole of testing via the router's other tools (Netwatch, Terminal (":put [:resolve a.b.c.d]"), Ping, etc) to trigger DNS queries and see what I could make out. Unsurprisingly, only ping, set to force traffic out the LAN interface "br1" to guarantee it would hit the IPSEC tunnel, worked as expected. This mirrored the issues I had encountered in the past, but I had never had time to dig into my suspicions about the cause.

Being pretty sure that the issue was something related to source interface and/or source IP selection, I decided to try methods of coaxing traffic over the tunnel for internally-generated traffic. I decided that the best option was the routing table, rather than the seemingly over-engineered packet/connection/route-marking schemes others had presented in other threads. I'm not 100% sure if I'm missing something behind the reasoning here, hence why I'm asking for feedback on why this is working, if it will cause other issues, or possibly is just working for me but would not have for others in the past (say, due to requiring a newer RouterOS feature), as it seems almost too simple to have gone unnoticed or untried for this long. As as far as I can see this is a dynamite, simple solution with no side effects at all (or at least none you wouldn't be able to spot should your situation be complex enough to worry about one little route that's seemingly redundant for the IPSEC policy already redirecting the traffic).

----

The solution is to just add a route for the destination subnet (192.168.2.0/24 in my case) that points to the local LAN interface itself, which is just a shim for tools that need the explicitness of the routing table to tell them how to originate their traffic and then IPSEC policy takes over from there as usual. Which LAN/bridge to pick? Presumably any LAN where traffic can properly be picked up by IPSEC routing rules ("policies", as they aren't technically routes as is shown here), meaning there are no filter or NATting rules that could interfere. Unless you've closed the tunnel off with your own rules, there shouldn't be anything impeding the traffic as long as you made the two bypass rules (NAT+RAW) as shown in the IPSEC manual entry (also, see the IPSEC sub-chart and example chart on the Packet Flow page in the manual to orient yourself further). I think whether you enter a preferred source IP in the rule isn't important unless you have multiple IPs on that interface that are meaningfully different (i.e. if which IP RouterOS picks is important in your situation), but I don't have that issue, so I went with the cautious approach here rather than possibly adding unnecessary complexity.

So, for my remote router, configured for wide-open access between the subnets via the IPSEC tunnel from 192.168.5.0/24 to 192.168.2.0/24, I've added this route:
/ip route add distance=1 dst-address=192.168.2.0/24 gateway=br1

So far it has tested just fine, working for all DNS-query-generating methods listed above, and honestly seems like it should be installed by me on every IPSEC tunnel to avoid this inconsistent behavior. I'd say it should be added automatically by RouterOS, but unless this is a slam-dunk and always a good thing, it's against MT's built-it-from-basics-yourself philosophy and would probably be unappreciated by someone wanting an IPSEC tunnel unencumbered by additional defaults like this.

---

At any rate, I hope this helps someone... I'll link this thread in some others I found that seemed unresolved. If anyone understands this or how source interface/IP selection works in RouterOS, I'd be very curious to know. I'd bet it's probably the default method used by the underlying Linux kernel version, so maybe I'll look that up later or post it as a separate question on here, but for now it's late and I'm tired. Good luck!
Last edited by bcbigb on Wed Aug 04, 2021 2:06 pm, edited 2 times in total.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Interface selection for internal/router-generated traffic -- A solution and request for feedback.

Wed Aug 04, 2021 12:05 pm

...In order to populate this with keywords to help others find this solution...
You do exactly the opposite, you have written so many words that almost any search is polluted.
Instead of being simple and concise, you wrote a novel.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10183
Joined: Mon Jun 08, 2015 12:09 pm

Re: Interface selection for internal/router-generated traffic -- A solution and request for feedback.  [SOLVED]

Wed Aug 04, 2021 12:36 pm

What you describe is the can of worms that you usually get when using plain IPsec tunnels.
Problems with source address selection, tricks required to avoid NAT rules triggering, complicated firewalls because traffic through and outside the tunnel is coming in on the same interface (and has to be selected using a special match), etc etc etc.

When you control both ends of the link I strongly advise you to migrate from an "IPsec tunnel" to a "GRE over IPsec transport" tunnel (or IPIP).
This is easy to configure: remove the IPsec tunnel, create a GRE tunnel interface at both ends, set an IP on each GRE tunnel interface (e.g. a /30), and setup either static routing or use BGP or OSPF.

Now, you have a clear situation w.r.t. routes, firewall rules, source address selection, etc.
And it also is much easier to maintain when you add extra sites, or extra transport methods (e.g. you want a 4G link to backup the fixed internet connection).
 
User avatar
bcbigb
just joined
Topic Author
Posts: 20
Joined: Sat Dec 28, 2013 5:48 pm

Re: Interface selection for internal/router-generated traffic -- A solution and request for feedback.

Wed Aug 04, 2021 1:28 pm

...In order to populate this with keywords to help others find this solution...
You do exactly the opposite, you have written so many words that almost any search is polluted.
Instead of being simple and concise, you wrote a novel.
Yes, in some sense, but I was documenting several different issues that people may come across and answering several questions at once. I considered splitting it up into multiple posts, but it would have been very weird and redundant if split up that way.

Also, this wasn't intended to be a quick fix (though it was outlined at the bottom, which I'll highlight now. Some people who are trying to figure out these interrelationships in context need more explanation than just "do this". I do respect the desire for conciseness in many situations, but this was intended for a different purpose than simply an isolated answer.
Last edited by bcbigb on Wed Aug 04, 2021 1:39 pm, edited 1 time in total.
 
User avatar
bcbigb
just joined
Topic Author
Posts: 20
Joined: Sat Dec 28, 2013 5:48 pm

Re: Interface selection for internal/router-generated traffic -- A solution and request for feedback.

Wed Aug 04, 2021 1:38 pm

What you describe is the can of worms that you usually get when using plain IPsec tunnels.
Problems with source address selection, tricks required to avoid NAT rules triggering, complicated firewalls because traffic through and outside the tunnel is coming in on the same interface (and has to be selected using a special match), etc etc etc.

When you control both ends of the link I strongly advise you to migrate from an "IPsec tunnel" to a "GRE over IPsec transport" tunnel (or IPIP).
This is easy to configure: remove the IPsec tunnel, create a GRE tunnel interface at both ends, set an IP on each GRE tunnel interface (e.g. a /30), and setup either static routing or use BGP or OSPF.

Now, you have a clear situation w.r.t. routes, firewall rules, source address selection, etc.
And it also is much easier to maintain when you add extra sites, or extra transport methods (e.g. you want a 4G link to backup the fixed internet connection).
Very interesting... I've set up more complicated networks elsewhere, but didn't want to do it for these tiny clients with only 2-4 sites, so this fix seemed to be a good solution for that, but perhaps it only seems simple because I am used to adding the couple of bypass rules and they're always available in the manual should I forget something. Using /30s does have its advantages for addressing and routing like you mentioned, and I can see why you would recommend it for maintenance, particularly at scale with a proper routing protocol. I just never consider BGP/OSPF at tiny scales like this, so I'm rolling over it in my mind for these small clients. If I were starting another WISP, it wouldn't even be a question.

Either way, I'm going to definitely lab that up and take a look, so thank you very much for the feedback.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10183
Joined: Mon Jun 08, 2015 12:09 pm

Re: Interface selection for internal/router-generated traffic -- A solution and request for feedback.

Wed Aug 04, 2021 3:33 pm

Your remarks may be valid for complicated routers you have seen in the past, but in a MikroTik router, setting up a GRE over IPsec tunnel is a minute of work, and setting up BGP is another minute or two.
(of course assuming you have done it before)
The /30 network is only used for the two peers on the GRE tunnel, i.e. you need that to define the static routes or to configure the peers in BGP.
Of course when you do nothing more you would find that when Router1 is itself sending traffic to a host on Router2's network, it will use the /30 address as the source address, and the host may not know about it and not have the correct route to it.
But you solve that by setting the preferred source address in either the static route or in a route filter in BGP.

It does not depend on the scale, it depends on the way of network design. You can do it with 2 sites, or with 200.
(above that you may want to think about other alternatives)

Who is online

Users browsing this forum: adimihaix, Bing [Bot], Google [Bot], rplant and 75 guests