Community discussions

MikroTik App
 
User avatar
mutluit
Forum Veteran
Forum Veteran
Topic Author
Posts: 765
Joined: Wed Mar 25, 2020 4:04 am

Why DNS-record updates not working?

Tue Nov 10, 2020 4:37 pm

Hi,
imagine this real scenario: for example the domain consent.youtube.com has one IP, but it changes often (like a dynamic IP, but I rather think YT changes the f*cking IP intentionally).
When the IP changes then the DNS server in RouterOS still gives the old, now invalid, IP, which of course isn't working anymore, leading to a connection error.
I just wonder why RouterOS cannot detect and update it automatically. IMHO this should be possible since normally even true dynamic IPs (ie. DynDNS) are working well with other DNS servers...
 
User avatar
eworm
Forum Veteran
Forum Veteran
Posts: 749
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: Why DNS-record updates not working?

Tue Nov 10, 2020 4:41 pm

The domain has a time to live (ttl) of 299 seconds. RouterOS caches the record for this time, see / ip dns cache.
This is correct behavior and should not be changed.
Manage RouterOS scripts and extend your devices' functionality: RouterOS Scripts
 
User avatar
eworm
Forum Veteran
Forum Veteran
Posts: 749
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: Why DNS-record updates not working?

Tue Nov 10, 2020 4:44 pm

Or for details on that specific domain:
[admin@MikroTik] > /ip dns cache print where name="consent.youtube.com"
Flags: S - static 
 #   NAME       TYPE  DATA                                         TTL         
 0   consent... A     172.217.17.142                               3m8s
Note that TTL here is the actual time remaining in cache, not what the upstream server gave.
Manage RouterOS scripts and extend your devices' functionality: RouterOS Scripts
 
User avatar
mutluit
Forum Veteran
Forum Veteran
Topic Author
Posts: 765
Joined: Wed Mar 25, 2020 4:04 am

Re: Why DNS-record updates not working?

Tue Nov 10, 2020 4:54 pm

The domain has a time to live (ttl) of 299 seconds. RouterOS caches the record for this time, see / ip dns cache.
This is correct behavior and should not be changed.
But this is not user-friendly. I mean just think practically....
 
User avatar
eworm
Forum Veteran
Forum Veteran
Posts: 749
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: Why DNS-record updates not working?

Tue Nov 10, 2020 4:58 pm

But that is how things work. What do you think this should work like?
Manage RouterOS scripts and extend your devices' functionality: RouterOS Scripts
 
User avatar
mutluit
Forum Veteran
Forum Veteran
Topic Author
Posts: 765
Joined: Wed Mar 25, 2020 4:04 am

Re: Why DNS-record updates not working?

Tue Nov 10, 2020 5:01 pm

But that is how things work. What do you think this should work like?
After each connection failure, RouterOS should check whether the IP/domain is covered in its local DNS, and if yes, then check/verify whether its A record is still valid...

Because: currently I have manually to do these 2 damn steps each time to fix this damn error:
[x@y] /ip/firewall/address-list> remove [find address=consent.youtube.com]               
[x@y] /ip/firewall/address-list> add list=allow address=consent.youtube.com
 
Sob
Forum Guru
Forum Guru
Posts: 6484
Joined: Mon Apr 20, 2009 9:11 pm

Re: Why DNS-record updates not working?

Tue Nov 10, 2020 6:13 pm

Bad news, if you're trying to use this as whitelist for hostnames, your expectations are wrong, and it won't work reliably.

Hostname can have multiple IP addresses, and DNS server can give different ones to different clients. And they can change all the time, because many websites and especially various CDNs use this for load balancing.

To have any chance for this to work at all, it's critical that both router and client have the same data, so client must use only this router as DNS resolver. It's big problem with things like DNS over HTTPS.

But even if you are able to force client to only use this router as resolver, it still won't be 100% reliable, because not everything is precise. It can easily happen that client resolves hostname, gets address from router with just expiring TTL, but tries to actually connect to that address later, when the record already expired. Problem is, even if it's just a second, router already refreshed address list and new address may be different, so the connection will be blocked, because address list no longer contains old address.
Excessive quoting is useless and annoying. If you use it, please consider if you could do without it.
 
User avatar
eworm
Forum Veteran
Forum Veteran
Posts: 749
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: Why DNS-record updates not working?

Tue Nov 10, 2020 6:36 pm

There's no need to blame DNS server in RouterOS - that works as expected and is completely unrelated to your problem.
The address list is something completely different, and it can not be use (reliably) the way you expect it.
Manage RouterOS scripts and extend your devices' functionality: RouterOS Scripts
 
User avatar
mutluit
Forum Veteran
Forum Veteran
Topic Author
Posts: 765
Joined: Wed Mar 25, 2020 4:04 am

Re: Why DNS-record updates not working?

Tue Nov 10, 2020 6:57 pm

There's no need to blame DNS server in RouterOS - that works as expected and is completely unrelated to your problem.
The address list is something completely different, and it can not be use (reliably) the way you expect it.
Maybe it's just an address list issue, not DNS.
I'm using the address list feature in RouterOS for allowing access to outbound (as well inbound). Ie. by default everything is blocked.
The problem with the address list is that it stores the IP, whereas for my use-case it would suffice if it would operate on domain name only. Somehow...
 
User avatar
eworm
Forum Veteran
Forum Veteran
Posts: 749
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: Why DNS-record updates not working?

Tue Nov 10, 2020 7:16 pm

The problem with the address list is that it stores the IP, whereas for my use-case it would suffice if it would operate on domain name only. Somehow...
This is an address list, it operates on addresses.
Creating dynamic address list entries from domain names is a convenient feature, but it is not made to work with constantly changing domain name records.
Manage RouterOS scripts and extend your devices' functionality: RouterOS Scripts
 
Sob
Forum Guru
Forum Guru
Posts: 6484
Joined: Mon Apr 20, 2009 9:11 pm

Re: Why DNS-record updates not working?

Tue Nov 10, 2020 7:20 pm

If we could just ignore how things work, it would make everything so much easier. ;)

It can't "operate on domain name only", because when you're connecting to some server, you're not connecting to name, packets have no place for that, they only work with numeric addresses. So client first resolves the name to get address, and then it connects to that address. There's no direct relation between that, in a way that router can see (there goes your previous idea that router could check for connection failures).

I currently need the same thing in one place and I had the same idea like you, to use address list with hostnames. But it's barely usable. Servers with static addresses are fine. Those that change addresses once per hour or more, there I can live with a tiny outage (few seconds) when it happens. But too many hostnames have too short TTLs, 60 seconds is very common. And the router often gets even less, because it comes from upstream resolver's cache and it's not always completely fresh. It means few seconds long problem every half a minute or so, so it's unusable.

I'd love to hear some idea how it could be solved in RouterOS, but I don't think there's a way.
Excessive quoting is useless and annoying. If you use it, please consider if you could do without it.
 
User avatar
eworm
Forum Veteran
Forum Veteran
Posts: 749
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: Why DNS-record updates not working?

Tue Nov 10, 2020 7:26 pm

That said I gave it a try myself... Looks like RouterOS actually does update the address list entry when ttl expires:
/ip firewall address-list add address=et-contents.s3.eu-west-1.amazonaws.com list=test
/ip firewall address-list print interval=5s follow where comment="et-contents.s3.eu-west-1.amazonaws.com"
Flags: X - disabled, D - dynamic 
 #   LIST            ADDRESS                              CREATION-TIME        TIMEOUT             
 0 D ;;; et-contents.s3.eu-west-1.amazonaws.com
     test            52.218.96.131                        nov/10/2020 18:24:00
 1 D ;;; et-contents.s3.eu-west-1.amazonaws.com
     test            52.218.56.96                         nov/10/2020 18:24:10
 2 D ;;; et-contents.s3.eu-west-1.amazonaws.com
     test            52.218.88.224                        nov/10/2020 18:24:20
 3 D ;;; et-contents.s3.eu-west-1.amazonaws.com
     test            52.218.105.187                       nov/10/2020 18:24:30
So it should work better than expected. Can you verify whether or not this happens for you?
Manage RouterOS scripts and extend your devices' functionality: RouterOS Scripts
 
User avatar
mutluit
Forum Veteran
Forum Veteran
Topic Author
Posts: 765
Joined: Wed Mar 25, 2020 4:04 am

Re: Why DNS-record updates not working?

Tue Nov 10, 2020 7:35 pm

So client first resolves the name to get address, and then it connects to that address. There's no direct relation between that, in a way that router can see (there goes your previous idea that router could check for connection failures).
I think the problem in RouterOS can be solved as follows: after each DNS resolve the address list should for this particular address be checked for invalid data (outdated IP) and updated accordingly.
I personally would use just a pointer or a reference to the DNS data, and not IP copies as is done currently. By this method the IP in the address list would always automatically be the same like in the DNS database...
 
Sob
Forum Guru
Forum Guru
Posts: 6484
Joined: Mon Apr 20, 2009 9:11 pm

Re: Why DNS-record updates not working?

Tue Nov 10, 2020 8:27 pm

Well, while it would not be completely impossible, you need to thing big. I don't know your numbers (hostnames in address list, clients using the router, amount of traffic, etc), but the solution would have to work even when you'd have thousands of hostnames in address list, thousands of clients using the router as resolver, thousands of packets checked against addresss list every second, etc. Also don't forget that router's main job is to route, it can't waste too much resources on stuff like this.

And again, what happens on router is not the only problem. TTL resolution for DNS records is in seconds. And things are asynchronous, client gets the address, but before it tries to connect, router already updates the records it has, and connection fails. The difference can be a millisecond or even less.

If you'd want 100% reliability (well, almost), the whole thing would have to work differently. DNS resolution on router would have to be synchronized with address list in some way, so when router would get new data from upstream resolver, it would have to immediatelly update address list and delay all clients asking for the hostname, in order to not give them the address before it's in address list. Your idea with pointers wouldn't work, because it would ruin all optimizations. Address lists use hashes, so they are fast even with huge amount of addresses. Pointers to DNS cache would require checking each address one by one, so it would completely kill the performance. Address list would also need to hold old addresses for a while, in case the client is slow in using them.

In short, it may seem easy, but in reality it's nearly impossible.
Excessive quoting is useless and annoying. If you use it, please consider if you could do without it.
 
Sob
Forum Guru
Forum Guru
Posts: 6484
Joined: Mon Apr 20, 2009 9:11 pm

Re: Why DNS-record updates not working?

Tue Nov 10, 2020 9:06 pm

@eworm: It works, but the problem occurs when address changes. I tried a simple test:

- hostname with TTL 60 seconds is updated on authoritative server every 20 seconds with different address (so client respecting TTL will get different address each time)
- the hostname is in address list on router
- client is sending http request to hostname in a loop; it doesn't do anything special itself, so it uses whatever DNS caching is done by OS (the router is used as resolver)
- router counts both allowed connections (when destination is in address list) and failed ones (when destination is not in address list)
- addresses are not actually reachable and client has one second timeout, so there's roughly one request per second

Result: ~87% connections are allowed, the rest is blocked. It should be better in real world scenario, because even though TTL 60 seconds is nothing extreme, addresses usually don't change every time.
Excessive quoting is useless and annoying. If you use it, please consider if you could do without it.
 
User avatar
eworm
Forum Veteran
Forum Veteran
Posts: 749
Joined: Wed Oct 22, 2014 9:23 am
Location: Oberhausen, Germany
Contact:

Re: Why DNS-record updates not working?

Tue Nov 10, 2020 9:56 pm

But you do not have to manually remove and add the address list entry, no?
So wondering why mutluit has to.
Manage RouterOS scripts and extend your devices' functionality: RouterOS Scripts
 
Sob
Forum Guru
Forum Guru
Posts: 6484
Joined: Mon Apr 20, 2009 9:11 pm

Re: Why DNS-record updates not working?

Tue Nov 10, 2020 10:11 pm

Maybe clients don't use this router as their only DNS resolver? If that's the case, it could be much worse. Even if they would both use same server, it may not be the same server at all. For example public servers like 8.8.8.8 are actually many different load balanced servers, each with different cached records, so same queries to 8.8.8.8 can get differerent responses, different addresses, different TTLs. So the address list on router would not match with what client is using at all.
Excessive quoting is useless and annoying. If you use it, please consider if you could do without it.

Who is online

Users browsing this forum: td32 and 11 guests