Mikrotik DNS server issues with Amazon S3 - low TTL 60sec

falz · Mon Jan 09, 2012 4:06 pm

Hello,

I've run in to an issue where a customer using a Mikrotik RB751 running the latest RouterOS 5 (5.8 at the time?) has a lot of issues using Amazon S3 apparently due to the very low TTL that Amazon uses (60 seconds). I've heard of others with this issue as well, is the only workaround to not use the DNS server within RouterOS?

feld · Mon Jan 09, 2012 5:20 pm

Hey, this explains the exact problem I was experiencing!

Is there an ETA for a fix?

Thanks

richardhkirkando · Mon Jan 09, 2012 5:30 pm

I've noticed this once or twice and didn't think much of it since I typically run my own DNS cache on a server, but yeah, that issue definitely sounds familiar.

remit · Mon Jan 09, 2012 9:54 pm

YES! This is the exact problem a few of my customers have reported, and I have confirmed. It seems to have started about 3-4 months ago. Running 5.5 caching google's DNS servers.

MCT · Mon Jan 09, 2012 10:54 pm

Why not use just mangle to raise the TTL?

TTL isn't measured in seconds, it's a hop count.

falz · Mon Jan 09, 2012 11:07 pm

TTL in DNS terms is indeed number of seconds to cache a DNS record:

* Wikipedia - DNS TTL

Amazon keeps the TTL low for various reasons. Mucking with it would likely cause you to be connecting to the wrong IP address.

MCT · Tue Jan 10, 2012 12:32 am

TTL in DNS terms is indeed number of seconds to cache a DNS record:

* Wikipedia - DNS TTL

Amazon keeps the TTL low for various reasons. Mucking with it would likely cause you to be connecting to the wrong IP address.

Ah, well I deal with network engineering mostly so my brain defaults to network terms.

R1CH · Tue Jan 10, 2012 2:18 am

Our entire LAN is using a MT RB750G unit as a DNS server, which randomly doesn't resolve some domains (notably Amazon S3 due to its widespread use). I've had to resort to giving out Google DNS IPs via DHCP until this is fixed which is annoying since it breaks all the internal LAN DNS which I had setup. Been having this problem for over a year, glad to see I'm not the only one!

remit · Wed Jan 11, 2012 7:26 pm

Just had another customer call and complain of this issue. Any ideas to resolve this issue, or information I can provide to assist in the resolution?

Xeron · Wed Mar 14, 2012 12:49 am

I have problem with random resolve issues since 5.x version too, I can't remember, but it's about 5.10, may be earlier.

And it's strange, browser, curl/wget and any other software can't resolve name, but nslookup works without problems. So usually I have:

xeron@macbook:~$ wget — can't resolve
xeron@macbook:~$ wget — can't resolve
xeron@macbook:~$ nslookup — resolved
xeron@macbook:~$ wget — can't resolve
wait 1-2 minutes
xeron@macbook:~$ wget — resolved

And really often this problem happens with Amazon S3 hosts, but not only S3.

I tried to increase max-udp-packet-size, but still no luck.

fuzzbawl · Thu Mar 15, 2012 6:36 pm

I have seen issues with this as well. We are currently running RouterOS 5.13 and have seen DNS TTL issues crop up but it's not exactly limited to Amazon S3 but that seems to be the worst offender. I have also seen issues if you have a large number of records expire in cache, it takes 100% cpu for a few seconds to clear those entries out. If you have a lot of requests coming in, and a lot of cache records expiring, it causes the entire router to slow down on all duties.

zerkalka · Mon Mar 19, 2012 8:27 pm

Exactly the SAME PROBLEM!
Not only amazon, but randomly ever!

Where is mikrotik team?

Defensor · Thu Mar 22, 2012 8:30 pm

I have the same problem with DNS.
And my dns settings randomly returns to the previos ones by themself some times.
ROS 5.14 - RB751U-2HnD.

remit · Fri Mar 23, 2012 5:29 pm

Still an issue, anyone from mikrotik about to weigh in on this? Very annoying

jandafields · Sun Mar 25, 2012 10:19 pm

If everyone in this thread would send an email to support@mikrotik.com, it would probably get seen as a high priority issue!

remit · Wed Mar 28, 2012 2:51 am

janisk · Tue Apr 03, 2012 1:36 pm

it is not clear what issues you are having, if you want to use router dns cache and chace time of the entry is very short - just adjust the max-limit to some lower value, or check if DNS server that responds gives you correct cache time to start with as all values below cache-max-ttl are set according to replied value.

About some other issues - try to increase replysize by increasing max-udp-packet-size to something large like 4096 (that will be default in later RouterOS versions but will not be changed via update to newer version) due to DNSSEC that returns huge replies.

voxframe · Tue Apr 03, 2012 4:45 pm

Holy crap! This might be the problem I had almost a year and a half ago!

I was using OpenDNS and randomly google and amazon DNS related stuff wouldn't resolve. I never had the time to really look into it as it crippled our network and I needed to just get it working. It was something TTL related for sure but couldn't investigate further.

I'm keeping an eye on this one!

rferroni · Wed May 23, 2012 5:38 pm

hi, I´m having the same dns problem. it´s seem to be random.
when I try to resolve several times I get not answer:
:resolve dns.domain.com
failure: dns name does not exist
and then without doing nothing works fine (previous I flushed the cache).
I can add some information but for now I can´t says why:
It´s seem work fine with a dns server on solaris 9 (bind 8.3.3) but is not working on dns server on debian (lenny with bind9 1.9.6). I hope this helps in some way!
thanks!

changeip · Wed May 23, 2012 5:46 pm

to get this resolved, take a packet capture of port 53 on the external interface and highlight the query and the response in wireshark. then send a supout along with those results. if you can prove that the response came back in but didnt get used, then maybe mikrotik will finally look at it.

rferroni · Thu May 24, 2012 10:53 pm

Today I have the problem in both dns servers.
Feeling that I`m going backwards.
As soon I have more time I will capture the packets and try to figure why, because works fine when I delete the second dns entry in the DNS configuration and flush the cache.
But for now I`ll change the dns server in dhcp server without using the mk.
Thanks.

Fri May 25, 2012 1:24 pm

There were few DNS improvements since 5.8 and new versions. Does anybody have DNS issues at the latest 5.16 version?

dboreham · Tue May 29, 2012 2:46 am

I'm seeing this problem with 5.16. Found this thread after pulling out a bunch of hair trying to figure out why random weird things were happening with certain hosts (the one that I'm looking at is a Sony bluray player) after we switched to using Mikrotik for DNS. I'm looking at a packet trace where the host continually submits the same DNS query. MT responds with what on the face of it looks like a kosher reply, but obviously the host no like it because it keeps sending the query again and again. I have similar complaints from customers -- iPad works but laptop doesn't work properly (CSS files not loading for some web sites, stuff like that).

dboreham · Tue May 29, 2012 3:10 am

Update : I increased the DNS cache size (from 2M to 4M) and flushed the cache. This magically fixed my Sony bluray player. Hopefully it doesn't un-fix once the cache fills up again.

dboreham · Tue May 29, 2012 5:27 am

Checking the subs who have MT CPE routers, I noticed that the customer who hasn't complained had their DNS set to 512 bytes max UDP packet size. The others were set to 4096, which I believe was the default since I haven't changed anything in that area myself. I've set all of them to 512 bytes max now. Subscribers reporting problems now say their problems have been resolved.

Tue May 29, 2012 4:15 pm

dboreham,
max-udp-packets-size=512 fixed your problem, is it correct?

dboreham · Sun Jun 10, 2012 5:57 pm

dboreham,
max-udp-packets-size=512 fixed your problem, is it correct?

It fixed that one specific customer problem, yes. But the broader saga continues.
I have customers with computers/routers that "don't like" a high setting for max-udp-packets-size,
and I have customers who have problems with a setting of 512.

I'm in the process of backing out our use of the MT DNS server since it seems not usable for production.
Unfortunate since for us the options are either use a remote DNS out on the 'net (adds to user latency) or deploy bind on a Linux box again (we just took down our Linux servers to save power and colocation space, hence the switch to MT as caching DNS).

mortin · Sun Jun 10, 2012 7:23 pm

For us the problem is serious as we use MK dns cache widely in all our routers.
The random resolve issues occurs since 5.x update and is still present in newest 5.17 one.

Unfortunately we are forced to switch from using MK DNS Cache to external DNS servers.
The DNS cache parameters is default for MK:
servers: list of remote DNS servers which are working correctly
allow-remote-requests: yes
max-udp-packets-size=512
cache-size: 2048KiB
cache-max-ttl: 1w

What I have noticed:
When the problem occurs nslookup doesn't resolve the domain.
Flushing the local dns cache of host after the problem occurs help to resolve the domain correctly.

atopcu · Tue Jun 12, 2012 2:26 am

We have using four pcs RB433UH. ROS are 5.17. There are same problem.

Devil · Fri Jun 22, 2012 11:41 am

For us the problem is serious as we use MK dns cache widely in all our routers.
The random resolve issues occurs since 5.x update and is still present in newest 5.17 one.
...
When the problem occurs nslookup doesn't resolve the domain.
Flushing the local dns cache of host after the problem occurs help to resolve the domain correctly.

I can confirm this as well. since this problem seems quite random, usually nslookup seems to be able to resolve the host while sometimes browsers don't. i thought it might have something to do with the way browsers resolve the dns, see This Article for more detail. however, i'm not able to confirm this at the moment. this problem occurs because for some reason mikrotik fails to send its cached result to the client. the result is in the cache, but for some reason the client seems to not receive it, or at least not fully. when this happens, if you look at 'ipconfig /displaydns' , you'll something like this:

by default, windows caches a negative record for 15 minutes, meaning the client wont be able to resolve the dns in that time. as you mentioned, flushing the dns cache will cause the client to request the dns again which will most likely resolve the issue for a while. other solutions would be disabling the dns cache service completely or yet better disabling only the negative result caching(i have not tried this one but should improve the client's experience): http://support.microsoft.com/default.as ... -us;318803

also it appears that this most likely happens when the resolved dns has multiple records. like:
download.windowsupdate.com
mail.yahoo.com
irc.rizon.net
and so on

Either ways, this is a serious bug that forced me to switch my dns server as well.
Also, i'm using hotspot, maybe some security feature added to protect the hotspot is conflicting with dns resolving, causing this behavior?

Fri Jun 22, 2012 11:48 am

Recently released v5.18 has some DNS changes, could you try it there?
We have so far not been able to repeat the issue locally, with any version.

Devil · Fri Jun 22, 2012 12:02 pm

Hi normis. alright, ill try v5.18. I'm using v5.17 at the moment by the way. two different routers and they're both suffering the same thing. sine i purely rely on my clients feedback, and since this issue seems quite random and could not be easily reproduced, it might take a while to see whether it happens in v5.18 too or not. I'll report back as soon as i could confirm something. meanwhile, ill try to narrow down the problem more.

Devil · Sat Jun 23, 2012 1:10 pm

Alright, i have a bad news and a good news. the bad news is that i can confirm now that the problem still exists in v5.18 . the good news however is that i was able to come up with a batch script to reproduce this problem within minutes. because of that, i was also able to capture the problematic packets and analyze them a bit. lets start with the script:

@echo off
ping -n 2 127.0.0.1 >nul
if not defined host echo Start time: %time% & set count=0 & set host=mail.yahoo.com
set /a count=count+1
title Number of runs = %count%
ipconfig /flushdns >nul
ping %host% -n 1 -w 0 | find "32 bytes" >nul
if errorlevel 1 goto checking
goto rerun

:checking
ipconfig /displaydns | find "%host%" >nul
if not errorlevel 1 goto exit
goto rerun

:rerun
"%~nx0"

:exit
title = Failed!
ipconfig /displaydns
echo Number of runs before the failure = %count%
echo End time: %time%
pause

copy/paste it into a file and change the extension to .bat . as i said, the problem happens with a lot of domains but it appears that it's most likely to happen with mail.yahoo.com. i have tested this script in windows 7 as well as windows xp. also i tried to put a simple test to reduce the chance of false positives. (even a non-responsive dns server, should not be considered as a failure by the script.) . it appears that its not about how fast you try to resolve the dns, in fact the faster you do, the less likely the problem would happen. 1 second between each try seems to be efficient (hence the ping to the localhost). you should be able to reproduce the problem in less that 10 minuets, here's what you should end up with:

the reason for choosing the ping approach for resolving the dns instead of nslookup, is that when it happens, unlike nslookup, the system caches the dns response. which is a lot like what happens when you request the site with a browser. and also it makes it easier to spot. nevertheless, i was able to reproduce this bug with nslookup as well.

This is actually different from NXDOMAIN response. if you take a look at the dns response when it happens:

You'll see that although the Flag indicates a normal reply, 'Answer RRS' and 'Additional RRS' are both empty. this is considered a valid response which is why windows caches the result as "No records of type A"

Mon Jun 25, 2012 4:51 pm

Your script is running for 50 minutes without problems. Can you post "/ip dns" configuration export?

Devil · Mon Jun 25, 2012 5:04 pm

Well, that means it probably depends on something else too. some other settings maybe, that triggers the bug. this is the export you requested:

# jun/25/2012 15:28:33 by RouterOS 5.18
# software id = xxxx-xxxx
#
/ip dns
set allow-remote-requests=no cache-max-ttl=1w cache-size=2048KiB \
    max-udp-packet-size=4096 servers=208.67.222.222,208.67.220.220
/ip dns static
add address=192.168.xxx.xxx disabled=no name=xxxxxxxx ttl=1d
add address=192.168.xxx.xxx disabled=no name=xxxxxxxx ttl=1d
add address=192.168.xxx.xxx disabled=no name=xxxxxxxx ttl=1d

to clear things up about why allow-remote-requests is set to no, let me point out that since i'm running hotspot, all the dns requests are already being forwarded to the hotspot dns port. so no need to set it to yes.
also, i tested this script dozen times. every time it fails in less than 10 minutes

Mon Jun 25, 2012 7:11 pm

Still running with the same DNS settings except allow-remote-request.

Devil · Mon Jun 25, 2012 7:30 pm

I just enabled allow-remote-requests, even disabled static dns rules but still getting the same result. this might not do much, but attached is a wireshark capture file, with 3 dns requests and their responses. the first two failed while the third one was successful

biomesh · Mon Jun 25, 2012 7:39 pm

How much of your cache is in use?

Mine is very low and I don't see the issue you are seeing - been running the batch file for two hours. The config is

/ip dns export
set allow-remote-requests=yes cache-max-ttl=1w cache-size=4096KiB \
max-udp-packet-size=4096 servers=208.67.222.222,208.67.220.220

/ip dns print
servers: 208.67.222.222,208.67.220.220
dynamic-servers:
allow-remote-requests: yes
max-udp-packet-size: 4096
cache-size: 4096KiB
cache-max-ttl: 1w
cache-used: 80KiB

Devil · Mon Jun 25, 2012 7:51 pm

Nope. i just flushed the cache and when the cache-used was as low as 15KiB , it happened again. it's quite clear that some specific setups trigger this bug as some people actually don't have this problem and even mikrotik couldn't reproduce it. i'm trying to change things one by one to see whether i can find whats causing this behavior.

changeip · Mon Jun 25, 2012 11:56 pm

are you using changed opendns configuration? Maybe their responses are different on an unconfigured network from their side.

Devil · Tue Jun 26, 2012 2:05 pm

No 'changeip', that is not the case. As i'm using another dns server running on the same link, pointing to opendns without any problem. I have actually found some new stuff regarding this bug. I'm now able to predict when its going to happen with high probability. but i need more time to find the exact cause of the problem (although i'm close). I'll keep you update

Xeron · Sun Jul 01, 2012 12:31 pm

As for me, problem fixed after 5.17.

My problem was described at http://forum.mikrotik.com/viewtopic.php ... 43#p307422.

UPD: After update from 5.17 to 5.18 problem returned.

Xeron · Mon Jul 02, 2012 9:47 pm

Got this problem with tcpdump running on client machine. Sorry for big copypaste.

Tried to go to some http://cl.ly/ link

22:13:31.696467 IP 192.168.88.104.53065 > mikrotik.lan.domain: 44437+ A? crl.thawte.com. (32)
22:13:31.915154 IP mikrotik.lan.domain > 192.168.88.104.53065: 44437 2/0/0 CNAME crl.verisign.net., A 199.7.48.190 (78)
22:13:32.916544 IP 192.168.88.104.50810 > mikrotik.lan.domain: 7352+ A? cl.ly. (23)
22:13:32.917165 IP 192.168.88.104.51687 > mikrotik.lan.domain: 14656+ A? api.cld.me. (28)
22:13:32.917483 IP 192.168.88.104.61730 > mikrotik.lan.domain: 16470+ A? assets.cld.me. (31)
22:13:32.917534 IP 192.168.88.104.61344 > mikrotik.lan.domain: 35881+ A? f.cl.ly. (25)
22:13:32.917647 IP 192.168.88.104.60021 > mikrotik.lan.domain: 60353+ A? www-google-analytics.l.google.com. (51)
22:13:32.937707 IP mikrotik.lan.domain > 192.168.88.104.61344: 35881 5/0/0 CNAME f.cl.ly.s3.amazonaws.com., CNAME s3-directional-w.amazonaws.com., CNAME s3-directional-w.geo.amazonaws.com., CNAME s3-1-w.amazonaws.com., A 207.171.163.196 (166)
22:13:32.944460 IP mikrotik.lan.domain > 192.168.88.104.60021: 60353 11/0/0 A 173.194.32.0, A 173.194.32.1, A 173.194.32.2, A 173.194.32.3, A 173.194.32.4, A 173.194.32.5, A 173.194.32.6, A 173.194.32.7, A 173.194.32.8, A 173.194.32.9, A 173.194.32.14 (227)
22:13:33.037923 IP mikrotik.lan.domain > 192.168.88.104.51687: 14656 11/0/0 CNAME cloudapp.herokuapp.com., CNAME ar.herokuapp.com., CNAME argon-stack-1879049447.us-east-1.elb.amazonaws.com., A 23.23.195.213, A 50.17.250.204, A 75.101.152.162, A 107.20.177.118, A 23.21.154.16, A 23.21.241.235, A 23.23.129.204, A 23.23.130.88 (270)
22:13:33.083854 IP mikrotik.lan.domain > 192.168.88.104.50810: 7352 ServFail 3/0/0 A 75.101.163.44, A 174.129.212.2, A 75.101.145.87 (71)
22:13:33.440868 IP mikrotik.lan.domain > 192.168.88.104.61730: 16470 10/0/0 CNAME d23tod3mb75lgr.cloudfront.net., CNAME d23tod3mb75lgr.arn1.cloudfront.net., A 205.251.219.63, A 205.251.219.65, A 205.251.219.80, A 205.251.219.122, A 205.251.219.151, A 205.251.219.169, A 205.251.219.199, A 205.251.219.224 (236)
22:13:33.934670 IP 192.168.88.104.49406 > mikrotik.lan.domain: 19944+ A? linkhelp.clients.google.com. (45)
22:13:33.954028 IP mikrotik.lan.domain > 192.168.88.104.49406: 19944 12/0/0 CNAME clients.l.google.com., A 173.194.32.200, A 173.194.32.201, A 173.194.32.206, A 173.194.32.192, A 173.194.32.193, A 173.194.32.194, A 173.194.32.195, A 173.194.32.196, A 173.194.32.197, A 173.194.32.198, A 173.194.32.199 (245)
22:13:34.126591 IP 192.168.88.104.64929 > mikrotik.lan.domain: 6136+ A? csi.gstatic.com. (33)
22:13:34.136641 IP mikrotik.lan.domain > 192.168.88.104.64929: 6136 1/0/0 A 74.125.239.15 (49)

Then mDNSResponder cached this failed request, so what I got:

nslookup can resolve (it does request to mikrotik's dns server)
ping/curl/browsers can't resolve (because they asking MacOS's mDNSResponder)

After mDNSResponder's cache purge or after some time when cache expired ping/curl/browsers can resolve too.

Another tcpdump of failed request (assets.cld.me didn't resolve):

22:22:07.523407 IP 192.168.88.104.51449 > mikrotik.lan.domain: 18424+ A? cl.ly. (23)
22:22:07.528335 IP mikrotik.lan.domain > 192.168.88.104.51449: 18424 3/0/0 A 174.129.212.2, A 75.101.145.87, A 75.101.163.44 (71)
22:22:07.870006 IP 192.168.88.104.49549 > mikrotik.lan.domain: 9249+ A? assets.cld.me. (31)
22:22:08.687931 IP mikrotik.lan.domain > 192.168.88.104.49549: 9249 ServFail 10/0/0 CNAME d23tod3mb75lgr.cloudfront.net., CNAME d23tod3mb75lgr.arn1.cloudfront.net., A 205.251.219.38, A 205.251.219.78, A 205.251.219.92, A 205.251.219.93, A 205.251.219.113, A 205.251.219.115, A 205.251.219.140, A 205.251.219.210 (236)
22:22:09.072335 IP 192.168.88.104.53981 > mikrotik.lan.domain: 40860+ A? api.cld.me. (28)
22:22:09.609727 IP mikrotik.lan.domain > 192.168.88.104.53981: 40860 11/0/0 CNAME cloudapp.herokuapp.com., CNAME ar.herokuapp.com., CNAME argon-stack-1879049447.us-east-1.elb.amazonaws.com., A 174.129.244.122, A 23.21.77.228, A 23.23.130.88, A 23.23.204.240, A 50.17.250.204, A 75.101.152.162, A 107.20.177.118, A 107.20.207.97 (270)
22:22:10.101057 IP 192.168.88.104.55715 > mikrotik.lan.domain: 57284+ A? f.cl.ly. (25)
22:22:10.147813 IP mikrotik.lan.domain > 192.168.88.104.55715: 57284 5/0/0 CNAME f.cl.ly.s3.amazonaws.com., CNAME s3-directional-w.amazonaws.com., CNAME s3-directional-w.geo.amazonaws.com., CNAME s3-1-w.amazonaws.com., A 72.21.203.149 (166)

Wait some time, repeat request

22:23:55.323760 IP 192.168.88.104.51425 > mikrotik.lan.domain: 47950+ A? assets.cld.me. (31)
22:23:56.325211 IP 192.168.88.104.51425 > mikrotik.lan.domain: 47950+ A? assets.cld.me. (31)
22:23:57.744442 IP mikrotik.lan.domain > 192.168.88.104.51425: 47950 10/0/0 CNAME d23tod3mb75lgr.cloudfront.net., CNAME d23tod3mb75lgr.arn1.cloudfront.net., A 205.251.219.36, A 205.251.219.49, A 205.251.219.81, A 205.251.219.95, A 205.251.219.103, A 205.251.219.120, A 205.251.219.133, A 205.251.219.211 (236)
22:23:58.113325 IP 192.168.88.104.51539 > mikrotik.lan.domain: 1350+ A? argon-stack-1879049447.us-east-1.elb.amazonaws.com. (68)
22:23:58.141764 IP mikrotik.lan.domain > 192.168.88.104.51539: 1350 8/0/0 A 50.17.220.21, A 50.17.250.204, A 174.129.244.122, A 184.73.155.93, A 23.21.154.16, A 23.21.241.235, A 23.23.130.88, A 50.17.184.83 (196)

Only difference I see it's "ServFail" before normal response in mikrotik's CNAME/A answer.

22:13:33.083854 IP mikrotik.lan.domain > 192.168.88.104.50810: 7352 ServFail 3/0/0 A 75.101.163.44, A 174.129.212.2, A 75.101.145.87 (71)
22:22:08.687931 IP mikrotik.lan.domain > 192.168.88.104.49549: 9249 ServFail 10/0/0 CNAME d23tod3mb75lgr.cloudfront.net., CNAME d23tod3mb75lgr.arn1.cloudfront.net., A 205.251.219.38, A 205.251.219.78, A 205.251.219.92, A 205.251.219.93, A 205.251.219.113, A 205.251.219.115, A 205.251.219.140, A 205.251.219.210 (236)

This problem never occurs when I use other DNS servers directly on client machine (tried Google's or my ISP's).
This problem often occurs when I use Mikrotik's DNS server on client machine and Mikrotik uses any (tried Google's and my ISP's) DNS servers.

On 5.17 I thought this problem disappeared, but on 5.18 it returned. See it every day.

Devil · Sun Jul 08, 2012 2:20 pm

Alright, im sorry for the late reply. i really thought i could spend more time on this issue. but seems its not going to happen any time soon. so im just going to share my findings, hoping it could be enough to identify the problem. all the domain names that ive reported, have something in common: they're using Round Robin with multiple CNAMES. meaning, CNAMES chain, might change with each request. let me give you an example with mail.yahoo.com . by requesting the host through a dns server, one could get this chain:

mail.yahoo.com -> login.yahoo.com -> login-global.lgg1.b.yahoo.com -> login.lga1.b.yahoo.com -> 98.139.241.94

but that's not the only chain. every once in a while, the dns response changes to:

mail.yahoo.com -> login.yahoo.com -> login-global.lgg1.b.yahoo.com -> eulogin.lga1.b.yahoo.com -> eu-eulogin.lga1.b.yahoo.com -> 217.12.8.76

at least that's the case for opendns servers being requested through my internet line (just for the record, i also have access to another line and it seems that opendns always resolves mail.yahoo.com to the second chain. i ran the script for hours and was not able to reproduce the bug).

of course each one of those records, have their own TTL. in the second chain, it appears that the last CNAME and A Records are sharing the same TTL but thats not the case in the first chain. as we go up in the chain, TTLs increase. now imagine we request mail.yahoo.com once and the dns server get the response from the first chain, if just when the A record expires, we try to resolve mail.yahoo.com again and we get the response from the second chain this time, things might go wrong. (this needs more testing but at this point i can confirm that the bug might only trigger right when the A records TTL hit 0). look at this screenshot for example:

the bug was triggered in about 7 seconds after the screenshot (this might also have something to do with latency. for example mikrotik dns server couldn't get the result from opendns fast enough so it tries to send what its got in its cache instead...).

Devil · Fri Jul 13, 2012 3:21 pm

Alright, so as it turned out switching DNS server, caused me some very hard to find problems. so i decided to switch back to mikrotik dns once more but this time with a new approach. if you still want to use mikrotik dns, and specially if you are using dhcp to hand out dns settings, read on:
I was able to come up with a regex to find when the dns bug is being triggered, and in the end what appears to be a rather effective workaround.
ill try to explain things step by step:

/ip firewall layer7-protocol add name="DNS bug zapper" regexp="^.\?.\?\\x81\\x80\\x01[\\x01-\?][a-z0-9][\\x01-\?a-z]*[\\x02-\\x06][a-z][a-z][a-z]\?[a-z]\?[a-z]\?[a-z]\?\\x01\\x01\$"

this regex will be triggered when the bug happens ( when there is no RRs record for type A) . original credit goes to Matthew Strait and Ethan Sommer for their work on DNS L7 Filter.

now we add a firewall rule for output dns packets matching our L7 filter, and we drop them:

/ip firewall filter add action=drop chain=output layer7-protocol="DNS bug zapper" out-interface=ether2-local protocol=udp src-port=53

adjust the code base on your settings. note that if you are using hotspot like me, the src-port should be set to 64872 . in the wiki its stated that you should create the same rule in input chain or it might not work as expected. well, i have found that its working very nicely without adding a rule to input chain, but if you want to add one, here's the code:

/ip firewall filter add action=passthrough chain=input layer7-protocol="DNS bug zapper" in-interface=ether2-local protocol=udp dst-port=53

again for hotspot, dst-port should be 64872.

well the first part is done. now the system should be able to recognize buggy packets and filter them. the client simply don't get any response so it tries again. however, there is still a big problem. at least in windows, the system tries up to five times to resolve the host before giving up. the problem is that it uses the same connection each time. and as you can see, according to Technical Details of l7-filter, once a packet matched an L7 filter, the whole connection will be considered as a match afterwards. so all the five attempts from the windows client will be considered a match and will be dropped.

So far we have been able to ease the bug's effect. the client will still get the error but it won't stick in his/her dns cache. so a simple refresh would most likely do the trick. but that's not good enough.
so this is how we get around this 5 times try using the same udp connection: by specifying an alternate DNS server in dhcp and redirecting DNS requests to that DNS server, to our own. so lets say we choose 8.8.8.8 (the address doesn't even need to be a valid dns server but i suggest you choose a valid one so your whole clients experience won't depend on a single nat rule) as secondary dns for our clients. then we add this nat rule:

/ip firewall nat add chain=dstnat action=redirect protocol=udp dst-address=8.8.8.8 dst-port=53 in-interface=ether2-local

if you're using hotspot, you do not need to add that nat rule as its already in place by default.

Well that's about it. now upon a failure, since the client doesn't get any response, it tries its secondary DNS which results in establishing a new udp connection. and it most likely will be able to resolve the host in question this time. i've been testing this for hours and was not able to reproduce this bug or even not get a proper response every time.

Edit 3: Hopefully the final edit. Stated the correct reason for why reusing the same connection that has matched L7 filtering once, would not work anymore. one might be able to find another way around this limitation as well, but i find it unnecessary at this point.

changeip · Fri Jul 13, 2012 7:03 pm

please mikrotik team - lets fix this dns server code instead of work around it with hacks : ) You should be able to identify it now and fix it since 'devil' has done such good work on tracking it down.

xphat · Mon Jul 16, 2012 4:49 am

I am having this SAME issue!

I thought i was going crazy for a bit. Please fix this issue ASAP!!

Regards.

jakkwb · Tue Jul 17, 2012 8:26 pm

I would like a fix also, not a workaround. I have three Mikrotiks and all are having this problem.

galileo · Thu Jul 19, 2012 4:37 pm

I can confirm a similar issue.
For me it manifest immediately when I attach a queue to global-in or global-out.
I doesn't even have to match dns traffic.
Something similar has been reported: http://forum.mikrotik.com/viewtopic.php?f=2&t=60582

I first noticed the issue when trying to ssh to a RHEL 6 server.
With the queue in place login takes about 35s, without 0.3s.
I'm using the mikrotik as a caching dns server and with a few static local entries.
I tried increasing the max-udp-packet-size parameter but it made no difference.
Anyway here are some tcpdumps.
I can attach full binary dumps if anyone needs them.

Without queue:

[root@vm ~]# tcpdump -i br0 -nn host 192.168.1.251 and 192.168.1.254
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 65535 bytes
15:26:21.053188 IP 192.168.1.251.47677 > 192.168.1.254.53: 7461+ PTR? 60.1.168.192.in-addr.arpa. (43)
15:26:21.055308 IP 192.168.1.254.53 > 192.168.1.251.47677: 7461 1/0/1 PTR galileo.local.test.net. (98)
15:26:21.055475 IP 192.168.1.251.42607 > 192.168.1.254.53: 1382+ A? galileo.local.test.net. (43)
15:26:21.061309 IP 192.168.1.254.53 > 192.168.1.251.42607: 1382 1/13/11 A 192.168.1.60 (456)
15:26:21.129686 IP 192.168.1.251.59587 > 192.168.1.254.53: 36114+ A? galileo.local.test.net. (43)
15:26:21.129698 IP 192.168.1.251.59587 > 192.168.1.254.53: 52686+ AAAA? galileo.local.test.net. (43)
15:26:21.135461 IP 192.168.1.254.53 > 192.168.1.251.59587: 36114 1/13/11 A 192.168.1.60 (456)
15:26:21.141500 IP 192.168.1.254.53 > 192.168.1.251.59587: 52686 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:26:21.142651 IP 192.168.1.251.41761 > 192.168.1.254.53: 42299+ A? galileo.local.test.net. (43)
15:26:21.142661 IP 192.168.1.251.41761 > 192.168.1.254.53: 30131+ AAAA? galileo.local.test.net. (43)
15:26:21.148496 IP 192.168.1.254.53 > 192.168.1.251.41761: 42299 1/13/11 A 192.168.1.60 (456)
15:26:21.154579 IP 192.168.1.254.53 > 192.168.1.251.41761: 30131 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:26:21.158360 IP 192.168.1.251.44583 > 192.168.1.254.53: 30041+ A? galileo.local.test.net. (43)
15:26:21.158371 IP 192.168.1.251.44583 > 192.168.1.254.53: 44297+ AAAA? galileo.local.test.net. (43)
15:26:21.164296 IP 192.168.1.254.53 > 192.168.1.251.44583: 30041 1/13/11 A 192.168.1.60 (456)
15:26:21.170322 IP 192.168.1.254.53 > 192.168.1.251.44583: 44297 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:26:21.171029 IP 192.168.1.251.54687 > 192.168.1.254.53: 7492+ A? galileo.local.test.net. (43)
15:26:21.171038 IP 192.168.1.251.54687 > 192.168.1.254.53: 25992+ AAAA? galileo.local.test.net. (43)
15:26:21.176820 IP 192.168.1.254.53 > 192.168.1.251.54687: 7492 1/13/11 A 192.168.1.60 (456)
15:26:21.183175 IP 192.168.1.254.53 > 192.168.1.251.54687: 25992 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:26:21.183323 IP 192.168.1.251.55833 > 192.168.1.254.53: 56609+ A? galileo.local.test.net. (43)
15:26:21.183332 IP 192.168.1.251.55833 > 192.168.1.254.53: 24005+ AAAA? galileo.local.test.net. (43)
15:26:21.189412 IP 192.168.1.254.53 > 192.168.1.251.55833: 56609 1/13/11 A 192.168.1.60 (456)
15:26:21.195418 IP 192.168.1.254.53 > 192.168.1.251.55833: 24005 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:26:21.196565 IP 192.168.1.251.43972 > 192.168.1.254.53: 52618+ A? galileo.local.test.net. (43)
15:26:21.196720 IP 192.168.1.251.43972 > 192.168.1.254.53: 19827+ AAAA? galileo.local.test.net. (43)
15:26:21.202721 IP 192.168.1.254.53 > 192.168.1.251.43972: 52618 1/13/11 A 192.168.1.60 (456)
15:26:21.208636 IP 192.168.1.254.53 > 192.168.1.251.43972: 19827 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:26:21.215342 IP 192.168.1.251.47937 > 192.168.1.254.53: 5586+ A? galileo.local.test.net. (43)
15:26:21.215353 IP 192.168.1.251.47937 > 192.168.1.254.53: 26093+ AAAA? galileo.local.test.net. (43)
15:26:21.221328 IP 192.168.1.254.53 > 192.168.1.251.47937: 5586 1/13/11 A 192.168.1.60 (456)
15:26:21.227266 IP 192.168.1.254.53 > 192.168.1.251.47937: 26093 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:26:21.227558 IP 192.168.1.251.59315 > 192.168.1.254.53: 3333+ A? galileo.local.test.net. (43)
15:26:21.227569 IP 192.168.1.251.59315 > 192.168.1.254.53: 50472+ AAAA? galileo.local.test.net. (43)
15:26:21.233455 IP 192.168.1.254.53 > 192.168.1.251.59315: 3333 1/13/11 A 192.168.1.60 (456)
15:26:21.239551 IP 192.168.1.254.53 > 192.168.1.251.59315: 50472 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:26:21.240019 IP 192.168.1.251.53482 > 192.168.1.254.53: 50545+ A? galileo.local.test.net. (43)
15:26:21.240029 IP 192.168.1.251.53482 > 192.168.1.254.53: 42440+ AAAA? galileo.local.test.net. (43)
15:26:21.245929 IP 192.168.1.254.53 > 192.168.1.251.53482: 50545 1/13/11 A 192.168.1.60 (456)
15:26:21.251927 IP 192.168.1.254.53 > 192.168.1.251.53482: 42440 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:26:21.252107 IP 192.168.1.251.56157 > 192.168.1.254.53: 9501+ A? galileo.local.test.net. (43)
15:26:21.252117 IP 192.168.1.251.56157 > 192.168.1.254.53: 52604+ AAAA? galileo.local.test.net. (43)
15:26:21.258037 IP 192.168.1.254.53 > 192.168.1.251.56157: 9501 1/13/11 A 192.168.1.60 (456)
15:26:21.264126 IP 192.168.1.254.53 > 192.168.1.251.56157: 52604 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)

time ssh root@vm uptime
 15:26:21 up 5 days, 13:10,  4 users,  load average: 0.08, 0.03, 0.03

real    0m0.230s
user    0m0.006s
sys     0m0.006s

With queue:

tcpdump -i br0 -nn host 192.168.1.251 and 192.168.1.254
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 65535 bytes
15:28:36.898396 IP 192.168.1.251.56631 > 192.168.1.254.53: 32214+ PTR? 60.1.168.192.in-addr.arpa. (43)
15:28:36.900594 IP 192.168.1.254.53 > 192.168.1.251.56631: 32214 1/0/1 PTR galileo.local.test.net. (98)
15:28:36.900759 IP 192.168.1.251.55916 > 192.168.1.254.53: 9512+ A? galileo.local.test.net. (43)
15:28:36.906538 IP 192.168.1.254.53 > 192.168.1.251.55916: 9512 1/13/11 A 192.168.1.60 (456)
15:28:41.898083 ARP, Request who-has 192.168.1.254 tell 192.168.1.251, length 28
15:28:41.898319 ARP, Reply 192.168.1.254 is-at d4:ca:6d:25:5e:8a, length 46
15:28:41.979784 IP 192.168.1.251.60792 > 192.168.1.254.53: 44335+ A? galileo.local.test.net. (43)
15:28:41.979798 IP 192.168.1.251.60792 > 192.168.1.254.53: 7494+ AAAA? galileo.local.test.net. (43)
15:28:41.985773 IP 192.168.1.254.53 > 192.168.1.251.60792: 44335 1/13/11 A 192.168.1.60 (456)
15:28:46.983275 IP 192.168.1.251.60792 > 192.168.1.254.53: 44335+ A? galileo.local.test.net. (43)
15:28:46.989198 IP 192.168.1.254.53 > 192.168.1.251.60792: 44335 1/13/11 A 192.168.1.60 (456)
15:28:46.989233 IP 192.168.1.251.60792 > 192.168.1.254.53: 7494+ AAAA? galileo.local.test.net. (43)
15:28:46.995368 IP 192.168.1.254.53 > 192.168.1.251.60792: 7494 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:28:46.996663 IP 192.168.1.251.35555 > 192.168.1.254.53: 44336+ A? galileo.local.test.net. (43)
15:28:46.996674 IP 192.168.1.251.35555 > 192.168.1.254.53: 3369+ AAAA? galileo.local.test.net. (43)
15:28:47.002547 IP 192.168.1.254.53 > 192.168.1.251.35555: 44336 1/13/11 A 192.168.1.60 (456)
15:28:52.001568 IP 192.168.1.251.35555 > 192.168.1.254.53: 44336+ A? galileo.local.test.net. (43)
15:28:52.007379 IP 192.168.1.254.53 > 192.168.1.251.35555: 44336 1/13/11 A 192.168.1.60 (456)
15:28:52.007410 IP 192.168.1.251.35555 > 192.168.1.254.53: 3369+ AAAA? galileo.local.test.net. (43)
15:28:52.013462 IP 192.168.1.254.53 > 192.168.1.251.35555: 3369 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:28:52.021389 IP 192.168.1.251.45126 > 192.168.1.254.53: 1509+ A? galileo.local.test.net. (43)
15:28:52.021400 IP 192.168.1.251.45126 > 192.168.1.254.53: 54702+ AAAA? galileo.local.test.net. (43)
15:28:52.027449 IP 192.168.1.254.53 > 192.168.1.251.45126: 1509 1/13/11 A 192.168.1.60 (456)
15:28:57.023015 IP 192.168.1.251.45126 > 192.168.1.254.53: 1509+ A? galileo.local.test.net. (43)
15:28:57.029083 IP 192.168.1.254.53 > 192.168.1.251.45126: 1509 1/13/11 A 192.168.1.60 (456)
15:28:57.029165 IP 192.168.1.251.45126 > 192.168.1.254.53: 54702+ AAAA? galileo.local.test.net. (43)
15:28:57.035279 IP 192.168.1.254.53 > 192.168.1.251.45126: 54702 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:28:57.036146 IP 192.168.1.251.36885 > 192.168.1.254.53: 17852+ A? galileo.local.test.net. (43)
15:28:57.036156 IP 192.168.1.251.36885 > 192.168.1.254.53: 17879+ AAAA? galileo.local.test.net. (43)
15:28:57.042324 IP 192.168.1.254.53 > 192.168.1.251.36885: 17852 1/13/11 A 192.168.1.60 (456)
15:29:02.040381 IP 192.168.1.251.36885 > 192.168.1.254.53: 17852+ A? galileo.local.test.net. (43)
15:29:02.046309 IP 192.168.1.254.53 > 192.168.1.251.36885: 17852 1/13/11 A 192.168.1.60 (456)
15:29:02.046355 IP 192.168.1.251.36885 > 192.168.1.254.53: 17879+ AAAA? galileo.local.test.net. (43)
15:29:02.052395 IP 192.168.1.254.53 > 192.168.1.251.36885: 17879 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:29:02.052618 IP 192.168.1.251.55202 > 192.168.1.254.53: 48400+ A? galileo.local.test.net. (43)
15:29:02.052627 IP 192.168.1.251.55202 > 192.168.1.254.53: 52566+ AAAA? galileo.local.test.net. (43)
15:29:02.058690 IP 192.168.1.254.53 > 192.168.1.251.55202: 48400 1/13/11 A 192.168.1.60 (456)
15:29:07.051830 IP 192.168.1.251.55202 > 192.168.1.254.53: 48400+ A? galileo.local.test.net. (43)
15:29:07.057750 IP 192.168.1.254.53 > 192.168.1.251.55202: 48400 1/13/11 A 192.168.1.60 (456)
15:29:07.057803 IP 192.168.1.251.55202 > 192.168.1.254.53: 52566+ AAAA? galileo.local.test.net. (43)
15:29:07.063956 IP 192.168.1.254.53 > 192.168.1.251.55202: 52566 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:29:07.067052 IP 192.168.1.251.51428 > 192.168.1.254.53: 23857+ A? galileo.local.test.net. (43)
15:29:07.067064 IP 192.168.1.251.51428 > 192.168.1.254.53: 36144+ AAAA? galileo.local.test.net. (43)
15:29:07.072946 IP 192.168.1.254.53 > 192.168.1.251.51428: 23857 1/13/11 A 192.168.1.60 (456)
15:29:12.071290 IP 192.168.1.251.51428 > 192.168.1.254.53: 23857+ A? galileo.local.test.net. (43)
15:29:12.077184 IP 192.168.1.254.53 > 192.168.1.251.51428: 23857 1/13/11 A 192.168.1.60 (456)
15:29:12.077214 IP 192.168.1.251.51428 > 192.168.1.254.53: 36144+ AAAA? galileo.local.test.net. (43)
15:29:12.083324 IP 192.168.1.254.53 > 192.168.1.251.51428: 36144 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:29:12.092956 IP 192.168.1.251.57697 > 192.168.1.254.53: 7458+ A? galileo.local.test.net. (43)
15:29:12.092968 IP 192.168.1.251.57697 > 192.168.1.254.53: 5630+ AAAA? galileo.local.test.net. (43)
15:29:12.099539 IP 192.168.1.254.53 > 192.168.1.251.57697: 7458 1/13/11 A 192.168.1.60 (456)
15:29:17.093128 IP 192.168.1.251.57697 > 192.168.1.254.53: 7458+ A? galileo.local.test.net. (43)
15:29:17.098918 IP 192.168.1.254.53 > 192.168.1.251.57697: 7458 1/13/11 A 192.168.1.60 (456)
15:29:17.098955 IP 192.168.1.251.57697 > 192.168.1.254.53: 5630+ AAAA? galileo.local.test.net. (43)
15:29:17.105166 IP 192.168.1.254.53 > 192.168.1.251.57697: 5630 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:29:17.105442 IP 192.168.1.251.44077 > 192.168.1.254.53: 30193+ A? galileo.local.test.net. (43)
15:29:17.105450 IP 192.168.1.251.44077 > 192.168.1.254.53: 56817+ AAAA? galileo.local.test.net. (43)
15:29:17.111389 IP 192.168.1.254.53 > 192.168.1.251.44077: 30193 1/13/11 A 192.168.1.60 (456)
15:29:22.110393 IP 192.168.1.251.44077 > 192.168.1.254.53: 30193+ A? galileo.local.test.net. (43)
15:29:22.116353 IP 192.168.1.254.53 > 192.168.1.251.44077: 30193 1/13/11 A 192.168.1.60 (456)
15:29:22.116384 IP 192.168.1.251.44077 > 192.168.1.254.53: 56817+ AAAA? galileo.local.test.net. (43)
15:29:22.122450 IP 192.168.1.254.53 > 192.168.1.251.44077: 56817 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:29:22.122927 IP 192.168.1.251.58699 > 192.168.1.254.53: 29974+ A? galileo.local.test.net. (43)
15:29:22.122936 IP 192.168.1.251.58699 > 192.168.1.254.53: 5448+ AAAA? galileo.local.test.net. (43)
15:29:22.128827 IP 192.168.1.254.53 > 192.168.1.251.58699: 29974 1/13/11 A 192.168.1.60 (456)
15:29:27.110208 ARP, Request who-has 192.168.1.254 tell 192.168.1.251, length 28
15:29:27.110474 ARP, Reply 192.168.1.254 is-at d4:ca:6d:25:5e:8a, length 46
15:29:27.127859 IP 192.168.1.251.58699 > 192.168.1.254.53: 29974+ A? galileo.local.test.net. (43)
15:29:27.133685 IP 192.168.1.254.53 > 192.168.1.251.58699: 29974 1/13/11 A 192.168.1.60 (456)
15:29:27.133714 IP 192.168.1.251.58699 > 192.168.1.254.53: 5448+ AAAA? galileo.local.test.net. (43)
15:29:27.139790 IP 192.168.1.254.53 > 192.168.1.251.58699: 5448 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)
15:29:27.140008 IP 192.168.1.251.51797 > 192.168.1.254.53: 17764+ A? galileo.local.test.net. (43)
15:29:27.140016 IP 192.168.1.251.51797 > 192.168.1.254.53: 21901+ AAAA? galileo.local.test.net. (43)
15:29:27.146065 IP 192.168.1.254.53 > 192.168.1.251.51797: 17764 1/13/11 A 192.168.1.60 (456)
15:29:32.143514 IP 192.168.1.251.51797 > 192.168.1.254.53: 17764+ A? galileo.local.test.net. (43)
15:29:32.149311 IP 192.168.1.254.53 > 192.168.1.251.51797: 17764 1/13/11 A 192.168.1.60 (456)
15:29:32.149403 IP 192.168.1.251.51797 > 192.168.1.254.53: 21901+ AAAA? galileo.local.test.net. (43)
15:29:32.155633 IP 192.168.1.254.53 > 192.168.1.251.51797: 21901 1/13/11 AAAA fe80::21f:d0ff:fe5f:c095 (468)

time ssh root@vm uptime
 15:29:12 up 5 days, 13:13,  4 users,  load average: 0.00, 0.01, 0.01

real    0m35.265s
user    0m0.009s
sys     0m0.004s

rzirzi · Fri Jul 20, 2012 2:14 pm

I have to confirm MT DNS caching problem

MikroTik team - plase DO SOMETHING TO REPAIR THIS PROBLEM.
I think it's UPD packet size problem.

Fri Jul 20, 2012 2:21 pm

We still are unable to reproduce this. Can you give us detailed info about your setup, and how you can repeat the issue?

rzirzi · Fri Jul 20, 2012 7:49 pm

I think MikroTik can't use feature: http://msmvps.com/blogs/systmprog/archi ... d-udp.aspx
Why MikroTik-to-MikroTik never use TCP?
it should: "UDP packets are smaller in size. Can't be greater then 512 bytes. So any application needs data to be transffered greter than 512 bytes uses TCP"
And also: http://tools.ietf.org/html/rfc5966
"DNS resolvers and recursive servers MUST support UDP, and SHOULD support TCP, for sending (non-zone-transfer) queries."
"In the absence of EDNS0 (Extension Mechanisms for DNS 0) (see below),
the normal behaviour of any DNS server needing to send a UDP response
that would exceed the 512-byte limit is for the server to truncate
the response so that it fits within that limit and then set the TC
flag in the response header. When the client receives such a
response, it takes the TC flag as an indication that it should retry
over TCP instead."

When I'm connected to MT directly from my PC - I can see that opening some sites (for instance yahoo.com) is generating TCP DNS requests to MT. And it responds good. When I'm connected to AP over PPPoE from another MT (CPE) - there is NO DNS traffic over TCP and there is a problem with opening some sites!!!
Please test it Normis, and find solution. Thanks.

rzirzi · Mon Jul 23, 2012 10:00 pm

We have explained it very precisely.
MikroTik team - what are You planning to do?

Tue Jul 24, 2012 4:00 pm

We want to fix problem, as soon as possible, as different users reported the problem exists.

We have tried to repeat the same problem countless times,
- on different boards;
- on different DNS servers;
- different RouterOS versions;
- on different networks;
- on different computers and OS;
MikroTik network is built on DNS caches, and we have never experienced problems with DNS resolving on any page (as well such as yahoo, windows update, etc.).

Perhaps there is something to do with the settings on /ip dns and specific DNS server, specific OS etc..

We will very appreciate, if anybody can post step by step instructions, that 100% of time (at least 50% is fine) can produce the issue, post your /ip dns settings. Thank you very much for the cooperation.

rzirzi · Tue Jul 24, 2012 9:23 pm

We want to fix problem, as soon as possible, as different users reported the problem exists.

We will very appreciate, if anybody can post step by step instructions, that 100% of time (at least 50% is fine) can produce the issue, post your /ip dns settings. Thank you very much for the cooperation.

Have You read my post?! If not - please read, if not - please read again!

I think MikroTik can't use feature: http://msmvps.com/blogs/systmprog/archi ... d-udp.aspx
Why MikroTik-to-MikroTik never use TCP?
it should: "UDP packets are smaller in size. Can't be greater then 512 bytes. So any application needs data to be transffered greter than 512 bytes uses TCP"
And also: http://tools.ietf.org/html/rfc5966
"DNS resolvers and recursive servers MUST support UDP, and SHOULD support TCP, for sending (non-zone-transfer) queries."
"In the absence of EDNS0 (Extension Mechanisms for DNS 0) (see below),
the normal behaviour of any DNS server needing to send a UDP response
that would exceed the 512-byte limit is for the server to truncate
the response so that it fits within that limit and then set the TC
flag in the response header. When the client receives such a
response, it takes the TC flag as an indication that it should retry
over TCP instead."

When I'm connected to MT directly from my PC - I can see that opening some sites (for instance yahoo.com) is generating TCP DNS requests to MT. And it responds good. When I'm connected to AP over PPPoE from another MT (CPE) - there is NO DNS traffic over TCP and there is a problem with opening some sites!!!
Please test it Normis, and find solution. Thanks.

Devil · Wed Jul 25, 2012 11:06 am

We will very appreciate, if anybody can post step by step instructions, that 100% of time (at least 50% is fine) can produce the issue, post your /ip dns settings. Thank you very much for the cooperation.

I think it got more to do with different links, dns forwarders and/or latencies. If you are interested, i could setup a sstp server, confirm that the problem still exists over the tunnel and give you the credentials so you could try to reproduce it over my link. worth a try i suppose.

rzirzi · Wed Jul 25, 2012 4:15 pm

We want to fix problem, as soon as possible, as different users reported the problem exists.

We will very appreciate, if anybody can post step by step instructions, that 100% of time (at least 50% is fine) can produce the issue, post your /ip dns settings. Thank you very much for the cooperation.

ONE VERY IMPORTANT SETTING: Max UDP Packet Size: 512
- i know that we can set 4096, but with 4096 is big problem with old CPE connected via PPPoE.
BUT, when Max UDP packet size is 512 and station id directly connected to MikroTik - it uses TCP for longer DNS replies.
The problem is as I described before: MikroTikOS DOES NOT USE TCP DNS QUERIES to other MikroTik DNS (only UDP) - WHY????
Could you repair this problem please?!

ankostis · Sat Aug 18, 2012 11:07 pm

I believe the cause for the my DNS problem is that MikroTik's DNS tries all defined dn-servers in round-robin fashion.
The standard behavior is to try the 1st, and if that fails, only then to proceed with the next one.

To be clear, the problem of mine is manifested by getting indeterminate bad resolutions for private domain-names.
A dname at occasions might resolve to an IP, or fail to.
And even if it resolves OK, later it may stop resolving, until flushing MikroTik's dns-cache, which starts the process all over again.
It may be happening only when CNAMEs are involved, or not, i can't be sure about that.

MY SETUP
-----------

[MikroTik] /ip dns> export
# aug/18/2012 22:00:38 by RouterOS 5.20
# software id = 12HY-1CWN
#
/ip dns
set allow-remote-requests=yes cache-max-ttl=1w cache-size=1024KiB max-udp-packet-size=4096 servers=10.176.3.130,8.8.8.8

* My DNS-server:               10.176.3.130 ## Provides dnames for the private network 'mydomain.wn', along with the internet.
* Google's DNS-server(B):      8.8.8.8      ## For backup, whenever the link to my DNS breaks (or so i assumed...).
* My PC's IP:                  10.176.9.71
* MikroTik IP:                 10.176.9.1
* domain:                      home.mydomain.wn
* search:                      home.mydomain.wn mydomain.wn
* dns-record@home.mydomain.wn: gw-for-lan AA    10.176.9.1
* dns-record@home.mydomain.wn: router     CNAME gw-for-lan.home.mydomain.wn

Then i run:

ping router

and it occasionally fails to resolv.
I tried 'dig router +search', host, nslookup etc, and they all behave similarly.

When i removed the 2nd dns-server@8.8.8.8, MikroTik's dns works correctly at all times.
If i added a 3rd or even more dns-servers, the frequency of failures increased!

Then i looked with wireshark and with cache flushed i got something like this (i can supply you the actual tcpdump):

#  MYPC            MIKROTIK       DNS_SERVER   INFO
1. 10.176.9.71 --> 10.176.9.1                  Standard query A: router.home.mydomain.wn
2.                 10.176.9.1 --> 8.8.8.8      Standard query A: router.home.mydomain.wn
3.                 10.176.9.1 <-- 8.8.8.8      Standard query response: No such name
4. 10.176.9.71 <-- 10.176.9.1                  Standard query response: No such name

5. 10.176.9.71 --> 10.176.9.1                  Standard query A: router.mydomain.wn
6.                 10.176.9.1 --> 10.176.3.130 Standard query A: router.mydomain.wn
7.                 10.176.9.1 <-- 10.176.3.130 Standard query response: No such name
8. 10.176.9.71 <-- 10.176.9.1                  Standard query response: No such name

9. 10.176.9.71 --> 10.176.9.1                  Standard query A: router
0.                 10.176.9.1 --> 8.8.8.8      Standard query A: router
1.                 10.176.9.1 <-- 8.8.8.8      Standard query response: No such name
2. 10.176.9.71 <-- 10.176.9.1                  Standard query response: No such name

If the query #2 were against my dns-server@10.176.3.130, it would have resolved ok,
and it wouldn't proceed to ask the next non-existent names.
But since MikroTik's DNS asks cascade dns-servers in a round-robin (at least in my occasion), it is indeterminate which server is to be asked next.

So the problem appears when the client performs multiple queries using 'domain search',
it is indeed non-deterministic, and it gets worse as more dns-servers are added into the configuration of MikroTik.

Also i noticed that the problem probably arises AFTER some bad-resolutions, that is,
immediately after flushing the cache, all names resolve OK until i enter a non non-existent.
But i cannot be sure about that.

Hope that helps to fix the bug, if indeed that round-robin behavior is unintentional.

Devil · Mon Aug 20, 2012 9:50 am

@ankostis
I don't know about linux but in windows, this is indeed not a normal behavior and second dns will only be used if the first one has failed. that being said, i believe this is a different story. it's a good thing to specify a backup dns server but in your case, you should supply that via dhcp to each client rather than adding it to mikrotik dns server list. i believe this is the standard approach that will eliminate lots of potential issues like this. So i think your problem, is not really related to this bug since as i explained in This post , when the bug happens, you'll see a standard query response with no error(Flags: 0x8180) but answer RRS is 0.

ankostis · Mon Aug 20, 2012 10:55 am

@devil
thank you for the suggested workround. It should solve my problem

Regarding MikroTik's dns functon,
* as you said, it should proceed to the next DNS only when the previous one has failed - not when it responds with 'No such name" response", which is a valid response.
* And even if this round-robin behavior were "by design", it should do that consistently, and not bounce from one behavior to the other. Mikrotik instead falls back to round-robin only after a upstream server responds with a "No such name", and then it sticks back to asking the same server as soon as it receives a query-response for an existent dname (checked it with tcpdump).

I believe that the proper behavior for the task is similar to that of dnsmaq[1], since this is a well tested GPL-software doing exactly the forwarding-dns-server job we are talking about.

[1] http://www.thekelleys.org.uk/dnsmasq/doc.html

fuzzbawl · Thu Aug 23, 2012 12:33 am

Our call center has been keeping track of customers having issues connecting to our MikroTik DNS routers. These home routers are always the ones customers with issues seem to have. The connections are usually in this manner:

Home Wifi Router -> MikroTik Wireless CPE -> PPPoE session on CPE -> MikroTik PPPoE Router/Server -> OSPF to Other routers in network

Our PPPoE Router/Servers are allowing remote requests for DNS and that's what the clients are issues in the PPPoE parameters. I can draw up a Visio diagram if desired.

List of Customer Wifi Routers that have issues with DNS resolution when using MikroTik router as server:
Belkin F9K110 v1
Belkin n750db
Cisco WRV210
LinkSys E1000
Netgear WPN824 v3
Netgear WRN2000

fuzzbawl · Fri Aug 24, 2012 5:56 pm

I was able to duplicate this at home.

Environment:
Comcast Internet
MikroTik RB2011LS w/RouterOS 5.20, on IP 192.168.42.1
Linux server running Bind 9.7, on IP 192.168.42.3
Mac Pro w/Mac OS X 10.8.0 (my workstation)
Macbook Pro w/Mac OS X 10.8.0 (my laptop)

Network layout:
ether1 of RB2011 plugged into Comcast modem
ether2 through ether10 are members of bridge1
bridge1 has IP of "192.168.42.1/24" and hands out DHCP
NAT Masquerade is obviously enabled

RB2011 DNS Settings:
servers: 192.168.42.3
dynamic-servers:
allow-remote-requests: yes
max-udp-packet-size: 4096
cache-size: 2048KiB
cache-max-ttl: 1w
cache-used: 336KiB

I was trying to browse the Xbox forums from my workstation and noticed that several images didn't show up. They all belonged to the same domain, nxeassets.xbox.com. So I started my investigative work. My workstation DNS settings point to the RB2011.

I right clicked the broken image icon and chose "Open image in new tab". Chrome tried to open the image, however it failed with an error of "The server at nxeassets.xbox.com can't be found, because the DNS lookup failed." The technical error at the bottom is "Error 105 (net::ERR_NAME_NOT_RESOLVED): Unable to resolve the server's DNS address."

I ran "dig nxeassets.xbox.com" from my desktop, as you can see in the attachment "workstation_dig.rtf" under "ATTEMPT 1" header. The result came back normal so I tried reloading the image, but it came back with the same error. I then did the dig again, which you can see under the "ATTEMPT 2" header. That result came back with nothing so I did the dig a third time as seen under "ATTEMPT 3" header. This time the DNS records resolved but the image still didn't load, same error as before. I then went to a server at work that had Bind running on it and did a "dig nxeassets.xbox.com" on that. The result is in the "server_dig.rtf" file and it was able to load the image just fine.

I went to my laptop, which has it's DNS settings set to my Linux box with Bind, and loaded the image which came up fine. This whole time I had winbox open, watching the DNS cache window.

In "RB2011-1.png" you can see the records in DNS cache. Since they are CNAME records, I looked up the other entries in the DNS cache and took screenshots of each as seen in the other two PNG files. The "msxb.vo.llnwd.net" actually resolves to a set of IP addresses, however the "content.xbox.com.edgesuite.net" resolves to yet another CNAME record of "a940.g.akamai.net". I looked for "a940.g.akamai.net" in the DNS cache but I couldn't find it. I went back to watching the DNS cache for record "nxeassets.xbox.com" to see what happens when it's TTL finally was met so that I could hit refresh a few seconds later and see if my image would then load. I noticed something VERY interesting. As the TTL timer ticked down, it got to 00:00:00 and then went UP. It counted up to 00:00:03 and then reset to zero and counted UP again to 00:00:03. It reset again and then disappeared. I thought that was really odd.

Now that the record was gone from DNS cache, I decided to refresh the page. That worked, all of the images loaded and I was able to see the entire page. Unfortunately I wasn't able to get a Wireshark grab of the incident, but I will try to do that next time it happens.

fuzzbawl · Fri Aug 24, 2012 6:02 pm

These are the other two relevant files to my post above. Sadly the attachment limit is 3 and I needed to attach more files!

flyingclover · Mon Aug 27, 2012 12:23 pm

Registered an account just to reply to this thread

Our Mikrotik Router PC is having the same problem too, most often when resolving mail.yahoo.com (and yahoo messenger), and s3 amazon .

The Version is 5.18, it is a dual core xeon with 2GB RAM, it run as a router and transparent web proxy.

I was trying to change Max UDP and dns cache size but it doesn't solve the problem.

What we did when the problem occur is to flush the dns cache in client's computer (ipconfig /flushdns). More often than not it would resolve successfully, but at some day the problem occur too often, it become annoying.

Meanwhile we set client's DNS to google public DNS.

I will gladly post the router setting if that would help you diagnose the problem - but there is nothing fancy in the setting.

remit · Sat Sep 08, 2012 7:19 pm

I can SHOW it to someone at Mikrotik any time they want. I have one site that one customer always complains about www.theoatmeal.com. The page loads fine, but the pictures dont, as the links have expired.

We can do a remote session any time, email me a jake at cranertech dot com, or call my cell, it's listed on my webpage.

TDPsGM · Sat Sep 08, 2012 8:58 pm

Another VERY consistent reoccurring issue is using the 'Amazons S3 Browser' to go directly to your 'Amazon S3 Storage Buckets' to manage your files!

If I Flush My MicroTik Router DNS Cache Then we are off to the races . . . for a couple of minutes anyway. Close it and come back, and we are back to square one and I have to Flush the DNS cache again. VERY VERY annoying.

amazonS3browserFailure.png

amazonS3browserFailure_Diagnosics.png

If I pull the MicroTik Router out and insert an off the shelf router (Dlink Dir-655) we have no issues at all.

I have applies just 1 DNS server to the DNS settings ( 8.8.8.8 ) to prevent what appears to be a "Round Robin" issue (I think I read that right) aggravating the situation, but I am not sure that that has really helped any.

FYI - By far the majority of problems I have, have to do with Amazon! Be it landing on a site that has a video that streams from it, or trying to manage my S3 account. Amazon resolutions are a REAL B!tch with regards to this issue.

TRY THE Amazon S3 Browser ====> wow, is that a pain to work with through this router! I am ALWAYS clearing the DNS cache to get it to work!

Hope that gives you something useful to use and test with.

TDPsGM · Sat Sep 08, 2012 9:16 pm

The above images were with regard to a specific bucket.

I don't know if this helps, but
this is just a couple of minutes later, and it can't resolve with my Amazon S3 Account as a whole now without clearing the DNS Cache:

amazonS3browserFailure2.png

amazonS3browserFailure_Diagnosics2.png

subscope · Sun Sep 09, 2012 1:03 pm

Hi Everyone,

I have the same problem with several RB routers after os upgrade from 4.x...
Mikrotik Team please provide us a solution for this very annoying problem!

Thx

remit · Sun Sep 09, 2012 8:10 pm

I see that a few people are caching googles DNS (8.8.8.8 and 8.8.4.4). Any chance this is related? I am caching Google's DNS while experiencing this issue. Is anyone caching something else and seeing it too?

fuzzbawl · Sun Sep 09, 2012 9:27 pm

I had the issue caching Google, OpenDNS, Comcast and my own Bind server with caching enabled.

TDPsGM · Mon Sep 10, 2012 1:04 am

I am seeing it with OpenDNS right now too:
208.67.222.222
208.67.220.220

janisk · Mon Sep 10, 2012 4:29 pm

I think it got more to do with different links, dns forwarders and/or latencies. If you are interested, i could setup a sstp server, confirm that the problem still exists over the tunnel and give you the credentials so you could try to reproduce it over my link. worth a try i suppose.

of course we are interested in this issue.

I have windows 7 laptop with IPv6 disabled that is running script for hours with no success (that is failure), so maybe it would be possible to arrange tunnel to run that over your network, maybe then we could see the issue and fix it. Please send details to support@mikrotik.com

remit · Mon Sep 10, 2012 6:56 pm

It seems to me like after a refresh, it takes DAYS, if not WEEKS to show up again. I flushed it yesterday, and still all is working properly. I will try to check it daily and get a timeline, and contact support as soon as it goofs again.

remit · Tue Sep 11, 2012 2:04 am

Okay, looks like I am starting to have expired DNS results on the site I was watching. I will email support my contact info.

TDPsGM · Mon Oct 01, 2012 12:13 am

Have we made any headway on this issue?

It's been kinda quiet for a while, and I am regularly having to still clear the DNS cache to make it work.

Thanks.

remit · Mon Oct 01, 2012 12:20 am

I have not heard back from support. Maybe someone else should give it a try.

TDPsGM · Mon Oct 01, 2012 1:00 am

I sure hope they can get on this as I am starting to think about walking away. I don't want to, but I am tired of having to flush my DNS Cache every time I touch ANYTHING that has to do with information being streamed from Amazon S3 Storage.

I basically have to leave Winbox open and minimized so that i can get to it quickly. That is no way to run this setup.

voxframe · Mon Oct 01, 2012 2:13 am

I know this isn't what people want to hear, and it may not be the best solution...

But I've cleared the problem on my network simply by getting rid of OpenDNS and Google's DNS.

I am now using a mix of DNS servers that were listed in the GRC DNS benchmark kit. I think I'm using Peer1 and H.E. for my area. Maybe Level3?

Either way I've had this problem for nearly 2 years now using OpenDNS and then suddenly I've noticed Google's DNS doing the same garbage. I thought it was a problem more with them so I moved away and the problem is fixed. I loved using OpenDNS for the statistics and stuff, but it just never worked correctly because of this.

So a quick fix, get away from large DNS servers. Stick with one from your upstream provider?

remit · Tue Oct 02, 2012 2:55 am

How long have you been trouble free since switching away from Google DNS servers? Why would only tiks goof up using Google DNS, meanwhile my computer caches just fine?

voxframe · Tue Oct 02, 2012 3:03 am

Trouble free the second I moved away from Google.

It first started with OpenDNS... Then I moved to Google for about a year or so... Then I started noticing problems in the last couple months and switched away.

Now no more problems.

I have a feeling OpenDNS and Google treat their records differently than what a "normal" response should be.

From what I understood with OpenDNS (Others asked about this on their forums, maybe it's related) was that OpenDNS was messing with the TTL values for sites hosted with large distributed systems (AKAMI etc). So this affected all kinds of major players like Yahoo, Youtube, Google, MSN, etc.

Maybe it's not the same issue, but it's just what I've experienced. Once I moved away from OpenDNS all the problems stopped. But it was only relatively recently when I noticed problems with Google as well. Not exactly the same, but similar.

remit · Tue Oct 02, 2012 3:05 am

Hell its worth a try! Still gotta be something tik related in there somewhere though....

pelish · Tue Oct 02, 2012 11:51 am

Hello,
I have the same problem with DNS on my network. It starts when I upgraded my x86 DNS server from 3.30 to 5.20. After downgrading that problem still remain. It finaly disappear when I changed max-udp-size back to 512 (since it was 4096 after upgrade to 5.20).

So for me - until this problem will be there I must use mikrotik 3.30 for DNS servers

remit · Sun Oct 07, 2012 1:54 am

Is it just me, or did this issue disappear? I have not made any changes, and I have not been able to reproduce this issue since my last post of emailing support.

pelish · Mon Oct 15, 2012 12:05 pm

Hello,
I would like to know if new version 5.21 fixed this problem. I am asking because of this line in Changelog:

*) dns - fix empty response;

Tahnk you for reply

janisk · Mon Oct 15, 2012 12:13 pm

it was a proposed possible fix for the issue. Please check if any of you still have issues.

sandov63 · Tue Oct 16, 2012 6:29 am

solved chek this

nslookup www.hotmail.com
;; Got SERVFAIL reply from 192.168.30.2, trying next server
;; Truncated, retrying in TCP mode.
Server: 192.168.30.2
Address: 192.168.30.2#53

Non-authoritative answer:
www.hotmail.com canonical name = dispatch.kahuna.glbdns.microsoft.com.
Name: dispatch.kahuna.glbdns.microsoft.com
Address: 65.55.72.199
Name: dispatch.kahuna.glbdns.microsoft.com
Address: 65.55.72.183

ooo retrying in tcp mode lol

sandov63 · Tue Oct 16, 2012 6:49 am

Normally, ordinary queries use UDP, and zone transfers use TCP.

However, DNS limits UDP queries and responses to about 500 bytes. If a
response would be larger than that, the server sends back up to 500 bytes
and sets the "truncated" flag. The client is then supposed to perform the
same query again using TCP, which is almost unlimited in the size of
response it can send (the limit is typically only exceeded by web hosting
organizations that feel the need to create a PTR record for every A record,
and they have thousands of names pointing to the same address)

Ivoshiee · Wed Oct 17, 2012 10:05 pm

I run v5.21 and I still get that described error.
Output of a .bat script:

Start time: 22:00:56,82

Windows IP Configuration

         1.0.0.127.in-addr.arpa
         ----------------------------------------
         Record Name . . . . . : 1.0.0.127.in-addr.arpa.
         Record Type . . . . . : 12
         Time To Live  . . . . : 576622
         Data Length . . . . . : 4
         Section . . . . . . . : Answer
         PTR Record  . . . . . : localhost


         Record Name . . . . . : 1.0.0.127.in-addr.arpa.
         Record Type . . . . . : 12
         Time To Live  . . . . : 576622
         Data Length . . . . . : 4
         Section . . . . . . . : Answer
         PTR Record  . . . . . : fazher.com


         fazher.com
         ----------------------------------------
         Record Name . . . . . : fazher.com
         Record Type . . . . . : 1
         Time To Live  . . . . : 576622
         Data Length . . . . . : 4
         Section . . . . . . . : Answer
         A (Host) Record . . . : 127.0.0.1


         mail.yahoo.com
         ----------------------------------------
         Record Name . . . . . : mail.yahoo.com
         Record Type . . . . . : 5
         Time To Live  . . . . : 17
         Data Length . . . . . : 4
         Section . . . . . . . : Answer
         CNAME Record  . . . . : login.yahoo.com


         localhost
         ----------------------------------------
         Record Name . . . . . : localhost
         Record Type . . . . . : 1
         Time To Live  . . . . : 576622
         Data Length . . . . . : 4
         Section . . . . . . . : Answer
         A (Host) Record . . . : 127.0.0.1


Number of runs before the failure = 954
End time: 22:46:01,56

gsloop · Thu Oct 18, 2012 5:29 pm

What's so goofy about the "fix" that's supposedly in 5.21 is that MikroTik never even admits there is a problem, from what I can see. They deny, deny, deny.

Then, suddenly
"Aha! That thing that was never broken, and implied was a user problem - well it's FIXED! Aren't we great!"

I'd laugh if it wasn't so utterly bat**it insane.

Fri Oct 19, 2012 9:52 am

I don't see anyone denying it. I see these two responses from MikroTik in this topic (after several people confirmed they have such issues):

1. We still are unable to reproduce this.
2. of course we are interested in this issue.

rzirzi · Fri Oct 19, 2012 10:33 am

I don't see anyone denying it. I see these two responses from MikroTik in this topic (after several people confirmed they have such issues):
1. We still are unable to reproduce this.
2. of course we are interested in this issue.

We have desribed You fully in detail how to reproduce it. Maybe You should employ any network specialist ;P

Fri Oct 19, 2012 10:35 am

I don't see anyone denying it. I see these two responses from MikroTik in this topic (after several people confirmed they have such issues):
1. We still are unable to reproduce this.
2. of course we are interested in this issue.
We have desribed You fully in detail how to reproduce it. Maybe You should employ any network specialist ;P

Your fully described instructions are not giving any result. This means that there are other variables at play.

janisk · Fri Oct 19, 2012 2:03 pm

configuration as follows:

DNS server <---> RouterOS router1 DNS <---> RouterOS router2 DNS <---> laptop runnin win7 with bat script- no joy. It was running for week w/o problems

what i am doing wrong?

win7 have ipv6 disabled.

DNS to router1 ipv6 dns requests
router1 to router2 ipv4 dns requests
router2 to laptop ipv4 dns requests

gsloop · Wed Oct 24, 2012 8:13 pm

I don't see anyone denying it. I see these two responses from MikroTik in this topic (after several people confirmed they have such issues):

I see comments in the changelog in 5.21 that seem to indicate you "fixed" something in DNS.

But since no-one at MikroTik will talk about it, and it sure is being taken as a "fix" to a problem MikroTik claims not to have, it seems a little odd.

See here:
http://forum.mikrotik.com/viewtopic.php ... 59#p337800

remit · Thu Oct 25, 2012 7:29 am

configuration as follows:

DNS server <---> RouterOS router1 DNS <---> RouterOS router2 DNS <---> laptop runnin win7 with bat script- no joy. It was running for week w/o problems what i am doing wrong?

win7 have ipv6 disabled.

DNS to router1 ipv6 dns requests
router1 to router2 ipv4 dns requests
router2 to laptop ipv4 dns requests

I wish I could be more help here, but this has been hard for me to track down. If I reboot my windows machine or manually flush my dns cache, it resolves the issue. So its almost like the Mikrotik is HANDING OUT too long of TTLs on queries? I always test it with www.theoatmeal.com because, well it makes me giggle and it uses amazonaws

Possibly try leaving the win7 machine on for an extended period to make it happen?

janisk · Thu Oct 25, 2012 9:15 am

i had it running for week (as in 7 days). i even hooked that laptop to a UPS
will set up it once more in near future.

remit · Thu Oct 25, 2012 7:47 pm

i had it running for week (as in 7 days). i even hooked that laptop to a UPS
will set up it once more in near future.

Ive had it happen from anywhere from 2 days to 2 weeks. Sorry, I know this isn't much info to go off of, but I don't really understand DNS queries enough to be of more help. I do know that this is an issue though, and have had it happen to me (Only while pulling DNS from mikrotik devices).

Devil · Mon Nov 05, 2012 2:10 pm

Ah, for some reason i did not get an email about any new post in this topic since my last post. and also, ever since i implemented my workaround, i completely forgot about this issue.
@mikrotik support: I'm very sorry that i kept you hanging like this. i will test the latest routeros version (5.21) that supposedly have the fix for this and keep you update.

fuzzbawl · Tue Nov 06, 2012 8:27 am

Since upgrading my test environment to 5.21, I have yet to see the issue come up again. I will keep monitoring however and let you know what I see.

Devil · Tue Nov 06, 2012 12:48 pm

I did a last minute test before upgrading from v5.18 to ensure that i can still reproduce the bug, and i could. however, after upgrading to v5.21 and hours of trying, i was not able to reproduce this anymore. i've also set my i7 filter rule to log any empty dns response packet. but after about a day, it still hasn't matched a single one. Well done guys, it does indeed appear that you've fixed this bug.

@Ivoshiee:
unfortunately the result that you provided, appears to be a false-positive result. or at least it doesn't prove that the bug still exists, as there is a valid entry for mail.yahoo.com in the log you provided. were you running multiple instance of that script at the same time on the same machine? the script is not designed for that and it would mostly likely give you not so reliable results or even false-positive ones.

@remit
did you experience this after upgrading to v5.21? if that's so, the next time that this happens, instead of flushing the cache or restarting your pc, could you open cmd and type 'ipconfig /displaydns' and post the output?

@Mikrotik support
did you find the root cause of this or does this version only introduce some sort of workaround to avoid sending empty responses? could you please share with us some details about this bug and how it was being triggered? i want to know how close my hypothesis was.

janisk · Wed Nov 07, 2012 11:22 am

logic of reply processing was changed. So, replies are processed in another way now.

Devil · Wed Nov 07, 2012 12:51 pm

logic of reply processing was changed. So, replies are processed in another way now.

Ah, ok. thanks for the info. and of course for fixing this bug :)

brandonrossl · Wed Nov 07, 2012 5:40 pm

Looks like I have no reason NOT to upgrade to 5.21 tonight!

flyingclover · Sat Dec 15, 2012 4:15 am

Kudos for mikrotik teams.

I'll try to upgrade the OS version ASAP.

rferroni · Fri Jan 04, 2013 5:36 pm

thanks everyone!
we`ll start to upgrade a lots of mk`s!

flyingclover · Wed Jan 16, 2013 1:06 pm

Just a report

After almost 1 month using the new mikrotik OS, the problem never occur.

Mikrotik 5.22

Thanks

Wed Jan 16, 2013 10:49 pm

Are these fixes in 6.0rc6 ??

TDPsGM · Fri Jan 18, 2013 6:52 pm

FYI - The caching issues that I was having seemed to have been fixed. Thanks for the fix guys! (running v5.22 now with this post).

Mon Jan 21, 2013 1:29 pm

This issue still exists in 6.0rc7.

Mikrotik, any ETA on it being in the 6.0 RC's ?

Xeron · Mon Jan 21, 2013 1:53 pm

I never saw this issue from latest 5 and 6rc versions. So for me it looks fixed.

fuzzbawl · Wed Jun 19, 2013 11:36 pm

It seems we still have customers with issues. I noticed two things today that I didn't notice before with this issue. First is that it appears the MikroTik DNS packets have the DF bit (Don't Fragment) set on replies. This is especially bad if the client making the query is behind a VPN tunnel or a PPPoE session and the query is larger than the MTU. With DNSSEC replies and IPv6 replies, it doesn't take much for a response to be large.

In the query I have attached, there are three IPs involved. My Mac OS 10.8 desktop (192.168.42.35), MikroTik RB2011 with RouterOS 6.1 (192.168.42.1) and a Linux server with Bind DNS (192.168.42.3). You can see the query to and reply from 192.168.42.3 comes back with no tcp flags set and it includes the SOA record for the DNS zone. The query to and reply from the MikroTik router 192.168.42.1 comes back with DF bit set and does not include SOA record.

I have sent this to MikroTik support but thought I would post here for everyone else to be in the loop.

Thu Jun 20, 2013 1:51 am

I never saw this issue from latest 5 and 6rc versions. So for me it looks fixed.

Me too

Thanks Mikrotik !

janisk · Thu Jun 20, 2013 1:06 pm

try 6.1 as there are domain name cache changes that could resolve the issues you had.

fuzzbawl · Thu Jun 20, 2013 5:28 pm

...MikroTik RB2011 with RouterOS 6.1 (192.168.42.1)...

I am using 6.1

Keron · Thu Aug 29, 2013 3:28 pm

the issue still exist

i just changed my rb433 to rb2011, the next day i had many clients calls with not working sites

in my case the problematic sites are: youtube.com, groupon.pl, gumtree.pl, facebook.com and few others ...

rb2011 os version - 5.25
ap mode rb433 version 5.21, 5.22

fast schema: rb2011dns <---- rb433 dns
|
/|\
|
client's router dns (connected through wifi to rb433 os ver. 5.22/5.21)
|
/|\
|
pc/ap router dns

I switched today from the default value of 4096 max udp dns packet size @ rb2011 to 512 bytes, and we will see if the issue still occurs...

Xeron · Fri Oct 04, 2013 11:31 pm

I see this issue again after 6.4 update (there were no issues since 6.0rc).

Xeron · Sun Feb 23, 2014 10:59 pm

I want to bump this thread again.

It's still a problem.

Tried with different public DNS servers (ISP, Yandex, Google DNS).

Ping from client side:

xeron@macbook:~$ ping assets.cld.me
ping: cannot resolve assets.cld.me: Unknown host

tcpdump from client side:

00:28:51.213841 IP (tos 0x0, ttl 255, id 29275, offset 0, flags [none], proto UDP (17), length 59)
    macbook.lan.59065 > mikrotik.lan.domain: [udp sum ok] 39469+ A? assets.cld.me. (31)
00:28:51.812088 IP (tos 0x0, ttl 64, id 16954, offset 0, flags [none], proto UDP (17), length 230)
    mikrotik.lan.domain > macbook.lan.59065: [udp sum ok] 39469 ServFail q: A? assets.cld.me. 9/0/0 assets.cld.me. [10m] CNAME d23tod3mb75lgr.cloudfront.net., d23tod3mb75lgr.cloudfront.net. [1m] A 54.230.98.27, d23tod3mb75lgr.cloudfront.net. [1m] A 54.230.96.197, d23tod3mb75lgr.cloudfront.net. [1m] A 205.251.219.4, d23tod3mb75lgr.cloudfront.net. [1m] A 54.230.97.215, d23tod3mb75lgr.cloudfront.net. [1m] A 205.251.219.26, d23tod3mb75lgr.cloudfront.net. [1m] A 54.230.96.214, d23tod3mb75lgr.cloudfront.net. [1m] A 54.230.97.20, d23tod3mb75lgr.cloudfront.net. [1m] A 54.230.98.50 (202)

MT configuration, RouterOS 6.10:

[admin@MikroTik] > /ip dns print 
                servers: 77.88.8.8,77.88.8.1
        dynamic-servers: 85.21.192.3,213.234.192.8
  allow-remote-requests: yes
    max-udp-packet-size: 4096
   query-server-timeout: 2s
    query-total-timeout: 10s
             cache-size: 2048KiB
          cache-max-ttl: 1w
             cache-used: 43KiB

DNS record appeared in MT DNS cache so looks like resolve was successful from this side, but MT set servfail flag in DNS response.

What MT got from public DNS server — in attachment (wireshark dump). Actually it was servfail from first and normal response from second DNS server.

I'm not sure about RFC, but should recursive DNS server set ServFail if it got normal response from second DNS server after ServFail from first?

leocarvalho001 · Fri Jan 22, 2016 4:48 pm

We have this problem with a Mikrotik hotspot servicing hundreds of people in my company.

They use MT as temporary thing for wifi, and are discussing what to do (Cisco probably).

I would upgrade, but this problem seems unsolvable since 2007???

DNS is such an important thing in networking that if MT can't solve this bug, then it should just open source its RouterOS and let people do it IMHO.

Xeron · Mon Jan 25, 2016 10:41 am

I would upgrade, but this problem seems unsolvable since 2007???

This exact issue was fixed in ROS 6.11 after my report to Mikrotik Support.

It has also been discussed here: https://www.linkedin.com/groups/1616717 ... 6799951874

Who is online