Problems: Changes to Static DNS causing serious issues.

Squidblacklist · Fri Sep 30, 2016 7:49 am

We provide domain name based blacklists, formatted for use with Mikrotik RouterOS Web Proxy access list, and for RoS DNS Static Entry List.

Recently our subscribers began complaining that our dns blacklists were no longer importing with the latest version of router OS.

We had to change name= to regexp= in our scripts because we block the topdomain.com as well as all subdomains in our blacklists. Which we did with little effort, the issue however is that we are unable to import our lists because something is broken in routeros now. Do you guys even test this stuff before rolling out changes? Well I dont know how to say this in a constructive way because I am so furiously pissed off at you right now Mikrotik that it is difficult to even type correctly when posting this message.

I thought I should come here before I go on our blog publicly speaking about the issue and justifiably, bashing you for causing a problem when in fact, my problem may be something I simply do not understand at the moment, I thought that I would come here and give you the chance to correct the problem in routeros or at least inform me if Im the one to blame here, the sooner we get this fixed right the better.

As such we would be happy to provide to your engineers access to our blacklists for testing purposes so that you can see there is definitely a problem or problems with large lists. Lists that used to load, 4000-10000 entries, now will not load at all now that we are using regex, we dont even get an error in the system log half the time which is another issue. It just simply stops importing sometimes itll tell us regex too complex , and it does this at random! There are issues with our list formatting, but we are also absolutely positive there are bugs in routeros, we will reach out in this threat to try and correct our regex formatting issues and bring to light some of the actual bugs.

Ok, so this is what we started with, straight conversion from name to regex, and presumably its choking on all of the unescaped dots.

#  TiK-DNS-Ads: Blacklist compiled by SquidBlacklist.org 09-29-2016.                                   -MADE IN USA-
#  Blacklists by Squidblacklist.org are licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License.
:log info "tik dns ads blacklist script import started"
:local redirectIP "127.0.0.1"
/ip dns static remove [find comment="sbl ads"]
/ip dns static
add regexp=.*004.frnl.de address="$redirectIP" comment="sbl ads"
add regexp=.*01s.net address="$redirectIP" comment="sbl ads"
add regexp=.*01viral.com address="$redirectIP" comment="sbl ads"
add regexp=.*039068a.dialer-select.com address="$redirectIP" comment="sbl ads"
add regexp=.*0427d7.se address="$redirectIP" comment="sbl ads"
add regexp=.*0702.de address="$redirectIP" comment="sbl ads"
add regexp=.*0ca.net address="$redirectIP" comment="sbl ads"
add regexp=.*0day.kiev.ua address="$redirectIP" comment="sbl ads"
add regexp=.*0gee.com address="$redirectIP" comment="sbl ads"
add regexp=.*100-100.ru address="$redirectIP" comment="sbl ads"
add regexp=.*100free.com address="$redirectIP" comment="sbl ads"
add regexp=.*100free.nl address="$redirectIP" comment="sbl ads"
add regexp=.*100suelle.com address="$redirectIP" comment="sbl ads"
add regexp=.*101com.com address="$redirectIP" comment="sbl ads"
add regexp=.*101order.com address="$redirectIP" comment="sbl ads"
add regexp=.*10central.com address="$redirectIP" comment="sbl ads"
add regexp=.*1100ad.de address="$redirectIP" comment="sbl ads"

And for your testing purposes, here is our latest release of our ads blacklist, which as you will find, will not load, every time you try to import it, it will fail on a random line inexplicably.

http://www.squidblacklist.org/downloads/tik-dns-ads.rsc

If you change rexexp= to name= and format the list accordingly, you will see that the list imports just fine as name, but when you try to import as regexp it chokes to death on the lists.

We propose that you consider enabling name PLUS regexp, so that we could add website.com as the name and \*. or something simple as the regexp to block all subsequent subdomains.

Understandably when somebody blocks a website like pornhub.com they mean it, they want it blocked and all subdomains, Anyway, its going to be a nightmare to create the backend script to format for the regex. But well get to work on it, in the meanwhile consider the issues. And I am totally open to any help I can get.

Thank you. Now lets make the world a better place, starting right now.
Signed,

Benjamin E. Nichols
http://www.squidblacklist.org

http://blog.squidblacklist.org/?p=853

Sob · Fri Sep 30, 2016 3:48 pm

Few comments:

- I actually like separate name and regexp fields, it makes it more clear what the entry is, and enables better input validation. But since now name field does not take regexps, it is a breaking change for some people. So not everyone will be excited about it.

- You're right that there is something very wrong with current RouterOS version. I tested your ~6000 entries list with CHR. After changing regexp= to name=, it takes only ten seconds to import it on 6.34.6. Then I tested 6.37 and 6.38rc7 with original file and it does not work well at all. The initial import speed is maybe a little slower, but still ok. But after ~3000 entries it slows down to crawl. It still does something, it eats up cpu and new entries do appear in list, but it's just one every few seconds. So yeah, there's clearly a bug.

- If you want it fixed, you have to contact support directly. This is user forum. MikroTik employees read it too, but you can't count on them noticing everything.

- Your regexps are not exactly right, e.g. regexp=.*01s.net will also match test01s.net, 01sxnet.com, etc.

Squidblacklist · Fri Sep 30, 2016 9:30 pm

You are correct, Ok so today, I woke up and somehow a router magically succeeded in importing the whole list, that I had scheduled ot run over night, so thats encouraging.

But then I began to read your messages, and again, after testing I verified what you said to be true, and so I made some changes to get things working how they should.

Regex has been changed to .\\*.somewebsite.net

Which seems to do precisely what we want ==> *.somewebsite.net and corrects the issue you described.

The sample is now up and running if youd like to test.

http://www.squidblacklist.org/downloads/tik-dns-ads.rsc

Only problem we have now is that the god damn mikrotik wont import both slashes, even though that is what works, its in the blacklists, the routeros is stripping one of the slashes during import process, anybody know how to get that to stop?

Sob · Sat Oct 01, 2016 12:13 am

Looks like a typo, because .\*.somewebsite.net (unquoted regexp) looks for literal "*", you probably meant .*\.somewebsite.net, but it still gives you false positives for anything.somewebsite-net.com and such.

The accurate no-false-positives version would be (including base domain without any subdomain):

^(.*\.)?somewebsite\.net$

Same as export:

/ip dns static
add address=127.0.0.1 regexp="^(.*\\.)\?somewebsite\\.net\$"

But it does not really influence the handling in RouterOS, there's definitely some bug. Btw, your current script dies here at this line:

add regexp=".\\*.ads-siteelb-g56fz8b4d9hz-801723328.us-east-1.elb.amazonaws.com" address="127.0.0.1" comment="sbl ads"

with "failure: regular expression too compex".

Squidblacklist · Sat Oct 01, 2016 12:26 am

Alright, scripts setup and working the way we want. However.

The problem remains, grrrr.

Failure at or around line 901:

add regexp=".\\\\*.ads-sexy-site.com" address="127.0.0.1" comment="sbl ads"
add regexp=".\\\\*.ads-sg.com" address="127.0.0.1" comment="sbl ads"
add regexp=".\\\\*.ads.shizmoo.com" address="127.0.0.1" comment="sbl ads"
add regexp=".\\\\*.ads-shotblast.com" address="127.0.0.1" comment="sbl ads"
add regexp=".\\\\*.ads-signs.com" address="127.0.0.1" comment="sbl ads"
add regexp=".\\\\*.ads-silleda.com" address="127.0.0.1" comment="sbl ads"

[admin@MikroTik1] > import tik-dns-ads.rsc                                                                                              
failure: regular expression too compex

Squidblacklist · Sat Oct 01, 2016 12:31 am

Squidblacklist · Sat Oct 01, 2016 12:34 am

But it does not really influence the handling in RouterOS, there's definitely some bug. Btw, your current script dies here at this line:
Code: Select all
add regexp=".\\*.ads-siteelb-g56fz8b4d9hz-801723328.us-east-1.elb.amazonaws.com" address="127.0.0.1" comment="sbl ads"
with "failure: regular expression too compex".

I am not sure I would call that a bug, apparently there is a character limitation in regex thats preventing large domain names and/or multiple subdomains to be added. Or just inherant to the way its processing the dots or dashes as regex? I suspect that is the case, which we will need to escape somehow to avoid the issue.

Sat Oct 01, 2016 1:47 am

Try escaping your dots.

An un-escaped dot is a single-character wildcard in REGEX.

Sob · Sat Oct 01, 2016 1:59 am

This problem with long regexp looks like an artificial limit in some sanity check. Usually you wouldn't need anything this long. Escaping dots does not change anything, even when you enter all basic character like letters or numbers, it takes maximum of 63.

By bug I meant the main problem, the extremely slow and dying import of items with regexp.

Squidblacklist · Sat Oct 01, 2016 3:23 am

This problem with long regexp looks like an artificial limit in some sanity check. Usually you wouldn't need anything this long. Escaping dots does not change anything, even when you enter all basic character like letters or numbers, it takes maximum of 63.

By bug I meant the main problem, the extremely slow and dying import of items with regexp.

I Agreed, and I will be frank with you, aside from any regexp formatting issues with our list, RouterOS in general has a BIG problem with large domain name ACL rules, we also have a web proxy access list, and it takes FOREVER to load, even on the most powerful Cloudcore routers, and the way they import is absurd, say you have 16 cores, the damn units only using 1 core at 100% to import, there so much they need to optimize for large acls to enable more people to harness the true power of routeros, and to be able to really use the hardware to its full potential. In terms of web filtering with large acls is concerned. Im not sure if the load times being so slow has more to do with poor disk io on nand flash, or what it is. But even if that is the case, theres definately a real software issue that could be fixed.

Allow me to elaborate, I know the hardware on the smaller less powerful routers will never handle large lists well, however, its clear after experimenting on Cloudcore routers with our blacklists that Mikrotik can do somethings on the backend to help alleviate these issues and sort of , supply us with a more streamlined, accommodating platform.
When you import a 9mb domain blacklist, the router consumes hundreds of MB of storage/ram which seems really absurd to me, and makes clear that something absurdly inefficient is happening on the backend within routeros, is it their encrypted file system causing bloating of file sizes within the os???.

Regardless of what conjecture I come up with, its clear that Mikrotik software engineers are not testing large domain acls for web proxy access list or static dns entries when they do their testing, or they are oblivious, or maybe they need faster nand and better IO, or maybe they simply dont give a shit.

But more or less I am also pissed off because now, I have to completely rewrite scripts in order to format the lists to make it work, and its going to be a fricking nightmare to get that working.,

Whats required now is

^(.*\.)?somewebsite\.com$

And so far this is where we are.

#  TiK-DNS-Ads: Blacklist compiled by SquidBlacklist.org 09-30-2016.                                   -MADE IN USA-
#  Blacklists by Squidblacklist.org are licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License.
:log info "tik dns ads blacklist script import started"
:local redirectIP "127.0.0.1"
/ip dns static remove [find comment="sbl ads"]
/ip dns static
add regexp="^(.*.)?004.frnl.de" address="$redirectIP" comment="sbl ads"
add regexp="^(.*.)?01s.net" address="$redirectIP" comment="sbl ads"
add regexp="^(.*.)?01viral.com" address="$redirectIP" comment="sbl ads"
add regexp="^(.*.)?039068a.dialer-select.com" address="$redirectIP" comment="sbl ads"
add regexp="^(.*.)?0427d7.se" address="$redirectIP" comment="sbl ads"
add regexp="^(.*.)?0702.de" address="$redirectIP" comment="sbl ads"
add regexp="^(.*.)?0ca.net" address="$redirectIP" comment="sbl ads"
add regexp="^(.*.)?0day.kiev.ua" address="$redirectIP" comment="sbl ads"

Prepending ^(.*\.)? is a snap, but writing some logic that identifies all gtld/tlds and applys a backslash behind only tlds is going to be a real pain in the fking @ss, since we cant just add a backslash before the last dot in ever line, we have .co.uk, .co.nz, etc etc, and that obviously would fk things up pretty quick, so thanks a lot Mikrotik!!!

We Appreciate the extra work! GRRRR!!!

Sob · Sat Oct 01, 2016 2:53 pm

To be fair to MikroTik, the actual format of regexp is not new. Even before they added separate regexp= field, you had false positives with your original script. Dot always meant any character, and to look for literal dot, it required "\.". And not having "$" at the end meant that it looked for given text in any part of domain, not just at the end.

They actually must have fixed some bugs in new version, because when I'm playing with it now and testing both old and new, I just can't make some perfectly valid regexpes work in old version. E.g. \.somewebsite\.net$ (to match any subdomain, but not the base domain) works fine in new version, but does nothing in old one.

About dots and backslashes, good news, there's no need for any advanced logic, you need backslash before every dot, otherwise the bare unslashed dot acts as a wildcard for any character. If you now have:

regexp="^(.*.)?004.frnl.de"

It looks from the beginning for any number of characters, including zero, followed by any one character, both of previous being optional, followed by 004, any character, frnl, any character, de, anything after that. These all match: test.004.frnl.de, xxx004.frnl.de, 004xfrnl.de, x004xfrnlxde.com. The correct regexp is:

^(.*\.)?004\.frnl\.de$

But this is how you'd add it in WinBox. When you export it, special characters like backslashes get quoted by using another backslash. So as an export, and how it needs to look in your script, is this:

regexp="^(.*\\.)\?004\\.frnl\\.de\$"

Squidblacklist · Sat Oct 01, 2016 8:22 pm

Ah a forward slash before every dot, now thats a lot easier to script.

You have solved my problem, I can script that up in no time.

Thank you!

Squidblacklist · Sat Oct 01, 2016 8:56 pm

#  TiK-DNS-Ads: Blacklist compiled by SquidBlacklist.org 10-01-2016.                                   -MADE IN USA-
#  Blacklists by Squidblacklist.org are licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License.
:log info "tik dns ads blacklist script import started"
TADA Now it is as it should be.  Thanks again! 

:local redirectIP "127.0.0.1"
/ip dns static remove [find comment="sbl ads"]
/ip dns static
add regexp="^(.*\\.)\?004\\.frnl\\.de\$" address="$redirectIP" comment="sbl ads"
add regexp="^(.*\\.)\?01s\\.net\$" address="$redirectIP" comment="sbl ads"
add regexp="^(.*\\.)\?01viral\\.com\$" address="$redirectIP" comment="sbl ads"
add regexp="^(.*\\.)\?039068a\\.dialer-select\\.com\$" address="$redirectIP" comment="sbl ads"
add regexp="^(.*\\.)\?0427d7\\.se\$" address="$redirectIP" comment="sbl ads"
add regexp="^(.*\\.)\?0702\\.de\$" address="$redirectIP" comment="sbl ads"
add regexp="^(.*\\.)\?0ca\\.net\$" address="$redirectIP" comment="sbl ads"
add regexp="^(.*\\.)\?0day\\.kiev\\.ua\$" address="$redirectIP" comment="sbl ads"
add regexp="^(.*\\.)\?0gee\\.com\$" address="$redirectIP" comment="sbl ads"
add regexp="^(.*\\.)\?100-100\\.ru\$" address="$redirectIP" comment="sbl ads"

I will leave this up for anybody who wants to play with it.

http://www.squidblacklist.org/downloads/tik-dns-ads.rsc

Also note that the error has a syntax booboo "Error: regular expression too compex"" Compex?

Squidblacklist · Sun Oct 02, 2016 9:46 pm

This problem with long regexp looks like an artificial limit in some sanity check. Usually you wouldn't need anything this long. Escaping dots does not change anything, even when you enter all basic character like letters or numbers, it takes maximum of 63.

By bug I meant the main problem, the extremely slow and dying import of items with regexp.

After correcting our incorrect attempt at regexp, with the syntax you suggested. We have no alternative but to agree, because it imports with the "regex too complex" errors , Do you have any suggested actions to get Mikrotik to fix this problem? If it truly is a bug this needs to be fixed as soon as possible.

I have updated the format and the updated file reflects the changes that you informed me needed to be made, however, as I stated, I have come to the same conclusion as yourself. But thats because I still dont know what the cause of the failures are.

You mentioned a maximum of 63 characters, which I suppose isnt too big of a pill to swallow, I did eliminate all lines greater than 60 characters from the source domain blacklist before converting and importing to tik-dns format, but that did not stop the error...

I have a suspicion the culprit is unescaped dashes, Is there any issue with the dashes commonly found in domain names, that may be a problem for regex?
( my tik isnt telling me which line is causing the errors, so im left with my own conjecture here )

[admin@MikroTik1] /ip dns static> add regexp="^(.*\\.)\?.ads-siteelb-g56fz8b4d9hz-801723328\\.us-east-1\\.elb\\.amazonaws\\.com\$" address="127.0.0.1" 
comment="sbl ads"
failure: regular expression too compex

Also note that the error has a syntax booboo "Error: regular expression too compex"" Compex?

Sob · Tue Oct 04, 2016 3:29 am

If you want something fixed, you need to contact support. Describe what's the problem and hope for the best. Try to be as clear as possible, so all details, but not too long, you don't want them getting lost in it. And be patient.

Dashes alone are not a problem, e.g. shortened regexp="^(.*\\.)\?.ads\\.us-east-1\\.elb\\.amazonaws\\.com\$" passes ok.

Finding an exact line which does not pass shouldn't be hard, just look in IP->DNS->Static, sort it by number (# column) then take the regexp from last item, find it in your script and the problem is on next line.

Squidblacklist · Tue Oct 04, 2016 7:01 am

If you want something fixed, you need to contact support. Describe what's the problem and hope for the best. Try to be as clear as possible, so all details, but not too long, you don't want them getting lost in it. And be patient.

Yup I went ahead and opened a support ticket, we shall see, hopefully its just a minor issue with the regexp syntax. Because getting a bug like this fixed might be a real agonizing wait.

Squidblacklist · Fri Oct 07, 2016 6:09 am

This is the bullshit response I got from Mikrotik support, they didnt investigate, My regex is not overly complex, and it would only lead to "infinate loops" if there were a bug present in RouterOS. This is a serious issue, we represent many entities who wish to use these blacklists, and if Mikrotik RouterOS is a bug ridden OS incapable of performing as advertised then I supposed it isnt worth promoting, or encouraging more people to waste money on expensive CloudCore routers that are unable to process reasonably large blacklists.

SHAME ON YOU MIKROTIK

Whomever is working at support who sent me this is an uninspired, careless twat who should be fired immediately.

Hello,

RouterOS does not allow to add very complex regexps which may lead to infinite loops or very slow operation.

Best regards,
Maris

--
MikroTik.com

Come to the MUM conferences, registration open in Hungary, Serbia, Indonesia, Middle East (Dubai), Netherlands, United Kingdom, Brazil, Bolivia!
http://mum.mikrotik.com/

Sob · Fri Oct 07, 2016 11:01 pm

You're not being patient. I get it, you're not exactly thrilled by this, but telling them what you think about them won't help you much. Instead it might be better strategy to explain that there are million different ways to shoot yourself in foot with wrong config, so why not let you do the same with too complex regexp, if you want to.

Personally I'd be more worried about the other part of problem, i.e. import speed. People will notice if they can import your list in few seconds or in several minutes, if at all.

Compared to that, this too complex regexp problem is just a minor inconvenience. You can always skip few of the longest hostnames when generating your list for RouterOS, and it won't make it noticeably worse. Or you could even go back to your old regexp format. It was not completely accurate and could have false matches, but if nobody complained so far... Or you can use something between. E.g. instead of regexp="^(.*\\.)\?01s\\.net\$" you can use regexp="01s\\.net\$", which is not perfect, but still better than original regexp=".*01s.net".

Sob · Wed Oct 12, 2016 5:44 am

What's new in 6.38rc12 (2016-Oct-11 10:35):
...
*) dns - improved static dns entry add speed when regexp is being used;

Tested and working nice and fast.

Squidblacklist · Mon Oct 17, 2016 7:48 am

Ok, I have to take a moment and give Mikrotik credit, where it is due. they actually resonded quite impressively with a speed enhancement for RouterOS, this has been a problem for a long time with large blacklist importing on RouterOS, so this was such a welcomed, and long overdue fix that so many people are going to benefit from, not only us.

Also, we have adjusted our listes to limit to less than 60 characters, which, when paired with the rc of RouterOS containing the applied patches, seems to load perfectly.

Thank you again to everybody here who helped to get this issue resolved, and a big thank you to Mikrotik Software engineers whom took it upon themselves to give us the solution to our format, as well as the patch to RouterOS.

Thank you all!

Signed,

Benjamin E. Nichols
http://www.squidblacklist.org

Shumkov · Mon Feb 22, 2021 10:42 am

Before FWD was implemented, I used layer7+NAT to conditionally forward DNS requests.
This is a working example of a regex:

/ip firewall layer7-protocol add name=opennic regexp="(\\x03bbs|\\x04chan|\\x03cyb|\\x03dyn|\\x04epic|\\x04geek|\\x06gopher|\\x04indy|\\x05libre|\\x03neo|\\x04null|\\x01o|\\x03oss|\\x02oz|\\x06parody|\\x06pirate|\\x05bazar|\\x04coin|\\x03emc|\\x03lib|\\x03fur|\\x02ku|\\x02te|\\x02ti|\\x02uu).\\x01\$"

And finally, the opportunity presented itself to get rid of this design.
Changed (simplified) the regular expression a little, and introduced the command to /ip/dns/static:

[admin@MikroTik] > /ip dns static add forward-to=<address> regexp="\\.(bbs|chan|cyb|dyn|epic|geek|gopher|indy|libre|neo|null|o|oss|oz|parody|pirate|bazar|coin|emc|lib|fur|ku|te|ti|uu)\$"
 
failure: regular expression too complex

failure: regular expression too complex

"Too complex"? Seriously? This is the simplest regular expression!

Further more interesting. Removed several domains from the regex, and the command ran without errors:

[admin@MikroTik] > /ip dns static add forward-to=<address> regexp="\\.(bbs|chan|cyb|dyn|epic|geek|gopher|indy|libre|neo|null|o|oss|oz|parody|pirate)\$"

So it's all about artificially limiting the length of the regular expression?
WTF?
Why did you do it?

"Foolproof"? So the fools have thousands of other ways to bring the device into a state of complete inoperability!

Now what? Break the already simplest regular expressions into even simpler ones? Increase the number of configuration lines instead of simplifying it?
Why are these limits? Why such "care" about the user? Moreover, the user did not ask for it.

Problems: Changes to Static DNS causing serious issues.

Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Re: Problems: Changes to Static DNS causing serious issues.

Who is online