Community discussions

 
Znuff
Member Candidate
Member Candidate
Topic Author
Posts: 139
Joined: Tue Sep 26, 2006 2:42 am
Contact:

Help Needed - Performance Issues - How can I improve this?

Sat Dec 20, 2014 7:52 pm

So for a while now I have been using this script:
:global FindList do={
    :local myListName $list
    :local myTimeout "1:00:00"
    
    /ip dns cache all {
        :foreach j in=[find where (name~$server)] do={
            :do {
                :local myName [get $j name]
                :local myType [get $j type]
                :local myData [get $j data]
                #:local myTime [get $j ttl]
                #:local myLine [get $j]

                :if ($myType = "A") do={
                    :do {
                        /ip firewall address-list add address=$myData list=$myListName comment="$myName" timeout=$myTimeout
                    } on-error={
                        #:put "DNSCACHE: $myLine"
                        :put "DNSCACHE: Error on $myName $myData $myType $myTime"
                        /ip firewall address-list set [find where address=$myData] timeout=$myTimeout
                    }
                }

                :if ($myType = "CNAME") do={
                    :local currentName $j
                    :local nextName [find where (name=$myData && type="CNAME")]

                    :while ($nextName != "") do={
                        :set currentName $nextName
                        :resolve $currentName
                        :set nextName [find where (name=[get $nextName data] && type="CNAME")]
                    }

                    :resolve [get $currentName data]

                    :foreach k in=[find where (name=[get $currentName data] && type="A")] do={
                        :set myData [get $k data]
                        #:set myTime [get $k ttl]
                        #:set myLine [get $k]
                        :do {
                            /ip firewall address-list add address=$myData list=$myListName comment="$myName" timeout=$myTimeout
                        } on-error={
                            #:put "DNSCACHE: $myLine"
                            :put "DNSCACHE: Error on $myName $myData $myType $myTime"
                            /ip firewall address-list set [find where address=$myData] timeout=$myTimeout
                        }
                    }
                }
            } on-error={
                :put "DNSCACHE: Missing Entry"
            }
        }
     }
 }
 
 $FindList server="youporn" list="porn"
 $FindList server="phncdn" list="porn"
 $FindList server="youtube" list="youtube"
 $FindList server="googlevideo" list="youtube"
This works just fine, functionality wise, but performance wise, it's horrible. It takes up to 10 minutes to run and it keeps one of my CPU Cores at 100% for that duration (so overall ~60% CPU Usage).

How can I improve this?

EDIT: In case someone needs further explanation - we search our DNS Cache for entries with "youporn", "phncdn", "youtube", "googlevideo" and add them to different address lists, temporarly, with a 1 hour expiration time. We're using this to send traffic to those destinations via a different connection.
 
User avatar
boen_robot
Forum Guru
Forum Guru
Posts: 2411
Joined: Thu Aug 31, 2006 4:43 pm
Location: europe://Bulgaria/Plovdiv

Re: Help Needed - Performance Issues - How can I improve thi

Sat Dec 20, 2014 9:58 pm

I don't have any performance measurements, but just theoretically... Using "print as-value" should be more efficient than issuing several get calls with the ID as an argument. Also, variables declared in loops tend to be expensive in most languages that don't have hoisting (and I'm guessing that's the case in RouterOS scripting too, since it doesn't do hoisting), and that penalty is only outweighed when the value is accessed multiple times and the single access time is on par with variable creation time.

So for starters:
:global FindList do={
    :local myListName $list
    :local myTimeout "1:00:00"
    
    /ip dns cache all {
        :foreach j in=[print as-value where (name~$server)] do={
            :do {
                :local myType ($j->type);

                :if ($myType = "A") do={
                    :do {
                        /ip firewall address-list add address=($j->data) list=$myListName comment=($j->name) timeout=$myTimeout
                    } on-error={
                        #:put "DNSCACHE: $myLine"
                        :put ("DNSCACHE: Error on " . ($j->name) . " " . ($j->data) . " $myType $myTime")
                        /ip firewall address-list set [find where address=($j->data)] timeout=$myTimeout
                    }
                }

                :if ($myType = "CNAME") do={
                    :local currentName ($j->name)
                    :local nextName [find where (name=($j->data) && type="CNAME")]

                    :while ($nextName != "") do={
                        :set currentName $nextName
                        :resolve $currentName
                        :set nextName [find where (name=[get $nextName data] && type="CNAME")]
                    }

                    :resolve [get $currentName data]

                    :foreach k in=[find where (name=[get $currentName data] && type="A")] do={
                        :set myData [get $k data]
                        #:set myTime [get $k ttl]
                        #:set myLine [get $k]
                        :do {
                            /ip firewall address-list add address=$myData list=$myListName comment="$myName" timeout=$myTimeout
                        } on-error={
                            #:put "DNSCACHE: $myLine"
                            :put "DNSCACHE: Error on $myName $myData $myType $myTime"
                            /ip firewall address-list set [find where address=$myData] timeout=$myTimeout
                        }
                    }
                }
            } on-error={
                :put "DNSCACHE: Missing Entry"
            }
        }
     }
 }
But perhaps it might be significantly more efficient to actually change the whole algorithm... Rather than expecting the cache to already be filled, and handling errors when it's not, why not preemptively call :resolve on the actual domain, and search the cache THEN? At that point, all IPs will be in the cache, and if another domain happens to be aliased to the existing ones, they too would then be matched by whatever rules to associate with those addresses.

Of course, this means the call should actually have the full domain in it.

So for example:
:global FindList do={
    :local myTimeout 01:00:00

    :local aliases (({}), $server)
    :local i 0
    :while ($i < [:len $aliases]) {
        :set server [:pick $aliases $i]
        :resolve $server
        :foreach j in=[/ip dns cache all print as-value where (name=$server)] do={

            :if (($j->type) = "CNAME") do={
                :set aliases ($aliases, ($j->data));
            } else={
                :if (($j->type) = "A") do={
                    :do {
                        /ip firewall address-list add address=($j->data) list=$list comment=($j->name) timeout=$myTimeout
                    } on-error={
                        :put ("DNSCACHE: Error on $j")
                        /ip firewall address-list set [find where address=($j->data)] timeout=$myTimeout
                    }
                }
            }

        }
        :set i ($i + 1)
    }
}

$FindList server="youporn.com" list="porn"
$FindList server="phncdn.com" list="porn"
$FindList server="youtube.com" list="youtube"
$FindList server="googlevideo.com" list="youtube"
Oh and BTW, with the above, you also get a nice syntax sugar available to the call:
$FindList server=({"youporn.com";"phncdn.com"}) list="porn"
$FindList server=({"youtube.com";"googlevideo.com"}) list="youtube"
PEAR2_Net_RouterOS(1.0.0b6) - My API client in PHP
(Rate my posts? If you want... no pressure...)
 
Znuff
Member Candidate
Member Candidate
Topic Author
Posts: 139
Joined: Tue Sep 26, 2006 2:42 am
Contact:

Re: Help Needed - Performance Issues - How can I improve thi

Sun Dec 21, 2014 1:25 am

Correct me if I'm wrong, but your last example wouldn't actually work properly for my usage, as I'm trying to get *everything* that contains "youtube" and "googlevideo" etc., as youtube has dozens of CDN hosts and I really have no idea how to get them all other than searching for everything with those keywords in them...?

But you make a good point about the first one.
 
User avatar
boen_robot
Forum Guru
Forum Guru
Posts: 2411
Joined: Thu Aug 31, 2006 4:43 pm
Location: europe://Bulgaria/Plovdiv

Re: Help Needed - Performance Issues - How can I improve thi

Sun Dec 21, 2014 6:00 am

Fair point... Then I think perhaps splitting this in two is in order - getting a list of all domain names, and then adding those domains in the list.

The above does the latter, so we could rename it, and add the first:
:global AddServerToList do={
    :local myTimeout 01:00:00

    :local aliases (({}), $server)
    :local i 0
    :while ($i < [:len $aliases]) {
        :set server [:pick $aliases $i]
        :resolve $server
        :foreach j in=[/ip dns cache all print as-value where (name=$server)] do={

            :if (($j->type) = "CNAME") do={
                :set aliases ($aliases, ($j->data));
            } else={
                :if (($j->type) = "A") do={
                    :do {
                        /ip firewall address-list add address=($j->data) list=$list comment=($j->name) timeout=$myTimeout
                    } on-error={
                        :put ("DNSCACHE: Error on $j")
                        /ip firewall address-list set [find where address=($j->data)] timeout=$myTimeout
                    }
                }
            }

        }
        :set i ($i + 1)
    }
}

:global FindList do={
    :foreach j in=[/ip dns cache all print as-value where (name~("\\.\?" . $server . "\\."))] do={
        $AddServerToList server=($j->name) list=$list
    }
}

$FindList server="youporn" list="porn"
$FindList server="phncdn" list="porn"
$FindList server="youtube" list="youtube"
$FindList server="googlevideo" list="youtube"
The regex is intentionally tweaked so that it matches only a full domain segment. So it matches youtube.com, youtube.us, youtube.akamai.com, i.youtube.com, but does NOT match myownyoutube.com or similar.

Though with that new addition, I'm even less sure if there will be a measurable difference.
PEAR2_Net_RouterOS(1.0.0b6) - My API client in PHP
(Rate my posts? If you want... no pressure...)
 
Znuff
Member Candidate
Member Candidate
Topic Author
Posts: 139
Joined: Tue Sep 26, 2006 2:42 am
Contact:

Re: Help Needed - Performance Issues - How can I improve thi

Sun Dec 21, 2014 6:04 pm

Throws an error at line 6
:while ($i < [:len $aliases]) {
--

nevermind, fixed that, there was a missing do={}

But there's still an issue at
        :set server [:pick $aliases $i]

...right, because that has to be either a local or a global variable, not "set".

Still not exactly working. Script runs, does nothing :(

----------------------

Actually I tried all variations of the script and none of them seemed to work, all come up empty, no results
 
User avatar
boen_robot
Forum Guru
Forum Guru
Posts: 2411
Joined: Thu Aug 31, 2006 4:43 pm
Location: europe://Bulgaria/Plovdiv

Re: Help Needed - Performance Issues - How can I improve thi

Mon Dec 22, 2014 4:53 pm

Serves me right for not testing...

Associative keys apparently MUST have quotes around them... I thought it was needed only on non-alphanumeric keys.

Also, there was a runtime error inside the inner function (I believe related to the fact the name of the local variable matched the name of a parameter) that's not at all reported on screen... And finally, my regex ends up matching domains that end with the string, which is not right...

Here's a version that corrects all of these, and makes the whole thing one function:
:global FindList do={
    :local myTimeout 01:00:00

    :foreach i in=[/ip dns cache all print as-value where (name~("(^" . $server . "|\\." . $server . ")\\."))] do={
        :local aliases ({$i->"name"})
        :local j 0
        :while ($j < [:len $aliases]) do={
            :local alias [:pick $aliases $j]
            :resolve $alias
            :foreach k in=[/ip dns cache all print as-value where (name=$alias)] do={

                :if (($k->"type") = "CNAME") do={
                    :set aliases ($aliases, ($k->"data"))
                } else={
                    :if (($k->"type") = "A") do={
                        :do {
                            /ip firewall address-list add address=($k->"data") list=$list comment=($k->"name") timeout=$myTimeout
                        } on-error={
                            :put ("DNSCACHE: Error on " . ($k->"name"))
                            /ip firewall address-list set [find where address=($k->"data")] timeout=$myTimeout
                        }
                    }
                }

            }
            :set j ($j + 1)
        }
    }
}


$FindList server="youporn" list="porn"
$FindList server="phncdn" list="porn"
$FindList server="youtube" list="youtube"
$FindList server="googlevideo" list="youtube"
This time, I have tested it, and it DOES work. You'll want to of course browse those sites in order to fill the initial cache. All of YouTube's domains alone (which are indeed many...) take only ~2 seconds (again: no "precise" measurements...) on my router (v6.23 on a 1.9Ghz single core x86 PC) and doesn't appear to spike at all during that. I imagine those 4 sites should add up to less than 8 seconds, though you probably have many others... I'll be curious to see whether it scales to something better than your original script.
Last edited by boen_robot on Tue Dec 23, 2014 3:09 pm, edited 1 time in total.
PEAR2_Net_RouterOS(1.0.0b6) - My API client in PHP
(Rate my posts? If you want... no pressure...)
 
CR24
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Wed Dec 04, 2013 10:32 pm

Re: Help Needed - Performance Issues - How can I improve thi

Tue Dec 23, 2014 5:31 am

You could try using filter rules that search for keywords you specify and add anything matching to your address lists. You will want to add an accept filter rule for your DNS IPs, if you don't it will add your DNS IPs to the address lists. Make sure you change the IPs below to match the IPs of your DNS servers.


/ip firewall address-list
add address=8.8.8.8 list=dns
add address=8.8.4.4 list=dns

/ip firewall filter
add chain=forward src-address-list=dns
add action=add-src-to-address-list address-list=porn address-list-timeout=\
    1h chain=forward content=porn
add action=add-src-to-address-list address-list=youtube address-list-timeout=\
    1h chain=forward content=youtube
 
User avatar
boen_robot
Forum Guru
Forum Guru
Posts: 2411
Joined: Thu Aug 31, 2006 4:43 pm
Location: europe://Bulgaria/Plovdiv

Re: Help Needed - Performance Issues - How can I improve thi

Tue Dec 23, 2014 2:59 pm

You could try using filter rules that search for keywords you specify and add anything matching to your address lists. You will want to add an accept filter rule for your DNS IPs, if you don't it will add your DNS IPs to the address lists. Make sure you change the IPs below to match the IPs of your DNS servers.


/ip firewall address-list
add address=8.8.8.8 list=dns
add address=8.8.4.4 list=dns

/ip firewall filter
add chain=forward src-address-list=dns
add action=add-src-to-address-list address-list=porn address-list-timeout=\
    1h chain=forward content=porn
add action=add-src-to-address-list address-list=youtube address-list-timeout=\
    1h chain=forward content=youtube
That won't work for at least two reasons:
1. It matches any site that mentions "youtube" in their content, not just the site YouTube.
2. Sites like YouTube use HTTPS, meaning you won't inspect their contents. You can only inspect the DNS packet's reply, but those firewall rules would add the DNS to the address list, not the IP that the DNS gives. The scripts above DO inspect the DNS reply, although via the router's DNS cache, as opposed to "on the fly".
PEAR2_Net_RouterOS(1.0.0b6) - My API client in PHP
(Rate my posts? If you want... no pressure...)
 
User avatar
boen_robot
Forum Guru
Forum Guru
Posts: 2411
Joined: Thu Aug 31, 2006 4:43 pm
Location: europe://Bulgaria/Plovdiv

Re: Help Needed - Performance Issues - How can I improve this?

Fri Feb 06, 2015 8:01 pm

@Znuff
It's been more than a month now... Any success on this? Performance stats you can share? How much (if at all) of an improvement was it compared to before?
PEAR2_Net_RouterOS(1.0.0b6) - My API client in PHP
(Rate my posts? If you want... no pressure...)

Who is online

Users browsing this forum: No registered users and 4 guests