Community discussions

MikroTik App
 
dany
just joined
Topic Author
Posts: 23
Joined: Thu Apr 27, 2006 4:19 pm

Help with script for DNS lookups and address lists

Sun Jan 12, 2014 1:14 pm

I'm using the script from
http://adminsline.wordpress.com/2012/08 ... k-youtube/
in order to make address list with YouTube video server IPs from DNS cache so I can make some redirections.

I wish to use the same script so I could grab IP addresses of some other video streaming sites,
like vimeo.com, imdb.com and so on. While for YouTube video servers I get A records in DNS cache, for Vimeo or IMDB
I actually get CNAME records and the above script can't resolve them.

There is other script on Mikrotik Wiki:
http://wiki.mikrotik.com/wiki/Sync_Addr ... _A_Records
which works with CNAME records as well but is not suitable as such for my situation (the first script as search string
takes part of the server name, for example "googlevideo", "vimeocdn" etc).

Unfortunately I'm not that good at scripting and can't figure out how to incorporate the CNAME resolving part of the second
script into the first one.

Has anyone done something like that and is willing to help? It would be much appreciated.
 
efaden
Forum Guru
Forum Guru
Posts: 1708
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Help with script for DNS lookups and address lists

Sun Jan 12, 2014 1:24 pm

I'll look at the scripts once I get back to my computer.

Sent from my SCH-I545 using Tapatalk
 
dany
just joined
Topic Author
Posts: 23
Joined: Thu Apr 27, 2006 4:19 pm

Re: Help with script for DNS lookups and address lists

Sun Jan 12, 2014 1:28 pm

Thank you very much!
 
efaden
Forum Guru
Forum Guru
Posts: 1708
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Help with script for DNS lookups and address lists

Sun Jan 12, 2014 3:54 pm

Thank you very much!
It wouldn't be that hard to merge the two scripts, but I don't have the time to do it right this second. It will most likely be at least a week or two before I would have time to tackle it, I had/have some family stuff going on that is taking up most of my time. I figure it would take about 2 hours to get it done.
 
dany
just joined
Topic Author
Posts: 23
Joined: Thu Apr 27, 2006 4:19 pm

Re: Help with script for DNS lookups and address lists

Sun Jan 12, 2014 5:38 pm

I have tried merging it but with no success.
It's not a pressing matter so if you could do it in a two weeks time,
when you have free time it would be great.

Best regards!
 
efaden
Forum Guru
Forum Guru
Posts: 1708
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Help with script for DNS lookups and address lists

Sun Jan 12, 2014 6:02 pm

I have tried merging it but with no success.
It's not a pressing matter so if you could do it in a two weeks time,
when you have free time it would be great.

Best regards!
I'll keep it on the to do list when I have some time. If you want it done sooner I could "consult" on it... if your interested in going that route just email me.
 
efaden
Forum Guru
Forum Guru
Posts: 1708
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Help with script for DNS lookups and address lists

Sun Jan 12, 2014 7:11 pm

I have tried merging it but with no success.
It's not a pressing matter so if you could do it in a two weeks time,
when you have free time it would be great.

Best regards!
I'll keep it on the to do list when I have some time. If you want it done sooner I could "consult" on it... if your interested in going that route just email me.
Also can you give me a run through of EXACTLY what you are wanting the script to do? You just want the script with CNAMEs to work on partial names?
 
dany
just joined
Topic Author
Posts: 23
Joined: Thu Apr 27, 2006 4:19 pm

Re: Help with script for DNS lookups and address lists

Sun Jan 12, 2014 9:29 pm

Here is the exact script I'm running on the router at the moment:
:foreach i in=[/ip dns cache find] do={
    :local bNew "true";
    :local cacheName [/ip dns cache all get $i name] ;
#    :put $cacheName;

    :if (([:find $cacheName "youtube"] != 0) || ([:find $cacheName "googlevideo"] != 0)) do={

        :local tmpAddress [/ip dns cache get $i address] ;
#   :put $tmpAddress;

# if address list is empty do not check
        :if ( [/ip firewall address-list find ] = "") do={
            :log info ("added entry: $[/ip dns cache get $i name] IP $tmpAddress");
            /ip firewall address-list add address=$tmpAddress list=VC-YouTube comment=$cacheName;
        } else={
            :foreach j in=[/ip firewall address-list find ] do={
                :if ( [/ip firewall address-list get $j address] = $tmpAddress ) do={
                    :set bNew "false";
                }
            }
            :if ( $bNew = "true" ) do={
                :log info ("added entry: $[/ip dns cache get $i name] IP $tmpAddress");
                /ip firewall address-list add address=$tmpAddress list=VC-YouTube comment=$cacheName;
            }
        }
    }
}

It searches through DNS cache looking for servers which have words "youtube" or "googlevideo" in their name and then adds IP
address of the server to address list named VC-YouTube.

If I change it to search for servers that have in their name "vimeocdn" it will find, lets say server10.vimeocdn.com (I've made
that up) but the Type field in DNS cache will be CNAME and the Data field will be something like a1806.dwskn.akamai.net.
In order to be able to add that to an address list I would need that a1806.dwskn.akamai.net further resolved to numeric IP address.

The second script I mentioned in my first post can do that, but I can't feed it a partial server name and let it find all the
servers that has the keyword in its name so it's unusable in my case.

I guess this might be interesting to others as well as it allows some fancy dynamic redirecting or routing of the sites that can be
bandwidth hogs. As I've said there is no rush about it so whenever you have free time, it would be nice to have this done and apply
it to more sites...

Regards!
 
efaden
Forum Guru
Forum Guru
Posts: 1708
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Help with script for DNS lookups and address lists

Mon Jan 13, 2014 6:07 pm

ros code

:local myServers { "google" }
:local myListName "myList"
:local myTimeout "00:01:00"

/ip dns cache all {
    :foreach i in=$myServers do={
        :foreach j in=[find where (name~$i)] do={
            :local myName [get $j name]
            :local myType [get $j type]
            :local myData [get $j data]

            :if ($myType = "A") do={
                :do {
                    /ip firewall address-list add address=$myData list=$myListName comment="DNSCACHE-$i-$myName" timeout=$myTimeout
                } on-error={
                    :put "DNSCACHE: Error on $myName $myData $myType"
                }
            }

            :if ($myType = "CNAME") do={
                :local currentName $j
                :local nextName [find where (name=$myData && type="CNAME")]

                :while ($nextName != "") do={
                    :set currentName $nextName
                    :set nextName [find where (name=[get $nextName data] && type="CNAME")]
                }

                :resolve [get $currentName data]

                :foreach k in=[find where (name=[get $currentName data] && type="A")] do={
                    :set myData [get $k data]
                    :do {
                        /ip firewall address-list add address=$myData list=$myListName comment="DNSCACHE-$i-$myName" timeout=$myTimeout
                    } on-error={
                        :put "DNSCACHE: Error on $myName $myData $myType"
                    }
                }
            }
        }
    }
}
 
dany
just joined
Topic Author
Posts: 23
Joined: Thu Apr 27, 2006 4:19 pm

Re: Help with script for DNS lookups and address lists

Mon Jan 13, 2014 9:26 pm

Thank you very very much Efaden!!! The script works like a charm.

I had to takeout the following part of the script though, as it was giving me an error,
but that doesn't decrease its usefulness at all!
timeout=$myTimeout
     } on-error={
        :put "DNSCACHE: Error on $myName $myData $myType"
     }
 
efaden
Forum Guru
Forum Guru
Posts: 1708
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Help with script for DNS lookups and address lists

Mon Jan 13, 2014 9:29 pm

Thank you very very much Efaden!!! The script works like a charm.

I had to takeout the following part of the script though, as it was giving me an error,
but that doesn't decrease its usefulness at all!
timeout=$myTimeout
     } on-error={
        :put "DNSCACHE: Error on $myName $myData $myType"
     }
The error catch will only work on 6.5 (I think) and beyond. .... I would actually suggest upgrading then using that statement. Otherwise the script will crash on duplicate entries. If you want me to make it work with duplicate entries and no on-error let me know... I can re-work it a bit...

And timeout is also only on the newer versions.. I think 6.7 and up.

Right now the script doesn't clear the list... so you will run into problems if you remove those and run it on an older version. If you want me to make it work on an older ROS let me know what version and I can retrofit.

-Eric
 
dany
just joined
Topic Author
Posts: 23
Joined: Thu Apr 27, 2006 4:19 pm

Re: Help with script for DNS lookups and address lists

Mon Jan 13, 2014 9:36 pm

Oh, I see.. I've tried it on a router with 6.3 version.
I don't want to bother you more than I have already, I will just upgrade the OS to a higher version.
Thanks again!
 
efaden
Forum Guru
Forum Guru
Posts: 1708
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Help with script for DNS lookups and address lists

Mon Jan 13, 2014 10:23 pm

Oh, I see.. I've tried it on a router with 6.3 version.
I don't want to bother you more than I have already, I will just upgrade the OS to a higher version.
Thanks again!
Up to you... won't take me long to retrofit it.
 
efaden
Forum Guru
Forum Guru
Posts: 1708
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Help with script for DNS lookups and address lists

Mon Jan 13, 2014 10:32 pm

Not tested... but something like this.

ros code

:local myServers { "google" }
:local myListName "myList"

/ip firewall address-list {
	remove [find list=$myListName]
}


/ip dns cache all {
	:foreach i in=$myServers do={
		:foreach j in=[find where (name~$i)] do={
			:local myName [get $j name]
			:local myType [get $j type]
			:local myData [get $j data]
	
			:if ($myType = "A") do={
				:local inList false
				:foreach k in=[find where (address=$myData && list=$myListName)] {
					:set inList true
				}

				:if (!inList) do={
					/ip firewall address-list add address=$myData list=$myListName comment="DNSCACHE-$i-$myName"
				}
			}
	
			:if ($myType = "CNAME") do={
				:local currentName $j
				:local nextName [find where (name=$myData && type="CNAME")]

				:while ($nextName != "") do={
					:set currentName $nextName
					:set nextName [find where (name=[get $nextName data] && type="CNAME")]
				}

				:resolve [get $currentName data] 

				:foreach k in=[find where (name=[get $currentName data] && type="A")] do={
					:set myData [get $k data]
					:local inList false
					:foreach l in=[find where (address=$myData && list=$myListName)] do={
						:set inList true
					}

					:if (!inList) do={
						/ip firewall address-list add address=$myData list=$myListName comment="DNSCACHE-$i-$myName"
					}
				}
			}
		}
	}
}
 
dany
just joined
Topic Author
Posts: 23
Joined: Thu Apr 27, 2006 4:19 pm

Re: Help with script for DNS lookups and address lists

Tue Jan 14, 2014 9:46 pm

Thank you for the modified version as well.

I have upgraded OS to v6.7 in the meantime and the original script now works without any modification.
However, when I run if for the second time (soon after the first run) it ends up in error. Please have a look:
DNSCACHE: Error on r4---sn-nvm-cxbe.googlevideo.com 193.105.163.207 CNAME
DNSCACHE: Error on r1---sn-nvm-cxbe.googlevideo.com 193.105.163.204 CNAME
DNSCACHE: Error on r1.sn-nvm-cxbe.googlevideo.com 193.105.163.204 A
DNSCACHE: Error on r8---sn-c0q7lne7.googlevideo.com 74.125.108.173 CNAME
DNSCACHE: Error on r8.sn-c0q7lne7.googlevideo.com 74.125.108.173 A
DNSCACHE: Error on r8.sn-c0q7lner.googlevideo.com 173.194.1.237 A
DNSCACHE: Error on r11.sn-c0q7ln7y.googlevideo.com 74.125.108.144 A
DNSCACHE: Error on r4.sn-nvm-cxbe.googlevideo.com 193.105.163.207 A
DNSCACHE: Error on r3.sn-nvm-cxbe.googlevideo.com 193.105.163.206 A
DNSCACHE: Error on r10.sn-c0q7lnee.googlevideo.com 74.125.108.207 A
DNSCACHE: Error on r10.sn-c0q7lney.googlevideo.com 173.194.2.143 A
DNSCACHE: Error on r11.sn-c0q7lnez.googlevideo.com 173.194.1.48 A
DNSCACHE: Error on r2.sn-nvm-cxbe.googlevideo.com 193.105.163.205 A
DNSCACHE: Error on r5.sn-c0q7lnek.googlevideo.com 173.194.1.202 A
DNSCACHE: Error on r5.sn-c0q7lnez.googlevideo.com 173.194.1.42 A
DNSCACHE: Error on r7.sn-c0q7lney.googlevideo.com 173.194.2.140 A
DNSCACHE: Error on r5.sn-c0q7lney.googlevideo.com 173.194.2.138 A
DNSCACHE: Error on r19.sn-c0q7lnek.googlevideo.com 173.194.1.216 A
DNSCACHE: Error on r9.sn-c0q7lne6.googlevideo.com 173.194.1.142 A
DNSCACHE: Error on r13.sn-c0q7lnee.googlevideo.com 74.125.108.210 A
DNSCACHE: Error on r18.sn-c0q7lnez.googlevideo.com 173.194.1.55 A
interrupted
           no such item (4)
I understand that those DNSCACHE: Error are because the address already exists in the address-list and that
is okay I presume? But I don't know why it got interrupted and what item is missing. An expired entry from the
DNS cache??
 
efaden
Forum Guru
Forum Guru
Posts: 1708
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Help with script for DNS lookups and address lists

Tue Jan 14, 2014 9:52 pm

Hmmm. That's my guess. I'll have to investigate and add a mechanism to catch that.

Sent from my SCH-I545 using Tapatalk
 
efaden
Forum Guru
Forum Guru
Posts: 1708
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Help with script for DNS lookups and address lists

Wed Jan 15, 2014 2:25 am

Try this.... I added a ":resolve" in the middle to make sure entries are valid when I loop... also added something to detect if an entry vanished while it was looping and just ignore that entry..... I haven't tested it... let me know.

ros code

:local myServers { "google" }
:local myListName "myList"
:local myTimeout "00:01:00"

/ip dns cache all {
    :foreach i in=$myServers do={
        :foreach j in=[find where (name~$i)] do={
            :local exists false
            :do {
                :local myName [get $j name]
                :local myType [get $j type]
                :local myData [get $j data]
                :set exists true
            } on-error={
                :put "DNSCACHE: Missing Entry"
            }

            :if ($exists && $myType = "A") do={
                :do {
                    /ip firewall address-list add address=$myData list=$myListName comment="DNSCACHE-$i-$myName" timeout=$myTimeout
                } on-error={
                    :put "DNSCACHE: Error on $myName $myData $myType"
                }
            }

            :if ($exists && $myType = "CNAME") do={
                :local currentName $j
                :local nextName [find where (name=$myData && type="CNAME")]

                :while ($nextName != "") do={
                    :set currentName $nextName
                    :resolve $currentName
                    :set nextName [find where (name=[get $nextName data] && type="CNAME")]
                }

                :resolve [get $currentName data]

                :foreach k in=[find where (name=[get $currentName data] && type="A")] do={
                    :set myData [get $k data]
                    :do {
                        /ip firewall address-list add address=$myData list=$myListName comment="DNSCACHE-$i-$myName" timeout=$myTimeout
                    } on-error={
                        :put "DNSCACHE: Error on $myName $myData $myType"
                    }
                }
            }
        }
    }
}
 
dany
just joined
Topic Author
Posts: 23
Joined: Thu Apr 27, 2006 4:19 pm

Re: Help with script for DNS lookups and address lists

Wed Jan 15, 2014 11:22 am

It gives me the following error:
expected variable name (line 39 column 26)
 
efaden
Forum Guru
Forum Guru
Posts: 1708
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Help with script for DNS lookups and address lists

Wed Jan 15, 2014 3:35 pm

It gives me the following error:
expected variable name (line 39 column 26)

Whoops... that was stupid. Scope problem...

ros code

:local myServers { "google" }
:local myListName "myList"
:local myTimeout "00:01:00"

/ip dns cache all {
    :foreach i in=$myServers do={
        :foreach j in=[find where (name~$i)] do={
            :do {
                :local myName [get $j name]
                :local myType [get $j type]
                :local myData [get $j data]

                :if ($myType = "A") do={
                    :do {
                        /ip firewall address-list add address=$myData list=$myListName comment="DNSCACHE-$i-$myName" timeout=$myTimeout
                    } on-error={
                        :put "DNSCACHE: Error on $myName $myData $myType"
                    }
                }

                :if ($myType = "CNAME") do={
                    :local currentName $j
                    :local nextName [find where (name=$myData && type="CNAME")]

                    :while ($nextName != "") do={
                        :set currentName $nextName
                        :resolve $currentName
                        :set nextName [find where (name=[get $nextName data] && type="CNAME")]
                    }

                    :resolve [get $currentName data]

                    :foreach k in=[find where (name=[get $currentName data] && type="A")] do={
                        :set myData [get $k data]
                        :do {
                            /ip firewall address-list add address=$myData list=$myListName comment="DNSCACHE-$i-$myName" timeout=$myTimeout
                        } on-error={
                            :put "DNSCACHE: Error on $myName $myData $myType"
                        }
                    }
                }
            } on-error={
                :put "DNSCACHE: Missing Entry"
            }
        }
    }
}
 
dany
just joined
Topic Author
Posts: 23
Joined: Thu Apr 27, 2006 4:19 pm

Re: Help with script for DNS lookups and address lists

Wed Jan 15, 2014 4:26 pm

This time it runs through with no problem. Even when I runnit for the second time immediately after the
first run it still doesn't get aborted and runs through to the end.

One funny thing, when I run it, the first run, it showed 30 of those "DNSCACHE: Error" errors, and that's exactly how many
entries it added to the address list. But please note, I already had those IP addresses but in another
address-list (collected by the old script I used before yours) so that might be the cause for such behaviour.

In any case, all addresses are there regardles of the "DNSCACHE: Error".

Thank you very much for your effort!
 
efaden
Forum Guru
Forum Guru
Posts: 1708
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Help with script for DNS lookups and address lists

Wed Jan 15, 2014 5:50 pm

This time it runs through with no problem. Even when I runnit for the second time immediately after the
first run it still doesn't get aborted and runs through to the end.

One funny thing, when I run it, the first run, it showed 30 of those "DNSCACHE: Error" errors, and that's exactly how many
entries it added to the address list. But please note, I already had those IP addresses but in another
address-list (collected by the old script I used before yours) so that might be the cause for such behaviour.

In any case, all addresses are there regardles of the "DNSCACHE: Error".

Thank you very much for your effort!
Yeah. Those are generated when there is a duplicate entry. For a lot of things (e.g. google, etc)... they use the same IP for multiple things.
 
wcsnet
Frequent Visitor
Frequent Visitor
Posts: 64
Joined: Mon Apr 29, 2013 12:43 pm
Location: South Africa

Re: Help with script for DNS lookups and address lists

Thu Feb 05, 2015 2:55 pm

this is what I use can be used to get ip from dns name or dns name from ip.

# wcsnet, copyright © 2010. all rights reserved
# script version : 1.0

:do {

:local content
:local newhost
:local oldhost

:log info "******************** starting - ip resolve ip*******************"
:log info "-"

:foreach i in=[/ip firewall address-list find list="resolve ip" and disabled=no] do={

:set content [/ip firewall address-list get $i address]
:set oldhost [/ip firewall address-list get $i comment]

:do {
:set newhost [:resolve $content]
} on-error={
:log error "dns resolve failure [ ip - resolve ip ]"
}

: if ($newhost != $oldhost) do={
/ip firewall address-list set $i comment=$newhost
:log warning "$[/system identity get name] $comment dns host name changed: $oldhost -> $newhost"
}
}

:log info "******************** ending - ip resolve ip********************"

:exec "o - after script run"

} on-error={
:log error "script failure [ ip - resolve ip ]"
}


:do {

:local content
:local newip
:local oldip

:log info "******************** starting - ip resolve host*******************"
:log info "-"

:foreach i in=[/ip firewall address-list find list="resolve host" and disabled=no] do={

:set content [/ip firewall address-list get $i comment]
:set oldip [/ip firewall address-list get $i address]

:do {
:set newip [:resolve $content]
} on-error={
:log error "dns resolve failure [ ip - resolve host ]"
}

: if ($newip != $oldip) do={
/ip firewall address-list set $i address=$newip
:log warning "$[/system identity get name] $comment ip address changed: $oldip -> $newip"
}
}

:log info "******************** ending - ip resolve host********************"

:exec "o - after script run"

} on-error={
:log error "script failure [ ip - resolve host ]"
}
 
efaden
Forum Guru
Forum Guru
Posts: 1708
Joined: Sat Mar 30, 2013 1:55 am
Location: New York, USA

Re: Help with script for DNS lookups and address lists

Sat Aug 27, 2016 12:45 am

Glad it was helpful.

Who is online

Users browsing this forum: DNAT and 40 guests