Community discussions

 
hjoelr
newbie
Topic Author
Posts: 29
Joined: Mon Apr 28, 2008 11:29 pm

Update address list with wildcard DNS names

Thu Aug 22, 2019 6:42 pm

I have a few online services that I'd like to pull under control of QoS. Some of them say that their traffic can be identified by wildscard DNS entries like this: *.download.windowsupdate.com.

I know in RoS you can put a non-wildscard DNS name in an address list and it will automatically do nslookups to update the address list with the resolved IPs. With wildcards, it's a little harder as you don't have the actual DNS name to resolve. I've created the following script to scan through /ip dns cache all and resolve IPs, but it's pretty slow if there's multiple CNAME levels that it has to wade through before getting to an actual IP. This also doesn't even handle keeping an address list updated. Is there some other built-in method to accomplish this?

# delete the function before defining it again
/system script environment remove [ find where name="fGetIpsForDomain" ];

:global fGetIpsForDomain do={
    /log debug message="fGetIpsForDomain (domain: $domain | regex: $regex | recurselevel: $recurselevel)"

    :if ( [:len $domain] = 0) do={
        /log debug message="fGetIpsForDomain() - No domain specified. Exiting"
        :return [:toarray ""];
    }

    :local level $recurselevel
    :if ( [:len $recurselevel] = 0) do={
        :set level 0
    }

    :local isregex false
    :if ( $regex = "true" || $regex = "yes" ) do={
        :set isregex true
    }

    :local cacheitems

    :if ( $isregex ) do={
        :set cacheitems [/ip dns cache all find name~($domain)]
    } else={
        
        /log debug message=("fGetIpsForDomain(level: $level) - resolving $domain")

        # Force a DNS resolve before we look at the cache.
        :do { [:resolve $domain] } on-error={}

        :set cacheitems [/ip dns cache all find name=$domain]
    }

    /log debug message=("fGetIpsForDomain(level: $level) - found cache items count: " . [:len $cacheitems])

    :local iparr [:toarray ""];

    :if ( [:typeof $currarrayitems] = "array" ) do={
        /log debug message=("fGetIpsForDomain(level: $level) - using passed array of " . [:len $currarrayitems] . " items.")
        
        :set iparr $currarrayitems
    }

    :local rectype;
    :local recname;

    :foreach result in=$cacheitems do={
        :set rectype [/ip dns cache all get $result type ]
        :set recname [/ip dns cache all get $result name ]

        # Only if this is a regex call do we want to resolve the domain for each result.
        # If not a regex call we already resolved earlier and don't want to be redundant.
        :if ( $isregex ) do={
            /log debug message=("fGetIpsForDomain(level: $level) - resolving $recname")

            :do { [:resolve $recname] } on-error={}
        }
        

        /log debug message=( "fGetIpsForDomain(level: $level) - record type: $rectype" )

        # We only want to find IPv4, IPv6 and CNAME items. We ignore the rest.
        :if ( $rectype = "A" || $rectype = "AAAA" || $rectype = "CNAME" ) do={

            :local itemdata [/ip dns cache all get $result data ]

            /log debug message=( "fGetIpsForDomain(level: $level) - itemdata: $itemdata | iparr(count): " . [:len $iparr] )

            :if ( $rectype = "A" || $rectype = "AAAA" ) do={
                :if ( ([:len $itemdata] > 0) && ( $itemdata != "0.0.0.0" ) ) do={
                    :global fArraySearch

                    :local itemexists [$fArraySearch array=$iparr find=$itemdata]

                    # Only add the item to the array if it doesn't already exist in there
                    :if ( $itemexists = false ) do={

                        /log debug message="fGetIpsForDomain(level: $level) - adding $itemdata to array"

                        :set iparr ( $iparr, $itemdata )
                    } else={
                        /log debug message="fGetIpsForDomain(level: $level) - $itemdata already exists in array. Discarding."
                    }
                    
                }
            } else={
                # Have to get a reference to its own global variable to call itself.
                :global fGetIpsForDomain

                # Pass the current array and reset it with returned array as it would include old and new items.
                :set iparr [$fGetIpsForDomain domain=$itemdata recurselevel=($level + 1) currarrayitems=$iparr]
            }
        }
    }

    /log debug message=( "fGetIpsForDomain(level: $level) - final result: " . [:tostr $iparr] )

    :return $iparr
}


# delete the function before defining it again
/system script environment remove [ find where name="fArraySearch" ];

:global fArraySearch do={

    :if ( [:typeof $array] = "nothing") do={
        /log debug message="fArraySearch() - No array specified."
        /log debug message=("fArraySearch() - premature end");
        :return false;
    }

    :if ( [:len $find] = 0 ) do={
        /log debug message="fArraySearch() - No value to find specified."
        /log debug message=("fArraySearch() - premature end");
        :return false;
    }

    /log debug message=( "fArraySearch( arraylen: " . [:len $array] . " | find: $find )" )

    :local itemfound false
    :local itemindex 0
    :while ( $itemfound = false && $itemindex < [:len $array] ) do={
        :if ( $find = $array->itemindex ) do={
            /log debug message=("fArraySearch(find: $find) - found at index $itemindex");

            :set itemfound true
        }
        :set itemindex ( $itemindex + 1 )
    }

    if ( $itemfound = false ) do={
        /log debug message=("fArraySearch(find: $find) - not found");
    }
    
    :return $itemfound;
}

Example function usage:

:put [$fGetIpsForDomain domain=("download\\.windowsupdate\\.com\$") regex=true]
Last edited by hjoelr on Fri Aug 23, 2019 3:53 am, edited 6 times in total.
 
pe1chl
Forum Guru
Forum Guru
Posts: 5697
Joined: Mon Jun 08, 2015 12:09 pm

Re: Update address list with wildcard DNS names

Thu Aug 22, 2019 7:11 pm

Unfortunately those domains usually do not even offer zone transfer, so it is impossible to determine what is behind *.domain.com.
For windows update you can still use some L7 match like:

/ip firewall layer7-protocol
add name=MSDO regexp=User-Agent:.Microsoft-Delivery-Optimization

and also the (of course much too broad):

/ip firewall layer7-protocol
add name=Download regexp=/download

Of course you should evaluate those only on outgoing TCP connects to port 80 and use it to mark a connection and later use the mark for the QoS.
 
hjoelr
newbie
Topic Author
Posts: 29
Joined: Mon Apr 28, 2008 11:29 pm

Re: Update address list with wildcard DNS names

Thu Aug 22, 2019 9:00 pm

@pe1chl Thanks for replying. I hadn't thought about L7 inspection, but that may be an option for some of them. Thanks for the idea.
 
pe1chl
Forum Guru
Forum Guru
Posts: 5697
Joined: Mon Jun 08, 2015 12:09 pm

Re: Update address list with wildcard DNS names

Thu Aug 22, 2019 9:08 pm

It works for Windows Update, but of course many other downloads have switched to https and then it does not work anymore.
Also, L7 matching is unfortunately not possible for IPv6.

Who is online

Users browsing this forum: No registered users and 7 guests