Page 1 of 1

[API bug/suggestion] Regex in queries

Posted: Thu Nov 17, 2011 4:22 pm
by boen_robot
I was recently made aware that the API doesn't support regular expressions in queries. While I don't personally have a need for that (yet), there are others who do, and I believe it's a reasonable feature request.

Reasons for having this feature and use cases are the same for which queries exist in the first place - to limit the amout of memory consumed by API clients, and allow RouterOS to minimize network traffic.

To keep consistency with other operations, using a syntax like
?~name=pattern
would probably be best IMO.

[edit]
Well, I was told that the following syntax works for making print filter by regex in API:
~name~"pattern"
However this appears more like a bug in API rather than a feature, since we have a completely inconsistent syntax from the rest of the API. Not to mention it's not part of the spec.
[edit]

Re: [API bug/suggestion] Regex in queries

Posted: Mon Nov 21, 2011 10:14 am
by janisk
hmm

prints all interfaces, so it does not work:
/interface/print
~name~"ether"
prints no interfaces, so it does work:
/interface/print
?name=ether
prints one interface, so it does work:
/interface/print
?name=ether1
as i have been told some long time ago, as it is, in most cases, cheaper for router to send packets than do regexp matching, API will not have this feature. And that has to be done on client side.

Re: [API bug/suggestion] Regex in queries

Posted: Mon Nov 21, 2011 1:40 pm
by boen_robot
The example (which you can see on the linked page) was
/ip/dhcp-server/lease/print
=count-only=
~active-address~"10.80."
Maybe it's a problem/"feature" that occurs only in that particular print.
as it is, in most cases, cheaper for router to send packets than do regexp matching, API will not have this feature. And that has to be done on client side.
In terms of router CPU, yes. But overall, in order for the client to process a response, it needs to first receive it in memory (which is also spending some client CPU time right there, not to mention network bandwidth) which may not be good for the client, plus the actual regexp matching itself, which may not be as efficient as RouterOS' regexp.

So although it would be more work for the router, it should be up to the user to decide where will they put the burden.

Re: [API bug/suggestion] Regex in queries

Posted: Sun Feb 26, 2012 10:57 pm
by amr
So although it would be more work for the router, it should be up to the user to decide where will they put the burden.
I couldnt agree more! We run routerOS on none mikrotik hardware - I think servers are capable of a little regex :/

Can anyone confirm that regex/partial value matching does not work?

"~key~value" does not filter anything, and "?~key~value" returns nothing.

You can do it via terminal ...
 /ip address print where address~"10.10"
So how is it any extra "burden" via the API?

Re: [API bug/suggestion] Regex in queries

Posted: Mon Jun 04, 2012 3:59 pm
by janisk
it adds a lot. As i explained before - it is cheaper to run them on client side than on the server side, taking into account that queries are possible.

I have not seen any powerful router placed somewhere, just to idle around. Usually that is done other way around - you are required to have powerful router, and when you want to automate some tasks, you want as little additional load (not routing/network related) as possible to avoid other hardware upgrade.

Re: [API bug/suggestion] Regex in queries

Posted: Mon Jun 04, 2012 4:30 pm
by boen_robot
But regex is available in scripting/SSH... I'd assume that if implemented in the API, it would use the same regex engine, so where does the difference come from?

I mean, in both cases you have a non-routing automated action. The only difference is that with scripting/SSH, the results are not sent over the network, but kept for further processing (which could in turn include a sending over the network, especially if we're talking SSH output). Or am I missing something else?

If the penalty is really high for some scenarios, people could always just... I don't know... not use this feature in those scenarios for example. A warning in the docs would suffice to alert people IMO.

Re: [API bug/suggestion] Regex in queries

Posted: Tue Mar 31, 2015 7:05 am
by chojrak11
But regex is available in scripting/SSH...
Yeah that mismatch is weird. So I can put a "burden" on a router using SSH, but I cannot do the same using API? Where's the logic?

You could implement this with at most 30 minutes of coding, and let users decide if it's useful or not. These autocratic arguments seems like corporation which knows better what's best for its customers/users.

And @janisk is well aware that his arguments don't make any sense at all, but for some reason defends them for years. If you really think that regex puts a lot of load on the router, the "~" operator should be removed from scripting as well. And then remove scripting altogether, because it can also cause 100% load, e.g. by using an infinite loop.

Don't consider your customers dumbasses, just please add a feature that is asked for.

Re: [API bug/suggestion] Regex in queries

Posted: Sun Jul 12, 2015 9:16 pm
by legrang
This has come up again, see: http://forum.mikrotik.com/viewtopic.php?p=490459

Moore's law and the recurring questions about this should surely be an indication that this feature should be reconsidered?

Gideon

Re: [API bug/suggestion] Regex in queries

Posted: Tue Jan 19, 2016 9:36 am
by Deantwo
as i have been told some long time ago, as it is, in most cases, cheaper for router to send packets than do regexp matching, API will not have this feature. And that has to be done on client side.
Unless the scripting/CLI engine is totally separate from the API engine and work totally different, I don't see how this augment makes any sense. The person that told you this "a long time ago" may have been right at the time, but it is clearly not an issue anymore.

You are also assuming that the client is always more powerful than the router, which does not have to be true at all anymore. I think a RaspberryPi using the API would prefer the work being done on the router.

Anyway, would very much like for an update on this issue/request.

Re: [API bug/suggestion] Regex in queries

Posted: Wed Jan 20, 2016 12:56 pm
by chojrak11
as i have been told some long time ago, as it is, in most cases, cheaper for router to send packets than do regexp matching, API will not have this feature. And that has to be done on client side.
Unless the scripting/CLI engine is totally separate from the API engine and work totally different, I don't see how this augment makes any sense. The person that told you this "a long time ago" may have been right at the time, but it is clearly not an issue anymore.

You are also assuming that the client is always more powerful than the router, which does not have to be true at all anymore. I think a RaspberryPi using the API would prefer the work being done on the router.

Anyway, would very much like for an update on this issue/request.
Come on guys, don't play fool, do this API regex matching already. And when you decide to do it, don't use those crappy GNU Extended Regex syntax anymore, just go for the widespread PCRE-compatible syntax.

Re: [API bug/suggestion] Regex in queries

Posted: Wed Jan 20, 2016 1:11 pm
by boen_robot
And when you decide to do it, don't use those crappy GNU Extended Regex syntax anymore, just go for the widespread PCRE-compatible syntax.
In fairness, they're already using POSIX regular expressions in scripting... Switching to PCRE is going to break a lot of scripts, and using PCRE just for the API while using POSIX for scripting is going to be confusing to people.

If the API uses POSIX regular expressions, that's enough IMHO. A switch to PCRE, if it happens at all, should be at v7, for both API and scripting.

Re: [API bug/suggestion] Regex in queries

Posted: Wed Jan 20, 2016 1:14 pm
by chojrak11
In fairness, they're already using POSIX regular expressions in scripting... Switching to PCRE is going to break a lot of scripts, and using PCRE just for the API while using POSIX for scripting is going to be confusing to people.

If the API uses POSIX regular expressions, that's enough IMHO. A switch to PCRE, if it happens at all, should be at v7, for both API and scripting.
Good points. I forgot about scripting and L7 matching. I hate POSIX regex, it's wrong on all fields. So let's vote for PCRE for RouterOS v7 :-)

Re: [API bug/suggestion] Regex in queries

Posted: Wed Jan 20, 2016 3:51 pm
by mrz
Please be patient, we will have some new features in ROSv7. :)

Re: [API bug/suggestion] Regex in queries

Posted: Wed Jan 20, 2016 3:55 pm
by boen_robot
Please be patient, we will have some new features in ROSv7. :)
Oh? Including this one?

(where's a firework emoticon when you need one...)

Excellent!!!!!!


Knowing that it would come eventually is all anyone wanted to know... As opposed to what we had until now, which was "Nope, won't have it, sorry".

Re: [API bug/suggestion] Regex in queries

Posted: Wed Jan 20, 2016 4:07 pm
by legrang
As API implementation developer (not just API user), it would be really nice if I could have some pre-release info so I can adapt the Java API and be ready to release an update when v7.0 leaves beta *hint* *hint* :D

But good news!