This is a RouterOS-API feature-request/request-for-comments.
An API-client is allowed to “tag” any command issued to a RouterOS device by specifying the .tag API-attribute-word with a given identifier: this is a critical feature in exploiting API’s ability to issue multiple commands, expecially when the listen command is involved, but also allows a properly designed API-client to differently manage the received replies based on the expected contents.
API docs clearly say that an API-client should not rely on the ordering of attribute-words in reply-sentences: from the implementation’s point of view this is acceptable for “non-API attributes” (i.e. attributes which are part of the actual command’s reply) and does not hamper anything, but is severely limiting when it comes to the .tag API-attribute.
Having the .tag API-attribute always returned as first attribute-word (i.e. as second word of a reply-sentence, just after the !re, !trap or !done first word) would greatly simplify API-clients inplementations by allowing an early recognition of received reply-sentences so to have them properly managed (an obvious example is a reply-sentence which holds the content of a file to be saved locally by the API-client).
Currently, as far as I have seen, the .tag API-attribute is always returned as the last word (i.e. just before the terminating null-word) in any reply-sentence to a tagged command: may be there are good reasons for such behaviour and I’m just missing some point.
I’m interested in any comment from Mikrotik guys and/or API developers.
If an API client is intertwining the response interpretation with the response marshaling, then yes, I agree that having an early peek at the .tag will be beneficial to its performance. For clients that separate the two stages, this makes no difference, so +1 for this suggestion from me.
That said, all API clients I’m aware of, including my own, are of the latter type - marshaling, if done at all (which sadly few clients do), occurs after the whole sentence is first interpreted. So at least at first, there won’t be anyone benefitting.
And I for one wouldn’t take advantage of that if implemented, because I strike to let users choose a limited set of features if they want to. With my client currently, you can use “parse & marshal”, which is what’s recommended throughout all docs… but you could also just “parse”, and do the marshaling yourself OR even just “read a word”, and then handle the rest yourself. Integrating the parsing with the marshaling will prevent that flexibility. Few people need that flexibility though, so if other implementations want to maybe improve performance (by microseconds per sentence I believe…) at the expense of that rarely used flexibility, I say “let them”.
Thank you for your comments; I only partly agree with them.
It is not just a matter of performances: it affects the actual implementation of any truly generic API-client library.
By using your same words, even just “read a word”, and then handle the rest yourself alone is not such a generic operation to be implemented by a client-library in a way independent of the word’s content. To be more accurate: from the semantics point of view it is of course a “generic operation” (API-words are API’s atomic-entities of encoding), but when it comes to the actual implementation it is not.
Let me clarify this point: an API-word can be up to 2 GB in size (…well, there is some inaccuracy into the API’s official doc about this point, but this is another story…): how do the API-clients libraries out there (including your own) manage such kind of received words (e.g. a word which holds the content of a big file: BTW, I already made this example in my original message)? Do they “just read the word” and return it to the caller to let it “handling the rest”?
This would mean allocating big (huge?) memory blocks just to temporarily holding data expected to be, presumably, saved on disk.
Apart of performances and memory efficiency matters, there are cases where this is simply not possible (i.e. API-clients running on embedded or otherwise RAM-constrained devices: I have written several clients of this kind).
how do the API-clients libraries out there (including your own) manage such kind of received words (e.g. a word which holds the content of a big file: BTW, I already made this example in my original message)? Do they “just read the word” and return it to the caller to let it “handling the rest”?
In my case, users can retrieve word contents into a “php://temp” stream. In PHP, these streams start writing to HDD as soon as their content reaches 2MB. That option is off by default for obvious reasons, but it’s there precisely for those circumstances. Similar things could be done in other languages (though not as easily), but I’m not aware of clients which actually implement that.
But I suppose you have a point here about the automatic nature of this… or lack there of rather… Right now, if users enable this option, it is applied on all words of all incoming responses, regardless of tag. If the tag is known earlier, that option can be enabled on a per-request basis instead, potentially giving a BIG performance boost to scenarios that currently face an overhead. Hmm… Come to think of it, perhaps I could adjust my implementation to automatically switch to streams when the word length is above a certain (user defined) threshold. In that scenario, whether the tag comes first or not is, again, irrelevant.
there are cases where this is simply not possible (i.e. API-clients running on embedded or otherwise RAM-constrained devices: I have written several clients of this kind).
If you are in an environment where you have less than 2GBs of RAM and/or disk, you won’t have anywhere to write 2GBs to, regardless of whether you know in advance that they are about to come or are coming “right now” - you have to end in an error (and/or discard data) anyway.
Again, not just a performance boost: it greatly simplifies and generalize implementation. Anyway, it seems you got my point.
The “word length trick” is exactly what I do in some of my implementations (the “read_word” proc simply calls a user-specified call-back handler by passing the socket and the word-length so to let it directly do the reading), but it has several drawbacks and is ugly-to-death; it is just a “work-around trick” which implements some “heuristics” (and, as such, not always reliable) to deduce something that can indeed be accurately specified by tagging a command, but cannot be used simply because the tag is not delivered in the proper place. Having a .tag at the end of the sentence, in my opinion, greatly reduces its usefulness.
How simple (and elegant) would be to read a .tag before any actual content and to properly dispatch the rest of the sentence to the specific receiving proc! Actually I already use tagging this way in multi-threading and/or event-driven applications along with multiple “listen” commands, but file transfers almost break the game.
Well, this is not an honest admission.
It is irrelevant because you don’t use it, and you don’t use it because you can’t.
(cfr. “Nondum matura est, nolo acerbam sumere.” – Fedro)
The fact that the tag does not come first forces you to implement an heuristic work-around which would be otherwise not needed: this is NOT irrelevant.
This is absolutely NOT TRUE.
I can describe you several applications where I am quite limited in RAM (either due to physical constraints or because the memory allocated to the process running the API-client is limited) but have LOT of disk or disk-like (i.e. CF/SD/SSD) storage space.
First off, I should probably remind you that I support this idea. Whether or not I would/could/should personally leverage it is a completely different matter.
Right. That does not apply to what I said then.
i.e.
if (haveEnoughRAM) {
//use RAM
} else if (haveEnoughDiskSpace) {//includes “disk-like” storages like CF/SD/SSD
//use diskOrDiskLikeSpace
} else {
//discard data and perhaps error
}
What heuristics? A pre-specified hard limit isn’t really a heuristic per se. And besides, how is that any more or less accurate than applying a write-to-disk on per-request basis? In both cases, you’re writing to disk conditionally, and anyway, wouldn’t you want to write to disk only the words that need to be written to disk (e.g. the contents of a large file, but not its name)? In that case, it would seem like the word length limit is the better approach anyway, since it would protect users from accidently using too much RAM (e.g. the user is reading a list of queues, and the length of one queue’s comment just happens to be gigantic, which they didn’t anticipate upon making the request, as they expect queues to have reasonable comment lengths).
How simple (and elegant) would be to read a .tag before any actual content and to properly dispatch the rest of the sentence to the specific receiving proc!
This is kind of what I started with, and why I support this in the first place. If you have the kind of implementation that both parses words of the response and dispatches it in the same phase (i.e. the same function within the same class, with the addition of calling socket reading functions) - yes, in that case, having the tag at the top will be beneficial, and because it doesn’t affect other implementations, it should definitely be done.
The part where we disagree seems to be as to whether the above is simple or elegant AND (I would add) flexible, as in “addressing multiple approaches the user may wish to use the client with”. I believe in allowing the user to NOT use the built in dispatching, while still recommending that they do and allowing them to use only the parsing if they wish. This is not really possible if you’re dispatching the sentence before it is received in whole. Or perhaps it is, with some object hierarchy that I feel would be uglier. If you’re willing to sacrifice that flexibility requirement (which, as I said, few people take advantage of anyway), then I guess the overall result is a simpler client indeed… I wouldn’t call it exactly “elegant” though, as flexibility contributes a lot to what I’d define as “elegant”.
Yes, it is clear to me: since your first message you acknowledged that the proposal is worthwhile to be supported.
The fact that you might or might not use it in your applications is, of course, your ultimate choice: anyone has its needs and “tastes”, its own development environment with more or less features and options and -last but not least- its own knowledge, experience, abilities and coding style; this point is out of question: if I gave you a different impression about it then I was not clear enough (but, really, I cannot see anything leading to this in my messages).
Ok, no need to further discuss this point.
It seems you’re making assumptions on something I never expressed (more on this later): the “read-file” case is just an example, the first which came to my mind; let’s consider it along with your last (good) example: a “read-file” and a “read-queue-list” requests (and, may be, several outstanding “asynchronous” requests such as “listened” counters, etc.), in other words, do not assume that requests/responses are sequenced (the RoutersOS API gives us this worderful ability to issue/manage non-sequenced commands and it’s worthwhile to be exploited, expecially in multi-threaded or anyway “concurrent” applications); what I know is that the file content will be probably saved on disk while the “queue-list” will not or will be anyway treated in a different way (e.g. will be processed in memory one entry at a time or whatever thing the app needs to do…); assume also that the “read-file” and the “read-queue-list” responses are similar in size, or even that the latter is bigger than the first one (in the end, size here really doesn’t matter) so the question is: how do I differentiate between the (several) sentences coming back from the device? The “word length trick”, along with some additional check (i.e. if the attribute happens to include the string “contents=” in the proper place then this is the file coming-in… etc.) might give me an hint, but you will agree that this IS heuristics, quite unreliable and “ugly-to-death”.
In more explicit terms, my objections are not just a matter of “memory size”, or things like that, but a more general question about the “semantics of the command issued and the related action required upon reception of the response”: tagging the command allows the designer of the API-library to accurately specify everything, from the command issued to the thread/task which sent it and to which direct the response/the state-machine to which dispatch an event-message, from a specific class/object instance and the way the response has to be handled (e.g. as “raw/parsed single words” or “raw/parsed single sentences” or “entire raw/parsed replies up to the final !done sentence” or even as just a socket to be directly handled maybe one byte at a time…) to a “user-defined-tag” (if ever needed); and all of this at the same time in a properly “layered-fashion” encoded API-tag. May be this clarifies that I want by no means reduce the “flexibility” of the library, but exactly the opposite! The ability to tag a command, provided that the tag is returned in the right place (that is: as early as possible in each reply-sentence), allows the designer to build a truly generic, hyper-flexible yet well structured and optimized client-library.
Here is where, as anticipated, it becomes apparent that you are making assumptions on things I never expressed. Please, do not intend the “specific receiving proc” in my above phrase as being neither an “API-client-library internal proc” nor an “user-level proc” (i.e. something “outside” the client-library), so as the expression “dispatching the sentence” does not mean that the sentence was raw or parsed neither it was entirely read in memory or not (what was surely read from the socket in the above model are just the first two words, i.e. the !whatever sentence-type word and the .tag API-attribute word). The above phrase is just to be intended as “(…) read a .tag before any actual content and to perform an action as dictated by the content of the tag”. This by no means reduces the flexibility (I would never introduce into the design of a library something reducing the flexibility or the performances, and for the very simple reason that the first user of my libraries is… myself): it does exactly the opposite; such approach, when properly exploited, allows the design of a truly-generic, structured yet optimized client-library and, by moving to an early stage (i.e. when the command is built) the choice of “how” to handle a response, gives the ultimate-flexibility to the user of the library. And this IS “elegant”, as it allows a client to leverage ALL API features in the most EFFICIENT and EASY way. Think of it as something similar to object-oriented programming, where “data” and “actions” are semantically joined into the same entity.
Anyway all of the above is, again, just a “concept example”: it might or might not relate to a client’s actual implementation. I described it because you, since your first message, still insists in discussing implementation-related aspects (parsing/marshalling yes/no or before/after, etc.) and consequences (i.e. more or less flexibility or efficiency) regarding an hypotetical (on my part?) actual use of the tagging feature, instead to stay at an higher, more abstract, semantical level; in the end, it seems you’re doing what you apparently blamed me for: whether or not I would/could/should personally leverage it, the way I would do it and its consequences.
So, let’s go back to the original, general, question (rewritten in a, maybe, clearer and more explicit form):
Given the semantics of the tagging feature,
would such feature result more useful if the .tag API attribute is returned as early as possible into the reply-sentences? (My answer: YES)
does returning the .tag API attribute as early as possible into the reply-sentences, by itself, directly implies degrading performances or flexiblity of an API-client? (My answer: NO)
is there any actual reason to return the .tag API attribute as the last word of any reply-sentence? (My answer: I don’t see any)
does returning the .tag API attribute as early as possible into the reply-sentences might affect currently existing API-clients? (My answer: NO, provided that the client conforms to current API specifications which dictate no assumptions on the ordering of attribute words in reply-sentences)
In conclusion, if your answers to the above four questions are the same as mine, we actually don’t disagree on anything and all of this post was just a misunderstanding due to your groundless assumptions on an hypotetical (my?) implementation.
If your answer to some of the above questions are different than mine, I would be greatly interested in your point. But, please, do not assume things I never expressed.
The latin “… not mature… bitter” bit is the primary thing that rubbed me the wrong way… It made it seem like you’re cleverly implying “You can’t use it… therefore you’re saying it’s bad and no one should use it and therefore there’s no point in it being done at all”, which is not my position.
Side note: I could actually use it. If you’re making the assumption PHP is single threaded, that’s because (like most people, including myself until two months ago) you haven’t seen the thread extension.
Yes, my answers to all of the above are the same.
But… You are the one who started arguing over whether this would be good in general for any implementation (hypothetical or actual current one). I merely stated that I don’t see my implementations benefitting, while still acknowledging that others might. But I guess I’m at fault too for later extending the conversation further by stretching the question “Is my current implementation’s state the best it could be?” to “Is my current implementation’s state the best for any implementation, not just mine?”.
Aha! Additional checks! Yes, that’s the part I missed, since you never mentioned it. I assumed the “word length trick” was the only “heuristic”. And yes, that is ugly and unreliable. However, I don’t think the word length trick alone, if applied to all requests is ugly or unreliable. On the contrary - it makes applications using such a client library more stable without making it more complex for the library user, assuming they set the limit themselves (if there’s a default limit, I can see havoc for applications that haven’t taken that limit into account).
The ultimate benefit then seems (No, this is not what you said, but this is what I’m gathering a hypothetical implementation could be…) to be setting receiving preferences both per request and per word. I can see that making things simpler for the user in a number of scenarios. I’m still not entirely sure how it would work when the parsing and dispatching are to be kept separated from one another, while the parsing is still integrated with the sentence (i.e. how this works in existing implementations, my own included), but that’s probably just me.
First, about the quote: it is from the classical fable “The vixen and the grape” (originally in greek by Fedro, then “covered” in latin by Esopo). It’s about a vixen struggling to catch a grape without being able to do it, being the grape too high on the plant so, rather than to admit the own inability, the vixen prefers to despise the fruit. The phrase, translated in english, would sound: “It is not still mature, I do not want to eat it sour”. It is frequently used here in Italy (where greek and latin are studied quite well… or at least were well teached when, ages ago, I was an high-school student) as a metaphor to criticize the inappropriate despising of an objective which is not reachable either due to the inability of the person who is trying to achieve it or due to an inherent characteristic of the objective itself; in both cases (and, just to be clear, I intended the second one) the phrase never implies that “the vixen” is claiming a “general banning of the objective” (no one should use it and therefore there’s no point in it being done at all) but the exact opposite: it always implies a very specific condition as, by being specific, it “saves the vixen” in the event that someone else would be able to “catch the grape finding that it is actually mature and not sour” (the “vixen” could then reply: “THAT grape WAS sour WHEN I wanted to eat it”).
Then about the context in which I used the quote; first you have to note that we were focused on different points: you were thinking about the ability to switch to “stream mode” based on the size of the received word while I was thinking on the general ability to recognize a specific response (the file-download, as already explained, was just an example: I think that this point should be clear enough, now) so for you the “word-length trick” was (and actually IS) a proper solution and the tagging feature was (and actually IS) irrelevant, while from my point of view the “trick” was (and actually IS) just an heuristic work-around due to the inability to properly use the tagging feature because of the non-ideal placement of the .tag attribute, which makes it (as actually IS) NOT irrelevant, hence the latin metaphor.
Hope that this definitely clarifies my intention.
I kindly asked you not to make any groundless assumption: I was not doing any assumption nor I was referring to any specific implementation detail. By the way I’m old, experienced and skilled enough to clearly know that “multi-threading” is nothing more than just a “concurrency model” and that you don’t need “threads” to write “concurrent multi-tasking” applications. It was not by chance that I said in multi-threaded or anyway “concurrent” applications and that I mentioned “state-machines”, “event-driven applications” and “event-message dispatching” (just to refer to some alternative “concurrency models”: there are really a lot more of them, you know).
Good. That is what I was actually interested in. From my point of view the original question is perfectly answered.
I just said that (a) (…) the (current) ordering of attribute-words in reply-sentences (…) is severely limiting when it comes to the .tag API-attribute and that (b) having the .tag API-attribute always returned as first attribute-word (…) would greatly simplify API-clients implementations by allowing an early recognition of received reply-sentences: since you confirmed my answers to all of the previous four questions you should agree to both of these points. These are anyway just general thoughts, as everything depends on if and how an actual client uses the tagging feature (and this, in turn, depends on what can be done with the tagging feature given its current way of working). So, the fact that your implementation could be improved or not is a specific matter: you should better go to answer your own last questions above. I can just say that if, as yourself admitted, the “word-length trick” alone suggested a possible optimization regarding the per-request switching to “stream mode” (and this is definitely a positive consequence of our discussion herein) then, IMHO, probably YES, your implementation could be improved and could benefit of a “revised” tagging feature. The final answer is anyway up to you (or anyone evaluating your implementation).
From my side I can clearly say: YES, there are cases where my implementations would definitely benefit of a .tag attribute-word returned as early as possible.
The misunderstanding should be finally solved, now. And YES, as I already said, the “word-length trick” is a proper solution to the use-cases you described (you can be sure I would never used it otherwise…).
As clearly said, that was just a “concept example” to explore the semantical power of the tagging feature: I anyway used such approach in several context (also not related to RouterOS API).
I’m indeed really happy if I have given you something to think about.
Already gathered that from your last post. Thanks for further elaborating anyway. And I know the fable too, which is exactly why the poor Google translation was enough for me to start extrapolating what I thought you meant with that part.
You assumed that I could not use “it”:
and I assumed “it” refers to the scenario you’re describing, which particularly focuses on how a multi-threaded environment benefits. The only possible reason (that I can think of) you would assume I would not be able to use that “it” is if you assume PHP can’t do any form of multi-thread-ness or another concurrency model thing… Which is actually true without the threads extension - PHP indeed does not support any form of concurrency without an extension like that, making it easy to jump to this being the “it”.
But then again, I can already see that’s probably not the “it” you were talking about, right?
If I have ever implied to even think otherwise, I apologize. I have gathered just that (well, except the “old” bit…) since your first post.
Yep. That sort of outcome is why I entered the rabbit hole anyway . I mean, I could’ve left your second post just hang in there, but nooo… Potentially productive conversations (like what this one has already turned out to be) are too rare in this forum to miss out on.
Right. I was just referring to that hypotetical use-case of the “word-length trick” you described, in which (as seen from my, already explained, point of view) the tagging feature could be used but wasn’t only because of it’s limitation, making the limitation itself not irrelevant. I was not referring to your actual implementation, moreover I could not refer to it by any means for the very simple reason that such “trick” was not (yet?) implemented by you.
About “focusing on multi-threading”: an “early-recognition” of the tag would be of great help even (and, may be, more) in “single-threaded” applications; just think of an application displaying some real-time values returned “asynchronously” by several “listen” commands (interface or queue counters, etc.) while still allowing the user to interact with the device: the receiver could early-recognize the “!re” sentences corresponding to the “listen” commands and dispatch them (parsed/unparsed/whatever: it does not care here, it’s all about the “concept”) to the code updating the screen, leaving the rest of the communication to the “foreground” user interaction. You don’t need multi-threading for this.
You can of course do this also with the tagging feature as it is currently provided by the API (i.e. tag attribute coming last in each sentence) but then you are forced to read an entire sentence before being able to know if it relates to a “background action” (i.e. counter updating) or to a “foreground one”. And if the “foreground action” involves “big-sized” responses (i.e. reading a file or a list of queues/firewall-rules/etc. including big elements, as you suggested) you are in trouble.
I already gave you this kind of example; well, just to help you to answer you own questions about enhancing your code: try to implement this (very simple) application with your client-library and you will soon be able to see if it would benefit of an “early-tag-recognition”.
“Don’t give in without a fight!” (not Fedro/Esopo this time, Pink Floyd instead).
You can do “concurrency” in almost any programming language on the planet, if you need it, and PHP is a quite feature-rich environment. I see different ways to implement an event-loop in “standard” PHP (at least three, may be more) and to generate i/o or timeout related “events”. Then you can implement “coroutines” (the PHP environment should be able to give you support even for them), or “static-closures”, or code in “continuation-passing style”, or might implement your own simple scheduler, maybe with an event-message queue, or just a simple “state-machine”… most probably a combination of all of the above.
After all… “A Computer is a state machine. Threads are for people who can’t program state machines.” – Alan Cox
(well, a quite strong admission, as in Alan’s style, but absolutely correct in its substance)
I agree. There is just a missing point into this conversation, definitely the most important one: a comment from Mikrotik’s people about the tagging feature. But they tend to be someway reluctant when it comes to discussing something related to RouterOS possible improvements. Or, maybe, they just missed this thread.
They aren’t reluctant to answer when the answer is a big fat “No”, so you can take the silence as a “maybe… once we resolve all higher priority issues… if we don’t forget about this by then” .
. I mean, I could’ve left your second post just hang in there, but nooo… 
Potentially productive conversations (like what this one has already turned out to be) are too rare in this forum to miss out on.
.
.