queue tree statistics when there is >2000 with longnames

RouterOS Scripting
1

I need to download statistics of queue tree. However, there is many of them ~ 2300. All of them have loong descriptive names which render tool /queue tree print bytes unusable, cos they simply don’t fit into terminal output and names are partialy replaced with dots.

I have managed to put statistics to file, but there is problem with max file size arround 4000b (then I got error -contents of file too long or such)


I tried to send stats by e-mail body, but this is not optimal for further parsing by our system and there is probably another limit too, forcing me to divide mails to 5 parts/500 queue.

one of my test

:global pocetQT;
:set pocetQT (get  [/queue tree print count-only]);
:global traf;
:global resulter " ";
:global datum [/system clock get date];
:global vrijeme [/system clock get time];
:global CustMark "none ";

/file print file=stats1;
/file set stats1.txt contents="";
/queue tree
print count-only
:for i from=0 to=499 do={
	:global CustDATA [/queue tree get $i bytes ]
    :global CustName [/queue tree get $i name]
	:global CustMark [/queue tree get $i packet-mark]
	:if ($CustDATA ="") do={
                :global CustDATA "0"
            }
:if ($CustMark ="") do={
                :global CustMark "none"
            }
:put (($i) . ";" . ($CustDATA) . ";" . ($CustName))
 :set resulter ("$resulter \n;  $CustMark;$CustName;$CustDATA;")
 /file set stats1.txt contents=$resulter;
}
/tool e-mail send from=me@me.org to=recepient@server.cpm subject="1_data_radio  $datum $vrijeme" body="$resulter" file=stats1.txt server=XXXXXXXXXXX

There must be a better solution for that, but I’m stuck and hopeless now.
Is there any way how to for example append contents to existing file ?

thanks for any suggestion

2

The scripting language can only handle files up to 4,096 bytes.

Your best bets would be access via the API, or via externally scripted telnet/SSH. To make sure names are fully printed use “/queue tree print detail” or “/queue tree print terse”.

On an unrelated note to the immediate issue at hand: 2300 queue trees is nuts. Have you considered using PCQ instead?

3

thanks for confirmation of scripting limit

you’re right, that amount of queue is a little bit nuts, now is running on DELL1850, 2x dualcore Xeon, with 2x100mbit internet lines, shaping arround 1000 clients, hundreds of APs, in transparent bridge mode and it works perfectly for now.

This solution is based mainly on historical reasons, we running that network for 10 years.

We wrote special modules to our complex accounting system so it is able to generate whole shaper rules and queues and CPE settings based on actual database records, we spent half of year with that :frowning:

The reasons why I don’t use PCQ (I have studied that many times specially Janis Megis MUM QOS presentation http://tiktube.com/index.php?video=LLco3bmqeEJoIplpJLovmypxKlEoJqmF= and Valens Riyadi’s http://tiktube.com/index.php?video=KJeq3cHHiHEGCnmCJCHrJxGvIlFoDEKK=.. Valens described why I am not using PCQ…) are

-dozens of different clients subsciption programs
-no detailed settings on leaf queue/individual treatment of customer
-no detailed stats per user
-no priority control over users in one group
-no burst on ROS 4.X which I’m using now if I’m not wrong

I am aware that PCQ would be times easier to work with and setup.

Unfortunately I can’t take numbers from mangle, because in order to keep mangle rules as small as possible, I just mangle trafic from IP and shaper uses the same connection marks/packet marks for upload and download queue, so clients uploads a downloads are mixed together at that point.

I need that statistics info at least to achieve kind of fair policy for users, to get lower priority for all-time downloaders and automatically give them better priority when they stop acting like that, and to give better priority for clients who using service ocasionally+that stats are further used for monitoring where at our network we have bottlenecks and where to improve transport capacity and many other financial accounting info

I just considering to use php api and I have done some tests of php API so far, however, I’m far from being php king and having difficulties to parse API’s output to get machine readable and processable data format just on queue tree.

(based on php API in the mikrotik wiki)
this code still needs some tuning to produce usable output :frowning:

<?php

require('routeros_api.class.php');

$API = new routeros_api();

$API->debug =false;

if ($API->connect('111.111.111.111', 'user', 'password')) {

   $ARRAY = $API->comm('/queue/tree/getall');
   print_r($ARRAY);

   $API->disconnect();

}

?>
4

Actually thinking about it now that I’ve had coffee, the best solution would be to use SNMP to query the router. That’s precisely what SNMP was designed to do.

Sorry for not realizing that earlier.

5

@jonashi

I wish I could PM you the following, but I seem to not have permissions for it, so I’m forced to say it publically, and in the process feeling like a spammer due to the fact this is now the 3rd topic I do it in:

Check out the client at my signature. It parses all responses, so gathering useful data should be easier.

6

@ boen_robot
thanks. As I am not a php expert, I’m still having difficulties to get some output of your API, so far I have solved all obstacles in environment to get your API running, (Phar. Pear… etc), as I can’t install it on our BSD production systems, so I had to prepare testing environment on XP using WAMP distribution.

However, still doesn’t ring a bell.

Please can you give me a code snippet to get some queue tree bytes and save the result to file on webserver?

It seem that I’m not allowed to send PM either. If I could, I would like to contact you directly.
I would pay some amount of money to anyone who help me get this task done.

Your message is not spam, and is highly appreciated by me.

Thank you very much.

7

Given your high number of queues, you’ll probably need to use sendAsync(), as sendSync() will fill up your web server’s memory pretty quick.

I must say “thank you”, because in trying to come up with a snippet, I saw a bug - completeRequest() doesn’t work properly when there is a callback. Until I fix this though, loop() is an equally good alternative, i.e.:

<?php
namespace PEAR2\Net\RouterOS;
require_once 'PEAR2/Net/RouterOS/Autoload.php';

$client = new Client('111.111.111.111', 'user', 'password');

$printRequest = new Request('/queue/tree/print .proplist=.id,packet-mark,name,bytes detail', 'print');
$file = fopen(
    /*
    Replace with the filename you want.
    Basing the name on the date would mean taking care
    of characters that can't be part of a valid file name though.
    */
    str_replace(':', '_', date('c')) . '.txt',
    'wb'
);

$client->sendAsync(
    $printRequest,
    function ($response) use ($file) {
        if ($response->getType() === Response::TYPE_DATA) {
            fwrite(
                $file,
                $response->getArgument('.id') . ';' .
                $response->getArgument('packet-mark') . ';' .
                $response->getArgument('name') . ';' .
                $response->getArgument('bytes') . "\n"
            );
        }
    }
);
$client->loop();

fclose($file);

Note that I don’t really use queue trees. This snippet is based on a single member in the queue tree that I added specifically to test the snippet. Because I don’t use them, I was never able to populate the “bytes” field (it was never returned in the response and therefore my call for it returned NULL). I’m willing to bet that if there’s any traffic on a queue, this argument will be present, so it should work.

BTW, what problems did you have with installing this on BSD? I don’t use anything Windows specific, and AFAIK, the authors of Pyrus/PEAR(2) are Linux guys, so they’d know if there’s a big UNIX related problem with Pyrus/PEAR(2). Or do you simply mean that you’re intentionally delaying the production deployment until you get it working on the test (not that there’s a problem installing it on BSD)?

Also, Pyrus/PEAR/Phar is not required; Just reccomended to make upgrades easier (and for those who already use Pyrus/PEAR for one reason or another - installation is easier too); You could just extract the archive and include the Autoload.php file.

8

@boen_robot karma+

thanks ! I owe you one !
this method works, I’m getting file with almost all required info, however, only one but important parameter missing - bytes.

There si no problem installing API on BSD, works like a charm. My problem was that I wasn’t allowed to test on our production machines, so finaly I had to install separate one for testing purposes as WAMP on XP doesn’t work properly with phar.

I’m almost done, now I have to just solve why I’m not getting queue bytes.

Output in the file looks like that :

*10005EE;10.64.15.68/32_pkt_dwn;cust_name_cust_program_dw;
*10005EF;10.64.15.70/32_pkt_dwn;cust_name_cust_program_dw;
*10005F0;10.64.15.72/32_pkt_dwn;cust_name_cust_program_dw;
*10005F1;10.64.15.74/32_pkt_dwn;cust_name_cust_program_dw
*10005F2;10.64.15.76/32_pkt_dwn;cust_name_cust_program_dw;
*10005F3;10.64.15.244/31_pkt_dwn;cust_name_cust_program_dw;
*10005F7;10.64.15.250/31_pkt_dwn;cust_name_cust_program_dw;
*10005F8;10.64.183.252/31_pkt_dwn;cust_name_cust_program_dw;
*10005F9;10.64.183.10/32_pkt_dwn;cust_name_cust_program_dw;

note: I have replaced customer name and customer program parameters here to keep their privacy.
There are all 2012 queues present in the file.

Maybe problem is that command /queue tree print bytes doesn’t return just number of bytes, but something like this

ID----NAME--------------------PARENT------PACKET-MARK-----------------------------  LIMIT-AT--  BYTES
2001  cust_name_cust_program  parent      10.64.67.10/32_pkt_dwn                    32k        21552            >
2002  cust_name_cust_program  parent      10.64.67.12/32_pkt_dwn                    32k        1761284          >
2003  cust_name_cust_program  parent      10.64.67.12/32_pkt_dwn                    32k        6449445          >

so I need to fetch an array for that variable and filter-out only BYTES from that array ? Or am I completely wrong?

9

you have to give same CLI command just adapted to API “/” instead of space etc.

then you will get all the required attributes

try =detail= or =stats= to get bytes

10

Yes. As janisk suggested, using stats instead of (or in addition to) details is the key. It worked even for me (i.e. it game me 0, since there isn’t traffic).

=>
replace

$printRequest = new Request('/queue/tree/print .proplist=.id,packet-mark,name,bytes detail', 'print');

with

$printRequest = new Request('/queue/tree/print .proplist=.id,packet-mark,name,bytes stats', 'print');

or

$printRequest = new Request('/queue/tree/print .proplist=.id,packet-mark,name,bytes detail stats', 'print');



The “.proplist” argument filters the response so that it contains only the specified properties. If you want to get additional data, you’ll either have to remove that argument (note: doing that would increase the memory consumed at your web server, especially if along with the long descriptive names you have long descriptive comments) or add the other stuff you want to it.

11

@janisk
I have tried suggested options without luck so far.
Maybe I’m not smart enough :confused:
I suppose that I should have implemented something like this in MT scripting.
[/queue tree get $i bytes ] where $i is the id of queue, but I don’t know how to implement into the snipet provided by boen_robot. Looks to me that method used in this example works with mikrotik “print” which isn’t probably the best solution for this case.
I’m going to study boen_robots API and Mikrotik API commands adn documentation and syntax deeper. Now I’m stuck again.

12

Did you tried it by just doing the adjustment in my last post, or did you used some other syntax?

Thanks to the fact you’re using sendAsync() with a callback, there isn’t really a problem. Without a callback, all entries in the queue list will be copied in your web server’s memory (and that’s bad). With a callback, only one entry at a time is stored in the web server’s memory. The callback itself writes it to a file, which is how your resulting file has all entries in it.

13

@boen_robot
yeap ! It does work. I finaly got it. Sorry for being such dumb. Would you accept €20 as my thanks?
If so, please contact me - by e-mail forboenrobot email.cz …I’d like to send you money thru paypal.
Thanks. :smiley: :smiley: :smiley:

14

That’s very kind of you, thanks.

(I’ve sent an email to that email already)

[edit]Reward received[/edit]

15

reward sent. My pleasure :slight_smile:

Have a nice day.