Community discussions

MikroTik App
 
rubence
just joined
Topic Author
Posts: 24
Joined: Mon Jan 24, 2011 10:06 am

need to save simple queue output to a file

Mon Feb 08, 2016 3:47 pm

hello all,

/queue simple print stats file=usage.txt

output is

0 name="queue1" target=192.168.230.52/32 rate=0bps/0bps
total-rate=0bps packet-rate=0/0 total-packet-rate=0 queued-bytes=0/0
total-queued-bytes=0 queued-packets=0/0 total-queued-packets=0
bytes=6751734/192510318 total-bytes=0 packets=90242/148282
total-packets=0 dropped=0/1894 total-dropped=0 pcq-queues=0/0

1 name="queue2" target=192.168.210.128/32 rate=0bps/0bps
total-rate=0bps packet-rate=0/0 total-packet-rate=0 queued-bytes=0/0
total-queued-bytes=0 queued-packets=0/0 total-queued-packets=0
bytes=40912239/1475112261 total-bytes=0 packets=625483/1019896
total-packets=0 dropped=0/9833 total-dropped=0 pcq-queues=0/0

2 name="queue3" target=192.168.250.61/32 rate=88.9kbps/2.3Mbps
total-rate=0bps packet-rate=161/201 total-packet-rate=0
queued-bytes=0/1500 total-queued-bytes=0 queued-packets=0/1
total-queued-packets=0 bytes=385383685/11254203955 total-bytes=0
packets=5979883/7744613 total-packets=0 dropped=0/1088 total-dropped=0
pcq-queues=1/1


but i want only name,target and bytes (if possible in megabytes) will be write in one line each and saved in file . it'll be like

0 name="queue1" target=192.168.230.52/32 bytes=6751734/192510318
1 name="queue2" target=192.168.210.128/32 bytes=40912239/1475112261
2 name="queue3" target=192.168.250.61/32 bytes=385383685/11254203955

thanks in advace
 
User avatar
Deantwo
Member
Member
Posts: 331
Joined: Tue Sep 30, 2014 4:07 pm

Re: need to save simple queue output to a file

Wed Feb 10, 2016 3:33 pm

Try this:
{
    :local filename "usage.txt"
    :local message "Queues:"
    :local queueList [/queue simple find]
    :foreach q in=$queueList do={
        :local qName [/queue simple get $q name]
        :local qTarget [/queue simple get $q target]
        :local qBytes [/queue simple get $q bytes]
        :set message ($message . "\r\n" . "name=\"" . $qName . "\" target=" . $qTarget . " bytes=" . $qBytes)
    }
    /file print file=$filename
    :delay 1
    /file set [/file find name=$filename] contents=$message
}
It is a little hacky maybe, but you can also format the file however you want here too.

If this is the file you are trying to transfer in your other thread, it could all be replaced by one simple API query.
/queue/simple/print
=.proplist=name,target,bytes
That is if you have any programming skills.
The API can be a lot of fun to play with.
Last edited by Deantwo on Wed Feb 10, 2016 4:40 pm, edited 2 times in total.
 
rubence
just joined
Topic Author
Posts: 24
Joined: Mon Jan 24, 2011 10:06 am

Re: need to save simple queue output to a file

Wed Feb 10, 2016 4:19 pm

{
:local message "Queues:"
:local queueList [/queue simple find]
:foreach q in=$queueList do={
:local qName [/queue simple get $q name]
:local qTarget [/queue simple get $q target]
:local qBytes [/queue simple get $q bytes]
:set message ($message . "\r\n" . "name=\"" . $qName . "\" target=" . $qTarget . " bytes=" . $qBytes) }
/file print file="usage.txt"
:delay 1
/file set "usage.txt" contents=$message
}

its not working . the content of usage.txt always showing the file list of files folder. can u check in a router by yourself plz ?
 
rubence
just joined
Topic Author
Posts: 24
Joined: Mon Jan 24, 2011 10:06 am

Re: need to save simple queue output to a file

Wed Feb 10, 2016 4:34 pm

hey,
your script is ok i think
its working in router where i've 20 queues . but with 700 queues same script is not working.
 
User avatar
Deantwo
Member
Member
Posts: 331
Joined: Tue Sep 30, 2014 4:07 pm

Re: need to save simple queue output to a file

Wed Feb 10, 2016 4:35 pm

its not working . the content of usage.txt always showing the file list of files folder. can u check in a router by yourself plz ?
Works on my end.
What is the error you are getting when running it in the terminal?

I did have a few issues with the file content not being written at first because the file took a second to be created, which is why I added the ":delay 1" call.
its working in router where i've 20 queues . but with 700 queues same script is not working.
700 queues? O_O
Well, what is the error message when you paste the script into the terminal?

Could be it is reaching the string limit...
 
rubence
just joined
Topic Author
Posts: 24
Joined: Mon Jan 24, 2011 10:06 am

Re: need to save simple queue output to a file

Wed Feb 10, 2016 7:44 pm

Well, what is the error message when you paste the script into the terminal?

Could be it is reaching the string limit...

[rubence@MikroTik] > {
{... :local message "Queues:"
{... :local queueList [/queue simple find]
{... :foreach q in=$queueList do={
{{... :local qName [/queue simple get $q name]
{{... :local qTarget [/queue simple get $q target]
{{... :local qBytes [/queue simple get $q bytes]
{{... :set message ($message . "\r\n" . "name=\"" . $qName . "\" target=" . $qTarget . " bytes=" . $qBytes) }
{... /file print file="usage.txt"
{... :delay 1
{... /file set "usage.txt" contents=$message
{... }
action timed out - try again, if error continues contact MikroTik support and send a supout file (13)
[rubence@MikroTik] >

string limit is ok . cause
/queue simple print stats file=usage.txt
is creating file with all information
Last edited by rubence on Wed Feb 10, 2016 8:10 pm, edited 2 times in total.
 
User avatar
boen_robot
Forum Guru
Forum Guru
Posts: 2400
Joined: Thu Aug 31, 2006 4:43 pm
Location: europe://Bulgaria/Plovdiv

Re: need to save simple queue output to a file

Wed Feb 10, 2016 7:48 pm

Make the delay 2s, not 1s.

The HDD writes every 1.5s, so if you're using 1s, you're gambling on exactly when the HDD read and writes are going to be. The other router is on a lucky streak, but it would eventually start to fail as well with 1s.
 
rubence
just joined
Topic Author
Posts: 24
Joined: Mon Jan 24, 2011 10:06 am

Re: need to save simple queue output to a file

Wed Feb 10, 2016 7:49 pm

working mikrotik router : hAP lite package version 6.28
NOT working router: CCR1036-12G-4S version 6.34
Make the delay 2s, not 1s.

The HDD writes every 1.5s, so if you're using 1s, you're gambling on exactly when the HDD read and writes are going to be. The other router is on a lucky streak, but it would eventually start to fail as well with 1s.
tried with 2 s as well as 10 s . result is same
 
User avatar
boen_robot
Forum Guru
Forum Guru
Posts: 2400
Joined: Thu Aug 31, 2006 4:43 pm
Location: europe://Bulgaria/Plovdiv

Re: need to save simple queue output to a file

Wed Feb 10, 2016 8:25 pm

hAP lite with 700 queues? No wonder it's failing... The script is probably resorting to using the HDD as RAM ("swap"), but by then it becomes so slow that the script times out.

EDT: Oh wait, you're saying that one is working... But is it one with just 20 queues? And the CCR has 700 queues? Well, either way...

I suggest you rethink your approach to the problem. The API seems to be the way to go - from a remote device, just call "print" (as Deantwo showed), and write the results to wherever in whatever format you want. If you insist on storing them at the router, that's fine too, just use the API to actually traverse and generate the thing. This will consume much less memory at the router, and instead consume that memory at the device where the API application is running.

Neither the installation nor the code needed is (too) complicated. The only real downside is the presence of an additional device in the overall scheme of things.
 
rubence
just joined
Topic Author
Posts: 24
Joined: Mon Jan 24, 2011 10:06 am

Re: need to save simple queue output to a file

Wed Feb 10, 2016 9:05 pm


I suggest you rethink your approach to the problem. The API seems to be the way to go - from a remote device, just call "print" (as Deantwo showed), and write the results to wherever in whatever format you want. If you insist on storing them at the router, that's fine too, just use the API to actually traverse and generate the thing. This will consume much less memory at the router, and instead consume that memory at the device where the API application is running.

Neither the installation nor the code needed is (too) complicated. The only real downside is the presence of an additional device in the overall scheme of things.
well, actually i was working with different approach. i was storing information with simple command "/queue simple print stats file=$filename " and sending the file to remote linux machine for further processing with the help of perl and php . it would be easy for me if i can sent only required information in one line .

i never tried mikrotik api . what u think ? should i study api or sending raw file to linux machine is ok .. ?

thank you both for help .
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4047
Joined: Wed May 11, 2011 6:08 pm

Re: need to save simple queue output to a file

Wed Feb 10, 2016 9:07 pm

This sounds like a billing issue - why not play with RADIUS accounting, since that naturally sends you statistics such as how much data the user transferred....
 
rubence
just joined
Topic Author
Posts: 24
Joined: Mon Jan 24, 2011 10:06 am

Re: need to save simple queue output to a file

Wed Feb 10, 2016 9:15 pm

This sounds like a billing issue - why not play with RADIUS accounting, since that naturally sends you statistics such as how much data the user transferred....
any simple documentation with examples ?? never did radius ...
 
User avatar
boen_robot
Forum Guru
Forum Guru
Posts: 2400
Joined: Thu Aug 31, 2006 4:43 pm
Location: europe://Bulgaria/Plovdiv

Re: need to save simple queue output to a file

Wed Feb 10, 2016 9:27 pm

i never tried mikrotik api . what u think ? should i study api or sending raw file to linux machine is ok .. ?
Using the API allows you to address the very problem with sending the raw file - information filtering.

The API can easily fetch just the stuff you want, and the parsing is pretty much done by the API client.

Using the API is closer to what you're already used to, although I'd have to agree that using RADIUS may be better in the long run for this case.
 
rubence
just joined
Topic Author
Posts: 24
Joined: Mon Jan 24, 2011 10:06 am

Re: need to save simple queue output to a file

Wed Feb 10, 2016 9:47 pm

although I'd have to agree that using RADIUS may be better in the long run for this case.
few times in googling i found documentation about radius+mikrtoik is for pppoe or hotspot. i am using static ip addresses for each client. is it possible to collect download upload data regularly from simple queue with queue name and ip address through radius server ??
 
User avatar
ZeroByte
Forum Guru
Forum Guru
Posts: 4047
Joined: Wed May 11, 2011 6:08 pm

Re: need to save simple queue output to a file

Wed Feb 10, 2016 10:38 pm

few times in googling i found documentation about radius+mikrtoik is for pppoe or hotspot. i am using static ip addresses for each client. is it possible to collect download upload data regularly from simple queue with queue name and ip address through radius server ??
No - RADIUS is for session authentication and accounting, so a static-assigned interface wouldn't be able to do anything with RADIUS.
Probably an SNMP-based solution would be helpful if each user has their own interface that you can poll - or it might be possible that the queues' statistics are available as SNMP-readable OIDs...
 
rubence
just joined
Topic Author
Posts: 24
Joined: Mon Jan 24, 2011 10:06 am

Re: need to save simple queue output to a file

Wed Feb 10, 2016 11:03 pm

Probably an SNMP-based solution would be helpful if each user has their own interface that you can poll - or it might be possible that the queues' statistics are available as SNMP-readable OIDs...
tried with cacti, probably its also not working because of 700+ queues :( check my other thread http://forum.mikrotik.com/viewtopic.php?f=2&t=104433

anyway, now i can collect data periodically from router. now i'll process the data through perl,mysql and php for reporting . thank you everyone.
 
IntLDaniel
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Apr 04, 2019 7:21 pm

Re: need to save simple queue output to a file

Tue Feb 01, 2022 1:19 pm

Try this:
{
    :local filename "usage.txt"
    :local message "Queues:"
    :local queueList [/queue simple find]
    :foreach q in=$queueList do={
        :local qName [/queue simple get $q name]
        :local qTarget [/queue simple get $q target]
        :local qBytes [/queue simple get $q bytes]
        :set message ($message . "\r\n" . "name=\"" . $qName . "\" target=" . $qTarget . " bytes=" . $qBytes)
    }
    /file print file=$filename
    :delay 1
    /file set [/file find name=$filename] contents=$message
}
It is a little hacky maybe, but you can also format the file however you want here too.

If this is the file you are trying to transfer in your other thread, it could all be replaced by one simple API query.
/queue/simple/print
=.proplist=name,target,bytes
That is if you have any programming skills.
The API can be a lot of fun to play with.
I know this is the old thread but I did find this script useful to export queue stats every months (and then reset them). BUT this script was (and still is) limited due to variable size 4096 bytes (see here viewtopic.php?t=109968#p655210 ). That is why if you try to export more than ~20 queues it does not work - the $message variable cannot hold so much strings :-( . So I was thinking if I could write every queue stats separately one by one and append it to the file (the same like /log print append file= ... does), but it is probably still not available (ROS 6.49.2 or ROS 7.2RC3) ? Otherwise you have to always read current content of the file first and it is the same problem :-(

Maybe read every queue to separate message variables (message1, message2...) and then somehow write to file all these messages (i.e. 500) one by one to that output file? Is there any solution to this task in 2022? Thanks.
 
User avatar
Jotne
Forum Guru
Forum Guru
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: need to save simple queue output to a file

Tue Feb 01, 2022 5:30 pm

You can send each data line to syslog server instead of a file. Then you can use an external server to examine logs.
Should be easy to implement to my Splunk/Mikrotik prosjekt. See signature.

Here is an example to test on your terminal:
{
	/queue simple
	:local queueList [find]
	:foreach q in=$queueList do={
		:local qName [get $q name]
		:local qTarget [get $q target]
		:local qBytes [get $q bytes]
		:local qStatus [get $q disabled]
		:if ($qStatus) do={
			:set $qStatus "no"
		} else={
			:set $qStatus "yes"
		}
		:put ("name=\"" . $qName . "\" target=" . $qTarget . " bytes=" . $qBytes . " enabled=" . $qStatus)
	}
}
To send to syslog, change ":put" with ":log info message="
 
IntLDaniel
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Apr 04, 2019 7:21 pm

Re: need to save simple queue output to a file

Thu Feb 10, 2022 3:34 pm

You can send each data line to syslog server instead of a file. Then you can use an external server to examine logs.
Should be easy to implement to my Splunk/Mikrotik prosjekt. See signature.

Here is an example to test on your terminal:
{
	/queue simple
	:local queueList [find]
	:foreach q in=$queueList do={
		:local qName [get $q name]
		:local qTarget [get $q target]
		:local qBytes [get $q bytes]
		:local qStatus [get $q disabled]
		:if ($qStatus) do={
			:set $qStatus "no"
		} else={
			:set $qStatus "yes"
		}
		:put ("name=\"" . $qName . "\" target=" . $qTarget . " bytes=" . $qBytes . " enabled=" . $qStatus)
	}
}
To send to syslog, change ":put" with ":log info message="
Thanks for reply! Could you be please more specific and put here the whole line with :log info message command? i have probably some syntax error there, does not work.
 
User avatar
Jotne
Forum Guru
Forum Guru
Posts: 3279
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: need to save simple queue output to a file

Thu Feb 10, 2022 3:43 pm

Try change from
:put ("name=\"" . $qName . "\" target=" . $qTarget . " bytes=" . $qBytes . " enabled=" . $qStatus)
to:
:local logmessage ("name=\"" . $qName . "\" target=" . $qTarget . " bytes=" . $qBytes . " enabled=" . $qStatus)
:log info message="$logmessage"
 
IntLDaniel
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Thu Apr 04, 2019 7:21 pm

Re: need to save simple queue output to a file

Thu Feb 10, 2022 11:04 pm

yes it works :D Thanks!
BTW Am I able to log it to independent log file..only output from this script?
 
DarielHG89
just joined
Posts: 3
Joined: Sun Dec 09, 2018 3:08 pm

Re: need to save simple queue output to a file

Thu Sep 22, 2022 4:17 am

Store statistics info in a remote server.
In this example i just post the name and bytes.
Script:
{
	/queue simple
	:local queueList [find]
	:foreach q in=$queueList do={
		:local qName [get $q name]
		:local qTarget [get $q target]
		:local qBytes [get $q bytes]
		:local qStatus [get $q disabled]
		:if (!$qStatus) do={
			:local logmessage ("{\"name\":\"" . $qName . "\",\"bytes\":\"" . $qBytes . "\"}")
			:put ($logmessage)
			/tool fetch http-method=post http-header-field="Content-Type: application/json" http-data=$logmessage url="http://192.168.100.17/queue.php"
		}
	}
}
queue.php:
<?php
  function isValidJSON($str) {
     json_decode($str);
     return json_last_error() == JSON_ERROR_NONE;
  }
  $json = file_get_contents('php://input');
  if (strlen($json) > 0 && isValidJSON($json)){
    $data = json_decode($json);
	date_default_timezone_set("America/New_York");
	$njsond = ["LastC"=>date("d-m-Y h:i:sa"),"bytesUD"=>$data->bytes];
	echo file_put_contents("Queue-" . $data->name . ".json",json_encode($njsond));
  }
?>
Last edited by DarielHG89 on Thu Sep 22, 2022 4:30 am, edited 3 times in total.
 
DarielHG89
just joined
Posts: 3
Joined: Sun Dec 09, 2018 3:08 pm

Re: need to save simple queue output to a file

Sat Sep 24, 2022 2:56 am

This will reset stats for every queue stat uploaded, schedule to 5s.

Script:
{
	/queue simple
	:local queueList [find]
	:foreach q in=$queueList do={
		:local qName [get $q name]
		:local qBytes [get $q bytes]
		:local qBLength [:len $qBytes]
		:local qStatus [get $q disabled]
		:if (!$qStatus && $qBLength>5) do={
			:local logmessage ("{\"name\":\"" . $qName . "\",\"bytes\":\"" . $qBytes . "\"}")
#			:log info message="$qBLength"
#			:log info message="$logmessage"
			reset-counters $q
			/tool fetch http-method=post http-header-field="Content-Type: application/json" http-data=$logmessage url="http://192.168.100.17/queue.php" output=none
		}
	}
}
Queue.php:
<?php
function isValidJSON($str)
{
    json_decode($str);
    return json_last_error() == JSON_ERROR_NONE;
}

$json = file_get_contents('php://input');
if (strlen($json) > 0 && isValidJSON($json))
{
    if (!is_dir('Queue'))
    {
        mkdir('Queue', 0777);
    }
	
    $data = json_decode($json);

    $lstchkbU = 0; $lstchkbD = 0;

    $Queuefile = "Queue/Queue-" . $data->name . ".json";

    if (file_exists($Queuefile))
    {
        $file = file_get_contents($Queuefile, FILE_USE_INCLUDE_PATH);
        $datajf = json_decode($file);
        $lbs = $datajf->bytes;
		$fbytesUD = explode("/", $lbs);
		$lstchkbU = $fbytesUD[0];
		$lstchkbD = $fbytesUD[1];
    }

    $bytesUD = explode("/", $data->bytes);
    $bytesUp = $bytesUD[0] + $lstchkbU;
    $bytesDn = $bytesUD[1] + $lstchkbD;
    $bytesUDstr = $bytesUp . "/" . $bytesDn;
    date_default_timezone_set("America/New_York");
    $njsond = ["LastC" => date("d-m-Y h:i:sa") , "bytes" => $bytesUDstr];
    echo file_put_contents($Queuefile, json_encode($njsond));
}

?>

Who is online

Users browsing this forum: nocivo and 18 guests