Insert PPPoe client bandwidth into mysql

Hi all,

I’m trying to design some quality automatic collection of pppoe clients average bandwidth data into mysql database.
What I did is wrote a shell script which pulls usernames from radius db, equals them with interface OIDs (bytes-in and bytes-out) from mikrotik routers in regular intervals (e.g. 3 secs) and calculates bandwidth.
I faced a problem where not all pppoe usernames were listed, bytes-in or bytes-out values were not pulled from router properly, maybe because of ssh connection or something else. Anyway, I think this is not the best approach for automatism.

Does anybody uses some monitoring tools (dude, cacti,…), MT API maybe, for pulling these informations from routers, is it possible, and how do you update your database with proper values?

It would probably be best if you fetch queue information. It’s trivial using simple queues, where you can simply capture the queue name and current bytes-in/out, and thus map the queue name to a username. I’m not sure how things would work out if you’re using queue trees. Then again, perhaps this OID way you’ve found is the right one in that case, I don’t know.

Anyway… the API protocol should provide a relatively stable way to do what you want. How does your current SSH based script look like? Perhaps I can help rewrite it with the API for you. Any preference/requirement for a language?

I don’t want to involve queues here…

Probably someone can use this script and it will work out of the box, for me it didn’t, I had problems with one of my routers where ssh values sometimes passed, sometimes didn’t…
Actually, logic built inside is not very good because of bandiwidth calculation.
Probably the best way would be to pull inoctets, outoctets, exact timestamp but from SNMP information, and then calculate bandwidth based on two timestamps…

It’s nice of you boen_robot that you offered to rewrite the code for MT API, but let’s figure out first the best and accurate way to pull RX/TX values and get true information.

#!/bin/bash

uname=
pass=
dbasez=
dbaser=
tablez=
tabler=
delta=2
community=
log=/tmp/pppoe_bw.log
echo `date '+%Y-%m-%d %H:%M:%S'` >> $log
# list nas names
for router in `mysql -B --default-character-set=latin2 -u$uname -p$pass --disable-column-names -e "use $dbaser; select nasname from $tabler"`
do
        # remember NAS name for DB insert
        nasshortname=`mysql -B --default-character-set=latin2 -u$uname -p$pass --disable-column-names -e "use $dbaser; select shortname from $tabler WHERE nasname='$router'"`
        echo `date '+%Y-%m-%d %H:%M:%S'` >> $log
        echo $router >> $log
        # list users from nas names and get tx/rx values
        ssh -l admin $router interface print where type=pppoe-in | grep "pppoe-in"|awk '{print $3}'|cut -f2 -d'-'|cut -f1 -d'>' > /tmp/users
        for user in `cat /tmp/users`
        do
                echo $user >> $log
                ssh -l admin $router interface print oid where name="<pppoe-$user>" |grep "bytes-in" | awk -F"=" '{print $NF}' > /tmp/byts_in_oid
                bytes_in_oid=`cat /tmp/byts_in_oid`
                echo $bytes_in_oid >> $log
                ssh -l admin $router interface print oid where name="<pppoe-$user>" |grep "bytes-out" | awk -F"=" '{print $NF}' > /tmp/byts_out_oid
                bytes_out_oid=`cat /tmp/byts_out_oid`
                echo $bytes_out_oid >> $log
				# control if OID variables are empty
                if [[ ! $bytes_in_oid == "" ]] && [[ ! $bytes_out_oid == "" ]]; then
                        # rx user
                        jedan=`snmpwalk -v1 -c $community $router $bytes_in_oid |awk '{print $4}'`
                        sleep $delta
                        dva=`snmpwalk -v1 -c $community $router $bytes_in_oid |awk '{print $4}'`
                        rx=`echo "($dva-$jedan)/$delta"|bc`
                        # tx user
                        jedan=`snmpwalk -v1 -c $community $router $bytes_out_oid |awk '{print $4}'`
                        sleep $delta
                        dva=`snmpwalk -v1 -c $community $router $bytes_out_oid |awk '{print $4}'`
                        tx=`echo "($dva-$jedan)/$delta"|bc`
                        date=`date '+%Y-%m-%d %H:%M:%S'`
                        echo "$user, $rx, $tx, $nasshortname" >> $log
                        mysql -B --default-character-set=latin2 -u$uname -p$pass -e "use $dbasez; INSERT into $tablez (username, timestamp, rxrate, txrate,nasshortname) VALUES ('$user', str_to_date('$date','%Y-%m-%d %H:%i:%s'), $rx, $tx, '$nasshortname')"
                fi
        done
done
exit 0

From the looks of it, the problem is you’re doing too many distinct commands, so the data from one isn’t nicely paired with the other, leading to cases where the data looks droped, when in fact it’s simply being “late”.

I wanted to see the script mostly because I wasn’t sure what data you’re after. I didn’t knew PPPoE creates a new interface for each user, so I suppose what you’re trying is the right way… it’s just using a bad tool for the job.

Here’s an attempt to rewrite this in PHP using the client from my signature, and the MySQLi extension:

<?php
namespace PEAR2\Net\RouterOS;
require_once 'PEAR2_Net_RouterOS-1.0.0b3.phar';

$config = array(
    'db' => array(
        'hostname'  => '127.0.0.1',
        'username'  => 'root',
        'password'  => '',
        'charset'   => 'latin2',
        'dst-db'    => 'statsDb',
        'src-db'    => 'usersDb',
        'dst-table' => 'users',
        'src-table' => 'stats'
    ),
    'router' => array(
        'hostname'  => '192.168.0.1',
        'username'  => 'admin',
        'password'  => ''
    )
);

try {
    //Estabilishing connections. This is where stuff is most likely to fail.
    $mysqli = new \mysqli($config['db']['hostname'], $config['db']['username'], $config['db']['password'], $config['db']['src-db']);
    $mysqli->setCharset($config['db']['charset']);
    $client = new Client($config['router']['hostname'], $config['router']['username'], $config['router']['password']);

    //Getting existing users
    $users = $mysqli->query("SELECT shortname FROM {$config['db']['src-table']} WHERE nasname = '{$config['router']['hostname']}'");
    
    //Constructing a query to later filter interfaces by.
    $query = Query::where('name', "<pppoe-{$users->fetch_object()->shortname}>");
    while ($user = $users->fetch_object()) {
        $query->orWhere('name', "<pppoe-{$user->shortname}>");
    }
    
    //Switching to the dst-db
    $mysqli->select_db($config['db']['dst-db']);
    
    //Because we'll be doing a lot of queries, we'll use a prepared statement.
    $insertStatement = $mysqli->prepare(
        "INSERET INTO {$config['db']['dst-table']}
            (username, timestamp, rxrate, txrate, nasshortname)
        VALUES (?, ?, ?, ?, '{$config['router']['hostname']}')"
    );
    $insertStatement->bind_param('ssss', $printedUser, $timestamp, $printedRx, $printedTx);
    
    //Prepare the printing request.
    $printRequest = new Request('/interface/print .proplist=name,rx-byte,tx-byte stats', $query);

    //If new users appear since we've started gathering stats, start collecting stats on them as well.
    $client->sendAsync(
        new Request('/interface/listen', null, 'listener'),
        function ($response) use (&$query) {
            if ($response->getType(Response::TYPE_DATA) && 0 === strpos($name = $response->getArgument('name'), '<pppoe-')) {
                $query->orWhere('name', $name);
            }
        }
    );
    
    //Starting the real deal.
    while (true) {
        //Because processing the entries might take over a second,
        //we'll set the time now.
        $timestamp = date('Y-m-d H:i:s');
        
        //Getting the current speeds.
        foreach ($client->sendSync($printRequest)->getAllOfType(Response::TYPE_DATA) as $response) {
            $printedUser = substr(strstr($response->getArgument('name'), '>', true), 7/*strlen('<pppoe-')*/);
            $printedRx = $response->getArgument('rx-byte');
            $printedTx = $response->getArgument('tx-byte');
            $insertStatement->execute();
        }
        
        //In case a new user was added while we were printing,
        //collect results for it at the next iteration.
        $printRequest->setQuery($query);
        sleep(3);
    }
} catch (\Exception $e) {
    die($e);
} 

You need to execute it from the command line, NOT from a browser, because it is intended to keep collecting stats in the background.

I don’t use PPPoE, so I haven’t tested this, but I see no reason why it shouldn’t work.

Actually, even if this works, scrap it… it will be very inefficient, and pose a relatively heavy burden on the router and server alike.

Here’s an alternative that makes a separate request for each interface using the monitor-traffic command:

<?php
namespace PEAR2\Net\RouterOS;
require_once 'PEAR2_Net_RouterOS-1.0.0b3.phar';

$config = array(
    'db' => array(
        'hostname'  => '127.0.0.1',
        'username'  => 'root',
        'password'  => '',
        'charset'   => 'latin2',
        'dst-db'    => 'statsDb',
        'src-db'    => 'usersDb',
        'dst-table' => 'users',
        'src-table' => 'stats'
    ),
    'router' => array(
        'hostname'  => '192.168.0.1',
        'username'  => 'admin',
        'password'  => ''
    ),
    'misc' => array(
        'interval' => 3
    )
);

try {
    //Estabilishing connections. This is where stuff is most likely to fail.
    $mysqli = new \mysqli($config['db']['hostname'], $config['db']['username'], $config['db']['password'], $config['db']['dst-db']);
    $mysqli->setCharset($config['db']['charset']);
    $client = new Client($config['router']['hostname'], $config['router']['username'], $config['router']['password']);
    
    //Because we'll be doing a lot of queries, we'll use a prepared statement.
    $insertStatement = array(
        $mysqli->prepare(
            "INSERET INTO {$config['db']['dst-table']}
                (username, timestamp, rxrate, txrate, nasshortname)
            VALUES (?, ?, ?, ?, '{$config['router']['hostname']}')"
        ),
        array(
            'user' => null,
            'timestamp' => null,
            'rx' => null,
            'tx' => null
        )
    );
    $insertStatement[0]->bind_param(
        'ssss',
        $insertStatement[1]['user'],
        $insertStatement[1]['timestamp'],
        $insertStatement[1]['rx'],
        $insertStatement[1]['tx']
    );
    
    //The interval is part of the command.
    //Make sure the socket timeout is larger than the value of "interval".
    $monitorRequest = new Request(
        '/interface/monitor-traffic .proplist=rx-bits-per-second,tx-bits-per-second'
    );
    $monitorRequest->setArgument('interval', $config['misc']['interval']);
    
    //This is the fun part. Upon receiving a response from the monitor,
    //the insert statement will be executed about it.
    $monitorListener = function ($response) use (&$insertStatement) {
        if ($response->getType(Response::TYPE_DATA)) {
            $insertStatement[1]['tiimestamp'] = date('Y-m-d H:i:s');
            
            $insertStatement[1]['user'] = substr(strstr($response->getArgument('name'), '>', true), 7/*strlen('<pppoe-')*/);
            $insertStatement[1]['rx'] = $response->getArgument('rx-bits-per-second');
            $insertStatement[1]['tx'] = $response->getArgument('tx-bits-per-second');
            
            $insertStatement[0]->execute();
        }
    };
    
    //If new users appear since we've started gathering stats, start collecting stats on them as well.
    $client->sendAsync(
        new Request('/interface/listen .proplist=.dead,type,name', null, 'listener'),
        function ($response, $client) {
            if ($response->getType(Response::TYPE_DATA)
                && 'yes' !== $response->getArgument('.dead')
                && 'pppoe-in' === $response->getArgument('type')
            ) {
                $client->sendAsync(
                    $monitorRequest->setArgument(
                        'interface', $name = $response->getArgument('name')
                    )->setTag($name),
                    $monitorListener
                );
            }
        }
    );

    //Start collecting stats on existing interfaces.
    foreach (
        $client->sendSync(
            new Request(
                '/interface/print .proplist=name',
                Query::where('type', 'pppoe-in')
            )
        )->getAllOfType(Response::TYPE_DATA) as $response
    ) {
        $client->sendAsync(
            $monitorRequest->setArgument(
                'interface', $name = $response->getArgument('name')
            )->setTag($name),
            $monitorListener
        );
    }
    
    //Activate event loop, i.e. the actual monitoring.
    $client->loop();
} catch (\Exception $e) {
    die($e);
} 

Using this approach, the server only ever performs requests at the start, and if new users appear, whereas previously, it did a request every 3 seconds. Not only does doing that make the server faster (because it doesn’t spend time sending requests), it also lightens the router load a little (since it doesn’t need to reinitialize the stats gathering process; it reuses the already initialized monitor).

And another thing… what’s with the users database and table? I’ve altered the script from the previous one, where now it takes all pppoe-in interfaces, and monitors them. After all, they’re the only ones that can actually be monitored for sure.

Did you tested this code?
I’m trying to run it, this is the output:

exception ‘PEAR2\Net\Transmitter\SocketException’ with message ‘Failed to initialize socket.’ in phar:///var/www/html/PEAR2_Net_RouterOS-1.0.0b3.phar/PEAR2_Net_RouterOS-1.0.0b3/src/PEAR2/Net/Transmitter/TcpClient.php:140
Stack trace:
#0 phar:///var/www/html/PEAR2_Net_RouterOS-1.0.0b3.phar/PEAR2_Net_RouterOS-1.0.0b3/src/PEAR2/Net/Transmitter/TcpClient.php(118): PEAR2\Net\Transmitter\TcpClient->createException(‘Failed to initi…’, 7)
#1 phar:///var/www/html/PEAR2_Net_RouterOS-1.0.0b3.phar/PEAR2_Net_RouterOS-1.0.0b3/src/PEAR2/Net/RouterOS/Communicator.php(111): PEAR2\Net\Transmitter\TcpClient->__construct(‘10.19.x.y’, 8728, false, NULL, ‘admin/password’, NULL)
#2 phar:///var/www/html/PEAR2_Net_RouterOS-1.0.0b3.phar/PEAR2_Net_RouterOS-1.0.0b3/src/PEAR2/Net/RouterOS/Client.php(121): PEAR2\Net\RouterOS\Communicator->__construct(‘10.19.x.y’, 8728, false, NULL, ‘admin/password’, NULL)
#3 /var/www/html/test.php(30): PEAR2\Net\RouterOS\Client->__construct(‘10.19.x.y’, ‘admin’, ‘password’)
#4 {main}

Next exception ‘PEAR2\Net\RouterOS\SocketException’ with message ‘Error connecting to RouterOS’ in phar:///var/www/html/PEAR2_Net_RouterOS-1.0.0b3.phar/PEAR2_Net_RouterOS-1.0.0b3/src/PEAR2/Net/RouterOS/Communicator.php:113
Stack trace:
#0 phar:///var/www/html/PEAR2_Net_RouterOS-1.0.0b3.phar/PEAR2_Net_RouterOS-1.0.0b3/src/PEAR2/Net/RouterOS/Client.php(121): PEAR2\Net\RouterOS\Communicator->__construct(‘10.19.x.y’, 8728, false, NULL, ‘admin/password’, NULL)
#1 /var/www/html/test.php(30): PEAR2\Net\RouterOS\Client->__construct(‘10.19.x.y’, ‘admin’, ‘password’)
#2 {main}

I didn’t, because I don’t use PPPoE, so I have nothing to test it on.

Still, this is a connection error, so I can definetly help with it…

  1. Make sure the API is enabled in the router:
/ip service enable api
  1. Make sure the PHP executable (“php.exe” on Windows; just “php” on UNIX) is allowed to make outgoing connections in your server’s firewall.

Yes, I figured out I need to enable API service on Mikrotik.

Usernames are not inserted in database… I would like to implement possibility to query the database table for all of my NAS mikrotiks in php loop and select users bandwidth from all routers in my network.
Also, we need to insert in code variable for timezone settings, because I get a warning:

PHP Warning: date(): It is not safe to rely on the system’s timezone settings. You are required to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected ‘Europe/Berlin’ for ‘CEST/2.0/DST’ instead in /var/www/html/test.php on line 65

Thank you!

For the timezone issue, you need to alter php.ini.

I’ll assume you haven’t altered PHP’s configuration before (judging by the error message…):

  1. Go to PHP’s folder.
  2. There should be two files - “php.ini-development” and “php.ini-production”. Choose one (I’d reccomend “development”, because by default, it displays all error messages on screen), and change its name to just “php.ini”.
  3. Open “php.ini”.
  4. Find “date.timezone” and change the whole line to
date.timezone=Europe/Berlin

(or whatever your real timezone is)

For the name issue… sorry, I missed adding the “name” to the monitor’s “.proplist”… but come to think of it, it’s not needed anyway. You can use the tag instead. Change:

$insertStatement[1]['user'] = substr(strstr($response->getArgument('name'), '>', true), 7/*strlen('<pppoe-')*/); 

to

$insertStatement[1]['user'] = substr(strstr($response->getTag(), '>', true), 7/*strlen('<pppoe-')*/); 

Thank you very much! This works as expected :slight_smile:

Can you tell me please, what will happen if connection gets lost between server executing the script and Mikrotik? Since this is wireless link I must count on such scenario… Will process fail?

Yes, further stats gathering will stop, and an exception is thrown. As written, the script will just “die” with the exception message, and you’d have to manually restart it when the connection is back up.

You could make it not die by surronding the whole try-catch statement with a loop, and in the catch just sleep() for a while before retrying to reconnect. I mean, like:

<?php
namespace PEAR2\Net\RouterOS;
require_once 'PEAR2_Net_RouterOS-1.0.0b3.phar';

$config = array(...);

while (true) {
    try {
        ...
    } catch (\Exception $e) {
        if ($e instanceof Exception || $e instanceof \PEAR2\Net\Transmitter\Exception) {//Exception from the router connection
            sleep(10);
        } else {//Any other exception (e.g. DB connection)
            die($e);
        }
    }
} 

Please tell me if I don’t want to insert zero values rxrate or txrate in DB, what’s the syntax?

Thank you

I assume you mean when both are 0 (you can’t do the insert with only one non-zero value; you need to supply both or neither).

The simplest way is to replace

            $insertStatement[0]->execute(); 

with

            if ($insertStatement[1]['rx'] > 0 && $insertStatement[1]['tx'] > 0) {
                $insertStatement[0]->execute();
            } 

nice work boen_robot . i have same problem in this topic http://forum.mikrotik.com/t/php-api-follow-1/61870/1
i want to run

/ip/firewall/address-list/print
=follow=1
.tag=adr

each time something adds or removes from specified address-lists the script should do the same in another router . may i ask you to please reply in that topic . thanks

For the sake of anyone who might stumble upon this topic… The code above needs some changes for later versions of the API client.

That, plus a slight performance optimization, can be seen in this topic where it was brought up:
http://forum.mikrotik.com/t/pear2-broken-signature/94032/15