Community discussions

 
Buckeye
just joined
Topic Author
Posts: 13
Joined: Tue Sep 11, 2018 2:03 am

CSS106-5G-1S v2.8 bogus TX and RX unicast counter with bit 32 set.

Tue Sep 11, 2018 5:58 am

Edit: This issue has been reported to support@mikrotik.com Auto assigned Ticket#2018091222007256 In the email I sent the subject was:
"CSS106-5G-1S SwOS v2.8 TX and RX unicast counters not correct. tuph and ruph appear to be too high."

Edit 2: From MikroTik on Sept 17: "I was able to reproduce the same results. We will look into this matter and try to fix this in upcoming releases, but I cannot say any ETA yet."
So it is on their plate of things to fix. That's all we can hope for. I don't consider this to be a critical issue, but worth reporting so they can know it isn't working as intended.

Just noticed this playing around with my new switch. I have a Raspberry Pi 3 connected to port 5 and an Edgerouter-X connected to port 1. The counters claim that over 4 billion unicast have been received on port 1 and transmitted on port 5. It is obviously incorrect, the total packets count is correct. Interpreting as a 64bit binary, bit 32 is set.
Bogus Unicast Tx and Rx counters on CSS106-5G-1S SwOS 2.8.PNG
This looks like a bug to me.

I just looked at a downloaded Atheros AR8327 manual (Google AR8327 manual). In section "2.12 MIB/Statistics Counters" on pages 43-45 it has a description of the 40 MIB counters that the chip has. Most of these are 32 bit counters, but the chip has the ability to copy and zero these counters periodically, so if implemented correctly, 64 bit counters can be done in software by a CPU.

What was surprising to me is that I didn't see any counter for unicast packets in the MIB list. It doesn't even appear that there are total packet counters, so the totals must have to be derived from the different size packet counters. So perhaps what SwOS is displaying as Unicast packets is a "derived" value based on the derived total minus all other frame types (broadcast, multicast, pause).

There may be a race condition that is being misinterpreted as an overflow, and the high 32bit word of the 64bit software count accumulator may being incremented when it should not.

I just got this switch which I received with 2.0p which I manually updated to 2.8. So I don't know if this is new or old "issue".
You do not have the required permissions to view the files attached to this post.
Last edited by Buckeye on Thu Sep 20, 2018 4:29 am, edited 2 times in total.
 
eider
newbie
Posts: 30
Joined: Thu Nov 30, 2017 10:14 pm

Re: CSS106-5G-1S v2.8 bogus TX and RX unicast counter with bit 32 set.

Tue Sep 11, 2018 8:16 pm

That's a nice find! It looks like SwOS is calculating and counting packets correctly, however JavaScript code was not updated to reflect recent changed of counters to 64-bit, hence the issue. In actual response from /stats.b you'll find proper values in 'rup' (ordered by port #, from 0 to 5). In my case it's something like this:
  rup: [
    0x0009677f,
    0x005886a8,
    0x0010e2d1,
    0x00c766c1,
    0x000038d7,
    0x00000000
  ],
which translates to 616319 unicast packets, however page shows 120259700607 which is 1c0009677f. Raught calculation of other types and comparison with total shows that raw received value is actually correct.
 
Buckeye
just joined
Topic Author
Posts: 13
Joined: Tue Sep 11, 2018 2:03 am

Re: CSS106-5G-1S v2.8 bogus TX and RX unicast counter with bit 32 set.

Tue Sep 11, 2018 8:55 pm

After running overnight while logged into the switch with web statistics page being displayed. The only significant traffic was the CSS106-5G-1S sending web updates.

Both of these unicast counters increased by unreasonable amounts.

Perhaps just the fact that the web page was being displayed has something to do with the counters being incorrect.

Since I am new to the forum, what is the official way to submit a bug report?
Bogus Unicast Tx and Rx counters on CSS106-5G-1S SwOS 2.8 after running overnight.PNG
@eider You obviously know a lot more about MikroTik than I do. Where do you enter the /stats.b and /sys.b ?

As far as the response about javascript not being updated correctly, can you explain why snmpwalk is also displaying incorrect 64 bit counters? I assume that the javascript code would be running in the browser (I am using Chrome Version 68.0.3440.106 (Official Build) (64-bit) on Win 7 pro). Or is javascript running on the Taifatech TF470 SoC in the CSS106?

iso.3.6.1.2.1.31.1.1.1.6.1 = Counter64: 57085300
iso.3.6.1.2.1.31.1.1.1.6.2 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.6.3 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.6.4 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.6.5 = Counter64: 566283
iso.3.6.1.2.1.31.1.1.1.6.6 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.7.1 = Counter64: 30065177702
iso.3.6.1.2.1.31.1.1.1.7.2 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.7.3 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.7.4 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.7.5 = Counter64: 4392
iso.3.6.1.2.1.31.1.1.1.7.6 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.8.1 = Counter64: 4139
iso.3.6.1.2.1.31.1.1.1.8.2 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.8.3 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.8.4 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.8.5 = Counter64: 35
iso.3.6.1.2.1.31.1.1.1.8.6 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.9.1 = Counter64: 11557
iso.3.6.1.2.1.31.1.1.1.9.2 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.9.3 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.9.4 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.9.5 = Counter64: 11
iso.3.6.1.2.1.31.1.1.1.9.6 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.10.1 = Counter64: 211479872
iso.3.6.1.2.1.31.1.1.1.10.2 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.10.3 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.10.4 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.10.5 = Counter64: 5720247
iso.3.6.1.2.1.31.1.1.1.10.6 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.11.1 = Counter64: 430854
iso.3.6.1.2.1.31.1.1.1.11.2 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.11.3 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.11.4 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.11.5 = Counter64: 21474839469
iso.3.6.1.2.1.31.1.1.1.11.6 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.12.1 = Counter64: 35251
iso.3.6.1.2.1.31.1.1.1.12.2 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.12.3 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.12.4 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.12.5 = Counter64: 39359
iso.3.6.1.2.1.31.1.1.1.12.6 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.13.1 = Counter64: 1188
iso.3.6.1.2.1.31.1.1.1.13.2 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.13.3 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.13.4 = Counter64: 0
iso.3.6.1.2.1.31.1.1.1.13.5 = Counter64: 12733
iso.3.6.1.2.1.31.1.1.1.13.6 = Counter64: 0
You do not have the required permissions to view the files attached to this post.
Last edited by Buckeye on Wed Sep 12, 2018 9:22 pm, edited 1 time in total.
 
eider
newbie
Posts: 30
Joined: Thu Nov 30, 2017 10:14 pm

Re: CSS106-5G-1S v2.8 bogus TX and RX unicast counter with bit 32 set.

Tue Sep 11, 2018 10:06 pm

Perhaps just the fact that the web page was being displayed has something to do with the counters being incorrect.
No, but web page constantly updates its stats so packets are being generated.

Where do you enter the /stats.b and /sys.b ?
These are just URI. You'll need to navigate to "http://192.168.88.1/!stats.b" (assuming default IP) - this will print raw counter data that's being interpreted by JavaScript in browser to display decimal. In fact, SwOS on this device is a very small integrated OS, no more than 64kB, so to save space, only a small HTML stub is being held in flash along with JavaScript file that renders whole UI.

can you explain why snmpwalk is also displaying incorrect 64 bit counters
That's actually something i thought about after posting previous reply - it's possible that this switch does not conform to 64-bit counters (due to hardware limitations or simply a bug in 2.8) and this causes rest of code that expects 64-bit value to go crazy while interpreting 32-bit value. I really don't know, I don't feel like sitting over that code, it's minified in order to save space and that makes it pretty hard to read, even after prettifying it. Either way, there is an issue with converting register value to decimal for printing.

I've linked this thread on viewtopic.php?f=21&p=685562#p685562 to make sure someone from Mikrotik sees this, as this is obviously a bug.
 
Buckeye
just joined
Topic Author
Posts: 13
Joined: Tue Sep 11, 2018 2:03 am

Re: CSS106-5G-1S v2.8 bogus TX and RX unicast counter with bit 32 set.

Wed Sep 12, 2018 5:24 am

@eider Thanks for the info about getting the raw values.

I generated around 8GB of traffic from the the Raspberry Pi 3 on port 5 to the Edgerouter-X on port 1 with iperf3.

I think the bug is in the code in the embedded processor in the CSS106. It appears there are multiple 64 bit values being sent, but the upper "high" 32 bits is being sent with tags that end with "h".

e.g. rb: is the low 32 bits of received bytes, rbh: is the high 32 bits.

rup: low 32 bits of received unicast packets, ruph: high 32 bits of received unicast packets.
{rbp:[0x0000425d,0x00000000,0x00000000,0x00000000,0x0000000d,0x00000000],rpp:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],rmp:[0x000017ca,0x00000000,0x00000000,0x00000000,0x00000023,0x00000000],rfcs:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],rae:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],rr:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],fr:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],r64:[0x0005d272,0x00000000,0x00000000,0x00000000,0x00000252,0x00000000],r65:[0x002b260b,0x00000000,0x00000000,0x00000000,0x000025a6,0x00000000],r128:[0x000028a8,0x00000000,0x00000000,0x00000000,0x00000ac0,0x00000000],r256:[0x000000be,0x00000000,0x00000000,0x00000000,0x0000008a,0x00000000],r512:[0x000104b2,0x00000000,0x00000000,0x00000000,0x00000008,0x00000000],r1k:[0x00001f23,0x00000000,0x00000000,0x00000000,0x0054544f,0x00000000],rmax:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],rtl:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],rb:[0x10633607,0x00000000,0x00000000,0x00000000,0xf421706a,0x00000000],rbh:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000001,0x00000000],rov:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],tbp:[0x000006cf,0x00000000,0x00000000,0x00000000,0x0000491d,0x00000000],tpp:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],tmp:[0x0000ca4f,0x00000000,0x00000000,0x00000000,0x0000e1f9,0x00000000],tur:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],t64:[0x0003d484,0x00000000,0x00000000,0x00000000,0x0000fc96,0x00000000],t65:[0x0000261b,0x00000000,0x00000000,0x00000000,0x002a37dd,0x00000000],t128:[0x000011de,0x00000000,0x00000000,0x00000000,0x000028a8,0x00000000],t256:[0x00011522,0x00000000,0x00000000,0x00000000,0x000000af,0x00000000],t512:[0x00041e62,0x00000000,0x00000000,0x00000000,0x000009ce,0x00000000],t1k:[0x005454b2,0x00000000,0x00000000,0x00000000,0x00001f23,0x00000000],tmax:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],ttl:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],tb:[0x052aef79,0x00000000,0x00000000,0x00000000,0x0cc122a3,0x00000000],tbh:[0x00000002,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],tcl:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],tec:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],tmc:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],tsc:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],ted:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],tdf:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],tlc:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],rtp:[0x003245b8,0x00000000,0x00000000,0x00000000,0x00548799,0x00000000],rup:[0x0031eb91,0x00000000,0x00000000,0x00000000,0x00548769,0x00000000],rte:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],ttp:[0x005d94b3,0x00000000,0x00000000,0x00000000,0x002b86bb,0x00000000],tup:[0x005cc395,0x00000000,0x00000000,0x00000000,0x002a5ba5,0x00000000],tte:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],rrb:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],trb:[0x00000040,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],rrp:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],trp:[0x00000001,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],ruph:[0x0000000b,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],rmph:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],rbph:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],tuph:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000009,0x00000000],tmph:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000],tbph:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000]}
It appears to me that there are values being sent in the ruph: and tuph: arrays that are not correct. They are too high.

ruph:[0x0000000b,0x00000000,0x00000000,0x00000000,0x00000000,0x00000000]
tuph:[0x00000000,0x00000000,0x00000000,0x00000000,0x00000009,0x00000000]

In this case both should still be zero.
Last edited by Buckeye on Wed Sep 12, 2018 9:11 pm, edited 1 time in total.
 
eider
newbie
Posts: 30
Joined: Thu Nov 30, 2017 10:14 pm

Re: CSS106-5G-1S v2.8 bogus TX and RX unicast counter with bit 32 set.

Wed Sep 12, 2018 9:11 am

Ah, I see. You are indeed right. didn't notice h-variant of these values are not zero. Yes, it looks like there's some issue with CPU counter improperly increasing upper bits. Would be interesting to catch a moment when it's being increased.
 
Buckeye
just joined
Topic Author
Posts: 13
Joined: Tue Sep 11, 2018 2:03 am

Re: CSS106-5G-1S v2.8 bogus TX and RX unicast counter with bit 32 set.

Wed Sep 12, 2018 8:42 pm

I have sent a message to support@mikrotik.com just now. I will update this with ticket number when I get one.

BTW, the only configuration change I made was to change the password. Everything else is "factory". The device is getting ip address from DHCP via edgerouter-x (SwOS v2.8 default action, DHCP with fallback)
 
Buckeye
just joined
Topic Author
Posts: 13
Joined: Tue Sep 11, 2018 2:03 am

Re: CSS106-5G-1S v2.8 bogus TX and RX unicast counter with bit 32 set.

Thu Sep 13, 2018 10:01 pm

This issue has been reported to support@mikrotik.com Auto assigned Ticket#2018091222007256

Also added to first post in thread to make it easy to find.

Who is online

Users browsing this forum: No registered users and 3 guests