Community discussions

MikroTik App
 
User avatar
inteq
Member Candidate
Member Candidate
Topic Author
Posts: 205
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

3 routers in a row defective possible?

Mon Jun 01, 2020 11:55 am

Hello,

So I have a FTTH location where internet drops completely between 10 seconds to 2 minutes, at random times, for 1-4 times a day.
Nothing in logs. No link downs on interface. Online UPS.
1st router: a RB4011. Because the router crashed couple of times, I thought those random internet dropouts were caused by it. Replaced it.
2nd router: a RB1100AHx4. New patch cable. Random dropouts continued. Bought a new one.
3rd router: a RB1100AHx4. New patch cable. Random dropouts continued.

Of course, ISP states they do not see anything wrong on ONT. Funny thing, after they first stated that there is nothing wrong with the connection and that the ONT has been online for ages, I just unplugged the fiber optic cable for 5 minutes, while keeping the ONT powered. Next time I opened a ticket, again, they stated the same thing. Go figure.

The question is: anyone had such a bad luck to have three defective routers in a row?
 
User avatar
inteq
Member Candidate
Member Candidate
Topic Author
Posts: 205
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: 3 routers in a row defective possible?

Wed Jun 17, 2020 10:42 am

Sorry for bump, but this one is strange.

Installed the 2nd RB1100AHx4 (the one purchased after the 1st RB4011) in another location. Same provider, same FTTH tech.
2 weeks already and absolutely stable. No packet loss at all.

Setup a PC with intel nic in place of 3rd RB1100AHx4, in the problematic location, allowing just incoming icmp.
To simulate some traffic, every 10 minutes I scheduled a speedtest-cli.
2 days already and no packet loss at all, both in and out.
I am monitoring from 2 external locations with a tool that pings every 1 second with 1 second timeout. Not a single ping lost.
With both RB1100AHx4, in 2 days I would've had several hundred lost pings already.

I am really out of ideas on this one.
 
anav
Forum Guru
Forum Guru
Posts: 4606
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada

Re: 3 routers in a row defective possible?

Wed Jun 17, 2020 8:53 pm

I feel your pain.
I have similar issues withe an RB450Gx4 but perhaps more insidious over 2 years but I put it down to internet quirkiness for 2 years (now out of warranty).
Significant packet loss on traceroute to my both my ISPs gateways.
Win MTR shows losses
Traffic flow showed gaps that clearly showed NO tx and NO rcve and yet no log entries by router saying it was down.
I just completed a neinstall and I will post the trace route to the ISP gateway, and winmtr to google.ca results as well as the gap image from earlier testing, which shows one ISP with no Gap and the other with a huge gap.

No change! :-( Originally using 6.46.6 firmware, now uising 6.45.9 firmware - no difference!


https://imgur.com/1y5Gd9s
https://imgur.com/DYPJGH4
https://imgur.com/OHdK0RH
I'd rather manage rats than software. Follow my advice at your own risk! (Sob & mkx forced me to write that!)
 
Sob
Forum Guru
Forum Guru
Posts: 5590
Joined: Mon Apr 20, 2009 9:11 pm

Re: 3 routers in a row defective possible?

Wed Jun 17, 2020 9:34 pm

@anav: Your packet loss is misleading. It can be one long gap, like the one in your graph, yes. But let it run longer and it will be much better. ISP can also rate limit icmp. There's no way you could use connection with continuous 40 or 60% packet loss for years.

But I would tend to blame ISP, rather than router. Of course if it does that with two, it's suspicious. In any case, you already mentioned that you'll have another router, so just for fun, you can reset this one, give it minimal config (just two subnets and nothing else) and test it in your LAN, connect one computer behind it and let it copy something huge between it and device in other subnet (main LAN). And see if there will be similar gaps then.
People who quote full posts should be spanked with ethernet cable. Some exceptions for multi-topic threads may apply. Not intended as incentive for masochists.
 
User avatar
inteq
Member Candidate
Member Candidate
Topic Author
Posts: 205
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: 3 routers in a row defective possible?

Tue Jun 23, 2020 9:59 pm

To recap,
3 routerboards in a pure routing config, no NAT, no DHCP, no STP and even tested without a bridge: random packet loss lasting from couple of seconds to 1 minute.
Did a test with only one interface setup with a public IP, without routing or anything else plugged into it: random packet loss.
PC with intel network card. No routing, just a public IP setup on it. No packet loss at all.
Same PC with an additional intel network card and IPFire installed on it. Same setup as on the 3 routerboards. Routing without NAT, DHCP. No packet loss at all.

Sent one of the "problematic" routerboards to another location, but with NAT and DHCP enabled: no packet loss
A bit of a stretch but not setting up NAT/masquerade and just using routing can cause packet loss on RB1100AHx4?

A minimum config that still presents random packet loss. Only one interface setup. No NAT or anything else.
# jun/18/2020 21:53:08 by RouterOS 6.47
# software id = ASE2-1E2N
#
# model = RB1100x4
# serial number = 91D80A0C859E
/interface ethernet switch port
set 0 default-vlan-id=0
set 1 default-vlan-id=0
set 2 default-vlan-id=0
set 3 default-vlan-id=0
set 4 default-vlan-id=0
set 5 default-vlan-id=0
set 6 default-vlan-id=0
set 7 default-vlan-id=0
set 8 default-vlan-id=0
set 9 default-vlan-id=0
set 10 default-vlan-id=0
set 11 default-vlan-id=0
set 12 default-vlan-id=0
set 13 default-vlan-id=0
set 14 default-vlan-id=0
set 15 default-vlan-id=0
/interface list
add name=WAN
add name=Local
/ip neighbor discovery-settings
set discover-interface-list=Local
/interface list member
add interface=ether1 list=WAN
add interface=ether2 list=Local
/ip address
add address=x.x.206.92/24 interface=ether1 network=x.x.206.0
/ip dns
set servers=1.1.1.1
/ip firewall filter
add action=drop chain=input comment="Drop invalid " connection-state=invalid
add action=accept chain=input comment="Accept established and related" \
    connection-state=established,related
add action=accept chain=input comment="Accept icmp" protocol=icmp
add action=drop chain=input comment="Drop everything else" in-interface-list=\
    !Local
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
set udplite disabled=yes
set dccp disabled=yes
set sctp disabled=yes
/ip route
add distance=1 gateway=x.x.206.1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=Europe/Bucharest
/system logging
set 3 action=disk
/system routerboard settings
set auto-upgrade=yes
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=Local

Dude outages:
outages.png
You do not have the required permissions to view the files attached to this post.
 
hostclub
newbie
Posts: 27
Joined: Tue May 12, 2020 2:23 am

Re: 3 routers in a row defective possible?

Tue Jun 23, 2020 11:55 pm

Sa mai zica careva ca Mikrotik nu e de cacao la faza asta....

I see that Mikrotik ARM CPU routers have issues, more than Tilera

Taking into account that the next generation of Mikrotik routers is ARM only.....I expect dark age for Mikrotik.
 
Sob
Forum Guru
Forum Guru
Posts: 5590
Joined: Mon Apr 20, 2009 9:11 pm

Re: 3 routers in a row defective possible?

Wed Jun 24, 2020 2:06 am

You can try to disable the 'drop invalid' rule. It shouldn't be it, but since there's nothing else in your config...
People who quote full posts should be spanked with ethernet cable. Some exceptions for multi-topic threads may apply. Not intended as incentive for masochists.
 
sindy
Forum Guru
Forum Guru
Posts: 5343
Joined: Mon Dec 04, 2017 9:19 pm

Re: 3 routers in a row defective possible?

Wed Jun 24, 2020 9:32 am

What does /interface ethernet print stats show on the problematic site? What happens if you place a stupid switch between the ISP gear and the 1100's WAN there (the stats only show Rx errors on the local interface, but cannot show Rx errors at the other end of the cable)? It sounds to me like an interface compatibility issue.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
inteq
Member Candidate
Member Candidate
Topic Author
Posts: 205
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: 3 routers in a row defective possible?

Wed Jun 24, 2020 12:16 pm

@sob I have tried without the Drop invalid rule when @tippenring suggested here viewtopic.php?f=2&t=162627&p=801420#p801383. No change.
@sindy Will have to reinstall the RB1100AHx4 to be able to check /interface ethernet print stats but if I recall there were some RX errors logged seen under Winbox interface stats screen. Will test with a dumb switch and get back with info.

Another thing I wanted to mention is that I tested multiple ports, covering all 3 switches on the RB, to rule out a defective port as much as possible. Tested on ether1, 6 and 11.
Tried also lowering the CPU frequency from 1400 to 1200. No change.

Thank you all for replies.
 
nje431
Frequent Visitor
Frequent Visitor
Posts: 53
Joined: Tue Sep 10, 2013 5:17 pm

Re: 3 routers in a row defective possible?

Wed Jun 24, 2020 3:00 pm

I know this is a shot in the dark, but try disabling 1000/half under auto negotiate. The switch chip doesn't support that option, and if your ISP connection does...... I've disabled it in all ours, and they seem to be a lot more stable since then. Or maybe it's just my imagination.
 
sindy
Forum Guru
Forum Guru
Posts: 5343
Joined: Mon Dec 04, 2017 9:19 pm

Re: 3 routers in a row defective possible?

Wed Jun 24, 2020 4:41 pm

Will have to reinstall the RB1100AHx4 to be able to check /interface ethernet print stats but if I recall there were some RX errors logged seen under Winbox interface stats screen. Will test with a dumb switch and get back with info.
I don't bother whether you use Winbox, WebFig or command line to check the Ethernet port statistics. But as long as you can see any Rx errors on Ethernet, it makes little sense to search for packet loss anywhere else. An Rx error means that the packet has been dropped as it arrived from the wire, big fat dot. There is no auto-correction on Ethernet - a packet with wrong CRC is simply not let through.

Another thing I wanted to mention is that I tested multiple ports, covering all 3 switches on the RB, to rule out a defective port as much as possible. Tested on ether1, 6 and 11.
I did not have in mind a manufacturing issue on a single port (or switch chip) of the 1100. What I had in mind was a design issue - an electrical incompatibility between the 1100's ports and the port of the ISP's device or, no matter how odd it may seem, a mechanical incompatibility between the connectors on the patchcord and on one of the devices. I have got some factory-made patchcords (i.e. not DIY crimped) which do not work in my hAP ac lite but do in my PC. Other patchcords from the same purchase, differing only by cable color, work in both.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
User avatar
inteq
Member Candidate
Member Candidate
Topic Author
Posts: 205
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: 3 routers in a row defective possible?

Thu Jun 25, 2020 4:43 am

Reinstalled the 3rd RB1100AHx4 tonight and did not have to wait longer than 2 hours for the first interruption. This time it lasted 2 minutes as reported by Dude.
So far used 4 different factory crimped ethernet cables and one cat 6 made by me.
/interface ethernet print stats taken after the 1st incident shows:
stats.png
Stats after 11 hours uptime:
stats11.png
You do not have the required permissions to view the files attached to this post.
 
quackyo
Member Candidate
Member Candidate
Posts: 126
Joined: Mon Nov 16, 2015 10:14 am

Re: 3 routers in a row defective possible?

Fri Jun 26, 2020 11:45 am

It seems quite probable that your ISP is having problems.
But just as a try. Disable dude monitoring of your router.
I have had some troubles with a RB1100AHx4 that seemed to be caused by monitoring by Dude.
 
sindy
Forum Guru
Forum Guru
Posts: 5343
Joined: Mon Dec 04, 2017 9:19 pm

Re: 3 routers in a row defective possible?

Fri Jun 26, 2020 1:42 pm

So actually no Rx errors at the 1100 side at all. Unfortunately (or fortunately?), this still does not mean that there is no L1 problem, so if you can put a dumb switch between the 1100 and the ISP's port, do that. If it is still bad, keep the same dumb switch in place and connect, instead of the 1100, something that you have tested before to be working, to make sure that the dumb switch itself doesn't have an issue with the ISP gear.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.

Who is online

Users browsing this forum: inteq, jvanhambelgium and 71 guests