Hello,
So I have a FTTH location where internet drops completely between 10 seconds to 2 minutes, at random times, for 1-4 times a day.
Nothing in logs. No link downs on interface. Online UPS.
1st router: a RB4011. Because the router crashed couple of times, I thought those random internet dropouts were caused by it. Replaced it.
2nd router: a RB1100AHx4. New patch cable. Random dropouts continued. Bought a new one.
3rd router: a RB1100AHx4. New patch cable. Random dropouts continued.
Of course, ISP states they do not see anything wrong on ONT. Funny thing, after they first stated that there is nothing wrong with the connection and that the ONT has been online for ages, I just unplugged the fiber optic cable for 5 minutes, while keeping the ONT powered. Next time I opened a ticket, again, they stated the same thing. Go figure.
The question is: anyone had such a bad luck to have three defective routers in a row?
Re: 3 routers in a row defective possible?
Sorry for bump, but this one is strange.
Installed the 2nd RB1100AHx4 (the one purchased after the 1st RB4011) in another location. Same provider, same FTTH tech.
2 weeks already and absolutely stable. No packet loss at all.
Setup a PC with intel nic in place of 3rd RB1100AHx4, in the problematic location, allowing just incoming icmp.
To simulate some traffic, every 10 minutes I scheduled a speedtest-cli.
2 days already and no packet loss at all, both in and out.
I am monitoring from 2 external locations with a tool that pings every 1 second with 1 second timeout. Not a single ping lost.
With both RB1100AHx4, in 2 days I would've had several hundred lost pings already.
I am really out of ideas on this one.
Installed the 2nd RB1100AHx4 (the one purchased after the 1st RB4011) in another location. Same provider, same FTTH tech.
2 weeks already and absolutely stable. No packet loss at all.
Setup a PC with intel nic in place of 3rd RB1100AHx4, in the problematic location, allowing just incoming icmp.
To simulate some traffic, every 10 minutes I scheduled a speedtest-cli.
2 days already and no packet loss at all, both in and out.
I am monitoring from 2 external locations with a tool that pings every 1 second with 1 second timeout. Not a single ping lost.
With both RB1100AHx4, in 2 days I would've had several hundred lost pings already.
I am really out of ideas on this one.
Re: 3 routers in a row defective possible?
I feel your pain.
I have similar issues withe an RB450Gx4 but perhaps more insidious over 2 years but I put it down to internet quirkiness for 2 years (now out of warranty).
Significant packet loss on traceroute to my both my ISPs gateways.
Win MTR shows losses
Traffic flow showed gaps that clearly showed NO tx and NO rcve and yet no log entries by router saying it was down.
I just completed a neinstall and I will post the trace route to the ISP gateway, and winmtr to google.ca results as well as the gap image from earlier testing, which shows one ISP with no Gap and the other with a huge gap.
No change!
Originally using 6.46.6 firmware, now uising 6.45.9 firmware - no difference!
https://imgur.com/1y5Gd9s
https://imgur.com/DYPJGH4
https://imgur.com/OHdK0RH
I have similar issues withe an RB450Gx4 but perhaps more insidious over 2 years but I put it down to internet quirkiness for 2 years (now out of warranty).
Significant packet loss on traceroute to my both my ISPs gateways.
Win MTR shows losses
Traffic flow showed gaps that clearly showed NO tx and NO rcve and yet no log entries by router saying it was down.
I just completed a neinstall and I will post the trace route to the ISP gateway, and winmtr to google.ca results as well as the gap image from earlier testing, which shows one ISP with no Gap and the other with a huge gap.
No change!
Originally using 6.46.6 firmware, now uising 6.45.9 firmware - no difference!https://imgur.com/1y5Gd9s
https://imgur.com/DYPJGH4
https://imgur.com/OHdK0RH
I'd rather manage rats than software. Follow my advice at your own risk! (Sob & mkx forced me to write that!)
MTUNA Certified, by the Ascerbic Llama!
MTUNA Certified, by the Ascerbic Llama!
Re: 3 routers in a row defective possible?
@anav: Your packet loss is misleading. It can be one long gap, like the one in your graph, yes. But let it run longer and it will be much better. ISP can also rate limit icmp. There's no way you could use connection with continuous 40 or 60% packet loss for years.
But I would tend to blame ISP, rather than router. Of course if it does that with two, it's suspicious. In any case, you already mentioned that you'll have another router, so just for fun, you can reset this one, give it minimal config (just two subnets and nothing else) and test it in your LAN, connect one computer behind it and let it copy something huge between it and device in other subnet (main LAN). And see if there will be similar gaps then.
But I would tend to blame ISP, rather than router. Of course if it does that with two, it's suspicious. In any case, you already mentioned that you'll have another router, so just for fun, you can reset this one, give it minimal config (just two subnets and nothing else) and test it in your LAN, connect one computer behind it and let it copy something huge between it and device in other subnet (main LAN). And see if there will be similar gaps then.
Excessive quoting is useless and annoying. If you use it, please consider if you could do without it.
Re: 3 routers in a row defective possible?
To recap,
3 routerboards in a pure routing config, no NAT, no DHCP, no STP and even tested without a bridge: random packet loss lasting from couple of seconds to 1 minute.
Did a test with only one interface setup with a public IP, without routing or anything else plugged into it: random packet loss.
PC with intel network card. No routing, just a public IP setup on it. No packet loss at all.
Same PC with an additional intel network card and IPFire installed on it. Same setup as on the 3 routerboards. Routing without NAT, DHCP. No packet loss at all.
Sent one of the "problematic" routerboards to another location, but with NAT and DHCP enabled: no packet loss
A bit of a stretch but not setting up NAT/masquerade and just using routing can cause packet loss on RB1100AHx4?
A minimum config that still presents random packet loss. Only one interface setup. No NAT or anything else.
Dude outages:
3 routerboards in a pure routing config, no NAT, no DHCP, no STP and even tested without a bridge: random packet loss lasting from couple of seconds to 1 minute.
Did a test with only one interface setup with a public IP, without routing or anything else plugged into it: random packet loss.
PC with intel network card. No routing, just a public IP setup on it. No packet loss at all.
Same PC with an additional intel network card and IPFire installed on it. Same setup as on the 3 routerboards. Routing without NAT, DHCP. No packet loss at all.
Sent one of the "problematic" routerboards to another location, but with NAT and DHCP enabled: no packet loss
A bit of a stretch but not setting up NAT/masquerade and just using routing can cause packet loss on RB1100AHx4?
A minimum config that still presents random packet loss. Only one interface setup. No NAT or anything else.
Code: Select all
# jun/18/2020 21:53:08 by RouterOS 6.47
# software id = ASE2-1E2N
#
# model = RB1100x4
# serial number = 91D80A0C859E
/interface ethernet switch port
set 0 default-vlan-id=0
set 1 default-vlan-id=0
set 2 default-vlan-id=0
set 3 default-vlan-id=0
set 4 default-vlan-id=0
set 5 default-vlan-id=0
set 6 default-vlan-id=0
set 7 default-vlan-id=0
set 8 default-vlan-id=0
set 9 default-vlan-id=0
set 10 default-vlan-id=0
set 11 default-vlan-id=0
set 12 default-vlan-id=0
set 13 default-vlan-id=0
set 14 default-vlan-id=0
set 15 default-vlan-id=0
/interface list
add name=WAN
add name=Local
/ip neighbor discovery-settings
set discover-interface-list=Local
/interface list member
add interface=ether1 list=WAN
add interface=ether2 list=Local
/ip address
add address=x.x.206.92/24 interface=ether1 network=x.x.206.0
/ip dns
set servers=1.1.1.1
/ip firewall filter
add action=drop chain=input comment="Drop invalid " connection-state=invalid
add action=accept chain=input comment="Accept established and related" \
connection-state=established,related
add action=accept chain=input comment="Accept icmp" protocol=icmp
add action=drop chain=input comment="Drop everything else" in-interface-list=\
!Local
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
set udplite disabled=yes
set dccp disabled=yes
set sctp disabled=yes
/ip route
add distance=1 gateway=x.x.206.1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=Europe/Bucharest
/system logging
set 3 action=disk
/system routerboard settings
set auto-upgrade=yes
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=LocalDude outages:
You do not have the required permissions to view the files attached to this post.
Re: 3 routers in a row defective possible?
Sa mai zica careva ca Mikrotik nu e de cacao la faza asta....
I see that Mikrotik ARM CPU routers have issues, more than Tilera
Taking into account that the next generation of Mikrotik routers is ARM only.....I expect dark age for Mikrotik.
I see that Mikrotik ARM CPU routers have issues, more than Tilera
Taking into account that the next generation of Mikrotik routers is ARM only.....I expect dark age for Mikrotik.
Re: 3 routers in a row defective possible?
You can try to disable the 'drop invalid' rule. It shouldn't be it, but since there's nothing else in your config...
Excessive quoting is useless and annoying. If you use it, please consider if you could do without it.
Re: 3 routers in a row defective possible?
What does /interface ethernet print stats show on the problematic site? What happens if you place a stupid switch between the ISP gear and the 1100's WAN there (the stats only show Rx errors on the local interface, but cannot show Rx errors at the other end of the cable)? It sounds to me like an interface compatibility issue.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
Re: 3 routers in a row defective possible?
@sob I have tried without the Drop invalid rule when @tippenring suggested here viewtopic.php?f=2&t=162627&p=801420#p801383. No change.
@sindy Will have to reinstall the RB1100AHx4 to be able to check /interface ethernet print stats but if I recall there were some RX errors logged seen under Winbox interface stats screen. Will test with a dumb switch and get back with info.
Another thing I wanted to mention is that I tested multiple ports, covering all 3 switches on the RB, to rule out a defective port as much as possible. Tested on ether1, 6 and 11.
Tried also lowering the CPU frequency from 1400 to 1200. No change.
Thank you all for replies.
@sindy Will have to reinstall the RB1100AHx4 to be able to check /interface ethernet print stats but if I recall there were some RX errors logged seen under Winbox interface stats screen. Will test with a dumb switch and get back with info.
Another thing I wanted to mention is that I tested multiple ports, covering all 3 switches on the RB, to rule out a defective port as much as possible. Tested on ether1, 6 and 11.
Tried also lowering the CPU frequency from 1400 to 1200. No change.
Thank you all for replies.
Re: 3 routers in a row defective possible?
I know this is a shot in the dark, but try disabling 1000/half under auto negotiate. The switch chip doesn't support that option, and if your ISP connection does...... I've disabled it in all ours, and they seem to be a lot more stable since then. Or maybe it's just my imagination.
Re: 3 routers in a row defective possible?
I don't bother whether you use Winbox, WebFig or command line to check the Ethernet port statistics. But as long as you can see any Rx errors on Ethernet, it makes little sense to search for packet loss anywhere else. An Rx error means that the packet has been dropped as it arrived from the wire, big fat dot. There is no auto-correction on Ethernet - a packet with wrong CRC is simply not let through.Will have to reinstall the RB1100AHx4 to be able to check /interface ethernet print stats but if I recall there were some RX errors logged seen under Winbox interface stats screen. Will test with a dumb switch and get back with info.
I did not have in mind a manufacturing issue on a single port (or switch chip) of the 1100. What I had in mind was a design issue - an electrical incompatibility between the 1100's ports and the port of the ISP's device or, no matter how odd it may seem, a mechanical incompatibility between the connectors on the patchcord and on one of the devices. I have got some factory-made patchcords (i.e. not DIY crimped) which do not work in my hAP ac lite but do in my PC. Other patchcords from the same purchase, differing only by cable color, work in both.Another thing I wanted to mention is that I tested multiple ports, covering all 3 switches on the RB, to rule out a defective port as much as possible. Tested on ether1, 6 and 11.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
Re: 3 routers in a row defective possible?
Reinstalled the 3rd RB1100AHx4 tonight and did not have to wait longer than 2 hours for the first interruption. This time it lasted 2 minutes as reported by Dude.
So far used 4 different factory crimped ethernet cables and one cat 6 made by me.
/interface ethernet print stats taken after the 1st incident shows: Stats after 11 hours uptime:
So far used 4 different factory crimped ethernet cables and one cat 6 made by me.
/interface ethernet print stats taken after the 1st incident shows: Stats after 11 hours uptime:
You do not have the required permissions to view the files attached to this post.
Re: 3 routers in a row defective possible?
It seems quite probable that your ISP is having problems.
But just as a try. Disable dude monitoring of your router.
I have had some troubles with a RB1100AHx4 that seemed to be caused by monitoring by Dude.
But just as a try. Disable dude monitoring of your router.
I have had some troubles with a RB1100AHx4 that seemed to be caused by monitoring by Dude.
Re: 3 routers in a row defective possible?
So actually no Rx errors at the 1100 side at all. Unfortunately (or fortunately?), this still does not mean that there is no L1 problem, so if you can put a dumb switch between the 1100 and the ISP's port, do that. If it is still bad, keep the same dumb switch in place and connect, instead of the 1100, something that you have tested before to be working, to make sure that the dumb switch itself doesn't have an issue with the ISP gear.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
Re: 3 routers in a row defective possible?
This is getting a bit ridiculous.
I setup a 4th Mikrotik in the location, a RB1100AHx4 Dude Edition. Problem still persists.
I bought a P-Unlimited CHR license and setup RouterOS inside ESXi on a Supermicro server equipped with Intel X722 nics. Problem still persists.
Setup a Netwatch to call a traceroute script when ping fails on 3 targets and write a log file for each incident.
Generates a filename: traceroute 1.1.1.1 from ros-esxi on 13:23:33 - 2020jul15.txt (could not figure out how to format the date so I have to download the file via ssh because of those : in time)
When the problem occurs, the content of the file is kinda empty, which is a bit weird: I am starting to think the place is haunted.
I setup a 4th Mikrotik in the location, a RB1100AHx4 Dude Edition. Problem still persists.
I bought a P-Unlimited CHR license and setup RouterOS inside ESXi on a Supermicro server equipped with Intel X722 nics. Problem still persists.
Setup a Netwatch to call a traceroute script when ping fails on 3 targets and write a log file for each incident.
Code: Select all
:execute {/tool traceroute 1.1.1.1 duration=5} file=("traceroute 1.1.1.1 from ".[/system identity get name] . " on " . [:pick [/system clock get time] 0 8] . " - " . [:pick [/system clock get date] 7 11] . [:pick [/system clock get date] 0 3] . [:pick [/system clock get date] 4 6])When the problem occurs, the content of the file is kinda empty, which is a bit weird: I am starting to think the place is haunted.
You do not have the required permissions to view the files attached to this post.
Re: 3 routers in a row defective possible?
No wonder if you took the task to connect the Poenari ruin to the internetI am starting to think the place is haunted.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
Re: 3 routers in a row defective possible?
Setup a Netwatch to call a traceroute script
Is this run on your border gateway? The first hop printed by traceroute is already router beyond the device running traceroute. If it's your border gateway running it, this means your link is failing. With that many different devices showing same symptoms it might actually be the line failing ... or ISP's gear. This is the reason I like to keep some ISP's gear at my place ... so that the line between ISP gear and my border gateway is well defined and easily replaceable (patch cord).
BR,
Metod
Metod
Re: 3 routers in a row defective possible?
No BGP.
Just ISP > ONT > Mikrotik.
Direct connected IP on one ether and a subnet on another ether.
The netwatch and script is run on Mikrotik
Simple diagram
Just ISP > ONT > Mikrotik.
Direct connected IP on one ether and a subnet on another ether.
The netwatch and script is run on Mikrotik
Simple diagram
You do not have the required permissions to view the files attached to this post.
Re: 3 routers in a row defective possible?
I mentioned border gateway (without P) ... in complex setups there might be multiple routers, but usually only one is border gatewa / router whic is demarcation point between your LAN and the rest of internet.
Anyhow, I'd say the problem is ONT or fibre between ONT and ISP ... specially so as you made a service break (by disconnecting optic connection) in controlled conditions and yet ISP claims everything was just fine ...
Anyhow, I'd say the problem is ONT or fibre between ONT and ISP ... specially so as you made a service break (by disconnecting optic connection) in controlled conditions and yet ISP claims everything was just fine ...
BR,
Metod
Metod
Re: 3 routers in a row defective possible?
The only argument against this assumption is that the test with nothing-to-do-with-RouterOS device as described in post #2 has shown no issues for two days, but I deem that adrgument quite strong.Anyhow, I'd say the problem is ONT or fibre between ONT and ISP ... specially so as you made a service break (by disconnecting optic connection) in controlled conditions and yet ISP claims everything was just fine ...
@inteq, what happens if you just ping the gateway with arp-ping=yes and interval=100ms instead of using traceroute?
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
Re: 3 routers in a row defective possible?
Will setup with arp-ping=yes and interval=100ms and get back with details.
Thank you.
Code: Select all
ping gatewayIP arp-ping=yes interface=ether1 interval=100msRe: 3 routers in a row defective possible? [SOLVED]
Quick update
I disabled Dude notification for some time and just stopped thinking about it too much.
Beginning of December I decided to just disable The Dude because in my mind, the program had a lot of false positives. I just became numb to connection issues reported by The Dude.
Lo and behold, 2 weeks now already and the problem described in the 1st post of this topic went away. Confirmed from multiple external sources.
Somehow, it seems The Dude is causing massive/total packet loss for short duration.
Cannot really explain this and cannot really figure out how a monitoring program can cause such problems, but in my case it seems it did.
No more Dude for me.
I disabled Dude notification for some time and just stopped thinking about it too much.
Beginning of December I decided to just disable The Dude because in my mind, the program had a lot of false positives. I just became numb to connection issues reported by The Dude.
Lo and behold, 2 weeks now already and the problem described in the 1st post of this topic went away. Confirmed from multiple external sources.
Somehow, it seems The Dude is causing massive/total packet loss for short duration.
Cannot really explain this and cannot really figure out how a monitoring program can cause such problems, but in my case it seems it did.
No more Dude for me.
Re: 3 routers in a row defective possible?
.Quick update
I disabled Dude notification for some time and just stopped thinking about it too much.
Beginning of December I decided to just disable The Dude because in my mind, the program had a lot of false positives. I just became numb to connection issues reported by The Dude.
Lo and behold, 2 weeks now already and the problem described in the 1st post of this topic went away. Confirmed from multiple external sources.
Somehow, it seems The Dude is causing massive/total packet loss for short duration.
Cannot really explain this and cannot really figure out how a monitoring program can cause such problems, but in my case it seems it did.
No more Dude for me.
I had troublesome monthly reboots on a RB1100AHx4de. Nothing was reported in the log except router rebooted without proper shutdown. I tried lots of changes but because it took so long it was difficult to make progress. In the end I disabled the DUDE and it didn't reboot in over 2 months. I changed to a brand new RB1100AHx4de with factory firmware 6.45.8 and manually recreated the config except I imported the DUDE database. I have currently 72days uptime with the new RB1100AHx4de and the DUDE on v6.45.9.
Does anyone think there could be code in the Dude that triggers a hardware fault in susceptible ARM routers?
.
Re: 3 routers in a row defective possible?
Enabled The Dude again for one week. Packet loss and problems started.
Disable again.No more packet loss.
I do not think it is related to ARM, because after trying 4 ARM Mikrotik routers I moved this particular location to a CHR unlimited and the same problem continues.
Funny thing is that RouterOS loses connectivity to a subnet behind it also with The Dude active, so I can rule out ISP problems.
Too bad it is not maintained and developed anymore. Nice piece of software not considering this problem.
Disable again.No more packet loss.
I do not think it is related to ARM, because after trying 4 ARM Mikrotik routers I moved this particular location to a CHR unlimited and the same problem continues.
Funny thing is that RouterOS loses connectivity to a subnet behind it also with The Dude active, so I can rule out ISP problems.
Too bad it is not maintained and developed anymore. Nice piece of software not considering this problem.
Re: 3 routers in a row defective possible?
Thanx for reporting, I have had problems with dude making my 1100 as well earlier, this means my find was correct.