Community discussions

MikroTik App
 
q89
just joined
Topic Author
Posts: 10
Joined: Mon Dec 11, 2017 2:34 pm

Router debugging and random RST packets.

Tue Feb 13, 2018 5:30 pm

Hi

I'm in a similar situation described in this topic viewtopic.php?t=61826

Some time ago we've bought RB2011UiAS-RM to replace an old and faulty linux proxy server in my workplace. The router accepts two WAN symetric connections on 100Mb ports, and combines them using load-balancing+failover method.

We have also a managed cisco switch (24 100Mb ports, 4 1Gb ports). Ports 1 and 13* are reserved for WAN1, ports 2 and 14* for WAN2 and port 1G for LAN, all other ports use vlans to distribute these three networks on our servers (small servers, just for office use). The router is connected to three cisco switch ports, 13 for WAN1, 14 for WAN2 and 1G for LAN. Other gigabyte ports on a router are used for company local network.

* - underneath

Most of the time the internet works just fine, but from time to time I can't get on some websites. A browser shows an error page saying connection has been reset and it's not secure. I've tried this on firefox and chrome but both browsers respond with the same error. The problem persisted on stackoverflow :shock: until RouterOS update from 6.3x to 6.41.2, now it occurs just randomly on some less visited websites for a few minutes.

Wireshark shows lots of RST packets send from external IP addreses to me, but I'm sure it's the router itself. TTL of each TCP Reset is short (50-60) and MAC address is always pointing at our router.

It happened a moment ago, I disabled WAN2 interface and bam! It works. .. unless it didn't solve the problem and I have to wait some time for another situation to happen.
/interface bridge
add admin-mac=MAC:MAC:MAC auto-mac=no comment=defconf fast-forward=no name=bridge
/interface ethernet
set [ find default-name=ether2 ] name=LAN
set [ find default-name=ether8 ] name=WAN-myIsp1
set [ find default-name=ether9 ] disabled=yes name=WAN-myIsp2
set [ find default-name=ether6 ] name=ether6-master
/interface list
add name=WAN-List
add name=mactel
add name=mac-winbox
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=default-dhcp ranges=10.1.5.10-10.1.6.254
/ip dhcp-server
add address-pool=default-dhcp authoritative=after-2sec-delay disabled=no interface=bridge name=defconf
/snmp community
set [ find default=yes ] addresses=0.0.0.0/0
/interface bridge port
add bridge=bridge comment=defconf interface=LAN
add bridge=bridge comment=defconf disabled=yes hw=no interface=ether6-master
add bridge=bridge comment=defconf disabled=yes hw=no interface=sfp1
add bridge=bridge interface=ether1
add bridge=bridge interface=ether3
add bridge=bridge interface=ether4
add bridge=bridge interface=ether5
/interface list member
add interface=WAN-myIsp1 list=WAN-List
add interface=WAN-myIsp2 list=WAN-List
add interface=bridge list=mactel
add interface=bridge list=mac-winbox
/ip address
add address=192.168.1.174/28 comment="WAN MyISP1" interface=WAN-myIsp1 network=192.168.1.160
add address=192.168.2.78/28 comment="WAN MyISP2" interface=WAN-myIsp2 network=192.168.2.64
add address=10.1.0.1/21 comment=LAN interface=bridge network=10.1.0.0
/ip dhcp-client
add comment=defconf dhcp-options=hostname,clientid interface=WAN-myIsp1
/ip dhcp-server network
add address=10.1.0.0/21 comment=defconf dns-server=8.8.8.8,8.8.4.4 gateway=10.1.0.1
/ip dns
set servers=8.8.8.8,8.8.4.4
/ip dns static
add address=192.168.88.1 name=router
/ip firewall filter
add action=drop chain=input comment="Blokada dost\EApu do WINBOX z zewn\B9trz" in-interface-list=WAN-List port=8291 protocol=tcp
add action=drop chain=input in-interface-list=WAN-List port=8291 protocol=udp
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept established,related" connection-state=established,related,new
add action=drop chain=input comment="defconf: drop all from WAN" in-interface-list=WAN-List
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related" connection-state=established,related,new
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new disabled=yes in-interface-list=WAN-List
/ip firewall mangle
add action=accept chain=prerouting connection-limit=0,32 dst-address=192.168.1.0/24 in-interface=bridge
add action=accept chain=prerouting connection-limit=0,32 dst-address=192.168.2.0/24 in-interface=bridge
add action=mark-connection chain=prerouting connection-limit=0,32 connection-mark=no-mark in-interface=WAN-myIsp1 new-connection-mark=WAN-myIsp1_conn passthrough=yes
add action=mark-connection chain=prerouting connection-limit=0,32 connection-mark=no-mark in-interface=WAN-myIsp2 new-connection-mark=WAN-myIsp2_conn passthrough=yes
add action=mark-connection chain=prerouting connection-limit=0,32 connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=WAN-myIsp1_conn passthrough=yes \
    per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting connection-limit=0,32 connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=WAN-myIsp2_conn passthrough=yes per-connection-classifier=\
    both-addresses:2/1
add action=mark-routing chain=prerouting connection-limit=0,32 connection-mark=WAN-myIsp1_conn in-interface=bridge new-routing-mark=to_WAN-myIsp1 passthrough=yes
add action=mark-routing chain=prerouting connection-limit=0,32 connection-mark=WAN-myIsp2_conn in-interface=bridge new-routing-mark=to_WAN-myIsp2 passthrough=yes
add action=mark-routing chain=output connection-limit=0,32 connection-mark=WAN-myIsp1_conn new-routing-mark=to_WAN-myIsp1 passthrough=yes
add action=mark-routing chain=output connection-limit=0,32 connection-mark=WAN-myIsp2_conn new-routing-mark=to_WAN-myIsp2 passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" disabled=yes src-address=10.1.0.0/21
add action=masquerade chain=srcnat out-interface=WAN-myIsp1
add action=masquerade chain=srcnat out-interface=WAN-myIsp2
/ip route
add check-gateway=ping distance=1 gateway=192.168.1.161 routing-mark=to_WAN-myIsp1
add check-gateway=ping distance=1 gateway=192.168.2.65 routing-mark=to_WAN-myIsp2
add check-gateway=ping distance=1 gateway=192.168.1.161
add check-gateway=ping distance=2 gateway=192.168.2.65
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes port=59184
set api disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=Europe/Warsaw
/system logging
add topics=interface,route,ipsec
/system ntp client
set enabled=yes primary-ntp=95.158.95.123 secondary-ntp=91.232.160.1
/tool mac-server
set allowed-interface-list=mactel
/tool mac-server mac-winbox
set allowed-interface-list=mac-winbox
/tool sniffer
set filter-interface=LAN filter-ip-address=10.1.4.10/32
Previous topic: Router is randomly sending TCP Reset packets.
Last edited by q89 on Mon Feb 19, 2018 10:30 am, edited 2 times in total.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Router is randomly sending TCP Reset packets.

Tue Feb 13, 2018 7:49 pm

What's connection-limit=0,32 in mangle rules supposed to do?
 
q89
just joined
Topic Author
Posts: 10
Joined: Mon Dec 11, 2017 2:34 pm

Re: Router is randomly sending TCP Reset packets.

Tue Feb 13, 2018 8:06 pm

I was fiddling (today) with configuration here and there, and forgot to change it back. It has nothing to do with the problem.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: Router is randomly sending TCP Reset packets.

Tue Feb 13, 2018 10:39 pm

Too bad, it was the only thing that looked strange.
 
q89
just joined
Topic Author
Posts: 10
Joined: Mon Dec 11, 2017 2:34 pm

Re: Router is randomly sending TCP Reset packets.

Wed Feb 14, 2018 5:24 pm

I think I spot source of this problem.
add check-gateway=ping distance=1 gateway=192.168.2.65 routing-mark=to_WAN-myIsp2
Distance here should be set to 2 and that should fix it all. I will look after it if the problem shows again.

Also, other faulty packets like TCP Keep Alive or TCP Dup Ack are not caused by the router, but more like by ISP or other faulty equipment outside our office.

Dual wan setup is based on https://wiki.mikrotik.com/wiki/Manual:PCC
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Router is randomly sending TCP Reset packets.

Wed Feb 14, 2018 11:03 pm

Wireshark shows lots of RST packets send from external IP addreses to me, but I'm sure it's the router itself.
Just to be even more sure, have you tried to use packet sniffer to watch the uplink (internet-facing) interface simultaneously to see whether the 'Tik really generates the RST packets on its own? You can forward (stream) the sniff taken at the internet-facing interface to the machine in the LAN wunning Wireshark using TZSP encapsulation if you have enough bandwidth there (i.e. if you can filter on the IP of the server with which the problem occurs).

MAC address is always pointing at our router.
That's no indicator. Any packet from any source IP addres outside the subnet where the receiving interface is will have a source MAC address of the router.
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Router is randomly sending TCP Reset packets.

Wed Feb 14, 2018 11:26 pm

I think I spot source of this problem.
add check-gateway=ping distance=1 gateway=192.168.2.65 routing-mark=to_WAN-myIsp2
Distance here should be set to 2 and that should fix it all. I will look after it if the problem shows again.
Unfortunately, this assumption is wrong.
Distance only distinguishes the priority between two routes with identical dst-address and identical routing mark. In your case, these are the two routes with no (visible) routing mark which actually means that they use the default routing mark "main". But for the route above, there is no twin route as no other route has the same routing mark, so setting its distance to 2 affects nothing.

I suspect your problem is caused by something else - namely, the way how connection tracking and, in particular, NAT works. When a new connection gets established to which a src-nat rule (like masquerade) is applied, connection tracking builds a context for that connection which holds the quadruple of internal and external sockets involved. It means that all further packets of that connection are sent out with the public IP and port of the WAN interface through which that connection's first outbound packet was routed. This is, unfortunately and not immediately obviously, true even if that WAN interface goes down. In such case, thanks to your backup configuration, the subsequent packets of that connection are sent out through the other WAN interface but still with the IP address of the one used initially, which likely makes some security element at the ISP providing that uplink reset the connection as it has no context for it (in Mikrotik's firewall, the "action=drop connection-state=invalid" rule would be responsible for that).
 
User avatar
acruhl
Member
Member
Posts: 371
Joined: Fri Jul 03, 2015 7:22 pm

Re: Router is randomly sending TCP Reset packets.

Thu Feb 15, 2018 2:51 am

I'm probably stating the obvious, and I have nothing really to add to sindy's response other than basics:

TCP resets are layer 4 and should only be originated by the layer 4 endpoints. You really should rule that out before you blame the router.

Sindy's explanation seems plausible (about NAT maybe not handling it properly based on config). But rule out the easy stuff first.

Maybe I'm missing something?

Maybe you already know all of this. I find that it helps to start from the basics though.
 
q89
just joined
Topic Author
Posts: 10
Joined: Mon Dec 11, 2017 2:34 pm

Re: Router is randomly sending TCP Reset packets.

Thu Feb 15, 2018 11:15 am

Yeah. The problem still happens after this change.

I'm setting up a computer with two nics to capture and analyze traffic (in case I miss something capturing packets on a router).
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Router is randomly sending TCP Reset packets.

Thu Feb 15, 2018 1:48 pm

@acruhl,
TCP resets are layer 4 and should only be originated by the layer 4 endpoints.
You are right that it should be like that, however firewalls quite often send TCP resets to both endpoints if they do not like the connection.

@q89,
I'm setting up a computer with two nics to capture and analyze traffic (in case I miss something capturing packets on a router).
To confirm that my assumption is correct, it should be enough to capture at WAN2 until the issue occurs, with the capture filter set to IP address of WAN1. If configured that way, the internal sniffing is a zero-effort and safe solution as it can run for quite a long time without risk - no packets should match that filter until the issue occurs so the capture file won't grow quickly.
 
q89
just joined
Topic Author
Posts: 10
Joined: Mon Dec 11, 2017 2:34 pm

Re: Router is randomly sending TCP Reset packets.

Thu Feb 15, 2018 3:50 pm

I did some testing using another computer with two bridged ethernet ports. Sadly both connections seem fine without any reset packets (these tests were done using tshark and wireshark, not on a router itself but between ISP and the router).

I started to blame cisco switch, bypassed it connecting wan1 and wan2 directly to the router, started to setup vlans for servers on it (not on a switch) when the issue showed up again while browsing the internet.

Now it's either routeros configuration or some unknown bug.
 
q89
just joined
Topic Author
Posts: 10
Joined: Mon Dec 11, 2017 2:34 pm

Re: Router is randomly sending TCP Reset packets.

Thu Feb 15, 2018 11:42 pm

Ok. I disabled fasttrack firewall rule, because it was the last thing suspicious to me. I'm going to test it tomorrow morning at work and see if it solves the problem.

This unwanted behaviour is really close if not exact to this:
https://www.reddit.com/r/mikrotik/comme ... an_policy/
Unwanted behavior? TCP traffic going out WAN2 would intermittently fail for a specific destination, possibly for a specific host/dst pair. As in, for minutes at a time; then, it would pass packets again. ICMP never failed. WAN1 not affected.
The issue i'm describing happens only on some new hosts, and just for a few minutes.
 
q89
just joined
Topic Author
Posts: 10
Joined: Mon Dec 11, 2017 2:34 pm

Re: Router is randomly sending TCP Reset packets.

Fri Feb 16, 2018 4:41 pm

This problem is now gone with disabled fasttrack rule. I will mark this topic as solved next week, when this issue won't show up again.
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Router is randomly sending TCP Reset packets.

Fri Feb 16, 2018 4:56 pm

I have completely missed that you use the mangle rules simultaneously with fasttracking when reading your configuration for the first time. But bear in mind that even with fasttracking disabled, what I wrote before can happen if one of your WANs fail while some connection through it is in use, so if an issue with same symptoms occurs during the quarantine period you've proclaimed before marking the topic as solved, first check the log for WAN going down and up again before concluding that disabling the fasttracking did not help. It did.
 
q89
just joined
Topic Author
Posts: 10
Joined: Mon Dec 11, 2017 2:34 pm

Re: Router is randomly sending TCP Reset packets.

Mon Feb 19, 2018 10:28 am

I'd like to apologize for blaming mikrotik router.

Today we found out this problem doesn't happen in the router directly, but in the building network we don't own and sadly we can't fix the problem. Disabling fasttrack routine indeed lowered chances of it happening.
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: Router is randomly sending TCP Reset packets.

Mon Feb 19, 2018 11:22 am

sadly we can't fix the problem. Disabling fasttrack routine indeed lowered chances of it happening.
If disabling fasttracking has "lowered chances of it to happen", there is still a good chance that the external network sends the resets in reaction to something what Mikrotik sends to it. So I'd still be all for sniffing on both WANs for what happens right before the TCP RST come.

Who is online

Users browsing this forum: Bing [Bot], GoogleOther [Bot], jaclaz, RichardDok, straightslant and 100 guests