Community discussions

MikroTik App
 
TheCiscoGuy
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Fri Jun 22, 2018 8:32 am

Strange Routing Issue

Sat Sep 01, 2018 8:51 pm

- A Site Core router (CCR-1036) acts as a PE for the EIT VRF with 1 peer (CCR-1016). A second BGP instance is configured to facilitate the route advertisement from the CE, that instance is a member of the EIT vrf and is sending default information (if exists)
- The site router is configured to redistribute all connected routes
- A Cisco 7600 PE is sending a default route for the EIT vrf
- The site router and the Cisco 7600 are using a mikrotik CCR-1016 as a route reflector and the route reflector is configured for vpnv4 address families
- Route advertisement is functioning as expected and connectivity for end user subnets at the CE are functioning (with odd impacts)
- Affected TCP flow to port 80 or 443 (other ports are untested)

Remote management to some destination IP addresses do not function, even though pings work. The following is the troubleshooting I have conducted and have come up with more questions than answers. The only thing I have identified as a common thread is all reported impacted destinations have IP addresses X.X.255.X and are using (formally) BOGON 11.0.0.0/8. Please note that all of these addresses are internal use only.

This feels like an ACL drop, however, I do not see any ACLs that could/would impact the flow. It appears to be silent drop within the mikrotik. I have set up numerous ACLs to track the connection and I see counts increase on input, prerouting, forward and postrouting rules and no counts on output. I have set up logging that shows that the correct interface is selected for outbound, however capture shows 0 packets. I even thought that perhaps this is an issue with the capture, and ran a capture on the radio directly connected to the outbound interface and validated no packets shown.

Just to be clear there are no blocking rules in the firewall, but (again for clarity) there is no packets counted on any blocking rule. ICMP packets are working as expected, other devices within the same subnet without the third octet of 255 are functioning completely. HTTP and HTTPS are disabled on all mikrotiks
Network Solutions Engineer and Trainer
Cisco | Juniper | Mikrotik | Ubiquiti
 
TheCiscoGuy
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Fri Jun 22, 2018 8:32 am

Re: Strange Routing Issue

Sat Sep 01, 2018 8:59 pm

I have attached the output of log files showing that this traffic passes prerouting, forward, and postrouting rules and the inbound/outbound interfaces are correct (redacted)
10:55:25 firewall,info prerouting: in:ether6.2000 out:(unknown 0), src-mac X:X:X:X:X:X, proto TCP (SYN), X.X.1.9:12345->X.X.255.219:80, len 114 

10:56:38 firewall,info forward: in:ether6.2000 out:ether5.1, src-mac X:X:X:X:X:X, proto TCP (SYN), X.X.1.9:12345->X.X.255.219:80, len 114 

10:53:06 firewall,info postrouting: in:(unknown 0) out:ether5.1, src-mac X:X:X:X:X:X, proto TCP (SYN), X.X.1.9:12345->X.X.255.219:80, len 114 
Network Solutions Engineer and Trainer
Cisco | Juniper | Mikrotik | Ubiquiti
 
sindy
Forum Guru
Forum Guru
Posts: 5381
Joined: Mon Dec 04, 2017 9:19 pm

Re: Strange Routing Issue

Sun Sep 02, 2018 10:56 pm

If your action=passthrough or action=log rules (you haven't stated which ones generate those logs lines but it doesn't really matter) show the packets to traverse the Mikrotik the right way, the next search should be on the destination devices (some ACLs restricting management access to other subnets/ranges than those from which you connect)? To be sure, use /tool torch or /tool sniffer on the output interface (facing towards the managed devices) - if you can see the connection attempts there and no responses, there is a trouble on the managed devices themselves. If you can see packets in both directions, the issue may be the Mikrotik's firewall blocking the responses. If so, follow the suggestion in my automatic signature.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
TheCiscoGuy
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 51
Joined: Fri Jun 22, 2018 8:32 am

Re: Strange Routing Issue

Mon Sep 03, 2018 8:24 am

Interesting signature, action is permit log=yes, however, torch was one of the first tools I used to diagnose the issue and it showed traffic (as expected) from the inbound interface, but nothing on the egress. I am setting up a similar, simplified, scenario in GNS3 this week and will continue to test and I will be able to post the configurations from the lab if I can replicate the issue.
Network Solutions Engineer and Trainer
Cisco | Juniper | Mikrotik | Ubiquiti
 
sindy
Forum Guru
Forum Guru
Posts: 5381
Joined: Mon Dec 04, 2017 9:19 pm

Re: Strange Routing Issue

Mon Sep 03, 2018 8:38 am

log=yes can be set even for an action=drop rule, so if mangle postrouting shows the packet but /tool torch does not, it can be
  • the logging rule dropping it,
  • some ipsec policy stealing and redirecting it,
  • a srcnat rule matching too widely and changing the packet's source address,
  • the torch filter being too selective and not showing the packets.
Check the packet flow diagram if you haven't yet.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
mducharme
Trainer
Trainer
Posts: 981
Joined: Tue Jul 19, 2016 6:45 pm

Re: Strange Routing Issue

Mon Sep 03, 2018 9:29 am

This feels like an ACL drop, however, I do not see any ACLs that could/would impact the flow. It appears to be silent drop within the mikrotik.
I have seen a similar kind of silent drop to this, although I don't know if it helps you with this case, but I wanted to share it anyway.

Last year we merged two subnets on our core router, two /25's merging into one /24 (both customer subnets with IPs assigned via dynamic addressing). IIRC, after this was done, two customers had their packets silently dropped, they happened to get an IP exactly mid-way through the range, on the .127 broadcast address for the first /25 subnet that used to exist on the router, and the .128 network address for the second /25 subnet that used to exist (before the merger). It seemed that the router cached the old subnet scheme in some way and still thought that the IP ending in .127 was the broadcast for the /25 that used to exist instead of being midway through the /24, and thought that the IP ending in .128 was the network for the second /25. Any packets sent from the .127 and .128 IPs were silently dropped even if allowed completely in the firewall, similar to your description here. A reboot of the router cleared the cached entry and resolved the problem.
 
User avatar
CZFan
Forum Guru
Forum Guru
Posts: 1694
Joined: Sun Oct 09, 2016 8:25 pm
Location: South Africa, Randburg
Contact:

Re: Strange Routing Issue

Mon Sep 03, 2018 10:18 am

Make the following change, from TheCiscoGuy to TheMikrotikGuy, reboot and test again :-)
MTCNA, MTCTCE, MTCRE & MTCINE

Who is online

Users browsing this forum: inteq, vilpalu and 111 guests