Hello,
I'm having trouble with a new RB750Gr3 (RouterOS v 6.49.20) which is a replacement for an old device that we reconfigured from scratch replicating manually the same configurations of the previous RB.
We have 3 GRE/IPSec tunnels, all of them with the exact same configuration.
- Tunnel 1 goes through ISP A to a Linux Server.
- Tunnel 2 goes through ISP B to another RB via ISP C.
- Tunnel 3 goes through ISP B to the other RB via ISP D.
ISP B has a lower MTU (1492) due to a PPPoE connection which is dialed by the routed ONT, transparent to the RB.
All of these tunnels have a /30 address, and OSPF configured as PTP interfaces on the backbone area.
The old setup (RB941-2nD ROS v6.48.3) used to work perfectly fine and we've never had any issues, even with previous RouterOS versions. The other RB and the Linux server remain unchanged and have other connections with the same configuration to other devices, including Mikrotik(different models and versions).
After replacing the device, at first with RouterOS 7 (the most recent stable version), we first noticed that the OSPF neighbors kept stuck in Exchange (or a few times ExStart) and took way too long to switch to Full, especially after a WAN link going down and up again. Most times it was stuck in Exchange and never turned to Full without manual intervention. We initially tough of some bug or incompatibility with the new ROS7 version, as all other devices run ROS6 and we haven't validated ROS7 in our network for this VPN/OSPF scenario yet. So we found it safer to keep using ROS6 on this device as it's intended for production on a remote site and we can't take any risks or use it for testing.
With ROS 6.49.20 (the most recent stable version), it seemed to be "more stable", but still took too long for the OSPF neighbors to change to FULL state on tunnels 2 and 3, with the "State changes" counter significantly higher than the other clients.
Also, when traffic is routed via either tunnel 2 or 3, some pages don't load correctly (or don't even load at all), file transfers with big file sizes (like around 10MB) fail, and Winbox connection to the router keeps dropping and reconnecting from time to time (especially when a window is opened or a change made).
It seemed A LOT like an MTU issue, and we've already addressed this in every possible way (changed MTU on the physical interfaces and tunnel, created change-MSS mangle rules etc) but it's apparently okay. We left the GRE Tunnel with the MTU manually set to 1280 now on both sides of all tunnels. 1280-byte pings inside the tunnel with do-not-fragment flag goes through with 0% loss and normal latency.
During troubleshooting we found that disabling IPSec on the tunnel (leaving just the GRE tunnel itself), makes everything work fine. So we started thinking of some issue with IPSEC, or MTU again.
Tunnel 1 is working completely fine with the original configuration. Tunnels 2 and 3 only work if ipsec is disabled. All three tunnels have the same GRE, IPSec and OSPF configuration. And all three tunnels used to work in the old RB.
I also remember having this Winbox disconnecting issue when configuring the router before sending it to the location it's currently installed (i.e. with another ISP), so I'd already discard any problems with the ISP-B connection. They also assured us everything is okay on their side.
I had already thought at some point about the possibility of a hardware issue regarding the IPSec Hardware Acceleration feature, but I dismissed it as one of the tunnels work normally.
After examining everything many times and comparing the current configuration with the old router, I found that the "in-state-protocol-errors" counter in ip ipsec statistics kept increasing every time a page wasn't being loaded or an OSPF neighbor was a long time in Exchange state. Then I started investigating IPSec more closely, and found these two threads (¹links below), making me suspect of a problem related to the HW Acceleration again.
I changed then the IPSec proposal (which by our standard was using sha1 / 3des) to different algorithms (sha512 / aes-256-cbc), thus disabling hardware acceleration. And everything started working perfectly! The "in-state-protocol-errors" counters stops increasing, OSPF neighbors go to FULL as quickly as before, all traffic goes well, all pages load correctly... But we obviously can't leave with that way because of security and performance issues, as well as the company's standard (I had to request to change the proposal on the other side too to perform this test).
So now I finally found out it's definitely some issue related to IPSec, most probably with Hardware Acceleration, possibly specific to this RB model/architecture (mmips) as reported by the other posts below, but I have no clue on how to fix this issue. The weirdest part is, as I said before, Tunnel 1 has the exact same configuration (including ipsec algorithms which use hw acceleration), and it works!
Does anyone have an idea on what to do? I have already talked to other MikroTik certified consultants and we don't have any clue. I'll also try to get in touch with Mikrotik support now.
Thanks in advance for everyone's time. Any help is appreciated.
¹ Related threads:
