RB5009 drops hardware vpn packets but not through another switch

HI there,

A weird question here. I have an RB5009 which works well, however I’ve been issued a corporate VPN device which is using some sort of L2TP/Ipsec tunnel through port 4500. I then connect my work device to this box to be permanently routed through the corporate VPN.

I can see the connection to the VPN being established from winbox on the rb5009. I’ve connected the VPN box to ether7, connection comes up, and I can ping an IP at the other end of the tunnel, however maybe only 60% of the packets get through. As a result, the VPN box decides things aren’t working as planned and then reconnects. This continues forever.

However, if I plug the VPN box into a Netgear managed switch (with no VLANs or anything configured, factory reset), then the netgear into the RB5009, it works absolutely perfectly.

What could the Netgear be doing that’s making this work? Is it an MTU issue? ether7 is default configuration (1500MTU, 1514 L2 MTU).

Thanks!

Hard to day without seeing your 5009 config
/export file=anynameyouwish ( minus router serial number, and any public IP information)

The router should be transparent to the device and its connectivity through the internet to office site using the office vpn.

I assume most (if not all) of the RB5009 parts are part of a bridge-interface? What happens if you plug it into ether6 of ether5 of so ?
You sure the physical interface is not giving errors ? Is the interface negotiated correctly at 1Gbits/sec ?

Thanks for the rapid replies - I’ve just had a quick look and found this reddit comment https://www.reddit.com/r/mikrotik/comments/1blxmet/comment/kw9k2kl/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button which appears to describe similar symptoms and indeed setting:

/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu passthrough=yes \
protocol=tcp tcp-flags=syn

Has done the trick!

For more info, I’m on PPPoE over fibre, with 1492 MTU which has caused issues in the past (with wireguard, for example).

New problem I’m facing is why the link is dropped at seemingly random intervals too, resulting in the DHCP de-assigning & reassigning instantly (ether7 link down, 3 seconds later ether7 link up (speed 1G, full duplex))… Have seen some earlier posts suggesting this is due to the DHCP Server Lease window being open Winbox! I’ll close it all down and see if it reoccurs…

Short follow up to the above issue with my ether port on rb5009 ‘flapping’: http://forum.mikrotik.com/t/port-flapping-rb5009-vodafone-modem/153058/1 has been discussed many times already. I’m testing auto negotiation disabled and setting to 1G full duplex. Hopefully that’s the end of this problem.

Well, it wasnt :joy:! I’ve had 10 disconnections today, and counting.
Tried different cable, disabling auto negotiation but nothing works. I’m going to try to downgrade the ROS now (am on 7.18.2)

I think you’ve tried the usual stuff. Sometimes gigabit auto kind of conspires to make your life difficult.

Downgrading etc. usually doesn’t help with these devices. (I mean devices with the Marvell switch chips.) The drivers are solid in this regard and simply don’t change version to version. When near line-rate data is exchanged between ports with different speeds is another thing altogether, here Mikrotik has changed default buffer allocations multiple times - but I don’t think you’re running into this, or you would have told us. You would have, wouldn’t you? :slight_smile:

The rb5009 actually has ether1 as a 2.5G port, which has a different and better PHY. Try it with that one. It will probably work.

I know it’s an eyesore, especially for technically minded people - but sometimes the easiest and quickest solution is to put a cheap dumb switch between the two.

There has been a time that I also regulary encountered link down/up/down/up with 1 of my Unifi AP’s connected. That “suddenly” dissapeared.
It would be hard to believe that the change-mss would have fixed it, especially the ultimate fix is inserting another L2-switch.
Can you test with going into Winbox then selecting “ether7” interface and then going to “Ethernet” and change some “Advertisement” parameters ? Try to only advertise 100Mbps Full Duplex ? Normally your VPN-box is also autoneg and will support 100/FD too.

Maybe the separate box, does not follow protocols properly?? Bad cables??

Thanks again for all the replies. I’m clearly not alone in this issue as it appears have been reported several times on this forum. The external box is a GLinet Beryl AX, so a relatively new device, but of course it is anybody’s guess if it truly follows standards. All I know is I didn’t have a problem with it when plugged in to the switch…

In any case, I’ll play with all the options I can see over the weekend. Probably worth raising a support ticket too as it appears to be a known issue still unresolved.

Well the problem could still be the config, which you have refused to provide. There may be some collision with the box protocols and the MT config for example.

Alright so latest update on this is that disabling auto negotiation and setting to 100MBit Full duplex has resulted in a full day of no dropped connections, thanks @jvanhambelgium for the suggestion. As for getting that tasty gigabit, I probably will just have to take @lurker888 suggestion and drop a switch between. I have gone through 6 cables, 5 cat 5e and 1 cat 6 (in case that made a difference) and it didn’t. The Change MSS sorted out the dropped packets (the title of this post), but subsequently the port flapping (evidently a well known Mikrotik issue, widely reported on this forum) came to face me.

Some relevant config (moved over to ether6 rather than ether7 to see if it was a malfunctioning port - it wasn’t), port goes onto guestvlan10 pvid 10:

/interface bridge
add admin-mac=FF:FF:FF:FF:FF:FF auto-mac=no comment=defconf name=bridge \
    vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] comment=lan
set [ find default-name=ether6 ] auto-negotiation=no speed=100M-baseT-full
set [ find default-name=ether8 ] comment="emergency access"
set [ find default-name=sfp-sfpplus1 ] auto-negotiation=no comment=wan speed=\
    1G-baseT-full
/interface vlan
add comment="guest vlan" interface=bridge name=guestvlan10 vlan-id=10
add comment="media vlan" interface=bridge name=mediavlan20 vlan-id=20
add comment="fibre vlan" interface=sfp-sfpplus1 name=vlan911 \
    vlan-id=911
add interface=bridge name=vlan4086 vlan-id=4086
/interface bridge port
add bridge=bridge comment=defconf interface=ether1
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=ether6 pvid=10
add bridge=bridge comment=defconf interface=ether7 pvid=10
/interface bridge vlan
add bridge=bridge tagged=bridge,ether1 vlan-ids=4086
add bridge=bridge tagged=ether1,bridge vlan-ids=10
add bridge=bridge tagged=bridge,ether1 vlan-ids=20

and some firewall rules if it makes any difference

/ip firewall filter
add action=drop chain=forward comment="deny guestvlan10 everything but WAN" \
    in-interface=guestvlan10 log-prefix="block guestvlan10" \
    out-interface-list=!WAN
add action=accept chain=input comment=wireguard dst-port=51820 protocol=udp
add action=accept chain=input comment="Allow mDNS" dst-address=224.0.0.251 \
    dst-port=5353 log-prefix=mDNS protocol=udp src-port=5353
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=accept chain=input comment="allow mediavlan20 chromecast" \
    in-interface=mediavlan20 log-prefix="allow vlan20"
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" disabled=yes \
    protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=forward comment="allow mediavlan20 internet only" \
    connection-state=new in-interface=mediavlan20 log-prefix=\
    "allow vlan internet:" out-interface-list=WAN
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN log-prefix="deny lan"
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
add action=accept chain=forward comment=\
    "chromecast connections to mediavlan20" connection-state=new \
    in-interface-list=LAN out-interface=mediavlan20
add action=accept chain=forward comment="allow within lan list" \
    connection-state=new in-interface-list=LAN out-interface-list=LAN
add action=drop chain=forward comment="disable inter-vlan routing" \
    dst-address-list=not_in_internet src-address-list=not_in_internet
/ip firewall mangle
add action=change-mss chain=forward comment=\
    "Clamp MSS to PMTU for Outgoing packets" new-mss=clamp-to-pmtu \
    out-interface=wireguard1 protocol=tcp tcp-flags=syn
add action=change-mss chain=forward comment="Fix L2TP/IPsec issues" new-mss=\
    clamp-to-pmtu out-interface-list=WAN protocol=tcp tcp-flags=syn
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN

The port flapping issue is definitely real and I am researching this for a long time. I have tried switching ports, cables, it does not matter, in 99% of cases its not cables, I am tired by how many people mention cables. In my life, the cable issues have hints that look totally different, for example, a bad cable will NEVER produce such an effect like being able to work for 1 month without a single flap and then suddenly start flapping non-stop, independently of traffic, where re-plugging does nothing (or only for a short time), the only thing that helps (for much longer, but still limited time) is power cycling the RB5009. What is more interesting, when flapping starts, even the reboot does not help for long, only power cycling manages to restore it for longer.

Also as an electronics engineer I can assure that at short lengths the cable quality does not even matter, it can be absolute junk and still work perfectly because it manages to not only meet, but even exceed the necessary signal quality/strength tolerances. It is almost impossible for a bad cable to create ANY problems if its short enough.

Anyway, in my case its the 1Gbit connection between RB5009 and Netgear GC110P managed switch. Sometimes it can work for a month, two or even more, but when the flapping starts the only solution is power cycling RB5009. Nothing else really helps, neither power cycling the Netgear switch, nor re-plugging the connectors on both ends, at least not for long. I have even tried setting the RB5009 port to 100Mbit when I noticed that flapping started, without power cycling, it worked for some time, but then started flapping even on 100Mbit, so it definitely is not the solution. Only power cycling helps.

Another curious thing that I noticed on 100Mbit, the link LED on Netgear switch can go dark after some time, but the link will still be working and not flapping. Its enough to re-plug the UTP cable in order to wake up the LED. It looks like some part of the ethernet auto-negotiation process in RB5009 stops working or starts working incorrectly. The signal quality reasons are definitely not the case here, especially when talking about 100Mbit where the cabling requirements for everything to work perfectly are simply laughable and easily overachieved even with cabling that is beyond poor.

Another possible reason that I’m thinking of is maybe it is some timing/clocking/pll issue with the switch chip 88E6393X that is used in RB5009, like maybe the clocking gets skewed over time, becomes more sensitive to something, that would explain why does it manage to work perfectly for quite a long time until something finally provokes it to become unstable, from where only the power cycling really helps, which would also mean that a soft reboot does not reset the chip internals where the clocking error accumulates in some way, and only reinitializes it. Hopefully correctable by software and not a chip/PCB design flaw.

Does the RB5009 provide time to the netgear switch (NTP). Stretch but thinking of things that may cause differences.
Wonder if you can borrow a different switch to see if the behaviour remains.

This would seem to be rather stupid. How did the MAC address get set like that?

Suspect simply hiding the mac address…??

Obviously a hidden MAC address!

I’ve since raised this whole issue with MT support and received some suggestions from their team:

We found ‘tx-queue-drop’ counters in your supout.rif file(as seen in /interface print stats), indicating that the device has experienced issues sending packets out of a specific interface due to congestion. Common scenarios include:

  1. Sending traffic from a higher-rate interface to a lower-rate interface (e.g., 1 Gbps to 100 Mbps).
  2. Forwarding traffic from multiple interfaces to a single interface (e.g., 2x 1 Gbps to 1x 1 Gbps).
  3. High traffic volumes from VLAN sub-interfaces.

When congestion occurs, the switch initially queues packets. However, if the device’s buffer resources are limited, packets will be dropped to manage the congestion until queue space becomes available. This can also lead to interface reset.

To mitigate this issue, we suggest enabling PFIFO(Packet First-In, First-Out) queues as a software-based buffering mechanism. This can help manage packet queuing more effectively. Use the following CLI commands to configure multi-queues:

/queue type set ethernet-default pfifo-limit=1000
/queue interface set [find where queue!=no-queue] queue=ethernet-default

>
> If you need further assistance, please don't hesitate to reach out.

This, however, didn't make any difference for my issue - maybe it'll help someone else as it seems like a reasonable thing to try. I've heard nothing back since replying to them saying it did nothing for me.

As mentioned above I've sandwiched a Netgear GS108T between the devices and all has been working perfectly for some time now.

It’s not obvious at all. Writing something like aa:bb:cc:dd:ee:ff is obvious.

Is there an obviousness scale?
Maybe:

  1. 12:34:56:78:90:AB
  2. aa:bb:cc:dd:ee:ff
  3. FF:FF:FF:FF:FF:FF

SE:CR:ET:MA:CA:DD