TL;DR RouterOS is always sending traffic for 81.0.0.208* out of the sfp-sfpplus1 interface, back to our upstream provider. It has a valid route for routing this address internally. Assigning this IP to a local interface has no effect.
Details:
I run a small ISP and this issue relates to our edge router. The edge route has BGP peering to our upstream provider. The subnet used for this is 81.0.0.94/30*. We also advertise the prefix 81.0.0.208/29*, which they then route to us. These IPs are used as follows:
- 81.0.0.208-209 - Routed internally, each as a /32
- 81.0.0.210-215 - CGNAT on the edge router, each IP attached to sfp-sfpplus1 as a /32
Previously, these IPs were attached to sfp-sfpplus1 using a /29 prefix (i.e. 81.0.0.208/29), which was a mistake. I’ve changed this to a /32 as I now want to route .208 & .209 internally.
The CGNAT works fine, and 81.0.0.209 is routed just fine. However, RouterOS always sends traffic to 81.0.0.208 out of the sfp-sfpplus1 interface (to the broadcast MAC address). That made sense when I was using 81.0.0.208**/29** as a subnet attached to sfp-sfpplus1, but that is no longer the case. Yet RouterOS is still behaving as though 81.0.0.208 is a broadcast address on that interface.
I have even assigned this IP to a local interface, which had no affect at all. I also created a manual ARP entry, which was completely ignored.
It is quite possible I am doing/thinking something silly here, but I have looked into this for several hours now. I’ve done a /export and checked for references to 81.0.0.208, but nothing seems out of place.
The only thought I have now is, “perhaps its internal state has got messed up”. So I’m considering doing a restart of the router, but that’ll take down our ISP for a few (nail-biting) minutes. Is my “turn it off and on again” solution a good one, or are there any other better options?
Router is an RB4011 running 6.47.10.
Any advice would be welcome.
*Not the actual IP
hello @ adam,
well, maybe you could give us a little bit of clue about your interface and ip configuration - which you think problematic . otherwise we would have no idea?
maybe along with bgp peering config?
and please… not the whole export - my eyes hurt 
Hi @wiseroute. Sure thing, and thank you. I think this is everything relevant:
/interface ethernet
set [ find default-name=sfp-sfpplus1 ] name=sfp-sfpplus1-iptelecom
/interface vlan
add interface=sfp-sfpplus1-iptelecom name=core-transit-iptelecom-1 vlan-id=230
add interface=ether1 name=core-vlan vlan-id=100
add interface=ether1 name=management vlan-id=84
/ip address
add address=10.100.0.2/16 interface=management network=10.100.0.0
add address=100.127.0.2 interface=loopback network=100.127.0.2
add address=81.0.0.94/30 interface=sfp-sfpplus1-iptelecom network=81.0.0.92
add address=81.0.0.210 interface=sfp-sfpplus1-iptelecom network=81.0.0.210
add address=81.0.0.211 interface=sfp-sfpplus1-iptelecom network=81.0.0.211
add address=81.0.0.212 interface=sfp-sfpplus1-iptelecom network=81.0.0.212
add address=81.0.0.213 interface=sfp-sfpplus1-iptelecom network=81.0.0.213
add address=81.0.0.214 interface=sfp-sfpplus1-iptelecom network=81.0.0.214
add address=81.0.0.215 interface=sfp-sfpplus1-iptelecom network=81.0.0.215
add address=100.125.0.2/24 interface=core-vlan network=100.125.0.0
/routing bgp instance
set default as=65000 client-to-client-reflection=no comment="[ ID:main ]" redistribute-other-bgp=yes redistribute-static=yes router-id=100.127.0.2
/routing bgp network
add network=81.0.0.208/29 synchronize=no
add network=81.0.0.210/32
add network=81.0.0.211/32
add network=81.0.0.212/32
add network=81.0.0.213/32
add network=81.0.0.214/32
add network=81.0.0.215/32
/routing bgp peer
add default-originate=if-installed hold-time=1m30s multihop=yes name=core out-filter=internal-out remote-address=100.127.0.1 remote-as=65000 update-source=loopback
add in-filter=iptelecom-in name=iptelecom-v4 out-filter=iptelecom-out remote-address=81.0.0.93 remote-as=29003 tcp-md5-key=IPT-GDN
add address-families=ipv6 in-filter=iptelecom-in name=iptelecom-v6 out-filter=iptelecom-out remote-address=2a02:8f0:a1ff:d2da::1 remote-as=29003 tcp-md5-key=redacted
# Relevant routes:
0 ADb dst-address=0.0.0.0/0 gateway=81.0.0.93 gateway-status=81.0.0.93 reachable via sfp-sfpplus1-iptelecom distance=20 scope=40 target-scope=10 bgp-as-path="29003"
bgp-origin=incomplete received-from=iptelecom-v4
11 ADb dst-address=81.90.51.208/32 gateway=100.127.2.1 gateway-status=100.127.2.1 recursive via 100.125.0.1 core-vlan distance=200 scope=40 target-scope=30 bgp-local-pref=100
bgp-origin=igp received-from=core
30 ADo dst-address=100.127.2.1/32 gateway=100.125.0.1 gateway-status=100.125.0.1 reachable via core-vlan distance=110 scope=20 target-scope=10 ospf-metric=45 ospf-type=inter-area
hello @ adam,
However, RouterOS always sends traffic to 81.0.0.208 out of the sfp-sfpplus1 interface (to the broadcast MAC address).
did you mean: inbound from internet to your network? or outbound from local to internet?
btw, do you have any loopback interface with ip addresses inside your router?
i am sorry, my mistake. i didn’t mean to export the config but :
- ip address print details
- interface print details
- ip route print details
- and, nevermind for that bgp .
No problem:
did you mean: inbound from internet to your network? or outbound from local to internet?
To rephrase for clarity: However, RouterOS always sends traffic to 81.0.0.208 out
of the sfp-sfpplus1 interface, headed out of our network, towards our
upstream provider (to the broadcast MAC address).
btw, do you have any loopback interface with ip addresses inside your router?
Yes. 100.127.0.2, see below
[admin@edge] /ip address> pr
Flags: X - disabled, I - invalid, D - dynamic
# ADDRESS NETWORK INTERFACE
0 10.100.0.2/16 10.100.0.0 management
1 100.127.0.2/32 100.127.0.2 loopback
2 81.0.0.94/30 81.0.0.92 sfp-sfpplus1-iptelecom
3 81.0.0.210/32 81.0.0.210 sfp-sfpplus1-iptelecom
4 81.0.0.211/32 81.0.0.211 sfp-sfpplus1-iptelecom
5 81.0.0.212/32 81.0.0.212 sfp-sfpplus1-iptelecom
6 81.0.0.213/32 81.0.0.213 sfp-sfpplus1-iptelecom
7 81.0.0.214/32 81.0.0.214 sfp-sfpplus1-iptelecom
8 81.0.0.215/32 81.0.0.215 sfp-sfpplus1-iptelecom
9 100.125.0.2/24 100.125.0.0 core-vlan
10 D 10.111.0.1/32 10.111.0.10 <ovpn>
[admin@edge] /interface> pr
Flags: D - dynamic, X - disabled, R - running, S - slave
# NAME TYPE ACTUAL-MTU L2MTU MAX-L2MTU MAC-ADDRESS
0 R ether1 ether 1500 1592 9578 48:8F:5A:99:B8:42
1 ether2 ether 1500 1592 9578 48:8F:5A:99:B8:43
2 ether3 ether 1500 1592 9578 48:8F:5A:99:B8:44
3 ether4 ether 1500 1592 9578 48:8F:5A:99:B8:45
4 ether5 ether 1500 1592 9578 48:8F:5A:99:B8:46
5 ether6 ether 1500 1592 9578 48:8F:5A:99:B8:47
6 ether7 ether 1500 1592 9578 48:8F:5A:99:B8:48
7 ether8 ether 1500 1592 9578 48:8F:5A:99:B8:49
8 ether9 ether 1500 1592 9578 48:8F:5A:99:B8:4A
9 ether10 ether 1500 1592 9578 48:8F:5A:99:B8:4B
10 R sfp-sfpplus1-iptelecom ether 1500 1600 9586 48:8F:5A:99:B8:4C
11 DR <ovpn> ovpn-in 1500
12 R core-transit-iptelecom-1 vlan 1500 1596 48:8F:5A:99:B8:4C
13 R core-vlan vlan 1500 1588 48:8F:5A:99:B8:42
14 R loopback bridge 1500 65535 46:2D:85:E0:5F:01
15 R management vlan 1500 1588 48:8F:5A:99:B8:42
[admin@edge] /ip route> pr
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 ADb 0.0.0.0/0 81.0.0.93 20
1 X S ;;; Enable if BGP peering goes down
0.0.0.0/0 81.0.0.93 1
2 A S 10.32.0.0/12 10.111.0.10 1
3 ADC 10.100.0.0/16 10.100.0.2 management 0
4 Do 10.100.0.0/16 100.125.0.1 110
5 ADo 10.110.1.0/24 100.125.0.1 110
6 ADC 10.111.0.10/32 10.111.0.1 <ovpn> 0
7 A S 10.200.0.0/16 10.111.0.10 1
8 A S 10.200.0.10/32 10.111.0.10 1
9 A S 10.200.0.11/32 10.111.0.10 1
10 ADC 81.0.0.92/30 81.0.0.94 sfp-sfpplus1-ip... 0
11 ADb 81.0.0.208/32 100.127.2.1 200
12 ADb 81.0.0.209/32 100.127.2.1 200
13 ADC 81.0.0.210/32 81.0.0.210 sfp-sfpplus1-ip... 0
14 ADC 81.0.0.211/32 81.0.0.211 sfp-sfpplus1-ip... 0
15 ADC 81.0.0.212/32 81.0.0.212 sfp-sfpplus1-ip... 0
16 ADC 81.0.0.213/32 81.0.0.213 sfp-sfpplus1-ip... 0
17 ADC 81.0.0.214/32 81.0.0.214 sfp-sfpplus1-ip... 0
18 ADC 81.0.0.215/32 81.0.0.215 sfp-sfpplus1-ip... 0
19 ADb 81.0.0.216/32 100.127.2.1 200
20 ADb 100.65.1.0/24 100.127.1.1 200
21 ADo 100.65.2.0/26 100.125.0.1 110
22 ADC 100.125.0.0/24 100.125.0.2 core-vlan 0
23 ADo 100.126.0.0/29 100.125.0.1 110
24 ADo 100.126.0.8/29 100.125.0.1 110
25 ADo 100.126.0.16/29 100.125.0.1 110
26 ADo 100.126.0.24/29 100.125.0.1 110
27 ADo 100.127.0.1/32 100.125.0.1 110
28 ADC 100.127.0.2/32 100.127.0.2 loopback 0
29 ADo 100.127.1.1/32 100.125.0.1 110
30 ADo 100.127.2.1/32 100.125.0.1 110
I bit the bullet and did a restart. It fixed the issue immediately.
I would love to know how I could have resolved this without doing a restart though. Can I clear out the router’s caches manually somehow?