Hi
I have connect two router through public IPs as VXLAN vteps.
The address for each VLAN has been set. Routers can ping each other VXLAN IP (192.168.210.0).
Route for forwarding DNS requests of router1 towards router2 works flawlessly through MAIN route table route.
When I mangle PPP clients on router1 and mark route the traffic to a table with VXLAN IP of router2 as gateway and set source NAT rule with VXLAN interface of router1 as outgoing interface and masquerade, clients can't connect to internet anymore.
In firewall connections tab C or Cs state apears due to mangle on output or preroute chain.
They can translate addresses through MAIN table route but can't access internet.

router1 config:

# apr/18/2023 09:29:28 by RouterOS 7.8

/interface sstp-server
add comment="TK Link" name=sstp-in-TK user=TK
/interface list
add name=WAN
add name=VXLAN

/ip pool
add name=pool-VXLAN ranges=192.168.2.11-192.168.2.99
/interface vxlan
add local-address=2.2.2.2 mac-address=CE:6A:2B:3B:86:E6 mtu=1500 \
name=vxlan-TK port=1350 vni=1 vrf=main vteps-ip-version=ipv4

/ppp profile

add dns-server=192.168.2.1 local-address=192.168.2.1 name=VXLAN \
remote-address=pool-VXLAN

/ppp secrets
add name=vxuser profile=VXLAN

/routing table
add fib name=VXLAN

/interface l2tp-server server
set allow-fast-path=yes enabled=yes l2tpv3-circuit-id=0 use-ipsec=required
/interface list member
add interface=ether1 list=WAN
add interface=vxlan-TK list=VXLAN

/interface sstp-server server
set certificate="VPN Server" default-profile=default-encryption enabled=yes \
keepalive-timeout=10
/interface vxlan vteps
add interface=vxlan-TK port=1350 remote-ip=1.1.1.1
/ip address
add address=2.2.2.2/24 interface=ether1 network=2.2.2.0
add address=192.168.210.2/30 interface=vxlan-TK network=192.168.210.0
/ip dns
set allow-remote-requests=yes cache-max-ttl=1d servers=\
94.140.15.15,94.140.14.14

/ip firewall filter
add action=accept chain=input comment=Winbox dst-port=7353 in-interface-list=\
WAN protocol=tcp
add action=accept chain=input comment=SSTP dst-port=443 in-interface-list=WAN \
protocol=tcp
add action=accept chain=input comment=L2TP dst-port=500,4500,1701 \
in-interface-list=WAN protocol=udp
add action=accept chain=input comment=DNS dst-port=53 in-interface-list=!WAN \
protocol=udp
add action=accept chain=input comment=ICMP in-interface-list=!WAN protocol=\
icmp
add action=accept chain=input comment=VXLAN dst-port=1350 in-interface-list=\
WAN protocol=udp
add action=accept chain=input comment=Related connection-state=\
established,related,untracked
add action=drop chain=input comment=Drop
/ip firewall mangle
add action=mark-routing chain=prerouting comment="VXLAN Mark" \
dst-address-type=!local new-routing-mark=VXLAN \
passthrough=no routing-mark=!VXLAN src-address=192.168.2.11-192.168.2.99
/ip firewall nat
add action=masquerade chain=srcnat comment="VXLAN NAT" out-interface-list=\
VXLAN routing-mark=VXLAN
add action=masquerade chain=srcnat comment="NAT" out-interface-list=WAN

/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=2.2.2.1

add comment=DNS1 disabled=no distance=1 dst-address=94.140.15.15/32 gateway=\
192.168.210.1 pref-src=0.0.0.0 routing-table=main scope=30 \
suppress-hw-offload=no target-scope=10
add comment=DNS2 disabled=no distance=1 dst-address=94.140.14.14/32 gateway=\
192.168.210.1 pref-src=0.0.0.0 routing-table=main scope=30 \
suppress-hw-offload=no target-scope=10
add comment="VPN VXLAN" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
192.168.210.1 routing-table=VXLAN scope=30 suppress-hw-offload=no \
target-scope=10
/ip service
set winbox port=7353


router2 config:

# apr/18/2023 09:12:46 by RouterOS 7.8
#

/interface vxlan
add local-address=1.1.1.2 mac-address=7E:32:B4:C1:32:DD mtu=1500 name=\
vxlan-IR0 port=1350 vni=1 vrf=main vteps-ip-version=ipv4
/ip address
add address=1.1.1.1/24 interface=ether1 network=1.1.1.0
add address=192.168.210.1/30 interface=vxlan-IR0 network=192.168.210.0
/ip dns
set servers=8.8.8.8,9.9.9.9
/ip firewall filter
add action=accept chain=input comment=Winbox dst-port=7353 in-interface=\
ether1 protocol=tcp
add action=accept chain=input comment=Ping in-interface=!ether1 protocol=icmp
add action=accept chain=input comment="Allow Related" connection-state=\
established,related,untracked
add action=accept chain=input comment=VXLAN dst-port=1350 in-interface=ether1 \
protocol=udp
add action=drop chain=forward comment="WAN !desnated forward drop" \
connection-nat-state=!dstnat connection-state=new in-interface=ether1
add action=drop chain=input comment="Input drop"
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=1.1.1.1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
set winbox port=7353
set api-ssl disabled=yes

Any recommendations?
There is no problem with same config on other tunnelling protocols and interfaces.
It seems like a routing problem related to vxlan interface.
Please advise.

I have no idea what you're trying to do here. You have two site and want to join them with a VXLAN, got that part. I'm not sure that's best protocol over a WAN given it's higher overhead than WG or EoIP+IPSec.

Your firewall config is pretty confusing and seemingly incomplete. I'm not sure you're covering all the cases need for using a seperate routing table. Blocking ping will not be helpful, but likely not the cause of your issues.

I'd start with the default firewall, then add the incoming VXLAN port as a accept on input in /ip/firewall/filter – since the benefit of VXLAN is that it's just one port that needs be opened. Then use routing rules to direct the direct to any desired routing table.

In my case, it's my last chance.
VXLAN is the last remaining one.
So I have no choise. I'll secure the vxlan later after solving my problem.
I have removed unconfigured or default parts like clock, timezone and other unused/unrelated parts from configuration.
I'm trying to connect two routers, one in my country and the other outside to route the traffic to outside.
As I said routers ping each other on the vxlan interface, so connection is up and there is no problem in firewall configuration. I just prevented ping from WAN.
I have access to the vxlan interface of router1 from router 2. I can ping it and send dns queries from router2 towards it.
Table had been created for vxlan and gateway has been set in the related routing table.(192.168.210.1)
Mangling has been setup to redirect the required traffic to the specified table by route mark but it doesn't work as expected.
Cs or C status of connections, show that traffic doesn't move as expected to the interface of vxlan.
There is no visible connections in firewall of router1 originated from router 2 except dns queries and test pings that are sent from the router2 itself from MAIN routing table and not the built VXLAN table.
Mangle rule that routes the traffic to the vxlan interface or the table itself doesn't do the job but I can't understand why.
Public IP on routers has been replaced by 1.1.1.1 and 2.2.2.2 in the above configuration due to security reasons.

Maybe this config is one tweak away – I have no idea. But it's not obvious what's going on here since there are things that are either mistakes or it's incomplete. There are secondary problems like speed would be slow over VXLAN because of the incorrect MTU.

You check the "Log" in your firewall rules and/or use sniffer or torch to figure it out. The packet flow diagram shows how all the packets will flow: https://help.mikrotik.com/docs/display/ ... n+RouterOS

As I said, I'd start fresh with the default firewall and allow VXLAN ports.
See https://help.mikrotik.com/docs/display/ ... d+Firewall

Then move on to routing using rules to direct traffic over WAN vs VXLAN tunnel:
https://help.mikrotik.com/docs/display/ ... cy+Routing

Thanks for your replies.
There is no problem with MTU and firewall.
The problem is beyond that.

I found something strange which implements a bug probably.

If I added a specific /32 route entry in VXLAN table, that host got reachable from clients But default route of table 0.0.0.0/0 didn't work!!!

I added 0.0.0.0/1 and it works!
Adding 128.0.0.0/1 cause the problem again after!
It seems there is a problem with 0.0.0.0/0 route to router1 VXLAN gateway.

Not that it may make any difference but try vers 7.9rc3 or rc3 vice 7.8.
For me 7.8 is flaky firmware

And something more strange!
When I add 128.0.0.0/1, the problem returns!!!
Only one of 128.0.0.1/1 or 0.0.0.0/1 should be active, else table loses it's functionality!!!
How can I report the bug?

I'm not sure there is a bug. It unclear what you're trying to do here. You have VLAN, PPP, and VXLAN on top of mangle routing table and /32 routing.

It's this kinda case where a diagram sure be helpful because you got a lot going on...

Thanks for advise.
I explained clearly and the config is basic.
I'm 100% sure it's a bug and I'm trying to find the exact destination network route which cause the routing table unstable.
I'm very thankful, you are caring about my problem.
I'm an administrator for about 20 years now and working around many network routers and firewalls.
Be sure I wouln't post here if I didn't know what I'm doing.
It's very clear what is the purpose of vxlan I'm doing and I found the problem in default route of custom table of VXLAN specified in configuration.
I'll post the working configuration of router2 to clear more

I'm not sure there is a bug. It unclear what you're trying to do here. You have VLAN, PPP, and VXLAN on top of mangle routing table and /32 routing.

It's this kinda case where a diagram sure be helpful because you got a lot going on...

This is complete configuration of router2, ports and some other configs has been changed since my first post but that's not the case. Just look at VXLAN table route

# apr/18/2023 20:35:47 by RouterOS 7.8
# software id =
#
/interface sstp-server
add comment="TK Link" name=sstp-in-TK user=TK
/interface list
add name=WAN
add name=VPN
add name=VXLAN
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec profile
set [ find default=yes ] dh-group=modp1024 enc-algorithm=3des
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=3des pfs-group=none
/ip pool
add name=pool-VPN ranges=192.168.0.11-192.168.0.99
add name=pool-IR ranges=192.168.1.11-192.168.1.99
add name=pool-VXLAN ranges=192.168.2.11-192.168.2.99
/interface vxlan
add local-address=103.215.xxx.243 mac-address=CE:6A:2B:3B:86:E6 name=vxlan-TK \
port=1000 vni=10 vrf=main vteps-ip-version=ipv4
/port
set 0 name=serial0
/ppp profile
set *0 only-one=yes use-ipv6=no
add change-tcp-mss=yes dns-server=217.218.155.155,217.218.127.127 \
local-address=192.168.1.1 name=Iran only-one=yes remote-address=pool-IR \
use-compression=yes use-encryption=yes use-ipv6=no
add dns-server=192.168.2.1 local-address=192.168.2.1 name=VXLAN only-one=yes \
remote-address=pool-VXLAN
set *FFFFFFFE dns-server=192.168.0.1,94.140.15.15 local-address=192.168.0.1 \
only-one=yes remote-address=pool-VPN use-compression=yes use-ipv6=no
/routing table
add disabled=no fib name=VPN
add fib name=VXLAN
/ip neighbor discovery-settings
set discover-interface-list=none
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface l2tp-server server
set allow-fast-path=yes enabled=yes l2tpv3-circuit-id=0 use-ipsec=required
/interface list member
add interface=ether1 list=WAN
add interface=sstp-in-TK list=VPN
add interface=vxlan-TK list=VXLAN
/interface ovpn-server server
set auth=sha1,md5
/interface sstp-server server
set certificate="VPN Server" default-profile=default-encryption enabled=yes \
keepalive-timeout=10
/interface vxlan vteps
add interface=vxlan-TK port=1000 remote-ip=185.211.xxx.10
/ip address
add address=103.215.xxx.243/24 interface=ether1 network=103.215.xxx.0
add address=192.168.210.2/30 interface=vxlan-TK network=192.168.210.0
/ip dns
set allow-remote-requests=yes cache-max-ttl=1d servers=\
94.140.15.15,94.140.14.14
/ip firewall address-list
add address=2.57.3.0/24 comment="Iran IPs" list=Iran
add address=217.219.192.0/21 list=Iran
add address=217.219.200.0/22 list=Iran
add address=217.219.204.0/24 list=Iran
add address=217.219.205.64/26 list=Iran
add address=217.219.205.128/25 list=Iran
add address=217.219.206.0/23 list=Iran
add address=217.219.208.0/20 list=Iran
add address=217.219.224.0/19 list=Iran
add address=185.211.xxx.10 comment="TK Link" list=Links
/ip firewall filter
add action=accept chain=input comment=Winbox dst-port=7353 in-interface-list=\
WAN protocol=tcp
add action=accept chain=input comment=SSTP dst-port=443 in-interface-list=WAN \
protocol=tcp
add action=accept chain=input comment=L2TP dst-port=500,4500,1701 \
in-interface-list=WAN protocol=udp
add action=accept chain=input comment=DNS dst-port=53 in-interface-list=!WAN \
protocol=udp
add action=accept chain=input comment=ICMP in-interface-list=!WAN protocol=\
icmp
add action=accept chain=input comment=VXLAN dst-port=1000 in-interface-list=\
WAN protocol=udp
add action=accept chain=input comment=Related connection-state=\
established,related,untracked
add action=drop chain=input comment=Drop
/ip firewall mangle
add action=mark-routing chain=prerouting comment="VPN Route" \
dst-address-list=!Iran dst-address-type=!local new-routing-mark=VPN \
passthrough=no src-address=192.168.0.0/24
add action=mark-routing chain=prerouting comment="VXLAN Mark" \
dst-address-list=!Iran dst-address-type=!local new-routing-mark=VXLAN \
passthrough=no src-address=192.168.2.11-192.168.2.99
/ip firewall nat
add action=masquerade chain=srcnat comment="VXLAN NAT" out-interface-list=\
VXLAN routing-mark=VXLAN
add action=masquerade chain=srcnat comment="VPN NAT" out-interface-list=VPN \
routing-mark=VPN
add action=masquerade chain=srcnat comment="Iran NAT" out-interface-list=WAN
add action=redirect chain=dstnat comment="80 to 443" disabled=yes dst-port=80 \
in-interface-list=WAN protocol=tcp to-ports=443
/ip proxy
set anonymous=yes port=444 src-address=0.0.0.0
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=103.215.xxx.254
add comment="VPN Gateway" disabled=no distance=1 dst-address=0.0.0.0/0 \
gateway=sstp-in-TK pref-src=0.0.0.0 routing-table=VPN scope=30 \
suppress-hw-offload=no target-scope=10
add comment=DNS1 disabled=no distance=1 dst-address=94.140.15.15/32 gateway=\
192.168.210.1 pref-src=0.0.0.0 routing-table=main scope=30 \
suppress-hw-offload=no target-scope=10
add comment=DNS2 disabled=no distance=1 dst-address=94.140.14.14/32 gateway=\
192.168.210.1 pref-src=0.0.0.0 routing-table=main scope=30 \
suppress-hw-offload=no target-scope=10
add comment="VPN VXLAN" disabled=no distance=1 dst-address=0.0.0.0/1 gateway=\
192.168.210.1 pref-src=0.0.0.0 routing-table=VXLAN scope=30 \
suppress-hw-offload=no target-scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
set winbox port=7353
set api-ssl disabled=yes
/ppp secret
add caller-id=185.211.xxx.10 comment="TK Link" local-address=192.168.200.2 \
name=TK remote-address=192.168.200.1 service=sstp
add name=Esfandiar profile=VXLAN
add name=Sahand profile=VXLAN
add name=Sahand2 profile=default-encryption
/system clock
set time-zone-name=Asia/Tehran
/system identity
set name=IR0
/system ntp client
set enabled=yes
/system ntp client servers
add address=0.asia.pool.ntp.org
add address=1.asia.pool.ntp.org
/system scheduler
add interval=1m name="Minutely sstpreset" on-event=sstpreset policy=\
ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
start-date=mar/28/2023 start-time=04:02:00
/system script
add dont-require-permissions=no name=sstpreset owner=admin policy=\
ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source="/\
ppp active\r\
\nremove number=[/ppp active find where service=\"sstp\" and encoding=\"\"\
]\r\
\n"

I've truncated the "Iran" IP list and secrets only to decrease the post volume and use "x" character to hide the public IPs for security reasons

More details on routing bug about v7.8:
As I mentioned here:
viewtopic.php?p=997060

There was a problem with routing when:
the gateway of a routing table is set to IP of VXLAN interface of opposite tunneled router
The problem shows itself when the traffic ignores the gateway route of table traffic routed to

Now I suggest that this bug happens whenever I use IP instead of interface in any routing table on gateway (0.0.0.0/0) even on other protocols such as IPIPv6 and GREv6.

The bug is confirmed on these protocols by me but I think it's more generic and has something to do with using IP as default route.

So it's a routing table bug around using IP instead of interface on default route and not a protocol specified bug.
No problem with routes, other than 0.0.0.0/0

The bug occures whenever IP of opposite router interface destined as default gateway instead of interface with the address itself.

To simulate the problem:
1. Make a tunnel based on VXLAN or IPIPv6 or GREv6 between two routers.
2. Assign an IP address to tunnel interfaces on each side and test the ping of other side IP.
3. Add a NAT rule on router2 with internet interface as outgoing interface if you want to test with internet hosts.
4. Make a table on router1 with gateway (0.0.0.0/0) of opposite router tunnel interface IP.
5. Mangle some clients and mark it with built routing table.
6. Set proper NAT rule for the outgoing interface of tunnel on router1 or make the returning route on the opposite router.
7. Test it on clients you have mangled. Try to ping the router2 IP of tunnel interface. Timeout happens. Try to ping an internet host and you get the same result.
8. To confirm the bug more obviously, edit the built table 0.0.0.0/0 route to a smaller route like 0.0.0.0/1 or even a /32 route.
9. Now test that destined subnet route or/32 destination from the same clients and it will be accessible. And even more strange, now you can ping router2 tunneling interface IP from clients!
10. Additionally you can edit the same route and assign appropriate interface as gateway instead of IP (if possible in tested protocol), and even 0.0.0.0/0 will work.

Fair enough. You could try disabling fast-path to see if that fixes. Maybe you should open a ticket at help.mikrotik.com. If you include a supout.rif file with your report it will likely get more attention.

Out of curiosity, have you looked at using ZeroTier for your national firewall situation? I suggest this since it's pretty aggressive of finding a path out, and you just need a ZeroTier controller instead of VTEP on the "open internet" side.

Fair enough. You could try disabling fast-path to see if that fixes. Maybe you should open a ticket at help.mikrotik.com. If you include a supout.rif file with your report it will likely get more attention.

Out of curiosity, have you looked at using ZeroTier for your national firewall situation? I suggest this since it's pretty aggressive of finding a path out, and you just need a ZeroTier controller instead of VTEP on the "open internet" side.

I have tried disabling fast path as I thought it should change the routing path, but it wasn't the case and I reported it to support.

Send me url to earn info about ZeroTier, it may help. Thanks.


National firewall has restricted most of tunneling protocols such as GRE and L2TP. About 90% of famous websites and all of social media are blocked.
There is no facebook, instagram, twitter,...
No access to news agents are allowed.
Because af bad economic situation of ordinary people, mikrotik low end products like hap lite is popular.
No enterprise solutions could be implemented due to regulations.
Admins here try to make free internet avaible to people as low price as possible. For example average monthly cost of home ADSL connection is about 1-2$ due to usage but people should pay 2-4$ more monthly, to obtain VPN for free access. And this is alot here for ordinary person who earns 150$ to 200$ a month at most.
So solutions over mikrotik low cost hardware is welcomed.

It's better not to mention the protocols and technical details because it increases the probility of blocking them.

But if you see something unusual such as VXLAN over public IP, that's not done to connect two LANs across internet for serious inter domain connections. I do that to obtain a NAT on out of country router. No sensitive data are sent on these links just ordinary people web browsing. And I limit the protocols in firewalls on both sides to opposite IP only and that's enough for the proposal security.

I found these on ZeroTier official website documents:

Don't use "symmetric" NAT. Use "full cone" or "port restricted cone" NAT. Symmetric NAT is extremely hostile to peer to peer traffic and will degrade VoIP, video chat, games, WebRTC, and many other protocols as well as ZeroTier.
No more than one layer of NAT should be present between ZeroTier endpoints and the Internet. Multiple layers of NAT introduce connection instability due to chaotic interactions between states and behaviors at different levels. No Double NAT.
NATs should have a port mapping or connection timeout no shorter than 60 seconds.
Place no more than about 16,000 devices behind each NAT-managed external IP address to ensure that each device can map a sufficient number of ports.

In my country most of ISPs which represent their services on LTE technology, dont provide the clients with a valid internet public IP and give them an Intranet IP which is natted at least once before reaching internet on the ISP net. Besides that, it's not possible to even use that interanet IP on a client either so another level of NAT happens in client LAN towards the modem, ISP has privided. This is the case of millions, so.

There is little to no restrictions in internal country connections so I rent some servers with valid Internet public IPs and make clients tunnel to that server with standard protocols, natively most of OSes support such as L2TP on iOS and SSTP on Windows, then I route the traffic to an outside if country router, by means of new v7 protocols which are not blocked.

These protocols are blocked from/toward international servers:
OpenVPN
SSTP
L2TP
PPTP
GRE
IPIP
EoIP
Wireguard
and IPv6 is not supported by ISPs or even datacenters here.

Is there anyway I use IPv6 without ISP support to gain access to free internet?

Thanks for the explanation.

and IPv6 is not supported by ISPs or even datacenters here.
Is there anyway I use IPv6 without ISP support to gain access to free internet?

Haven't used it in years. But Hurricane Electric runs IPv6 tunnel broker:
https://tunnelbroker.net

But I'd have to imagine that be blocked, but maybe not.

Hi esipoko,
Have you find any solution for the problem of VxLAN tunnel traffic?

Hi @esipoko,

I can confirm the bug you explained exists as of 7.12.1. Described it here and reported to the support (SUP-136716).