Hello community, I’ve been struggling to figure this problem for a few days without any success.
The setup is next - a linux computer with static IP is connected to the Mikrotik, that acts as a wireless adapter (in station-pseudobridge mode, also static IP).
As for AP - matters not, it can be any device, another Mikrotik, Huawei or Xiaomi, acting as a dumb AP. The real routing is done by the other Mikrotik.
The issues are next:
- Sometimes there is a traffic loss to linux computer (best observable on ICMP loss, around 5-10 packets), that might be enough to break an SSH session to it, or to cause any other service interruption. Mikrotik is reachable during this time.
- Quite often there is a traffic loss to a Mikrotik acting as a wireless client. During this time, linux computer is reachable.
Both issues are valid, no matter from where both devices are pinged - from any computer, or from core Mikrotik itself.
The setup is acting quite unstable, going from case 1 to case 2 several times, without no reason.
Tried different devices, different distance between AP and client - this happens even at close proximity (AP lies down next to client).
Literally this setup works on the different location, where (presumingly) routing is not done via Mikrotik, or configured differently (there is no opportunity to figure out how - call it a black box).
What was attempted:
- playing with wireless modes (idea was as if it caused by interference on the first glance)
- playing with queues on mikrotik client
- playing with diffrerent versions of router os
- playing with bridge mode (with and without RSTP on both core and client Mikrotik)
- playing with bridge hardware offloading on used ports on client Mikrotik device
- playing with broadcast\multicast flood on client Mikrotik device
- playing with firewall rules on the core Mikrotik
- playing with ARP mode both core and client Mikrotik
Speaking of configuration:
Linux, clean OOB Ubuntu 20.04, with static netplan config
Case 1
network:
ethernets:
enp2s0:
dhcp4: no
dhcp6: no
addresses: [10.11.0.114/25,172.17.17.252/24]
gateway4: 10.11.0.1
nameservers:
addresses: [8.8.8.8]
version: 2
network:
ethernets:
enp2s0:
dhcp4: no
dhcp6: no
addresses: [10.233.17.114/25,172.17.17.252/24]
gateway4: 10.233.17.1
nameservers:
addresses: [8.8.8.8]
version: 2
Mikrotik acting as a wirelss client
/interface bridge
add name=lan protocol-mode=none
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
add authentication-types=wpa2-psk,wpa2-eap mode=dynamic-keys name=\
security_profile supplicant-identity=MikroTik wpa2-pre-shared-key=\
*******
/interface wireless
set [ find default-name=wlan1 ] antenna-gain=0 band=2ghz-b/g/n channel-width=\
20/40mhz-Ce country=no_country_set disabled=no distance=indoors \
frequency=auto frequency-mode=manual-txpower mode=station-pseudobridge \
security-profile=security_profile ssid=ap_name station-roaming=enabled tx-power=\
21 tx-power-mode=all-rates-fixed wireless-protocol=802.11 wps-mode=\
disabled
/queue type
set 1 pfifo-limit=500
/interface bridge port
add bridge=lan interface=wlan1
add bridge=lan broadcast-flood=no interface=ether2 unknown-multicast-flood=no \
unknown-unicast-flood=no
add bridge=lan broadcast-flood=no hw=no interface=ether4 \
unknown-multicast-flood=no unknown-unicast-flood=no
add bridge=lan fast-leave=yes interface=ether3
add bridge=lan interface=ether1
add bridge=lan interface=ether6
add bridge=lan interface=ether7
add bridge=lan interface=ether8
add bridge=lan interface=ether9
add bridge=lan interface=ether10
add bridge=lan interface=ether5
/ip address
add address=10.11.0.115/25 interface=lan network=10.11.0.0
add address=172.17.17.254/24 interface=lan network=172.17.17.0
/ip dhcp-client
add interface=lan
/ip dhcp-server network
add address=172.17.17.0/24 gateway=172.17.17.254
/ip dns
set allow-remote-requests=yes
/ip firewall filter
add action=drop chain=forward out-interface=lan src-address=172.17.17.0/24
/ip route
add check-gateway=ping distance=1 gateway=10.11.0.1
As for core Mikrotik (the one handling routing), at least 2 different configurations were tested, and issue can be reproduced on both:
Core Mikrotik case 1
/interface bridge
add admin-mac=4C:5E:0C:2C:8B:BE auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=sfp1 ] disabled=yes
/interface ethernet switch port
set 6 vlan-mode=fallback
set 7 vlan-mode=fallback
set 8 vlan-mode=fallback
set 9 vlan-mode=fallback
set 10 vlan-mode=fallback
set 12 vlan-mode=fallback
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] authentication-types=wpa2-psk,wpa2-eap mode=\
dynamic-keys supplicant-identity=MikroTik
add authentication-types=wpa2-psk mode=dynamic-keys name=\
wlan1-charlie0-1-repeater supplicant-identity=MikroTik
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-b/g/n default-authentication=no \
frequency=2447 mode=station-pseudobridge security-profile=\
wlan1-charlie0-1-repeater ssid=charlie0-1
add mac-address=4E:5E:0C:2C:8B:C7 master-interface=wlan1 name=wlan2 \
security-profile=wlan1-charlie0-1-repeater ssid=charlie0-1
/ip dhcp-server option
add code=121 name=route_10.233.17.0 value=0x00000D0AA91001180A0A000001
/ip pool
add name=dhcp ranges=10.0.0.2-10.0.0.254
/ip dhcp-server
add address-pool=dhcp interface=bridge name=defconf
/port
set 0 name=serial0
/interface ppp-client
add add-default-route=no comment=ISP2 dial-on-demand=no disabled=no \
modem-init=ATZ name=ppp-out1 phone=*99***3# pin=**** port=usb2 \
use-peer-dns=no
/ppp profile
add change-tcp-mss=yes name=ovpn-client only-one=yes use-encryption=yes \
use-ipv6=no use-mpls=no
/routing table
add fib name=vpn_traffic
/interface bridge port
add bridge=bridge comment=defconf ingress-filtering=no interface=ether2
add bridge=bridge comment=defconf ingress-filtering=no interface=ether3
add bridge=bridge comment=defconf ingress-filtering=no interface=ether4
add bridge=bridge comment=defconf ingress-filtering=no interface=ether5
add bridge=bridge comment=defconf ingress-filtering=no interface=ether6
add bridge=bridge comment=defconf ingress-filtering=no interface=ether7
add bridge=bridge comment=defconf ingress-filtering=no interface=ether8
add bridge=bridge comment=defconf ingress-filtering=no interface=ether9
add bridge=bridge comment=defconf ingress-filtering=no interface=ether10
add bridge=bridge comment=defconf ingress-filtering=no interface=sfp1
add bridge=bridge interface=wlan2
/ip neighbor discovery-settings
set discover-interface-list=LAN
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set disable-ipv6=yes forward=no max-neighbor-entries=8192
/interface detect-internet
set detect-interface-list=all lan-interface-list=LAN wan-interface-list=WAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
/interface lte settings
set mode=serial
/interface ovpn-server server
set auth=sha1,md5
/interface wireless connect-list
add comment=wlan1-repeater interface=wlan1 mac-address=28:D1:27:1A:68:9E
/ip address
add address=10.0.0.1/24 comment=defconf interface=bridge network=10.0.0.0
add address=10.11.0.1/24 interface=bridge network=10.11.0.0
/ip dhcp-client
add comment=defconf interface=ether1 use-peer-dns=no
/ip dhcp-server network
add address=10.0.0.0/24 comment=defconf dns-server=8.8.8.8 gateway=10.0.0.1 \
netmask=24
/ip dns
set servers=8.8.8.8
/ip dns static
add address=10.0.0.1 name=router.lan
/ip firewall address-list
add address=10.0.0.0/8 list=local_traffic
/ip firewall filter
add action=accept chain=input comment=\
"defconf: accept established,related,untracked" connection-state=\
established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
"defconf: accept established,related, untracked" connection-state=\
established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
connection-state=invalid
add action=drop chain=forward comment=\
"defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
ipsec-policy=out,none out-interface-list=WAN
add action=masquerade chain=srcnat out-interface=ppp-out1
add action=src-nat chain=srcnat protocol=udp src-port=123 to-ports=12300
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
/ip route
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=ppp-out1 pref-src="" \
routing-table=main suppress-hw-offload=no
/ip service
set telnet disabled=yes
set api-ssl disabled=yes
/ip smb
set allow-guests=no
/ip ssh
set allow-none-crypto=yes forwarding-enabled=remote
/lcd
set backlight-timeout=30s read-only-mode=yes touch-screen=disabled
/lcd pin
set hide-pin-number=yes pin-number=9262
/lcd interface pages
set 0 interfaces=wlan1
/system clock
set time-zone-name=Europe/Kiev
/system identity
set name=78
/system logging
set 3 action=memory
add action=echo disabled=yes topics=wireless
add action=echo disabled=yes topics=debug
add action=echo topics=ntp
add action=echo topics=script
/system ntp client
set enabled=yes mode=multicast
/system ntp client servers
add address=time.cloudflare.com
/tool netwatch
add disabled=no down-script="/ip route set [find where comment=\"WAN1\"] dista\
nce=3;\
\n:log warning \"WAN1 down, switching to WAN2\"" host=1.1.1.1 \
test-script="" type=simple up-script="/ip route set [find where comment=\"\
WAN1\"] distance=1;\
\n:log warning \"WAN1 restored\""
Core Mikrotik case 2
/interface bridge
add arp=proxy-arp name=bridge1
/interface ethernet
set [ find default-name=ether1 ] mac-address=08:55:31:FC:53:D7
set [ find default-name=ether5 ] mac-address=08:55:31:FC:53:E5
set [ find default-name=sfp1 ] mac-address=08:55:31:FC:53:D3
/interface sstp-client
add authentication=mschap1,mschap2 connect-to=x.y.a.b http-proxy=\
0.0.0.0 name=sstp-legacy user=users \
verify-server-address-from-certificate=no
/interface wireguard
add listen-port=41194 mtu=1420 name=vpn00-wg-client
/interface list
add name=ISP
add name=LAN
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] authentication-types=wpa2-psk mode=dynamic-keys \
supplicant-identity=MikroTik
/ip dhcp-server option
add code=66 name=option66 value="s'172.16.88.93'"
add code=67 name=option67 value="'ipxe/netboot.xyz.efi'"
add code=121 name=route_1023317024_via_10233171 value=\
0x0000790AE911010A0AE91101
add code=121 name=route_121_original value=0x000AE91101180AE9110AE91101
add code=121 name=route_1023317024_via_17216881 value=\
0x00089AD91101180AC1688101
/ip dhcp-server option sets
add name=set-pxe options=option66,option67
/ip pool
add name=dhcp_pool0 ranges=172.16.88.2-172.16.88.128
/ip dhcp-server
add address-pool=dhcp_pool0 interface=bridge1 lease-time=5m name=dhcp1
/port
set 0 name=serial0
/ppp profile
add name=ovpn-client only-one=yes use-compression=no use-encryption=yes \
use-ipv6=no use-mpls=no
add name=ovpn-bare only-one=yes use-compression=no use-encryption=no \
use-ipv6=no use-mpls=no
/routing table
add fib name=ISP1-route
add fib name=ISP2-route
add fib name=ISP1
add fib name=ISP2
add fib name=via-ovpn
/interface bridge filter
add action=drop chain=forward in-interface-list=LAN ip-protocol=udp \
mac-protocol=ip src-port=67
/interface bridge port
add bridge=bridge1 ingress-filtering=no interface=ether3
add bridge=bridge1 ingress-filtering=no interface=ether4
add bridge=bridge1 ingress-filtering=no interface=ether6
add bridge=bridge1 ingress-filtering=no interface=ether7
add bridge=bridge1 ingress-filtering=no interface=ether8
add bridge=bridge1 ingress-filtering=no interface=ether9
add bridge=bridge1 ingress-filtering=no interface=ether10
add bridge=bridge1 interface=ether5
add bridge=bridge1 interface=wlan1
add bridge=bridge1 interface=sfp1
add bridge=bridge1 interface=ether2
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface detect-internet
set detect-interface-list=static
/interface list member
add interface=ether1 list=ISP
add interface=sfp1 list=LAN
add interface=ether2 list=LAN
add interface=ether10 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=LAN
add interface=ether5 list=LAN
add interface=ether6 list=LAN
add interface=ether7 list=LAN
add interface=ether8 list=LAN
add interface=ether9 list=LAN
add interface=wlan1 list=LAN
/interface ovpn-server server
set auth=sha1,md5
/interface wireguard peers
add allowed-address=0.0.0.0/0 endpoint-address=x.x.x.x endpoint-port=\
41194 interface=vpn00-wg-client persistent-keepalive=10s public-key=\
"*******"
/ip address
add address=172.16.88.1/24 interface=bridge1 network=172.16.88.0
add address=10.233.17.1/24 interface=bridge1 network=10.233.17.0
add address=192.168.20.2/24 interface=vpn00-wg-client network=192.168.20.0
/ip dhcp-client
add interface=ether1
/ip dhcp-server network
add address=10.233.17.0/24 boot-file-name=ipxe/netboot.xyz.kpxe dns-server=\
8.8.8.8,1.1.1.1 gateway=10.233.17.1 netmask=24 next-server=172.16.88.93
add address=172.16.88.0/24 boot-file-name=ipxe/netboot.xyz.efi dns-server=\
8.8.8.8,1.1.1.1 gateway=172.16.88.1 netmask=24 next-server=172.16.88.93
/ip dns
set servers=8.8.8.8,1.1.1.1
/ip firewall filter
add action=accept chain=input
add action=accept chain=input protocol=gre
add action=accept chain=forward
add action=drop chain=forward src-mac-address=60:E3:27:26:03:E7
add action=drop chain=input src-mac-address=60:E3:27:26:03:E7
/ip firewall mangle
add action=mark-connection chain=input comment=ISP1 disabled=yes \
in-interface=sfp1 new-connection-mark=isp1-conn passthrough=yes
add action=mark-routing chain=output connection-mark=isp1-conn disabled=yes \
new-routing-mark=ISP1-route passthrough=no
add action=mark-connection chain=input comment=ISP2 disabled=yes \
in-interface=ether2 new-connection-mark=isp2-conn passthrough=no
add action=mark-routing chain=output connection-mark=isp2-conn disabled=yes \
new-routing-mark=ISP2-route passthrough=yes
add action=mark-routing chain=prerouting disabled=yes new-routing-mark=ISP1 \
passthrough=yes src-address=192.168.1.0/24
add action=mark-routing chain=prerouting disabled=yes new-routing-mark=ISP2 \
passthrough=yes src-address=192.168.1.0/24
/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=ISP
add action=masquerade chain=srcnat disabled=yes out-interface=bridge1
add action=masquerade chain=srcnat src-address=192.168.20.0/24
add action=dst-nat chain=dstnat dst-port=8001 in-interface-list=all protocol=\
tcp to-addresses=172.16.88.20 to-ports=8001
add action=masquerade chain=srcnat dst-address=172.16.88.20 dst-port=8001 \
protocol=tcp src-address=172.16.88.0/24
/ip route
add disabled=no dst-address=192.168.248.0/24 gateway=192.168.159.1 \
routing-table=main suppress-hw-offload=no
add disabled=no distance=1 dst-address=192.168.20.0/24 gateway=192.168.20.1 \
pref-src="" routing-table=main suppress-hw-offload=no
add disabled=no distance=1 dst-address=11.10.0.0/24 gateway=192.168.20.1 \
pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
target-scope=10
add disabled=yes distance=1 dst-address=11.233.17.0/24 gateway=192.168.20.1 \
pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
target-scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www address="10.233.17.0/24,172.16.88.0/24,213.109.137.6/32,11.10.0.0/24,1\
92.168.20.0/24"
set ssh address="10.233.17.0/24,172.16.88.0/24,213.109.137.6/32,11.10.0.0/24,1\
92.168.20.0/24"
set api address=172.16.88.0/24
set winbox address=172.16.88.0/24,213.109.137.6/32 disabled=yes
set api-ssl address=172.16.88.0/24
/lcd interface pages
set 0 interfaces="sfp1,ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8\
,ether9,ether10"
/routing rule
add action=lookup src-address=172.16.88.0/24 table=via-ovpn
/system clock
set time-zone-name=Europe/Kyiv
/system identity
set name=DeusRoboticsCore
/system logging
add action=echo disabled=yes topics=wireguard
/tool netwatch
add disabled=yes down-script="/ip route set [find where comment=\"ISP5\"] dist\
ance=3;\
\n:log warning \"sfp1 has failed, switching to ether2\"" host=1.1.1.1 \
interval=15s test-script="" timeout=1s type=simple up-script="/ip route se\
t [find where comment=\"ISP5\"] distance=1;\
\n:log warning \"sfp1 restored\""
P.S. Going with Mikrotik AP, and using station-bridge mode is not an option.
P.S2. MAC addresses on any port on mikrotik and ethenet adapter of linux computer are different.
Any ideas on what’s (might be) wrong here are appreciated.