Community discussions

MikroTik App
 
gdanov
Member Candidate
Member Candidate
Topic Author
Posts: 150
Joined: Thu Jan 17, 2019 1:10 pm

WG tunnel UDP is 5x faster than TCP

Fri Dec 30, 2022 12:23 pm

I'm testing WG tunnel between two locations in different countries. Mean ping is 50ms. With MT's bandwidth test i get 100Mbps UDP and 15-20Mbps TCP. No obvious problems like dropped packets. In the tested direction the client's downlink is 200Mbps so we are limited by the server's uplink.

configs...

client
# dec/30/2022 <mac address> by RouterOS 7.6
# software id = 1FCH-PMGC
#
# model = RBM33G
# serial number = whatever
/interface bridge
add fast-forward=no name=bridge
add name=bridge-580
/interface lte
set [ find default-name=lte1 ] allow-roaming=no band=3,7 disabled=yes network-mode=lte
/interface ipip
add name=ipip-tunnel1 remote-address=xx.xx.xx.xx
/interface wireguard
add listen-port=51821 mtu=1420 name=mineWG
add listen-port=13231 mtu=1420 name=wireguard-pp
/interface vlan
add interface=ether2 name=ether1-580 vlan-id=580
/interface list
add name=WAN
add name=LAN
/interface lte apn
set [ find default=yes ] apn=drei.at authentication=pap default-route-distance=1 ip-type=ipv4
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
add authentication-types=wpa2-psk eap-methods="" mode=dynamic-keys name=psychoiho supplicant-identity=MikroTik
/ip pool
add name=dhcp ranges=xx.xx.xx.xx-xx.xx.xx.xx
add name=native ranges=xx.xx.xx.xx-xx.xx.xx.xx
add name=vlan-05 ranges=xx.xx.xx.xx-xx.xx.xx.xx
add name=vlan-99 ranges=xx.xx.xx.xx-xx.xx.xx.xx
add name=pool1 ranges=xx.xx.xx.xx-xx.xx.xx.xx
/ip dhcp-server
add add-arp=yes address-pool=native interface=bridge lease-time=1h name=native
/port
set 0 name=serial0
set 1 name=serial1
/queue type
add fq-codel-ecn=no fq-codel-interval=50ms fq-codel-limit=1024 fq-codel-quantum=300 fq-codel-target=20ms kind=fq-codel name="fq codel"
add cake-ack-filter=filter cake-autorate-ingress=yes cake-diffserv=diffserv4 cake-flowmode=flows cake-nat=yes cake-overhead=25 cake-rtt=30ms cake-rtt-scheme=\
    regional kind=cake name=cake
add kind=pfifo name=default-big pfifo-limit=500
/queue simple
add burst-limit=55M/999M burst-threshold=50M/900M burst-time=5s/5s disabled=yes dst=ether1 limit-at=0/900M max-limit=50M/900M name=queue1 queue=\
    "fq codel/default-big" target=xx.xx.xx.xx/16 total-queue=default
/queue tree
add burst-limit=45M burst-threshold=35M burst-time=1s limit-at=30M max-limit=35M name=egress packet-mark=no-mark parent=ether1 queue="fq codel"
add burst-limit=250M burst-threshold=225M burst-time=1s disabled=yes limit-at=200M max-limit=220M name=ingress packet-mark=no-mark parent=bridge queue="fq codel"
/routing bgp template
set default disabled=no output.network=bgp-networks
/routing table
add fib name=""
add disabled=no fib name=pleven
/system logging action
set 3 remote=xx.xx.xx.xx
/caps-man manager
set enabled=yes
/interface bridge filter
# no interface
add action=drop chain=forward comment=DHCP dst-port=67-68 in-interface=*1C ip-protocol=udp mac-protocol=ip
add action=drop chain=input comment="isolate vlan 580 from xx.xx.xx.xx/24" disabled=yes dst-address=xx.xx.xx.xx/24 in-bridge=bridge-580 mac-protocol=ip
/interface bridge port
add bridge=bridge ingress-filtering=no interface=ether2
add bridge=bridge ingress-filtering=no interface=*9
add bridge=bridge-580 interface=ether1-580
add bridge=bridge interface=ether3
/ip firewall connection tracking
set enabled=yes tcp-established-timeout=1h
/ip neighbor discovery-settings
set discover-interface-list=LAN
/ip settings
set accept-redirects=yes accept-source-route=yes rp-filter=loose
/ipv6 settings
set accept-redirects=no disable-ipv6=yes forward=no max-neighbor-entries=8192
/interface detect-internet
set lan-interface-list=LAN wan-interface-list=WAN
/interface list member
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether1 list=WAN
add interface=mineWG list=LAN
add interface=bridge list=LAN
add interface=wireguard-pp list=WAN
add interface=lte1 list=WAN
/interface lte settings
set mode=mbim
/interface ovpn-server server
set auth=sha1,md5
/interface wireguard peers
add allowed-address=xx.xx.xx.xx/32 comment=samsunga interface=mineWG public-key="0dmvOJCac4UpbCsonRZe0cWHdz9Z4yDNjLcJld9+yXQ="
add allowed-address=xx.xx.xx.xx/32 comment=hackmack interface=mineWG public-key="hGxvXEGf50vuADtMpE/us8t8crx2nT9a0CdSE1oj9HQ="
add allowed-address=xx.xx.xx.xx/32,xx.xx.xx.xx/24 comment="hetzner danov.biz " endpoint-address=danov.biz endpoint-port=51871 interface=mineWG persistent-keepalive=20s \
    public-key="uKmLcRMJT1YOUCgsds8v2KmPSY+ccrS2scR3LCUvekA="
add allowed-address=xx.xx.xx.xx/32 comment="15 mbpro" interface=mineWG public-key="FqJ390Owg0BnH8HFOc1Z9ipW2liWaJJ5zWU+voeu5yw="
add allowed-address=xx.xx.xx.xx/16 comment=front-balancer-2 endpoint-address=xx.xx.xx.xx endpoint-port=51820 interface=wireguard-pp persistent-keepalive=20s \
    public-key="PqZ49a4pi9ENsRCOKVvmGk8A4Gargeh0ALV2iKA8Fjs="
add allowed-address=xx.xx.xx.xx/16 comment=front-balancer disabled=yes endpoint-address=xx.xx.xx.xx endpoint-port=51820 interface=wireguard-pp \
    persistent-keepalive=15s public-key="PpFrmR0QnOsgUu0TK1fiWj0Qd4AA1lQGFfq1piBjlwg="
add allowed-address=xx.xx.xx.xx/32 comment=Fabrikata endpoint-address=c5d10de057de.sn.mynetname.net endpoint-port=51821 interface=mineWG persistent-keepalive=20s \
    public-key="HmnE7+Pkq+BVyFKh6vKZSdEw5qQruAAyGIgzVfvcOks="
add allowed-address=xx.xx.xx.xx/32,xx.xx.xx.xx/24 comment="pleven home" endpoint-address=xx.xx.xx.xx endpoint-port=51821 interface=mineWG persistent-keepalive=\
    20s public-key="mu+MA/IsacDGyubGZzAT7EO/1M18PyUA9BmIGxi3fxA="
add allowed-address=xx.xx.xx.xx/32 comment=iPhone interface=mineWG public-key="3tdsqf3R1qpcZsgv4ggi3FS/Oz/wBjg2JPt09jMdy3k="
add allowed-address=xx.xx.xx.xx/32 comment=Bodganchovci endpoint-address=a2fd0cb1dad7.sn.mynetname.net endpoint-port=51871 interface=mineWG persistent-keepalive=15s \
    public-key="YfZFSu0TVNth81L80w3MyH2Jg4joVh1MEiCDIIVvaVc="
add allowed-address=xx.xx.xx.xx/32 comment=shinobi endpoint-address=xx.xx.xx.xx endpoint-port=51871 interface=mineWG public-key=\
    "dI0H82rOyFQtRq+5Nj6exoV7mIZ8Pjtt/1JkICI8r2c="
/interface wireless access-list
add comment="j work" mac-address=<mac address> vlan-id=99 vlan-mode=use-tag
add allow-signal-out-of-range=5s disabled=yes signal-range=-75..120 vlan-mode=no-tag
add authentication=no disabled=yes forwarding=no signal-range=-120..-75 vlan-mode=no-tag
/interface wireless connect-list
add disabled=yes interface=*10 security-profile=psychoiho
add disabled=yes interface=*12 security-profile=psychoiho
/interface wireless sniffer
set streaming-enabled=yes streaming-server=xx.xx.xx.xx
/interface wireless snooper
set multiple-channels=no receive-errors=yes
/ip address
add address=xx.xx.xx.xx/24 interface=bridge network=xx.xx.xx.xx
add address=xx.xx.xx.xx/24 interface=mineWG network=xx.xx.xx.xx
add address=xx.xx.xx.xx interface=wireguard-pp network=xx.xx.xx.xx
add address=xx.xx.xx.xx/24 interface=ipip-tunnel1 network=xx.xx.xx.xx
add address=xx.xx.xx.xx/24 interface=bridge-580 network=xx.xx.xx.xx
add address=xx.xx.xx.xx/24 interface=ether1 network=xx.xx.xx.xx
add address=xx.xx.xx.xx/24 interface=bridge network=xx.xx.xx.xx
/ip cloud
set ddns-enabled=yes ddns-update-interval=1m
/ip dhcp-client
add disabled=yes interface=bridge
add interface=ether1 use-peer-dns=no
/ip dhcp-server lease
add address=xx.xx.xx.xx client-id=<mac address> disabled=yes mac-address=<mac address> server=native
add address=xx.xx.xx.xx client-id=<mac address> mac-address=<mac address> server=native
add address=xx.xx.xx.xx disabled=yes mac-address=<mac address> server=native
add address=xx.xx.xx.xx client-id=<mac address> comment="Eli Kindle" mac-address=<mac address> server=native
add address=xx.xx.xx.xx comment="Joro Kindle" mac-address=<mac address> server=native use-src-mac=yes
add address=xx.xx.xx.xx client-id=<mac address> comment="Jeny Kindle" mac-address=<mac address> server=native
add address=xx.xx.xx.xx client-id=<mac address> mac-address=<mac address> server=native
add address=xx.xx.xx.xx client-id=<mac address> mac-address=<mac address> server=native
add address=xx.xx.xx.xx client-id=<mac address> mac-address=<mac address> server=native
add address=pool1 client-id=<mac address> mac-address=<mac address> server=native
add address=pool1 client-id=<mac address> mac-address=<mac address> server=native
add address=pool1 client-id=<mac address> mac-address=<mac address> server=native
add address=xx.xx.xx.xx client-id=<mac address> mac-address=<mac address> server=native
add address=xx.xx.xx.xx client-id=<mac address> mac-address=<mac address> server=native
/ip dhcp-server network
add address=xx.xx.xx.xx/24 comment="was xx.xx.xx.xx" dns-server=xx.xx.xx.xx gateway=xx.xx.xx.xx ntp-server=xx.xx.xx.xx
add address=xx.xx.xx.xx/24 comment="no adblock" dns-server=xx.xx.xx.xx gateway=xx.xx.xx.xx netmask=24
add address=xx.xx.xx.xx/24 dns-server=xx.xx.xx.xx,xx.xx.xx.xx,xx.xx.xx.xx gateway=xx.xx.xx.xx netmask=24
add address=xx.xx.xx.xx/24 dns-server=xx.xx.xx.xx,xx.xx.xx.xx,xx.xx.xx.xx gateway=xx.xx.xx.xx
/ip dns
set allow-remote-requests=yes cache-size=40096KiB max-concurrent-queries=500 max-concurrent-tcp-sessions=50 query-server-timeout=3s servers=xx.xx.xx.xx \
    verify-doh-cert=yes
/ip dns static
add address=xx.xx.xx.xx name=mx.danov.biz
add forward-to=xx.xx.xx.xx regexp="^.*\\.weint\$" type=FWD
add address=xx.xx.xx.xx name=cloudflare-dns.com
/ip firewall address-list
add address=xx.xx.xx.xx/8 comment=RFC6890 list=not_in_internet
add address=xx.xx.xx.xx/12 comment=RFC6890 list=not_in_internet
add address=xx.xx.xx.xx/16 comment=RFC6890 list=not_in_internet
add address=xx.xx.xx.xx/8 comment=RFC6890 list=not_in_internet
add address=xx.xx.xx.xx/16 comment=RFC6890 list=not_in_internet
add address=xx.xx.xx.xx/8 comment=RFC6890 list=not_in_internet
add address=xx.xx.xx.xx/4 comment=Multicast list=not_in_internet
add address=xx.xx.xx.xx/15 comment=RFC6890 list=not_in_internet
add address=xx.xx.xx.xx/24 comment=RFC6890 list=not_in_internet
add address=xx.xx.xx.xx/24 comment=RFC6890 list=not_in_internet
add address=xx.xx.xx.xx/24 comment=RFC6890 list=not_in_internet
add address=xx.xx.xx.xx/24 comment=RFC6890 list=not_in_internet
add address=xx.xx.xx.xx/10 comment=RFC6890 list=not_in_internet
add address=xx.xx.xx.xx/4 comment=RFC6890 list=not_in_internet
add address=xx.xx.xx.xx/24 comment="6to4 relay Anycast [RFC 3068]" list=not_in_internet
add address=xx.xx.xx.xx/16 list="my LAN"
add address=a2fd0c77e0ed.sn.mynetname.net list="WAN addresses"
/ip firewall filter
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes
add action=accept chain=forward connection-nat-state=srcnat connection-state=established,related in-interface=bridge-580
add action=reject chain=forward connection-state="" in-interface=bridge-580 out-interface=!ipip-tunnel1 reject-with=icmp-network-unreachable
add action=accept chain=forward comment="Established, Related" connection-state=established,related,untracked
add action=accept chain=input dst-port=51821,13231 in-interface-list=WAN protocol=udp
add action=accept chain=input disabled=yes in-interface-list=WAN protocol=icmp
add action=drop chain=forward comment="Drop invalid" connection-state=invalid log-prefix="FW Invalid Drop"
add action=drop chain=forward comment="Drop incoming packets that are not NATted" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN log=yes \
    log-prefix=!NAT:
add action=drop chain=forward comment="Drop incoming from internet which is not public IP" disabled=yes in-interface-list=WAN log=yes log-prefix=!public \
    src-address-list=not_in_internet
add action=drop chain=forward comment="Drop packets from LAN that do not have LAN IP" disabled=yes in-interface-list=LAN src-address-list="my LAN"
add action=drop chain=input dst-port=8080,53,1080,80,443,8291 in-interface-list=WAN protocol=tcp
add action=drop chain=input connection-state=!established,related,untracked in-interface-list=WAN log-prefix="\?\?"
/ip firewall mangle
add action=change-ttl chain=postrouting comment="avoid throttling\?" disabled=yes new-ttl=<mac address> out-interface=lte1 passthrough=yes
add action=change-ttl chain=postrouting comment="avoid throttling\?" disabled=yes new-ttl=<mac address> out-interface=ether1 passthrough=yes
add action=change-mss chain=forward new-mss=clamp-to-pmtu passthrough=yes protocol=tcp tcp-flags=syn
add action=mark-packet chain=postrouting new-packet-mark=out out-interface=ether1 passthrough=yes
add action=mark-packet chain=prerouting in-interface=ether1 new-packet-mark=ingress passthrough=yes
add action=mark-routing chain=prerouting in-interface=bridge-580 new-routing-mark=pleven passthrough=yes
add action=mark-connection chain=prerouting connection-state=new disabled=yes dst-port=993,143,691 new-connection-mark=IMAP passthrough=yes protocol=tcp
add action=mark-connection chain=prerouting comment="also apple push and similar" connection-state=new disabled=yes dst-port=5223 new-connection-mark=XMPP \
    passthrough=yes protocol=tcp
add action=mark-connection chain=prerouting connection-state=new disabled=yes dst-port=19305 new-connection-mark=Hangouts passthrough=yes protocol=tcp
add action=mark-connection chain=prerouting connection-state=new disabled=yes dst-port=5222,5228 new-connection-mark="Google misc" passthrough=yes protocol=tcp
add action=mark-connection chain=prerouting connection-state=new disabled=yes dst-port=8291,8728,8729 new-connection-mark=Mikrotik passthrough=yes protocol=tcp
add action=mark-connection chain=prerouting connection-state=new disabled=yes dst-port=19305 new-connection-mark=Hangouts passthrough=yes protocol=udp
add action=mark-connection chain=prerouting connection-state=new disabled=yes dst-port=3478-3481 new-connection-mark="MS Teams" passthrough=yes protocol=udp
add action=mark-connection chain=prerouting connection-state=new disabled=yes dst-port=80,443 new-connection-mark=HTTP passthrough=yes protocol=tcp
add action=mark-connection chain=prerouting connection-state=new disabled=yes dst-port=1900 new-connection-mark=UPnP passthrough=yes protocol=udp
add action=mark-connection chain=prerouting connection-state=new disabled=yes dst-port=445 new-connection-mark=Samba passthrough=yes protocol=tcp
add action=mark-connection chain=prerouting comment=VOIP disabled=yes new-connection-mark=VOIP passthrough=yes port=5060-5062,10000-10050 protocol=udp
add action=mark-connection chain=prerouting comment=DNS connection-state=new disabled=yes new-connection-mark=DNS passthrough=yes port=53 protocol=udp
add action=mark-connection chain=postrouting connection-state=new disabled=yes new-connection-mark=DNS passthrough=yes port=53 protocol=udp
add action=mark-packet chain=prerouting connection-mark=DNS disabled=yes new-packet-mark=DNS passthrough=no
add action=mark-packet chain=postrouting connection-mark=DNS disabled=yes new-packet-mark=DNS passthrough=no
add action=mark-connection chain=postrouting content=amazonaws disabled=yes new-connection-mark=aws passthrough=yes
add action=mark-connection chain=postrouting disabled=yes dst-port=443,80 new-connection-mark=quic passthrough=yes protocol=udp
add action=mark-connection chain=postrouting disabled=yes dst-port=4000,4001 new-connection-mark=IBKR passthrough=yes protocol=tcp src-address=xx.xx.xx.xx
add action=change-mss chain=forward comment="WG performance experiment" new-mss=1380 out-interface=mineWG passthrough=yes protocol=tcp tcp-flags=syn tcp-mss=\
    1381-65535
add action=add-dst-to-address-list address-list="WAN addresses" address-list-timeout=none-dynamic chain=input comment=\
    "MUST BE LAST!!!!!!!!!!! capture the WAN IP on every new connection" connection-state=new in-interface-list=WAN log-prefix="WAN IP:" protocol=tcp
/ip firewall nat
add action=dst-nat chain=dstnat comment="WG hairpin when the zte shit is NOT in bridge mode" dst-address-list="WAN addresses" dst-port=51821 protocol=udp \
    src-address=xx.xx.xx.xx/24 to-addresses=xx.xx.xx.xx to-ports=51821
add action=masquerade chain=srcnat comment=defconf out-interface-list=WAN
add action=masquerade chain=srcnat dst-address=xx.xx.xx.xx out-interface=mineWG
add action=masquerade chain=srcnat dst-address=xx.xx.xx.xx/24 out-interface=mineWG src-address=xx.xx.xx.xx/24
add action=masquerade chain=srcnat dst-address=xx.xx.xx.xx out-interface=mineWG
add action=masquerade chain=srcnat comment="without this iphone -> wie -> bogdanchovci does not work" dst-address=xx.xx.xx.xx out-interface=mineWG
add action=masquerade chain=srcnat dst-address=xx.xx.xx.xx out-interface=mineWG
add action=masquerade chain=srcnat out-interface=ipip-tunnel1
add action=masquerade chain=srcnat dst-address=xx.xx.xx.xx/24 src-address=xx.xx.xx.xx/24
add action=masquerade chain=srcnat dst-address=xx.xx.xx.xx/24 src-address=xx.xx.xx.xx/16
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set sip disabled=yes
set udplite disabled=yes
set dccp disabled=yes
set sctp disabled=yes
/ip proxy
set enabled=yes max-cache-size=2048KiB src-address=xx.xx.xx.xx
/ip proxy access
add src-address=xx.xx.xx.xx/16
add action=deny src-address=xx.xx.xx.xx
/ip route
add disabled=no distance=1 dst-address=xx.xx.xx.xx/24 gateway=mineWG pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=xx.xx.xx.xx/16 gateway=wireguard-pp pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=xx.xx.xx.xx/0 gateway=ipip-tunnel1 pref-src="" routing-table=pleven scope=30 suppress-hw-offload=no target-scope=10
add disabled=no dst-address=xx.xx.xx.xx/24 gateway=bridge routing-table=pleven suppress-hw-offload=no
add disabled=yes distance=1 dst-address=xx.xx.xx.xx/0 gateway=xx.xx.xx.xx pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=xx.xx.xx.xx/24 gateway=xx.xx.xx.xx pref-src=xx.xx.xx.xx routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
/ip service
set telnet address=xx.xx.xx.xx/24 disabled=yes
set ftp disabled=yes
set www address=xx.xx.xx.xx/24,xx.xx.xx.xx/16
set ssh address=xx.xx.xx.xx/24 disabled=yes
set api address=xx.xx.xx.xx/24,xx.xx.xx.xx/16 disabled=yes
set winbox address=xx.xx.xx.xx/16,xx.xx.xx.xx/16
set api-ssl address=xx.xx.xx.xx/24,xx.xx.xx.xx/16 disabled=yes
/ip upnp
set enabled=yes
/ip upnp interfaces
add interface=bridge type=internal
add interface=*7 type=external
add interface=*6 type=internal
add interface=*8 type=internal
/snmp
set enabled=yes trap-generators=temp-exception,start-trap,interfaces trap-interfaces=all trap-version=2
/system clock
set time-zone-name=Europe/Vienna
/system identity
set name=RBM33G#1
/system leds
add leds=,,,,,,,,,user-led type=off
/system logging
set 0 topics=info,!dhcp
set 3 action=memory
add topics=firewall
add topics=wireless
add topics=lte
add action=remote disabled=yes topics=lte
add topics=system
add disabled=yes topics=dns
add topics=script
add disabled=yes topics=wireguard
/system ntp client
set enabled=yes
/system ntp server
set broadcast=yes enabled=yes manycast=yes multicast=yes use-local-clock=yes
/system ntp client servers
add address=europe.pool.ntp.org
/system routerboard settings
set auto-upgrade=yes
/system scheduler
add interval=3m name=watch-WG on-event=\
    ":global myFunc [:parse [/system script get watch-WG-pp source]]\r\
    \n\r\
    \n\$myFunc wgcheckip=xx.xx.xx.xx endpointip=\"danov.biz\"\r\
    \n" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-date=dec/16/2021 start-time=<mac address>
add interval=3m name=watch-pp-wg on-event=\
    ":global myFunc [:parse [/system script get watch-WG-pp source]]\r\
    \n\r\
    \n\$myFunc wgcheckip=xx.xx.xx.xx endpointip=xx.xx.xx.xx\r\
    \n" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-date=dec/16/2021 start-time=<mac address>
add interval=3m name=watch-pleven-wg on-event=\
    ":global myFunc [:parse [/system script get watch-WG-pp source]]\r\
    \n\r\
    \n\$myFunc wgcheckip=xx.xx.xx.xx endpointip=cc210e9e2427.sn.mynetname.net\r\
    \n" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-date=dec/16/2021 start-time=<mac address>
add interval=3m name=watch-fac-wg on-event=\
    ":global myFunc [:parse [/system script get watch-WG-pp source]]\r\
    \n\r\
    \n\$myFunc wgcheckip=xx.xx.xx.xx endpointip=c5d10de057de.sn.mynetname.net\r\
    \n" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-date=dec/16/2021 start-time=<mac address>
add interval=3m name=watch-bogd on-event=\
    ":global myFunc [:parse [/system script get watch-WG-pp source]]\r\
    \n\r\
    \n\$myFunc wgcheckip=xx.xx.xx.xx endpointip=a2fd0cb1dad7.sn.mynetname.net" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-date=\
    dec/16/2021 start-time=<mac address>
/system script
add dont-require-permissions=no name=watch-WG owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":if ([/ping xx.xx.xx.xx inte\
    rval=1 count=5] =0) do={\r\
    \n  :log info \"WG down\"\r\
    \n  /interface disable mineWG;\r\
    \n  :delay 5\r\
    \n  /interface enable mineWG;\r\
    \n  :log info \"WG up again\"\r\
    \n}\r\
    \n"
add dont-require-permissions=no name=watch-WG-pp owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source="#:log info \"wg check-i\
    p \$wgcheckip \"\r\
    \n\r\
    \<mac address> ([/ping \$wgcheckip interval=1 count=5] =0) do={\r\
    \n  :log info \"WG down \$wgcheckip\"\r\
    \n  /interface/wireguard/peers/disable [find endpoint-address=\$endpointip];\r\
    \n  :delay 60\r\
    \n  /interface/wireguard/peers/enable [find endpoint-address=\$endpointip];\r\
    \n  :log info \"WG up again \$wgcheckip\"\r\
    \n}\r\
    \n"
/tool bandwidth-server
set authenticate=no
/tool graphing interface
add allow-address=xx.xx.xx.xx/24 interface=*7
/tool graphing queue
add allow-address=xx.xx.xx.xx/24
/tool graphing resource
add allow-address=xx.xx.xx.xx/24
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool netwatch
add disabled=no down-script=":log warn \"INET is down\"" host=xx.xx.xx.xx interval=10s timeout=1s type=simple
add comment=srvy disabled=no host=xx.xx.xx.xx interval=5s timeout=1s type=simple
add comment=danov disabled=no host=xx.xx.xx.xx interval=1m timeout=100ms type=simple
add comment=front-balancer disabled=no host=xx.xx.xx.xx interval=1m timeout=1s type=simple
add comment=front-balancer-2 disabled=no host=xx.xx.xx.xx interval=1m timeout=1s type=simple
add disabled=no host=xx.xx.xx.xx interval=1m timeout=1s type=simple
add disabled=no host=xx.xx.xx.xx interval=1m timeout=1s type=simple
add disabled=no host=xx.xx.xx.xx interval=1m timeout=1s type=simple
add disabled=no host=xx.xx.xx.xx interval=1m timeout=1s type=simple
add disabled=no host=xx.xx.xx.xx interval=1m timeout=1s type=simple
add disabled=no down-script=":log warning \"WAN is down\"" host=xx.xx.xx.xx interval=5s timeout=100ms type=simple
add disabled=no host=xx.xx.xx.xx interval=1m timeout=1s type=simple
add disabled=no host=xx.xx.xx.xx interval=1m timeout=1s type=simple
add disabled=no host=xx.xx.xx.xx interval=1m timeout=1s type=simple
/tool romon
set enabled=yes
/tool sms
set port=*7 receive-enabled=yes
/tool sniffer
set filter-ip-address=xx.xx.xx.xx/32 filter-operator-between-entries=and filter-stream=yes memory-limit=500KiB streaming-enabled=yes streaming-server=\
    xx.xx.xx.xx
server
# dec/30/2022 <mac address> by RouterOS 7.6
# software id = BCWV-TL8G
#
# model = RB750Gr3
# serial number = xxxx
/interface bridge
add admin-mac=<mac address> auto-mac=no comment=defconf name=bridge
/interface eoip
add arp=disabled disabled=yes mac-address=<mac address> name=eoip-tunnel-52 remote-address=xx.xx.xx.xx tunnel-id=52
add disabled=yes mac-address=<mac address> name=eoip-tunnel1 remote-address=xx.xx.xx.xx tunnel-id=1
/interface ipip
add name=ipip-tunnel1 remote-address=xx.xx.xx.xx
/interface wireguard
add listen-port=51821 mtu=1420 name=wireguard1
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=default-dhcp ranges=xx.xx.xx.xx-xx.xx.xx.xx
/ip dhcp-server
add address-pool=default-dhcp interface=bridge name=defconf
/port
set 0 name=serial0
/queue type
add fq-codel-limit=1024 fq-codel-quantum=300 kind=fq-codel name="fq codel"
/queue simple
add burst-limit=95M/200M burst-time=3s/3s max-limit=95M/200M name=queue1 target=xx.xx.xx.xx/16,xx.xx.xx.xx/8 total-queue="fq codel"
/routing table
add fib name=""
add fib name=opti-local
/interface bridge filter
add action=drop chain=forward in-interface=eoip-tunnel-52 ip-protocol=udp mac-protocol=ip src-port=67-68
add action=drop chain=forward dst-port=67-68 ip-protocol=udp mac-protocol=ip out-interface=eoip-tunnel-52
/interface bridge port
add bridge=bridge comment=defconf ingress-filtering=no interface=ether2
add bridge=bridge comment=defconf ingress-filtering=no interface=ether3
add bridge=bridge comment=defconf ingress-filtering=no interface=ether4
add bridge=bridge comment=defconf ingress-filtering=no interface=ether5
add bridge=bridge interface=eoip-tunnel1
add bridge=bridge interface=eoip-tunnel-52
/ip firewall connection tracking
set enabled=yes
/ip neighbor discovery-settings
set discover-interface-list=LAN
/ip settings
set accept-source-route=yes rp-filter=loose
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=eoip-tunnel1 list=LAN
add interface=ipip-tunnel1 list=LAN
add interface=wireguard1 list=LAN
/interface ovpn-server server
set auth=sha1,md5
/interface wireguard peers
add allowed-address=xx.xx.xx.xx/32,xx.xx.xx.xx/32 comment=wie endpoint-address=a2fd0c77e0ed.sn.mynetname.net endpoint-port=51821 interface=wireguard1 \
    persistent-keepalive=20s public-key="OrNS1nsHA4Ks6xy/UKuZTkNT3VRCk+cDVyUBDHn97jI="
add allowed-address=xx.xx.xx.xx/32 comment=factory endpoint-address=c5d10de057de.sn.mynetname.net endpoint-port=51821 interface=wireguard1 persistent-keepalive=20s \
    public-key="HmnE7+Pkq+BVyFKh6vKZSdEw5qQruAAyGIgzVfvcOks="
add allowed-address=xx.xx.xx.xx/32,xx.xx.xx.xx/32 comment=danov.biz endpoint-address=danov.biz endpoint-port=51871 interface=wireguard1 persistent-keepalive=15s \
    public-key="uKmLcRMJT1YOUCgsds8v2KmPSY+ccrS2scR3LCUvekA="
add allowed-address=xx.xx.xx.xx/32 comment=samsunga interface=wireguard1 public-key="0dmvOJCac4UpbCsonRZe0cWHdz9Z4yDNjLcJld9+yXQ="
add allowed-address=xx.xx.xx.xx/32 comment=macbook interface=wireguard1 public-key="MobwnvcWMHz0c5EJfIThPgeF5ybRWkv22PQUVKVMUxA="
add allowed-address=xx.xx.xx.xx/32 comment=bogdanchovci endpoint-address=a2fd0cb1dad7.sn.mynetname.net endpoint-port=51871 interface=wireguard1 public-key=\
    "YfZFSu0TVNth81L80w3MyH2Jg4joVh1MEiCDIIVvaVc="
/ip address
add address=xx.xx.xx.xx/24 interface=wireguard1 network=xx.xx.xx.xx
add address=xx.xx.xx.xx/24 interface=bridge network=xx.xx.xx.xx
add address=xx.xx.xx.xx/24 interface=eoip-tunnel1 network=xx.xx.xx.xx
add address=xx.xx.xx.xx/24 interface=ipip-tunnel1 network=xx.xx.xx.xx
/ip cloud
set ddns-enabled=yes
/ip dhcp-client
add comment=defconf dhcp-options=clientid interface=ether1 use-peer-dns=no
/ip dhcp-server network
add address=xx.xx.xx.xx/24 comment=defconf dns-server=xx.xx.xx.xx gateway=xx.xx.xx.xx netmask=24
/ip dns
set allow-remote-requests=yes cache-max-ttl=1h servers=xx.xx.xx.xx,xx.xx.xx.xx verify-doh-cert=yes
/ip dns static
add address=xx.xx.xx.xx name=mx.danov.biz
/ip firewall filter
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=input comment="defconf: accept ICMP" log-prefix=ICMP protocol=icmp
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=accept chain=input dst-port=51821 in-interface-list=WAN protocol=udp
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid log-prefix=FINV
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN log-prefix=\
    NDST
add action=drop chain=input dst-port=8291,8080,53,1080,443,80 in-interface=ether1 protocol=tcp
add action=drop chain=input dst-port=53 in-interface=ether1 protocol=udp
add action=drop chain=input connection-state=!established,related,untracked in-interface=ether1 log-prefix=Nstate
/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu passthrough=yes protocol=tcp tcp-flags=syn
add action=mark-routing chain=prerouting dst-address=xx.xx.xx.xx/23 in-interface=bridge new-routing-mark=opti-local passthrough=yes
add action=mark-routing chain=output disabled=yes dst-address=xx.xx.xx.xx/23 new-routing-mark=opti-local passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
add action=masquerade chain=srcnat dst-address=xx.xx.xx.xx/24
/ip firewall service-port
set ftp disabled=yes
/ip proxy
set enabled=yes
/ip proxy access
add src-address=xx.xx.xx.xx
add action=deny
/ip route
add disabled=no distance=1 dst-address=xx.xx.xx.xx/23 gateway=xx.xx.xx.xx pref-src="" routing-table=opti-local scope=30 suppress-hw-offload=no target-scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www address=xx.xx.xx.xx/16,xx.xx.xx.xx/8
set ssh disabled=yes
set api disabled=yes
set winbox address=xx.xx.xx.xx/16,xx.xx.xx.xx/8
set api-ssl disabled=yes
/ip socks
set enabled=yes max-connections=500 version=5
/ip upnp
set enabled=yes
/ip upnp interfaces
add interface=ether1 type=external
/routing rule
add action=lookup disabled=no routing-mark=opti-local table=opti-local
/snmp
set enabled=yes
/system clock
set time-zone-name=Europe/Sofia
/system identity
set name=Pleven
/system logging
set 0 topics=info,!dhcp
add topics=wireguard
/system routerboard settings
set auto-upgrade=yes
/system scheduler
add interval=1d name="Reboot Router Daily" on-event="/system reboot" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-date=jan/01/1970 \
    start-time=<mac address>
add interval=5m name=watch-WIE on-event="/system script run watch-WIE" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-date=\
    dec/10/2021 start-time=<mac address>
/system script
add dont-require-permissions=no name=watch-WIE owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":if ([/ping xx.xx.xx.xx inter\
    val=1 count=5] =0) do={\r\
    \n  :log info \"WIE WG down\"\r\
    \n  /interface disable wireguard1;\r\
    \n  :delay 30\r\
    \n  /interface enable wireguard1;\r\
    \n  :log info \"WG up again\"\r\
    \n}\r\
    \n"
/tool bandwidth-server
set authenticate=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool netwatch
add disabled=no host=xx.xx.xx.xx interval=1m timeout=1s type=simple
add disabled=no host=xx.xx.xx.xx interval=1m timeout=1s type=simple
add disabled=no host=xx.xx.xx.xx interval=1m timeout=1s type=simple
/tool romon
set enabled=yes
/tool romon port
add disabled=no forbid=yes interface=ether1
/tool sniffer
set filter-direction=rx filter-ip-protocol=tcp filter-operator-between-entries=and filter-port=http-alt
 
holvoetn
Forum Guru
Forum Guru
Posts: 5317
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: WG tunnel UDP is 5x faster than TCP

Fri Dec 30, 2022 1:56 pm

It's normal UDP is faster then TCP on itself.
Secondly, Wireguard is using UDP as underlying protocol.
They way you're testing is also of importance.

You should have two devices testing to each other which are not involved in the setting up of the tunnel.
They need to use the tunnel but not create it.
iperf between 2 devices is better suited for this.
 
gdanov
Member Candidate
Member Candidate
Topic Author
Posts: 150
Joined: Thu Jan 17, 2019 1:10 pm

Re: WG tunnel UDP is 5x faster than TCP

Fri Dec 30, 2022 2:17 pm

5x difference is too much to call “normal”
I don’t have device on the other end that can run iPerf unfortunately.

Are you suggesting not to use MTs own bandwidth test because it’s unreliable?

Using the same client and connecting via WG to cloud servers with similar latency gives me 200Mbps TCP and that’s the limit of my line.
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 26287
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: WG tunnel UDP is 5x faster than TCP

Fri Dec 30, 2022 2:21 pm

Of course! This is a known fact. Do not use the device itself to generate test and also run the tunnel.
The CPU must generate random data that it will send, this is an intensive task. You must use iperf3 on some powerful machine, so that the tunnel is only a tunnel and traffic goes "over" it, not from it.
 
gdanov
Member Candidate
Member Candidate
Topic Author
Posts: 150
Joined: Thu Jan 17, 2019 1:10 pm

Re: WG tunnel UDP is 5x faster than TCP

Fri Dec 30, 2022 2:27 pm

Of course! This is a known fact. Do not use the device itself to generate test and also run the tunnel.
The CPU must generate random data that it will send, this is an intensive task. You must use iperf3 on some powerful machine, so that the tunnel is only a tunnel and traffic goes "over" it, not from it.
I smell bullshit. TCP can't be 5x more cpu intensive. Not at that speed.

client

Image

server

Image
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: WG tunnel UDP is 5x faster than TCP

Fri Dec 30, 2022 4:18 pm

I smell bullshit. TCP can't be 5x more cpu intensive. Not at that speed.

One thing is CPU load when running ROS bandwidth test. Another thing is (bandwidth times delay) problem of TCP. There are plenty of articles describing it, here's link to a random one. In essence: without properly working TCP window enlargements, with mentioned 50ms RTT one is hitting throughput ceiling already at mere 12Mbps ... with some TCP window size adjustments, throughput does get better. But that process is a slow one, it takes seconds to increase considerably. So TCP test should last at least a minute or so to reach max sustainable speed.

UDP doesn't have such problems - apart from the one involving running bandwidth test directly on router.

And, please, watch your language. The bandwidth test problems are very well known (and accepted), if this is news for you you can express it without throwing insults around.
 
gdanov
Member Candidate
Member Candidate
Topic Author
Posts: 150
Joined: Thu Jan 17, 2019 1:10 pm

Re: WG tunnel UDP is 5x faster than TCP

Fri Dec 30, 2022 4:33 pm

CPU isn't loaded. Tested on second, single-CPU mikrotik and it reaches 50Mbps with minimal differences between TCP and UDP, at 100% CPU. Exactly same setup — WG with MT bandwidth test.

The argument "sometimes the CPU is bottleneck therefore bandwidth tests are unreliable" is bullshit. None of that explains 5 fold diffence between TCP and UDP. I know about TCP windows, that's why I mention latency. Any suggestions how to improve bandwidth (which is the topic of the thread)?
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: WG tunnel UDP is 5x faster than TCP

Fri Dec 30, 2022 4:42 pm

Any suggestions how to improve bandwidth (which is the topic of the thread)?

You're not accepting already given suggestions (being: use external test devices), so why should we bother providing any different ones?
 
holvoetn
Forum Guru
Forum Guru
Posts: 5317
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: WG tunnel UDP is 5x faster than TCP

Fri Dec 30, 2022 5:07 pm

Are you suggesting not to use MTs own bandwidth test because it’s unreliable?
No.
Just that you should not use it on devices making the tunnel.

4 routers
A b c d
B and c make the tunnel.
A and d can do testing.
And be aware bandwidth test is single core.

Iperf is still better.
 
tangent
Forum Guru
Forum Guru
Posts: 1329
Joined: Thu Jul 01, 2021 3:15 pm
Contact:

Re: WG tunnel UDP is 5x faster than TCP

Fri Dec 30, 2022 5:19 pm

You’re mixing two separate issues. CPU usage involved with on-device bandwidth test, and TCP vs UDP. They’re entirely orthogonal. TCP isn’t slow because of the CPU. It’s slow because TCP-in-TCP is always bad, on all CPUs, everywhere.
 
gdanov
Member Candidate
Member Candidate
Topic Author
Posts: 150
Joined: Thu Jan 17, 2019 1:10 pm

Re: WG tunnel UDP is 5x faster than TCP

Fri Dec 30, 2022 5:41 pm

You’re mixing two separate issues. CPU usage involved with on-device bandwidth test, and TCP vs UDP. They’re entirely orthogonal. TCP isn’t slow because of the CPU. It’s slow because TCP-in-TCP is always bad, on all CPUs, everywhere.
no I'm not. WG is TCP-in-UDP.

eventually it turned out to be yet another simple queue that looks properly set-up on the surface but severely limited the throughput.

I'm getting now ~90 vs ~65 Mbps UDP/TCP which is waaay better and logical throughput and ratio.
 
User avatar
Znevna
Forum Guru
Forum Guru
Posts: 1347
Joined: Mon Sep 23, 2019 1:04 pm

Re: WG tunnel UDP is 5x faster than TCP

Sat Dec 31, 2022 1:09 pm

How can it be a bad config and barking at the wrong moon?
No way.
 
gdanov
Member Candidate
Member Candidate
Topic Author
Posts: 150
Joined: Thu Jan 17, 2019 1:10 pm

Re: WG tunnel UDP is 5x faster than TCP

Sat Dec 31, 2022 3:00 pm

How can it be a bad config and barking at the wrong moon?
No way.
yet another extremely helpful answer, thanks!
 
User avatar
Znevna
Forum Guru
Forum Guru
Posts: 1347
Joined: Mon Sep 23, 2019 1:04 pm

Re: WG tunnel UDP is 5x faster than TCP

Sat Dec 31, 2022 3:05 pm

There's no right answer to a non-issue topic.
That one fits fine.
Welcome!
 
gdanov
Member Candidate
Member Candidate
Topic Author
Posts: 150
Joined: Thu Jan 17, 2019 1:10 pm

Re: WG tunnel UDP is 5x faster than TCP

Sat Dec 31, 2022 3:46 pm

There's no right answer to a non-issue topic.
That one fits fine.
Welcome!
of course it's an issue. miss-configuration, apparently (my default assumption). asking for help to debug it, and getting random shots in the dark or smirky unhelpful answers like yours.
 
User avatar
Znevna
Forum Guru
Forum Guru
Posts: 1347
Joined: Mon Sep 23, 2019 1:04 pm

Re: WG tunnel UDP is 5x faster than TCP

Sat Dec 31, 2022 4:20 pm

Nobody will go through 500 lines of config trying to find what you screwed up.
When you want to point out an issue like this, you take out the usual suspects (you don't run the speedtests on the devices that also do the routing and VPN, you disable all the bandwidth limiting queues etc) since you did none of the above, happy new year!
Also you can mark your post from above as a solution so this gets 'solved' status, so you don't get more smirky comments.
And maybe request to close the topic.
 
User avatar
tutugreen
just joined
Posts: 13
Joined: Fri Oct 06, 2017 3:14 pm

Re: WG tunnel UDP is 5x faster than TCP

Sun Feb 05, 2023 2:09 pm

Try lowering the mtu of the tunnel on both site. try 1280.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18959
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: WG tunnel UDP is 5x faster than TCP

Sun Feb 05, 2023 4:11 pm

I was going to suggest a better option, go back into the womb and try again. ;-)
 
hapoo
newbie
Posts: 45
Joined: Wed Apr 24, 2019 1:35 am

Re: WG tunnel UDP is 5x faster than TCP

Sun Feb 05, 2023 5:10 pm

He said it was an issue with the queue. Can someone explain how that is tested for and how it is resolved? I’m pretty certain I’m running into a similar issue. I get about 400mbps in iperf3 on a symmetrical gigabit connection with <50% cpu usage.
 
aoakeley
Member Candidate
Member Candidate
Posts: 170
Joined: Mon May 21, 2012 11:45 am

Re: WG tunnel UDP is 5x faster than TCP

Mon Feb 06, 2023 3:25 pm

He said it was an issue with the queue. Can someone explain how that is tested for and how it is resolved? I’m pretty certain I’m running into a similar issue. I get about 400mbps in iperf3 on a symmetrical gigabit connection with <50% cpu usage.
Getting off track from the OP's post... but to answer this....

Even with the following the best I ever managed with real world tests was between 450mbps and 500mbps
- 2 x CCR routers (tried a variety of different hardware)
- connected simply by gigabit switch,
- with wireguard in between them

I could get a bit more if I just threw garbage UDP through the tunnel using IPERF or the mikrotik speed test (dedicated routers running the speedtest), but in every real world type scenario I struggled to get much over 500mbps (often slightly less)
- iperf in TCP mode
- or copying an ISO,
- or running iperf or copying an ISO on 4 laptops (two behind each router to give some concurrency)

If anyone has been able to achieve any faster I would love to know how, as I tested the crap out of this.

(and yes I could easily sustain gigabit simply routing the traffic, and near gigabit using hardware accelerated ipsec)

Andy

Who is online

Users browsing this forum: No registered users and 25 guests