Community discussions

MikroTik App
 
polaroidik
just joined
Topic Author
Posts: 4
Joined: Sun Nov 01, 2020 12:02 am

Dual PPOE WAN, strange connection mark misshandling

Tue Nov 17, 2020 4:14 pm

Greetings to all,
as expected I already used wiki, forums and videos to make my dual VDSL with PPC configuration working and i am more or less satisfied with speed (reason behind all of this was idiotic policy of our ISP to disallow bonding for certain speeds).
Problem is indeed with portforwarding... it is kinda chaotic.

details:
VDSL1~pppoe1~VLAN1(id848)~public static ip~1.1.1.1
VDSL2~pppoe2~VLAN2(id848)~public static ip~2.2.2.2
both VDSLs do have same GW (same ISP)
RDP port through WAN: 5000
bridge ~ LAN ~ DHCP server ~ 192.168.100.0/24

Fundamentally i need to access my home workstation from my workplace through forwarded rdp port. But when i am checking the state of connections in winbox (IP-Firewall-Connections (filter: Reply Src. Address/Port=3389) I can sometimes see in destination address column/field value "1.1.1.1:5000" and connection mark column value "GTW2" (or "2.2.2.2:5000" together with connection mark "GTW1"), this state results in failed TCP handshake, ofc.
Other attemps may work but only when its paired correctly like 1.1.1.1:5000 -connection mark ->GTW1 or 2.2.2.2:5000 - connection mark -> GTW2.
On certain different types of connections I have even seen connection mark to be altering between GTW1 and GTW2, for same connection it may change in time. Ups

Problem:
From my understanding when packet goes from one VDSL to my home network, reply packet does not always return through the same VDSL line to internet (incoming GW and outgoing GW are not the same).

Question:
Is following config working as intended and portwarding cannt work ever at all? Or do i have to make some workaround that would included PCC bypass for certain incoming connections?

I am not a mikrotik professional nor networking mastermind. I tried to get training but this human malware is obstructing everything....
Please, any comment is welcome. I am slowly loosing my mind doubting everything what works so far.
/ip firewall mangle
add action=accept chain=prerouting in-interface=pppoe1
add action=accept chain=prerouting in-interface=pppoe2
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe1 new-connection-mark=GTW1 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe2 new-connection-mark=GTW2 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=GTW1 passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=GTW2 passthrough=yes per-connection-classifier=both-addresses-and-ports:2/1
add action=mark-routing chain=prerouting connection-mark=GTW1 in-interface=bridge new-routing-mark=GTW1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=GTW2 in-interface=bridge new-routing-mark=GTW2 passthrough=yes
add action=mark-routing chain=output connection-mark=GTW1 new-routing-mark=GTW1 passthrough=yes
add action=mark-routing chain=output connection-mark=GTW2 new-routing-mark=GTW2 passthrough=yes

/ip firewall nat
add action=masquerade chain=srcnat out-interface=pppoe1 src-address=192.168.100.0/24
add action=masquerade chain=srcnat out-interface=pppoe2 src-address=192.168.100.0/24
add action=dst-nat chain=dstnat dst-port=5000 protocol=tcp to-addresses=192.168.100.120 to-ports=3389

/ip route
add check-gateway=ping distance=1 gateway=pppoe1 routing-mark=GTW1
add check-gateway=ping distance=2 gateway=pppoe2 routing-mark=GTW2
add check-gateway=ping distance=1 gateway=pppoe1
add check-gateway=ping distance=2 gateway=pppoe2
add distance=3 gateway=pppoe1
add distance=5 gateway=pppoe2
Last edited by polaroidik on Tue Nov 17, 2020 6:56 pm, edited 1 time in total.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19352
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Dual PPOE WAN, strange connection mark misshandling

Tue Nov 17, 2020 5:03 pm

(1) Not sure if the routes are okay (unclear to me what the differences in distance on the rule may do or not do)

(2) the dstnat rule is funky
add action=dst-nat chain=dstnat dst-port=5000 protocol=tcp to-addresses=192.168.100.120 to-ports=3389
Is missing the in-interface-list=WAN for example.

(3) Dont think you need to put in source address on the masquerade rules.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
But the main thing to address is requirements.
IS IT really a requirement to go from work to home using both home WANIP connections?????
This adds needless complication IMHO.

Without any mangling, any query coming in on VDSL2 (the secondary WANISP for routing) will have returns going out VDSL1 as that is the primary route.
So if you enter your workstation via VDSL1, you will never have an issue and you dont need to mangle.

The question for me is why complicate life and add mangling and try to come in on VDSL2??
Especially because there is no redundancy (if one connection goes down they both will as the source is the same).

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
WeWiNet
Long time Member
Long time Member
Posts: 597
Joined: Thu Sep 27, 2018 4:11 pm

Re: Dual PPOE WAN, strange connection mark misshandling

Tue Nov 17, 2020 5:41 pm

First of all question is if your LAN and BRIDGE set up as well as IP is all correct.

But what is definitely wrong is the route setup.... ;-)

What you have is that EVERYTHING goes out to PPPOE1!
Only if PPPOE fails, it goes over to PPPOE2...
So if you go inside from PPOE2 the respons will go out to PPPOE1 as the route for PPPOE2 has higher distance as the
0.0.0.0 route.

If you want to do dual WAN (same WAN IF working with PCC) you would do (for example) :
/ip route
add check-gateway=ping distance=1 gateway=pppoe1 routing-mark=GTW1
add check-gateway=ping distance=1 gateway=pppoe2 routing-mark=GTW2
add check-gateway=ping distance=2 gateway=pppoe1
add check-gateway=ping distance=3 gateway=pppoe2
add distance=4 gateway=pppoe1
add distance=5 gateway=pppoe2
The last two routes are not even needed anymore (distance 4 and 5) as you cover them as well with distance 2 and 3 in my example.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19352
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Dual PPOE WAN, strange connection mark misshandling

Tue Nov 17, 2020 6:46 pm

I would agree, the last two do seem not necessary at all but still interested in the requirements question.
 
polaroidik
just joined
Topic Author
Posts: 4
Joined: Sun Nov 01, 2020 12:02 am

Re: Dual PPOE WAN, strange connection mark misshandling

Wed Nov 18, 2020 6:24 pm

replying

anav
-in-interface-list has following options : dynamic /none / static ; typo from your side?
-i did add in-interface to each rule, but no change
-masquerade indeed works fine w/o source address
-for work2home connection i dont care about bandwith, i may use one specific VDSL only, thats the reason why i asked about workaround (binding certain traffic to certain WAN only), i dont have any positive experience nor solution for this while load balancing is active
-load balancing is used to get as much bandwith as i can, when i am at home

WeWiNet
-lan/bridge was already checked by our local support (i will post full config later on)
-my previous config was correctly spreading traffic equally with preference of one gateway (route marked as DAC with Pref.Source visible)
-i had updated config based on your route suggestion, this had no effect, no benefit from bandwith nor GTW1/2 misshandling

# nov/18/2020 17:13:27 by RouterOS 6.47.7
# software id = NU4W-HCE5
#
# model = RB4011iGS+
# serial number = 
/interface bridge
add name=bridge
/interface vlan
add interface=ether1 name=vlan1 vlan-id=848
add interface=ether2 name=vlan2 vlan-id=848
/interface pppoe-client
add disabled=no interface=vlan1 max-mru=1492 max-mtu=1492 name=pppoe1 \
    password=bucket use-peer-dns=yes user=metronet
add disabled=no interface=vlan2 max-mru=1492 max-mtu=1492 name=pppoe2 \
    password=bucket use-peer-dns=yes user=metronet
/interface ethernet switch port
set 0 default-vlan-id=0
set 1 default-vlan-id=0
set 2 default-vlan-id=0
set 3 default-vlan-id=0
set 4 default-vlan-id=0
set 5 default-vlan-id=0
set 6 default-vlan-id=0
set 7 default-vlan-id=0
set 8 default-vlan-id=0
set 9 default-vlan-id=0
set 10 default-vlan-id=0
set 11 default-vlan-id=0
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=pool1 ranges=192.168.100.2-192.168.100.254
/ip dhcp-server
add address-pool=pool1 disabled=no interface=bridge name=dhcp1
/interface bridge port
add bridge=bridge interface=sfp-sfpplus1 trusted=yes
add bridge=bridge interface=ether3
add bridge=bridge interface=ether4
add bridge=bridge interface=ether5 trusted=yes
/ip firewall connection tracking
set tcp-syn-received-timeout=2s tcp-syn-sent-timeout=2s
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/ip settings
set tcp-syncookies=yes
/ip address
add address=192.168.100.1/24 interface=bridge network=192.168.100.0
/ip dhcp-server network
add address=192.168.100.0/24 dns-server=8.8.8.8,1.1.1.1 gateway=192.168.100.1 \
    netmask=24
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,1.1.1.1
/ip firewall filter
add action=drop chain=input comment="\"Drop invalid connections\"" \
    connection-state=invalid
add action=accept chain=input comment="\"Accept established connections\"" \
    connection-state=established
add action=accept chain=input comment="\"Accept related connections\"" \
    connection-state=related
add action=accept chain=input comment="\"UDP\"" protocol=udp
add action=accept chain=input comment="\"Allow limited pings\"" limit=\
    50/5s,2:packet protocol=icmp
add action=drop chain=input comment="\"Drop excess pings\"" protocol=icmp
add action=accept chain=input comment="\"From our LAN\"" in-interface=all-ppp \
    src-address=192.168.100.0/24
add action=accept chain=input comment="winbox fix" dst-address=192.168.100.1 \
    dst-port=8291 protocol=tcp src-address=192.168.100.0/24
add action=log chain=input comment="\"Log rest\"" log=yes log-prefix=\
    "DROP INPUT "
add action=drop chain=input comment="\"Drop rest\""
/ip firewall mangle
add action=accept chain=prerouting in-interface=pppoe1
add action=accept chain=prerouting in-interface=pppoe2
add action=mark-connection chain=prerouting connection-mark=no-mark \
    in-interface=pppoe1 new-connection-mark=GTW1 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark \
    in-interface=pppoe2 new-connection-mark=GTW2 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface=bridge new-connection-mark=GTW1 \
    passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface=bridge new-connection-mark=GTW2 \
    passthrough=yes per-connection-classifier=both-addresses-and-ports:2/1
add action=mark-routing chain=prerouting connection-mark=GTW1 in-interface=\
    bridge new-routing-mark=GTW1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=GTW2 in-interface=\
    bridge new-routing-mark=GTW2 passthrough=yes
add action=mark-routing chain=output connection-mark=GTW1 new-routing-mark=\
    GTW1 passthrough=yes
add action=mark-routing chain=output connection-mark=GTW2 new-routing-mark=\
    GTW2 passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface=all-ppp
add action=dst-nat chain=dstnat comment="\"to workstation rdp port\"" \
    dst-port=5000 in-interface=all-ppp protocol=tcp to-addresses=\
    192.168.100.100 to-ports=3389
/ip route
add check-gateway=ping distance=1 gateway=pppoe1 routing-mark=GTW1
add check-gateway=ping distance=1 gateway=pppoe2 routing-mark=GTW2
add check-gateway=ping distance=2 gateway=pppoe1
add check-gateway=ping distance=3 gateway=pppoe2
add distance=4 gateway=pppoe1
add distance=5 gateway=pppoe2
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api address=192.168.100.0/24
set winbox address=192.168.100.0/24
set api-ssl address=192.168.100.0/24
/system clock
set time-zone-name=Europe/Prague
/system identity
set name=gw
/system ntp client
set enabled=yes primary-ntp=195.113.144.201 secondary-ntp=195.119.144.238
Last edited by polaroidik on Wed Nov 18, 2020 6:36 pm, edited 1 time in total.
 
polaroidik
just joined
Topic Author
Posts: 4
Joined: Sun Nov 01, 2020 12:02 am

Re: Dual PPOE WAN, strange connection mark misshandling

Wed Nov 18, 2020 6:34 pm

Interesting observation/explanation of change after 2-3 recent routeros updates:
-current config with current routeros had to be changed a bit in masquerade's out-interface value, i had to merge both rules into one using "all-ppp".
Reason behind was that, if one pppoe connection dropped, then its NAT rule went red and mikrotik refused to re-dial connection automatically
 
WeWiNet
Long time Member
Long time Member
Posts: 597
Joined: Thu Sep 27, 2018 4:11 pm

Re: Dual PPOE WAN, strange connection mark misshandling  [SOLVED]

Wed Nov 18, 2020 10:20 pm

-my previous config was correctly spreading traffic equally with preference of one gateway (route marked as DAC with Pref.Source visible)
I don't think so with the distance you have had before. You marked traffic equally, but it all went out on PPPOE1.
Only if it failed it went to PPPOE2.
Have you checked the outbound traffic on the out interface if traffic actually hit that one?

You should also remove the first two mangle rules.
/ip firewall mangle
add action=accept chain=prerouting in-interface=pppoe1
add action=accept chain=prerouting in-interface=pppoe2
Else traffic coming in originally from here will not hit your marking rules... or do you want to accept traffic towards upstream modems or gateway?
 
User avatar
CZFan
Forum Guru
Forum Guru
Posts: 2098
Joined: Sun Oct 09, 2016 8:25 pm
Location: South Africa, Krugersdorp (Home town of Brad Binder)
Contact:

Re: Dual PPOE WAN, strange connection mark misshandling

Wed Nov 18, 2020 11:08 pm

-my previous config was correctly spreading traffic equally with preference of one gateway (route marked as DAC with Pref.Source visible)
I don't think so with the distance you have had before. You marked traffic equally, but it all went out on PPPOE1.
Only if it failed it went to PPPOE2.
Have you checked the outbound traffic on the out interface if traffic actually hit that one?

You should also remove the first two mangle rules.
/ip firewall mangle
add action=accept chain=prerouting in-interface=pppoe1
add action=accept chain=prerouting in-interface=pppoe2
Else traffic coming in originally from here will not hit your marking rules... or do you want to accept traffic towards upstream modems or gateway?

The distances In Routes will not cause traffic to only go out pppoe1 as they are connection / routing mark.

agree that those 2 mangle rules is cause of problem and must removed, what is happening is there are no pssthrough set on these, so incoming traffic is accepted and rule processing stops with these and does not get the necessary connection marks
 
polaroidik
just joined
Topic Author
Posts: 4
Joined: Sun Nov 01, 2020 12:02 am

Re: Dual PPOE WAN, strange connection mark misshandling

Fri Nov 20, 2020 8:41 am

great thanks to WeWiNet ,CZfan, anav
issue with loosing proper marking of the wan's path is gone, those two rules were indeed the cause of observed annoying issue

one last question:
-how to bind certain traffic (defined by either internal ip+port, or external ip+port ) to certain wan

using this feature, for example:
-i have synology in internal network and its providing me with ddns, now its kinda hopping every three minutes between available wans
(i think that synology uses outgoing traffic stream from internal network through wan with specific port 80ish or 5001ish to reach their ddns service)

idea:
-i was thinking about adding 1 ( i think ddns is initiated by device) mangle rule in front of any other rules to be processed first
add action=mark-routing chain=prerouting in-interface=bridge \
    log=yes log-prefix=syno_ddns new-routing-mark=GTW1 out-interface=all-ppp \
    passthrough=yes protocol=tcp src-address=192.168.100.150 src-port=5001
Would this have a chance to work as is? Or am i forgetting something?

Once again many thanks for previous help
 
WeWiNet
Long time Member
Long time Member
Posts: 597
Joined: Thu Sep 27, 2018 4:11 pm

Re: Dual PPOE WAN, strange connection mark misshandling

Fri Nov 20, 2020 10:40 am

I would remove the out interface and add the
dst-address-type=!local
, then the rule should indeed work.
You might not want to specify the port if you just consider that this NAS will always use same outgoing WAN
as long as that WAN is up and running (keep in mind, if WAN1 goes down, everything goes over to WAN2 so you are save).

But ok, you can add one (or more) specific port only.

Who is online

Users browsing this forum: anav and 19 guests