Port is open when I scan from the internet, but I cannot access it from local network

adrianTNT · Thu Mar 21, 2019 1:20 pm

// I have a separate topic but I get new issues and questions along the way and it gets confusing and unproductive, I think is best to ask / solve one by one

With the attached setup, what I am trying to do is have 3 pppoe connections from my ISP, both .. sorry all three are made trough their same ethernet cable insert in port 1.
Each of the different pppoe connections need to be used by a different computer on the network, in both directions (they need to also be servers that respond on that dedicated IP www, ftp, etc).

It seems to be 90% done, the problem is that ... services work from the internet when I access them using domain name or public IP of these ppoe connections, but when I try to access the services (www, ftp, etc) from a local computer in same network normally (using domain or public ip), I seem to reach the router, unless I access them using local ip (e.g 192.168.1.62) but that is not great, I need to access them like from the internet (using public IP or domain name).

# mar/21/2019 12:52:27 by RouterOS 6.44.1
# software id = S70D-L6ES
#
# model = RouterBOARD 3011UiAS
# serial number = 8EEXXXXX
/interface bridge
add name=bridge1
/interface pppoe-client
add add-default-route=yes disabled=no interface=ether1 name=my-pppoe-1 user=\
    CRPTM256XXXXX
add add-default-route=yes disabled=no interface=ether1 name=my-pppoe-2 user=\
    CRPTM2559XXXXX
add add-default-route=yes disabled=no interface=ether1 name=my-pppoe-3 user=\
    TM509XXXXX
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=vpn ranges=192.168.89.2-192.168.89.255
add name=dhcp_pool1 ranges=192.168.1.10-192.168.1.254
/ppp profile
set *FFFFFFFE local-address=192.168.89.1 remote-address=vpn
/interface bridge port
add bridge=bridge1 interface=ether2
add bridge=bridge1 interface=ether3
add bridge=bridge1 interface=ether4
add bridge=bridge1 interface=ether5
add bridge=bridge1 interface=ether6
add bridge=bridge1 interface=ether7
add bridge=bridge1 interface=ether8
add bridge=bridge1 interface=ether9
add bridge=bridge1 interface=ether10
/interface detect-internet
set detect-interface-list=all internet-interface-list=all lan-interface-list=\
    all wan-interface-list=all
/interface l2tp-server server
set enabled=yes use-ipsec=yes
/interface list member
add interface=ether1 list=WAN
add interface=bridge1 list=LAN
add interface=my-pppoe-1 list=WAN
add interface=my-pppoe-2 list=WAN
add interface=my-pppoe-3 list=WAN
/interface pptp-server server
set enabled=yes
/interface sstp-server server
set default-profile=default-encryption enabled=yes
/ip address
add address=192.168.1.1/24 interface=ether2 network=192.168.1.0
/ip cloud
set ddns-enabled=yes
/ip dhcp-client
add dhcp-options=hostname,clientid disabled=no interface=ether1
/ip dhcp-server network
add address=192.168.1.0/24 gateway=192.168.1.1
/ip dns
set servers=8.8.8.8,8.8.4.4
/ip firewall mangle
add action=mark-routing chain=prerouting dst-address=!192.168.1.0/24 \
    new-routing-mark=my-mark-pppoe-1 passthrough=yes src-address=192.168.1.16
add action=mark-routing chain=prerouting dst-address=!192.168.1.0/24 \
    new-routing-mark=my-mark-pppoe-2 passthrough=yes src-address=192.168.1.62
/ip firewall nat
add action=masquerade chain=srcnat comment=\
    "local computers access to public internet" out-interface=my-pppoe-1
add action=masquerade chain=srcnat out-interface=my-pppoe-2
add action=masquerade chain=srcnat out-interface=my-pppoe-3
add action=dst-nat chain=dstnat comment="public internet to local computer" \
    in-interface=my-pppoe-1 to-addresses=192.168.1.16
add action=dst-nat chain=dstnat in-interface=my-pppoe-2 to-addresses=\
    192.168.1.62
add action=masquerade chain=srcnat comment="masq. vpn traffic" src-address=\
    192.168.89.0/24
/ip route
add distance=1 gateway=my-pppoe-1 routing-mark=my-mark-pppoe-1
add distance=1 gateway=my-pppoe-2 routing-mark=my-mark-pppoe-2
add distance=1 gateway=my-pppoe-1
/ip service
set telnet port=523
set ftp port=521
set www port=580
set ssh port=522
set www-ssl disabled=no port=5443
set api port=58728
set winbox port=58291
set api-ssl port=58729
/ppp secret
add name=vpn
/system clock
set time-zone-name=Europe/Bucharest
/tool graphing interface
add
/tool sniffer
set filter-interface=my-pppoe-2

mikrotik_cannot_reach_services_locally.png

Sob · Thu Mar 21, 2019 1:43 pm

Sounds like you need https://wiki.mikrotik.com/wiki/Hairpin_NAT

anav · Thu Mar 21, 2019 1:53 pm

// I have a separate topic but I get new issues and questions along the way and it gets confusing and unproductive, I think is best to ask / solve one by one

With the attached setup, what I am trying to do is have 3 pppoe connections from my ISP, both are made trough their same ethernet cable inset in port 1.
Each of the different pppoe connections need to be used by a different computer on the network, in both directions (they need to also be servers that respond on that dedicated IP www, ftp, etc).

I couldnt get past the basic math. No coffee yet this morning.

.....
@sob

Sounds like you need https://wiki.mikrotik.com/wiki/Hairpin_NAT

.........
"Amazing deduction, Sob". ..."What did you expect dear Watson!"
My question is since there seems to be three different (I think) wan Ips, does this mean there needs to be three hairpin nat sets of rules??

Sob · Thu Mar 21, 2019 1:55 pm

Hint, inspect the required srcnat rule closely, how many public addresses do you see?

mkx · Thu Mar 21, 2019 2:13 pm

Hint, inspect the required srcnat rule closely, how many public addresses do you see?

Without coffee? Around 254 (a whole /24 subnet of them)

After coffee? Ooopsie, none at all.

anav · Thu Mar 21, 2019 2:20 pm

WTF.........ahhhh sorry Sob, my bad. I have hole in my pocket and the peanuts I stuffed in there must have left a trail for MKX to find.................. ;-P

adrianTNT · Thu Mar 21, 2019 2:38 pm

With the attached setup, what I am trying to do is have 3 pppoe connections from my ISP, both are made trough their same ethernet cable inset in port 1.
I couldnt get past the basic math. No coffee yet this morning.

I am trying to set this up for the last 10 days, so that's my excuse. And I keep experimenting with one of the interfaces, sometimes I have 2 sometimes I have 3 online.

adrianTNT · Thu Mar 21, 2019 2:42 pm

Hint, inspect the required srcnat rule closely, how many public addresses do you see?

Do I need to manually add the public IP ? If so, to which rule(s) ? And I have no ideea if they should be added as "source", "destination", etc, it gets very confusing, I started using Mikrotik just days ago.

Sob · Thu Mar 21, 2019 3:28 pm

First, sorry, I was too quick and didn't study your config in detail.

The right solution depends on what you have. If you have static public addresses, simply replace in-interface=my-pppoe-x with dst-address=<public address on WANx> in dstnat rules and add hairpin rule:

/ip firewall nat
add chain=srcnat src-address=192.168.1.0/24 dst-address=192.168.1.0/24 action=masquerade

If you don't have static public addresses, you'll need another approach, add another set of dstnat rules:

/ip firewall address-list
add list=wan1ip address=<hostname pointing to WAN1 address>
add list=wan2ip address=<hostname pointing to WAN2 address>
/ip firewall nat
add action=dst-nat chain=dstnat dst-address-list=wan1ip to-addresses=192.168.1.16
add action=dst-nat chain=dstnat dst-address-list=wan2ip to-addresses=192.168.1.62

And also the above srcnat rule. These new dstnat rules could in theory replace your current ones, but since there's a delay before the address list is updated when hostname changes, and it could cause service interruptions, it's better to keep both. Old rules will work for connections from internet a these new ones for connections from LAN.

adrianTNT · Thu Mar 21, 2019 3:36 pm

Normally I have 3 static public IPs but while doing the experiments I have 2 available and one that is assigned when connecting its pppoe.

Before getting your reply, I was able to make it work from local and from internet with these rules based on the hairpin url:

add chain=dstnat dst-address=82.X.X.X action=dst-nat to-address=192.168.1.16 comment="hairpin experiments"
add chain=srcnat src-address=192.168.1.0/24 dst-address=192.168.1.16 out-interface=bridge1 action=masquerade comment="hairpin experiments"  
add chain=srcnat out-interface=my-pppoe-1 action=masquerade comment="hairpin experiments"

So now I am thinking to start with this from zero and make the same for all interfaces.
Does it look OK ? Besides the above, do I still need some rules for the other direction (e.g to tell each local server what route to use to reach the internet) ?

Sob · Thu Mar 21, 2019 3:53 pm

Great, then it's the simple case ("replace in-interface=my-pppoe-x with dst-address=<public address on WANx> in dstnat rules"). Do the same with other dstnat rules, broaden dst-address in srcnat hairpin rule (dst-address=192.168.1.0/24), so that the rule will work for all, and you're done.

For routes, you already mark routing in your config.

anav · Thu Mar 21, 2019 4:24 pm

First, sorry, I was too quick and didn't study your config in detail.

......
Premature you say!............Ahh your such a softy.......

Sob · Thu Mar 21, 2019 4:34 pm

Oh anav, only now I realize how boringly technical forum we had here before you showed up.

adrianTNT · Thu Mar 21, 2019 4:37 pm

When I think I am almost done, something goes wrong

Right now, the ports open on 192.168.1.16 cannot be reached from the internet. But I can connect from other local computers to it, even if I use the public IP of that target.
This is what I have now (attached).

adrianTNT · Thu Mar 21, 2019 4:50 pm

Hmm ... that is because I don't have the routes set yet and the reply is not sent trough the right route ?!

Trying now ...
Yes, that was it.

Thank you Sob

anav · Thu Mar 21, 2019 6:00 pm

Oh anav, only now I realize how boringly technical forum we had here before you showed up.

Say what? This is a technical forum? I came hear to meet people and make friends so that I would have a place to stay when I travel!!
Speaking of which, how many spare beds do you have at your place. You are in Spain right.

Seriously, glad the OPs issues have been resolved!!

Sob · Thu Mar 21, 2019 7:22 pm

@adrianTNT: You're welcome.

@anav: I'm tempted to believe your explanation. Although I'm not sure if everyone will welcome you with open arms, it seems to me that sometimes you're a little bit too intense. But hey, I have big heart, I'm willing to lend you a place for tent and maybe access to wifi.

anav · Thu Mar 21, 2019 7:30 pm

Too intense LOL. I disagree and perhaps we should have a mud wrestling contest to settle this.
I suppose you are one of those folks that stifles a sneeze and hates orgasms too. ;-P

Sob · Thu Mar 21, 2019 8:16 pm

You missed the "little bit" part. And call me picky, but I have certain ideas who I'd like to mud wrestle with, and I'm not sure if you'd fit. Anyway, there's one advantage of strictly technical forums, in there you can have great relation even with people who you'd want to strangle in real life, because you skip million of uninportant things to argue about, and all you focus on is common enemy in form of stray packet or something.

anav · Thu Mar 21, 2019 8:33 pm

I will be sure to wear a neck collar when I come to visit.

adrianTNT · Sun Mar 24, 2019 1:15 pm

I need some more help

I think previously I didn't have things set right but it just worked for some reason.

With current setup, the local computer 192.168.1.16 cannot reach public IP 82.x.x.7 UNLESS I add that blue mangle routing rule and the blue route, allow to reach that public IP by local bridge (?!).

Is that a must ?
Is there a simpler way ? (e.g just by one rule in NAT ?)
Don't other NAT rules there already do that ? allow 192.168.1.16 to reach 82.x.x.7 locally ?

Sob · Sun Mar 24, 2019 6:13 pm

Simpler way is routing rule:

/ip route rule
add action=lookup-only-in-table dst-address=192.168.1.0/24 table=main

Problem is that prerouting mangle (where you mark routing) occurs before dstnat. So you have packet going to public address which matches dst-address=!192.168.1.0/24 in rules #0-1 and it gets marked. Then dstnat changes destination to 192.168.1.x, but routing is already marked, so destination is looked up only in my-routing-mark-pppoe-x table and it doesn't contain route to 192.168.1.0/24, so packet goes out to internet.

Options are:
a) What you did
b) Add dst-address-type=!local to mangle rules #0-1, to also exclude all addresses on router
c) The routing rule I posted (it tells the router to always look up 192.168.1.0/24 only in main routing table)

adrianTNT · Tue Mar 26, 2019 2:17 am

OK, I left that as it was since you said it is one way to do it, and I have a hard time understanding all the rules.

I am very close to smashing it right now, I think what I need is very simple (e.g 3 pppoe accounts each going to/from different computer) I can't understand why I need tons of rules for it, each of them overwrites each other and cause conflicts, etc. It's a nightmare, it is over 16 days already, every day I "work" on this thing, it's ridiculous.

Now ... I cannot reach it by internet in order to manage it, in ip > services, I set www port to 380, I understand the 192.168.1.1 is the one replying on the web port, no ?!
So for testing, I changed that first NAT (dst-nat) to forward all traffic of that public 82.x.x.6 IP to 192.168.1.1 instead of the 192.168.1.16 as it was in the images.
And nothing, doesn't reply on port :380, only from local network.
What am I doing wrong ?

Help

The 82.x.x.6 is also the internet connection I see under "quick set", I have no idea why another "internet connection" shows there when I already have tons of rules in order to connect to the internet.

anav · Tue Mar 26, 2019 2:20 am

Dont play in quickset once you start configuring the router...........
The router creates dynamic routing rules if thats what you are seeing (normal).

adrianTNT · Tue Mar 26, 2019 2:22 am

Great.
Now it works and I have no idea why, I am going crazy.
Does it need reboots on each change ?

anav · Tue Mar 26, 2019 2:25 am

Strategy to take down a country - drop hex routers, like leaflets, on the population and watch them self destruct and go stark raving mad. No bullets need be fired, no animals hurt in the process.

adrianTNT · Tue Mar 26, 2019 2:26 am

Dont play in quickset once you start configuring the router...........
The router creates dynamic routing rules if thats what you are seeing (normal).

I was thinking it does that, but I wasn't sure, and I applied "quick" settings there in order for it to make the local bridge, enable DHCP and VPN. I cannot find another location for VPN settings eider.

adrianTNT · Tue Mar 26, 2019 2:27 am

Strategy to take down a country - drop hex routers, like leaflets, on the population and watch them self destruct and go stark raving mad. No bullets need be fired, no animals hurt in the process.

OK, I need sleep and will see what this^ is tomorrow

Sob · Tue Mar 26, 2019 4:57 am

Take a deep breath and don't panic, everything is ok, but you need some more rules.

With multi-WAN config, router doesn't automatically send replies via the same WAN from where the request came in. So the basic idea is to mark incoming connections based on incoming interface:

/ip firewall mangle
add chain=prerouting in-interface=my-pppoe-1 connection-mark=no-mark action=mark-connection new-connection-mark=ISP1_conn
add chain=prerouting in-interface=my-pppoe-2 connection-mark=no-mark action=mark-connection new-connection-mark=ISP2_conn
add chain=prerouting in-interface=my-pppoe-3 connection-mark=no-mark action=mark-connection new-connection-mark=ISP3_conn

And then send replies the right way (this is for services on router):

/ip firewall mangle
add chain=output connection-mark=ISP1_conn action=mark-routing new-routing-mark=my-mark-pppoe-1
add chain=output connection-mark=ISP2_conn action=mark-routing new-routing-mark=my-mark-pppoe-2
add chain=output connection-mark=ISP3_conn action=mark-routing new-routing-mark=my-mark-pppoe-3

And this would be for replies from internal servers (you may not need it for your current config, if each server is accessible only from one WAN interface and always uses the same interface for outgoing traffic):

/ip firewall mangle
add chain=prerouting connection-mark=ISP1_conn in-interface=bridge1 action=mark-routing new-routing-mark=my-mark-pppoe-1
add chain=prerouting connection-mark=ISP2_conn in-interface=bridge1 action=mark-routing new-routing-mark=my-mark-pppoe-2
add chain=prerouting connection-mark=ISP3_conn in-interface=bridge1 action=mark-routing new-routing-mark=my-mark-pppoe-3

And that's it, no need to hurt poor little router.

adrianTNT · Tue Mar 26, 2019 1:31 pm

For the 2 connections I have now, I added the rules you mentioned, except the last group which you said might not be needed, this is what I have now.
Problem is ... I still cannot reach it on port 380 when accessing from outside, but from local computers I can see the web interface at 192.168.1.1:380
- I was expecting it to be a simple port forward from public to 192.168.1.1 since I can access it locally.
- I am not sure the NAT and mangle to/from 192.168.1.1 is correct or needed (red arrows), I edited that trying to get a connection to router port 380.

adrianTNT · Tue Mar 26, 2019 1:34 pm

Sob, if you think you can help me by some live chat method it would be great and I think I would leave you alone faster

anav · Tue Mar 26, 2019 1:40 pm

In plain English,
(1) All external requests (originating) are connection marked by the router, depending upon which WAN interface they come in on.
(2) When traffic is exiting the router and has these marks, assign the applicable routing marks to match the same interface.
This does nothing by itself but is something that the admin directs the router to follow so that when the admin is making the IP route rules, the router will be able to physically route the traffic appropriately
(3) When server traffic replies to the requests, and we know they have the original marks the router put on them, we assign router marks as we did in 2.
@sob (4) What is missing is the IP Route rules that tie everything together.........

@sob My question is why do we bother with 2?
In part 1. we connection mark the incoming traffic per WAN.
In part 3. we track the replies by their connection marks and route mark the responses.
In part 4. we apply IP route rules to ensure the connection marked traffic goes out the applicable WAN by the route mark
(where does 2 come in to play)

adrianTNT · Tue Mar 26, 2019 2:58 pm

I am so lost, I don't even know what to ask anymore

anav · Tue Mar 26, 2019 3:34 pm

Hi Adrian, that is why I am trying to put advice in language that is digestible. The MT geeks here speak config as though its our mother tongue LOL.
I have taken it upon myself, through humour, to attempt to bring them down to earth. I could have put on my signa, I would rather herd rats than big MT egos ;-P
Seriously, stick with it! Between Sob's expert advice and my amateurish fumbling, we will get there. Heck its not about the end result its about the journey and the people we meet along the way.

mkx · Tue Mar 26, 2019 4:02 pm

The MT geeks here speak config as though its our mother tongue LOL.

To me, ROSish and English are both alien languages ... so I tend to use the less appropriate one whenever there's a chance

anav · Tue Mar 26, 2019 4:17 pm

No argument from me mkx. At least you admitted RoS was alien LOL. I do have to admit though there is nothing sexier than a person of the opposite (hmmmm lets say gender that attracts you) speaking your language with a foreign accent. In my case even those purporting to speak english (UK, Aussies etc), so hot! Hmm, mkx's plan was to distract me from the thread at hand so I didn't provide any further incorrect, confusing or damaging advice................ it worked!!

adrianTNT · Tue Mar 26, 2019 4:58 pm

Does this show any tips on why it would not open the web interface over port 380 ?
This is an attempt over internet (failing), but it loads from local network if I access both 192.0.0.1 or public IP too.

adrianTNT · Tue Mar 26, 2019 5:09 pm

OK, I added another route "ip routes" to use pppoe-1 interface for that ISP1_conn connection mark (seen in the image) and it seems to work.
Now I need to try to understand them all and find other things that don't work

mkx · Tue Mar 26, 2019 5:13 pm

Port 380 is quite uncommon and it wouldn't surprise me if some your ISP was blocking it. Try to forward some more standard port (e.g. port 80) to verify that your port forwarding rule actually works. You might want to enable logging on port forwarding rule to see if it actually gets triggered when accessed from internet ...

adrianTNT · Tue Mar 26, 2019 5:21 pm

Nah, it is open, lucky for them

I was able to fix this part, now I am trying to understand them and see if any of the rules are not needed.

adrianTNT · Tue Mar 26, 2019 5:37 pm

Sob · Tue Mar 26, 2019 10:45 pm

So... fixed? For now at least.

@anav: About your points, if by route rules you mean routes in other routing tables, seemingly missing from my last post, those were already in original config:

/ip route
add distance=1 gateway=my-pppoe-1 routing-mark=my-mark-pppoe-1
add distance=1 gateway=my-pppoe-2 routing-mark=my-mark-pppoe-2

And we need (2), because otherwise packets from router would use default gateway, even when they should really use another WAN.

If you really mean rules (/ip route rule), then actually, since this config is all static, it should be possible to do it solely with them (no firewall mangle rules):

/ip route rule
add action=lookup-only-in-table dst-address=192.168.1.0/24 table=main
add action=lookup src-address=<WAN1 address> table=my-mark-pppoe-1
add action=lookup src-address=<WAN2 address> table=my-mark-pppoe-2
add action=lookup src-address=192.168.1.16 table=my-mark-pppoe-1
add action=lookup src-address=192.168.1.62 table=my-mark-pppoe-2

Sorry for not mentioning it earlier, but it's not the first thing that comes to minds, because people usually have dynamic addresses, or they want internal servers accessible from more than one WAN, etc. And then it can no longer be done with routing rules alone, and in fact, they can get in the way. So marking connections and routing is the usual config.

anav · Wed Mar 27, 2019 12:35 am

Thanks Sob, my point was that if the return traffic from the servers (which is already connection marked, and now is route marked) then how the heck will this traffic when routed according to the route mark applied by our rule go out the wrong WAN? (in other words why mark route outgoing packets when they are already route marked - seems like route marking packets twice).

adrianTNT · Wed Mar 27, 2019 1:02 am

So... fixed? For now at least.

Yes, for now, it seems to be most stable setup so far

Can you tell if anything looks wrong ? e.g rules that can be removed because they don't do anything or have duplicate functions ?
Maybe removing passtrough needed in any rule ?
CPU usage is 1% average and 7% max, I guess I didn't cause infinite loops or anything like that.

Sob · Wed Mar 27, 2019 1:24 am

Nothing seems wrong, as in breaking things. But it's possible to make it simpler.

First step, without changing anything else, you can set passthrough=no for all rules, because you either mark routing and don't need to do anything after that, or you mark connections, but you don't need to do anything futher with the same packets either.

Then I don't like whole local-computer-to-local-server-1, it's better to not mark this traffic at all, e.g. by adding this as first rule:

/ip firewall mangle
add chain=prerouting dst-address-type=local action=accept

It will stop processing when destination is any address on router. And on second look, I don't think dst-address=!192.168.1.0/24 is doing anything useful in those two other rules. Packets from .16 or .62 can go back to them, but not directly, only if you connect to public address and dstnat sends them back. But it happens after prerouting. So you can try to remove dst-address and instead add dst-address-type=!local. It will mark packets not going to any of router's addresses. And then you won't need even the above accept rule. But it's not wrong to keep it as one common rule and then don't have any dst-addres(-type) on the other two.

@anav: I thought we were now talking about return traffic from router. And that doesn't go through prerouting chain, but through ouput.

anav · Wed Mar 27, 2019 1:32 am

I like that dst address rule accept to remove local traffic from being mangled LOL. Brilliant!
As for my point, who cares about marking output chain traffic with a second set of route marks.
As I stated return traffic from the server which came from the various wans to begin with got connection marked inbound. I am assuming that the router keeps connection and mark tracking on this traffic.
A query hits the server, the server replies and guess what, the return traffic also has the marks attached and that is why we then say that traffic with that connection mark tis on its way out of the router but before it goes anywhere, we are to route mark it. Then later to route the traffic the router sees that hey for any traffic with route mark 1, we should route that to wan1.

So basically no need for this output chain BS

The larger point being is that you are still unable to walk a mile in my shoes and see the world from an untrained perspective............... otherwise you would synthesized my shortcomings and efficiently provided a succinct and accurate education piece that makes sense.

@adrian, are you trying to hack into my brain with a unique flashing LED sequence?

Sob · Wed Mar 27, 2019 1:42 am

Do you see that dstnat rule for port 380? (Which btw doesn't need to be "action=dst-nat to-addresses=192.168.1.1", simpler "action=redirect" would work as well) When you connect to that, it's connection to router itself, not to server behind router. So think about steps that happen to these packets and you'll find out that you should give the output rules a chance.

adrianTNT · Wed Mar 27, 2019 2:09 am

@adrian, are you trying to hack into my brain with a unique flashing LED sequence?

Not really, but now I am thinking ... I hope someone doesn't fetch my data by reading the leds / traffic

anav · Wed Mar 27, 2019 2:19 am

And around we go in circles, because we have already stated that data coming into any one of the wan ports is already marked with a connection mark, whether tis destined for port 380, 666 or whatever. Traffic not ORIGINATING on the LAN side.

Sob · Wed Mar 27, 2019 2:35 am

It's not circle when you go only one way. So far you covered only first incoming packet, which got the connection marked. Now what about response packet that the service on router wants to send to client?

anav · Wed Mar 27, 2019 12:47 pm

It's not circle when you go only one way. So far you covered only first incoming packet, which got the connection marked. Now what about response packet that the service on router wants to send to client?

They get routed marked as per below!! and now the router can send them out the right WAN interface based on the admin configured IP route rules

firewall mangle
add chain=prerouting connection-mark=ISP1_conn in-interface=bridge1 action=mark-routing new-routing-mark=my-mark-pppoe-1
add chain=prerouting connection-mark=ISP2_conn in-interface=bridge1 action=mark-routing new-routing-mark=my-mark-pppoe-2
add chain=prerouting connection-mark=ISP3_conn in-interface=bridge1 action=mark-routing new-routing-mark=my-mark-pppoe-3

Sob · Wed Mar 27, 2019 2:15 pm

Are you a magician? Service on router, so for example WinBox, it got a packet from client and wants to send a reply back. How will this packet get in prerouting chain and match a rule with in-interface=bridge1? Next try...

anav · Wed Mar 27, 2019 5:20 pm

Seems like you should draw a diagram of how this works.............
Pre routing, pouting, means little to me LOL. See my granular diagram below!

Session A.
WANTRAFFICINCOMINGWanX---------------------->ROUTER PREROUTING--------->Mangle-----> Apply WANX connection marks to session A connection--------------> forward to server (1)

SERVER_Reply to Session A connection----------------------->Note Replies also have connection marksWanX still attached-------------->Prerouting------------->ApplyWANX route marks to connectionWANX packets with ROUTEMARK WANX--------------------------->Forward to WAN (but which one???)----------------> Check IP ROUTE RULES----------------------->

route to destination WANX Gateway Route mark WANx distance=3
route to destination WANY Gateway Route mark WANy distance=3
route to destination WANZ Gateway Route-mark WANz distance=3

----------------> move connection session A return traffic to WANX........................

Conclusion: take your output chain mangles rules and flush down toilet LOL. Obviously I am missing something and even more obvious you have no way of explaining it. I hope you are not a certified trainer (certifiable just not certified).

Sob · Wed Mar 27, 2019 6:58 pm

Don't you know the nice diagrams from MikroTik?

https://wiki.mikrotik.com/wiki/Manual:Packet_Flow_v6

But ok, I'll be gentle. Before you get lost in those, just know that:

a) forwarded traffic (e.g. to and from internal servers) goes through prerouting, forward and postrouting
b) traffic to router (e.g to WebFig) goes through prerouting and input
c) traffic from router (e.g. response packets from WebFig) goes through output and postrouting

So if you want to handle c), you can't rely on rules in prerouting. That's it.

anav · Wed Mar 27, 2019 9:29 pm

So the conclusion I can make is that.
Marking the server responses with mark routes is not enough!
All traffic with mangled anything needs to be identified on the way out the door with the same mark route markings.
For some unknown reason the IP Route instructions from the admin are not enough???
Since I dont mangle anything I have not encountered this phenomena LOL

Its like hey you know this forwarded traffic with marked route coming from that busy server, which by the way I am supposed to send it out wanx according to the IP Route directions, has no corresponding output instructions. So I cannot process the packets, cause I am really stupid and need more information.

Okay obviously this is granularity that is required for probably more complicated setups, otherwise it should be an automatic postrouting function.

Sob · Wed Mar 27, 2019 10:21 pm

I thought that you understood this already, it's the basic packet flow. I really believed in you!

If a packet comes to router, it's either for the router itself (= to address assigned to router, i.e. something from "/ip address") or some other device (dstnat can change that, and we're now only interested in the final destination). If it's for router, it's case b) in my previous post. If it's for something else and router has route to it, it will send it there and it's case a). You can think there's a1) for one direction (= to server) and a2) for the other (= from server), if it makes things easier. Both a1) and b) have one thing in common (= prerouting) and that's where connection is marked. If it was for some other device, response will take a2) route and that's where you mark routing for previously marked connections (= in prerouting). If it's reponse from router itself, it also needs its routing marked, but because it's case c) and it doesn't inlude prerouting, it must be done in output. And it's only for packets from router itself, because case a) doesn't include output.

You better understand it now, because I'm not sure if I'm able to come up with yet another version.

anav · Wed Mar 27, 2019 11:26 pm

hahah, you know what I got out of your explanation right............. well slightly more than that LOL.
I got the fact that the output chain mangle rules are strictly for traffic that went to the router and now has to leave the router but has to go out the right wanx and thus we mark route this traffic as well. So it has nothing to do with the return traffic from the servers. If true then we can both be right LOL. Where is my estrella? I need it to peruse this MOTHER OF ALL PACKET FLOWS
viewtopic.php?f=2&t=72736&start=50#p418449

Sob · Thu Mar 28, 2019 4:52 am

Yeah... don't start with that one. Try something simple:

routing.png

It's the diagram from manual, minus IPSec. Packets enter the router from (I) and go either to some service on router (right circle) or through router to (L). This is for any direction. Combine it with:

And when you master this, you'll be unstoppable.

adrianTNT · Thu Mar 28, 2019 3:44 pm

This image should be added to all MikroTik product images, it is a accurate representation of it's simplicity

anav · Thu Mar 28, 2019 6:17 pm

Hi Sob,
Follow, but here is my issue (mental block).
If we are going to mangle output chain with route marks based on connection marks that were applied way back at the entrance of originating traffic coming in on wans x, y, z.)
then why the heck are we route marking those packets earlier after they have left the server (replies going back to originators)??? We are already putting those route marks on that traffic again as it is almost out the door (when route and connection marks are dropped and useless anyway once passed the wan). How many times do we have to route mark the friggen packets on their journey????

My answer I thought was that the purpose of marking the original incoming packets on the WAN was to ensure REGARDLESS of the destination of those packets, the response would go out the same WAN. So in case the packets were not going to be to any servers, such as to the router itself, upon exit they would be directed to the right WAN. I think I am close.
But dont you see we are then duplicating the mark route for packets that ARE going to servers???

Sob · Thu Mar 28, 2019 7:15 pm

Is it my misunderstanding, or are you suggesting that packets from internal servers are passing through output too? Because they are not, look at the image. The small simple one I posted, not that scary thing (bad adrianTNT!

).

So in case the packets were not going to be to any servers, such as to the router itself, upon exit they would be directed to the right WAN.

No. All packets will by default use default route in main routing table. And it might not be in the right WAN. If you want a specific one, all packets need a hint where they should go.

anav · Thu Mar 28, 2019 8:38 pm

Okay so the lessons learned are...
With multiple wans
a. mark all incoming packets to ensure they will exit the same wan using connection marks (consistent and logical config management)
b. for all packets destined to the router, they will go back to the internet via the output chain.
c. thus output chain packets need to be route marked based on the original incoming connetion marks to ensure they will exit the proper wan
d. ip route rules using the route marks will ensure that they do!
e. for all traffic, originating on the WAN side being forwarded to the LAN side, the incoming connections are also marked.
d. when they reach their destination, and replies are forthcoming from lan devices, we mangle these return connections (being tracked and thus still marked with connection marks) by route marking them.
e. when the returns are heading out toward the exit and need to be IP routed, they will be sent out the appropriate wan according to the route marks entered at IP route rules.

Thus the moral of the story could have been a two liner way way back B.C and not 2019 AC. ;-P
"Initial inbound connection marks are applied to multi-wan configurations to ensure that traffic headed to the router(input) and back(output) and traffic heading to the lan (forwarded) and then returned to the WAN interface (forwarded) can be routed through IP route rules. These rules ensure the traffic goes out the same wanx it arrived from. This is done by ensuring that router centric traffic is route-marked on the output chain and lan centric traffic is route-marked when being forwarded back to the internet.

The only questions I have left are the mangles themselves.
Clearly the connection marks inbound could be applied using the input chain but would that cover only the traffic headed to the router.
If so, that would explain why some use pre-routing because pre-routing covers both input to router and forward to lans.
I never see both input and prerouting for the marking of connections on the wan inbound schemas??

Then I see mangle forwarding and mangle postrouting and am thinking why not apply route marks in post routing only??
Both forward and output ends up on postrouting. Wouldnt that be more efficient.

And why are you using prerouting for traffic comeing back/replies from servers. In the diagrams that seems to be mangle forward????
Arggg getting confused gain.

Sob · Thu Mar 28, 2019 9:52 pm

And here I though that I was explaining exactly what you finally came up with.

It's "yes" on all points, except perhaps d), because I'm not sure if you're not mixing together routes (/ip route) with routing rules (/ip route rule), but if we sum it up as "decision where the packet goes to", then it's also "yes".

For questions:

1) You answered it yourself: "pre-routing covers both input to router and forward to lans"
2) When you mark routing, you help router decide where the packet will be routed to. Doing it in postrouting wouldn't work well (try if you can spot the hint).
3) Forward comes also after the routing is decided.

anav · Sat Mar 30, 2019 3:44 pm

Thanks I will try to have those tattooed on my forehead LOL.
Now I understand why those input rules can be done either through input chain or prerouting chain.
That Postrouting mangling has nothing to do with changing anything routing wise, but to respect the routing already done and apply it?

What I wish now is to speak Russian to understand this presentation because it looks DAMN interesting and has germane aspects to this thread.
https://mum.mikrotik.com/presentations/ ... 517470.pdf

Who is online