Issues with routes with package/routing marks

rbuserdl · Fri Mar 15, 2019 6:05 pm

Hello,

I have a Mikrotik router with Routeros and Firmware v6.44, which has 2 internet connections (2 different ISPs) and 2 different internal networks: 192.168.1.0/24 and 192.168.10.0/24
I configured a time ago, the router to set marks to get 192.168.10.0/24 devices, to use ISP1 for Internet access, and 192.168.1.0/24 devices to use ISP2
This configuration was made alike https://wiki.mikrotik.com/wiki/Manual:PCC, but changing some settings, this work fine
Yesterday someone configured and connected a new NVR with many cameras connected to it, and I set a dst-nat to access it from Internet (port 8001)
Different mobiles phones should connect from outside, using an app, these devices can view the cameras using ISP2 public IP but they cannot access using the ISP1 public IP
If I disable the only route with the ISP1 routing mark, it works
If I dont mark the connections when they are using the port 8001, it works, but I prefer to understand why it is happening and I need to make it work using marks
When I see the connection in

ip firewall connection

it appear with the correct connection mark
ISP2 is with less distance in default route table for 0.0.0.0/0, but if I change it to set ISP1 with less distance, the behavior is the same, still can access just using ISP2 public IP
Here I attach the settings:

/ip firewall nat
add action=dst-nat chain=dstnat comment="NVR 2" dst-port=8001 in-interface=ether2-WAN2 protocol=tcp to-addresses=192.168.1.X to-ports=8001
add action=dst-nat chain=dstnat comment="NVR 2" dst-port=8001 in-interface=ether1-WAN1 protocol=tcp to-addresses=192.168.1.X to-ports=8001


/ip firewall mangle
add action=accept chain=prerouting in-interface-list=VPNs
add action=accept chain=prerouting dst-address=WAN2_Network in-interface-list=LAN
add action=accept chain=prerouting dst-address=WAN1_Network in-interface-list=LAN

add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether1-WAN1 in-interface-list=!VPNs new-connection-mark=ISP1_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether2-WAN2 in-interface-list=!VPNs new-connection-mark=ISP2_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list="!Internal" in-interface-list=!VPNs new-connection-mark=ISP1_conn passthrough=yes src-address=192.168.10.0/24

add action=mark-routing chain=prerouting connection-mark=ISP1_conn in-interface-list=!VPNs new-routing-mark=to_ISP1 passthrough=yes src-address-list=Local
add action=mark-routing chain=prerouting connection-mark=ISP2_conn in-interface-list=!VPNs new-routing-mark=to_ISP2 passthrough=yes src-address-list=Local
add action=mark-routing chain=output connection-mark=ISP1_conn new-routing-mark=to_ISP1 out-interface-list=!VPNs passthrough=yes
add action=mark-routing chain=output connection-mark=ISP2_conn new-routing-mark=to_ISP2 out-interface-list=!VPNs passthrough=yes

/ip route
add check-gateway=ping distance=1 gateway=ISP1_Default_Gateway routing-mark=to_ISP1
add check-gateway=ping distance=1 gateway=ISP2_Default_Gateway routing-mark=to_ISP2
add check-gateway=ping distance=10 gateway=ISP2_Default_Gateway
add check-gateway=ping distance=15 gateway=ISP1_Default_Gateway

Anyone knows what could be happening?
Thanks in advance.
Regards

rbuserdl · Fri Mar 15, 2019 6:10 pm

Also:
telnet works with the port 8001 in any interface
I published 3389 port to rdp to a machine with IP address: 192.168.10.X, for test purpose, and it worked thought any ISP

sebastia · Fri Mar 15, 2019 7:21 pm

Hey


/ip firewall mangle
# why the in-interface-list=!VPN?
# passthrough not needed here
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether1-WAN1 new-connection-mark=ISP1_conn
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether2-WAN2 new-connection-mark=ISP2_conn

# mark routing for int->out: dst-address-list=!Local if local are your local addresses
# but if marked and from internal it's bound to be for outside...
add action=mark-routing chain=prerouting connection-mark=ISP1_conn in-interface-list=Internal new-routing-mark=to_ISP1
add action=mark-routing chain=prerouting connection-mark=ISP2_conn in-interface-list=Internal new-routing-mark=to_ISP2
# out-inter is irrelevant / can change
add action=mark-routing chain=output connection-mark=ISP1_conn new-routing-mark=to_ISP1 passthrough=yes
add action=mark-routing chain=output connection-mark=ISP2_conn new-routing-mark=to_ISP2 passthrough=yes

anav · Fri Mar 15, 2019 8:24 pm

Just to be clear, if you mangled marked one subnet network ....... and put the distance for that route as 10 for example and then for the other subnet and ISP you didnt put anything except a route rule of distance=5 for that route. Then the subnet without markings will go to distance=5 ISP and the rest marked will go out ISP2. Just trying to simplify before you get complicated.

The unknown is how do people know which ISP to come in on when accessing internal servers (ie the cameras or is that fixed at ISP1 or ISP2) ???

rbuserdl · Mon Mar 18, 2019 2:37 pm

Thanks both,

Anav, there is not any route rule. I know both public IPs, so I know which ISP to come in on, we give to the users both public IPs or names pointing to these IPs so they have a choice when some link is down (This is happening often, Internet service is not good in those places).
I marked just outgoing connections for 192.168.10.0/24 and leave 192.168.1.0/24 without any mark to use routes without mark (Which will use the other ISP)

Sebastia, here I answer your questions

/ip firewall mangle
# why the in-interface-list=!VPN? ---> Large story, it is not needed but it does not disturb
# passthrough not needed here ---> Ok
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether1-WAN1 new-connection-mark=ISP1_conn
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether2-WAN2 new-connection-mark=ISP2_conn

# mark routing for int->out: dst-address-list=!Local if local are your local addresses ---> This is for do not mark anything when it goes from one subnet to the other, for instance, if I need to access from 192.168.1.20 to 192.168.10.20. those connections/routes should not be marked because they will try to go out throught ISP1 or ISP2
# but if marked and from internal it's bound to be for outside...
add action=mark-routing chain=prerouting connection-mark=ISP1_conn in-interface-list=Internal new-routing-mark=to_ISP1
add action=mark-routing chain=prerouting connection-mark=ISP2_conn in-interface-list=Internal new-routing-mark=to_ISP2
# out-inter is irrelevant / can change ---> I dont understand what you mean
add action=mark-routing chain=output connection-mark=ISP1_conn new-routing-mark=to_ISP1 passthrough=yes
add action=mark-routing chain=output connection-mark=ISP2_conn new-routing-mark=to_ISP2 passthrough=yes

sebastia · Mon Mar 18, 2019 11:42 pm

Hey

This were actually my suggestions on how to do mangling!

Regarding the comments:
# why the in-interface-list=!VPN? ---> Large story, it is not needed but it does not disturb
if in-interface is matching that's all you need to know, there is no need for "!VPN", as I'm guessing wan1 / 2 are not part of it

# mark routing for int->out: dst-address-list=!Local if local are your local addresses ---> This is for do not mark anything when it goes from one subnet to the other, for instance, if I need to access from 192.168.1.20 to 192.168.10.20. those connections/routes should not be marked because they will try to go out throught ISP1 or ISP2
# but if marked and from internal it's bound to be for outside...
As stated in last line, if already marked and coming from internal it can't be for internal, it must be for outside, so the additional conditions are not needed

# out-inter is irrelevant / can change ---> I dont understand what you mean
original route had also conditions for out-interface, but given that it's going to be re-routed anyway, the only thing that matters is the connection mark WAN1 or 2. So the suggested conditions are enough.

rbuserdl · Tue Mar 19, 2019 2:10 pm

Thanks Sebastia,

How I see, this Mikrotik is routing even inside the network, because devices with an IP in 192.168.1.0/24 does not know how to reach devices in 192.168.10.0/24, so they go to their default Gateway, which is the Mikrotik, so the Mikrotik decide what to do. These connections, as I see also can be mangled and if I mangle them, they will try to go out through ISP1 or ISP2

Anyway, do you think this could be the problem?

sebastia · Tue Mar 19, 2019 9:22 pm

Won't happen: you have only 3 connection marking rules. First two are for incoming from wan, last is this:
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list="!Internal" in-interface-list=!VPNs new-connection-mark=ISP1_conn passthrough=yes src-address=192.168.10.0/24

and it won't apply to local<->local traffic given the condition dst-address-list="!Internal"

If you want to play safe: create accept rule for local-local traffic in begin of mangle table.

rbuserdl · Thu Mar 21, 2019 4:55 pm

Ok,
Do you think the issue is related with this?
I dont think so, maybe this is not necessary but it should not cause the issue
What do you think?

sebastia · Thu Mar 21, 2019 5:33 pm

What is the content of "Internal" list?

rbuserdl · Thu Mar 21, 2019 9:31 pm

Hello
This address-list has all the networks in all the sites (connected to the affected one, through VPN) and the networks of VPNs IPs
For example: 192.168.1.0/24, 192.168.10.0/24, 192.168.20.0/24, 172.16.1.0/30...... and so

Thanks in advance

sebastia · Thu Mar 21, 2019 10:27 pm

Just to check: you don't have VRF or routing rules do you?

Assuming negative to above, the connections initiated from outside to inside will stick to original WAN interface.
So the only question that remains is: how is the app talking to nvr and are in process of this conversation any new connections created, which are initiated from inside? This would then follow your default gateway which is ISP2. => And that is exactly what you see!

So my suspicion is that a new connection is created from nvr to app. Try to confirm / verify.

rbuserdl · Fri Mar 22, 2019 1:51 pm

That is right, I dont have any VRF, I dont have any routing rule
I really dont know how the NVR comunícate with clients but I could find out
I understand what you say, I agree with this.but what really surprise me is:
Why does it work only by disabling the only route with the ISP1 mark?
If the issue is that the NVR creates a new connection to the client, and it is going out through ISP2, this route should not be consider. Am I right?
Sorry, my english is not so good, I hope you understand.

sebastia · Fri Mar 22, 2019 1:58 pm

"Why does it work only by disabling the only route with the ISP1 mark?"
Please indicate which rule you're disabling.

rbuserdl · Fri Mar 22, 2019 2:26 pm

This is the route:

add check-gateway=ping distance=1 gateway=ISP1_Default_Gateway routing-mark=to_ISP1

If I only disable this route, I can view the cameras using any WAN public IP
Thanks

sebastia · Fri Mar 22, 2019 2:38 pm

If you disable that rule, responses will not be able to go out over matching isp1 => this basically disables ISP1 routing

So then if you have a client in ISP1 range it will connect over ISP2 ip (=that's the only functioning ip connectivity) and since that is your default route for connection from inside->outside, they will properly match.

rbuserdl · Fri Mar 22, 2019 4:12 pm

Thanks Sebastia,

As I think, if you have a connection with the "to_ISP1" routing mark, and you dont have any route considering this mark (in my case, the only route considering this mark is disabled), then this connection will use the default routing table, in which is going to use the default route with less distance, which is ISP2 DG right now
ISP2 is with less distance in default route table for 0.0.0.0/0, but if I change it to set ISP1 with less distance, the behavior is the same, still can access just using ISP2 public IP
So it seems that doesnt matter if the Mikrotik is answering with the same public IP, it is like the route with the ISP1 mark is not working, but it is the same with the default one, except by the mark.

Regards,
Damián

sebastia · Fri Mar 22, 2019 4:28 pm

Try to understand how client and nvr communicate first. otherwise it's just guessing.

rbuserdl · Wed Apr 03, 2019 10:21 pm

Hello,

I finally could talk with the person who installed the NVR, he said that the only port involved is the 8001, which is configured in NVR and APP, but he added that the video is sending using the port 554 or 1024

However I saw different behavior:
Now I am not marking anything to the port 8001
I connect from my phone (in other place) to the ISP1 public IP, in iVMS-4500 (The app used to view the cameras) and I can see the cameras
I filtered to see all the connections with my public IP as dst-address but nothing appeared, the list was empty
I filtered to see all the connections with my public IP as src-address and appeared the following 2 connections:

 #          PR.. SRC-ADDRESS           DST-ADDRESS           TCP-STATE   TIMEOUT     ORIG-RATE REPL-RATE ORIG-PACKETS REPL-PACKETS      ORIG-BYTES
 1  SAC   d tcp  OutsidePublicIP:44455  LocalPublicIP:8001     established 4m59s         7.7kbps 148.4kbps        7 771       10 575         351 044

I excluded my winbox connection, wich is from the same public IP
When I close the app, the connection with the port 8001 is closed automatically
The connection with the port 8001 has not "ISP_conn" as connection mark but the winbox conection has this mark, it is because I am not marking anything going to the port 8001
When I change to see another camera in my phone, other connections are opened, also with the port 8001 but after few seconds only appear 1 connection with this port

It seems that only is using the port 8001 and all connections are generated from the app.
Any idea?

Thanks in advance.
Regards

rbuserdl · Mon Apr 08, 2019 3:13 pm

Hello

I dont know what happened but it seems it is working now
I changed the routes to get everything going out through WAN1 and disabled the mangle rule which avoid mark anything on port 8001, worked
I set again less distance to WAN2 and leave this mangle rule disabled, continue working
I dont know why

Anyway, thanks, if you want to me to do any test just ask.
Regards

Issues with routes with package/routing marks

Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Re: Issues with routes with package/routing marks

Who is online