mangle not marking all packets

RouterOS General
1

Hi Guys

I am using Mikrotik 3.23 on a RB450 with three interfaces enabled. WAN1, WAN2 and LOCAL
LOCAL ip = 192.168.10.254 and connected to a ISA server (forefront threat management gateway) whose ip is 192.168.10.1

WAN1 ip is 192.168.11.1 and connects to a adsl modem with ip = 192.168.11.254
WAN2 ip is 196.30.94.12 and is connected to a adsl router so its an external ip (5 available)

I have two nat rules setup for both wan ports, but I have noticed I could also create 1 nat (masquerade) rule and set the interface to !LOCAL
Is this better than having two nat rules?

I also mangle traffic so I can choose which gateway - reading the manuals I cant find the best way to do it. I see in most examples you first do a connection mark, then using that connection mark you do a packet mark. with that approach you tick passthrough in the first rule so the second one take effect but no passthrough on the second one. I understand that but I dont know how to route through different gateways without doing a routing mark - I can’t find any docs on whether you must first do the routing mark, or first do a connection mark etc.

Could someone advice me the best approach eg: First do a connection mark, then packet mark, then routing mark… or maybe I got it totally wrong.
What I also notice is that not all packets are marked - when I examine the connections in the firewall I can see other connections to the same dst port I specified in mangle. eg I create a mangle rule for dst port = 443 , in interface = local , action = mark connection, chain = prerouting but I see other connections popping up all the time destined for 443 but without the connection mark.

Again - am I doing it correct, I have tried the anyport option, and I seem to be marking more of them that way. Should I choose the interface the traffic arrives on or leave it for all interfaces. I need to access the ISA server using ssl from the web and I can get that right. I know how to do the inbound nat to the isa server wan nic, but this breaks when I forward all unknown traffic through wan2 and try to enter using ssl from WAN1

I create a new routing mark - ssl-route , the add a route that routes all traffic with that mark through WAN1 so from inside the ISA server ssl traffic goes out on WAN1

i do similar for DNS, FTP, and some other protocals. Inbound VPN is also working through the Mikrotik to the ISA server.
I create one final routing mark for all other outbound traffic and that breaks everyting if I route it through a different WAN (wan2)- when I say breaks everything I am only referring to access from outside. From inside to outside everything always works.

I’ve tried so many different configs, using either prerouting, or forward , passthrough , no passthrough, dst port, src port, any port, etc in so many different setups I can’t remember all I have tired. Can anyone post a config of a stable system - I can figure it out from there…

Thanks in advance

Regards

Basil

2

Take a look at this thread. It has a routing mark example for two gateways.
http://forum.mikrotik.com/t/2-gateways/26753/1

3

I had a look but that topic refers to having multiple IP’s on the lan side and grouping them so that certain IPs route through one gateway and the rest through the other. The responce to his question refered to something else also.

I’m going to post my configs so its clearer what I am struggling with.

4

ok here goes..

here is my nat config

Flags: X - disabled, I - invalid, D - dynamic
0 chain=srcnat action=masquerade out-interface=ether2-COZA
1 chain=srcnat action=masquerade out-interface=ether3-ADEPT
2 ;;; VPN access
chain=srcnat action=masquerade protocol=gre
3 ;;; VPN access
chain=dstnat action=dst-nat to-addresses=192.168.10.1 protocol=gre in-interface=!ether1-FTMG
4 ;;; VPN access
chain=dstnat action=dst-nat to-addresses=192.168.10.1 to-ports=1723 protocol=tcp in-interface=!ether1-FTMG dst-port=1723
5 ;;; VPN access
chain=dstnat action=dst-nat to-addresses=192.168.10.1 to-ports=1701 protocol=udp in-interface=!ether1-FTMG dst-port=1701
6 ;;; VPN access
chain=dstnat action=dst-nat to-addresses=192.168.10.1 to-ports=500 protocol=udp in-interface=!ether1-FTMG dst-port=500
7 ;;; Remote Web workplace access
chain=dstnat action=dst-nat to-addresses=192.168.10.1 to-ports=80 protocol=tcp in-interface=!ether1-FTMG dst-port=80
8 ;;; External Access for https (outlook via http)
chain=dstnat action=dst-nat to-addresses=192.168.10.1 to-ports=443 protocol=tcp in-interface=!ether1-FTMG dst-port=443

my inbound interface = “!local” interface so I should in theory be able to come in from either wan but currently I can only access the internal server from outside using the wan that I am using for my default route. I understand why this is happening but I just dont know how set it so I can come in from either wan.

here is my mangle config

Flags: X - disabled, I - invalid, D - dynamic
0 chain=prerouting action=mark-connection new-connection-mark=http_conn passthrough=yes protocol=tcp in-interface=ether1-FTMG dst-port=80
1 chain=prerouting action=mark-packet new-packet-mark=http_pack passthrough=yes in-interface=ether1-FTMG connection-mark=http_conn
2 chain=prerouting action=mark-routing new-routing-mark=http_route passthrough=no in-interface=ether1-FTMG connection-mark=http_conn
3 chain=prerouting action=mark-connection new-connection-mark=ssl_conn passthrough=yes protocol=tcp in-interface=ether1-FTMG dst-port=443
4 chain=prerouting action=mark-packet new-packet-mark=ssl_pack passthrough=yes in-interface=ether1-FTMG connection-mark=ssl_conn
5 chain=prerouting action=mark-routing new-routing-mark=ssl_route passthrough=no in-interface=ether1-FTMG connection-mark=ssl_conn
6 chain=prerouting action=mark-connection new-connection-mark=pop3_conn passthrough=yes protocol=tcp in-interface=ether1-FTMG dst-port=110
7 chain=prerouting action=mark-routing new-routing-mark=pop_route passthrough=no in-interface=ether1-FTMG connection-mark=pop3_conn
8 chain=prerouting action=mark-connection new-connection-mark=smtp_route passthrough=yes protocol=tcp in-interface=ether1-FTMG dst-port=25
9 chain=prerouting action=mark-routing new-routing-mark=smtp_route passthrough=no in-interface=ether1-FTMG connection-mark=smtp_route
10 chain=prerouting action=mark-connection new-connection-mark=ftp_conn passthrough=yes protocol=tcp in-interface=ether1-FTMG dst-port=21
11 chain=prerouting action=mark-routing new-routing-mark=ftp_route passthrough=no in-interface=ether1-FTMG connection-mark=ftp_conn
12 chain=prerouting action=mark-connection new-connection-mark=p2p_conn passthrough=yes p2p=all-p2p in-interface=ether1-FTMG
13 chain=prerouting action=mark-routing new-routing-mark=p2p Traffic passthrough=no in-interface=ether1-FTMG connection-mark=p2p_conn
14 chain=prerouting action=mark-connection new-connection-mark=other_conn passthrough=yes in-interface=ether1-FTMG
15 chain=prerouting action=mark-routing new-routing-mark=other_route passthrough=no in-interface=ether1-FTMG connection-mark=other_conn

here is my route config

0 A S dst-address=0.0.0.0/0 gateway=196.213.190.49 interface=ether3-ADEPT gateway-state=reachable distance=1 scope=30 target-scope=10
routing-mark=http_route

1 A S dst-address=0.0.0.0/0 gateway=192.168.11.254 interface=ether2-COZA gateway-state=reachable distance=1 scope=30 target-scope=10
routing-mark=pop_route

2 X S ;;; Default Route for all Other Traffic - ADEPT
dst-address=0.0.0.0/0 gateway=196.213.190.49 distance=1 scope=30 target-scope=10

3 A S dst-address=0.0.0.0/0 gateway=196.213.190.49 interface=ether3-ADEPT gateway-state=reachable distance=1 scope=30 target-scope=10
routing-mark=ftp_route

4 A S ;;; Default Route for all Other Traffic - COZA
dst-address=0.0.0.0/0 gateway=192.168.11.254 interface=ether2-COZA gateway-state=reachable distance=1 scope=30 target-scope=10

5 A S dst-address=0.0.0.0/0 gateway=196.213.190.49 interface=ether3-ADEPT gateway-state=reachable distance=1 scope=30 target-scope=10
routing-mark=smtp_route

6 A S dst-address=0.0.0.0/0 gateway=192.168.11.254 interface=ether2-COZA gateway-state=reachable distance=1 scope=30 target-scope=10
routing-mark=ssl_route

7 ADC dst-address=192.168.10.0/24 pref-src=192.168.10.254 interface=ether1-FTMG distance=0 scope=10

8 ADC dst-address=192.168.11.0/24 pref-src=192.168.11.1 interface=ether2-COZA distance=0 scope=10

9 ADC dst-address=196.213.190.48/29 pref-src=196.213.190.50 interface=ether3-ADEPT distance=0 scope=10

I’ve tried adding a route for traffic with a routing mark “ssl_route” but as soon as I change my default route or route unknown traffic through the other wan connection I can no longer reach my internal server via ssl.

I am assuming this is something to do with not all 443 traffic getting the ssl_conn or ssl_route marks so when I swing my default route I try come in through COZA but the traffic prob tries to leave through ADEPT.

How do I fix this?
tracking.jpg

5

Ok, it seems quite a few people have a similar problem to me, traffic entering on a certain wan leaves through the default gateway. Is there still no fix for this? I’m running 3.23 and surely by now there is a solution to this problem. I used a edimax br6244 gateway previously - it has 2 wan ports and I could load balance with it and come in from the internet on either wan port and connect to an internal server via remote desktop or vnc without any issues.
What gets me about my issue is even if I mangle traffic to dst port 443 and route it through gateway 2, I can access secure web sites using gateway 2, but can still only come in from the internet using gateway1 (default route) assuming I am hosting a secure web service that I want to reach from outside.
I’m sure I dont have the nat setup correctly from the outside - I have read through so many posts on this forum and see no solution yet. I have read some posts with the same issue - and when the poster found a solution he/she just says its been solved but does not post the solution - thats a great help for the rest of us.

With the edimax I even bound ssl to a specific wan port so ssl always used that gateway and I could still come in from the internet on either gateway reaching my internal server via ssl.

6

Come on guys, somebody has to be able to help me out with this??? I’ve been reading a bit more seeing as no one is replying to this topic beside myself. I see that to protect the router you create inbound filters for new connections, and also related ones, etc. I am guessing that maybe thats a similar path I should follow in order to get all traffic marked, or all packets so I can route SSL out a gateway other than the default gateway and also be able to come in from the internet through said gateway.

I’ll give it a bash and post if I have any luck. This is kind of like a self help forum, :slight_smile:

7

you need to get the basic ‘go out the same interface it comes in on’ piece working first, then add some additional rules to get specific ports going elsewhere.

http://forum.mikrotik.com/t/policy-based-routing/19589/9

I have some rules there that should help you get the 2 wans working independently. Once you get that working, you can fool with forcing things to go other directions. Basically you have to make sure your input chain stuff is all handled as well as forward and output. If you ping the router, and aren’t handling the output chain (where ping replies originate) then it will reply with the wrong IP address out the wrong gateway.

8

ok - I think I can work it out from that post but can you please explain the following…

The following section is from that post - why do you mangle the second interface differently to the first interface. For the second you do prerouting input and output but only for the second interface - would it not work if you did the same for both interfaces? Also what do the last two lines do? the action lookup table, can you elaborate a bit please?

/ip firewall mangle

add action=mark-connection chain=prerouting comment=“inbound DSL connections”
disabled=no in-interface=1-DSL new-connection-mark=in-dsl-conn
passthrough=yes
add action=mark-packet chain=prerouting comment=“” connection-mark=in-dsl-conn
disabled=no new-packet-mark=in-dsl-packet passthrough=yes
add action=mark-connection chain=prerouting comment=“inbound T1 connections”
disabled=no in-interface=2-T1 new-connection-mark=in-t1-conn
passthrough=yes
add action=mark-packet chain=prerouting comment=“” connection-mark=in-t1-conn
disabled=no new-packet-mark=in-t1-packet passthrough=yes
add action=mark-packet chain=input comment=“” connection-mark=in-t1-conn
disabled=no new-packet-mark=in-t1-packet passthrough=yes
add action=mark-packet chain=output comment=“” connection-mark=in-t1-conn
disabled=no new-packet-mark=in-t1-packet passthrough=yes
add action=mark-routing chain=prerouting comment=“” disabled=no
new-routing-mark=t1 packet-mark=in-t1-packet passthrough=yes
add action=mark-routing chain=input comment=“” disabled=no new-routing-mark=t1
packet-mark=in-t1-packet passthrough=yes
add action=mark-routing chain=output comment=“” disabled=no
new-routing-mark=t1 packet-mark=in-t1-packet passthrough=yes

and then this had to be added:

/ip route rule
add action=lookup comment=“” disabled=no routing-mark=t1 table=t1

This works perfect - pings coming in one interface go back out the right interface, etc.

9

because the primary interface is always the primary on that configuration - I didnt have to try to force things out that way because the default gateway already handled them. if you need to use either connection in a failover then you can expand on those rules and make sure you have duplicated them for each interface.

the route rules are sometimes necessary because otherwise the alternate routing table wouldnt become active… don’t know if that was a bug in some versions or what.

10

Ok, I created the following rules

Flags: X - disabled, I - invalid, D - dynamic
0 chain=prerouting action=mark-connection new-connection-mark=http_conn passthrough=yes protocol=tcp in-interface=ether1-FTMG dst-port=80

1 chain=prerouting action=mark-packet new-packet-mark=http_pack passthrough=yes in-interface=ether1-FTMG connection-mark=http_conn

2 chain=prerouting action=mark-routing new-routing-mark=http_route passthrough=no in-interface=ether1-FTMG packet-mark=http_pack

3 chain=prerouting action=mark-connection new-connection-mark=ssl_conn passthrough=yes protocol=tcp in-interface=ether1-FTMG dst-port=443

4 chain=prerouting action=mark-packet new-packet-mark=ssl_pack passthrough=yes in-interface=ether1-FTMG connection-mark=ssl_conn

5 chain=prerouting action=mark-routing new-routing-mark=ssl_route passthrough=no in-interface=ether1-FTMG connection-mark=ssl_conn

6 chain=prerouting action=mark-connection new-connection-mark=pop3_conn passthrough=yes protocol=tcp in-interface=ether1-FTMG dst-port=110

7 chain=prerouting action=mark-routing new-routing-mark=pop_route passthrough=no in-interface=ether1-FTMG connection-mark=pop3_conn

8 chain=prerouting action=mark-connection new-connection-mark=smtp_route passthrough=yes protocol=tcp in-interface=ether1-FTMG dst-port=25

9 chain=prerouting action=mark-routing new-routing-mark=smtp_route passthrough=no in-interface=ether1-FTMG connection-mark=smtp_route

10 chain=prerouting action=mark-connection new-connection-mark=ftp_conn passthrough=yes protocol=tcp in-interface=ether1-FTMG dst-port=21

11 chain=prerouting action=mark-routing new-routing-mark=ftp_route passthrough=no in-interface=ether1-FTMG connection-mark=ftp_conn

12 chain=prerouting action=mark-connection new-connection-mark=p2p_conn passthrough=yes p2p=all-p2p in-interface=ether1-FTMG

13 chain=prerouting action=mark-routing new-routing-mark=p2p Traffic passthrough=no in-interface=ether1-FTMG connection-mark=p2p_conn

14 chain=prerouting action=mark-connection new-connection-mark=other_conn passthrough=yes in-interface=ether1-FTMG

15 chain=prerouting action=mark-routing new-routing-mark=other_route passthrough=no in-interface=ether1-FTMG connection-mark=other_conn

16 X chain=prerouting action=mark-connection new-connection-mark=inbound-ether2-conn passthrough=yes in-interface=ether2-COZA

17 X chain=prerouting action=mark-packet new-packet-mark=inbound-ether2-pack passthrough=yes connection-mark=inbound-ether2-conn

18 X chain=input action=mark-packet new-packet-mark=inbound-ether2-pack passthrough=yes connection-mark=inbound-ether2-conn

19 X chain=output action=mark-packet new-packet-mark=inbound-ether2-pack passthrough=yes connection-mark=inbound-ether2-conn

20 X chain=prerouting action=mark-routing new-routing-mark=inbound-ether2-route passthrough=yes packet-mark=inbound-ether2-pack

21 X chain=prerouting action=mark-connection new-connection-mark=inbound-ether3-conn passthrough=yes in-interface=ether3-ADEPT

22 X chain=prerouting action=mark-packet new-packet-mark=inbound-ether3-pack passthrough=yes connection-mark=inbound-ether3-conn

23 X chain=input action=mark-packet new-packet-mark=inbound-ether3-pack passthrough=yes connection-mark=inbound-ether3-conn

24 X chain=output action=mark-packet new-packet-mark=inbound-ether3-pack passthrough=yes connection-mark=inbound-ether3-conn

25 X chain=prerouting action=mark-routing new-routing-mark=inbound-ether3-route passthrough=yes packet-mark=inbound-ether3-pack

I have the bottom ones disabled because if I enable them and create two routes in ip route, then nothing works. I am trying to work it out in my mind how I can still route specific traffic through a certain interface - and still come in from the internet on either wan interface with out the traffic trying route through a specific interface.

IE: if I set ssl to route out through WAN1 - that should only take effect on traffic originating from within my organization, however traffic originating from outside should be able to come in on either interface leave on said interface.

My routing table is
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
0 A S dst-address=0.0.0.0/0 gateway=196.213.190.49 interface=ether3-ADEPT gateway-state=reachable distance=1 scope=30 target-scope=10 routing-mark=http_route

1 A S dst-address=0.0.0.0/0 gateway=192.168.11.254 interface=ether2-COZA gateway-state=reachable distance=1 scope=30 target-scope=10 routing-mark=pop_route

2 X S ;;; Default Route for all Other Traffic - ADEPT
dst-address=0.0.0.0/0 gateway=196.213.190.49 distance=1 scope=30 target-scope=10

3 A S dst-address=0.0.0.0/0 gateway=196.213.190.49 interface=ether3-ADEPT gateway-state=reachable distance=1 scope=30 target-scope=10 routing-mark=ftp_route

4 A S ;;; Default Route for all Other Traffic - COZA
dst-address=0.0.0.0/0 gateway=192.168.11.254 interface=ether2-COZA gateway-state=reachable distance=1 scope=30 target-scope=10

5 A S dst-address=0.0.0.0/0 gateway=192.168.11.254 interface=ether2-COZA gateway-state=reachable distance=1 scope=30 target-scope=10 routing-mark=smtp_route

6 A S dst-address=0.0.0.0/0 gateway=192.168.11.254 interface=ether2-COZA gateway-state=reachable distance=1 scope=30 target-scope=10 routing-mark=ssl_route

7 A S dst-address=0.0.0.0/0 gateway=192.168.11.254 interface=ether2-COZA gateway-state=reachable distance=1 scope=30 target-scope=10
routing-mark=inbound-ether2-route

8 A S dst-address=0.0.0.0/0 gateway=196.213.190.49 interface=ether3-ADEPT gateway-state=reachable distance=1 scope=30 target-scope=10
routing-mark=inbound-ether3-route

9 ADC dst-address=192.168.10.0/24 pref-src=192.168.10.254 interface=ether1-FTMG distance=0 scope=10

10 ADC dst-address=192.168.11.0/24 pref-src=192.168.11.1 interface=ether2-COZA distance=0 scope=10

11 ADC dst-address=196.213.190.48/29 pref-src=196.213.190.50 interface=ether3-ADEPT distance=0 scope=10