I have one router (network gateway) with two public interfaces.
WAN1 has /24 range of public addresses of ISP 1
WAN2 has one /24 public address of adsl ISP2
One LAN interface connects (with routing tables) to all my LAN networks.

To test port forwarding for incoming traffic via ISP1 network I want to route my LAN residing PC out via ISP2
I have policy routing where via addrs-list local clients get either a routing mark to route to ISP1 connected interface or they get a routing mark to route to ISP2 connected interface. Also a default route set to use ISP2.

I can now simply switch my PC to use either ISP1 or ISP2. Just give him an other address listing and after some time (secs) the new routing mark arranged this PC to go to the internet via chosen ISP. Works like a charm.

My problem is now, if I want to traceroute a public IP of ISP1 that resides on the WAN1 interface of my network gateway the traceroute shows that traffic hits that IP immediately. After some LAN routers it shows immediately the public IP of the network gateway! Not even the LAN IP of that network gateway!

Any other traceroute to an external IP on the internet shows the LAN IP of the network gateway and any next router down the road...

In my mangle I even created a rule that traffic from my lan with destination the public IP of the network gateway is forced to go out to the interface that is connected to ISP2. This way my expectation is that the ping packet enters the internet via ISP2 and than search and finds the route to an ISP1 connected users. (me). This should show a loop over the internet back to my network gateway.

But since the public IP is on the router itself that also have to send the ping out to WAN2 it looks like the router is so intelligent it replies to the ping since it knows the WAN1 owns that IP. Even if I tell router to send the package out WAN2 first. This just doesn't happen now....

Any suggestions?

It basically has to do with the routing table.
The most specific route will prevail over other routes for the same destination.
Since the IP you are trying to trace is assigned to your interface, it will
create a dynamic route, which is more specific than other you may try
to create using mangle.

Why it doesn't happens anymore, i don't know. What changed since it doesn't?
Take a look at your routing table and see the rules which would point to that IP.

Caci99 is spot on. The problem is that the directly connected route is more specific that the route you currently have, which is presumably a default route. The routing table will always choose the most specific route for a destination it can find. That's how fundamental TCP/IP routing works.

You can override that by inserting a route for the /32 public IP on interface 1 and refer to the routing mark, and assign a next hop of ISP 2. Now that route will be even more specific and will be chosen for traffic to that public IP, but only if the routing mark also matches.

OK guys, that seems to make some sense. I presume with "more specific" you mean the distance for dynamically created routes is "0" where my policy routes are at least "1"?

Ok, the dynamically route for the public IP is reading: How do I now make a more specific route for the specific Ip to go to another gateway?

I made this: It has as extra argument the routing mark "WAN2" (The routing mark parameter doesn't print with the print command!)

I also have route rule set: In mangle rule is set: This rule is the top one, so this traffic should get marked and leave the chain.

It is still not working....

ok fewi, I haven't taken all your words in advice yet.
You talk about assigning a next hop?
How to do that? I haven't got a glue. I see a "Nexthop" tab in winbox, but to be honest never used that at all... can you show me how?

You probably can't assign "WAN2" as a gateway as it's presumably not a point to point interface. You need to assign as a gateway the IP address of the ISP router on that network. The directly connected route shows that because it's a directly connected route, which the specific /32 route is not.

If you need specific help please post the output of "/ip route print detail", "/ip address print detail", and "/interface print detail".
If you feel like you need to edit public IPs because your routers are insufficiently hardened against attacks (on a side note you should then also consider hardening your routers so they aren't vulnerable to attacks) please find/replace the first two octets with two other consistent numbers. Asterisks and x's make things hard to read. Turning 192.168.1.1 into 16.16.1.1 obfuscates just as well but is consistent and easy to read.

Next hop is just another word for gateway. The gateway of the specific route has to be an IP address. You won't need that route rule.

Ok, I changed the second code part (for the specific route) into following:
so made the gateway specific. Still not working...

You probably can't assign "WAN2" as a gateway as it's presumably not a point to point interface. You need to assign as a gateway the IP address of the ISP router on that network. The directly connected route shows that because it's a directly connected route, which the specific /32 route is not.

See my previous post.
How much text do you want. I have 18 routes, 7 route rules, 259 IP addresses and 4 interfaces. You want all that info?

If you feel like you need to edit public IPs because your routers are insufficiently hardened against attacks (on a side note you should then also consider hardening your routers so they aren't vulnerable to attacks) please find/replace the first two octets with two other consistent numbers. Asterisks and x's make things hard to read. Turning 192.168.1.1 into 16.16.1.1 obfuscates just as well but is consistent and easy to read.

You're right. But to change all the info you'd requested takes me some hours into the night!. I am happy to send them to you by mail, I trust you.

Next hop is just another word for gateway. The gateway of the specific route has to be an IP address. You won't need that route rule.

But does it harm? Or should I remove it?

How much text do you want. I have 18 routes, 7 route rules, 259 IP addresses and 4 interfaces. You want all that info?

Sure. I'd always rather have too much info than too little.

You're right. But to change all the info you'd requested takes me some hours into the night!. I am happy to send them to you by mail, I trust you. 8)

Presumably your public IPs (the only ones that need obfuscating) are only on a few prefixes. Let's say they all start with 210.34. Gather all that information, paste it into notepad, edit > replace... and find "210.34" and replace it with "2.2". Then paste THAT result here. Takes all but a minute, even if there's 5 or 6 prefixes to edit.

I've tried explaining it in the abstract but apparently that isn't working, so the only way I can help you is if you post specific information from your router so I can write the configuration changes for you.

OK, I am going to do this tomorrow. I have to put it down in such a way it all makes sense. It's now way too late in the night to do this...

Ok, I changed the second code part (for the specific route) into following:

Code: Select all

#        DST-ADDRESS       PREF-SRC  GATEWAY            DISTANCE
2 A S  **.***.92.111/32              10.10.20.1              1

so made the gateway specific. Still not working...

I don't think this is going to work. The routing table will check who has the shortest distance first.
Actually I don't know of any solution to overcome the automatic routing with distance=0

Still, a /32 is more specific than e.g. a /29.
So a /32 route with metric 100 will always win against a /29 route with metric 0.

Still, a /32 is more specific than e.g. a /29.
So a /32 route with metric 100 will always win against a /29 route with metric 0.

Hmm, I tested and you are right. My mistake .

I checked a little in the wiki to see if there is any documentation to refresh my memory,
about how routes are selected. And I found this:

Criteria for selecting candidate routes

To participate in route selection process, route has to meet following criteria:

route is not disabled.
distance is not 255. Routes that are rejected by route filter have distance value of 255.
pref-src is either not set or is a valid local address of the router.
routing-mark is either not set or is referred by firewall or policy routing rules.
If type of route is unicast and it is not a connected route, it must have at least one reachable nexthop.

http://wiki.mikrotik.com/wiki/Manual:IP ... ate_routes

SInce the route in discussion is a connected route, does this mean that it will always prevail over others?

I can't say I've tried mostly because when I need to test something like Rudy has to I just use an outside machine or a 3G modem or something.

But if you look at the PCC wiki example there are explicit rules to prevent policy routing from messing up the path of packets going to directly connected networks, so it should conversely also be possible to force that issue. Other routing platforms can certainly do this, but i have never tried on a Linux based one.

Ok, it took some time but I have the info fewi asked for.
Its edited a bit and some clarifications are made.

I made it in a word document as attachment.
If this is not working, I can copy and past it directly in this topic. It is only some 3 pages long....

Could you do a text file, pretty please? I'll be spending most of my day on devices that don't read Word documents.

well, here it is. but now most of my editing is gone down the drain... Its a bit more messy to read and coloured markings are gone....

in .pdf
This at least is readable in original format and with coloured remarks etc..

Remove all those route rules for this ping test that could affect the packets you're testing with. They are interfering.

Your gateway on the WAN2 interface is 10.10.20.1.

If you want test traffic to 1.1.1.1 through WAN2, mark the traffic with some route mark, e.g 'route-test'. Then add a route like this: Now, don't take the next bit personally: you're making this way, way, way too complicated. This is a very simple question, the only reason I ever asked for those exports was because you didn't seem to immediately get my generic explanation, and I wanted to give a specific example for your specific network. There was no need to mark anything in color. I didn't need to read a nicely formatted document on how your network is run, I just wanted some IP addresses to use in examples, and I wanted them in text form. Even if I could have opened a Word document I would have just copied and pasted everything into my favorite text editor - see below for why.
Unfortunately you didn't seem to follow my advice to run find/replace on the entire document because it still contains your public IP addresses in the comments and mangle rules (at least, might be in more places), which is why I ended up NOT using an IP from your network in my code above. You may want to remove those documents if it worries you.
You also ended up not even showing the output from the commands I asked for: I asked for "/ip route print detail", for example, and you showed the output from "/ip route print". There's a huge difference between the two: only detail shows routing marks, for example. It's also formatted nicely for be parsed by computers. I have scripts I shoot the output from these commands through that removes unimportant bits and formats things nicely for me to read. That's why I want text format. I don't care if it's messy to read, I'm used to CLI output.

Now, that said, I do appreciate all your effort - it was just unnecessary to spend all that time. It would have been absolutely sufficient to paste the commands I asked for into a CLI, right click > select all > copy > paste > find/replace on the first two octets of IPs if you're worried about posting them. 2 minutes, tops.

But, to come back around to the topic at hand: don't use route rules (or figure out how to do this with route rules, I don't use those) and remove all of the ones that could apply to the packets in the (TCP/IP) connection you're testing. Make a mangle rule that applies the route mark to the traffic you want to test. Make a route with a default distance for the /32 you're testing, refer to the routing mark, and set the gateway of that route to the ISP next hop you want the packet to go through - for your WAN2 interface that appears to be 10.10.20.1.

I sincerely hope that helps, and explains how to approach this. Again, don't take the side bar personally. I truly enjoy reading your posts on wireless topics.

OK, followed you advices,

Removed the ping test route rules. Without them they still work I see, so they were not needed

Made the routing marker "route-test" in mangle as given in your code example but for my IP I want to test. No result. Traffic is still not going out through WAN2 gateway.


ip route print detail
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
0 A S ;;; Default WAN2 Route [> 3x ADSL lines]
dst-address=0.0.0.0/0 gateway=10.10.20.1 gateway-status=10.10.20.1 reachable WAN2 check-gateway=ping distance=1 scope=30 target-scope=10
routing-mark=WAN2

1 S ;;; Backup WAN2 Route [> 20/20 Line]
dst-address=0.0.0.0/0 gateway=33.333.92.1 gateway-status=33.333.92.1 reachable WAN1 check-gateway=ping distance=2 scope=30 target-scope=10
routing-mark=WAN2

2 A S ;;; Default WAN1Route [> 20/20 line]
dst-address=0.0.0.0/0 gateway=33.333.92.1 gateway-status=33.333.92.1 reachable WAN1 check-gateway=ping distance=1 scope=30 target-scope=10
routing-mark=WAN1

3 S ;;; Backup WAN1Route fr failing symmetric [> 3x ADS Lines]
dst-address=0.0.0.0/0 gateway=10.10.20.1 gateway-status=10.10.20.1 reachable WAN2 check-gateway=ping distance=2 scope=30 target-scope=10
routing-mark=WAN1

4 A S ;;; Ping 209.85.229.147 via ADSL1 (Vodafone1)
dst-address=209.85.229.147/32 gateway=10.10.20.1 gateway-status=10.10.20.1 reachable WAN2 distance=1 scope=30 target-scope=10
routing-mark=dns_test_adsl1

5 A S ;;; Ping 209.85.229.00 via ADSL2 (Vodafone2)
dst-address=209.85.229.99/32 gateway=10.10.20.1 gateway-status=10.10.20.1 reachable WAN2 distance=1 scope=30 target-scope=10
routing-mark=dns_test_adsl2

6 A S ;;; Ping 209.85.229.104 via ADSL3 (Telefonica)
dst-address=209.85.229.104/32 gateway=10.10.20.1 gateway-status=10.10.20.1 reachable WAN2 distance=1 scope=30 target-scope=10
routing-mark=dns_test_adsl3

7 A S dst-address=33.333.92.111/32 gateway=10.10.20.1 gateway-status=10.10.20.1 reachable WAN2 check-gateway=ping distance=1 scope=30 target-scope=10
routing-mark=route-test

8 A S ;;; Default Route [> 3x ADSL lines]
dst-address=0.0.0.0/0 gateway=10.10.20.1 gateway-status=10.10.20.1 reachable WAN2 check-gateway=ping distance=1 scope=30 target-scope=10

9 S ;;; Backup fr Default Route [> 20/20 line]
dst-address=0.0.0.0/0 gateway=33.333.92.1 gateway-status=33.333.92.1 reachable WAN1 check-gateway=ping distance=2 scope=30 target-scope=10

10 ADC dst-address=10.10.10.0/30 pref-src=10.10.10.2 gateway=WAN1 gateway-status=WAN1 reachable distance=0 scope=10

11 A S ;;; Route to ADSL Gateway + modems
dst-address=10.10.20.0/22 gateway=10.10.20.1 gateway-status=10.10.20.1 reachable WAN2 distance=1 scope=30 target-scope=10

12 ADC dst-address=10.10.20.0/30 pref-src=10.10.20.2 gateway=WAN2 gateway-status=WAN2 reachable distance=0 scope=10

13 A S ;;; To ALL Router networks
dst-address=10.50.48.0/20 gateway=10.50.50.2 gateway-status=10.50.50.2 reachable LAN check-gateway=ping distance=1 scope=30 target-scope=10

14 ADC dst-address=10.50.50.0/29 pref-src=10.50.50.1 gateway=LAN gateway-status=LAN reachable distance=0 scope=10

15 ADC dst-address=33.333.92.0/24 pref-src=33.333.92.2 gateway=WAN1 gateway-status=WAN1 reachable distance=0 scope=10

16 A S ;;; To ALL Client networks
dst-address=172.25.48.0/20 gateway=10.50.50.2 gateway-status=10.50.50.2 reachable LAN check-gateway=ping distance=1 scope=30 target-scope=10

17 ADC dst-address=192.168.1.0/24 pref-src=192.168.1.1 gateway=ether4 gateway-status=ether4 unreachable distance=0 scope=200




Regarding your remarks on the side;
I can't find any of my public addresses in the prints that are of importance except these of the ADSL lines. They are well fire-walled so to no use for anyone anyway. I did a ´search and replace´ for my public address of the symmetric line and word also can't find these any more.... The 33.333.92.0/24 is fake for the first two octets.

I never knew a command like ip / route / print / detail existed! Hence I probably looked it over.

Upon your advice I tried to remove the two route rules for the WAN1 and WAN2 marked traffic. The moment I disable these traffic dies to that interface..... So they are definitely needed! See hereafter.

New print:
[adminrudy@Main Router] > ip route rule print detail
Flags: X - disabled, I - inactive
0 ;;; Needed to route WAN1 marked traffic out of WAN1 interface
routing-mark=WAN1 interface=LAN action=lookup table=WAN1

1 ;;; Needed to route WAN2 marked traffic out of WAN2 interface
routing-mark=WAN2 interface=LAN action=lookup table=WAN2

2 src-address=0.0.0.0/0 dst-address=0.0.0.0/0 action=lookup table=main

End of story, I still cannot ping my WAN1 public IP from the internet by forcing traffic out of my network via WAN2 interface.
There must be something more to this....

(Indeed I used to use a G3 modem to do this kind of test, or just hook up directly to the adsl modem. But since some months I don't use G3 any longer and for the adsl modems I have to travel to remote location. So this looked more convenient to me... I'd like to work from my control room where there is this nice chair and aircon!)

After some testing in a lab with three routers and a packet sniffer it appears that this is impossible. You can route packets destined to IP addresses on directly connected networks out other interfaces, but only if the destination IP address is not actually local (i.e., is not implemented on a router interface). Those just get consumed, regardless of any routing marks.

So the only solution for this would be to bounce things off another router somewhere out on the Internet and use destination NAT to rewrite the IP to that target, and to then use destination NAT on that router to bounce it back to you.

This might also work if proxy ARP is on instead of all IPs being implemented on a network interface - it should work, in fact. The usual caveats with proxy ARP apply: it's ugly as sin, and you can't use it unless you own ALL the IPs in the network as you'd interfere with other users on the shared network. If you want to use it you could migrate slowly by turning it on for the WAN interface in question and then removing the IP addresses from that WAN interface. Once you're done testing you could put them back and turn proxy ARP back off.

After some testing in a lab with three routers and a packet sniffer it appears that this is impossible. You can route packets destined to IP addresses on directly connected networks out other interfaces, but only if the destination IP address is not actually local (i.e., is not implemented on a router interface). Those just get consumed, regardless of any routing marks.

That's what I already was afraid off. I hoped I was just not clever enough to get it done and you guys would help me out. Now it proves I am almost a smart as you in this simple issue!

Anyway, I can use a free port from a router before this network gateway and connect it straight to my adsl gateway. This way I bypass the main gateway and can do the test that way. pfff, I have to leave my comfortable office now to pull some cables. What a misery!

I'll come back with a new issue soon. Port forwarding. The reason I need to test the WAN1 IP address is that I actually can't set-up port-forwarding from the internet into my network. I use port forwarding at some stages in my local network (clients are even in my LAN still behind 1 or 2 more nat-firewalls.) and have no issues to get that done. I used all the wiki info and manuals I could find (hair pin forwarding) but it doesn't work.

I will prepare some prints and open a new topic on that one tonight or tomorrow. Thanks for the support so far....

can't VRF help in this case? I haven't played with it, so just guessing - maybe somebody knows VRF in more details =)

can't VRF help in this case? I haven't played with it, so just guessing - maybe somebody knows VRF in more details =)

I don't even know the abbreviation!
What is VRF?