Show me /interface list member print.
That looks correct to me
/interface list member
add interface="TTNET1 VLAN" list="WAN"
add interface="TTNET2 VLAN" list="WAN"
To me too. Do the mangle rules count (/ip firewall mangle print stats where action~“mark-routing”)? They should, as otherwise they couldn’t break anything 
So post the complete config, not just the pieces you deem relevant.
OK. Here is the whole thing. I hope it helps. When I append your mangle commands, the servers stop accessing the internet 
reducted
How did you manage to get in-interface-name=“!WAN” (with quotes)???
I am unable to do this, neither from commandline nor from winbox, so I don’t know how that rule behaves in such case.
It is impossible to debug a configuration I cannot see - you have posted a working one, not the broken one.
Hah. Honestly, I don’t know about the quotes. I just use winbox to select from the dropdown and check the negation checkbox. If quotes are not meant to be there, I can remove them.
The broken one is just your lines appended to it. Here it is:
reducted
Okay, so print shows that differently from export, and the reason for the quotes is that there is the space in the name. Why the ! is shown inside the quotes rather than outside is a different point but it works the expected way (on my test CHR, that is). So no idea again.
With “my” rules disabled, when you ping 8.8.4.4 from the command line, indicating routing-mark=“TTNET1 ROUTING”, do the pings get responded?
If you modify “my” rules to assign some nonexistent routing-mark values and enable them, do they prevent the servers from getting to the internet as well?
This really weird.
If both rules are enabled: ping doesn’t work
if only one rule is enabled(sometimes first rule, sometimes second I think. Can not be sure): ping works
if rules have a XXXYYYZZZ-1 and XXXYYYZZZ-2 routing mark: ping works
So what can be wrong when both of them are enabled?
Ok. I figured it out. I don’t know the reason, but at least I have it working.
For some reason, when
dst-address-type=!local
is used. Things stop working as well.
Just adding
dst-address=!10.0.0.0/8
didn’t work either.
However when I tried
reducted
192.168.xx.yy is not part of my local network really. they are just the subnets for the modem and the dhcp client. I was assuming !local had the same effect. But maybe not.
However since it works, I want to thank Sindy for all the help, and others as well.
Now that this is working. Why do i dirextly mark routes? Rather then first mark connections and then routes just like the examples? Is there a difference?
This is a popular misunderstanding, likely powered by a no less popular wishful thinking.
dst-address-type=local (or src-address-type=local) matches when the address is question is any of the router’s own addresses. It does not match on the whole subnets to which the router’s own addresses belong (these are called “connected subnets”).
Could it be that the servers themselves are checking availability of the gateways (192.168.x0.1)? Since there are no ports in ICMP protocol itself, it could be that check pings got always delivered to the “wrong” gateway thanks to the routing-mark (if so, swapping the 2/0 and 2/1 values between the rules would make it work).
Connection marking has two main purposes:
- to match all packets of the connection except the initial one to only a single property (the connection-mark value) when deciding what treatment to give them, and evaluate the complex conditions needed to choose the treatment only once, when handling the first (or at latest the second) packet
- storing information which can only be obtained/generated while handling packets in one direction of the connection (or when handling the first packet of the connection) for further use while handling packets in the opposite direction - this is what you do when you assign a connection-mark in chain=input depending on the in-interface, and then use it to choose the routing table (=assign the routing-mark) to packets sent by the router itself in chain=output
Matching on per-connection-classifier is a specific case where all packets of the same direction of the same connection always match on the same rule; by translating the connection-mark to routing-mark instead, you can use other type of rules (like random) to control the distribution of the connections among the WAN interfaces - the first packet of a connection is sent to a randomly chosen WAN, and the connection-mark is a way to remember that WAN so that subsequent packets of the same connection would use it too, whilst they might not match the same random rule again.
See some related thoughts here.
Thank you for all your explanations
I am just surprised with we got it to work with just a route marking, rather than a bunch of connection markings and then route markings on top of those as described in the PCC wiki.
So all is working.
How do I change from gateway address check to an ip address like 8.8.8.8 and 8.8.4.4, so that mikrotik checks the actual internet connection and route traffic to the healthy ISP?
Thank you
The principle of that solution is that the topmost recursive routes are only active if the destination address of the bottommost one is available via its “physical” gateway. If the rest of the configuration is correct, which cannot be seen from just a list of routes, disconnection of one WAN (or its failure further in the ISP’s network) only affects connections which use that WAN, but the connections using the other WAN must not suffer any drop-outs (except if caused by overload of that remining WAN by the other traffic).
A typical case would be that you have several routing tables:
- two for traffic which must use a particular WAN (e.g. incoming connections from outside); these contain only a default route via the respective WAN each, so it doesn’t matter whether that route’s gateway is the “physical” one or the “virtual” (recursive) one. If the path is down, it’s down, nothing to do about it, so no need to monitor it.
- two for traffic which should prefer one of the WANs; these are used for the traffic which can be freely distributed among the WANs. In one of these latter routing tables, the default route via WAN A’s virtual gateway is preferred and the default route via WAN B’s virtual gateway is a backup one (lower distance); in the other one, the roles of the WANs are swapped. So as long as both WANs work, the “distributable” traffic is distributed in the ratio imposed by the rules assigng routing-marks; if one of the WANs fails, the traffic which is marked to prefer that WAN is also routed via the remaining one as there’s the backup route in that routing table.
/ip route print detail while one of the WANs is down will show you which routes are active, which should help you analyse what’s going on. The topmost recursive route through the WAN which is down should become inactive in 10 seconds at the latest (check-gateway is sending a group of pings every 10 seconds and if none of them is responded, it considers the gateway unreachable and deactivates the route).
Thank you for all the insight. I have anothef question.
Lets say i get 2 more internet connections. So 4 in total.
Can i PPC only two at a time and assign it to a specific subnet?
So for example
Subnet 1 uses isp 1 and isp 2
Subnet 2 uses isp 3 and isp 4
And if for subnet 1, both isp 1 and isp 2 fails, use usp 3 and isp 4
Is this possible?
Of course it is. For this purpose, you’ll have 4 routing tables:
two for local subnet 1, with the following order of preference of the WANs:
a: 1,2,3,4
b: 2,1,4,3
two for local subnet 2, with the following order of preference of the WANs:
c: 3,4
d: 4,3
And two pairs of PCC rules, one matching on src-address(-list)=subnet1 and assigning routing marks a and b, and the other pair matching on src-address(-list)=subnet2 and assigning routing marks c and d.