ecmp load balanced router not accessible by ip, only mac tel

RouterOS General
1

we have a mikrotik router deployed at a customers business which we implemented as a load balancing/ failover solution.
they recently called us to add some dst nat rules to my dismay when i went to login to the wan ip i cant get in at all.
i tried logging into the 2nd wan ip and same result. we cant access or ping the router from behind our office router.
however when i plug into a switch thats not behind our office router, and pull a dhcp address from our noc i can mac telnet in because im pretty much on the same segment.
i took screenshots of the mikrotik config if someone would like to look.
i will post the config details if needed but im hoping someone else has seen or heard of this happening.
the really weird part is once mac telnetted into this device if i try to ping google.com from the mikrotik i get timeouts along with status of no route to host. obviously its getting out i can see traffic passing through the interfaces.
also i’m mac telnetted into it so its allowing some form of traffic.
I’m at a loss on this one guys. from the customers network everythings working fine except the nat rules i tried to setup for their cisco asa vpn.
I’ve been working with their cisco consultant and he’s urging me to contact mikrotik support to have them look into it however my company doesn’t want to have to pay a consultant. I really need some help here please.
talk about rock and a hard place.

quick summary of config:

the router has 2 wan interfaces and 1 lan interface.
one wan interface is a public ip from our isp network the other wan ip is a public from ameritech i believe.
ether1 = 66.xxx.xxx.xxx/24
ether2 = 192.168.2.1/29
ether3 = 80.xxx.xxx.xxx/29

for routes we have the 3 dynamic routes for the ip’s on the 3 interfaces.
In addition we have
active static routes setup as such

0.0.0.0/0 gateway 66.96.xxx.xxx int= ether1 distance2 routing mark= REST pref source 66.96.xxx.xxx
this route also has check gateway set to arp unicast scope 30 target scope 10.

0.0.0.0/0 gateway 66.96.xxx.xxx int= ether1 distance2 routing mark= POP3 pref source 66.96.xxx.xxx
this route also has check gateway set to arp unicast scope 30 target scope 10.

0.0.0.0/0 gateway 68.xxx.xxx.xxx int= ether3 distance1 routing mark= SMTP
this route also has check gateway set to PING unicast scope 30 target scope 10.

we have these routes in there as well though they aren’t active routes just static.

0.0.0.0/0 gateway 66.96.xxx.xxx int= ether1 distance2 routing mark= SMTP pref source 66.96.xxx.xxx
this route also has check gateway set to PING unicast scope 30 target scope 10.

0.0.0.0/0 gateway 68.xxx.xxx.xxx int= ether3 distance2 routing mark= REST
this route also has check gateway set to PING unicast scope 30 target scope 10.

0.0.0.0/0 gateway 68.xxx.xxx.xxx int= ether3 distance4 routing mark= POP3
this route also has check gateway set to ARP unicast scope 30 target scope 10.

those are all the routes.

the firewall rules are as follows accept input icmp
accept input tcp dst port 8291 in interface ether1
the above rules should allow us to winbox into the router.
we have a few rules for vpn traffic gre and port 1723.

which no traffic goes over currently.

mangle rules are as follows.

0 chain=prerouting proto=tcp dst port=110 action = mark routing new routing mark= POP3

1 chain=prerouting proto=tcp dst port=25 action=mark routing new routing mark= SMTP

2 chain= prerouting action=mark routing new routing mark= REST

NAT rules

0 chain= srcnat out interface= ether1 src address list= allowed internet action= masquerade

1 chain=srcnat out interface= ether3 src address list= allowed internet action = masquerade

below chain 1 we have 6 dst nat rules in place for the ether3 ip mapped to convert 68.xxx.xxx.xxx to 192.168.2.2
I have a feeling its something to do with mangle and the routing. the address list “allowed internet” consists of 192.168.2.1/29 so the lan can get out fine im told.

please i could really use some help figuring out the problem.

2

ok i read this article

http://wiki.mikrotik.com/wiki/Load_Balancing_Persistent

and i found this.

"With all multi-gateway situations there is a usual problem to reach router from public network via one, other or both gateways. Explanations is very simple - Outgoing packets uses same routing decision as packets that are going trough the router. So reply to a packet that was received via wlan1 might be send out and masqueraded via wlan2.

To avoid that we need to policy routing those connections. "

I thought our mangle rules present in the above post was our policy routing.
My main concern at this point is the inability to access the router via wan ip address.
If i run torch on our edge router i can see traffic coming from the mikrotiks 66.96.xxx.xxx address
i can even ping the device from our edge mikrotik but from our office either behind the router or directly pulling a dhcp ip from our edge fails to allow us to ping this address.
to me the symtoms seem like a routing issue, problem is our canopy network is pretty much flat.
customers have either static or dhcp assigned address from our edge router our edge router does aggregate routing for 7 subnets (static 3 subnets dhcp 4 subnets)all subnets being /24
i can ping from our office any other active ip address on the same /24 as the router having the problems.
so if i can ping other elements within this subnet i should be able to ping the router as well.
This is what leads me to believe theres a problem with our mangle, or maybe lackthereoff for our traffic.
pretty much traffic is being classified in the preroute chain as smtp traffic, pop3 traffic, any traffic not qualified by the above rules gets routing marked as REST so in theory any of our icmp traffic or anything inputted to the router should be allowed through?
My feeling is im missing some mangle and routes pertinent to our admin traffic icmp, winbox etc.
The only thing here that from understanding this is, if we were missing these rules why would we be able to ping this router from other mikrotiks on our wan but just cant communicate from our office at all.

any suggestions?
I know alot of you do consulting and as such are presented with conflict of interest to help me, however i’ll ask nicely and hope someone will remember what it was like starting out with mikrotik, and having your boss yelling at you to fix something you don’t quite understand.

3

See here http://wiki.mikrotik.com/wiki/PCC maybe this example can help you understand … or I could fix it for you for ~20 bucks :wink:

4

i’d actually be willing to pay you the 20 bucks out of my own pocket if you would teach me how you fix it.
teach me obi wan lol.
let me know if you have some way to accept payment from me.
I’m hungry and want to learn mikrotik that guide you linked to me looks like what my colleague used to base his router config upon.
problem here is because i can only mac telnet in atm im not sure how you would gain access to the router.
I have to plug into a switch thats not behind our office router to pull dhcp from our edge which puts me on the same segment as the router in question and once i open winbox and click … i can then login to it by mac address.
another thing, i can mac ping this device from anywhere on our network including behind our office router but cant ping public ip.

5

I have a method for this situation, don’t worry. I will make the config, fixing the issue, and you will take a look at it and if you do not understand something I will answer your questions :slight_smile:

6

ok sounds good one thing ima bit worried about is in the guide you linked to me earlier, I there are certain elements that we haven’t employed nor do i think our mt has the capability

/ ip firewall mangle
add chain=input in-interface=wlan1 action=mark-connection new-connection-mark=wlan1_conn
add chain=input in-interface=wlan2 action=mark-connection new-connection-mark=wlan2_conn

add chain=output connection-mark=wlan1_conn action=mark-routing new-routing-mark=to_wlan1
add chain=output connection-mark=wlan2_conn action=mark-routing new-routing-mark=to_wlan2

add chain=prerouting dst-address-type=!local in-interface=Local per-connection-classifier=both-addresses:2/0
action=mark-connection new-connection-mark=wlan1_conn passthrough=yes
add chain=prerouting dst-address-type=!local in-interface=Local per-connection-classifier=both-addresses:2/1
action=mark-connection new-connection-mark=wlan2_conn passthrough=yes

add chain=prerouting connection-mark=wlan1_conn in-interface=Local action=mark-routing new-routing-mark=to_wlan1
add chain=prerouting connection-mark=wlan2_conn in-interface=Local action=mark-routing new-routing-mark=to_wlan2

all of these elements are missing from our current config .

the below excerpt worries me as i don’t see this type of setting on the mikrotik in question.

Action mark-routing can be used only in mangle chain output and prerouting, but mangle chain prerouting is capturing all traffic that is going to the router itself. To avoid this we will use dst-address-type=!local. And with the help of the new PCC we will divide traffic into two groups based on source and destination addressees.

7

PCC is a new feature ot the latest RouterOS v3 but the actual thing I wanted to show you was the logic how traffic is mangled and then policy routed. Don’t worry about it, let’s not waste time and hurry up and do it - contact me on Skype ASAP please!

8

I reconfigured this load balancing/fail-over router and it was a pleasure working with Michael!

The problem was - only policy routes but no default route + misconfigured subnet mask on one itnerface.

Thanks again! It was a pleasure.