Hi
I have a CRS125-24G-1S v6.15 level 5 license. I have 38 eoip tunnels linking rb750 to crs from different locations and isp’s. All eoip interfaces are bridged together and to ether2 (eth2 is master port of all other ethernet ports). Everything is working greate. But now i need to make 12 more eoip tunnels and here is the problem… I create a new eoip interface (nothing is connected to it), everything seams ok, but when i add it to the bridge one of my old tunnels stops working (its still connected to crs, i get an ip from dhcp on the rb750 side, but i don’t have access to my network, nothing on the crs side responds to ping or anything). The moment i remove the new tunnel from the bridge, the old one starts working again. When i add two new tunnels to the bridge, two old ones stop working. Is there a limited number of simultaneously working tunnels? I found an information that it should be unlimited. Please help.
P.S.
Sorry for my bad english.
-
I recommend to install MikroTik RouterOS 6.27 on your router,
http://wiki.mikrotik.com/wiki/Manual:Upgrading_RouterOS -
Make sure new EoIP tunnel does not create network loop with older tunnel (that is disabled automatically).
Upgrading to 6.27 made it worse. All of my tunnels stopped working after upgrade. I’ve disabled all of them and got about half working by enabling one by one, but when i try to enable any one of the rest another one stops working and behaves just like i described in my first post.
There can’t be any loop betwean tunnels, the remote rb750 are at least few kilometers from eachother and the problem started when i created a new eoip interface and added it to the bridge (there was nothing connected to the interface, the rb750 that was supposed to connect to the tunnel is still on my desk in a box). Before that everything was working greate. How can i check if stp disabled any interface? I tried "interface bridge port monitor " but it shows “learning: yes forwarding: yes” on every interface.
You won’t have a problem with STP blocking ports on CRS125, because it doesn’t support STP.
[Edit]…in hardware. It does support STP on bridges [which are processed on the CPU].
i thought it does there are settings for stp and rstp i bridge settings.
Now all my tunnels are connected, but theres another problem and i think its with arp propagation over the tunnels. When i connect my laptop on any remote site, i get an ip address from pool 192.168.1.0/24, its ok. I can ping devices on any of the romote sites, but not 192.168.1.1 or 192.168.2.1 (crs ips), i cann access my server (192.168.1.100) connected physicaly to crs (so i can’t access internet). On the rb750 in ip/arp there arent any records showing up for those ips. When i add the record to arp manually and a route to the ip through eoip-tunnel1 it works for some time and then it stops. Any ideas why? Previously to adding more tunnels on crs everything worked greate without any static records. I’ve read about proxy-arp, but from what i understand it should be used when there are different subnets. I have the same subnet on all locations. 192.168.1.0/24 for devices and 192.168.2.0/24 for rb750.
Whoops, my mistake! I had only looked in the Switch menu.
stp is off. The only problem is with arp (like i described in previous post) and access to ip on crs that are set on bridge that is connectin tunnels and ethernet ports together. Now i noticed that sometimes there are problems with access from crs side to some devices connected to rb750. For example there are two devices connected to one rb750, i can ping one but not the other. When i logon to rb750 both reply to ping. After some time both start to reply to ping from crs, but they still cant access crs ip and internet (rb750 also cant).
tunnels are working. I can access every device no matter where it is, the problem was “Clamp TCP MSS” turned on on some tunnels. I turned it off and its working, i can ping crs from every rb750. The problem now is access to internet, i can logon to 192.168.1.1 but no internet access (internet access is only on devices connected directly to crs). I noticed that when i run IP-SCAN on 192.168.1.1 on any of routerboards i get that address with 12 different mac addresses (none of them belong to any of my mikrotiks) but snmp shows crs name. Does any one have a clue what is going on?
Those MAC addresses are probably the culprits behind the initial response from sergejs… suspecting a network loop.
I try to avoid a large broadcast segment exactly for reasons like this. One guy plugging in some mis-configured device in one location can sink the entire battleship. (rogue DHCP servers can really cause havoc on large broadcast domains, for instance)
Start looking in the hosts tables of your bridges to find where these MAC addresses are coming from. Probably some device has proxy-arp turned on at the very least.
There’s no posibility of a physical loop or a misconfigured device being connected. At every remote site there’s a industrial controler connected to rb750 and an nvr on few of them. In total we have about 50-60 devices in the network. Only 10 people have access to those places (if anyone enters we see an alarm) and only my colegue and i work with conectivity (separate key to a box with rb750 and controler).
I checked, every device has arp set to enable. When i do arp ping on bridge or eoip-tunnel i get timeout, but on ether1 (wan port) it replies with all of those mac addresses. This happens at every rb750. We have 4 different isp, so how is it possible that sudenly on every one of them i can see multiple devices with ip 192.168.1.1?
Today i set additional ip 192.168.3.1/24 on bridge on crs. Just for a test. My laptop (static ip set) connected directly to crs works (i have internet access on the 192.168.3.0/24 network). But when i connect on a remote site i don’t get any reply from arp, so no entry for 192.168.3.1 in arp table. When i add a entry manually to my arp table it works, have internet access and can logon to 192.168.3.1. After deleting the entry it stops working…
I don’t have any more ideas what can be the source of these problems… Could have someone injected me with fake arp or something? Two tunnels are done by rb 411 (wifi antenna, dont have access to them) managed by my ex-employer, who is pissed that i quit. Earlier he managged all the tunnels. Could he have done this somehowe?
It sounds like broadcasts are being filtered somewhere - possibly split horizon bridging, or a bridge firewall rule.
Or maybe WDS got disabled on the RB411 bridge - that could explain this behavior as well.
I would start to track down the MAC addresses that show up as 192.168.1.1 whenever you do the arp ping.
Note the MACs, find them in the hosts tab of the bridge, and start following the trail until you identify which physical interface they’re coming from. Check the vendor MAC identifier on coffer.com to see if that helps identify what type of device (Linksys, etc).
If you can ARP ping almost any IP and still get replies, and always from the same MAC, then something has proxy-arp turned on. You just have to find where the leak is and plug it.
If you can ARP ping almost any IP and still get replies, and always from the same MAC, then something has proxy-arp turned on. You just have to find where the leak is and plug it.
You misunderstood me. I see many different mac addresses for ip 192.168.1.1 and only on ether1 (wan port) of almost every mikrotik. Some of them see 3-5 different mac’s and some see more than a dozen other see none. I see no reply on bridge or eoip-tunnel.
Anything isn’t filtered i have only masquarade for subnets 192.168.1.0/24 and 192.168.3.0/24 and 4 port redirections.
I checked the hosts tabs of bridges on every mikrotik. None of the mac reported by arp ping that suppose to be associated with 192.168.1.1 are in there. I noticed that every mikrotik reports a different set of mac addresses for 192.168.1.1. Few of rb750 are in the same subnet of our isp. So if they are visible through ether1 they should be the same…
Some of the mac from ip scan resolve to elitegroup, dell, cisco etc. (checked at coffer.com) We dont have any devices from those companies.
Today suddenly about half of the industrial controllers disconnected from the server. I saw them when scanning the subnet and i could ping them. Some of them are connected through tunnels and some directly to internet (no tunnel yet). All tunnels were working, sending and receiving data. Nothing helped to get them to connect to the server. Only after rebooting crs (as a desperate messure), all of them connected to server the moment crs went back online. Could it be a problem with crs? I’m starting to thing of reseting it to factory defaults any configuring from scratch (as a last resort). In the past few times some random tunnel stopped working (every time a different one). Disabling it and enabling again solved the problem.
I see.
Have you considered trying something like L2TP with a bridging profile? That might be a little easier to manage settings and so forth, as well as give you the option to have dynamic IP endpoints on the client side in the future. Maybe try adding a new site as L2TP+bridging, and see if that makes the EoIP links burp. If it’s better, then try migrating.
The only thing I didn’t see in the profile was an option for split horizon - that could come in handy for some deployments.
Thank you for all your help
I will think about L2TP, maybe it will be better i the future. But for now i need to make my current setup working again. I have a GSM backup link on most of the remote sites (sim card in controler). When i set the local ip of the server on the controler it connects throu the tunnel, but i loose the GSM backup. I need to setup the routable ip of my crs on the controler for GSM and internet connection to work. Now it’s not possible because of the problems with connecting to 192.168.1.1. So i have a choice:
- internet connection, no GSM backup (servers local ip set on controler)
- GSM connection only, VERY slow (crs routable ip set on controler)
Normaly when there’s a problem with internet connection it automatically sends data through GSM and when internet is back it switches to it.
L2TP would totally fix that. The client device would only ever use the server’s internal IP, and you would remove the NAT pinhole on the main site and improve security. L2TP would be configured with the public IP of the main server, and have a userID/password. The main site won’t care what IP the connection comes from, so if it comes from GSM, fine. If it comes from terrestrial Internet connection, fine also.
The tunnel isn’t connecting through gsm. RB750 is making a eoip-tunnel to my crs, so i bridge the tunnel and ether 2-5 together and i have my local network and subnet on the remote site on ports 2-5. On the controler i set its ip on the ethernet port (ip from my local subnet) and the ip of the server it should connect to. I can’t access 192.168.1.1, so no internet. But i can access my whole subnet (all other ips from the subnet), so i can connect to the server on its local ip. GSM is a whole different connection it has nothing to do with tunnels. It connects directly through the GSM network, so i need to set the routable ip on the controler. I can connect to local ip through tunnel or routable ip through gsm network. Before these problems started, i had internet access on all remote sites through the tunnel and crs. So i set routable ip on controler and controler connected to server (crs routable ip and port redirection) through internet. When the internet connection stopped working on the romote site (no tunnel, no access to crs and internet), the controller automatically switched and send data through gsm network.
“controller” - These are the actual devices you’re supporting with this bridged network, right?
(Just want to make sure I’m following you correctly)
So when you go GSM, you put a public IP on the controller and configure it to access the server on some public IP at the main site? So the controller supports layer3 - it doesn’t have a limitation that it must be on the same LAN as the server?
Is there a reason (other than technical consequences of one big bridged network) that all Internet access should bridge to home site before going out to Internet?
You could configure ‘anycast’ on 192.168.1.1 - put this address on EVERY RB750, and also block ARP requests for 192.168.1.1 off of the tunnel. This would create split tunneling where internet goes out local connection at all sites, but access to other inside hosts will work as normal.
Finally, if you use L2TP tunnelling, then the 750 can have the public IP for GSM connection also, and it will just connect L2TP tunnel across GSM whenever normal internet goes down. Whenever normal internet is restored, GSM connection will drop and the tunnel will re-connect over the normal internet. All of this will happen automatically. Controller will not see any difference other than speeds and temporary loss of connection while tunnel detects failure and re-builds.
“controller” - These are the actual devices you’re supporting with this bridged network, right?
Yes you are right. This is a industrial controler, with analog/digital inputs and outputs for measurement sensors and controling other devices
So when you go GSM, you put a public IP on the controller and configure it to access the server on some public IP at the main site? So the controller supports layer3 - it doesn’t have a limitation that it must be on the same LAN as the server?
The GSM modem is build-in inside of the controler. Neither the modem nor the ethernet has that limitation. I just set “server ip” on it and it tries to connect. Right now with these problems i’m experiencing i can go through local address through tunnel because internet access stopped working (before i could connect through local and routable). GSM connects through GSM network so it has to have servers routable ip set.
Is there a reason (other than technical consequences of one big bridged network) that all Internet access should bridge to home site before going out to Internet?
Yes. These controllers shouldn’t be visible/accessible/detectible directly from the internet. One reason is the data they collect and we can remotelly controll devices connected to it. Soon we will be adding ip cameras on the sites and they also shouldn’t be accessible from internet. It would be preferred that at the remote site we have access to local network and internet.
You could configure ‘anycast’ on 192.168.1.1 - put this address on EVERY RB750, and also block ARP requests for 192.168.1.1 off of the tunnel. This would create split tunneling where internet goes out local connection at all sites, but access to other inside hosts will work as normal.
It would be preferred that the data is going through the tunnel.
Finally, if you use L2TP tunnelling, then the 750 can have the public IP for GSM connection also, and it will just connect L2TP tunnel across GSM whenever normal internet goes down. Whenever normal internet is restored, GSM connection will drop and the tunnel will re-connect over the normal internet. All of this will happen automatically. Controller will not see any difference other than speeds and temporary loss of connection while tunnel detects failure and re-builds.
The modem is build into the controler as a integral part of it (sim card is inside of controler). Routerboard can’t access the gsm connection.
Good reason 
I thought maybe you had GSM usb modems on the Mikrotiks.
Anyway, it looks like you’re going to have to resort to packet captures. If 192.168.1.1 does not successfully answer ARP requests from the remote sites, you’re going to have to determine where the process breaks down - do you see the requests going out EoIP tunnel at a site? Do you see them coming in the EoIP tunnel at main site? Do you see the reply coming from the CSR onto the bridge? Do you see the reply going out the EoIP tunnel unicast to the requester’s MAC address? Do you see the reply coming in the EoIP tunnel? etc - the arp request vanishes somewhere. Find out where, and that will help you determine your next step.
‘silly’ things to check before getting that detailed -
The ARP type on the master bridge didn’t get changed to listen-only did it?
There isn’t a filter rule on the bridge firewall which blocks/redirects/interferes with ARP requests or broadcasts coming from the sites, is there?
I checked more than a dozen times, just to be sure and arp is set to enable everywhere on every interface.
Did some tests today and had the same conclusion 
Arp requests aren’t getting to crs through the bridge or arent answeard by crs so i did some packet sniffing. Just to be 100% sure i just configured a new eoip tunnel from my home router (rb751u) to crs. I have a dhcp client set up on eoip-tunnel1 interface and it got a ip from crs. In ip->routes i see a dynamic route saying 192.168.1.0/24 reachable through eoip-tunnel1. But ping to 192.168.1.1 shows timeout.
Next i did some packet sniffing on the tunnel on my router. I see many arp packets with no src and dst address with source mac from one of my devices, but no arp request from my router (i got ip 192.168.1.8 from dhcp). Why don’t arp requests go through the tunnel (don’t enter it) if the subnet is accessible through it?
The same thing happens on all rb750. “Use ip firewall” is off on every bridge.
When i add a static arp entry on my router it can access 192.168.1.1. But i can’t do than on a controller.
When i do ip scan from crs, a arp entry shows up on my router and stays as long as i ping 192.168.1.1. When i stop ping, it disappears and won’t showup again and communication stops.