Hi,
I want to setup layer2 spanning over layer3.
I see that IPIP has the lowest overhead of any protocol available out there including those of VPNs and ones like VxLAN and EoIP.
What can VxLAN and EoIP can do that IPIP tunnel can not do? If none then why people want to use VxLAN or EoIP while IPIP is the fastest and very good solution.
Thanks,
The complexity and overhead (header size) is usually pretty directly related to how complicated of a problem it is that the tunneling mechanism is being designed to solve.
IPIP is a really simplified tunneling mechanism. Encapsulating IP-in-IP really is meant for devices acting as a router because the only thing you’ll find past the IP header is … the next IP header. Furthermore, the protocol uses the IP protocol field to indicate the next header so that the endpoint can know to recirculate the inside packet again for a routing decision. In summary, IP-in-IP is ultimately point-to-point and is incapable of emulating Ethernet at all.
Putting an Ethernet frame or some other type of non-IP PDU into an IP packet sounds easy enough. Maybe you could use the IP protocol to indicate that the next header is Ethernet. If you do that, you’re only limited to one Ethernet for a pair of IP addresses. How else is the recipient supposed to know which LAN the Ethernet frame is supposed to be associated with? That’s limiting. Also, you might want to send a number of other types of PDUs in this way … do there really need to be X-on-IP protocol numbers reserved for every possible tunneled PDU type? That’s also limiting. If you meet these challenges, you basically have reinvented the most important header fields defining GRE: a field to tell you what’s going to be inside and a key field so you can separate tunnels from each other. Why bring up GRE? Mikrotik EoIP is just Ethernet-over-GRE-over-IP [1]. Still, GRE is point-to-point [2] and can emulate an Ethernet segment, but really only one link.
VXLAN might as well be a VLAN tag in a lot of respects, but it’s designed to be carried over IP and to take advantage of hardware TCP/UDP offload. It’s also not designed to be point to point like GRE and IPIP were. This means it can support Ethernet BUM traffic (broadcast, unknown, and multicast) by way of multicast in the underlying transport network. It also has a much bigger identifier field, yet again adding to the size of the header. In summary, VXLAN is fully multipoint and can fully emulate an Ethernet segment, not just one link of it.
You didn’t include it, but pseudowires and VPLS are really just the GRE and VXLAN of the MPLS universe. Pseudowires are simple and work okay for building point-to-point non-IP connectivity that can emulate an Ethernet link over the MPLS core. VPLS is fully multipoint but is more complicated as a result.
Despite the number of choices you have … make absolutely sure you really need to be emulating Ethernet at all. Very few things really require it. Ethernet has been asked to accomplish terrible things beyond the wildest imagination of its original design requirements and the greatest gift you can give yourself is to not have to do any of this at all 
[1] https://wiki.mikrotik.com/wiki/Manual:Interface/EoIP
[2] Well okay Cisco supports multipoint GRE but it’s not really a change to the protocol that’s mostly a control-plane hack to make it easier to administer hundreds or thousands of GRE tunnels.
By one link you mean GRE can be one tunnel? (maybe link is the right term). I think you mean I can’t have multiple GRE connections running at the same time. If so that will be limiting to me because I want to make four tunnel and bond them too.
Sounds like VxLAN does what I need vs IPIP because IPIP can’t emulate ethernet you said - correct?
VPLS I was told I need MPLS running which I can’t do with this network.
Interesting. So Cisco can do multiple GREs but Mikrotik can only do one tunnel. What is a “control-plane” hack?
I don’t think I have many choices. Please see below requirements and let me know what options I have:
My network diagram:
Branch A Los Angles client router => ISP-X point to point fiber 100Mbps/100Mbps => Chicago datacenter => third party ISP-Y rents me space in data center and ip transit (my Mikrotik-A is here) => ip transit => 4x Cable coax ISP 360Mbps/30Mbps each => Toronto (my Mikrotik-B is here) => Branch B client router
*Client expects Branch A to be connected to Branch B using layer-2 or have that experience. Doesn’t matter how it is delivered.
My requirements:
Reading your scenario, I get why you’re investigating how to do it. I mostly try and discourage people because they look at the features/documentation, see that it’s possible, and then end up stitching LANs together for “convenience” rather than for a solid reason. If you’re providing essentially an Ethernet virtual circuit service to a customer, that’s what you have to provide and one of these is likely going to be the answer.
IPIP is out because it just can’t transport non-IP packets as the next header by design. All of the others are still on the table and it sounds like you currently have EoIP (basically Ethernet over GRE) in place. Frankly, if I was just doing this for a few customers, I’d do it exactly the way you’re already doing it with EoIP.
Even though the terminology can be imprecise, it’s also important to establish that what you’re really doing is a virtual circuit service that happens to use Ethernet framing. Fully emulating Ethernet over a provider network requires contending with the reality that “native” switched Ethernet is a multipoint protocol. Frames addressed to the broadcast or multicast MAC addresses are replicated by native transport hardware (usually a switch) and flooded out all of the relevant ports:
If your client asks for three or more branches to share this same Ethernet segment, then that’s when things become more complicated. This is where my thing about GRE not being a multipoint protocol comes in. If you are trying to fully emulate Ethernet with GRE, then each branch office would need its own GRE tunnel to one central router that will act as a bridge. If the sites are really far apart, then this can be a real pain:
If you use something like VXLAN, then you either have to support multicast routing in the provider network or your VTEP hardware has to support head-end replication. RouterOS does not appear to support head-end replication, so multicast would be a requirement. On top of that, really a multicast RP is still just the root of a tree, so you’d have to pick a router in the middle that was an acceptible location for the replication to happen. That may or may not be any better than the GRE hub and spoke. Alternatively, you could have routers that support an anycast RP but that also doesn’t appear to be supported in RouterOS. That would look something like this:
It gets complicated fast. I would hold the line with the client, use EoIP like you’re already doing, and only provide point-to-point virtual circuit service if at all possible.
The rest of the situation with trying to bond multiple connections is a completely different wrinkle in this. Can you produce a quick sketch of the routers and links in the entire path between these two branch offices? I’m not really sure how much this is going to factor into what tunneling mechanism you choose, though.
Start with the definitions, then try working the answer out on your own. If you succeed, you will know what a control-plane hack is and why Cisco’s alternative to this works. If not, then it won’t matter. 
Regardless, it’s off-topic here, since RouterOS can’t do that.
How do you square this fact with your wish for no encryption? If I had private WAN traffic transiting a commercial data center, I’d want it encrypted six ways from Sunday. Ditto with your cable ISP.
If you can give up on that requirement, and you’re willing to choose ARM-only RouterOS hardware, your problem sounds like an ideal application for RouterOS 7’s ZeroTier feature.
Have the least overhead…Be the fastest
I don’t see how you’re going to get less overhead and faster I/O than EoIP, short of leased lines.
Should I bond the 4 WAN Coax Cable internet connections together or bond the tunnels together (be it EoIP, VxLAN, or any other)?
Are we talking single-stream or many users collectively achieving the “100Mbps upload and download” rate you specify elsewhere?
Getting bonding to yield aggregate data rates faster than a single line when there aren’t at least as many simultaneous users as the number of lines is difficult, particularly with protocols like TCP where packet order is important to performance.
Bonding is better used when your goal is redundancy rather than increasing aggregate bandwidth.
Given your “requirements” L2 connectivity … Doesn’t matter how it is delivered. Does that mean they would accept a link via a Geostationary satellite, with at RTT of ~500 ms (one half second)?
Is the LA to Chicago link in place? Is the customer satisfied with the performance?
You can emulate the ethernet, but you won’t be able to reproduce the low latencies normally expected with a LAN, there’s that pesky “C” problem that can’t be overcome. And the velocity factor of glass and wire is lower than 1.
Here’s what I am talking about: Global Ping Statistics → Los Angeles and Chicago So as of 2022-04–04 58ms RTT is the floor. That’s ok for remote desktop, but would be terrible for applications designed for 0.1ms RTT. So you may want to use a linux system with two interfaces and tc netem to emulate a high latency link to verify that the applications will work acceptably with the higher latency.
It would be better to make sure they understand what is possible, as many people think that they can solve latency problems by adding bandwidth. The only time extra bandwidth helps is when there are delays due to queuing. There is no overcoming speed of light other than reducing the distance or increasing the velocity factor to closer to 1 (hollow-core fiber).
Or doing the things that really require low latency locally, and using some remote access solution to access the “results”.
We had a case where users were trying to access a database remotely with odbc and they wanted a “higher speed” network. The solution was to run the app on a VM that was near the data source, and they accessed that vm with remote desktop. They were much happer.
I’ve been telling myself that the clients in question actually just want a virtual circuit to connect branch routers. Most of the networks that want point-to-point layer 2 just want a cheaper alternative to a dedicated wave and that they don’t have to do anything gross like BGP with site-of-origin. Note: “telling myself”. Who knows what they want 
.
If they really want “a LAN” though, absolutely @Buckeye is right re: latency. The Wikipedia article on the bandwidth delay product [1] and the US ESnet resources on long-fat-networks (LFNs) [2] will be enlightening reading if you haven’t had to dig this deep into performance before.
Regarding the link bonding thing, I agree that @tangent is right to be skeptical and they mostly covered why. What I will add though is that even if you decide that there is enough bulk (not single flow) traffic to warrant it, I would be very careful to test the supported flow hashing characteristics of the hardware and software of the routers in question there.
GRE is famous for hashing poorly across bond/LAG/portchannel members because most strategies can only look at layer 2, 3 and 4 headers — and usually, layer 4 is TCP and UDP only. For one tunnel of GRE without explicit software/hardware support this usually means that the entire tunnel hashes to the same link. If you add more tunnels, it still won’t matter because the difference is going to be in the GRE header.
From a brief glance at the available hash algorithms in the RouterOS documentation here [3], this is probably the case across Mikrotik’s entire lineup. I have not tested but if you go down this road, test test test and beware. VXLAN won’t do better in a point to point scenario because even though the hash could operate on the UDP outer header, all traffic will still be headed to the same destination VTEP address .
(And to be clear, I’m not pointing a crooked finger at Mikrotik about this. I’ve only really seen GRE or VXLAN header support in flow hashing on boxes ten or a hundred times the price.)
[1] https://en.m.wikipedia.org/wiki/Bandwidth-delay_product
[2] https://fasterdata.es.net/
[3] https://help.mikrotik.com/docs/display/ROS/Bonding#Bonding-balance-xor
ZeroTier would be your best bet here