How to make a ten 10-Gig port x86 ROS virtual router
Required items:
One VmWare ESXi server (to host your x86 ROS operating system)
The free level-1 x86 will do just fine Optional items:
A second VmWare ESXi server
A 10-gig interface on one or both VmWare ESXi servers
A 10-gig external physical switch
Upgrade your virtual x86 ROS routers to level 5 to support 802.1q Vlans and OSPF
The problems and issues this procedure will overcome:
10-gig interfaces (E1000E) instead of 1-gig E1000 ethernet interfaces
Using more than 4 Ethernets. The problem with more than 4 x86 ethernet interfaces is the cards get moved around and what you thought was Ethernet 2 or 3 or something is renumbered to a different card. With 5 of more Ethernet interfaces, Ethernet interfaces appear broken.
Lets get started building our ten 10-gig port x86 ROS router: #1 VmWare ESXi; New Virtual Machine;
Configuration: Custom
Name: create a name
Storage: pick your datastore
Virtual Machine: Make your selection. I picked Virtual Machine Version 8
Guest Operating System: Other
Guest Operating System – Version: Other 32-bit (also type in x86-ROS in the other white box field
CPUs and ram and disk:
Use 2 CPUs & Memory Size (between 512K ram up to 2 gig ram)
Network: Add 4 Networks
SCSI Controller: select the default LSI Login Parallel
Select a Disk: Create a new virtual disk
Create a disk – Capacity: up to 8GB –with- Thin Provision
Advanced Options: Select “IDE” do not use the default “SCSI”
Finish
Now go back and edit your new virtual machine:; Remove all the “Network adapter”s
Remove the Floppy drive
Now – re edit your virtual machine and add quantity 10 ethernet adaptors using this information:
Type: E1000E
Now – re edit your virtual machine: On each network card 1-through-10 change the mac address to the following:
Network adapter 1: 22:22:22:00:00:01
Network adapter 2: 22:22:22:00:00:02
-through-
Network adapter 9: 22:22:22:00:00:09
Network adapter 10: 22:22:22:00:00:10
Now boot and install your x86. After installed – Winbox to your Ethernet1
In Winbox; rename your ethernet1 –through-ethernet10 interfaces to ethernetA –through- ethernetJ
In Winbox – interface: On each Ethernet interface perform the following:
“Reset MAC Address” – then look at the new MAC address. The last 2 digits of the new MAC address is what you need to rename your Ethernet interface to. Example 22:22:22:00:00:05 would be ether5 You are almost finished
Now shutdown your x86 ROS and edit the virtual machine. Change the MAC address from Manual to Automatic for each network interface.
Re-boot your x86 ROS. And again reset the MAC Address for each of your 10 network interfaces.
You are done: You now have a quantity 10 port all 10-gig x86 ROS router which can route to other local virtual machines or to other external outside machines at 10 gig. OOOO – yea !!!
At this time, I would like to suggest you get a free level 1 license or even better yet, purchase a Mikrotik level 5 license. With the level 5 license you can use 801.1q vlans on every interface and also use OSPF and other functions.
Mikrotik – thank you for a great ROS and hardware and great software.
If you found this informative and helpful – please post your findings. If you build it, please post your bandwidth throughput you are able to achieve.
With VMware, I would start out with 2 CPUs & 2 Gig of ram. After you build it. you can always shutdown, tweak the setting up or down and reboot. I usually use 1 CPU and 1 Gig of ram -or- 2 CPUs and 2 Gig of ram.
My VMware ESXi boxes all have the latest greatest and fastest Intel CPUs with the largest amount of CPU cache available on an Intel CPU.
I get 17 Gig udp throughput on a bandwidth test to 127.0.0.1 -however your throughput may be faster or slower depending on your physical hardware platform.
Note: So far, I have not seen any advantage in adding multiple CPUs. But there is a huge advantage in using a good Intel CPU to start with. If you can get the entire x86 ROS to run completely in CPU cache, it is worth going for it. Running in CPU cache is up to 100+ times faster than running in RAM memory.
All of my Physcal servers are using a SuperMicro chassis. Each one has two physical CPUs. Each CPU has 10 cores. And each core has hyper-threading enabled. So in all total, it looks like I have 40 CPUs.
My VMware reports the following:
CPU Cores: 20 CPUs
Processor type: Intel Xeon CPU E5-2690 @ 3.00 Ghz
Logical Processors: 40
RAM: 131 Gig of ram memory (note expandable up to 256 Gig of ram).
HDD - I am using Segate sas6 4TB hard drives with the largest amount of disk cache available in each drive.
My typical VMware ESXi server has up to 16 TB HDD - and my FreeNAS NAS server has 54 TB of hdd space using the ZFS-2 file system with real-time LZ4 file/disk compression (with still plenty of empty drive bays)
All of my physical servers have three 1-gig network cards and two or more of the Intel dual-10-gig network cards.
I pretty much do not use any 1-gig interfaces any more. I only use 10-gig interfaces internally and externally with 10-gig physical & virtual routers -and- 10-gig physical switches. Except I do use a 1-gig interface for my IPMI (virtual keyboard-mouse-video) so that I can get into the physical machine BIOS if I am somewhere out in the field and not in the office.
What is nice about VMware ESXi, it that I can software shutdown/power-off a virtual server and make a complete copy/clone on my NAS server. Thus if I ever crash one of my VMware ESXi servers, I just copy the clone of the host machine I want to restore from my NAS to another of my VMware ESXi servers. Total backup time to fully clone a typical hosted server is usually under 5 minutes.
Note: When a virtual host is writing to my FreeNAS server, it will sustain 8 gig while writing huge files.
Although you can get Xeon CPUs with more cores, I went with the 10 core because it had the largest amount of internal CPU cache memory.
Most of my VMware servers are running 15 to almost 40 virtual servers. My VMware ESXi boxes report very little CPU load - I would suspect I can take some of my ESXi servers up to 80+ virtual machines.
In answer to your question about what I recommend - I recommend doing the exact same thing:
-two or more SuperMicro servers running VMware ESXi with all virtual hosted machined running directly on the local datastore.
-one (or two) FreeNAS servers which provides either NFS or ISCSI mounts to your ESXi VMware servers for external datastores & and where to backup to.
Forget the price - just go for the best & fastest you can get. It is better to have 3 of the fastest servers you can get instead of 10 not-so-fast servers.
At North Idaho College we use quite a few virtualized RouterOS instances, but the 1gb limitation (and lack of a traditional support model, ala Cisco TAC) has kept us from using them for much more than VPN gateways or remote site routers.
I’m interested to see what the performance overhead is using 10gb emulated NIC.
I wish MikroTik would start selling a supported vSphere/Virtualized edition of RouterOS with vmxnet3 and vmware tools integration. I would imagine using the paravirtualized vmxnet3 would be a lot more efficient at 10gb speeds than than the emulated adapters. MikroTik, if you are listening, we’d easily pay $200 an L4 for license for this.
Just to add to the conversation on specs for a RouterOS VM, here is what we are using (typically):
2 x vCPU
256MB of vMem
256MB vmdk (thin provisioned)
1 - 4 E1000 NIC’s, depending on the number of VLANS/subnets we need to connect to.
On our highest bandwidth site-to-site virtual RouterOS deployment, between Coeur d’Alene and Post Falls, we use queues for traffic shaping and VoIP / Video QoS at rates up to 150mbps and as I recall, I have never seen the CPU hit more than 15-20% or so.
Looks like my E1000E virtual adapters are only connecting at 1gbps… Maybe I missed something, but could you check the link status on yours and see if you are seeing 10gbps connectivity? Based on your post I think the only difference config-wise is that the virtual-machine version of my test VM is 10, rather than 8 as I believe yours was.
Here’s what I am seeing:
[admin@MikroTik] /interface ethernet> prin det
Flags: X - disabled, R - running, S - slave
0 R ;;; VPN Access & Mgmt
name="ether1" default-name="ether1" mtu=1500 l2mtu=9014 mac-address=00:50:56:87:A0:CA orig-mac-address=00:50:56:87:A0:CA
arp=enabled disable-running-check=yes auto-negotiation=yes advertise=10M-half,10M-full,100M-half,100M-full,1000M-full
full-duplex=yes tx-flow-control=off rx-flow-control=off cable-settings=default speed=1Gbps
1 R name="ether2" default-name="ether2" mtu=1500 l2mtu=9014 mac-address=00:50:56:87:4F:9E orig-mac-address=00:50:56:87:4F:9E
arp=enabled disable-running-check=yes auto-negotiation=yes advertise=10M-half,10M-full,100M-half,100M-full,1000M-full
full-duplex=yes tx-flow-control=off rx-flow-control=off cable-settings=default speed=1Gbps
[admin@MikroTik] /interface ethernet> mon 0
;;; VPN Access & Mgmt
name: ether1
status: link-ok
auto-negotiation: done
rate: 1Gbps
full-duplex: yes
tx-flow-control: no
rx-flow-control: no
advertising: 10M-half,10M-full,100M-half,100M-full,1000M-full
link-partner-advertising:
default-cable-setting: standard
[admin@MikroTik] /interface ethernet> mon 1
name: ether2
status: link-ok
auto-negotiation: done
rate: 1Gbps
full-duplex: yes
tx-flow-control: no
rx-flow-control: no
advertising: 10M-half,10M-full,100M-half,100M-full,1000M-full
link-partner-advertising:
default-cable-setting: standard
[admin@MikroTik] /interface ethernet>
I attached a screenshot of the network adapter config in vSphere as well.
vRouter_cfg.png
Let me know if I missed a step!
My x86-ROS also shows 1Gbps when I look at the Ethernet Status. However, When I performed a bandwidth test between two physical different VMware servers connected by a 10 Gig link, I was able to get almost 10 gig from X86 to my other x86 in the other ESXi box. Both x86-ROS were using E1000E network interfaces. Both ESXi boxes had Intel 10-Gig physical network cards running through a Cisco 4500 10-Gig switch.
Note: Also - I found out that NOT using Vlans on the x86-ROS is a little faster than using 802.1q tagged vlans. I suspect this is because of the additional bytes in the packets for the 802.1q encapsulation of the trunked vlans.
What do you get when you do the following:
udp bandwidth test to 127.0.0.1
udp bandwidth test to another x86-ROS hosted on the same physical ESXi server
udp bandwith test to another x86-ROS hosted on another ESXi server (hopefully with 10-Gig physical network cards between your two ESXi servers.
edit - note: One thing I almost forgot about, I did go into my ESXi configs and changed all of the buffer sizes to the maximum size permitted in VMware. Also, I think I also configured my VMware servers to use Delayed Ack. It might be these tweaks also made a difference - I don’t know for sure.
edit - note: Another thought - A virtual network card does not have the same throughput limitations that a physical network card has. The only limiting factor for a virtual network card is how well the drivers are written and how well the VMware ESXi can handle the software network emulation of a network card. However Vmxnet3 would still be faster if Mikrotik had driver support.
Do you know Martin H. there with North Idaho College? Ask him about me - Tom Jones. We worked together at the CDA Tribe prior to him going to work at NIC.
I might like to talk to you on the phone and exchange information about how you use x86-ROS and PfSense. And possibly talk about what you use for a high-end NAS solution & how you rate bandwidth limit networks and individual computers and how handle Captive Portal -aka- Walled Garden environments using a Radius database.
