RouterOS x86 vs x86_64 discussion (RE: CHR suggestions for new functionality)

Talking hardware - A 64 bit CPU has more address lines and more data lines than a 32 bit CPU — 2x . A 64 bit system performing a single read or write to memory would take 1/2 the CPU clock time required by a 32 bit system to r/w the same amount of memory. Think of it this way, you have 64 people to move. You can use a 64 passenger bus in 1 trip, or make two trips with a 32 passenger bus.

re: It’s not that simple in reality. Being that straightforward shows to me that you do not really understand how this 32bit vs 64bit stuff works
Well - actually I do understand. I’ve designed and build several Intel based motherboards & I used to design and manufacture computer interface I/O boards & I used to write really screaming fast drivers in machine code - - - a very long time ago (like almost 38+ years ago). Anybody remember a company named SYZYGY that started back in mid-late 1970s and our RS-232 interface products ? That was me.

I don’t think you are still running RouterOS on a 32-bit CPU. You are rather running x86 RouterOS on a modern x86_64 CPU. Any modern x86_64 CPU uses full width of it’s memory bus (64 bit for a single-channel DDR4) no matter what kind of OS (32- or 64-bit) you use.

but what if you need to move 32 people (like IPv4 address, for example)? what’s more efficient - use a 32- or 64-passenger bus? =)

Correct
However, even though a 64-bit hardware based system is actually moving more I/O bits/bytes in a single CPU clock cycle, the underlying 32-bit software code is still processing only 32 bits at a time in those 64-bit registers.

In a 64-bit hardware/software system, a single register or memory location can hold any combination of bits of 2 to the 64th. In a 64-bit hardware system using a 32-bit program, the program will only use any combination of bits of 2 to the 32nd.

Another reason I would like to see a CHR ISO running on a bare metal box is:

  • A hyper-visor hosting a virtual 64-bit client will have much slower raw I/O throughput than a 64-bit bare metal system.
  • To get Ethernet packets from the outside physical world to the hosted client the following procedure is taken:
    1 - The hyper-visor services the interrupt (possibly re-writing the built-in CPU cache and consuming some CPU clock cycles)
    2 - The hypervisor transfers the network data contents of the physical interface (NIC) to a software based switch and/or a software based emulated (NIC) which the host system has access to. A software driven interrupt is performed to the hosted system so that the host system can then r/w to/from the emulated NIC. (possibly re-writing the built-in CPU cache and consuming some CPU clock cycles)
    3 - By the time the virtual hosted system actually starts processing the data, some if not all of the high-speed built-in CPU cache has been re-written and many CPU clock cycles have been lost due to hyper-visor overhead. (this creates I/O propagation overhead and slows down the raw maximum throughput).
    4 - When a routine in the CPU built-in cache is no longer cached, the CPU now is forced to run at memory speed , which can be anywhere from 1/4 to 1/2 the speed of built-in CPU cache speed(because the hyper-visor re-used the CPU cache).
    5 - With a hyper-visor, there is no guarantee the some or all parts of the hosted system are not being hyper-visor swapped out in a memory-to-disk & disk-to-memory procedure while the hosted system is running. (more lost CPU cache and more propagation delay).

Anyways - I think we all agree, that we might like to see a bare-metal system that can handle multiple 10-gig or 40-gig interfaces and deliver very close to network interface throughput.

But what do I know, I’ve been out of the hardware design and machine coding business for several decades now - lol

North Idaho Tom Jones

re: I’d take it if your are only looking for something to run btest on. Comparing how fast btest is on a given hardware is just that- comparing the btest performance. Btest is known to be (a) single-threaded and (b) highly inefficient (even Mikrotik stuff have admitted that here multiple times). Result of such comparison will have no direct indication of how fast the other tasks run on the same hardware.

Re btest, if you have 10 different btest sessions running to 10 different systems at the same time, are all of the btest routines running under a single thread on a single CPU ? -or- are the multiple individual btest sessions now using multiple cores ? (or possibly multiple meta-routers with each virtual meta-router also running a btest session to something external) (I don’t know …)

Using btest on different platforms does give you a really good idea the difference in speed is on various platforms for single threaded programs. If btest was re-written to be multi-threaded and could use multiple cores, I still think a new Xeon with 3x to 5x the clock speed and 256+ meg of built-in CPU cache can blow away many other processors doing the same exact test.

So could you just install 32-bit routeros on bare metal, run tests and then upgrade it to 64-bit and recheck? :slight_smile:

But what about 10-gig network card support? Sure there is at least a 1-gig E1000 driver

you mean, there’s no 10G support in x86_64?..

I think the following:

  • x86 ROS 32-bit does have drivers for 1-Gig Intel E1000 series of network cards.

  • If you install x86 ROS 32-bit on a VMware ESXi hypervisor and use the E1000E network drivers & and push it beyond 1-gig (such as I did with the original 207.32.195.2 btest server, the TCP/IP stack will often die - although it still runs at the console , All IP traffic comes to a grinding halt.

  • CHR ROS 64-Bit does have support for 1-Gig Intel1000 series of network cards and also has paravirtual vmxnet3 drivers.

  • I don’t think CHR ROS 64-Bit has any built-in drivers for 10-gig network cards (excluding the paravirtual vmxnet3 drivers).

  • I don’t think CHR has drivers for 10-Gig network cards such as the Intel 82599EB 10-Gigabit SFI/SFP+ cards

So why don’t you just try and see exact results? :slight_smile:

Capture.PNG

Very nice
So there is some built-in 10-gig drivers for physical 10-gig cards - great :slight_smile:

Am I correct in guessing you are running this bare metal or are you running CHR under a hypervisor ?

Hypervisor — too many VMware devices :slight_smile: So you still have a chance to check x86_64 on bare metal =)

Now I just need to locate a really fast bad-ass box to try a bare metal directly running it on.

Tom, have you found the “bad-ass box” :smiley: ? I’m very curious about drivers for physical 10-gig cards..

re have you found the “bad-ass box”

lol
I am expecting to have some new SuperMicro servers here soon (with all of the fastest possible components money can buy). Although they are slated for another department’s use for something else, I am going to attempt to route then through me for a day or so and see if I can turn one of them into a test bad-ass-box running CHR directly without a hyper-visor.

North Idaho Tom Jones

news?

News on what?

News on the test that the topic author was planning to do