Page 1 of 1

ROS under kvm with intel-iommu (VT-d directed i/o)

Posted: Fri Sep 02, 2011 2:33 am
by Anticimex
Just sharing that I managed to setup a virtualized ROS as guest on a Debian sid (with qemu-kvm from git for pci-assign).

Some data:
ASUS P8B WS -- has dual intel 82574L
Intel core-i7 2600
RouterOS 5.6 (free)

resources: ... T-d_in_KVM (don't miss adding intel_iommu=on to kernel boot options... :) )

I did this mostly to test ROS on x86 wrt windows, ipad etc to pptp / sstp etc.

Debian Sid eth0, mikrotik (debian sid's re-assigned eth1), linux laptop, windows laptop all connected to the same stupid GE switch.

Basic tests done, three groups:
A) wget to linux laptop from lighttpd @ Debian Sid, serving a 1TB sparse file.
- A1) Through NAT on mikrotik: 97% of wirespeed ( ~95kpps), ~30% cpu & irq load @ mikrotik (crashed webfig's apparently, lost pps updating data there completely)

B) Chrome web dl of same sparse file:
- B1) through vanilla(? ie what most wiki pages suggest, default-encryption etc) pptp & NAT: ~270 Mbps (~34 MB/s) -- cpu-load at client, core i5 420 - (2.4GHz) ~97% on all four cores -- cpu load at mikrotik ~ 97% as well.
- B2) B1, but taking NAT away, replacing it with Proxy-ARP between PPTP and ether1 (no IP header srcnat/masquerading), 340 Mbps (~40 MB/s) -- cpu load as B1.

C) Shooting small icmp echo reply packets from Debian Sid onto Linux Laptop, routed through mikrotik (that is, out eth0 onto GE-switch, back into eth1 on loan to mikrotik, packets arriving straight into the virtualized KVM machine, and then going back to GE switch after ROS routing, and finally out from GE switch onto Linux Laptop):
root@davinci:~# tcpreplay -i eth0 -l 0 -t /dev/shm/packets2.pcap
sending out eth0
processing file: /dev/shm/packets2.pcap
processing file: /dev/shm/packets2.pcap
processing file: /dev/shm/packets2.pcap
Actual: 37125047 packets (3638254606 bytes) sent in 103.41 seconds
Rated: 35182812.0 bps, 268.42 Mbps, 359008.28 pps

... this seems to have hit the ~limit of what I'm able to tcpreplay out... so need better methods to send more packets than that.
This resulted in a ~85% cpu / irq load on the ROS VM.

Ideas for more tests has to be quick, the free license expires soon. :)


Re: ROS under kvm with intel-iommu (VT-d directed i/o)

Posted: Fri Sep 02, 2011 8:23 am
by janisk
just shut down the test machine when you leave - it has 24 hour license and when router is not working countdown is stopped.

Also, if you by chance have several boxes around you can inside RouterOS run /tool traffic-generator with settings like this:

/tool traffic-generator packet-template add header-stack=mac,ip,udp ip-src=<address> ip-dst=<address2> interface=<outgoing> name=test

/tool traffic-generator quick tx-template=test mbps=<speed>
for packet template you can set mac addresses also, if required. As result you have quite powerful traffic gen that can push a lot of traffic (at the moment in one direction)

Also, i am interested in that part where webfig stopped working, can you shed some more details?

Re: ROS under kvm with intel-iommu (VT-d directed i/o)

Posted: Fri Sep 02, 2011 12:38 pm
by Anticimex
thanks for the hints on traffic-generator. i don't have more useful computers lying around atm however.

webfig yeah.. i started pushing traffic, ROS was sending redirects which were getting ignored; traffic kept flowing through the ROS-kvm.
then i was suddenly prompted with the webfig login screen, all the while traffic was flowing pretty heavily.
when i logged back in again, interface counters are dead in webfig. even after /system reboot. they just won't count. the CLI is counting fine however, in /interface print stats-detail, etc.

started pushing traffic again and webfig was still dead... but as i'm writing this message, i checked back and i was again prompted with the login screen in that tab.. so i log back in and now it's working. not very stable! (doesn't matter to me.. i do snmp polling for graphs anyway, and configure with cli, was just playing around with webfig)


Re: ROS under kvm with intel-iommu (VT-d directed i/o)

Posted: Mon Sep 05, 2011 11:37 am
by janisk
57 should improve on this. At least, it should not close the session so often.