Traffic Generator - Big vs small packets (strange) results

RouterOS General
1

I am currently doing some Traffic Generator tests using hEX routers and I am kind of puzzled with results when dealing with big packets (i.e. 1500 bytes) vs small ones (i.e. 100 bytes).

Overall setup is quite simple: one hEX is used as Traffic Generator and the second one is used as switch (switching incoming traffic back to the Traffic Generator - so we can get the Rx info).

IP setup:
First hEX has IP = 169.254.0.201
Second hEX has IP = 169.254.0.100

Packet template setup:
Type: TCP
Source IP = 169.254.0.200 (dummy IP just for the tests)
Destination IP = 169.254.0.201
Gateway IP = 169.254.0.100 (just to be 100% sure that we are forcing traffic towards second hEX)

For the purpose of this tests, both hEX devices are connected with single 1m CAT6 UTP patch cable.

Now comes the puzzling part. I am including two Traffic Generator report with different packet sizes:

Traffic Generator output for 1500 bytes packets / 100Mbps / no PPS limit:

SEQ    ID      TX-PACKET   TX-RATE     RX-PACKET   RX-RATE
...
63     0           8 333  99.9Mbps         8 333  99.9Mbps
64     0           8 335 100.0Mbps         8 333  99.9Mbps
65     0           8 332  99.9Mbps         8 334 100.0Mbps
66     0           8 333  99.9Mbps         8 333  99.9Mbps
67     0           8 334 100.0Mbps         8 334 100.0Mbps
68     0           8 333  99.9Mbps         8 333  99.9Mbps
69     0           8 334 100.0Mbps         8 334 100.0Mbps
70     0           8 333  99.9Mbps         8 333  99.9Mbps
71     0           8 333  99.9Mbps         8 333  99.9Mbps
72     0           8 333  99.9Mbps         8 333  99.9Mbps
73     0           8 334 100.0Mbps         8 334 100.0Mbps
74     0           8 333  99.9Mbps         8 333  99.9Mbps
75     0           8 334 100.0Mbps         8 334 100.0Mbps
76     0           8 333  99.9Mbps         8 333  99.9Mbps
77     0           8 334 100.0Mbps         8 332  99.9Mbps
78     0           8 332  99.9Mbps         8 334 100.0Mbps
79     0           8 334 100.0Mbps         8 334 100.0Mbps
80     0           8 333  99.9Mbps         8 333  99.9Mbps
81     0           8 333  99.9Mbps         8 333  99.9Mbps
TOT    0         674 993  99.9Mbps       674 993  99.9Mbps

(Everything looks OK here - Tx and Rx are the same.)

Traffic Generator output for 100 bytes packets / 100Mbps / no PPS limit:

SEQ    ID      TX-PACKET   TX-RATE     RX-PACKET   RX-RATE
...
35     0         125 000 100.0Mbps       125 000 100.0Mbps
36     0         125 002 100.0Mbps       124 962  99.9Mbps
37     0         125 000 100.0Mbps       124 937  99.9Mbps
38     0         125 000 100.0Mbps       124 994  99.9Mbps
39     0         124 941  99.9Mbps       124 863  99.8Mbps
40     0         125 058 100.0Mbps       125 063 100.0Mbps
41     0         125 003 100.0Mbps       124 987  99.9Mbps
42     0         125 000 100.0Mbps       124 987  99.9Mbps
43     0         124 999  99.9Mbps       124 984  99.9Mbps
44     0         124 996  99.9Mbps       124 999  99.9Mbps
45     0         125 004 100.0Mbps       124 988  99.9Mbps
46     0         124 998  99.9Mbps       124 977  99.9Mbps
47     0         125 003 100.0Mbps       125 000 100.0Mbps
48     0         124 829  99.8Mbps       124 747  99.7Mbps
49     0         125 172 100.1Mbps       125 183 100.1Mbps
50     0         124 997  99.9Mbps       124 980  99.9Mbps
51     0         125 001 100.0Mbps       124 967  99.9Mbps
52     0         124 997  99.9Mbps       124 952  99.9Mbps
53     0         125 001 100.0Mbps       124 993  99.9Mbps
TOT    0       6 624 817  99.9Mbps     6 623 899  99.9Mbps

(Note the difference between Tx and Rx packets in this example.)

Question is: why is this happening? Or more specifically: can first hEX (Traffic Generator) even capture all those small packets at this rate and generate reliable report? Or are these numbers kind of “misleading”? Meaning: the packets are actually arriving back, but they are not properly counted in? Is there any packet size / link speed / PPS data sheet available for hEX I could check / refer to?

2

To “properly” test a router, you need to test traffic / packets through the router, not generate traffic from the router, traffic generator is just to use to give some guideline and see if there is something major wrong, i.e. you get 50Mb/s instead of 100Mb/s, etc

Post results as through the router then we can see if there is a problem or not

3

CZFan, I am not interested in “router testing” per se. What I am really trying to do is understand the Traffic Generator capabilities of MikroTik. Unfortunately, there isn’t much materials on that topic online. That’s why I designed this small-scale test-rig. I’m looking to get some answers.

It seems to me (solely based on hunch) that I managed to hit CPU-capabilities wall in my Traffic Generator tests… But I do not have a solid way to confirm that. Looking at just lost packets, it isn’t clear if packets are really lost or if it’s CPU’s capture fault. Any ideas how to possibly check that?

4

@dadox, can you briefly describe what is so puzzling for you?

Update: ok, got it: you mean the difference between Tx and Rx packets in the 2nd table…
Easy explanation: some “TCP resend” packets occured, that’s IMO normal.
Similar differences are present also in 1st table, maybe you overlooked them.

5

I am fine with Tx / Rx not matching perfectly in each row due to resend, but if I am understanding MikroTik’s logic correctly, the TOT row should count that in. (Right?)

That being said:

In first case, we have…

TOT    0         674 993  99.9Mbps       674 993  99.9Mbps

…and that looks OK - 0 packets are lost.

In the second case, this…

TOT    0       6 624 817  99.9Mbps     6 623 899  99.9Mbps

…looks much worse - there’s 918 lost packets.

To sum my questions:

  1. Are those packets really lost?
  2. If not, how can I differentiate lost packets from un-captured ones (meaning mostly: is there any way to track CPU usage regarding packets generation / capture)?
  3. Is there a better way to setup such Traffic-Generator test-rig (maybe I am missing something crucial here)?

Like I said in OP, documentation on this particular topic is unfortunately quite minimalistic… So any help is much appreciated.

6

Traffic generator just sends frames - even if TCP template, there is no protocol state machine to manage connection setup, retransmissions, connection teardown, etc.

I looked at your numbers for % loss and note that it is around 0.01%. So ‘much worse’ depends on what you need for your use and the problem you are trying to solve. For $60 or so for a Hex (retail price on the Mikrotik website), I consider this pretty good. You can always buy an Ixia packet generator for $50K+…

Did you check the DuT switch and see if it dropped any of the frames? It may not be traffic generator (though I would suspect this, too, as it is done in software).

7

“Much worse” = comparing it to the 1st set of results (with 0% lost packets). That test proves that the cable and both hEX devices are capable of 100% correct communication using that particular set of packet templates.

The problem with the second test is that, and it seems that you confirmed my suspicions, there is no way of knowing exactly where things actually start to choke. Is it the seer number of packets, some buffers, CPU, overhead, or anything else. I was kind of hoping there might be some workaround to get this info.

I guess I have no other choice than try to play with packet templates until I got some working PS / LS / PPS templates that I’m sure MikroTik can handle?

P.S.
I am not trying to bash MikroTik here for the lost packets, quite contrary - my hats off to them for manage to pack such versatile and low-priced devices. I am just trying to learn more about Traffic Generation limitations, so I can design some testing scenarios and be sure that the lost packets are lost due to network problems and not hEX (or any other MikroTik) devices themselves.

8

The testing method you are trying to design is fundamentally flawed, you placing a shit load on the CPU of the router by using traffic generator, so with this heavy load, you can expect packet loss, etc.

As I said before, test “through” the router using tools i.e. iPerf, etc