I use hEXPoE for some embedded PoE devices in a (mostly) disconnected field application…
These devices can deliver a lot of data, depending on some specific events/requirements.
Using a laptop over Eth works very well…
Now trying to use a tabletPC over Wifi… thefore added a simple mAP lite (in simple AP bridge mode). The limit of 100MBit Eth is acceptable in this setup…
There is NO routing involved; the WAN port of the hEXPoE is in the field normally not used.
All the test/measurements here are done with iperf3, to avoid discussions about my software.
So the setup:
HexPoE:
<eth/poe> mAP lite pc
<eth/poe> emb.device
The problem is, the data rate (from emb. device to pc) is very slow (e.g. around 30MBit).
The key seems to be that the hEXPoE does NOT use fast path inside the bridge!; just looking at the interfaces with Winbox, the traffic does NOT show FP-Tx/Rx.
The mAP lite DOES show traffic as FP!
I also have here a LAN (WAN for hEXPoE) used for aditional testing (see c,d).
Variants tested:
a. Using inversed traffic (PC → device) also shows higher rate, ~95MBit, but still no FP.
b. pc connected over eth to hEXPoE (mAP bypassed) → ~600 MBit/s, but still without FP inside hEXPoE bridge.
c. using another PoE switch (in my LAN) instead of hEXPoE → 95 MBit/s (the Eth limit of mAP lite); the mAP has no other obvious limitation, CPU is almost idle.
d. !!! connecting the embedded device to my LAN (via WAN port of hEXPoE, but the PC left on the mAP Wifi) works with full speed (>95MBit).
The hEXPoE shows the traffic as FP, between the bridge and WAN port!!!
In this case, there is also the routing/NAT involved from hEXPoE LAN to “WAN”!
Obvious questions:
Why is hEXPoE NOT using FP inside the bridge?? Of course there is no bridge rule, or IP firewall for the bridge…
Why (b) is bridging via mAP much slower than directly to Eth/PC? hEXPoE works in both cases over the same Eth port, mAP can send data much faster…
I could understand a small difference…
Why (a) is the other direction also much faster?
Rethorical question: why (d) is FP working from bridge to WAN?
FastPath is a way how the traffic is handled in the CPU, so it is not necessary between bridge ports if “hardware accelerated bridging” (in fact, forwarding of frames between switch ports without them even reaching the CPU) is enabled at both ports involved. If there is the H among the flags in the flag area (the unnamed column between row index and interface name) of /interface bridge port print output, this is the case. If the H it is not there, you must have disabled it or you must be using vlan-filtering, horizon, or some other bridge functionality that is not compatible with switch chip forwarding on hEX PoE. But if it was the case, FastPath should be involved also in forwarding between bridge ports.
Packets routed between LAN and WAN do travel through the CPU, so FastPath makes sense for them regardless whether hardware accelerated forwarding is permitted at the LAN bridge or not.
But if switch chip forwarding is enabled, it is even more mysterious why the emb.device → etherX → etherY → mAP → PC path is throtled down to 30 Mbit/s.
My experince with older (and underpowered) MT WiFi devices is that CPU is simply to weak to utilize 100% of what wireless radio can transfer. Uplink (client → AP) is easier on AP’s CPU, hence higher throughput in the opposite direction.
Another thing with mAP lite: it has quite low-power transmitter (e.g. 7 to 8 dB lower power than RB951G) and if wireless client is some distance away from mAP lite, this affects wireless performance as well.
Note that in TCP connections round-trip delay (specially so if it’s variable) affects throuhput very much. If there are many devices between both endpoints, their effect can, in worse case, multiply due to inter-device interactions.
If you read the situation, mAP is NOT the problem.
It is just 50cm away in my test, and it saturates the 100MBit eth with either external switch (c) or in routing case (d). The CPU is virtually idle.
The problem seems to be in the switching part of hexPoE
Gabriel
I suspected as much. The Atheros/Qualcomm switch chip only has a 1Mb (around 85 full-sized ethernet frames) buffer memory to handle traffic between differing speed domains so is easily overflowed leading to packet loss and retries in higher-level protocols when mixing gigabit and fast ethernet connections.
If as @sindy suggests you disable hardware offload on the bridge ports so the traffic is processed entirely by the CPU you will likely find throughput improves dramatically.
As @tdw wrote - the HW offload could only be “improved” by using a different switch chip. HW offload means that the switch chip forwards the frames between ports on its own, and since its buffer is too short, frames get dropped before the transport protocol notices the low bandwidth available and reduces the sending speed. So no software upgrade at the Mikrotik side can change this.
I understand this very well now…
But the forwarding can be de/activated individually (per port)… that means that the chip accepts some sort of setup.
It would possible to have some kind of “port groups”, and the RouterOS to dynamically create the groups depending on speed and/or other logic.
Of course, only if the switch chip supports this.
How is the “port isolation” implemented? Only in software?
I am wondering why noone suggested/asked about eth flow-control… any why is it disabled by default…
I have just activated it on the Gbit port, and it works perfectly… with HW Offload activated…
Is there any reason NOT to use it?
The switch chip accepts a setup but not a change of the memory size (nor can it borrow more RAM from the CPU) - hw=no just tells it to forward received frames always to the CPU rather than trying first to deliver them autonomously.
The rate adaptation from 1000 down to 100 Mbit/s is a specific use case; the switch chip can provide a non-blocking full-duplex gigabit throughput for unicast traffic between pairs of ports, which the CPU of the hEX PoE is unable to provide - at first place because the link between the CPU and the switch chip has only 1 Gbit/s capacity (per direction).
So I’m afraid few people would practically benefit from any “AI” algorithm adjusting the hw offload dynamically to the traffic pattern, and Mikrotik has a long backlog of bugs and more useful features to deal with…
In the particular case of hEX PoE, port isolation can be implemented both in software (see the horizon property of /interface bridge port) and in hardware (see the new-dst-ports parameter of /interface ethernet switch rule). But I’m not sure whether port isolation does what you expect it to do if you still consider it in relationship to the rate adaptation.
Regarding flow control, it is useful in this point-to-point case, but it can cause trouble in point-to-multipoint cases, so few people usually even think about it.
I was mentioning the Port Isolation just as an example of port-setup, not as a real solution.
I’m not talking about “AI” or anything else… basically I can imagine a “switch group ports based on the same link speed domain”.
Whether the chip supports this or not, I don’t know.
Small update: I can see (from my emb. device) that 2 other PoE Switches I was testing (test c) are using flow control.
I assume this is the easiest for normal LAN connections…
