Community discussions

MikroTik App
 
User avatar
sebastia
Forum Guru
Forum Guru
Topic Author
Posts: 1802
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

hEX block diagram

Fri Dec 15, 2017 7:27 pm

Hi

On hEX (RB750Gr3) product page there are two block diagrams, one with disabled and one with enabled switching.
Qs:
why two, is that a user option?
if so, how to do it?
and why choose one over the other? What are the (dis)advantages of each?

Thanks
Seb
 
User avatar
sebastia
Forum Guru
Forum Guru
Topic Author
Posts: 1802
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: hEX block diagram

Fri Dec 22, 2017 2:04 pm

I've contacted support, but info received was insufficient

Anyway, what I guess:
the two modes of operation may be chosen, with 'disabled switching" and "enabled switching
see https://mikrotik.com/product/RB750Gr3#fndtn-downloads

It is a user configurable, depending whether or not two or more ports are "switched" together.
There is a dependency on the version of RouterOS though:
* pre 6.41: if at least one port is slave to a master -> "enabled switching" applies
* post 6.41: if at least two ports are bridged in hw(=yes) -> "enabled switching" applies
* otherwise "disabled switching" is effective.

edit: removed dead links
Last edited by sebastia on Fri Mar 12, 2021 12:09 pm, edited 1 time in total.
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Fri Mar 12, 2021 9:03 am

Sorry to bump such an old thread but after an ISP upgrade and currently using a RB750Gr3 here, I wanted to give the things mentioned above a shot, and well, not true.
With ethernet1 out of any bridges, and with only one single bridge present with ports 2,3,4 & 5 added to it, all hardware offloaded, the "Block Diagram with disabled switching" seems to apply.
So if you have an ISP that gives you over 500Mbps up&down you have to use port 1 for wan (example) and port 2 & 4 for LAN.
Or maybe use one of port 2 or 4 for WAN and use 1,3 and 5 for LAN, that way you have more LAN ports. *WRONG, see below: viewtopic.php?f=3&p=848197#p848151
Any way to change this behaviour? I guess not.
But worth mentioning somewhere instead of leaving the user wondering where the bottleneck is.
RouterOS: 6.46.8
Below some fast tests made with iperf:
eth1-eth2.PNG
eth1-eth3.PNG
eth1-eth4.PNG
eth1-eth5.PNG

interface bridge port print 
Flags: X - disabled, I - inactive, D - dynamic, H - hw-offload 
 #     INTERFACE              BRIDGE         HW  PVID PRIORITY  PATH-COST INTERNAL-PATH-COST    HORIZON
 0   H ;;; defconf
       ether2                 bridge        yes    1     0x80         10   10       none
 1 I H ;;; defconf
       ether3                 bridge        yes    1     0x80         10   10       none
 2   H ;;; defconf
       ether4                 bridge        yes    1     0x80         10   10       none
 3 I H ;;; defconf
       ether5                 bridge        yes    1     0x80         10   10       none
You do not have the required permissions to view the files attached to this post.
Last edited by Znevna on Sat Apr 17, 2021 10:02 am, edited 2 times in total.
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: hEX block diagram

Fri Mar 12, 2021 10:37 am

This is the real situation and if you enable switching (bridge) then all work is still done (emulated) in the processor. To the processor you see two lanes of each 1Gbit/s. When using port 1 as WAN then ports two and four provide maximum speed. Ports three and five have to share the 1Gbit/s with port one (WAN).

When plugging looking at the traffic to/from the WAN and best would be 1<->2-4 and less 1<->3-5.

In the hEX-S it even worse if you use the SPF as WAN. Less suited, SPF<->1-2-3-4-5 (1,2,3,4,5 share the same 1Gbit/s lane)

Even with it's limitations it is still a very nice little box.
hEX.JPG
You do not have the required permissions to view the files attached to this post.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Fri Mar 12, 2021 10:43 am

Yes, but we also have the Enabled Switching diagram, which looks like this:
https://i.mt.lv/cdn/product_files/RB750 ... 190642.png
hEX RB750Gr3 - Enabled Switching.PNG
How can we use that? I wrote to support about it, we'll see if we can do anything about it.
You do not have the required permissions to view the files attached to this post.
 
User avatar
sebastia
Forum Guru
Forum Guru
Topic Author
Posts: 1802
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: hEX block diagram

Fri Mar 12, 2021 12:18 pm

you are using it (1Gb/s) already!

see second test:
Image
Tx + Rx ~1Gb/s for ports ether1 & ether4

the 1Gb/s from diagram is TOTAL bandwidth available, for BOTH sending and receiving

I expect an apology now ;-)
 
User avatar
Paternot
Forum Veteran
Forum Veteran
Posts: 912
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: hEX block diagram

Fri Mar 12, 2021 12:22 pm

There isn't a fixed attribution of links.

1) If we are talking about 5 independent ports: the two 1Gbps links will be used, as needed. There is no hard assignment of a link to a group os ports.
2) If we are talking about a mix of independent ports and slave ports: One 1Gbps link is given to the independent ports, the other is given to the grouped ports.

I'm not sure how it would handle, if someone made two or more bridges. This isn't a recommended config, anyway.

My setup: ISP on ether5, the other four ports grouped under one bridge.
hEX_speed.png
Edited to correct typos.
You do not have the required permissions to view the files attached to this post.
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Fri Mar 12, 2021 12:33 pm

you are using it (1Gb/s) already!

see second test:
Image
Tx + Rx ~1Gb/s for ports ether1 & ether4

the 1Gb/s from diagram is TOTAL bandwidth available, for BOTH sending and receiving

I expect an apology now ;-)
By your logic, in two of my tests (1 and 3, eth1 wan - eth2 lan; eth1 wan - eth4 lan) I have two ~1.6Gb/s links, with a total of 3.2Gb/s to the CPU, that isn't in the diagrams.
The 1Gb/s links are full duplex.
Explain otherwise.
PS: shame that after 3 years since you've posted this you came with that explanation.

@Paternot your posted speedtest is one way at a time, I covered all the ports and tests in the screenshots posted above.
Also the Block Diagram posted above (disabled switching) seems pretty accurate to my findings. https://i.mt.lv/cdn/product_files/RB750 ... 161117.png
Having WAN on ETH5(or 3 or 1) will give best up & down simultaneous speed with LAN on ports 2 and 4, you can test that with two speedtests, starting another one just before the current one ends, on different servers maybe, so that one of them does the upload part and another one the download part.
If you switch your testing machine with the current setup on ports 3 or 1 you'll see half of the simultaneous transfer speed. (you'll still get 1Gb/s up OR down, though, so running just ONE speedtest will not show that.)
Like this, here I was doing just that, two speedtests:
RCS-RDS Speed Test 20210312.PNG
You do not have the required permissions to view the files attached to this post.
 
User avatar
sebastia
Forum Guru
Forum Guru
Topic Author
Posts: 1802
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: hEX block diagram

Fri Mar 12, 2021 12:44 pm

in tests 1 & 3 you don't go to cpu, but are using the hardware switching in the switch chip (= off-loading) -> hence the limitations of the link to cpu don't apply and you get full bandwidth of the 1Gb/s connection / port

So how about that admission :-D ?
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Fri Mar 12, 2021 12:48 pm

facepalm.
All the tests are done with a routing setup, not switching. See the screenshot above. CPU usage ain't for graphics.
 
User avatar
sebastia
Forum Guru
Forum Guru
Topic Author
Posts: 1802
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: hEX block diagram

Fri Mar 12, 2021 12:49 pm

1) If we are talking about 5 independent ports: the two 1Gbps links will be used, as needed. There is no hard assignment of a link to a group os ports.
Don't believe that to be the case: I think the port attribution of all independent links is fixed, (but I haven't tested it...)
 
User avatar
sebastia
Forum Guru
Forum Guru
Topic Author
Posts: 1802
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: hEX block diagram

Fri Mar 12, 2021 12:54 pm

The 1Gb/s links are full duplex.
Just for posteriority, this is NOT the case: Mikrotik always reports full bandwidth over all directions -> that 1Gb/s is shared for both directions!
(as supported by your own tests)
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Fri Mar 12, 2021 12:56 pm

Ok then, for Pete's sake, explain the screenshot above if you please, with the PPPoE client.
Take ether1, 529+908= ?? :) explain.
 
User avatar
sebastia
Forum Guru
Forum Guru
Topic Author
Posts: 1802
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: hEX block diagram

Fri Mar 12, 2021 1:16 pm

I you want some assistance or information you should be a bit more polite. Most of us on this forum are not here because we are paid for it.

And how would you explain it then, considering that this test goes right against the results of your tests number 2 & 4 from you first post here???
both were full duplex tests across two involved ports
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: hEX block diagram

Fri Mar 12, 2021 1:26 pm

Yes, but we also have the Enabled Switching diagram, which looks like this:
https://i.mt.lv/cdn/product_files/RB750 ... 190642.png
hEX RB750Gr3 - Enabled Switching.PNG
How can we use that? I wrote to support about it, we'll see if we can do anything about it.
The magic word is "integrated". Integrated in the CPU I assume. When you route (WAN), then the CPU is always used. Port one to CPU and traffic to three have to share the same lane.

When traffic stays in the bridge then the switch in the CPU will direct the traffic and is offloading in hardware. Don't put your WAN in the bridge that is that I am told.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Fri Mar 12, 2021 1:29 pm

I you want some assistance or information you should be a bit more polite. Most of us on this forum are not here because we are paid for it.

And how would you explain it then, considering that this test goes right against the results of your tests number 2 & 4 from you first post here???
both were full duplex tests across two involved ports
So I guess you can't explain the data that destroys your theory about the 1Gb/s links beeing a sum of upload+download, can you?
I'm not the impolite one. All the data is posted pretty clear above, the tests explained, they confirm that only the "Disabled Switching" diagram is used.
Yet, you insist on supporting bogus data and making false claims based on 2 out of 4 posted tests, out of 5 if you count the PPPoE one.
I've explained why those two tests are showing less(half) data passing through (because they use the same 1Gb link to the CPU). The other 2 tests including the one with the PPPoE client active are using BOTH of the 1Gb/s links to the CPU.
Read up again, maybe you learn something.
Make some tests of your own, repeat mine, whatever.
From all that beeing said, you are the ignorant one, I'm sorry.
Cheers!

@msatter I've posted above the bridge ports, ether1 is NOT part of the bridge.
 
User avatar
Paternot
Forum Veteran
Forum Veteran
Posts: 912
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: hEX block diagram

Fri Mar 12, 2021 2:02 pm

@Paternot your posted speedtest is one way at a time, I covered all the ports and tests in the screenshots posted above.
Also the Block Diagram posted above (disabled switching) seems pretty accurate to my findings. https://i.mt.lv/cdn/product_files/RB750 ... 161117.png
Having WAN on ETH5(or 3 or 1) will give best up & down simultaneous speed with LAN on ports 2 and 4, you can test that with two speedtests, starting another one just before the current one ends, on different servers maybe, so that one of them does the upload part and another one the download part.
If you switch your testing machine with the current setup on ports 3 or 1 you'll see half of the simultaneous transfer speed. (you'll still get 1Gb/s up OR down, though, so running just ONE speedtest will not show that.)
Like this, here I was doing just that, two speedtests:
RCS-RDS Speed Test 20210312.PNG
This isn't how it works. The system is a SOC, with a switch integrated right into the CPU. This integrated switch uses two connections (as seen here: http://www.t-firefly.com/download/FireW ... MT7621.pdf) to the CPU. Each of these connections runs at 1Gbps.

There is not a physical assignment to a given port: it works just our modern x86 CPUs, that have two memory controllers, and any core can use any controller at any time.

But the links on the hEX ARE full duplex. How can I be sure? Easy. Take a look at the speedtest, posted by Mikrotik.
https://mikrotik.com/product/RB750Gr3#fndtn-testresults

Take a look at "routing, fast path". The hEX can do almost 2Gbps. The speed test is done using one direction only - it goes in from one port and goes out from another.

So, say it goes in at eht1 and out at eth2. We need 1Gbps in at th1 and 1Gbps out at eth2. So far, so good. We have 2 links after all - even half duplex ones would work. But this gives us 1Gbps of traffic - you don't count twice. To get the 1,9Gbps stated, the only way would be with two full duplex internal links: say, eth1 and eth2 in, eth3 and eth4 out. Otherwise You would need 4 halfduplex links.
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Fri Mar 12, 2021 2:36 pm

That doesn't contradict my findings, using just two ports, based on the first screenshots:
screenshot1:
~842Mbps IN ether1 -> ~850Mbps OUT ether2 AND
~817Mbps IN ether2 -> ~815Mbps OUT ether1 -> 2x 1Gb/s links, right? (correct based on the Disabled Switching Diagram, because ports 1 and 2 are on different lanes to the CPU).

screenshot2:
~635Mbps IN ether1 -> ~631Mbps OUT ether3 AND
~321Mbps IN ether3 -> ~317Mbps OUT ether1 -> only 1x 1Gb/s link, right? (correct based on the Disabled Switching Diagram, because ports 1 and 3 are on the same lane to the CPU).

screenshot3:
~799Mbps IN ether1 -> ~799Mbps OUT ether4 AND
~869Mbps IN ether4 -> ~869Mbps OUT ether1 -> 2x 1Gb/s link, right? (correct based on the Disabled Switching Diagram, because ports 1 and 4 are on different lanes to the CPU).

screenshot4:
~626Mbps IN ether1 -> ~623Mbps OUT ether5 AND
~331Mbps IN ether5 -> ~329Mbps OUT ether1 -> only 1x 1Gb/s link, right? (correct based on the Disabled Switching Diagram, because ports 1 and 5 are on the same lane to the CPU).

Ports are fixed, 1,3,5 on one lane, ports 2,4 on another lane. They don't jump randomly from lane to lane. Based, again, on the Disabled Switching Block Diagram presented on the support page.
hEX RB750Gr3 - Disabled Switching.PNG
Routing performance on the test results advertises 1.1Gbps for routing with 25 filter rules and 1518 byte packets, I have only 14 filter rules and I get ~1.6Gbps routing (screenshots 1 and 3), that's pretty good for this little router.
You do not have the required permissions to view the files attached to this post.
 
User avatar
Paternot
Forum Veteran
Forum Veteran
Posts: 912
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: hEX block diagram

Fri Mar 12, 2021 5:00 pm

One last time: the links ARE full duplex. And, as You can see on the pdf I linked, they are NOT assigned to a given port. Think about it.
I posted a speedtest, crossing the router, from eth5 to eth1. If the ports were assigned to a given link, and by Mikrotik's schematic, It wouldn't be possible to cross 1Gbps through a single half duplex link. Only a full duplex one can do this.

Same thing with the speed results: one cannot pass through 1,9Gbps of traffic with two half duplex 1Gbps links. And, Yes, contradict your findings. You claimed the links where half duplex. They aren't. And by Your screenshot1: we have about 1,7Gbps of crossing traffic. In order to it be even possible, both links must be full duplex.

Take a look at the pdf I posted here. It's the documentation of the SOC used by Mikrotik. There You will see that the switch is connected to the CPU with two links that aren't hardwired to any single switch port.
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Fri Mar 12, 2021 6:06 pm

I didn't say anything about half duplex links, there are two 1Gb/s full duplex links, one link for ports 1,3,5, and one link for ports 2,4.
@sebastia is the one claiming half duplex links, not me.
The datasheet doesn't say how MikroTik configured those links, but the MikroTik posted diagrams say how they did, like above mentioned and tested.
You can't saturate one full duplex link with ~900Mbps unidirectional traffic at a time, like your speedtest, which does download, THEN upload.
You CAN saturate one full duplex link with bidirectional tests, like I did above:
1<=>2 = TWO full duplex links in use to/from CPU, bidirectional traffic of ~1.6Gbps, ~800Mbps both ways of both links, no bottleneck.
1<=>4 = same as above, no bottleneck.
3<=>2 = same as above, no bottleneck.
3<=>4 = same as above, no bottleneck.
5<=>2 = same as above, no bottleneck.
5<=>4 = same as above, no bottleneck.
1<=>3 = ONE full duplex link in use to/from CPU, bidirectional traffic of ~900Mbps, ~600Mbps one way, ~300 the other, bottleneck.
1<=>5 = same as above, bottleneck.
3<=>5 = same as above, bottleneck.
2<=>4 = same as above, bottleneck.
There is a bottleneck when using WAN and LAN ports from the same link (1<=>3 OR 1<=>5 OR 3<=>5 OR 2<=>4).
For optimal performance and avoid any bottlenecks you have to use WAN ports from one link and LAN ports from the other link.
I just wanted to know if there is any way around this, besides using port 2(or 4) for WAN and ports 1,3,5 for LAN. <- *WRONG, see below: viewtopic.php?f=3&p=848197#p848151
Or port 1(or 3 or 5) for WAN and ports 2,4 for LAN.
I don't understand what are you trying to prove. That my tests lie? Feel free to do your own BIDIRECTIONAL tests, two ports at a time, post the results and compare our findings.
Until then, Cheers!
Last edited by Znevna on Sat Apr 17, 2021 10:01 am, edited 1 time in total.
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: hEX block diagram

Fri Mar 12, 2021 6:46 pm

If you take the hEX-S things gets simpler if you insert a SFP. Ports 1-5 share all the same lane and the SFP gets its own private lane to the CPU.

Image
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Fri Mar 12, 2021 7:11 pm

Exactly. Thank you. Using the SFP port as WAN gives it a full 1Gb/s lane to the CPU while the other 1Gb/s lane is for the remaining 5 ethernet ports.
One could buy 1x hEX-S + 1x S-RJ01 just for that if you have only ethernet ports, but at that price you can get the hAP ac2 which has a 2Gb/s lane from the CPU to the switch and you can use the switch ports however you want.
And judging by this topic, that is not a happy combination (hEX-S + S-RJ01): viewtopic.php?t=149599
Any hEX-S users using S-RJ01 like that out there? care to make some tests? is the issue mentioned in that topic above fixed?
Back to current topic, bottom line, I just wanted to see if it is possible to assign ONE ethernet port to one lane and the remaining 4 ports to the other lane, apparently not.
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: hEX block diagram

Fri Mar 12, 2021 7:30 pm

Then looking at at the CPU. Two lane go up from the switch to the RBUS and trafic can only return through the right PBUS. Traffic can't flow from the switch through the PBUS to the CPU.

According to this schematic, five into two lanes, returning one lane (PBUS) 5->2->1 Gbit/s when forwarding.
Screenshot_20210312_180027.jpg
You do not have the required permissions to view the files attached to this post.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: hEX block diagram

Fri Mar 12, 2021 7:42 pm

Exactly. Thank you. Using the SFP port as WAN gives it a full 1Gb/s lane to the CPU while the other 1Gb/s lane is for the remaining 5 ethernet ports.
One could buy 1x hEX-S + 1x S-RJ01 just for that if you have only ethernet ports, but at that price you can get the hAP ac2 which has a 2Gb/s lane from the CPU to the switch and you can use the switch ports however you want.
And judging by this topic, that is not a happy combination (hEX-S + S-RJ01): viewtopic.php?t=149599
Any hEX-S users using S-RJ01 like that out there? care to make some tests? is the issue mentioned in that topic above fixed?
Back to current topic, bottom line, I just wanted to see if it is possible to assign ONE ethernet port to one lane and the remaining 4 ports to the other lane, apparently not.
I have two hEX-S and replaced one with a 4011 connected to the internet. The other hEX-S is stil in between the 4011 and the internal network. It work fine as long you remember not put bilk traffic on the same lane as you noticed. The encrypting power of the 4011 is much higher than of the hEX-S and thst is why I got me the 4011.

I would only advise to use an optical SFP in in ahEX-S.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Fri Mar 12, 2021 7:55 pm

The PBUS is used for something else, found it described in another PDF:
MT7621A - Diagram.PNG
I also have an RB4011 in use, even used a S-RJ01 on it until this happened: viewtopic.php?f=3&t=173219#p847923
But this little 750Gr3 is a little beast too, it's worth knowing how to use it at its full potential :) and understand when limitations occur and why.
You do not have the required permissions to view the files attached to this post.
Last edited by Znevna on Fri Mar 12, 2021 8:47 pm, edited 1 time in total.
 
mkx
Forum Guru
Forum Guru
Posts: 6652
Joined: Thu Mar 03, 2016 10:23 pm

Re: hEX block diagram

Fri Mar 12, 2021 7:59 pm

Here's how I understand the block diagram of a RB750Gr3 when ether1 is stand-alone interface (e.g. to piggy-back PPPoE) while ether2-ether5 are members of same bridge which has HW offload enabled (and active):

Image

If there was another stand-alone interface, e.g. ether2, then ether1 and ether2 would share one interconnect (the upper one) while switched ports (ether3-ether5) would share the other interconnect.

I'm not sure how it would handle, if someone made two or more bridges. This isn't a recommended config, anyway.
All but one of bridges would be entirely done in software ... HW offload is limited to single bridge per switch chip and RB750Gr3 has single switch chip.
Last edited by mkx on Fri Mar 12, 2021 8:07 pm, edited 1 time in total.
BR,
Metod
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Fri Mar 12, 2021 8:07 pm

Yeah, that's what I thought it works like too, until I did the above tests. The first 4 tests are done with ether1 standalone, other 4 ports in a bridge with hardware offload enabled and active (bridge protocol-mode=none, since STP disables Bridge HW Offload on MT7621).
 
mkx
Forum Guru
Forum Guru
Posts: 6652
Joined: Thu Mar 03, 2016 10:23 pm

Re: hEX block diagram

Fri Mar 12, 2021 8:12 pm

From the CPU profile it's obvious that most of CPU is consumed for interrupt handling. Since any given interrupt can be handled by any of CPU cores (however, usual linux drivers in recent versions of stock kernel tend to stick to a particular CPU core), it could well happen that during certain tests the interrupt handling is distributed differently (you'd have to run CPU profiler during all the tests and compare load distribution). If too much interrupt handling is allocated to one of cores, it can become a bottleneck.
BR,
Metod
 
User avatar
Paternot
Forum Veteran
Forum Veteran
Posts: 912
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: hEX block diagram

Fri Mar 12, 2021 8:32 pm

I didn't say anything about half duplex links, there are two 1Gb/s full duplex links, one link for ports 1,3,5, and one link for ports 2,4.
@sebastia is the one claiming half duplex links, not me.
Ah, sorry. My mistake.
The datasheet doesn't say how MikroTik configured those links, but the MikroTik posted diagrams say how they did, like above mentioned and tested.
You can't saturate one full duplex link with ~900Mbps unidirectional traffic at a time, like your speedtest, which does download, THEN upload.
You CAN saturate one full duplex link with bidirectional tests, like I did above:
1<=>2 = TWO full duplex links in use to/from CPU, bidirectional traffic of ~1.6Gbps, ~800Mbps both ways of both links, no bottleneck.
1<=>4 = same as above, no bottleneck.
3<=>2 = same as above, no bottleneck.
3<=>4 = same as above, no bottleneck.
5<=>2 = same as above, no bottleneck.
5<=>4 = same as above, no bottleneck.
1<=>3 = ONE full duplex link in use to/from CPU, bidirectional traffic of ~900Mbps, ~600Mbps one way, ~300 the other, bottleneck.
1<=>5 = same as above, bottleneck.
3<=>5 = same as above, bottleneck.
2<=>4 = same as above, bottleneck.
There is a bottleneck when using WAN and LAN ports from the same link (1<=>3 OR 1<=>5 OR 3<=>5 OR 2<=>4).
For optimal performance and avoid any bottlenecks you have to use WAN ports from one link and LAN ports from the other link.
I just wanted to know if there is any way around this, besides using port 2(or 4) for WAN and ports 1,3,5 for LAN.
Or port 1(or 3 or 5) for WAN and ports 2,4 for LAN.
I don't understand what are you trying to prove. That my tests lie? Feel free to do your own BIDIRECTIONAL tests, two ports at a time, post the results and compare our findings.
Until then, Cheers!
No, not a lie. I just thought they would be wrong.

Well, I can't get as high as 1Gbps up/down at the same time (ISP throttling). I did get some differences, while using eth1<->eth5 and eth2<->eth5. Not that much, but it's there.

Looks like You are right about the link distribution. I still find it really weird, but can't argue with the numbers. :D

Yes, I know they are on the block diagram. I just thought that they weren't that specific. After all, one can't make a diagram dividing 5 ports into 2 links...
You do not have the required permissions to view the files attached to this post.
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: hEX block diagram

Fri Mar 12, 2021 10:23 pm

Glad that we see the same thing and thinking about using eth 5 as WAN. Yes it looks logical and you have a symmetrical distribution of the other ports.

However looking again, better is to use port 2 or 4 for WAN. Why? Using one of those tow port give you possible maximum speed on three ports and a lower speed on one port, which you can use for a less important device connected to you hEX.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
Paternot
Forum Veteran
Forum Veteran
Posts: 912
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: hEX block diagram

Fri Mar 12, 2021 10:55 pm

Glad that we see the same thing and thinking about using eth 5 as WAN. Yes it looks logical and you have a symmetrical distribution of the other ports.

However looking again, better is to use port 2 or 4 for WAN. Why? Using one of those tow port give you possible maximum speed on three ports and a lower speed on one port, which you can use for a less important device connected to you hEX.
Yes. I'm thinking about moving my internet to ether4. It wasn't a problem, until 5 days ago - when I got a speed upgrade. The hEX is just a router - it's connected to a switch, and everything else goes there. I can't move the LAN from eth1 as I use PoE on this hEX.
 
mada3k
Member
Member
Posts: 463
Joined: Mon Jul 13, 2015 10:53 am
Location: Sweden

Re: hEX block diagram

Sat Mar 13, 2021 10:22 am

I think it's hilarious that we are discussing a 1Gbit bottleneck on a router that costs as much as the monthly fee as the connection :)
CCR/CRS/hEX/wAP • Ansible • NetXMS
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Sat Mar 13, 2021 10:37 am

That connection (1000/500) costs here 1/6 of the routers price. How is that relevant in any way to those two diagrams of this router?
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: hEX block diagram

Sat Mar 13, 2021 12:41 pm

My 500/500 connection costs more then 17 hEX a year.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
mada3k
Member
Member
Posts: 463
Joined: Mon Jul 13, 2015 10:53 am
Location: Sweden

Re: hEX block diagram

Sat Mar 13, 2021 12:57 pm

Here a 1000/1000 FTTH connection is about €60-99/mo. A hEX costs about €65.

If I would to spend that money on an Internet connection, then I would probably also invest in a router on that level (like a RB4011 or so) instead of spend hours figuring out how to achieve wire-speed performance on an entry level device.

If this is a case of internal/inter-vlan traffic, then I must excuse myself for being ignorant.
CCR/CRS/hEX/wAP • Ansible • NetXMS
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Sat Mar 13, 2021 1:19 pm

Here the hEX RB750Gr3 can be currently found at ~55€ including shipping, the 1000/500 connection for residential customers is ~8.2€ at most providers, most of them offer the services via PPPoE which itself is taxing enough on the CPU.
At work, yes, I have an RB4011 in one place for similar connection, also PPPoE which is only 10€ more than the residential one, but the RB4011 is there just because of the many IKEv2 clients connected to it, the traffic can otherwise easily be sustained by this little RB750Gr3 if using the right ports.
For home, moving the cables around and adjusting the config a little in this RB750Gr3 is cheaper than a RB4011, smaller sized, less power consumption, heat etc.
There's nothing wrong trying to figure out how to use the full potential of that connection on this router. Or on any router for that matter, no matter the price of it :)
And even if you'd use it for inter-vlan routing, you'd like to know which ports cause a bottleneck and which don't, no? :)
As Paternot mentioned below, I'll mention it too, all of this in a country where the minimum wage, on paper, is 470€, but in hand you only get ~283€, rest of it goes to taxes.
Last edited by Znevna on Sat Mar 13, 2021 4:48 pm, edited 2 times in total.
 
User avatar
Paternot
Forum Veteran
Forum Veteran
Posts: 912
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: hEX block diagram

Sat Mar 13, 2021 1:40 pm

I think it's hilarious that we are discussing a 1Gbit bottleneck on a router that costs as much as the monthly fee as the connection :)
Not as hilarious when my hEX is bought and paid for. It's already in place, was bought when my connection was 30/30, and here in Brazil we pay 50% up the suggested price, given our taxes.

So, no. A lousy hAP AC2 costs US$ 145,00 here.
An RB4011 (without wifi) costs US$ 365,00 here.
And my internet connection (officially a 350/350 one) costs US$ 22,12.
All of this with a minimum wage of US$ 196,00.

Now You know why people try to get the most out of their hardware.
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Sat Mar 13, 2021 6:56 pm

So, guess what? this looks like a bug, since the bridge actually does seem to control which port gets on which lane, since I wanted to move my WAN to port 2, SURPRISE (see below).
I redid all the below tests, three times each. Including the one which started this "search" (port 1 stand-alone, rest of them bridged). Results:
bridged: 2,3,4,5; stand-alone: 1
1<=>2 - ok
1<=>3 - notok
1<=>4 - ok
1<=>5 - notok
------------------
bridged: 1,3,4,5; stand-alone: 2
2<=>1 - notok
2<=>3 - notok
2<=>4 - ok
2<=>5 - notok
------------------
bridged: 1,2,4,5; stand-alone: 3
3<=>1 - notok
3<=>2 - ok
3<=>4 - ok
3<=>5 - notok
------------------
bridged: 1,2,3,5; stand-alone: 4
4<=>1 - notok
4<=>2 - ok
4<=>3 - notok
4<=>5 - notok
------------------
bridged: 1,2,3,4; stand-alone: 5
5<=>1 - notok
5<=>2 - ok
5<=>3 - notok
5<=>4 - ok

So the bridge config does seem to do something, but doesn't do it right?
There might be hope for a fix!
Too bad there's no tested config above where you can have atleast 3 OK bridged ports. I have not done tests with fewer bridged ports. Maybe some other time.
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: hEX block diagram

Sat Mar 13, 2021 9:34 pm

I switched this afternoon stand alone one to two and two to one. I could not test properly but I saw a lower throughput than before.

I use a hEX-S with no SPF installed. There seems to be something out of wack in that little box.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Sat Mar 13, 2021 10:33 pm

Yeah, I thought switching port 2 or 4 to stand-alone and using the rest in a bridge would be the best, but apparently those two are the worst options of them all. As seen above:
Using port 2 as stand-alone and 1,3,4,5 in a bridge, throughput is over 1Gb/s only between ports 2 and 4, no more.
Using port 4 as stand-alone, and 1,2,3,5 in a bridge, throughput is over 1Gb/s only between ports 4 and 2.
Since in the other tests port 2 and 4 seem to be on the same lane.. in these two tests they seem to be on different lanes.
Something looks broken, I've wrote to support about this too.. we'll see.
PS: I have only tested this as a router so packets HAVE to pass through the CPU. I have not made any tests between the bridged ports. That isn't the thing tested here.
 
msatter
Forum Guru
Forum Guru
Posts: 2317
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: hEX block diagram

Sat Mar 13, 2021 11:09 pm

Thanks for testing and using standalone port one is the best results expected on ports two and four and I went back to my previous setup.

I looked in bridge which switch ports are assigned to the physical ports and surprise and that could be caused by me using a hEX-S that also can use an SPF besides 5 ether ports.

Physical port and then switch port in the CPU:
1 -> 5
2 -> 1
3 -> 2
4 -> 3
5 -> 4

Looking at the schematics posted earlier of the MT7621 there is a RGMII and I assume the SPF is using that as entrance to the switch/CPU. Also notice the fifth arrow is not straight but curved.

Check in Bridged Port - Status if on a non hEX-S the ordering of the switch different.
Loving my freedom and so, no Twitter, no Facebook/Instagram/WhatsApp, no Apple and no Google/Alphabet, no Amazon/Cloudfront/AWS. However, 'happy' with giving money to Italy.

Running:
RouterOS 7RC4 and 6.49RC / Winbox 3.31 64bits
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Sat Mar 13, 2021 11:34 pm

Looks the same here:
/interface bridge port> monitor [find interface=ether2]
interface: ether2
port-number: 1
/interface bridge port> monitor [find interface=ether3]
interface: ether3
port-number: 2
/interface bridge port> monitor [find interface=ether4]
interface: ether4
port-number: 3
/interface bridge port> monitor [find interface=ether5]
interface: ether5
port-number: 4

PS: What I wrote so far above, regarding ports used during tests, I was referring to the etherX interfaces listed in the interface menu which coincide with the labels on the physical ports (except ether1 which is labeled "Internet" but that is pretty obvious since the others are 2,3,4,5)
For example "bridged: 2,3,4,5; stand-alone: 1" means "bridged: ether2,ether3,ether4,ether5; stand-alone: ether1;"
To avoid any confusion.
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Tue Mar 30, 2021 5:39 pm

Good news! they managed to reproduce my findings and they will try to fix the issue in an upcoming RouterOS version, no ETA for now though.
So there is indeed hope for even greater power from this tiny box.
Thank you all for testing and feedback (even the negative ones, the world needs you too!).
Cheers!
 
User avatar
Paternot
Forum Veteran
Forum Veteran
Posts: 912
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: hEX block diagram

Tue Mar 30, 2021 8:33 pm

Good news! they managed to reproduce my findings and they will try to fix the issue in an upcoming RouterOS version, no ETA for now though.
So there is indeed hope for even greater power from this tiny box.
Thank you all for testing and feedback (even the negative ones, the world needs you too!).
Cheers!
So, the two links are used by any port? Well, will be. Is it right? Great news! :D
 
User avatar
Znevna
Member
Member
Posts: 345
Joined: Mon Sep 23, 2019 1:04 pm

Re: hEX block diagram

Tue Mar 30, 2021 9:17 pm

The two links in a bridged configuration will (hopefully) be used as described in the "Enabled Switching Diagram": the ports outside the bridge on a link, and the bridged ports on the other link.
They mentioned that for now the only "predictable CPU lane layout" setup is the disabled switching one (no bridge?, didn't test, will do).
Or any preferable config from my tests.
I went with WAN on ether1, and bridged 2,3,4,5; but with a device connected to ether2 and a switch connected to ether4, and all other devices behind that switch; nothing connected on ether3 and ether5 on RB750gr3 (since any traffic on ether3 and ether5 will "eat" from the bandwidth available for ether1, and I want ether1 (WAN) to have a full lane for itself to the CPU).

Who is online

Users browsing this forum: No registered users and 4 guests