OK, so a brief summary of my setup, I am running proxmox on a dell r720 which has a quad 1gb erthernet that I created a bond with. my CRS328 is connected to an RB4011 through an sfp+ 10GB tx.
iface bond0 inet manual
bond-slaves eno1 eno2 eno3 eno4
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
auto vmbr0
iface vmbr0 inet dhcp
bridge-ports bond0
bridge-stp off
bridge-fd 0
I also created an ad Layer2+3 bond on my mikrotik CRS328
Flags: X - disabled, R - running
0 R name="R720_bond" mtu=1500 mac-address=DC:2C:6E:64:05:E8 arp=enabled
arp-timeout=auto slaves=ether9,ether11,ether13,ether15 mode=802.3ad
primary=none link-monitoring=mii arp-interval=100ms arp-ip-targets=""
mii-interval=100ms down-delay=0ms up-delay=0ms lacp-rate=30secs
transmit-hash-policy=layer-2-and-3 min-links=0
I started with speedetest from a wired client and was getting about 1Gbiit/s, but from my r720 with the bond I was getting about 500 Mbit/s. I read that this can be inaccurate due to cpu load, so I installed iperf3, and seemed to get about 1Gbit/s between 2 wired clients. I currently only have the server wired, but I will string a cat6 back up to test anything anyone suggests. However, I created a google cloud vm to test iperf3 outside of my network which is where I am really wanting to see.
I am in northern SC for context.
iperf3 from wireless client to proxmox - 294 Mbits/sec
iperf3 from server vm on proxmox to proxmox - 18.0 Gbits/sec
iperf3 from wireless client to google VM - 31.4 Mbits/sec
iperf3 from server vm to google VM - 28.6 Mbits/sec
speedtest from google cloud vm (atlanta, GA) - 969.70 Mbit/s (12ms ping)
speedtest from wireless client (rockhill, SC) - 453.96 Mbit/s (6ms ping)
speedtest from proxmox (Charlotte, NC) - 529.03 Mbit/s (42ms ping)
speedtest from VM on proxmox (Charlotte, NC) - 483.05 Mbit/s (34ms ping)
So I think the google vm isnt a great solution to test bandwidth either. Ultimately 500 Mbit is obviously not that slow, but I just dont know why plugging in my wireless client to ethernet gets me 1 GBit/sec but my server doesnt seem to. I just dont understand how a wired client could get 1Gbit and the server could not (seems like an issue with the proxmox bond), but I could also get 1Gbit between the server and the wired client, which makes it seem like an issue with the mikrotik...so not sure what to check next.