Hi, thanks for your attention.
I was expecting to see the queue configuration in the export, but never mind, the issue seems to be clear, but please confirm my understanding.
There is a gateway in the subnet (.1); you expect that if you put the CHR to the same subnet with an address .3, and tell the PC (address .4) that its gateway is the .3 instead of the .1, all traffic from the PC will flow to .3 which will forward it to .1, and the .1 will send all the responses to .3 which will forward them to .4?
Yes, the purpose will be Queue to limit the download and upload speed, but currently, as you see, the traffic is not passing through my CHR yet!
The scenario that you explained is exactly what I did, I thought it work!
If so, this will not work unless you configure both the PC(s) and the actual gateway in a very special way. Normally, if a router finds out that when forwarding a packet, the IP address of the gateway is in the same subnet like the source address of the packet, it forwards the packet but it also informs the sender that there is a better gateway in the subnet; the sender normally caches this information and sends the subsequent packets to the same destination via the new gateway. You have to disable this either by setting send-redirects under /ip settings to no on the CHR, or by configuring all the other hosts in that subnet to ignore the redirects.
I have disabled the "send-redirects" on CHR, but same results! we may have like 500 VMs that are using by customers, so we can not make changes on those, one by one.
For the responses for .4 coming from the outside, there is no way to convince the real gateway (.1) to send them to .3 rather than directly to .4, because no gateway is necessary to send a packet to an address in your own subnet - the .1 sends an ARP request for .4, and the .4 will respond with its own MAC address.
The clean way would be to use another subnet to connect the CHR with the actual gateway - you can do that even without creating a separate port group for that subnet, but this requires that you can configure the .1 to use the address of the .3 in that other subnet as a gateway to the original subnet.
This solution makes sense, even though we do not have direct access to (.1) since it is under datacenter/provider control. we may be able to contact them to configure it for us.
So, please explain this solution in details:
we should ask the provider to add a small IP subnet (/29 or /30) to this server. and what configuration should we do on CHR(.3) & on the main gateway(.1)?
Do we need to change the gateway address for each VMs(.4) in the network?
------------------------------------
Also, I wonder, is there any other way like using MAC address or etc ? or Am I doing the worst solution(CHR in same subnet) to see BW and limit speed? what do other people do for the same scenario in first place?
Btw, The dirty way won't work for us since the remote client should have their own IP and see the actual/public IP totally.