Hi everyone. I 'll try to tell you the scenario and then tell you the bug.
In my corporate we have an rb1100 x2 AH, and I want to limit(traffic shape) the Internet speed of LAN users, so I needed to disable the fasttrack rule in firewall and made some queues but finally found that CPU of the board is weak and it utilization goes to about 95 percent. So I decide to use a mikrotikx86 install beside the rb1100. So I installed it and unfortunately after hours of headaches I found that there’s a bug in nat table corroponding to route tables.
Let me explain the topology in a nutshell.
10.0.254.0/24 for internet uplink; RB1100: 10.0.254.253 and GW:10.0.254.254
192.168.0.0/24 for LAN segment; RB1100: 192.168.0.1
172.16.1.0/24 to forward the traffic to MikrotikX86; RB1100:172.16.1.253 X86:172.16.1.252
172.16.2.0/24 to receive traffic shaped traffic from X86; RB1100:172.16.2.253 X86:172.16.2.252
I used two routing tables. One before the client traffic would be traffic shaped (before-TS) which will be marked as for the packets which received on eth2 (LAN side) and another routing table for packets received on Eth4 (after-TS)
[admin@MikroTik] /ip/address> print
Columns: ADDRESS, NETWORK, INTERFACE
# ADDRESS NETWORK INTERFACE
0 10.0.254.253/24 10.0.254.0 ether1
1 192.168.0.253/24 192.168.0.0 ether2
2 172.16.1.253/24 172.16.1.0 ether3
3 172.16.2.253/24 172.16.2.0 ether4
[admin@MikroTik] /ip/route> print detail
Flags: D - dynamic; X - disabled, I - inactive, A - active; c - connect, s - static, r - rip, b - bgp, o - ospf, i - is-is, d - dhcp, v - vpn, m - modem, y - bgp-mpls-vpn; H - hw-offloaded; + - ecmp
0 As dst-address=0.0.0.0/0 routing-table=main pref-src="" gateway=10.0.254.254 immediate-gw=10.0.254.254%ether1 distance=1 scope=30 target-scope=10 suppress-hw-offload=no
DAc dst-address=10.0.254.0/24 routing-table=main gateway=ether1 immediate-gw=ether1 distance=0 scope=10 suppress-hw-offload=no local-address=10.0.254.253%ether1
DAc dst-address=172.16.1.0/24 routing-table=main gateway=ether3 immediate-gw=ether3 distance=0 scope=10 suppress-hw-offload=no local-address=172.16.1.253%ether3
DAc dst-address=172.16.2.0/24 routing-table=main gateway=ether4 immediate-gw=ether4 distance=0 scope=10 suppress-hw-offload=no local-address=172.16.2.253%ether4
DAc dst-address=192.168.0.0/24 routing-table=main gateway=ether2 immediate-gw=ether2 distance=0 scope=10 suppress-hw-offload=no local-address=192.168.0.253%ether2
1 As dst-address=172.16.1.0/24 routing-table=Before-TS pref-src="" gateway=ether3 immediate-gw=ether3 distance=1 scope=10 target-scope=10 suppress-hw-offload=no
2 As dst-address=172.16.2.0/24 routing-table=Before-TS pref-src="" gateway=ether4 immediate-gw=ether4 distance=1 scope=10 target-scope=10 suppress-hw-offload=no
3 As dst-address=192.168.0.0/24 routing-table=Before-TS pref-src="" gateway=ether2 immediate-gw=ether2 distance=1 scope=10 target-scope=10 suppress-hw-offload=no
4 As dst-address=0.0.0.0/0 routing-table=After-TS pref-src="" gateway=10.0.254.254 immediate-gw=10.0.254.254%ether1 distance=1 scope=30 target-scope=10 suppress-hw-offload=no
5 As dst-address=10.0.254.0/24 routing-table=After-TS pref-src="" gateway=ether1 immediate-gw=ether1 distance=1 scope=10 target-scope=10 suppress-hw-offload=no
6 As dst-address=172.16.2.0/24 routing-table=After-TS pref-src="" gateway=ether4 immediate-gw=ether4 distance=1 scope=10 target-scope=10 suppress-hw-offload=no
[admin@MikroTik] /routing/table> print
Flags: D - dynamic; X - disabled, I - invalid; U - used
0 D name="main" fib
1 name="Before-TS" fib
2 name="After-TS" fib
[admin@MikroTik] /ip/firewall/mangle> print
Flags: X - disabled, I - invalid; D - dynamic
0 chain=prerouting action=mark-connection new-connection-mark=Before-TS passthrough=yes in-interface=ether2 log=no log-prefix=""
1 chain=prerouting action=mark-routing new-routing-mark=Before-TS passthrough=no connection-state=established,related,new connection-mark=AfterC in-interface=ether2 log=no log-prefix=""
2 chain=prerouting action=mark-connection new-connection-mark=After-TS passthrough=yes in-interface=ether4 log=no log-prefix=""
3 chain=prerouting action=mark-routing new-routing-mark=After-TS passthrough=no connection-state=established,related,new connection-mark=After-TS in-interface=ether4 log=no log-prefix=""
[admin@MikroTik] /ip/firewall/nat> print
Flags: X - disabled, I - invalid; D - dynamic
0 chain=srcnat action=masquerade out-interface=ether1 log=no log-prefix=""
In Mikrotik X86 there’s no firewall rule and it’s just used for traffic shaping and queues only.
In the RB1100 there are two routing tables and corrosponding mangle rules.(Mangles are on prerouting chains and set on “In Interface” parameter)
The traffic which comes from LAN and goes to eth2 will be marked as the Before-TS and will be forwarded to X86, and the traffic that comes from X86 and goes to eth4 will be marked as After-TS and goes to the main internet Gateway. All of these works but the problem and the bug that I captured is that the traffic that comes back (step3) will not be nat-ed and the packet wil be send with the original source LAN client Address ( I found it via packet sniffer).
I 'm suspected that because the received back packet from srcip and dstip in step3 are the same as srcip and dstip step2, firewall nat rule could not handle it correctly.