Bridge interface not showing traffic

RouterOS Beginner Basics
1

Hi there,

I am having this bridge issue on one of the CCR1009-7G-1C-1S+ for a long time now and I can’t figure it out.
I have created a WAN interface bridge where I have added the SFP+ETH0 combo (uplink from ISP) and ETH1, where my firewall is connected to.
All of my users are behind the firewall – that’s where 99% of the traffic gets generated from.

On the LAN side, I have ETH2 connected to a Cisco switch where our VoIP phones get connected to a PBX on the cloud.

The problem is that I barely see any traffic on the WAN bridge, where its participant ports are having 200+ Mbps traffic, the WAN bridge barely shows any traffic, like 20Kbps or something.

Apparently, this is an issue for me because I can’t do anything with the queue trees. Necessary to make sure that our QoS is working properly.
Let me know if I need to post any config in here!

Any insights on where should I look for would be much appreciated!

2

Honestly this sounds like a misconfiguration. It seems that you are trying to use the CCR in the same manner as a traditional routerboard style device. Can you provide your configuration?

3

Sorry just catching up with this post. I surely can. Note here that I have the same router and configuration at another site and I don’t have this issue.

But let me DM the config to you.

4 
# may/01/2019 22:13:31 by RouterOS 6.44.3
# software id = Q4VP-P7RQ
#
# model = CCR1009-7G-1C-1S+
# serial number = 6F53040122100
/interface bridge
add fast-forward=no name=bridge-fw-management
add fast-forward=no name=bridge-lan
add comment=voip+paloalto fast-forward=no name=bridge-wan
/interface ethernet
set [ find default-name=combo1 ] auto-negotiation=no comment=cogent-sfp name=\
    combo-eth0-sp1
set [ find default-name=ether1 ] comment=paloalto speed=100Mbps
set [ find default-name=ether2 ] comment=cisco-switch-voip speed=100Mbps
set [ find default-name=ether3 ] speed=100Mbps
set [ find default-name=ether4 ] speed=100Mbps
set [ find default-name=ether5 ] speed=100Mbps
set [ find default-name=ether6 ] comment=palo-alto-management speed=100Mbps
set [ find default-name=ether7 ] speed=100Mbps
set [ find default-name=sfp-sfpplus1 ] advertise=\
    10M-full,100M-full,1000M-full name=sfp-plus1
/interface vlan
add comment=phones interface=bridge-lan name=vlan2 vlan-id=2
add comment=management interface=bridge-lan name=vlan5 vlan-id=5
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=voip ranges=10.10.2.20-10.10.2.240
add name=managment ranges=10.10.5.211-10.10.5.230
add name=dhcp_pool2 ranges=192.168.88.20-192.168.88.200
add name=palo_alto_management ranges=10.10.0.252-10.10.0.254
add name=dhcp_pool4 ranges=10.1.10.20-10.1.11.254
add name=dhcp_pool5 ranges=10.1.15.200-10.1.15.254
add name=dhcp_pool6 ranges=10.10.1.254
/ip dhcp-server
add address-pool=voip disabled=no interface=vlan2 name=dhcp_voip
add address-pool=managment disabled=no interface=vlan5 name=dhcp_managment
add address-pool=palo_alto_management disabled=no interface=\
    bridge-fw-management lease-time=1d name=dhcp_fw_management
/queue tree
add max-limit=50M name=parent_download parent=bridge-lan priority=1
add max-limit=40M name=1.child_rtp_in packet-mark=PBX_RTP_IN parent=\
    parent_download priority=1
add max-limit=10M name=2.child_sip_in packet-mark=PBX_SIP_IN parent=\
    parent_download priority=2
add max-limit=270M name=3.child_all_other_traffic_in packet-mark=\
    ALL_TRAFFIC_IN parent=parent_download
add max-limit=50M name=parent_upload parent=bridge-wan
add max-limit=250M name=paloalto packet-mark=ALL_TRAFFIC_OUT parent=global
add max-limit=250M name=paloalto_in packet-mark=ALL_TRAFFIC_IN parent=global
add max-limit=40M name=1.child_rtp_out packet-mark=PBX_RTP_OUT parent=\
    parent_upload priority=1
add max-limit=10M name=2.child_sip_out packet-mark=PBX_SIP_OUT parent=\
    parent_upload priority=2
add max-limit=270M name="3. child_all_other_traffic_out" packet-mark=\
    ALL_TRAFFIC_OUT parent=parent_upload
add burst-limit=260M burst-threshold=250M burst-time=30s disabled=yes \
    max-limit=250M name=palo-alto packet-mark=fortigate parent=global \
    priority=6 queue=pcq-download-default
/interface bridge port
add bridge=bridge-wan interface=combo-eth0-sp1
add bridge=bridge-lan hw=no interface=ether2
add bridge=bridge-wan interface=ether1
add bridge=bridge-fw-management interface=ether6
add bridge=bridge-fw-management interface=ether4
add bridge=bridge-fw-management interface=ether5
/ip settings
set tcp-syncookies=yes
/ip address
add address=103.32.5.126/29 interface=combo-eth0-sp1 network=38.88.202.192
add address=10.10.2.1/24 interface=vlan2 network=10.10.2.0
add address=10.10.5.1/24 interface=vlan5 network=10.10.5.0
add address=10.10.0.1/24 interface=bridge-fw-management network=10.10.0.0
/ip dhcp-server lease
add address=10.10.0.254 mac-address=08:30:6B:8D:D3:00 server=\
    dhcp_fw_management
add address=10.10.0.253 mac-address=C4:24:56:13:78:00 server=\
    dhcp_fw_management
/ip dhcp-server network
add address=10.10.0.0/24 gateway=10.10.0.1
add address=10.10.1.0/24 gateway=10.10.1.1
add address=10.10.2.0/24 gateway=10.10.2.1
add address=10.10.5.0/24 gateway=10.10.5.1
add address=192.168.88.0/24 gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes servers=66.28.0.45,66.28.0.61
/ip firewall address-list
add address=163.222.95.123 comment=freePBX list=voip-server
/ip firewall filter
add action=accept chain=forward
add action=accept chain=input
add action=accept chain=output
add action=accept chain=input src-address=163.222.95.123
add action=accept chain=forward src-address=163.222.95.123
add action=accept chain=output dst-address=163.222.95.123
add action=accept chain=input dst-port=53 protocol=udp src-address=\
    163.222.95.123
add action=drop chain=input dst-port=22 protocol=tcp
add action=drop chain=input dst-port=22 protocol=udp
add action=drop chain=input dst-port=53 in-interface=bridge-wan protocol=tcp
add action=drop chain=input dst-port=53 in-interface=bridge-wan protocol=udp
add action=accept chain=input dst-port=8191 protocol=tcp src-address-list=\
    mikrotik
add action=drop chain=input dst-port=8191 protocol=tcp
add action=accept chain=input comment="Allow all Site24x7 ICMP" protocol=icmp \
    src-address-list=site24x7
add action=accept chain=input comment="Allow all freePVX ICMP" protocol=icmp \
    src-address-list=voip-server
add action=accept chain=input comment="Allow all mikrotik ICMP" protocol=icmp \
    src-address-list=mikrotik
add action=drop chain=input comment="Drop all other ICMP" in-interface=\
    bridge-wan protocol=icmp
add action=accept chain=input comment="Permit established connections" \
    connection-state=established
add action=accept chain=input comment="Permit related connections" \
    connection-state=related
add action=accept chain=input comment="Allow whitelisted sources" \
    src-address-list=Whitelist
add action=drop chain=input comment="WAN - default deny" in-interface=\
    bridge-wan
add action=add-src-to-address-list address-list=blocked-addr \
    address-list-timeout=1d chain=input connection-limit=40000,32 protocol=\
    tcp
add action=tarpit chain=input connection-limit=3,32 protocol=tcp \
    src-address-list=blocked-addr
add action=jump chain=forward comment="SYN Flood protect" connection-state=\
    new jump-target=SYN-Protect protocol=tcp tcp-flags=syn
add action=accept chain=SYN-Protect connection-state=new limit=400,5:packet \
    protocol=tcp tcp-flags=syn
add action=drop chain=SYN-Protect connection-state=new protocol=tcp \
    tcp-flags=syn
add action=accept chain=forward connection-state=established,related
add action=accept chain=input connection-state=established,related
add action=accept chain=input protocol=icmp
add action=accept chain=input in-interface=bridge-wan
add action=drop chain=input in-interface=bridge-lan
add action=drop chain=forward connection-state=invalid
add action=drop chain=forward connection-nat-state=!dstnat connection-state=\
    new in-interface=bridge-lan
/ip firewall mangle
add action=mark-packet chain=prerouting comment=pbx-rtp-in in-interface=\
    bridge-wan new-packet-mark=PBX_RTP_IN passthrough=no protocol=udp \
    src-address=163.222.95.123 src-port=!5060
add action=mark-packet chain=prerouting comment=pbx-sip-in in-interface=\
    bridge-wan new-packet-mark=PBX_SIP_IN passthrough=no protocol=udp \
    src-address=163.222.95.123 src-port=5060
add action=mark-packet chain=prerouting comment=all-traffic in-interface=\
    bridge-wan new-packet-mark=ALL_TRAFFIC_IN passthrough=no src-address=\
    !163.222.95.123
add action=mark-packet chain=postrouting comment=pbx-rtp-out dst-address=\
    163.222.95.123 dst-port=!5060 new-packet-mark=PBX_RTP_OUT out-interface=\
    bridge-wan passthrough=no protocol=udp
add action=mark-packet chain=postrouting comment=pbx-sip-out dst-address=\
    163.222.95.123 dst-port=5060 new-packet-mark=PBX_SIP_OUT out-interface=\
    bridge-wan passthrough=no protocol=udp
add action=mark-packet chain=postrouting comment=all-traffic-out dst-address=\
    !163.222.95.123 new-packet-mark=ALL_TRAFFIC_OUT out-interface=bridge-wan \
    passthrough=no
/ip firewall nat
add action=src-nat chain=srcnat out-interface=bridge-wan src-address=\
    10.10.0.0/16 to-addresses=103.32.5.126
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set sip disabled=yes ports=5060
set pptp disabled=yes
/ip proxy
set max-client-connections=1 port=8888 src-address=103.32.5.126
/ip proxy access
add dst-address=10.10.5.100 src-address=10.10.5.230
/ip route
add distance=1 gateway=38.88.202.193
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh address=103.32.5.127/32 disabled=yes
set api disabled=yes
set winbox port=8191
set api-ssl disabled=yes
/ip socks access
add src-address=103.32.5.127 src-port=1080
/ip ssh
set allow-none-crypto=yes forwarding-enabled=remote
/lcd
set default-screen=stats
/system clock
set time-zone-name=America/Iowa City
/system identity
set name=Mikrotik
/system logging
add topics=firewall
5

A bridge in ROS has two personnalities: one is “something like a switch” which more or less moves packets between member ports. The other is “an L2 port” which allows ROS to interact with that traffic. The later is implicit member of self former (when you create the former, you also get the later).

In your case the traffic passing bridge (in the “like a switch” personnality) does not pass the bridge interface and hence doesn’t affect any counters. The numbers you’re seeing are probably some broadcasts, STP and neighbour discovery packets.
If you want to do something with passing traffic (PQs, firewall, …), you have to force traffic through CCR’s CPU. I’m not familiar with CCR, but you might have to disable HW offload on bridge member ports and/or enable bridge filtering and/or disable fastpath and/or something else.

6

Yes sir, that’s what I am actually struggling with.

I am doing everything I can to send a copy of the packet to the CPU. Mikrotik wifi recommends to run this command,

/interface ethernet switch rule
add copy-to-cpu=yes dst-mac-address=4C:5E:0C:4D:12:4B/FF:FF:FF:FF:FF:FF ports=ether1 switch=switch1

The issue I am having is with the command itself, I don’t seem to be able to add ports=ether1 (that’s my actual WAN as well) I keep getting “input does not match any value of port”. Same thing goes for the switch value. It seems like every value I insert on the ports and switch I get an error from the terminal.

7

CCR doesn’t have switch chip so majority (if not all) switch chip commands will fail.

8

Interesting, I kind of thought they didn’t. and I suppose the attribute “ports” won’t work either?

I honestly lost on what should I do at this point.

9

Show us diagram how everything is interconnected

10

It is right what is written above: traffic between ports of a switch is not considered “bridge traffic” in the counters, only traffic to/from the router is.
It is possible to make firewall rules (including the mangle rules you want for queue operation) operate on traffic between the bridge ports by setting an option in the bridge menu.
(bridge->settings->use IP firewall)

Then you can examine the traffic in normal firewall rules. Of course make sure you don’t lock yourself out when enabling this.

11

Got me wondering: CCR1009-7G-1C-1S+ doesn’t have any switch chip, https://i.mt.lv/cdn/rb_files/CCR1009-7G-1C-1Splus-170321154504.png

So traffic between ports, part of bridge will need to be passed on by cpu in software. Hence I would expect all traffic should be visible and accounted for?
BUT it does have HW acceleration, which in essence is a switch, which bypasses cpu…

That doesn’t match with the earlier posts and observations. So what is it?

Edited

12

The newer CCR1009 devices do not have a switch (the older ones with 8G in the type number do have a switch on ports 1-4).
That indeed means that all bridged traffic goes through the CPU.
But that does NOT mean that you can always see it. Bridged traffic is normally not seen by the firewall but you can configure that it should be seen.

13

I have attached a simple diagram how our networks are connected. The idea is that the eth1 (firewall) and the SPF are in bridge so that we can sort out and prioritize the VoIP traffic from the traffic that’s coming from the firewall - where all users are behind and the traffic that gets generated it’s pretty heavy and affecting the VoIP quality.

I can tell that the phone calls are fine most of the time, but when the burse limit for the firewall hits the MAX, we start getting lower VoIP quality during the call. There is no issues with dropped calls or anything of that nature, the issue I guess is that the traffic is not being prioritized by the queue tree because the traffic doesn’t get recognized by the bridge. Hence the reason of this post.

I know that you all know the issue, but I wanted to summarize the issue once again. Any workaround/help/suggestions on what we can do in practice for this case would be much appreciated!
corp-simple-diagram.png

14

I think there is an error in your configuration. The external ip address of your router is bound to the physical interface “combo1” but this port is member of the bridge “bridge-wan”. Can you please try to fix this (if you are remote you should consider doing that with safe mode).

Additionally your first three firewall rules allow everything thus making all following rules inoperative:

add action=accept chain=forward
add action=accept chain=input
add action=accept chain=output
15

I already answered it in reply #10.
When you want a bridge and you want to mark and queue traffic that crosses that bridge, you need to enable IP firewall on bridges.
(note that this is a global setting, so when you enable this you must first make sure that you have appropriate filter allow rules to allow all existing and correct bridge traffic in all of your bridges)

16

Thanks guys! I’ve honestly tried everything above mentioned, putting the IP address on the bridge-wan as opposed to the combo port, and I’ve tried to use the IP firewall on the bridge settings too. Also those three firewall rules is just me testing by the time I exported the config.

Either way, with that said, I will do all the changes again tonight and post my config here (hopefully this will get fixed!).

What’s tripping me off though is that I have the same exact mikrotik with the same exact config in another location and everything there is working fine there. Here is the config for the other mikrotik. See also attached the pic that shows the amount of traffic generated in the bridge-wan interface - just like how’s supposed to be.

# may/08/2019 07:28:16 by RouterOS 6.43.4
# software id = 2RR8-49EL
#
# model = CCR1009-7G-1C-1S+
# serial number = 7AEW0101D74B
/interface bridge
add fast-forward=no name=bridge-lan
add fast-forward=no name=bridge-mgm
add fast-forward=no name=bridge-wan
/interface ethernet
set [ find default-name=ether1 ] comment=comcast-paloalto speed=100Mbps
set [ find default-name=ether2 ] comment=paloalto-comcast-lan-test speed=100Mbps
set [ find default-name=ether3 ] comment=cisco-voip-sw speed=100Mbps
set [ find default-name=ether4 ] comment=paloalto-towerstream-redundancy speed=100Mbps
set [ find default-name=ether5 ] comment=test-cradlepoint speed=100Mbps
set [ find default-name=ether6 ] comment=palo-alto-management speed=100Mbps
set [ find default-name=ether7 ] comment=mikrotik-backup-net speed=100Mbps
set [ find default-name=sfp-sfpplus1 ] advertise=10M-full,100M-full,1000M-full
set [ find default-name=combo1 ] comment=comcast name=sfp1-eth0-combo
/interface vlan
add comment=switches interface=bridge-lan name=management vlan-id=15
add interface=ether7 name=mgm vlan-id=15
add interface=ether7 name=scanprint vlan-id=17
add interface=ether7 name=tablets vlan-id=16
add interface=ether7 name=timeclocks vlan-id=18
add interface=ether7 name=users vlan-id=10
add comment=phones interface=bridge-lan name=voip vlan-id=12
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip dhcp-server option
add code=66 name=DHCP66
/ip pool
add name=dhcp_voip ranges=10.10.12.20-10.10.12.240
add name=dhcp_management ranges=10.10.15.240-10.10.15.245
add name=dhcp_pool7 ranges=10.10.1.254
add name=dhcp_pool8 ranges=10.10.1.2-10.10.1.254
add name=dhcp_pool9 ranges=10.10.10.20-10.10.11.220
add name=dhcp_pool10 ranges=10.10.15.200-10.10.15.254
add name=dhcp_pool11 ranges=10.10.16.2-10.10.16.254
add name=dhcp_pool12 ranges=10.10.17.2-10.10.17.254
add name=dhcp_pool13 ranges=10.10.18.200-10.10.18.254
/ip dhcp-server
add address-pool=dhcp_management disabled=no interface=management lease-time=2m name=dhcp2
add address-pool=dhcp_voip disabled=no interface=voip name=dhcp1
add address-pool=dhcp_pool8 disabled=no interface=bridge-mgm name=dhcp3
add address-pool=dhcp_pool9 disabled=no interface=users name=dhcp4
add address-pool=dhcp_pool10 disabled=no interface=mgm name=dhcp5
add address-pool=dhcp_pool11 disabled=no interface=tablets name=dhcp6
add address-pool=dhcp_pool12 disabled=no interface=scanprint name=dhcp7
add address-pool=dhcp_pool13 disabled=no interface=timeclocks name=dhcp8
/queue tree
add max-limit=200M name=parent_download parent=bridge-lan priority=1
add max-limit=200M name=parent_upload parent=bridge-wan priority=1
add max-limit=20M name=1.child_rtp_in packet-mark=PBX_RTP_IN parent=parent_download \
    priority=1
add max-limit=5M name=2.child_sip_in packet-mark=PBX_SIP_IN parent=parent_download \
    priority=2
add max-limit=20M name=1.child_rtp_out packet-mark=PBX_RTP_OUT parent=parent_upload \
    priority=1
add max-limit=5M name=2.child_sip_out packet-mark=PBX_SIP_OUT parent=parent_upload \
    priority=2
add max-limit=170M name=3.child_all_other_traffic_in packet-mark=ALL_TRAFFIC_IN parent=\
    parent_download
add max-limit=170M name="3. child_all_other_traffic_out" packet-mark=ALL_TRAFFIC_OUT \
    parent=parent_upload
/interface bridge port
add bridge=bridge-wan interface=sfp1-eth0-combo
add bridge=bridge-lan interface=ether3
add bridge=bridge-wan interface=ether1
add bridge=bridge-wan hw=no interface=ether5
add bridge=bridge-wan interface=ether2
add bridge=bridge-mgm interface=ether6
/ip address
add address=70.134.175.42/30 interface=sfp1-eth0-combo network=70.134.175.40
add address=50.217.129.190/28 interface=ether1 network=50.217.129.176
add address=10.10.1.1/24 interface=bridge-mgm network=10.10.1.0
add address=10.10.12.1/24 interface=voip network=10.10.12.0
add address=10.10.15.1/24 interface=management network=10.10.15.0
add address=10.10.10.1/23 interface=users network=10.10.10.0
add address=10.10.15.1/24 disabled=yes interface=mgm network=10.10.15.0
add address=10.10.17.1/24 interface=scanprint network=10.10.17.0
add address=10.10.16.1/24 interface=tablets network=10.10.16.0
add address=10.10.18.1/24 interface=timeclocks network=10.10.18.0
/ip dhcp-server network
add address=10.10.1.0/24 gateway=10.10.1.1
add address=10.10.10.0/23 gateway=10.10.10.1
add address=10.10.12.0/24 dns-server=10.10.12.1 gateway=10.10.12.1
add address=10.10.15.0/24 dns-server=10.10.15.1 gateway=10.10.15.1
add address=10.10.16.0/24 dns-server=10.10.16.1 gateway=10.10.16.1
add address=10.10.17.0/24 dns-server=8.8.8.8 gateway=10.10.17.1
add address=10.10.18.0/24 dns-server=8.8.8.8 gateway=10.10.18.1
/ip dns
set allow-remote-requests=yes servers=208.67.222.222,69.38.184.2,75.75.75.75,8.8.8.8
/ip firewall address-list
add address=112.211.41.119 comment=freePBX list=voip-server
/ip firewall filter
add action=accept chain=input comment="Allow all Traffic from DNS" src-address-list=\
    allow-DNS
add action=accept chain=forward comment="Allow all Traffic from DNS" src-address-list=\
    allow-DNS
add action=accept chain=output comment="Allow all Traffic from DNS" src-address-list=\
    allow-DNS
add action=accept chain=input comment="Allow all PBX Traffic" src-address=112.211.41.119
add action=accept chain=forward comment="Allow all PBX Traffic" src-address=112.211.41.119
add action=accept chain=output comment="Allow all PBX Traffic" dst-address=112.211.41.119
add action=drop chain=input comment="Drop Malicious IPs" src-address-list=blocked-ips
add action=drop chain=forward comment="Drop Malicious IPs" src-address-list=blocked-ips
add action=drop chain=output comment="Drop Malicious IPs" src-address-list=blocked-ips
add action=accept chain=input comment="Allow admin SSH Requests" dst-port=22 protocol=tcp \
    src-address=28.38.102.191
add action=accept chain=input comment="Allow admin SSH Requests" dst-port=22 protocol=udp \
    src-address=28.38.102.191
add action=drop chain=input comment="Drop all SSH Requests" dst-port=22 protocol=tcp
add action=drop chain=input comment="Drop all SSH Requests" dst-port=22 protocol=udp
add action=drop chain=input comment="Drop all Input DNS Requests" dst-port=53 in-interface=\
    bridge-wan protocol=tcp
add action=drop chain=input comment="Drop all Input DNS Requests" dst-port=53 in-interface=\
    bridge-wan protocol=udp
add action=accept chain=input comment="Allow WinBox from Corp IP - Only!" dst-port=3189 \
    protocol=tcp src-address=28.38.102.191
add action=drop chain=input comment="Drop all other WinBox Requests" dst-port=3189 \
    protocol=tcp
add action=accept chain=input comment="Allow all Site24x7 ICMP" protocol=icmp \
    src-address-list=site24x7
add action=accept chain=input comment="Allow all mikrotik ICMP" protocol=icmp \
    src-address-list=mikrotik
add action=drop chain=input comment="Drop all other ICMP" in-interface=bridge-wan protocol=\
    icmp
add action=accept chain=input comment="Permit established connections" connection-state=\
    established
add action=accept chain=input comment="Permit related connections" connection-state=related
add action=accept chain=input comment="Allow whitelisted sources" src-address-list=\
    Whitelist
add action=drop chain=input comment="WAN - default deny" in-interface=bridge-wan
add action=add-src-to-address-list address-list=blocked-addr address-list-timeout=1d chain=\
    input connection-limit=40000,32 protocol=tcp
add action=tarpit chain=input connection-limit=3,32 protocol=tcp src-address-list=\
    blocked-addr
add action=jump chain=forward comment="SYN Flood protect" connection-state=new jump-target=\
    SYN-Protect protocol=tcp tcp-flags=syn
add action=accept chain=SYN-Protect connection-state=new limit=400,5:packet protocol=tcp \
    tcp-flags=syn
add action=drop chain=SYN-Protect connection-state=new protocol=tcp tcp-flags=syn
add action=accept chain=forward comment="Allow Established Forwarding Requests" \
    connection-state=established,related
add action=accept chain=input comment="Allow Established Input Requests" connection-state=\
    established,related
add action=drop chain=input connection-state=invalid in-interface=bridge-lan
add action=drop chain=forward comment="Drop Forward Invalid Packets" connection-state=\
    invalid
add action=drop chain=forward connection-nat-state=!dstnat connection-state=new \
    in-interface=bridge-lan
/ip firewall mangle
add action=mark-packet chain=prerouting comment=pbx-rtp-in in-interface=bridge-wan \
    new-packet-mark=PBX_RTP_IN passthrough=no protocol=udp src-address=112.211.41.119 \
    src-port=!5060
add action=mark-packet chain=prerouting comment=pbx-sip-in in-interface=bridge-wan \
    new-packet-mark=PBX_SIP_IN passthrough=no protocol=udp src-address=112.211.41.119 \
    src-port=5060
add action=mark-packet chain=prerouting comment=all-traffic in-interface=bridge-wan \
    new-packet-mark=ALL_TRAFFIC_IN passthrough=no src-address=!112.211.41.119
add action=mark-packet chain=postrouting comment=pbx-rtp-out dst-address=112.211.41.119 \
    dst-port=!5060 new-packet-mark=PBX_RTP_OUT out-interface=bridge-wan passthrough=no \
    protocol=udp
add action=mark-packet chain=postrouting comment=pbx-sip-out dst-address=112.211.41.119 \
    dst-port=5060 new-packet-mark=PBX_SIP_OUT out-interface=bridge-wan passthrough=no \
    protocol=udp
add action=mark-packet chain=postrouting comment=all-traffic-out dst-address=\
    !112.211.41.119 new-packet-mark=ALL_TRAFFIC_OUT out-interface=bridge-wan passthrough=no
/ip firewall nat
add action=src-nat chain=srcnat comment=nat-voip-to-comcast-business out-interface=\
    bridge-wan src-address=10.10.0.0/16 to-addresses=70.134.175.41
add action=dst-nat chain=dstnat disabled=yes dst-address=70.134.175.41 dst-port=8080 \
    protocol=tcp src-address=28.38.102.191 to-addresses=10.10.12.214 to-ports=80
add action=dst-nat chain=dstnat disabled=yes dst-address=70.134.175.41 dst-port=8081 \
    protocol=tcp src-address=28.38.102.191 to-addresses=10.10.15.11 to-ports=80
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set sip disabled=yes ports=5060
set pptp disabled=yes
/ip route
add distance=1 gateway=70.134.175.41
add distance=2 gateway=29.18.182.57
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh address=28.38.102.191/32
set api disabled=yes
set winbox address=28.38.102.191/32 port=3189
set api-ssl disabled=yes
/system clock
set time-zone-name=America/Los_Angeles
/system identity
set name=mikrotik_Bensenville
/system logging
add topics=firewall
/system routerboard settings
set silent-boot=no
/tool sniffer
set file-limit=10000KiB file-name=sweet_phone filter-interface=bridge-lan \
    filter-ip-address=10.10.12.221/32

@pe1chl I am not using the IP firewall on this mikrotik either, but the brige traffic is working correctly.
Screen Shot 2019-05-08 at 09.45.28.png

17

I think the problem is the wan bridge itself. For QOS to work, one needs to control the transmission. But in your case traffic is bypassing queue on bridge (because its in hardware / accelerated) which results in unpredictable queueing to ISP.

18

I am going to try to bridge two different ports. Maybe this combo port is causing issues? That’s the only thing I still haven’t tried.

The only difference I have noticed between these mikrotiks is that on the mikrotik that everything works fine, the bridge-wan shows all its participant ports as designated port

As opposed to the mikrotik that im having issues with, shows the combo-eth0-sp1 as the root port. (see attached for both instances.)
I know that the root path cost defines the role of the port, but not sure why they would have different properties anyways.

I am just trying to think out loud since this is the only difference that I see on both mikrotiks.
Screen Shot 2019-05-08 at 13.51.14.png
Screen Shot 2019-05-08 at 13.50.41.png

19

I finally found where the problem was and fixed it!

For those of you who are interested on knowing where the problem was, here you go.

The problem was that the interface that is connected directly to the firewall, who is participant on the bridge-wan, did NOT have an IP address assigned to it.

The other interface had a WAN IP address assigned to it. It makes no difference if the IP address is assigned to the root interface that is participant to the bridge-wan, or if the IP address is assigned to the bridge-wan itself.

You simply need to assign IP addresses on both interfaces that are participant on the bridge-wan - for the bridge to read the whole traffic that is being generated - as opposed to reading only the broadcast traffic that is going on that bridge.

So I called up Cogent and ask them to assign us another subnet, and ask to add a static route of our existing subnet to the old subnet. I didn’t want to lose the current public IP’s that we had for our firewall because we’re using those IP’s to whitelist a ton of cloud servers. And I also don’t like the idea of connecting the router with the firewall with a private subnet and then NAT that subnet to the internet, because the users would have double-natting and I simply don’t like that when there are better ways of doing it.

And so, on the root interface of the bridge-wan, I assigned one of the IP’s from the new subnet, and then on the other interface where the firewall is connected directly to the mikrotik, I assigned an IP from our old subnet and used that IP as the default gateway for the firewall. This way we preserved our old IP, and also made possible to static route the old subnet through the new subnet.

Yes, ultimately this added another hop in our route, but I don’t think that matters?

And bam! The issue was resolved instantly!

I am not entirely sure in details why this had to work this way, but I am just glad I was able to fix it after a long time.

Maybe you guys can help me out on understanding the story behind this issue.

See attached for the actual results.
Screen Shot 2019-07-10 at 11.21.55 copy.png
Screen Shot 2019-07-10 at 11.21.05 copy.png

20

Now your Internet traffic is not L2 switched through Mikrotik but L3 routed (additional hop as you say) and that is why it goes via Bridge.