Cross Connect between CCRs

mjezierski · August 17, 2021, 2:26pm

So I have a customer who wants to have the option to cross connect their Mikrotik CCR1009’s and join two portable networks together. They currently connect each CCR to a WAN (which is us) and cross connect their own ASAs together. They want to eliminate cross-connecting their ASAs and use the Mikrotik as a cross connect. Here’s a couple diagrams that I think will help explain what I want to do here:
Conf A.png
This is with each CCR connected separately to the WAN. There could be a cross connect between the ASAs but not always depending upon whether both portable networks are at the same location or not. The diagram shows one LAN but its two /24 LAN subnets, users freely traverse between the two subnets.
Conf B.png
This is with one CCR connected to the WAN, one not connected to a WAN and cross connected using port ETH4 on the CCRs

# model = CCR1009-7G-1C
/interface bridge
add name=bridge1
/interface ethernet
set [ find default-name=ether1 ] speed=100Mbps
set [ find default-name=ether2 ] speed=100Mbps
set [ find default-name=ether3 ] speed=100Mbps
set [ find default-name=ether4 ] speed=100Mbps
set [ find default-name=ether5 ] speed=100Mbps
set [ find default-name=ether6 ] speed=100Mbps
set [ find default-name=ether7 ] mac-address=6C:3B:6B:EB:2A:8E speed=100Mbps
/interface bonding
add mode=active-backup name=bonding1 slaves=ether7,ether6
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc
/ip pool
add name=pool1 ranges=dmz.ip.assigned.to.asa.via.dhcp
/ip dhcp-server
add address-pool=pool1 disabled=no interface=ether5 name=isp_single_address
/snmp community
**SNMP STUFF**
/user group
***BLAH BLAH***
/interface bridge filter
add action=drop chain=forward comment="Drop all DHCP Requests over EOIP bridge" dst-port=67 ip-protocol=udp mac-protocol=ip
/interface bridge port
add bridge=bridge1 interface=combo1
add bridge=bridge1 interface=ether1
add bridge=bridge1 interface=ether2
add bridge=bridge1 interface=ether3
#
/ip address
add address=outside.wan.ip comment="Outside IP - TEAMNET" interface=bonding1 network=w.x.y.z
add address=inside.lan.ip comment="Inside IP - LAN" interface=bridge1 network=a.b.c.d
add address=dmz.to.customer.asa.ip interface=ether5 network=e.f.g.h
add address=cross-connect.to.second.mikrotik.ip interface=ether4 network=j.k.l.m
/ip dhcp-server network
add address=dmz.to.customer.ip.dhcp dns-server=dns.server.ip gateway=dmz.to.customer.asa.ip
##
/ip firewall filter
add action=drop chain=input comment="Drop Invalid Connections" connection-state=invalid
add action=drop chain=forward comment="Drop Invalid Connections" connection-state=invalid
add action=accept chain=input comment="Allow Connections From LAN (DNS...)" src-address=inside.lan.network
add action=accept chain=input src-address=outside.wan.network
add action=accept chain=input src-address=dmz.to.customer.asa.network
add chain=input comment="Allow Established Connections" connection-state=established
add chain=input comment="Allow EOIP Tunnels" protocol=gre
add chain=input comment="Allow Pings" protocol=icmp
add chain=input comment="WinBox Wan Administration" dst-port=8291 protocol=tcp
add action=accept chain=input comment="ZABBIX SNMP" dst-port=161 protocol=udp src-address=zabbix.proxy.ip
add action=accept chain=input comment="SSH from Teams Net (IC)" dst-port=22 protocol=tcp src-address=network.range.for.ssh
add action=accept chain=input dst-port=443 protocol=tcp
add action=drop chain=input comment="Drop Remaining Inputs" log=yes
add action=accept chain=forward comment="Restrict New Connections to being sourced from LAN only" connection-state=new src-address=inside.lan.network
add action=accept chain=forward src-address=outside.wan.network
add action=accept chain=forward src-address=dmz.network
add chain=forward comment="Allow Related Connections" connection-state=related
add chain=forward comment="Allow Established Connections" connection-state=established
add action=accept chain=forward comment="Allow Team Router WAN out network" in-interface=ether5 out-interface=bonding1
add action=drop chain=forward comment="Drop Remaining Forward Chain" log=yes
/ip firewall nat
add action=masquerade chain=srcnat comment="Can be used to NAT through one IP" out-interface=bonding1
/ip firewall service-port
set pptp disabled=yes
/ip route
add distance=1 gateway=wan.gateway.ip.address
/ip service
**BUNCH OF DISABLED SERVICES
**SNMP AND SNTP CONFIGURATIONS
/tool netwatch
add down-script="interface ethernet disable ether5" host=wan.gateway.ip.address interval=9s up-script="interface ethernet enable ether5"

These are the CCRs above and below.

# model = CCR1009-7G-1C
/interface bridge
add name=bridge1
/interface ethernet
set [ find default-name=ether1 ] speed=100Mbps
set [ find default-name=ether2 ] speed=100Mbps
set [ find default-name=ether3 ] speed=100Mbps
set [ find default-name=ether4 ] speed=100Mbps
set [ find default-name=ether5 ] speed=100Mbps
set [ find default-name=ether6 ] speed=100Mbps
set [ find default-name=ether7 ] speed=100Mbps
/interface bonding
add mode=active-backup name=bonding1 primary=ether7 slaves=ether7,ether6
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc
/ip pool
add name=pool1 ranges=dmz.ip.for.asa.assigned.by.dhcp
/ip dhcp-server
add address-pool=pool1 disabled=no interface=ether5 name=isp_single_address
/snmp community
**SNMP STUFF
/user group
**USER GROUP STUFF
/interface bridge filter
add action=drop chain=forward comment="Drop all DHCP Requests over EOIP bridge" dst-port=67 ip-protocol=udp mac-protocol=ip
/interface bridge port
add bridge=bridge1 interface=combo1
add bridge=bridge1 interface=ether1
add bridge=bridge1 interface=ether2
add bridge=bridge1 interface=ether3
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/ip address
add address=outside.wan.ip comment="Outside IP - TEAMNET" interface=bonding1 network=w.x.y.z
add address=inside.lan.ip comment="Inside IP - LAN" interface=bridge1 network=a.b.c.d
add address=dmz.to.customer.asa.ip interface=ether5 network=e.f.g.h
add address=cross-connect.to.second.mikrotik.ip interface=ether4 network=j.k.l.m
/ip dhcp-server network
add address=dmz.to.customer.ip.dhcp dns-server=dns.server.ip gateway=dmz.to.customer.asa.ip
#
/ip firewall filter
add action=drop chain=input comment="Drop Invalid Connections" connection-state=invalid
add action=drop chain=forward comment="Drop Invalid Connections" connection-state=invalid
add action=accept chain=input comment="Allow Connections From LAN (DNS...)" src-address=inside.lan.network
add chain=input comment="Allow Established Connections" connection-state=established
add chain=input comment="Allow EOIP Tunnels" protocol=gre
add chain=input comment="Allow Pings" protocol=icmp
add chain=input comment="WinBox Wan Administration" dst-port=8291 protocol=tcp
add action=accept chain=input comment="Zabbix SNMP" dst-port=161 protocol=udp src-address=zabbix.proxy.ip
add action=accept chain=input comment="SSH from IC Teams Net" dst-port=22 protocol=tcp src-address=network.range.for.ssh
add action=drop chain=input comment="Drop Remaining Inputs"
add action=accept chain=forward comment="Restrict New Connections to being sourced from LAN only" connection-state=new src-address=inside.lan.network
add chain=forward comment="Allow Related Connections" connection-state=related
add chain=forward comment="Allow Established Connections" connection-state=established
add action=accept chain=forward comment="Allow Team Router WAN out network" in-interface=ether5 out-interface=bonding1
add action=drop chain=forward comment="Drop Remaining Forward Chain"
/ip firewall nat
add action=masquerade chain=srcnat comment="Can be used to NAT through one IP" out-interface=bonding1
/ip route
add distance=1 gateway=wan.gateway.ip.address
**SNMP SNTP AND DISABLED SERVICES
/tool netwatch
add down-script="interface ethernet disable ether5" host=wan.gateway.ip.address interval=9s up-script="interface ethernet enable ether5"

In any case, the internal networks will still be behind the ASAs. The only reason there is an internal LAN address assigned to the bridge is so the customer’s IT person can plug into ETH-ETH3 and get to the skinned webfig for troubleshooting. I warned the customer about the possibility of network loops in this scenario and they assured me they won’t make any loops. Famous last words.

My first attempt was making a 2nd bonding connection (active/backup) of ETH4 and bonding1 and set the masquerade NAT to bonding2 but I couldn’t get that to work.

sindy · August 17, 2021, 6:26pm

I’m quite confused. You mention a danger of loops, and there are some brifge filter rules whose comments mention a relationship to EoIP, but there are no EoIP interfaces in the configurations. Also, ether4 and ether5 are not member ports of the same bridge, nor of any bridge at all for that matter, so I can see no danger of an L2 loop once you interconnect the ether4s even if both ether5s are connected to the ASA(s).

But I also cannot see any routes except the default one via WAN, so - does the ASA act as a bridge to the customer network, i.e. is there L2 transparency from CCR1.ether5 to CCR2.ether5 when each is connected to the ASA? If not, how does a CCR know that the ASA is the gateway towards the customer network subnet(s)?

If the LAN of CCR1 and the LAN of CCR2 use different subnets, I’d use OSPF to let the company network and the two CCRs to find the routes dynamically depending on the currently available interconnection path, but maybe the setup is actually different?