My network setup looks like this: WAN -------> RB4011 -------> pfSense. I want the pfSense box to get the WAN IP (I don’t want double NAT), and use the MikroTik as a passthrough box of sorts (for AT&T FTTH gateway bypass). What is the most efficient way to get this done? I suspect there is more than one way, and I want to learn the most efficient (CPU cycles, etc) way.
I have attempted this:
Move ether9 and ether10 into a bridge named bridge_wan. Connect modem (for testing) to ether10, it gets WAN IP address. Connect ether9 to pfSense box. When I do this, ether9 has no connection (pfSense cannot pull DHCP from ISP). I feel like I need some kind of route entry, but my networking isn’t strong enough to know, exactly.
How many IP addresses did you get from your ISP? Maybe only one! And the RB4011 has taken one already. Release the IP address and disable DHCP on the bridge to check if the pfSenese router gets it.
Yes, I have one dynamic IP from my ISP. Here is what I attempted (and failed to succeed at):
ether9 - pfSense
ether10 - WAN
Set ether9 and ether10 into a bridge named bridge_wan. Disable all DHCP clients and servers in this bridge. Remove addresses from the ports in this bridge. Make firewall rules to pass absolutely everything. Still, my pfSense box can’t get an IP address. In my mind, I should be able to send the connection from the modem, through the MikroTik, to the pfSense box without anything set, as long as the ports are bridged together, they should act like a dumb switch, no?
If I enable DHCP client/server + DHCP network, I am able to pass internet just fine, albeit the pfSense box gets a 192.xxx address - internet does work fine when I do this. This isn’t my goal right now, however.
It looks as if you are doing NAT on the WAN interface.
The bridge should be on the “LAN interface list”, or in no list at al, if the default firewall rules are in place.
The default NAT rule in the firewall will do NAT on interfaces that are in the WAN interface list
DON’T put the bridge_wan nor ether10 in the WAN list, and ether9 in the LAN list. : WAN /LAN names don’t have the intuitive meaning of facing WAN or LAN. It are just “lists” where the name itself has no function, to be able to write simpler firewall rules and the lists are used in other settings (like MAC server) as well . Even if you are at the WAN-side of the pfSense, you want to make a transparant Local Area Network , a level 2 LAN, not a masqueraded connection to the Internet, at least the masqerading is not in the Mikrotik.
ether9 and ether10 are slave interfaces to the bridge. So the “bridge” named “bridge_wan” matters, not the interfaces.themselves. Don’t spoil your ISP’s IP address on the bridge though. Don’t do NAT on that bridge (look at the NAT rules in ‘IP/ Firewall/ NAT’ for the above default rule, it forces NAT if the bridge is in the “WAN” list !)
Thank you for your continued assistance. Sorry, I did forget to mention that I disabled the NAT rules in the firewall. Also, I made new filter rules in the firewall: one for forward, one for input, one for output - these are accept rules with no restrictions.
So, I have ether9 connected to pfSense, and ether10 connected to my modem. ether9 and ether10 are alone on a bridge named bridge_wan. There is no DHCP client, or DHCP server on ether9, ether10, or bridge_wan. There are no firewall filter or NAT rules to interfere, but my modem is still not handing an IP address to pfSense when I go through the MikroTik.
If I put a DHCP client on the ether10 modem interface, it gets an IP from the modem instantly. If I set up a DHCP server (+DHCP network) on ether9 for pfSense, it gets an IP from the MikroTik instantly and traffic flows correctly. What am I missing? It’s almost like the bridge isn’t working for some reason.
Some ISP-provided modems (or upstream devices) have enabled “feature” which limits communication through that modem only to single device at a time. It usually takes longer time (half an hour, a few hours) after last communication to reset that limitation so that another device can start communicating. Sometimes it helps to cold boot modem itself (if that’s the limiting device).
This is a good point and I did look into that. The pfSense box is set to get the WAN IP. But even if that didn’t work, it would still get a 192.168.xx.xx IP, as the modem is modem/switch combo unit. The MikroTik (when DHCP client is enabled on ether10) gets a 192.168.77.0/24 address (because it’s not allowed access to the WAN IP due to modem settings).
so something that you are doing in Mikrotik device is you set up as bridge, 4011 will act as “transparent” try to do a factory reset without setting up default configurations.
Hmmmmm … some cases that may be related but are different … I had a recent setup for a friend with an ISP for “public wifi” … tried to make him a wifi repeater … the bridge with DHCP client when connected via the WLAN interface set as “station pseudobridge”, could not get an IP address. On the other hand the WLAN interface itself when used alone had no problem for getting an IP address on the same public wifi. ROS version was 6.45.7 or 6.46.2. Never had this experience before !!! (My usual devices are ROS 6.44.5) Solved this then by using NAT and local DHCP server, but my intention for a “repeater” style setup failed.
Other case : DHCP lease can be a complex operation. And sometimes it fails … it always fails for a wireless client if you setup an “universal repeater” aka “pseudo bridge”/ “AP bridge” combination , to connect to MKT with a Mikrotik DHCP server. But in this case the MAC address is changed and the last DHCP handshake never reaches the client. (MKT uses 255.255.255.255 to broadcast , but the MAC is not FF:FF:FF:FF:FF:FF.)
Both experiences are different from this RB4011 case. But maybe also here sniffing the ether9 and ether10 DHCP traffic can learn if all passes through as expected. Maybe then it can be understood why after passing the MKT the pfSense is not getting an IP address. (Timing? Relaying broadcast? disabled DHCP server sending DHCP NACK?, …)
That provider limit is well known. They use small (single slot) DHCP and ARP tables with long timeout (hours). Sometimes releasing the previous DHCP lease or using MAC cloning helps. But here the pfSense is known by the ISP, when directly connected. No problem here.
But also STP timing in the bridge has been known to cause problems. “RSTP” as STP protocol mode is better than STP, but here it can also be “none”.
Wow, I would have never guessed. It was RSTP. If I set STP protocol to none, it works immediately. I switch it back on, Internet dies. I tested this a few times, results are clear. I have another connection to this RB4011, but that connection is on a management network, and I am absolutely not creating a network loop. Must be a bug in STP.
Seriously, thank you for saving the little sanity I had left!
Bridge and reserved MAC addresses
Consider the following scenario, you want to transparently bridge two network segments together, either those are tunnel interfaces like EoIP, Wireless interfaces, Ethernet interface or any other kind of interfaces that can be added to a bridge. Such setups allows you to seamlessly connect two devices together like there was only a physical cable between them, this is sometimes called a transparent bridge from DeviceA to DeviceB.
Configuration
For both devices DeviceA and DeviceB there should be a very similar configuration.
/interface bridge
add name=bridge1 protocol-mode=rstp
/interface bridge port
add interface=ether1 bridge=bridge1
add interface=eoip1 bridge=bridge1
Problem
Both devices are able to communicate with each other, but some protocols do not work properly. The reason is that as soon as you use any STP variant (STP, RSTP, MSTP), you make the bridge compliant with IEEE 802.1D and IEEE 802.1Q, these standards recommend that packets that are destined to 01:80:C2:XX:XX:XX should NOT be forwarded. In cases where there are only 2 ports added to a bridge (R/M)STP should not be used since a loop cannot occur from 2 interfaces and if a loop does occur, the cause is elsewhere and should be fixed on a different bridge. Since (R/M)STP is not needed in transparent bridge setups, it can be disabled. As soon as (R/M)STP is disabled, the RouterOS bridge is not compliant with IEEE 802.1D and IEEE 802.1Q and therefore will forward packets that are destined to 01:80:C2:XX:XX:XX.
Symptoms
Below is a list of possible symptoms that might be as a result of this kind of a misconfiguration:
LLDP neighbors not showing up
802.1x authentication (dot1x) not working
LACP interface not passing traffic
Solution
Since RouterOS v6.43 it is possible to partly disable compliance with IEEE 802.1D and IEEE 802.1Q, this can be done by changing the bridge protocol mode.
/interface bridge
set bridge1 protocol-mode=none
Icon-warn.png
Warning: The 802.1x standard is meant to be used between a switch and a client directly. If it is possible to connect a device between the switch and the client, then this creates a security threat. For this reason it is not recommended to disable the compliance with IEEE 802.1D and IEEE 802.1Q, but rather design a proper network topology.