I am having difficulties configuring a dual internet setup. I can successfully access the internet through isp1 but for the second isp it works for incomming connections to our webservers, sending out the web pages as expected but if the webservers try to go out themselves (for instance a ping), they fail.

Ether1 is Cerberus ISP, PPPoE connection (which works for connection initiated from internet only)
Ether2 is LAN 192.168.1.0/24 on ether-2 which can successfully browse the internet (via Brsk ISP on Ether5 as default)
Ether3 and Ether4 bridged as DMZ 192.168.0.0/24 (this is where the webservers are)
Ether5 is Brsk ISP, DHCP client connection (is default and works well)

If I have no rules then the DMZ goes out via Ether5 (default) and can browse the internet but I want DMZ to go out via Ether1 so have created mangle rules. Obviously I'm doing something wrong as with this rule in place the Webservers can successfully serve pages to the internet but if I try to do a ping from them to the internet it fails. It seems that if a connection is initiated from outside they work but if the connection is initiated inside they dont.

Config attached.

Your WAN for Cerberus is PPPoE?
If not what is the PPPoE for?

I think your masquerade rule is wrong
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface=ether1-WAN-Cerberus
Change to
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface=Cerberus_PPPoE

And if your WAN for cerberus is PPPoE you don't need the DHCP Client on ether1-WAN-Cerberus

My confusion is the statement.......... "but if the webservers try to go out themselves".
That is not the function of web servers??

What you are really getting at is that what is happening is users are reaching the web servers externally ( probably by dydns url unless you have a static IP - missing info! ) but the REPLY TRAFFIC is going out the other ISP and thus not working.

Why are you creating a DMZ??
Do you mean ether2 is not on the bridge and has its own subnet and ether3,and ether4 are on a bridge and that bridge has its own subnet??

+++++++++++++++++++++++++++++++++++++++++++++++

After looking at the config, its very confused... suggest you read and implement this --> viewtopic.php?t=143620

Hi guys, thanks for your replies.

@aoakeley - Thanks, the change you suggested to the NAT now allows a successful ping from the webserver consoles out to the internet

@anav - Sorry if my post wasn't clear enough, I am new to this and obviously wrongly assumed that the fixed IPs and bridges were obvious in the config.

For clarity, the Brsk line uses DHCP and has no fixed internet IP. The Cerberus line uses PPPoE and has 3 available static IPs ending 77, 78 and 79, each of which are NATted through on ports 80 and 443 to different webservers. As I said in my initial post, if someone on the internet accesses one of the websites hosted on these servers, they do receive a reply (all be it VERY slowly) so the reply traffic IS leaving by the Cerberus line as expected.

By "trying t to out themselves" I mean if the webservers try to access the intenet in their own right rather than replying to an externally initiated session. For instance on of the webservers talks to the ebay API to retrieve data but even though the ping now works these API calls fail as no reply is received from ebay. There has been no coding change and it works if I remove the mangle telling the 192.168.0.0/24 subnet to go out via Cerberus.

Ether2 is in LAN_bridge on it's own on subnet 192.168.1.0/24. Ether 3 and 4 are on DMZ_Bridge on subnet 192.168.0.0/24 so yes, they are separate subnets.

So my problem persists, although I can now ping the internet from the Webservers, they are unable to retrieve data from the internet themselves and anyone using the websites from the internet gets unuseably slow responses from the webservers though they do evenutally respond (a minute to fully load a page tha normally takes a second).

Thanks for the link to the article but it's about VLANs and I don't use VLANs

Well your forward chain rules are default so.
a. dst-nat from wan interface is permitted to reach LAN
b. there is no LAN to WAN blocking going on.

Which means there is no logic to explain why your servers cannot reach the net.
Just for giggles attach your PC to the bridge port going to the server (vice the server) and see if you can access the internet.

Set this all to none: /interface detect-internet
set detect-interface-list=WAN internet-interface-list=WAN lan-interface-list=\
LAN wan-interface-list=WAN

++++++++++++++++++++++++++++++++++++

Besides the MAIN issue of not having pppoe-out1 in your SRC-NAT RULE .......... already described and fixed.

The other ISSUE IS FW RULES and INTERFACE LIST. You are inconsistent as well, why do you assign bridge to LAN interface but not the ether ports, but then elect to put ether3 or 4 to the DMZ interface, but not their bridge??

From this:
/interface list member
add comment=defconf interface=LAN_bridge list=LAN
add comment=defconf interface=ether1-WAN-Cerberus list=WAN
add interface=ether5-WAN-Brsk list=WAN
add interface=ether3 list=DMZ
add interface=ether4 list=DMZ

TO:
/interface list member
add comment=defconf interface=Cerberus_PPPoE list=WAN
add interface=ether5-WAN-Brsk list=WAN
add comment=defconf interface=LAN_bridge list=LAN
add interface=DMZ_bridge list=LAN

Note: There is no need for an interface list DMZ that I can see based on the config.

ALTERNATIVELY
/interface list member
add comment=defconf interface=Cerberus_PPPoE list=WAN
add interface=ether5-WAN-Brsk list=WAN
add comment=defconf interface=LAN_bridge list=LAN
add interface=DMZ_bridge list=DMZ

and modify any fw rules that refer to LAN........ currently one!

From:
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
in-interface-list=!LAN

TO:
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
in-interface-list=!LAN
add action=drop chain=input comment="defconf: drop all not coming from DMZ" \
in-interface-list=!DMZ

In summary either use the LAN interface list as per normal to cover off all lan subnets, or keep two interface lists for lan subnets and make the additional fw rules required.

Hi Mesquite

Thanks for your suggestions, they moved me forward to finding the solution.

I implemented your suggested config changes and could then browse the web from the servers but just like when they replied to external requests it was basically unuseably slow (like 1980's dial up).

As you suggested, I tried my PC on Ether4 in the DMZ along with the servers and then it got the very slow internet (via Cerberus as expected) but as soon as it was plugged back into the LAN it was fast again but obviously going out via Brsk. That confirmed it wasn't just something to do with the servers themselves.

I then added Cerberus at distance 2 in routes so it was a backup for the default LAN in the case that Brsk failed. When I disabled Brsk, Cerberus then became the default route and suprise suprise it was fast! That confirmed that it was probably something to do with the mangle rule.

I then looked at how I might route the DMZ to Cerberus without using mangle and found routing rules so I set up a rule for the DMZ subnet to "lookup only in table" To_Cerberus. When I did that I wasn't able to communicate with the DMZ servers from the LAN so added another rule so that any DMZ traffic destined for LAN used "lookup" main. I could then talk to the servers from the LAN so I disabled the DMZ mangle rule.

I then tested internet browsing from the servers and it was full speed Checking with whatismyip.com confirmed that they were using Cerberus and checking the same from my PC it was using Brsk. Checking from an external device on the internet the servers were also now responding at full speed. Problem solved!

So the cause of the slow response was the prerouting mangle rule marking the routing. Doing the same in routing rules instead solved the issue. No idea why this should be the case but I'm happy anyway.

Thanks for your support guys, you pointed me in the right direction to find my solution.

Mangle rules and fastrack dont play nice so maybe something there too.
Ensure in marking connections you state connection-mark=no-mark and then in the fastrack rule add the same connection-mark=no-mark

Senario

2 X WAN connections- 1 X Ethernet 1 X PPPoE (both fixed IP's)
Ethernet connection is the primary connection for the LAN
PPPoE services all the DMZ traffic and acts as failover for the ethernet
1 X LAN 10.0.1.0/24
1 X DMZ with 3 web servers 10.0.2.0/24
SRCNAT works fine as does failover on both WAN connections
DSTNAT forwarding works perfectly for incoming web server requests on the PPPoE connection
2 route tables Main and DMZ
DMZ route table has 'look up only in table' rule

BUT I can't for the life of me access any of the web servers using the external PPPoE resolved IP address and now I can't get past the router DMZ address from the LAN. I've tried umpteen configurations to no avail. Gone round in circles actually. I tried hairpin and simple routing. If I get one side working it breaks the other.

Rather than share a clearly incorrect set of configs can anyone share a working configuration to try?

I would have to see the full config to comment, especially as the explanation is not clear.

@TheIBM

Your config sounds pretty similar to mine with 1 DHCP WAN and 1 PPPoE WAN and your problems sound similar. Mesquite's pointers in his first post certainly helped so I'd check those out but I did have to play around to get it working after.

It's difficult to say without your config but you possibly need to manually set up another "duplicate" route to your PPPoE but with a specific mark on it (I used "To_Cerberus") so that you can then set up a route rule to mark DMZ traffic to use that WAN connection. I do that with 2 route rules, 1 to force DMZ traffic to be marked to main if it's destined for LAN and 1 to mark it "To_Cerberus" for all other destinations. If I didn't have the first rule then I couldn't access the servers in the DMZ from LAN.

I've attached my working config so you can take a look.

Hope that helps.