routing between seperate 'protected' LANs

patr · October 8, 2009, 7:36pm

Hi

so I am new to microtik but not routers/routing but I’m having issues that I think I should have sorted but its not making sense to me:

here is the situation. (note: I upgraded to 3.30 which solved another issue I had)

I have three LAN connections (seperate and must be filtered/firewalled), “e2, e3 and e4” and 2 WAN links “e1-gateway” and “e5-gateway”

traffic from each of the three LANs gets routed to each of the 2 WANs based on IP, priority, service, failover etc. AND ALL OF THAT WORKS properly.

I can ping the router from each LAN. I can ping each LAN from the router. I can ALSO ping the routers LAN interface, from another LAN. But no traffic is working LAN to LAN.
I thought of course it must be firewall rules but even with all of the off, and a general “forward accept” rule, I am still getting no dice. I did a packet trace, and the router is getting the packets, but they aren’t leaving the router, as if the routing table was bad.

someone give me a tip here:

I added this to /ip firewall filter, JUST TO TEST:

0 chain=forward action=accept

here is what I have in /ip route (other stuff deleted):

7 ADC 192.168.0.0/24 192.168.0.1 0 e4
8 ADC 192.168.1.0/24 192.168.1.1 0 e3
9 ADC 192.168.2.0/24 192.168.2.1 0 e2

here is what I have in /ip firewall nat:

Flags: X - disabled, I - invalid, D - dynamic
0 chain=srcnat action=masquerade out-interface=e1-gateway
1 chain=srcnat action=masquerade out-interface=e5-gateway2

But I cant get ANYTHING to go from e2<->e4 !! help me !! It should work, if I can ping the router’s interface …

fewi · October 8, 2009, 8:04pm

Can you post all your firewall filter, nat and mangle rules?

patr · October 8, 2009, 9:52pm

ip firewall filter: (note #12 and #13 are just for testing to try and resolve this… but they dont work anyways)

[admin@RR_gate] /ip firewall filter> p
Flags: X - disabled, I - invalid, D - dynamic
 0   ;;; Added by webbox
     chain=input action=accept protocol=icmp

 1   ;;; Added by webbox
     chain=input action=accept connection-state=established in-interface=e1-gateway

 2   chain=input action=accept connection-state=related in-interface=e1-gateway

 3   chain=input action=drop in-interface=e1-gateway

 4   ;;; fwd from e1-gw to customer chain
     chain=forward action=jump jump-target=customer in-interface=e1-gateway

 5   chain=customer action=accept connection-state=established

 6   chain=customer action=accept connection-state=related

 7   chain=customer action=drop

 8   chain=input action=accept connection-state=established in-interface=e5-gateway2

 9   chain=input action=accept connection-state=related in-interface=e5-gateway2

10   ;;; fwd from e5-gw to customer chain
     chain=forward action=jump jump-target=customer in-interface=e5-gateway2

11   chain=input action=drop in-interface=e5-gateway2

12 X ;;; fwd LAN to VOIP accept
     chain=forward action=accept dst-address-list=VOIP-SUBNET in-interface=e4-168-0-1

13 X ;;; fwd VOIP to LAN accept
     chain=forward action=accept dst-address-list=LAN-SUBNET in-interface=e2

ip firewall nat:

[admin@RR_gate] /ip firewall nat> p
Flags: X - disabled, I - invalid, D - dynamic
 0   ;;; Added by webbox
     chain=srcnat action=masquerade out-interface=e1-gateway

 1   chain=srcnat action=masquerade out-interface=e5-gateway2

ip firewall mangle (note #1 is disabled because I dont need it, it is also for testing):

[admin@RR_gate] /ip firewall mangle> p
Flags: X - disabled, I - invalid, D - dynamic
 0   chain=prerouting action=mark-routing new-routing-mark=voip passthrough=yes src-address-list=VOIP-SUBNET

 1 X chain=prerouting action=mark-routing new-routing-mark=voip passthrough=no dst-address-list=VOIP-SUBNET

 2   chain=prerouting action=mark-routing new-routing-mark=other passthrough=yes src-address-list=LAN-SUBNET

 3   chain=prerouting action=mark-packet new-packet-mark=voip passthrough=no routing-mark=voip

 4   chain=prerouting action=mark-packet new-packet-mark=other passthrough=no routing-mark=other

and just for completeness:

ip address

[admin@RR_gate] /ip address> p
Flags: X - disabled, I - invalid, D - dynamic
 #   ADDRESS            NETWORK         BROADCAST       INTERFACE
 0   192.168.2.1/24     192.168.2.0     192.168.2.255   e2
 1   192.168.0.1/24     192.168.0.0     192.168.0.255   e4-168-0-1
 2   192.168.1.1/24     192.168.1.0     192.168.1.255   e3
 3 D aa.bb.cc.dd/24    aa.bb.cc.0       aa.bb.cc.255    e1-gateway
 4 D ww.xx.yy.zz/24    ww.xx.yy.0      ww.xx.yy.255   e5-gateway2

ip route

[admin@RR_gate] /ip route> p detail
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
 0 ADS  dst-address=0.0.0.0/0 gateway=64.180.60.254 interface=e1-gateway gateway-state=reachable distance=0 scope=30
        target-scope=10

 1   S  dst-address=0.0.0.0/0 gateway=aa.bb.cc.254 interface=e1-gateway gateway-state=reachable distance=2 scope=30
        target-scope=10 routing-mark=other

 2 A S  dst-address=0.0.0.0/0 gateway=aa.bb.cc.254 interface=e1-gateway check-gateway=ping gateway-state=reachable
        distance=1 scope=30 target-scope=10 routing-mark=voip

 3 A S  dst-address=0.0.0.0/0 gateway=ww.xx.yy.254 interface=e5-gateway2 check-gateway=ping gateway-state=reachable
        distance=1 scope=30 target-scope=10 routing-mark=other

 4   S  dst-address=0.0.0.0/0 gateway=ww.xx.yy.254 interface=e5-gateway2 gateway-state=reachable distance=2 scope=30
        target-scope=10 routing-mark=voip

 5  DS  dst-address=0.0.0.0/0 gateway=ww.xx.yy.254 interface=e5-gateway2 gateway-state=reachable distance=0 scope=30
        target-scope=10

 6 ADC  dst-address=aa.bb.cc.0/24 pref-src=aa.bb.cc.dd interface=e1-gateway distance=0 scope=10

 7 ADC  dst-address=192.168.0.0/24 pref-src=192.168.0.1 interface=e4-168-0-1 distance=0 scope=10

 8 ADC  dst-address=192.168.1.0/24 pref-src=192.168.1.1 interface=e3 distance=0 scope=200

 9 ADC  dst-address=192.168.2.0/24 pref-src=192.168.2.1 interface=e2 distance=0 scope=10

10 ADC  dst-address=ww.xx.yy.0/24 pref-src=ww.xx.yy.zz interface=e5-gateway2 distance=0 scope=10

as you can see, voip traffic is on e2, and works out and in (as does nat from/to the asterisk server). regular lan traffic is on e4 and works out and in. LAN traffic goes out through e5. voip traffic goes out through e1. they fall back to each other when one fails, no problem.

I just cant route from e2 to e4 !! (and vice-versa, and same with e3).

fewi · October 8, 2009, 11:30pm

Try these:

/ip interface filter
add chain=input action=accept protocol=icmp
add chain=input action=accept connection-state=established in-interface=e1-gateway
add chain=input action=accept connection-state=related in-interface=e1-gateway
add chain=input action=drop in-interface=e1-gateway
add chain=input action=accept connection-state=established in-interface=e5-gateway2
add chain=input action=accept connection-state=related in-interface=e5-gateway2
add chain=input action=drop in-interface=e5-gateway2
add chain=input action=accept
add chain=forward action=accept connection-state=established in-interface=e1-gateway
add chain=forward action=accept connection-state=related in-interface=e1-gateway
add chain=forward action=drop in-interface=e1-gateway
add chain=forward action=accept connection-state=established in-interface=e5-gateway2
add chain=forward action=accept connection-state=related in-interface=e5-gateway2
add chain=forward action=drop in-interface=e5-gateway2
add chain=forward action=accept

instead of the set you have.

patr · October 13, 2009, 9:59pm

that still doesn’t work.

To be honest, by adding ‘chain=foward action=accept’ to my original script, IMHO it should do the same thing.

This is what I have, still cant ping or anything else through the routerboard 450 (from e2 to either of e3 or e4). the packet trace show it is getting the packets… but not sure where its dropping them:

[admin@ww.xx.yy.zz] /ip firewall filter> p
Flags: X - disabled, I - invalid, D - dynamic
 0   chain=input action=accept protocol=icmp

 1   chain=input action=accept connection-state=established in-interface=e1-gateway

 2   chain=input action=accept connection-state=related in-interface=e1-gateway

 3   chain=input action=drop in-interface=e1-gateway

 4   chain=input action=accept connection-state=established in-interface=e5-gateway2

 5   chain=input action=accept connection-state=related in-interface=e5-gateway2

 6   chain=input action=drop in-interface=e5-gateway2

 7   chain=input action=accept

 8   chain=forward action=accept connection-state=established in-interface=e1-gateway

 9   chain=forward action=accept connection-state=related in-interface=e1-gateway

10   chain=forward action=drop in-interface=e1-gateway

11   chain=forward action=accept connection-state=established in-interface=e5-gateway2

12   chain=forward action=accept connection-state=related in-interface=e5-gateway2

13   chain=forward action=drop in-interface=e5-gateway2

14   chain=forward action=accept

fewi · October 13, 2009, 10:21pm

That should work.

How are you testing - just via ping? What are you pinging from, and what are you pinging to? Are you sure the hosts you’re pinging don’t have host firewalls that don’t permit ICMP?

patr · October 14, 2009, 12:22am

yes

but just to prove it:

from the router:

[admin@ww.xx.yy.zz] /tool sniffer> print
          interface: all
       only-headers: no
       memory-limit: 10
          file-name: ""
         file-limit: 10
  streaming-enabled: no
   streaming-server: 0.0.0.0
      filter-stream: yes
    filter-protocol: ip-only
    filter-address1: 192.168.2.198/32:0-65535
    filter-address2: 192.168.0.216/32:0-65535
            running: no
[admin@RR_gate] /tool sniffer> start

from a wintel box:

C:\Documents and Settings\patr>ipconfig

Windows IP Configuration

Ethernet adapter Local Area Connection:

        Connection-specific DNS Suffix  . : fubar.yourdomain.com
        IP Address. . . . . . . . . . . . : 192.168.0.216
        Subnet Mask . . . . . . . . . . . : 255.255.255.0
        Default Gateway . . . . . . . . . : 192.168.0.1

C:\Documents and Settings\patr>ping 192.168.2.198

Pinging 192.168.2.198 with 32 bytes of data:

Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 192.168.2.198:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

C:\Documents and Settings\patr>

and then the result, from the router again:

[admin@ww.xx.yy.zz] /tool sniffer> stop
[admin@ww.xx.yy.zz] /tool sniffer> packet
[admin@ww.xx.yy.zz] /tool sniffer packet> print
 # TIME    INTERFACE SRC-ADDRESS                                 DST-ADDRESS
 0 3.888   e4-168... 192.168.0.216                               192.168.2.198
 1 3.888   e4-168... 192.168.0.216                               192.168.2.198
 2 8.898   e4-168... 192.168.0.216                               192.168.2.198
 3 8.898   e4-168... 192.168.0.216                               192.168.2.198
 4 14.398  e4-168... 192.168.0.216                               192.168.2.198
 5 14.398  e4-168... 192.168.0.216                               192.168.2.198
 6 19.898  e4-168... 192.168.0.216                               192.168.2.198
 7 19.898  e4-168... 192.168.0.216                               192.168.2.198
[admin@ww.xx.yy.zz] /tool sniffer packet>

no packets on e2 (which is where 192.168.2.198 is, as you can see from above configuration). Again, I can ping from the wintel box to 192.168.2.1 (the router’s interface to that subnet), and I can ping from the router to 192.168.2.198, I just cant ping across. Same for all other protocols.

If it is indeed a firewall filter issue, it should have been resolved by adding “chain=forward action=accept” to the top of the original config. Unless I am misunderstanding, but I am really stumped by this.

patr · October 14, 2009, 12:40am

here is an outbound sip call from 192.168.2.198 (going through e1-gateway). As mentioned, I just cant talk across internal subnets. e2<->e3<->e4 oe e2<->e4. Its got to be something in the config but I cant see it, nor find it.

[admin@RR_gate] /tool sniffer> print
          interface: all
       only-headers: no
       memory-limit: 10
          file-name: ""
         file-limit: 10
  streaming-enabled: no
   streaming-server: 0.0.0.0
      filter-stream: yes
    filter-protocol: ip-only
    filter-address1: 192.168.2.198/32:0-65535
    filter-address2: 0.0.0.0/0:0-65535
            running: no
[admin@RR_gate] /tool sniffer> start
[admin@RR_gate] /tool sniffer> stop
[admin@RR_gate] /tool sniffer> packet print
 # TIME    INTERFACE SRC-ADDRESS                                 DST-ADDRESS
 0 4.479   e2        192.168.2.198:41874                          192.168.2.1:53 (dns)
 1 4.479   e2        192.168.2.198:41874                          192.168.2.1:53 (dns)
 2 4.48    e2        192.168.2.1:53 (dns)                        192.168.2.198:41874
 3 4.758   e2        192.168.2.198:5060 (sip)                     ee.ff.gg.hh:5060 (sip)
 4 4.758   e2        192.168.2.198:5060 (sip)                     ee.ff.gg.hh:5060 (sip)
 5 4.796   e1-gat... ee.ff.gg.hh:5060 (sip)                    192.168.2.198:5060 (sip)
 6 4.796   e2        ee.ff.gg.hh:5060 (sip)                    192.168.2.198:5060 (sip)
 7 4.798   e1-gat... ee.ff.gg.hh:5060 (sip)                    192.168.2.198:5060 (sip)
 8 4.798   e2        ee.ff.gg.hh:5060 (sip)                    192.168.2.198:5060 (sip)
 9 4.798   e2        192.168.2.198:5060 (sip)                     ee.ff.gg.hh:5060 (sip)
... 
[admin@RR_gate] /tool sniffer>

fewi · October 14, 2009, 12:40am

Me neither, I’m afraid.

I hope someone else can help you.

patr · October 14, 2009, 1:10am

ok. More debugging. I think I found the issue, but to me, this is a bug in the routing table, or else a major doc issue:

I added a filter rule to log the same packet (i.e. anything from 192.168.0.216 to 192.128.2.198)

then I went to 192.168.0.216 and did a ping. This is what I got:

23:19:46 firewall,info forward: in:e4-168-0-1 out:e5-gateway2, src-mac 00:15:58:32:f9:8e, proto ICMP (type 8, code 0), 192.168.0.216->192.168.2.198, len 40

*** note that the out interface is e5-gateway2 !!! ***

but this is wrong, because (scroll to the top to see), my routing table looks like this (I’ve dropped out the irrelevant routes):

[admin@ff.gg.hh.ii] /ip route> p detail
 0 ADS  dst-address=0.0.0.0/0 gateway=aa.bb.cc.254 interface=e1-gateway gateway-state=reachable distance=0
        scope=30 target-scope=10

 1   S  dst-address=0.0.0.0/0 gateway=aa.bb.cc.254 interface=e1-gateway gateway-state=reachable distance=2
        scope=30 target-scope=10 routing-mark=other

 2 A S  dst-address=0.0.0.0/0 gateway=aa.bb.cc.254 interface=e1-gateway check-gateway=ping
        gateway-state=reachable distance=1 scope=30 target-scope=10 routing-mark=voip

 3 A S  dst-address=0.0.0.0/0 gateway=ww.xx.yy.254 interface=e5-gateway2 check-gateway=ping
        gateway-state=reachable distance=1 scope=30 target-scope=10 routing-mark=other

 4   S  dst-address=0.0.0.0/0 gateway=ww.xx.yy.254 interface=e5-gateway2 gateway-state=reachable
        distance=2 scope=30 target-scope=10 routing-mark=voip

 5  DS  dst-address=0.0.0.0/0 gateway=ww.xx.yy.254 interface=e5-gateway2 gateway-state=reachable
        distance=0 scope=30 target-scope=10

 6 ADC  dst-address=aa.bb.cc.0/23 pref-src=aa.bb.cc.dd interface=e1-gateway distance=0 scope=10

 7 ADC  dst-address=192.168.0.0/24 pref-src=192.168.0.1 interface=e4-168-0-1 distance=0 scope=10

 8 ADC  dst-address=192.168.1.0/24 pref-src=192.168.1.1 interface=e3 distance=0 scope=200

 9 ADC  dst-address=192.168.2.0/24 pref-src=192.168.2.1 interface=e2 distance=0 scope=10

10 ADC  dst-address=ww.xx.yy.0/24 pref-src=ww.xx.yy.zz interface=e5-gateway2 distance=0 scope=10
[admin@ff.gg.hh.ii] /ip route>

this packet falls into “routing-mark=other”

routerOS is looking at the routing table and IGNORING the IP address over the routing mark, in other words, I only have a set of default routes for routing-mark=other. but this is right, I just want the DEFAULT route for routing-mark=other to go over e5, not ALL ROUTES. Shouldn’t the routes defined for local networks match the IP over the routing-marks ? so now I need to setup static routes for every routing mark I have ? this cant be correct…

basically, we need to move the IP address matching “up” the list.

patr · October 14, 2009, 1:24am

so it works now, but this is what I would call a bug. Why ?

well:

I have to add a static route for existing, already established routes (with no specific routing marks), for each attached route, for each routing mark I have defined. BUT I dont have to add a route to the originating subnet, of any packets that have routing marks on them. This subtelty shows that by default, it WILL check the IP table and route based on no routing marks, unless its to a different subnet.

In other words, all I have to do is:

add dst-address=192.168.2.0/24 routing-mark=other gateway=192.168.2.1

this adds the route for “other” packets to the other subnet, even though there is already a rule for this without routing-mark defined.

and

add dst-address=192.168.0.0/24 gateway=192.168.0.1 routing-mark=voip

this sets up the reverse direction. Now multiply all your routes by all subnets, and that is what you have to add.

By default, I have 5 LANs (and that is assuming I have no interfaces with more than 1 address), and if I even only have 2 routing marks, that is 10 static routes defined, all of which IMHO should work out of the box. If you define a route, to a subnet, with no routing marks, it should be followed.

With no routing-mark defined, it should match !

what do you guys think ?

Chupaka · October 15, 2009, 12:16am

please read http://wiki.mikrotik.com/wiki/Route

especially “Routing table lookup” and near that

actually, if you use ‘routing-test’ package, you can simply

/ip route rule
add disabled=no routing-mark=other action=lookup table=main

now for all packets with ‘other’ mark, first ‘main’ routing table will be checked (make sure you don’t have a default route here), then - ‘other’ routing table (which should contain default route)