RouterBoard: lvl 2 switching threatening lvl 3 security

elgo · Tue Apr 12, 2011 5:43 pm

Hi,

I'm mainly using a RB 450G (v4.17 so far) as a firewall, with 4 defined security zones (each on a level 3 interface): WAN, DMZ, LAN, TEST.

As I've 5 ports available, I've allocated 2 ports to LAN zone, by defining the first LAN port as master-port for the second LAN port. So this result in this, following wiki logic:

                       [CPU]
                         |
 ---------------------------------------------------
   |           |           |                      |
  [  ]        [  ]       [  ]                    [  ]     
   |           |           | -----------          |
   |           |           |           |          |
   |           |         [  ]        [  ]         |
 [WAN]        [DMZ]       [LAN]       [LAN]      [TEST]

As you can see, I need 4 lvl 3 interfaces, and 2 lvl 2 ports. I thought I could achieve that only with using master-port on the second LAN port.

Configuration detailed here:

/interface ethernet print
Flags: X - disabled, R - running, S - slave 
 #    NAME                          MTU   MAC-ADDRESS       ARP        MASTER-PORT                        SWITCH
 0 R  ether1-WAN                    1500  00:0C:42:7F:40:73 enabled   
 1 R  ether2-DMZ                    1500  00:0C:42:7F:40:74 enabled    none                               switch1 
 2 R  ether3-LAN                    1500  00:0C:42:7F:40:75 enabled    none                               switch1 
 3 RS ether4-LAN                    1500  00:0C:42:7F:40:76 enabled    ether3-LAN                         switch1 
 4 R  ether5-TEST                   1500  00:0C:42:7F:40:77 enabled    none                               switch1 

/interface ethernet switch port print 
Flags: I - invalid 
 #   NAME                                        SWITCH                                        VLAN-MODE VLAN-HEADER  
 0   ether2-DMZ                                  switch1                                       secure    leave-as-is   
 1   ether3-LAN                                  switch1                                       secure    leave-as-is   
 2   ether4-LAN                                  switch1                                       secure    leave-as-is   
 3   ether5-TEST                                 switch1                                       secure    leave-as-is   

/ip address print 
Flags: X - disabled, I - invalid, D - dynamic 
 #   ADDRESS            NETWORK         BROADCAST       INTERFACE
 0   192.168.10.254/24  192.168.10.0    192.168.10.255  ether2-DMZ
 1   192.168.0.254/24   192.168.0.0     192.168.0.255   ether3-LAN      
 2 D <WAN IP ADDR>/32    XXXXXXXX       0.0.0.0         pppoe-fibre

Ahem, pppoe-fibre is a pppoe client over a vlan sub-interface over ether1-WAN (Eeeerk, Orange ISP, prehistoric technologies...). I dont post this configuration here because it's not relevant.
And finally:

/interface ethernet switch
set switch1 mirror-source=none mirror-target=none name=switch1 switch-all-ports=no
/interface ethernet switch vlan
add disabled=no ports=cpu,ether2-DMZ,ether3-LAN,ether4-LAN,ether5-TEST switch=switch1 vlan-id=0

Fact is I can sniff on the [TEST] port and get sometimes packets going from DMZ to LAN (with src and dst being unicast IP addresses).

So I try to think about doing something clean like I would do on a real level 3 switch: port VLAN isolation.

I play with some vlan rules to set a vlan-id (without tagging) on ports, like:

interface ethernet switch rule
add copy-to-cpu=no disabled=no mirror=no new-vlan-id=102 ports=ether2-DMZ redirect-to-cpu=yes switch=switch1 vlan-header=not-present
add copy-to-cpu=no disabled=no mirror=no new-vlan-id=103 ports=ether3-LAN,ether4-LAN redirect-to-cpu=no switch=switch1 vlan-header=not-present
add copy-to-cpu=no disabled=no mirror=no new-vlan-id=105 ports=ether5-Phone redirect-to-cpu=yes switch=switch1 vlan-header=not-present

Well, no effect.

Then I thought about tagging this frames within the same rules, but I quickly came to something blocking: I can set a trunk-like link between ethernet port and cpu port (from the switching chip point of view), but I can't handle these tagged frames at level 3 after that. Because level 3 vlan interfaces are bound to an interface (like on *nix) but "cpu" is not a valid supporting interface. So I cant brig tagged frames to cpu but it can't untag them and process them?

So what can I do to do proper isolation on interface that aren't the WAN port (which is ok after switch-all-ports=no)?

I don't want to set an elaborate security policy and see that due to switch chip "messing up", security zones aren't really isolated.

--
edit: more details on tests and some precise output of packets sniffing will be posted here shortly, as I have to do them again).

elgo · Wed Apr 13, 2011 6:54 pm

After reading again MT switching documentation, I decided to conduct some more tests.

I've enabled uPnP on LAN interface (declared as internal interface).
So routeur is sending multicast packets to LAN zone. BUT I CAN SNIFF THEM ON TEST PORT!
Yeah, right, as they are sent with a multicast MAC adress, I guess switch chip flood all my switched port (then all but WAN thanks to switch-all-ports=no) with these packets.
Then I guess anyone can bypass IP filtering by crafting traffic to be a level 2 broadcast...

It's a real security concern, I must find out how to prevent switching from being a security breach. I thought I had a "simple so" clean setup...

Then my real question is now: how to do VLAN port isolation to circumvent this switching zealot chip?

Jeroen1000 · Thu Apr 14, 2011 12:32 pm

If you read around a bit, you will see that ROS cannot support tagged and untagged frames on the same interface (supposedly a hardware limitation). I practically wrote a book on it here on the forums before support said it could not be done. Read my threads and all this may become a lot clearer (it confused the *** out of me though)

from memory:

When you create a routed VLAN interface on an ether port, you can only send it tagged frames (note that you are trying to reach the CPU port). From that point on, untagged ports (access ports if you will) on the routerboard can no longer ping this interface (= the CPU-port). The option to specify 'add vlan-header-if-missing' for the CPU-port is an option that we would need to achieve this. Since we do not have this option, we can build VLANs using the switch chip (and isolate ports) but these ports would be unable to communicate with the CPU.
You see, the rules you built do not actually add a VLAN-HEADER. You only specifiy a 'new-vlan-id' and this id WILL be used to tag frames when a port has 'add vlan-header-if-missing' specified for it. The net result is we cannot have untagged ports on the routerboard talking the the CPU, which sucks.

The only way I could solve this annoying problem was to buy a VLAN-capable switch. Again, read through my posts as I've done a lot of testing with this.

Then I thought about tagging this frames within the same rules, but I quickly came to something blocking: I can set a trunk-like link between ethernet port and cpu port (from the switching chip point of view), but I can't handle these tagged frames at level 3 after that. Because level 3 vlan interfaces are bound to an interface (like on *nix) but "cpu" is not a valid supporting interface. So I cant brig tagged frames to cpu but it can't untag them and process them?

So what can I do to do proper isolation on interface that aren't the WAN port (which is ok after switch-all-ports=no)?

I don't want to set an elaborate security policy and see that due to switch chip "messing up", security zones aren't really isolated.

I think that is was you are trying to say in the text I quote, but I'm not sure what exactly you are explaining there.

Jeroen1000 · Thu Apr 14, 2011 12:39 pm

For the rest of your problem, could you present some more information? Ip addresses and your testing secenario would be of a lot of help. I think you are not getting many replies because it is a bit vague. I'm sure we can figure out what is happening.

kirshteins · Thu Apr 14, 2011 12:41 pm

Traffic from DMZ to LAN is processed through CPU in this configuration. Try adding IP firewall filter rules for port isolation.

elgo · Thu Apr 14, 2011 2:49 pm

In fact, I did read you post, I clearly remember this taged & untaged issue, Jeroen1000

I think that is was you are trying to say in the text I quote, but I'm not sure what exactly you are explaining there.

You're right, after all we are talking about the same thing

As for level 2, yes, I totally agree with you (as you previously discussed as port or VLAN isolation in other threads):

*I knew I didn't tagged my frames by only declaring a new-vlan-id, but I hoped that would modify switching logic (a vlan is supposely being isolated by definition...). Ingress maybe...
the need for 'add vlan-header-if-missing' for the CPU-port. I hoped that as of v5.x, as cpu port is listed now (errr, I think I saw that briefly), we could do this (but I can't use v5.0 so far, and v5.1 is buggy, so waiting to verify this idea).
[edit: on demo, what could work:
Code: Select all
```
[demo@demo2.mt.lv] /interface ethernet switch port> set switch1_cpu vlan-header= 
add-if-missing  always-strip  leave-as-is
```

@kirshteins: traffic entering DMZ port is ok (ingress forced to be redirected to cpu), but still "egress" traffic going from cpu port to an ethernet port suffers from being "overswitched" (see multicast frames). Level 3 firewalling won't help at all.

I will edit my first post with explicit configuration as Jeroen1000 says, and report my tests accordingly. And search for all your post on this subject, Jeroen1000

elgo · Fri Apr 15, 2011 6:27 pm

So, let me explain the "multicast" test, that points out that switching can be a security hole on a routerboard.

I plug a host sniffing on ether5-TEST port. Neither this routerboard port nor the host do have IP address assigned to them on this segment.
I do insert on top of my routerboard firewall rules logging statements, so that I know on what interface the board intended to send multicast.

/ip firewall filter
add action=log chain=output comment="" disabled=no dst-address-type=multicast log-prefix=""
add action=log chain=forward comment="" disabled=no dst-address-type=multicast log-prefix=""

I erased all vlan rules in the switch section of the routerboard, the rest of the configuration is detailled in the first post.

Let's begin the test!

I then enable or disable upnp feature on routerOS, to make the board send multicast. LAN interface is configured as internal interface, WAN as external.

ip upnp set enabled=no

Then I immediately get something on the sniffing port:

 # tcpdump -i eth1  -vv
tcpdump: WARNING: eth1: no IPv4 address assigned
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
17:17:02.906501 IP (tos 0x0, ttl 4, id 0, offset 0, flags [DF], proto UDP (17), length 260) 192.168.0.254.1900 > 239.255.255.250.1900: UDP, len
gth 232
17:17:02.906761 IP (tos 0x0, ttl 4, id 0, offset 0, flags [DF], proto UDP (17), length 229) 192.168.0.254.1900 > 239.255.255.250.1900: UDP, length 201
17:17:02.906887 IP (tos 0x0, ttl 4, id 0, offset 0, flags [DF], proto UDP (17), length 261) 192.168.0.254.1900 > 239.255.255.250.1900: UDP, length 233

Notice that the source is the LAN routerboard interface, which is OK according to board upnp setup.

In routerboard logs, I confirm this:

17:17:02 system,info upnp config changed by elgo 
17:17:03 firewall,info output: in:(none) out:ether3-LAN, proto UDP, 192.168.0.254:1900->239.255.255.250:1900, len 240 
17:17:03 firewall,info output: in:(none) out:ether3-LAN, proto UDP, 192.168.0.254:1900->239.255.255.250:1900, len 240 
17:17:03 firewall,info output: in:(none) out:ether3-LAN, proto UDP, 192.168.0.254:1900->239.255.255.250:1900, len 209

But I should never had got this on the ether5-TEST port that is not part of the virtual switch built by master-port feature:

/interface ethernet print
Flags: X - disabled, R - running, S - slave 
 #    NAME                           MTU   MAC-ADDRESS       ARP        MASTER-PORT                           SWITCH                          
 0 R  ether1-WAN                     1500  00:0C:42:7F:40:73 enabled   
 1 R  ether2-DMZ                     1500  00:0C:42:7F:40:74 enabled    none                                  switch1 
 2 R  ether3-LAN                     1500  00:0C:42:7F:40:75 enabled    none                                  switch1  
 3 RS ether4-LAN                     1500  00:0C:42:7F:40:76 enabled    ether3-LAN                            switch1 
 4 R  ether5-TEST                    1500  00:0C:42:7F:40:77 enabled    none                                  switch1

Conclusion?
Seems to me that broadcasted frames get switched, without considering master-port setup.

Jeroen1000 · Fri Apr 15, 2011 8:41 pm

I'll read this thoroughly. Looks like this may be an issue. Hmm ^^

Seeing I'm not a network specialiast or anything I'll voice some wild guess first thought. It is a bug and UPNP gets enabled on every interface? Can you also get normal broadcast traffic to show up on the TEST interface?

elgo · Sat Apr 16, 2011 3:40 pm

I'll read this thoroughly. Looks like this may be an issue. Hmm ^^

Seeing I'm not a network specialiast or anything I'll voice some wild guess first thought. It is a bug and UPNP gets enabled on every interface? Can you also get normal broadcast traffic to show up on the TEST interface?

Rough answers to wild guess, let's go

uPnP on every interface: I first thought about this too. But:
1- ether5-TEST has no IP address.
2-I enabled firwall logs on every multicast packet. If uPnP daemon was bugged and tried to send out an interface, I should have a log line about this. I only see "output on ether3-LAN" log entries.

This makes me think that this multicast seen on ether5-TEST interface is not processed @ level 3 by the board.
Then this brings me to say that level 2 chip is involved. That would make sense, as multicast frames sent are broadcast ethernet frames. This frames come from cpu port to the virtual switch (remember master-port feature, 2 ports are "LAN"), the wrong part being that this frames shouldn't be flooded on a port outside the virtual switch (ether5-TEST in this case).

I'll try do do some more tests, on different scenarios. Have to find out other good and explicit tests first

Jeroen1000 · Sat Apr 16, 2011 8:03 pm

I'll try to recreate what you are doing when I have some more time. I can't break my current setup now as I have to many other stuff waiting to be done.

So, ATM I do not have a master port and stuff.

My setup is very very simple actually. I just created 2 VLANs on ether1 (ether1 is a trunk port, the interface is connected to a VLAN switch):

- VLAN10 (internal LAN 192.168.0.x/24)
- VLAN20 (DMZ zone => devices in this VLAN receive a public routable IP address)

Then on ether5 I have a connection to a cable modem (you could call ether 5 the WAN-port). Ether5 is bridged together with VLAN20.

What I did was sniff on ether2 and ether3 (using a laptop and wireshark). I received no broadcast frames nor did I receive anything when I enabled UPNP.
I enabled UNPN on:

Ether1, VLAN10, VLAN20, ether2, ether3.

Remark that ether2 and 3 were not assigned an IP address.

So without master port nothing is coming through. Or perhaps the fact that ether1 is in 2 VLANs prevents anything from coming through. Ether5 is not in a VLAN though.

elgo · Sun Apr 17, 2011 12:47 am

I'll assume vlan 10 is your internal and bridge your external uPnP interface (strange, a bridge with an IP address? well ok

)
Then upnp multicast is send through vlan 10 interface.
2 points:

is switch-all-ports=no? You see where I'm going in this case, no switching on ether1 at all
Let's say switch-all-ports=yes. Frames going through your ether1 port are tagged. That may bring switching logic to be "correct" (an not flood others ether2 to ether5 ports), depending on vlan implementation vs master-ports implementation.

This may be worth some expert from MT to join the discussion to this point

Jeroen1000 · Sun Apr 17, 2011 3:11 am

If you can make a setup without the UPNP variable which leaks broadcasts, I'll setup my router the exact same way for verification.
I don't think my current setup was a good indication for the switch leaking broadcasts. I just wanted to see whether it worked correctly. In the meantime I propose we try to find out more if that is OK with you?

Let's go on with the issue at hand:

Just to be clear as we possibly can: esstentially, setting the master port for certain ports, creates a 'mini-switch' containing only these ports.
The CPU port is, or should be, the only way traffic can get out to ports that are NOT part of this mini-switch.

Correct so far?

So, you set ether3 as master port for ether4. You would assume all unicast frames from a host connected to ether3 destined for a host connected to ether4 WILL NOT leak to any other ports? This traffic must not show up on ether1, ether2 or ether5.
But what would happen with broadcast traffic? Will it reach other ports that are not part of the switch? It it does, it gets there via the CPU port. But the question is, is this behaviour correct?

What do you think about this?

Side note: a switch behaves like a hub for the first packet (it needs to build its switching database).

I'm trying to wrap my head around this but I'm commited to help you test this ^^

elgo · Sun Apr 17, 2011 4:55 pm

If you can make a setup without the UPNP variable which leaks broadcasts, I'll setup my router the exact same way for verification.

I'll think bout this.

Just to be clear as we possibly can: esstentially, setting the master port for certain ports, creates a 'mini-switch' containing only these ports.
The CPU port is, or should be, the only way traffic can get out to ports that are NOT part of this mini-switch.

I dont' think this is exact. I mean, cpu port is (according to the wiki page, or should be, depending of the exact implementation) like any other port of the switch. Not a "default port" (like a default route). See, we can set allowed vlans on cpu port in routerOS switch menu. But well, can't say exactly who is right without seeing the exact specs or code of this master-port feature.

So, you set ether3 as master port for ether4. You would assume all unicast frames from a host connected to ether3 destined for a host connected to ether4 WILL NOT leak to any other ports? This traffic must not show up on ether1, ether2 or ether5.

Can't say so far, as I didn't let 2 hosts up in LAN zone while sniffing. I'll look at this.

But what would happen with broadcast traffic? Will it reach other ports that are not part of the switch? It it does, it gets there via the CPU port. But the question is, is this behaviour correct?

Nah, I don't think so. In the precise case of ethernet broadcast like in my multicast example, no need for frames to go to cpu port to have a switching problem. As soon as they enter a port, they are about to be (uncorrectly or correctly) switched. In fact, if frames went to cpu port, the level 3 interface wouldn't forward ethernet broadcast frames anyway (since it's a router by definition). Moreover, I would get a "forward - interface name" log entry during my test.

Side note: a switch behaves like a hub for the first packet (it needs to build its switching database).

Exactly, have to keep this in my mind

Additionnal info:
Without modifying my setup, ether5-TEST is catching some unicast traffic too (from LAN to DMZ in this example, a windows share mounted on a *nix box):

14:59:55.693805 IP (tos 0x0, ttl 4, id 0, offset 0, flags [DF], proto UDP (17), length 316) 192.168.0.254.1900 > 239.255.255.250.1900: UDP, length 288
15:06:08.752878 IP (tos 0x0, ttl 64, id 27206, offset 0, flags [DF], proto TCP (6), length 124) 192.168.0.2.43352 > 192.168.10.1.microsoft-ds: P 410178:410250(72) ack 767873 win 501 <nop,nop,timestamp 19479640 401382229>
15:11:08.764442 IP (tos 0x0, ttl 64, id 27227, offset 0, flags [DF], proto TCP (6), length 124) 192.168.0.2.43352 > 192.168.10.1.microsoft-ds: P 410928:411000(72) ack 769277 win 501 <nop,nop,timestamp 19509640 401457224>

I'll think about this.

Thank you very much for your interest about this, and for the time you take speaking of this with me

Mon Apr 18, 2011 4:35 pm

elgo,

The isolation is working correctly, the configuration with all the ports in the switch group with vlan ID 0 is making the problem (vlan ID 0 means no vlan and the switch becomes flat for the L2 frames).

It is enough to do just "/interface ethernet set ether4 master-port=ether3". In such setup ether3 and ether4 would be completely isolated from other ports. By isolated here I mean that for packet to get forwarded to another interface it would have to go as described in RouterOS packet flow: http://wiki.mikrotik.com/wiki/Manual:Packet_Flow.

Following vlan configuration is not needed, it makes packets with vland id 0 (special meaning - means no tag) switched among all ports again:

/interface ethernet switch vlan
add disabled=no ports=cpu,ether2-DMZ,ether3-LAN,ether4-LAN,ether5-TEST switch=switch1 vlan-id=0

Also if you have no entries in "/interface ethernet switch vlan" table there is no point in doing "/interface ethernet switch port [find] vlan-mode=secure". It does not matter what you have there, so you can leave it to default "fallback" setting.

From switch chip wiki (http://wiki.mikrotik.com/wiki/Switch_Chip_Features):

Vlan tables specifies certain forwarding rules for packets that have specific 802.1q tag. Those rules are of higher priority than switch groups configured using 'master-port' property.
Packets without vlan tag are treated just like if they had a vlan tag with vlan id = 0. This means that if "vlan-mode=check or secure" to be able to forward packets without vlan tags you have to add a special entry to vlan table with vlan id set to 0.

elgo · Tue Apr 19, 2011 4:57 pm

mmmm, I see. Thank you for this explanation sergejs.

I then have 2 questions:

Should "vlan-mode=secure" be considered as "best practice" or not? Ok, right now it may raise some problems on a routerboard, but I somehow fail to see how a security device could tolerate "fallback" setting (vlan hoping and other issues).
"Study case" I set vlan-mode back to fallback. Take my original setup: 1 DMZ port, 2 ports in LAN (master-port), 1 TEST port. I have wire speed switching between 2 LAN ports, great. But is DMZ port isolated from TEST port and vice versa (from level 2 point of view, FW takes care of level 3)?

elgo · Wed Apr 20, 2011 11:54 am

Owww, I may have been so frightened by my previous "biased" tests I maybe didn't realized something: hardware switching isn't occuring at all unless master-port feature is used, and then occurs only within this defined virtual switch?
So basically, no need to do port-vlan for isolating ether ports without IP address? I'm a bit confused by this sentence of the wiki page "Here you can see that, a packet that gets received by one of the ports always passes through the switch logic at first. "

Jeroen1000 · Wed Apr 20, 2011 4:36 pm

Owww, I may have been so frightened by my previous "biased" tests I didn't realized something: hardware switching isn't occuring at all unless master-port feature is used, and then occurs only within this defined virtual switch?

That is what I make of it, without a master-port every ethernet port is isloated at L2. This virutal switch is what I meant with 'mini-switch'. So if you want, for instance, a routboard to work as a regular consumer router, you put all but 1 port in a switch, the remaining port would then be your WAN-port.
Without the hardware switching, ports that should be able to talk to each other are grouped together in a bridge. This will limit througput as bridging is done in software and hence speed is then largely dependent on the CPU.

So basically, no need to do port-vlan for isolating ether ports without IP address? I'm a bit confused by this sentence of the wiki page "Here you can see that, a packet that gets received by one of the ports always passes through the switch logic at first. "

VLANs are a L2 technology and do not really care about IP-addresses. So if you create a virutal switch (as you put it) then you could use VLANs to restrict port talking to each other. Even that is not strictly required with routerboard as you can define rules in the rule table to prevent ports from talking to eachother. The configuration of VLANs on a routerboard is a bit confusing as they do not line up with how most switches implement VLANs.

However, you have still raised a valid point. Are there frames passed up to the CPU and routed/bridged to ports where they should not belong?

RouterBoard: lvl 2 switching threatening lvl 3 security

RouterBoard: lvl 2 switching threatening lvl 3 security

Re: RouterBoard: lvl 3 switch but how to control lvl 2 switc

Re: RouterBoard: lvl 2 switching threatening lvl 3 security

Re: RouterBoard: lvl 2 switching threatening lvl 3 security

Re: RouterBoard: lvl 2 switching threatening lvl 3 security

Re: RouterBoard: lvl 2 switching threatening lvl 3 security

Re: RouterBoard: lvl 2 switching threatening lvl 3 security

Re: RouterBoard: lvl 2 switching threatening lvl 3 security

Re: RouterBoard: lvl 2 switching threatening lvl 3 security

Re: RouterBoard: lvl 2 switching threatening lvl 3 security

Re: RouterBoard: lvl 2 switching threatening lvl 3 security

Re: RouterBoard: lvl 2 switching threatening lvl 3 security

Re: RouterBoard: lvl 2 switching threatening lvl 3 security

Re: RouterBoard: lvl 2 switching threatening lvl 3 security

Re: RouterBoard: lvl 2 switching threatening lvl 3 security

Re: RouterBoard: lvl 2 switching threatening lvl 3 security

Who is online