Community discussions

MikroTik App
 
jkroon
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 58
Joined: Thu Apr 03, 2008 2:18 am
Contact:

RouterOS 7: routing "in" matcher against ip firewall ipsets

Wed May 24, 2023 8:32 am

Hi All,

Just wondering if this is a bug or expected behaviour. Currently Running RouterOS 7.8 - upgrade is scheduled.

Let's say for example we have (stripped down as far as possible):
/ip firewall address-list
add address=192.168.0.0/16 comment=RFC1918 list=bogon4
There are not more specifics, just loading all the more specifics for that single /16 is 128k prefixes going all the way to /32, even just going to /24 is 511 entries.

Our discard-bogons filter looks like this:
/routing filter rule
add chain=discard-bogons disabled=no rule=\
    "if (afi ipv4 && dst-len > 24) { set bgp-communities 857:60424; reject }"
add chain=discard-bogons disabled=no rule=\
    "if (afi ipv6 && dst-len > 48) { set bgp-communities 857:60648; reject }"
add chain=discard-bogons disabled=no rule=\
    "if (afi ipv4 && dst in bogon4) { set bgp-communities 857:60401; reject }"
add chain=discard-bogons disabled=no rule=\
    "if (afi ipv6 && dst in bogon6) { set bgp-communities 857:60601; reject }"
add chain=discard-bogons disabled=no rule=return
The set bgp-communities are so that we can see *why* routes are getting rejected. As a test I further added this filter:
add chain=in-test disabled=no rule="jump discard-bogons; set bgp-communities 857:0; reject"
Now, receiving a route for exactly 192.168.0.0/16 rejects correctly, and sets the bgp-community to 857:60401 indicating this was caught by the bogon filter above.

However, 192.168.0.0/17 gets rejected with in-test with community 857:0, indicating it was rejected by in-test, not the bogon filter.

Given that 192.168.0.0/17 really is in (ie, a subset of) 192.168.0.0/16 I expected the prefix to be rejected by the bogon filter.

If I add an explicit entry to the filter such as (after the 857:60401 reject rule, before the 857:60601 rule):
    "if (afi ipv4 && dst in 192.168.0.0/16) { set bgp-communities 857:60402; reject }"
add chain=discard-bogons disabled=no rule=\
Then the /17 gets correctly rejected and community set to 857:60402.

The reason for using the firewall ipsets is to enable sharing of things like bogon data between route filters and the firewall itself.

My question is whether this is a misunderstanding on my part, or whether this is a bug in ROS7?

PS: I'd love to be able to import data from BGP routes into those ipsets but this seems not possible at this stage to be able to do things like "add prefixes received from our route reflects marked with a specific community into an ipset to enable rejecting them when received from eBGP peers" for example.
Last edited by jkroon on Wed May 24, 2023 9:46 am, edited 1 time in total.
 
wiseroute
Member
Member
Posts: 352
Joined: Sun Feb 05, 2023 11:06 am

Re: RouterOS 7: routing "in" matcher against ip firewall ipsets

Wed May 24, 2023 8:56 am

hello
Given that 192.168.0.0/15 really is in (ie, a subset of) 192.168.0.0/16 I expected the prefix to be rejected by the bogon filter.
do you mean:

you want to know which rule will be executed first by the filter? in ordered manner?

or

you want to know whether the filter will match the longest or the supernet prefix first ?
 
jkroon
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 58
Joined: Thu Apr 03, 2008 2:18 am
Contact:

Re: RouterOS 7: routing "in" matcher against ip firewall ipsets

Wed May 24, 2023 9:11 am

Given that 192.168.0.0/15 really is in (ie, a subset of) 192.168.0.0/16 I expected the prefix to be rejected by the bogon filter.
do you mean:

you want to know which rule will be executed first by the filter? in ordered manner?

or

you want to know whether the filter will match the longest or the supernet prefix first ?
I want to know why the "dst in bogon4" does not cause 192.168.0.0/15 (dst) to match against 192.168.0.0/16 (bogon4). Seeing that dst is more specific than bogon4 I consider the "in" condition to be logically true. Yet ROS7 here will only match exact matches, ie, 192.168.0.0/16 will match the bogon4 set, but not 192.168.0.0/15 - which I find odd.
 
wiseroute
Member
Member
Posts: 352
Joined: Sun Feb 05, 2023 11:06 am

Re: RouterOS 7: routing "in" matcher against ip firewall ipsets

Wed May 24, 2023 9:26 am

firstable,
Given that 192.168.0.0/15 really is in (ie, a subset of) 192.168.0.0/16 I expected the prefix to be rejected by the bogon filter.
/15 is not a subset of /16.

/15 is the upper net (cidr) of /16, /22, /24 so forth.

/16 is the subnet (vlsm) of /15, /12, /8 so forth.

second,
let us go back to history of firewall or filters:

the old days firewall config utilize upper net filter to dismiss its next subnet filter execution. in ordered manner. very efficient.

current modern day config, do the opposite thing, which is any filter will match the longest prefix found in the list. cpu and brain intensive.

so, I hope you get the idea 💡

hope this helps.
 
jkroon
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 58
Joined: Thu Apr 03, 2008 2:18 am
Contact:

Re: RouterOS 7: routing "in" matcher against ip firewall ipsets

Wed May 24, 2023 9:45 am

/15 is not a subset of /16.
Ah shit, you're right. Same applies to the /17 though, it doesn't match when I think it should. Not sure I can fix up the initial posts but will try a bit later. In my actual case (details I'm not at liberty to disclose we've got a /21 outermost, and then some idiot is handing us our own /23 inside that which we're also advertising from another location, so the /23 along with a the /22s passes through the bogon filter, only the actual /21 gets filtered.

I screwed up the modifications in the copy&paste.
let us go back to history of firewall or filters:

the old days firewall config utilize upper net filter to dismiss its next subnet filter execution. in ordered manner. very efficient.

current modern day config, do the opposite thing, which is any filter will match the longest prefix found in the list. cpu and brain intensive.

so, I hope you get the idea 💡

hope this helps.
No it doesn't. Other than using the ipset from ip/firewall this has nothing to do with firewalls (incidentally, I've been around since the days of ipchains, so I know a fair bit of that, as irrelevant as it may be, the core mechanism hasn't changed significantly since then, nftables seems to be a bit different but from what I can see really just merges iptables and ip6tables back into a single firewall allowing you to do things like match on tcp only and be ignorant of what the IP layer was, and a whole new configuration syntax).

This is all about BGP and filtering the routes advertised by (eBGP) peers.
 
wiseroute
Member
Member
Posts: 352
Joined: Sun Feb 05, 2023 11:06 am

Re: RouterOS 7: routing "in" matcher against ip firewall ipsets

Wed May 24, 2023 9:57 am

hello
This is all about BGP and filtering the routes advertised by (eBGP) peers
well, I am sure you know it better than me 😉

there are many how to guide bgp filters on the net saying it will match the longest prefix.

ok. good day and good luck 👍🏻
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7042
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: RouterOS 7: routing "in" matcher against ip firewall ipsets

Wed May 24, 2023 10:08 am

Documentation has all the info:

https://help.mikrotik.com/docs/display/ ... xOperators

dst in address_list means "match if prefix is in address list", not "match if prefix is in each address list network"
 
jkroon
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 58
Joined: Thu Apr 03, 2008 2:18 am
Contact:

Re: RouterOS 7: routing "in" matcher against ip firewall ipsets

Wed May 24, 2023 10:50 am

Documentation has all the info:

https://help.mikrotik.com/docs/display/ ... xOperators

dst in address_list means "match if prefix is in address list", not "match if prefix is in each address list network"
Not sure what you mean with "match if prefix is in each address list network" - did you mean to say "match if prefix is in any of the prefixes listed in the address list"?

I note the documentation on the page you referenced states:
Return true if the prefix is the subnet of the provided network. If an operator is used to match prefixes from the address list (e.g "dst in list_name"), then it will match only the exact prefix.
Which matches the behaviour I'm observing, but doesn't match the logical expectation since the more specific received route is in the address list (ie, subset thereof, which is the implication of the in operator). I don't recall it this way but given the changelog and last update I obviously read right over that last part (which was added Dec 14, 2021).

Is Mikrotik looking to update/change this at some point? It should be fairly easy to do, I'm somewhat certain that the routing daemon isn't using the set in the kernel to match against, so a userspace copy is presumably available, so doing doing a shortest-prefix match against dst (ignoring the prefix length), and then then just verifying that the dst prefix length is longer-equal than the prefix-length of the matching entry in ipset should not have significant performance impact. For example, 192.168.0.0/16 and 192.168.0.0/24 in the address list (logically having the /24 doesn't really add anything, but it's also not illegal). We're looking to see if 192.168.0.0/15 matches, so we find the shortest prefix match for 192.168.0.0, this finds 192.168.0.0/16 but since 15<16 we don't match. For 192.168.0.0/17 we find the same and since 17>=16 we match. The reason for shortest-prefix is if we select 192.168.0.0/24, then for /17 it will be 17<24 and we won't match, so it's important to match shortest-prefix here.

This would be a change of behaviour though and one always need to be wary of those. It would make sense to implement it this way though, especially in combination with (from https://help.mikrotik.com/docs/display/ ... l+Overview) "Routing filter add a prefix to address list" which is not yet available. That way one could from your reflectors export the list of bogons using a special community, install unreachable/blackhole (depending on the type of bogon) routes on edge routers as well as inject these prefixes into an address list in order to dynamically filter these things on the fly.

As it stands, I'm not sure there is real benefit to using in address_list, current behaviour would be better suited to "dst == address_list" in my opinion.
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7042
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: RouterOS 7: routing "in" matcher against ip firewall ipsets

Wed May 24, 2023 11:32 am

Not sure what you mean with "match if prefix is in each address list network" - did you mean to say "match if prefix is in any of the prefixes listed in the address list"?
Yes that is the same as I wrote. And is the logical expectation.

Let me explain. Lets say we have address list with entry 1.1.0.0/24,
and we have two destinations:
dst1: 1.1.0.0/16
dst2: 1.1.1.0/24

Logical expectation:
is dst1 in the address list? Yes
is dst2 in the address list? no.
current behaviour would be better suited to "dst == address_list" in my opinion.
that would indicate that dst is a string equal to address list name, or dst is the actual address list that is equal to provided address_list which does not make sense in particular case, because dst cannot be neither string equal to address list nor address list itself.

If you want to use more advanced matching you can create a chain and do the matching
/routing filter rule
add chain=my-list-chain rule="if (dst in 1.1.0.0/16) {accept}"
add chain=my-list-chain rule="if (dst in 2.2.0.0/16) {accept}"

add chain=main-chain rule="if (chain my-list-chain) {accept}"
 
jkroon
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 58
Joined: Thu Apr 03, 2008 2:18 am
Contact:

Re: RouterOS 7: routing "in" matcher against ip firewall ipsets

Wed May 24, 2023 12:49 pm

Another example, let's say we have in the address list:

192.168.0.0/24
192.168.2.0/24

(my brain's logical) expectations:

dst=192.168.X.0/16-23: no
dst=192.168.1,3-255.X/24-32: no
dst=192.168.0.X/24-32: yes
dst=192.168.2.X/24-32: yes

Where X is a subset of 0 to 255 such that the resulting HOST portion of the specific prefix-length under consideration is 0.

As such, give the above two entries in the address list, logically I would expect 192.168.0.128/25 to match since it's a subnet of 192.168.0.0/24.

Yes, I can duplicate the address list to a route-filter, and this will have to happen currently, but it's an absolute waste in terms of duplication of configuration, especially when bogon list contains say 20 or so prefixes, and a much, much larger set if we can export CYMRU's bogon (unallocated space) into an address list, which contains a few /17s etc still, and a number of /8s on IPv6. Manually syncing that just seems like a total waste of effort.
 
jkroon
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 58
Joined: Thu Apr 03, 2008 2:18 am
Contact:

Re: RouterOS 7: routing "in" matcher against ip firewall ipsets

Wed May 24, 2023 2:13 pm

Just to clarify, whenever new ranges gets opened (or we add a transit client), that amends the bogon list, requiring us to head out to ALL edge routers, and adding it in two places, rather than simply adding the new ranges to our reflects, having those pushed into the ipsets, and using that to filter other peers.

For now we will continue to manually add these ranges to multiple routers whenever changes happen, it is, however fast becoming a burden, and we were hoping that this one function at least would form one small step closer to the ideal solution of signalling this to edge routers from the reflectors.

So requesting that Mikrotik please update the behaviour to be truly "in" and not "==", as well as implementing the function to export prefixes into address lists from the route filters.
 
wiseroute
Member
Member
Posts: 352
Joined: Sun Feb 05, 2023 11:06 am

Re: RouterOS 7: routing "in" matcher against ip firewall ipsets

Wed May 24, 2023 2:19 pm

jkroon,
we've got a /21 outermost, and then some idiot is handing us our own /23 inside that which we're also advertising from another location, so the /23 along with a the /22s passes through the bogon filter, only the actual /21 gets filtered.
is that part your current problem?

that wasn't other as'es fault. that's how ebgp works. all providers do the same thing: which is do some filter, prepend etc. bgp is about mind games between friends.

now... as i said the simplest thing - why would you comfortable on doing hard math logic just for filtering while your network on loops??

just find which prefix go back to you and from which peer, then - analyze those prefix, are they in /24 form or /23 like you say.

my normal guess would be you advertise /24.

if those prefix in /24 (even though they would form a /23 eventually) - then just do basic /24 filter. done. network secured.

then when you get home - you can continue figuring your math logic.

good luck 👍🏻
 
jkroon
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 58
Joined: Thu Apr 03, 2008 2:18 am
Contact:

Re: RouterOS 7: routing "in" matcher against ip firewall ipsets

Wed May 24, 2023 11:36 pm

is that part your current problem?
This is but one symptom of a larger problem, a problem that needs to be addressed, and the issue reported here would have been an ideal solution, especially when combined with the ability to push prefixes into address lists from route filters.
that wasn't other as'es fault. that's how ebgp works. all providers do the same thing: which is do some filter, prepend etc. bgp is about mind games between friends.
Agreed, but it's not mind games either, if you think it is perhaps you need to re-evaluate some of your assumptions. Please understand what the problem is before posting so called solutions. I've been at this for nearly two decades, and ROS7 is finally touching on some aspects for which I've been asking and hoping for for more than a decade. Combined with FRR it seems we're finally starting to get some of the building blocks that's required for defence of the internet infrastructure as a whole.

I can do the required using as per mrz using a sequence of filters like:
if (dst in 192.168.0.0/16) { reject }
I have a few problems with this:

1. It's hard to dynamically add/remove IPT customer ranges to the bogon filter.
2. I can't sensibly import CYMRU bogon lists into route-filters to prevent receiving these routes as advertisements (installing blackhole routes is easy enough, but not good enough).
3. The list of bogon's isn't shared between the firewall and the route filters (a single source of truth should obviously always be preferred).
4. Maintaining such in two places on even a handful of edge routers is hard - imagine scaling that to several tens. Now you're beginning to feel the pain.

At the moment we've got a mechanism to update the address list semi dynamically by building it from from frr using a cron on a central server serving acting as a route-reflector on top of Linux (which means availability of bash), then performing some checksum magic and based on the checksum a rsc script can determine if the address list needs to be updated or not, we can then download the updated set as a rsc script (generated by shell script) and import that. So for a period of at most a few minutes we may have sub-optimal or non-existent routes for IPT customers where their L2 to us fails. This doesn't happen frequently, but it does happen, and in the one case where it happens most frequently it's usually one of their internal links that fails resulting in us falling back to a much less than optimal path for a few minutes compared to then just picking up their routes from a shared INX. Are you starting to get that my question is about much, much more than the over-simplified symptom raised here?
now... as i said the simplest thing - why would you comfortable on doing hard math logic just for filtering while your network on loops??
It's but one possible symptom of much larger eco-system that is significantly more complex than most people realise. Simplify the issue to get sensible answers. Solving hard math is, well, hard. So tackle sub problems and solve those. Then coalesce the smaller solutions into a larger more comprehensive but also more complex solution. Then go back and question everything and try and simplify again. As simple as possible but no simpler than required.
just find which prefix go back to you and from which peer, then - analyze those prefix, are they in /24 form or /23 like you say.
Right, so as it stands in the bogon filter there are just under 20 prefixes which is static, and another 10 or so that's dynamic (explanation above). The proposal you make here is exactly what got me to making this post. We'd have liked to reference address lists in the route filters rather than duplicating the address lists to the route filters.
my normal guess would be you advertise /24.
Incorrect assumption, typically anything between (including) /20 and /23 with only a handful of /24s. We already filter anything more specific than a /24 (v4) or a /48 (v6), but we can't outright just filter all /24s. We need this to be dynamic, so the bogon filter will based on the information from this topic need to be something like:
if (afi ipv4 && dst-len > 24) { reject; }
if (afi ipv6 && dst-len > 48) { reject; }
if (dst in 10.0.0.0/8) { reject; }
if (dst in 192.168.0.0/16) { reject; }
if (dst in 0.0.0.0/8) { reject; } <-- specifically added as a nasty example that cannot be handled by way of address list.
if (dst in ...) { reject; }
... incl some of which we'd like to be dynamic.
return;
if those prefix in /24 (even though they would form a /23 eventually) - then just do basic /24 filter. done. network secured.
Very, very wrong. Oh I don't even know where to begin with how wrong this statement is. Your total number of prefixes that can be generated for your network can be calculated, so let's assume a total of public space of exactly a /19. This means:

1 x /19
2 x /20
4 x /21
8 x /22
16 x /23
32 x /24

Or 2^(24-19+1)-1 = 2^6-1 = 63.

Drop a few because most likely the space is fragmented, so let's say a /20 and two /21s, that's 2(2^(24-21+1)-1) + 2^(24-20+1)-1 = 61 ... that's not even such a significant space, and already a fair number of possible prefixes. Can be filtered using 4 static route filter rules, or two if using address_lists, but we don't want 61 prefixes in the address list, only 3. With address_lists being much more dynamic in that it's easier to modify them programmatically compared to route filters. I'm guessing 2^(24-tsize+1)-number_of_base_prefixes but I've not spent time now to mathematically prove this one.

Not really the issue though since every base prefix can be dealt with in a single rule, but the moment we use address lists, we have to insert all of those routes. For filtering 0.0.0.0/8 (assuming more specific than /24 has already been axed) that's a whopping 2^(24-8+1)-1 = 131071 prefixes to be inserted into the address list. So the if (dst in 10.0.0.0/8) { reject } mechanism is much simpler, except it's definitely not dynamic (neither are address lists but their easier to manipulate programmatically).
then when you get home - you can continue figuring your math logic.
The math has already been done, over and over again. Been working on it for many years. More than I care to count. You're missing this point. The idea behind this topic was to simplify the complexity of the algorithmic nature (rules) of calculating (deploying) the math.
good luck 👍🏻
I don't believe in or function on luck. I believe in and function on cold hard calculation and precision. And improving precision, and preventing manual mistakes makes perfect sense for me. Our almost ideal solution:

1. Originate bogon routes from route reflector with a specific community, eg 857:65500 (RFC1918, LSN, 0.0.0.0/8, multicast etc ..., but not RIR assigned prefixes)
2. Originate ASN prefixes that's been RIR assigned from route reflectors with 857:65500 (so that they're bogon's too) as well a community indicating how/where they should be advertised.
3. On edge routers install routes received from RRs with community 857:65500 as unreachable routes.
4. Copy all prefixes with community 857:65500 into address_list.
5. Use address_list to filter prefixes received on eBGP.
6. Use same address_list in firewall to filter ingress traffic - this one unfortunately forces traffic through CPU, so it's probably best to rather use switch chip ACLs to filter the static rules, and just periodically sync these ACL rules with the static bogon lists (ie, traffic from any prefix with 857:65500 on a RED interface should get dropped). No branching or anything, so if the RED interface is a 4x10G LACP bridge port this gets nasty, so this point we still need to consider, however, it's perfectly possible to redirect things like DNS, NTP and SMTP from the switch chip to CPU and filter (for example) just those - since these are the protocols we found is most abused. As it stands in order to maintain routing performance this one will at least partially remain a pipe dream.

There are other use-cases for these things which I won't go into now, but needless to say, the possibilities are insane, if you can send address lists from RRs to edge routers, there is just more options available for doing dynamic filtering. One such use-case is to peer with CYMRU and get the total unallocated space from RIRs (both v4 and v6) from them, now push that to edge routers as per above with 857:65500 - since these are filtered, and unreachable routes gets installed, you're effectively closing down on a very common phenomenon whereby unallocated space gets jacked by less-than-savoury characters and protecting your clients from those. We inherited space that was abused in this way - so this is a sensitive topic. RPKI is supposed to help here, but that only works if RIRs are allowed to publish a "reject" signature for these ranges. Meantime in the real world we need alternative solutions to protect our networks and our customers - something for which this whole discussion is but a small part.

But I digress ... please let's just stick to the topic at hand, for now:

Can we have the in operator match as in, and not as ==? For now we're getting the prefixes into the address list by means other than BGP, but we would like to use address_lists to help with filtering BGP advertisements, for now, since fixing up the filters by way of routeros scripting is actually quite hard. Getting to put stuff into those address lists using BGP would be a future endeavour once that functionality is available from ROS as per "Routing filter add a prefix to address list" from https://help.mikrotik.com/docs/display/ ... l+Overview

Who is online

Users browsing this forum: No registered users and 15 guests