Fortunately I had migrated from IP accounting to netflow some time ago. I do not need an expensive netflow analyzer package, I just want to keep
some logs, but I wanted to include port numbers. So I wrote a simple netflow receiver in perl using the Net::Flow package from CPAN, that just receives
the netflow stream and writes it into a tab-separated file similar to IP accounting. The deamon starts writing a new file (with date in the filename)
every day and the previous ones are compressed and kept for some limited time. Normally they are never read, they are just kept in case some
abuse report arrives.
I expect this to work in v7 (not tested yet).
Enabling L3 offloading on my CRS309 completely ignores switch ACL rules then all traffics are permitted, even those only for L2 switching.
Disable L3 offloading and reboot makes ACL rules working again.
Are ACL rules broken with L3 offloading enabled in v7.1beta6?
ACL rules should work with L3 HW offloading with exception of Fasttrack offloading. Offloaded Fasttrack connections have higher priority and bypass ACL rules. Other than that, hardware routing should obey ACL rules. Can you post your interface config please, so we try reproducing your issue on our side?
/interface export hide-sensitive
Hi,
Sure.
In my following config, I added the 3rd switch rule to completely disable the sfpplus1 port. However, it has no effects when l3 offloading is on. If I turn off l3 offloading, that port will be completely blocked and cannot send out any packets, but if turn l3 offloading on again, the port is working again.
I don’t have any stateful firewall configuration in /ip/firewall.
/interface bridge
add admin-mac=C4:AD:34:D5:C2:53 auto-mac=no comment=defconf igmp-snooping=yes name=bridge pvid=90 vlan-filtering=yes
/interface vlan
add interface=bridge name=uplink vlan-id=80
add interface=bridge name=vlan10 vlan-id=10
add interface=bridge name=vlan20 vlan-id=20
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge comment=defconf frame-types=admit-only-untagged-and-priority-tagged interface=ether1 pvid=90
add bridge=bridge comment=defconf interface=sfp-sfpplus1 pvid=20
add bridge=bridge comment=defconf interface=sfp-sfpplus2 pvid=20
add bridge=bridge comment=defconf interface=sfp-sfpplus3 pvid=20
add bridge=bridge comment=defconf interface=sfp-sfpplus4 pvid=20
add bridge=bridge comment=defconf interface=sfp-sfpplus5
add bridge=bridge comment=defconf interface=sfp-sfpplus6
add bridge=bridge comment=defconf interface=sfp-sfpplus7
add bridge=bridge comment=defconf interface=sfp-sfpplus8
/interface bridge vlan
add bridge=bridge comment=MGMT tagged=sfp-sfpplus7,sfp-sfpplus8,sfp-sfpplus6,sfp-sfpplus5 vlan-ids=90
add bridge=bridge comment="Uplink to Transit" tagged=sfp-sfpplus5,bridge vlan-ids=80
add bridge=bridge comment="User VLAN" tagged=sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,bridge,sfp-sfpplus5 vlan-ids=10
add bridge=bridge comment="Service VLAN" tagged=bridge,sfp-sfpplus5 vlan-ids=20
/interface ethernet switch rule
add comment="Accept MGMT" dst-address=100.100.16.0/24 ports=sfp-sfpplus8,sfp-sfpplus7,sfp-sfpplus6,sfp-sfpplus5 src-address=100.100.16.0/24 \
switch=switch1
add comment="disallow non-MGMT" dst-address=100.100.16.0/24 new-dst-ports="" ports=sfp-sfpplus8,sfp-sfpplus7 src-address=0.0.0.0/0 switch=\
switch1
add comment="completely disable sfpplus1" new-dst-ports="" ports=sfp-sfpplus1 switch=switch1
/interface list member
add interface=ether1 list=WAN
add interface=sfp-sfpplus1 list=LAN
add interface=sfp-sfpplus2 list=LAN
add interface=sfp-sfpplus3 list=LAN
add interface=sfp-sfpplus4 list=LAN
add interface=sfp-sfpplus5 list=LAN
add interface=sfp-sfpplus6 list=LAN
add interface=sfp-sfpplus7 list=LAN
add interface=sfp-sfpplus8 list=LAN
We reproduced the issue and confirm that, unfortunately, ACL rules do not work with L3 HW in v7.1beta6. The bug report has been submitted to the developers. Hopefully, it will get resolved in the next beta.
Thank you for the feedback!
Any news on whether route filters are working? Is there somewhere else that people who paid for a beta license can get support?
Please be more specific, in general route filters are working.
I think he is talking about this link (not updated yet to 7.1v5)
https://help.mikrotik.com/docs/display/ROS/v7+Routing+Protocol+Status
“Routing filter match prefix with address list”
“Routing filter add prefix to address list”
Is filtering working at v6? It would be great if Mikrotik updated this link.
Just rolled back from beta6 due to my crs328-24p-4s+rm becoming unresponsive after a couple of days after boot. It continues to pass packets, but no response to its internal IP.
Unfortunately, I didn’t get a chance to grab logs or delve deeper, but rolling back has fixed it.
There’s nothing funky in the config except a couple of 802.3ad bonds.
Hi …
I was talking about my post a few days back. BGP out-bound route filters
don’t seem to work; in-bound is ok.
Reiney
Yes, mine (Chateau LTE12) crashes within minutes after enabling cake simple queue.
Is there any known issue with Traffic Flow in beta6? It looks like I can’t get any netflow info into logstash (and I have checked everything in terms of config, FW, etc.)
If no one has an idea, I will open a support case to check if it really is an issue in MT or not.
Cheers,
anthonws.
Hopefully Cake will be stable. Currently using it (on RB750gr3) and random reboot twice so far. First one was nearly a day after setting up and the second just about 1 hour after.
edit: gonna stick to fq_codel the meantime.
Got ECMP?
Upgraded from v6-longterm to v7.1beta6, on an older RB953 with two LTE modems (Sierra MC7354, Telit LE960), to check it out. The modem both seem to working fine (once reconfig’d for MBIM & a seemingly a few reboots initially to get the ports/fw/carrier aligned, certainly needs the init delay.
BUT my issue is actually I can’t figure out how to do ECMP with the two LTE modems. I can’t seem to add multiple interface (or IP-based) “gateway”, in main or a specific table, using an interface nor another connected gateway IP. CLI or Winbox, all seem not want to take multiple gateways.
Basically is ECMP suppose to work? Maybe there some syntax for the /ip/route/add gateway=… butI tried a bunch of possible things for the two interface: “lte1,lte2” and “1.2.3.4,1.5.4.3”, “lte1 lte2”, etc., etc.
Here is something more specific:
/ip/route/add routing-table=ecmptable dst-address=0.0.0.0/0 gateway="lte1,lte2"
which appear to work with out error, however, this the resulting entry:
/ip/route/print detail
2 IsH dst-address=0.0.0.0/0 routing-table=ecmptable pref-src="" gateway=0.0.0.0 immediate-gw="" distance=1 scope=30 target-scope=10 suppress-hw-offload=no
Normally it the LTE chips sometimes fight…but that seems much improved! But now I’d like to use both of them in LTE mode 
I can’t figure out ECMP in v7.1beta6 however.
What happens if you add them one at a time? i.e. Your command line but with gateway=lte1
Then, same command with gateway=lte2. Do you get two entries? The conversion of “lte1, lte2”
to 0.0.0.0 seems odd. I’d try it here but I downgraded my test router back to v6, swapped it into
production, then forgot to remove the SATA drive that had the licensed v7 on it. 
In v7 gateway can be only one “IP/interface”, you need to add two route entries and that will form ECMP route.
Thanks MRZ - that worked - knew I was missing something… Pretty much as described, although avoiding doing anything winbox with ROS v7beta6 helped.
For completeness, this worked to create the routes.
/routing/table/add name=ecmp2 comment=ecmp2
/ip/route/add routing-table=ecmp2 gateway=lte1 comment=ecmp2
/ip/route/add routing-table=ecmp2 gateway=lte2 comment=ecmp2
which then show up in:
/routing/route/print detail
As + ;;; ecmp2
afi=ip4
contribution=active dst-address=0.0.0.0/0 routing-table=ecmp2 pref-src="" gateway=lte2 immediate-gw=lte2 distance=1 scope=30 target-scope=10
belongs-to="Static route"
debug.fwp-ptr=0x20202000
As + ;;; ecmp2
afi=ip4
contribution=active dst-address=0.0.0.0/0 routing-table=ecmp2 pref-src="" gateway=lte1 immediate-gw=lte1 distance=1 scope=30 target-scope=10
belongs-to="Static route"
debug.fwp-ptr=0x20202060
In my case the MBIM stuff just worked in v7beta6 -VERY COOL - why there is even a “lte1” and “lte2” interfaces to even be used by ECMP. The rest of the post is just documenting what I did in case it helps someone else.
I created a new VLAN/route table to test those in my lightly modified from quickset defaults RB953, using PBR for simplicity:
# vlan for ECMP route
/interface/vlan/add name=vlan-ecmp2 vlan-id=2 interface=bridge comment=ecmp2
/ip/address/add interface=vlan-ecmp2 address=203.0.113.1/24 comment=ecmp2
# dchp-server on ECMP route
/ip/dhcp-server/network/add address=203.0.113.0/24 gateway=203.0.113.1 comment=ecmp2
/ip/pool/add name=ecmp2 ranges=203.0.113.101-203.0.113.199 comment=ecmp2
/ip/dhcp-server/add address-pool=ecmp2 name=ecmp2 interface=vlan-ecmp2 disabled=no
# with default firewall, adding to LAN interface list will do NAT
/interface/list/member/add interface=vlan-ecmp2 list=LAN
# many ways to do this part, a policy rule seemed simple to show this working (NOTE: v6 had this under "/ip route rule")
/routing/rule/add src-address=203.0.113.0/24 table=ecmp2 comment=ecmp2
Skipping un-tagging the new VLAN 2 since that more config dependant… But using an access port from VLAN 2 above, a quick test using BitTorrent to download Ubuntu ISO goes out both MBIM-based LTE connections:
/interface/lte> monitor lte1,lte2
status: connected connected
model: LM960A18 MC7354
revision: 32.00.144 SWI9X15C_05.05.58.01
current-operator: AT&T Verizon
data-class: LTE LTE
session-uptime: 1h33m40s 1h28m28s
imei: 356299100000000 359225050000000
imsi: 310410000000000 311480000000000
uicc: 89014100000000000000 89148000000000000000
rssi: -93dBm -75dBm
/interface/lte> /interface/monitor-traffic lte1,lte2
name: lte1 lte2
rx-packets-per-second: 2 409 4 440
rx-bits-per-second: 13.0Mbps 25.2Mbps
fp-rx-packets-per-second: 2 409 4 440
fp-rx-bits-per-second: 13.0Mbps 25.2Mbps
rx-drops-per-second: 0 0
rx-errors-per-second: 0 0
tx-packets-per-second: 547 1 083
tx-bits-per-second: 285.6kbps 625.4kbps
fp-tx-packets-per-second: 0 0
fp-tx-bits-per-second: 0bps 0bps
tx-drops-per-second: 0 0
tx-queue-drops-per-second: 0 0
tx-errors-per-second: 0 0
It was actually doing higher speeds, but just captured one. Anyway v7 seems to have come a long way, considering we have to use PPP with Sierra modems in production today.
How does that actually work? I presume you rely on the ECMP only for the first packet of each connection to be routed to a random external connection, and then the connection tracking (NAT table) will force all the other traffic out of the chosen interface?
Otherwise it would only be useful for UDP traffic where a single UDP exchange constitutes the entire connection (DNS, NTP etc).
I would consider ECMP mostly to balance traffic over two links (e.g. VPN tunnels) where both ends are under my control and no NAT is involved.
Is there any advantage of using ECMP in your scenario, vs the “standard” solution of setting up route marking via some form of per-connection classification (based on IP+port or N’th matchers)?
Like others, the Let’s Encrypt worked (with port 80/firewall changes) - been waiting for that almost as long as MBIM: don’t like the self-signed certs but too lazy to deal with a cert hierachy.
But did run into some oddities in v7beta6 so far (hardware is RB953 in this case)…
winbox/webfig static route issues
Tried a few v7 beta now, so kinda know to use the CLI for now. But I tried in the ECMP/PBR stuff in both winbox and webfig, seem like there were still issues, noticed at least the following:
- In winbox, using “@” in the “Gateway” field in a listed IP>Routes crashes winbox
- Similarly, using IP>Route>Rule in winbox causes random crashes and displays different things from CLI sometimes.
- In webfig, the IP>Route display either shows empty or one route. Selecting the same item from webfig menu on left, then showed entire list.
Now that we have MBIM… some suggestions/issues
- LTE interface only shows “rssi” several TelIt and Sierra modems in MBIM mode - think I’ve seen it show in other beta, but RSSI is helpful be good to know rest of stats.
- Using ECM mode with TelIt LC960 in this v7.1beta6 shows the RSRQ etc (Sierra can’t do ECM, so only RSSI for those RN)
- While “at-chat” works - which is great - it strips “newlines” from some output… in particular ones that show CQI etc. See below. While remove newlines/“trimming” the AT output make sense most times, for the “big” AT commands with formated output, not as useful
Sierra’s output from these commands has columns with newlines so you can tell which numbers go to what:
/interface/lte/at-chat lte2 input="AT!LTEINFO\?"
output: !LTEINFO: Serving: EARFCN MCC MNC TAC CID Bd D U SNR PCI RSRQ RSRP RSSI RXLV 2100 311 480 7939 007A230C 4 3 3 -10 390 -11.1 -70.9 -42.6 53
IntraFreq: PCI RSRQ RSRP RSSI RXLV 390 -11.1 -70.9 -42.6 53 374 -19.0 -79.2 -50.2 53 InterFreq: EARFCN ThresholdLow ThresholdHi Priority PCI RSRQ
RSRP RSSI RXLV GSM: ThreshL ThreshH Prio NCC ARFCN 1900 valid BSIC RSSI RXLV WCDMA: UARFCN ThreshL ThreshH Prio PSC RSCP ECN0 RXLV CDMA 1x: Chan BC
Offset Phase Str CDMA HRPD: Chan BC Offset Phase Str OK
/interface/lte/at-chat lte2 input="AT!GSTATUS\?"
output: !GSTATUS: Current Time: 5268 Temperature: 38 Bootup Time: 0 Mode: ONLINE System mode: LTE PS state: Attached LTE band: B4 LTE bw: 10 MHz LTE Rx
chan: 2100 LTE Tx chan: 20100 EMM state: Registered Normal Service RRC state: RRC Connected IMS reg state: No Srv IMS mode: Normal RSSI (dBm): -41
Tx Power: 0 RSRP (dBm): -71 TAC: 1F03 (7939) RSRQ (dB): -15 Cell ID: 00... (80...) SINR (dB): 15.0 OK
For example, with v6 in serial-console, it show something like this:
at!gstatus?
!GSTATUS:
Current Time: 7483 Temperature: 64
Reset Counter: 1 Mode: ONLINE
System mode: LTE PS state: Attached
LTE band: B7 LTE bw: 20 MHz
LTE Rx chan: 3148 LTE Tx chan: 21148
LTE SSC1 state:INACTIVE LTE SSC1 band: B7
LTE SSC1 bw : 20 MHz LTE SSC1 chan: 2950
LTE SSC2 state:INACTIVE LTE SSC2 band: B3
LTE SSC2 bw : 15 MHz LTE SSC2 chan: 1275
LTE SSC3 state:NOT ASSIGNED
LTE SSC4 state:NOT ASSIGNED
EMM state: Registered Normal Service
RRC state: RRC Connected
IMS reg state: No Srv
PCC RxM RSSI: -62 PCC RxM RSRP: -100
PCC RxD RSSI: -65 PCC RxD RSRP: -105
SCC1 RxM RSSI: -74 SCC1 RxM RSRP: -103
SCC1 RxD RSSI: -80 SCC1 RxD RSRP: -108
SCC2 RxM RSSI: -70 SCC2 RxM RSRP: -93
SCC2 RxD RSSI: -75 SCC2 RxD RSRP: -97
Tx Power: 3 TAC: 204c (8268)
RSRQ (dB): -17.9 Cell ID: 080ba821 (134981665)
SINR (dB): -5.8
The TelIt seem to handle these status ones better:
/interface/lte>> at-chat lte1 input="AT#RFSTS"
output: #RFSTS: "310 410",800,-93,-59,-14,8B1E,255,-5,1280,19,2,A20...,"310410...","AT&T",3,2
OK
/interface/lte>> at-chat lte1 input="AT#moni"
output: #MONI: AT&T RSRP:-92 RSRQ:-17 TAC:8B1E Id:A204... EARFCN:800 PWR:-54dbm DRX:1280
But it too has output that the newlines/tab/etc are helpful:
at-chat lte1 input="AT#FIRMWARE"
output: HOST FIRMWARE : 32.00.005_1 MODEM FIRMWARE : 4 INDEX STATUS CARRIER VERSION TMCFG CNV LOC 1 Generic 32.00.115 1025 empty 1 2 Verizon 32.00.124 2020
empty 2 3 Activated ATT 32.00.144 4021 empty 3 4 TMUS 32.00.153 5004 empty 4 OK
In the above to switch firmware, it’s not that easy to parse the numbers (I had to look in their manual to be sure).
Otherwise, we’ll keep trying this one out. So far so good - great work!
When scripting, I found that
/ip dns static;
:put [get [ find type=A ]]
does not return the static DNS entries of type A. The attribute “type” seems to be nil for all entries of type A. Is this expected behaviour?