MikroTik

Posted: **Fri Feb 28, 2020 11:36 pm**

ccr2004-1g-12s+2xs
ccr2016-1g-12xs-2xq
ccr2004-2g-2xc-2s+
ccr2004-2g-2xc-2s+/r2
ccr2004-2g-2xc-2s+/r2-60

All referenced in a kernel object file in v7.0b5...

Posted: **Fri Feb 28, 2020 11:53 pm**

Also this...

Posted: **Tue Mar 03, 2020 2:39 am**

This is getting better and better...

https://sg.alagas.shop/mtik-ccr2004-1g- ... s-2xs.html
https://tw.alagas.shop/mtik-ccr2016-1g-12xs-2xq.html

Posted: **Tue Mar 03, 2020 3:56 am**

I don't understand XS, XC, and XQ.

XG makes sense.

Are they deprecating C+ ?

Posted: **Tue Mar 03, 2020 12:02 pm**

I don't understand XS, XC, and XQ.

According to the links @Paternot found (but are gone now), XS would be '40 Gigabit' and XQ '40 Gigabit combo'. With '40 Gigabit' being (I assume) 40GBASE-T (because of the 'combo' bit, and we already have 'Q+' for QSFP+ 40G).

To my great surprise 40GBASE-T actually exists as a standard (it's defined in IEEE 802.3bq), but who in his right mind would ever use it? It needs a CAT8 cabling plant, which nobody has, and makes no sense to install even on a greenfield because just going to OM4 is cheaper. And even if you had one it then only reaches 30 meters! And if 10GBASE-T is any indication, you can probably use the tranceivers to boil your tea...

Nope, not seeing it.

Posted: **Tue Mar 03, 2020 12:37 pm**

Any guess what can be this 3 variations also what 60ayD could mean?

ccr2004-2g-2xc-2s+
ccr2004-2g-2xc-2s+/r2
ccr2004-2g-2xc-2s+/r2-60

Posted: **Tue Mar 03, 2020 12:41 pm**

According to the links @Paternot found (but are gone now), XS would be '40 Gigabit' and XQ '40 Gigabit combo'. With '40 Gigabit' being (I assume) 40GBASE-T (because of the 'combo' bit, and we already have 'Q+' for QSFP+ 40G).

Gone? Stupid me - should've got some screens...

Posted: **Tue Mar 03, 2020 12:43 pm**

Intresting. The ccr2016 variant, AL73400 possibly?

40GBASE-T seems crazy.

Posted: **Tue Mar 03, 2020 12:50 pm**

All hail Google cache!

ccr2004-1g-12s+2xs.png

ccr2016-1g-12xs-2xq.png

Posted: **Tue Mar 03, 2020 1:00 pm**

Ah! Didn't think of google cache. Interesting prices too; 10K NT$ is only US $334,=. Somehow I don't think they'll be *that* cheap

Posted: **Tue Mar 03, 2020 1:13 pm**

Ah! Didn't think of google cache. Interesting prices too; 10K NT$ is only US $334,=. Somehow I don't think they'll be *that* cheap

I don't think they are the real prices. Many sites do this: put an impossible price, when the item is out of stock.

Posted: **Tue Mar 03, 2020 1:31 pm**

Intresting. The ccr2016 variant, AL73400 possibly?

Yeah, might well be an AL73400, even if the network specs seem a bit low for that (https://en.wikipedia.org/wiki/Annapurna_Labs). It's 16-core, and MT has experience with the Alpine platform in de RB4011.

@normis, 'fess up. What are the lot of you cooking up in the dark cellars of the MT Skunk Works?

Posted: **Tue Mar 03, 2020 2:10 pm**

I don't understand XS, XC, and XQ.

According to the links @Paternot found (but are gone now), XS would be '40 Gigabit' and XQ '40 Gigabit combo'. With '40 Gigabit' being (I assume) 40GBASE-T (because of the 'combo' bit, and we already have 'Q+' for QSFP+ 40G).

To my great surprise 40GBASE-T actually exists as a standard (it's defined in IEEE 802.3bq), but who in his right mind would ever use it? It needs a CAT8 cabling plant, which nobody has, and makes no sense to install even on a greenfield because just going to OM4 is cheaper. And even if you had one it then only reaches 30 meters! And if 10GBASE-T is any indication, you can probably use the tranceivers to boil your tea...

Nope, not seeing it.

My point was:

- G means GBASE-T so X would just mean 10
- XQ should actually be QC if XC replaces C+
- There's nothing that says 40 in XS and it would be unrelated to SFP

Posted: **Tue Mar 03, 2020 2:49 pm**

My point was:

- G means GBASE-T so X would just mean 10
- XQ should actually be QC if XC replaces C+
- There's nothing that says 40 in XS

I agree that 'XS' as 40GBASE-T doesn't make much sense, but the specs that shop gave for them do indicate that 'XS' means '40 Gigabit ethernet' of some sort, and 'XQ' meaning '40 Gigabit ethernet (combo)'. And the strings from the 7.0beta libs also point in that direction. That 'XC' seems to replace 'C+' as 10G combo port I agree with.

Al in all, it's confusing.

Posted: **Tue Mar 03, 2020 3:27 pm**

what 60ayD could mean?

Terragraph?

Posted: **Tue Mar 03, 2020 3:46 pm**

Some clues here? https://wiki.mikrotik.com/wiki/Manual:P ... ng_details

Posted: **Tue Mar 03, 2020 3:50 pm**

what 60ayD could mean?
Terragraph?

Yeah, might be 802.11ay.

Posted: **Tue Mar 03, 2020 3:59 pm**

Some clues here? https://wiki.mikrotik.com/wiki/Manual:P ... ng_details

So this was in a wiki modification from July 2018 that lacks a comment.

Posted: **Tue Mar 03, 2020 4:01 pm**

Some clues here? https://wiki.mikrotik.com/wiki/Manual:P ... ng_details

OK, now that makes a lot more sense. a CCR2004-1G-12S+2XS has 12 SFP+ ports and 2 25G SFP+ ports, and the CCR2016-1G-12XS-2XQ has...<blink>...12x25G & 2x100G?

I want to see prices for that CCR2016... And I want to know what's pushing those boxes. I don't think the Alpine SOCs will cut it for speeds like that, not even the AL73400/Graviton.

Posted: **Tue Mar 03, 2020 4:03 pm**

what 60ayD could mean?
Terragraph?

Yeah, might be 802.11ay.

What else? Dual chain.

Posted: **Tue Mar 03, 2020 4:41 pm**

what 60ayD could mean?
Terragraph?

Yeah, might be 802.11ay.
What else? Dual chain.

Yes make sense Wifi 802.11ay 60 GHz Dual chain but what really doesn't make sense is to put a wifi interface in a Core router that has 2xGiga port + 10G SFP+ ports and 2x 25G SFP+

Posted: **Tue Mar 03, 2020 4:47 pm**

Yes make sense Wifi 802.11ay 60 GHz Dual chain but what really doesn't make sense is to put a wifi interface in a Core router that has 2xGiga port + 10G SFP+ ports and 2x 25G SFP+

The port configuration is actually quite minimal for SOHO.

Some people would actually look for Q+.

Posted: **Tue Mar 03, 2020 5:11 pm**

The port configuration is actually quite minimal for SOHO.

Some people would actually look for Q+.

We've been here before: I want what you are smoking. Really. We're getting to the point that 10G is useful and affordable for prosumers and enthousiasts. Calling a router with 25G ports 'quite minimal for SOHO' is lunacy.

Posted: **Tue Mar 03, 2020 5:15 pm**

The port configuration is actually quite minimal for SOHO.

Some people would actually look for Q+.

We've been here before: I want what you are smoking. Really. We're getting to the point that 10G is useful and affordable for prosumers and enthousiasts. Calling a router with 25G ports 'quite minimal for SOHO' is lunacy.

It is not lunacy when a 40G card costs about 300 euro.

Professionals also work at home.

Posted: **Tue Mar 03, 2020 5:30 pm**

It is not lunacy when a 40G card costs about 300 euro.

Professionals also work at home.

Vortex, there is a difference between 'there is a market for 40G for home' and 'a router with 25G ports is quite minimal for SOHO'. I agree with the first, small as I think that market to be. The second does not follow from the first. If you don't see that, this discussion is pointless.

Posted: **Tue Mar 03, 2020 5:52 pm**

It is not lunacy when a 40G card costs about 300 euro.

Professionals also work at home.

Vortex, there is a difference between 'there is a market for 40G for home' and 'a router with 25G ports is quite minimal for SOHO'. I agree with the first, small as I think that market to be. The second does not follow from the first. If you don't see that, this discussion is pointless.

40G is the new 10G.

When I was asking for home routers capable of 10G switching some years ago, it was professional level. Now not only is 10G cheap, some people even have 10G WAN at home.

Why would you buy a couple of 25G cards when they are not much cheaper than 40G? Workstations and NAS that can take them are not cheaper.

I did not buy a 10G NAS because it was slower than TB2. I will similarly avoid buying a 25G NAS because it is slower than TB3. If I really needed a serious NAS, I would want 40G (with NVMe slots of course).

Posted: **Tue Mar 03, 2020 6:28 pm**

It is not lunacy when a 40G card costs about 300 euro.

Professionals also work at home.

Vortex, there is a difference between 'there is a market for 40G for home' and 'a router with 25G ports is quite minimal for SOHO'. I agree with the first, small as I think that market to be. The second does not follow from the first. If you don't see that, this discussion is pointless.
40G is the new 10G.

When I was asking for home routers capable of 10G switching some years ago, it was professional level. Now not only is 10G cheap, some people even have 10G WAN at home.

Why would you buy a couple of 25G cards when they are not much cheaper than 40G? Workstations and NAS that can take them are not cheaper.

I did not buy a 10G NAS because it was slower than TB2. I will similarly avoid buying a 25G NAS because it is slower than TB3. If I really needed a serious NAS, I would want 40G (with NVMe slots of course).

That all is plausible, what I don't get it is why to put a Wifi interface on CCR device? in POV it totally wastes of resources

Posted: **Tue Mar 03, 2020 6:33 pm**

That all is plausible, what I don't get it is why to put a Wifi interface on CCR device? in POV it totally wastes of resources

Even if it can route much faster than 10000/10000, it can make sense for SOHO for IDS or VPN.

Posted: **Tue Mar 03, 2020 6:45 pm**

That all is plausible, what I don't get it is why to put a Wifi interface on CCR device? in POV it totally wastes of resources

Look at it from a WISP perspective. You'd use the CCR2004-2G-2S+-2XS-60ayD a your wireless "POP". Put it right on a signal tower / post. Connect your 10/25G fibre uplinks to it, and spread those to various locations via 60GHz, and from then on.
Example: 2 25G Uplinks to the CCR. The CCR connects to 2x4=8 distribution wireless systems. Imagine those as "LHG 5XP-60ay" or "LHG 5P+-60ay" (comparable to LHG 60G) with 5 Multigigabit PoE ports, each connecting to a pair of "wAP 60Gx3 AP" or different 60GHz APs with a smaller degree field of view.
That would net you up to 128 100Mbit Clients with "Cube Lite60" as customer client systems slaved to a single CCR2004.
Or quite a bit more if a .ay variant was used to connect the customer client devices.

Just my 2 cents

Posted: **Tue Mar 03, 2020 6:49 pm**

Some clues here? https://wiki.mikrotik.com/wiki/Manual:P ... ng_details

OK, now that makes a lot more sense. a CCR2004-1G-12S+2XS has 12 SFP+ ports and 2 25G SFP+ ports, and the CCR2016-1G-12XS-2XQ has...<blink>...12x25G & 2x100G?

I want to see prices for that CCR2016... And I want to know what's pushing those boxes. I don't think the Alpine SOCs will cut it for speeds like that, not even the AL73400/Graviton.

I think CN98XX.

Yesterday:
viewtopic.php?f=3&t=158246

Posted: **Tue Mar 03, 2020 6:55 pm**

That all is plausible, what I don't get it is why to put a Wifi interface on CCR device? in POV it totally wastes of resources
Look at it from a WISP perspective. You'd use the CCR2004-2G-2S+-2XS-60ayD a your wireless "POP". Put it right on a signal tower / post. Connect your 10/25G fibre uplinks to it, and spread those to various locations via 60GHz, and from then on.
Example: 2 25G Uplinks to the CCR. The CCR connects to 2x4=8 distribution wireless systems. Imagine those as "LHG 5XP-60ay" or "LHG 5P+-60ay" (comparable to LHG 60G) with 5 Multigigabit PoE ports, each connecting to a pair of "wAP 60Gx3 AP" or different 60GHz APs with a smaller degree field of view.
That would net you up to 128 100Mbit Clients with "Cube Lite60" as customer client systems slaved to a single CCR2004.
Or quite a bit more if a .ay variant was used to connect the customer client devices.

Just my 2 cents

so you are putting the ccr as an very over blaster overkill AccessPoint? also in top of the Tower/post?

Posted: **Tue Mar 03, 2020 9:29 pm**

The 40G cards (QSFP+) do not support 25G. They do not cost much more than a SFP28 25G card.

A 50G card (QFSP28) supporting both 40G and 25G is not much cheaper than a similar 100G card.

Who needs a CCR2016-1G-5XQ first? Or more Q+ CRS?

Posted: **Tue Mar 03, 2020 11:50 pm**

I extrapolated these codes out:

ccr2004-1g-12s+2xs 		4 Core, 1x 1G, 12x 10G SFP+, 2x 25G SFP28
ccr2016-1g-12xs-2xq		4 Core, 1x 1G, 12x 25G SFP+, 2x 100G QSFP28
ccr2004-2g-2xc-2s+		4 Core, 2x 1G, 2x 10G/25G SFP28, 2x 10G SFP+
ccr2004-2g-2xc-2s+/r2 		4 Core, 2x 1G, 2x 10G/25G SFP28, 2x 10G SFP+, 2.4Ghz WiFi
ccr2004-2g-2xc-2s+/r2-60	4 Core, 2x 1G, 2x 10G/25G SFP28, 2x 10G SFP+, 2.4Ghz WiFi, 802.11ay 60Ghz

CCR2004-2G-2S+2XS-60ayD		Shorter Production model name for ccr2004-2g-2xc-2s+/r2-60

It all seems possible, but not completely logical.

I find it VERY odd that these are referenced in a 32bit driver blob for ARM... Especially the ccr2016. This strikes me as either an intentional troll/hint or devs being sloppy.

Posted: **Wed Mar 04, 2020 12:05 am**

I think CN98XX.

Marvell does not show only 16 cores in this subfamily.

Posted: **Wed Mar 04, 2020 2:11 am**

I _really_ wish it was Marvell Octeon TX2, but this platform looks like Annapurna Labs ARM SoC's.

Possibly Mikrotik have combined Annapurna Labs ARM core's with Marvell Prestera Switch ASIC's, where the AL chip does the initial packet pass (slow path) then the fast-path is handled by the Marvell ASIC. This would allow them to scale port counts independently of the ARM core and would also allow them to scale the platform from low-cost designs (CCR2004) with just an Annapurna Labs ARM processor, to higher-cost designs (CCR2016) with a Annapurna Labs core and a Marvell Prestera switch ASIC.

Posted: **Wed Mar 04, 2020 3:00 am**

So AWS runs on 25GbE...

I now pay attention and see that the first filename is al.ko .

Posted: **Wed Mar 04, 2020 12:30 pm**

Possibly Mikrotik have combined Annapurna Labs ARM core's with Marvell Prestera Switch ASIC's, where the AL chip does the initial packet pass (slow path) then the fast-path is handled by the Marvell ASIC. This would allow them to scale port counts independently of the ARM core and would also allow them to scale the platform from low-cost designs (CCR2004) with just an Annapurna Labs ARM processor, to higher-cost designs (CCR2016) with a Annapurna Labs core and a Marvell Prestera switch ASIC.

Yes, that's what it looks like. I don't think the ARM core can do it alone in the CCR2004s; the fastest quadcore Alpine is the AL21400 MT already uses in the RB4011, AFAIK, and even at 2GHz it doesn't have the horsepower for that. Combining it with a PresteraDX ASIC to do the heavy lifting would make sense. Cisco has been doing something like it for ages with NetFlow switching and CEF; the 'route processor' CPU does the control plane and programs the ASICs to do the real work.

Posted: **Wed Mar 04, 2020 1:30 pm**

I just used the Marvell Prestera's as an example, because I know the ASIC's are capable of it, and that Mikrotik are familiar with them due to their use in CRS3xx series. They could use almost any of the modern L3+ switch ASIC's.

The Marvell Octeon TX2 and possibly NXP ARM based SOC's are really the only ARM SOC's capable of the port counts the above Mikrotik CCR20xx model codes indicate natively, the Annapurna Labs stuff that is public does not have support for that number of interfaces so if they are using the Annapurna Labs SOC's it is either some unknown version of it, or running external switch ASIC's for the fast-path as I suggested above.

Posted: **Wed Mar 04, 2020 2:07 pm**

I just used the Marvell Prestera's as an example, because I know the ASIC's are capable of it, and that Mikrotik are familiar with them due to their use in CRS3xx series. They could use almost any of the modern L3+ switch ASIC's.

The Marvell Octeon TX2 and possibly NXP ARM based SOC's are really the only ARM SOC's capable of the port counts the above Mikrotik CCR20xx model codes indicate natively, the Annapurna Labs stuff that is public does not have support for that number of interfaces so if they are using the Annapurna Labs SOC's it is either some unknown version of it, or running external switch ASIC's for the fast-path as I suggested above.

Fully agree. Annapurna Labs ARM SOC + Marvell PresteraDX would make the most sense from a historical point of view, as MT already has experience with both and glueing them together would be a logical next step. And it matches the hints we have (like the 'al.ko' module name the strings are from indicating AL involvement somewhere).

Of course it could be a red herring, but I don't really think so. AL21400 & AL73400 match the 4 & 16 core counts indicated for the new boxes, and although the L3 switch ASIC could in theory be anything, it being a Prestera of some stripe makes the most sense.

Interesting stuff to hypothesize about

Posted: **Wed Mar 04, 2020 4:27 pm**

I only saw the Prestera port extenders matching these speeds.

Posted: **Wed Mar 04, 2020 4:35 pm**

I only saw the Prestera port extenders matching these speeds.

Marvell have already made custom Prestera DX ASIC's for Mikrotik...

Posted: **Wed Mar 04, 2020 5:05 pm**

Maybe a CCR2016-1G-6VQ-2XQ would be more interesting than a CCR2016-1G-12XS-2XQ.

Posted: **Thu Mar 05, 2020 10:43 am**

If we really are talking about multiple 25G and QSFP28 - then the forwarding must be done in hardware on a switch ASIC. No ARM or Intel CPU can shuffle that amounts of data.

It will probably then be a V7-only device with this specific forwarding delegation is implemented.

Posted: **Thu Mar 05, 2020 1:17 pm**

If we really are talking about multiple 25G and QSFP28 - then the forwarding must be done in hardware on a switch ASIC. No ARM or Intel CPU can shuffle that amounts of data.

It will probably then be a V7-only device with this specific forwarding delegation is implemented.

That's an interesting though. Does it mean that V7 final is closer than we thought? We already saw one site with the hardware, even if still "unavailable" and pulled out later.

Posted: **Thu Mar 05, 2020 3:12 pm**

Marvell have already made custom Prestera DX ASIC's for Mikrotik...

Do you know for which devices?

Posted: **Thu Mar 05, 2020 11:42 pm**

Marvell have already made custom Prestera DX ASIC's for Mikrotik...
Do you know for which devices?

The 98DX8208 in the CRS309

Posted: **Fri Mar 06, 2020 3:48 pm**

Anyone think this is related?

https://www.servethehome.com/new-marvel ... x-5g-socs/

Posted: **Mon Mar 09, 2020 6:16 pm**

http://www.broadbandworldnews.com/autho ... _id=758024
really need the new ccr2004 line now for ATT bypass

Posted: **Tue Mar 10, 2020 2:37 am**

Anyone think this is related?

https://www.servethehome.com/new-marvel ... x-5g-socs/

I certainly hope that is the case, but all the evidence above points to CCR2000 series being an Annapurna Labs based architecture.

Posted: **Tue Mar 10, 2020 2:50 am**

To my great surprise 40GBASE-T actually exists as a standard (it's defined in IEEE 802.3bq), but who in his right mind would ever use it? It needs a CAT8 cabling plant, which nobody has, and makes no sense to install even on a greenfield because just going to OM4 is cheaper. And even if you had one it then only reaches 30 meters!

If fibre is easy to break, some home users would prefer copper.

Posted: **Tue Mar 10, 2020 8:34 am**

Maybe there will be an announcement when the European mum was supposed to commence....

Posted: **Mon Mar 16, 2020 11:15 pm**

I extrapolated these codes out:

ccr2004-1g-12s+2xs 		4 Core, 1x 1G, 12x 10G SFP+, 2x 25G SFP28
ccr2016-1g-12xs-2xq		4 Core, 1x 1G, 12x 25G SFP+, 2x 100G QSFP28
ccr2004-2g-2xc-2s+		4 Core, 2x 1G, 2x 10G/25G SFP28, 2x 10G SFP+
ccr2004-2g-2xc-2s+/r2 		4 Core, 2x 1G, 2x 10G/25G SFP28, 2x 10G SFP+, 2.4Ghz WiFi
ccr2004-2g-2xc-2s+/r2-60	4 Core, 2x 1G, 2x 10G/25G SFP28, 2x 10G SFP+, 2.4Ghz WiFi, 802.11ay 60Ghz

It all seems possible, but not completely logical.

imagine a device - the ccr2004-2g-xxx - which is compact. as compact as a powerbox, maybe thicker. has decent fiberoptic options for high speed fixed backhaul.
2.4GHz wifi for easier management if it is mounted to a tower / pole.
and has 2 independent .11ay radios, each with 180 degree FOV - using 3 RF stages each, similarly to the current APx3.
because this is what this looks like.
and i would buy them on first sight.

Posted: **Mon Mar 16, 2020 11:56 pm**

ccr2004-1g-12s+2xs
ccr2016-1g-12xs-2xq
ccr2004-2g-2xc-2s+
ccr2004-2g-2xc-2s+/r2
ccr2004-2g-2xc-2s+/r2-60

All referenced in a kernel object file in v7.0b5...

well, they are also to be found in 6.47b35 (checked the arm build)

along with this baby: CRS319-1G-16P-2S+
and this is not the already known netPower device (CRS318-1Fi-15Fr-2S)

Posted: **Tue Mar 17, 2020 9:03 am**

I extrapolated these codes out:
Code: Select all
ccr2004-1g-12s+2xs 		4 Core, 1x 1G, 12x 10G SFP+, 2x 25G SFP28
ccr2016-1g-12xs-2xq		4 Core, 1x 1G, 12x 25G SFP+, 2x 100G QSFP28
ccr2004-2g-2xc-2s+		4 Core, 2x 1G, 2x 10G/25G SFP28, 2x 10G SFP+
ccr2004-2g-2xc-2s+/r2 		4 Core, 2x 1G, 2x 10G/25G SFP28, 2x 10G SFP+, 2.4Ghz WiFi
ccr2004-2g-2xc-2s+/r2-60	4 Core, 2x 1G, 2x 10G/25G SFP28, 2x 10G SFP+, 2.4Ghz WiFi, 802.11ay 60Ghz
It all seems possible, but not completely logical.
imagine a device - the ccr2004-2g-xxx - which is compact. as compact as a powerbox, maybe thicker. has decent fiberoptic options for high speed fixed backhaul.
2.4GHz wifi for easier management if it is mounted to a tower / pole.
and has 2 independent .11ay radios, each with 180 degree FOV - using 3 RF stages each, similarly to the current APx3.
because this is what this looks like.
and i would buy them on first sight.

I think you are on the right track here, and this could be a device in a square plastic enclosure that has connectors and a mounting bracket on the bottom to act as a Terragraph node.

I have no real idea, but it makes sense and the speculation is fun

Posted: **Tue Mar 17, 2020 9:26 am**

I think you are on the right track here, and this could be a device in a square plastic enclosure that has connectors and a mounting bracket on the bottom to act as a Terragraph node.

I have no real idea, but it makes sense and the speculation is fun

i built a fairly big network out of rev.5 terragraph nodes. their next gén stuff is in early prototype phase.
now this network is converted into a wap60G based deployment - and i have now different issues

the terragraph mac&phy had more controls over things how links are built, etc.
but lacked on some critical part where routeros excels. so i definitely want to have the best of both worlds.

btw, i also dissected some binaries in the arm driver stack and found stuff for qualcomm's talyn baseband (6436), so .11ay is definitely there.

Posted: **Wed Apr 22, 2020 4:06 pm**

Well one of these is now semi-announced:

https://mt.lv/CCR2004_1G_12S_2XS

It supports the -48volt DC PSU for the r2 CCR's as well!

Posted: **Wed Apr 22, 2020 4:37 pm**

Well one of these is now semi-announced:

https://mt.lv/CCR2004_1G_12S_2XS

It supports the -48volt DC PSU for the r2 CCR's as well!

Very cool! I'm looking forward to the 16 core version for replacing some CCR1072 BGP routers, but those numbers for the 4 core one are pretty promising.

The only head scratcher is the RAM... "RouterOS v6 1792MB ECC / RouterOS v7 4GB ECC", so for some reason we can't get the full 4GB of RouterOS 6? That's a bit of a weird limitation and until RouterOS 7 starts coming out with stable releases, having it in the core network makes me pretty uneasy.

Posted: **Wed Apr 22, 2020 5:40 pm**

Very cool! I'm looking forward to the 16 core version for replacing some CCR1072 BGP routers, but those numbers for the 4 core one are pretty promising.

The only head scratcher is the RAM... "RouterOS v6 1792MB ECC / RouterOS v7 4GB ECC", so for some reason we can't get the full 4GB of RouterOS 6? That's a bit of a weird limitation and until RouterOS 7 starts coming out with stable releases, having it in the core network makes me pretty uneasy.

I don't think they are all that promising actually. If this box is on par with the CCR1016, that's about 20 Gbps routing performance going downhill and with a stiff breeze at its back. Yes, the SOC is far more powerful per core than a Tilera, which will do wonders for things like BGP convergence (and 3.5 Gbps of single tunnel IPSEC is nice), but 20Gbps of IP forwarding is pretty anemic for a box you can pump 170 Gbps into...

I didn't really expect wire speed, but it's not the 'Marvell PresteraDX data plane + Annapurna Labs control plane' we were philosophizing about in this thread.

And I agree that the RAM bit is weird

Posted: **Thu Apr 23, 2020 3:14 am**

Well one of these is now semi-announced:

https://mt.lv/CCR2004_1G_12S_2XS

It supports the -48volt DC PSU for the r2 CCR's as well!
Very cool! I'm looking forward to the 16 core version for replacing some CCR1072 BGP routers, but those numbers for the 4 core one are pretty promising.

The only head scratcher is the RAM... "RouterOS v6 1792MB ECC / RouterOS v7 4GB ECC", so for some reason we can't get the full 4GB of RouterOS 6? That's a bit of a weird limitation and until RouterOS 7 starts coming out with stable releases, having it in the core network makes me pretty uneasy.

I think actual routeros V6 for ARM is 32 bit, i think this it's a sign that it will continue to be 32bit forever in V6

if you want routeros 64 bit for ARM you have to go to V7

mikrotik seems to be turning to routeros 7 (finally)

CCR1036 with 12.000 simple queues uses only 1.5GB of RAM

which scenarios require vast amounts of RAM beyond 1.7GB ??? They are many? Jumping from 2GB to 4GB is enough for that scenarios??

Posted: **Thu Apr 23, 2020 3:34 am**

which scenarios require vast amounts of RAM beyond 1.7GB ??? They are many? Jumping from 2GB to 4GB is enough for that scenarios??

One I can think of: full BGP tables. With the improvements made on RoS, and the higher single core speed, these new CCRs should work really well with this kind of load.

Posted: **Thu Apr 23, 2020 10:27 am**

no 40G port support?

In the text i read about it but in technical overiew i only see 25G ?

thanks,
glueck

Posted: **Thu Apr 23, 2020 1:21 pm**

no 40G port support?

In the text i read about it but in technical overiew i only see 25G ?

Nope, no 40G. The new CCRs hinted at in the new software (and now with the first one in datasheet) are 10/25/100G:

ccr2004-1g-12s+2xs 		 4 Core, 1x 1G, 12x 10G SFP+, 2x 25G SFP28
ccr2016-1g-12xs-2xq		16 Core, 1x 1G, 12x 25G SFP+, 2x 100G QSFP28
ccr2004-2g-2xc-2s+		 4 Core, 2x 1G, 2x 10G/25G SFP28, 2x 10G SFP+
ccr2004-2g-2xc-2s+/r2 		 4 Core, 2x 1G, 2x 10G/25G SFP28, 2x 10G SFP+, 2.4Ghz WiFi
ccr2004-2g-2xc-2s+/r2-60	 4 Core, 2x 1G, 2x 10G/25G SFP28, 2x 10G SFP+, 2.4Ghz WiFi, 802.11ay 60Ghz

Posted: **Thu Apr 23, 2020 3:22 pm**

The market has moved on:

10G/40G: SOHO/SMB
25G/100G: datacenter

Posted: **Thu Apr 23, 2020 10:55 pm**

10G/40G: SOHO/SMB
25G/100G: datacenter

sort of. 40G was _the_ datacenter thing for quite a while, but almost entirely for switching / spine+leaf. there was for example no 'real' cisco router with 40G interfaces - they had 10GE or 100GE. the thing with 40G is that the form factor is really not efficient today. the same space can accommodate a qsfp28 with 100G and it can be breaked out to 4x25G, in the same way as 40GE can be split into 4x10G. the fact of the matter is, that forwarding ASICs/NPs have a huge amount of bandwidth available, but we struggle to find enough faceplate real-estate to accommodate the connectors. so there are already a truckload of various 32x or 64x 100GE pizza boxes on the market. honestly, they might even sacrifice the space for the lights out interface and require an app to display link states if they could put an extra port on the device.

other than that, 25G/50G/100G really pair up. and the story goes further, to 200G/400G/800G - not quite my tempo for now. with FlexE you also have the possibility to build ethernet pipes of arbitrary size - albeit the concept of a 'leased line' is a blast from the past. 50G is not that common today, the whole working group was set up just about 4 years ago. probably we have a lot more that we could handle - but it's good to see how far we got from the 3xFE router boards. even the so called 'carrier grade' networking is totally available for a reasonable price. IMO, the classic hierarchical network approach will soon be disappearing, as the demand for content grows faster as we can build networks with bigger and bigger pipes in the core, so CDNs will reach soon the edges and maybe go even further.

Posted: **Fri Apr 24, 2020 6:15 am**

For SOHO, a 100G switch with 24x 10G and 2x 100G would be very interesting.

Btw, here are some 100G switches with prices in the range €4k to €11k:
https://www.fs.com/de-en/c/100g-switches-3258
From that list I find the "S5850-48S2Q4C 48-Port 10Gb SFP+ L2/L3 Carrier Grade Switch with 6 Hybrid 40Gb/100Gb Uplink Ports for € 5.117,00" interesting (https://www.fs.com/de-en/products/29124.html ), but for my taste too pricey for SOHO; this has to come down to <€1k.

I personally will simply skip/ignore everything between 10G and 100G, because the true evolution each time was 10-fold:
Ethernet (10 Mbps), FastEthernet (100Mbps), GigaBit (1000 Mbps), 10G (10000 Mbps), so consequently the next one has to be 100G

Forget 2.5G, 5G, 25G, 40G, 50G etc..

Posted: **Sat Apr 25, 2020 8:27 am**

Mikrotik cannot make the ideal device! Why it wasn’t easy to make support for USB, memory cards and several miniPCI-E slots and of course the ability to connect or immediately complete an SSD.
In my opinion better to make one universal device that will be easier to maintain.
Marketing, damn it ...

Posted: **Sat Apr 25, 2020 9:39 am**

Mikrotik cannot make the ideal device! Why it wasn’t easy to make support for USB, memory cards and several miniPCI-E slots and of course the ability to connect or immediately complete an SSD.
In my opinion better to make one universal device that will be easier to maintain.
Marketing, damn it ...

Buy a pc

Posted: **Sat Apr 25, 2020 10:24 am**

Mikrotik cannot make the ideal device! Why it wasn’t easy to make support for USB, memory cards and several miniPCI-E slots and of course the ability to connect or immediately complete an SSD.
In my opinion better to make one universal device that will be easier to maintain.
Marketing, damn it ...
Buy a pc

Want to buy Mikrotik's "PC". Buying PC I can use more powerful OS.

Posted: **Sat Apr 25, 2020 12:17 pm**

Mikrotik cannot make the ideal device! Why it wasn’t easy to make support for USB, memory cards and several miniPCI-E slots and of course the ability to connect or immediately complete an SSD.
In my opinion better to make one universal device that will be easier to maintain.
Marketing, damn it ...

You have to understand that MikroTik routers are built around "System-on-a-Chip" devices (SoC) which contain a certain mix of CPU (of course) and I/O ports.
When a certain SoC has been selected that has many nice features for a router, but does not include features that you desire, the router would have to include extra chips to provide those features, and it would increase complexity and cost. That could make it less attractive for buyers who do not need those features.
Also, in such a solution the performance of the extra features (like USB or SSD) is often poor, because they connect to the SoC via a slow bus such as I2C or SPI. See the (older) Raspberry Pi and similar boards for an example of what that would lead to.
Of course in a modern SoC one could expect that it has some PCIe channels for connecting fast peripherals and storage, but I have no detail on the situation on the SoC used in this product.

Posted: **Sat Apr 25, 2020 12:30 pm**

Why the need for SSD storage in a router?

https://mt.lv/CCR2004_1G_12S_2XS

The texts and descriptions feels a bit unpolished. Impressive power consumption and very nice with ECC memory. Looking forward to a block diagram as well.

Posted: **Sat Apr 25, 2020 5:09 pm**

Why the need for SSD storage in a router?

https://mt.lv/CCR2004_1G_12S_2XS
The texts and descriptions feels a bit unpolished. Impressive power consumption and very nice with ECC memory. Looking forward to a block diagram as well.

ftp, dude, etc.
Using internal 128 mbytes will kill it flash in a little time and divice goes into thrash.
I understand Mikrotik routers uses not from a good life, for the same reason I want several uses for one device.

Posted: **Sat Apr 25, 2020 5:24 pm**

I understand Mikrotik routers uses not from a good life, for the same reason I want several uses for one device.

What do you mean, man?

Posted: **Sat Apr 25, 2020 8:35 pm**

ftp, dude, etc.
Using internal 128 mbytes will kill it flash in a little time and divice goes into thrash.
I understand Mikrotik routers uses not from a good life, for the same reason I want several uses for one device.

Of course it will kill the NAND flash after a while, it's only designed to run the OS and store configuration.

I don't use dude myself but it does run far better in a CHR on a cheap PC/server. That goes for FTP as well.

Posted: **Sat Apr 25, 2020 8:47 pm**

ftp, dude, etc.
Using internal 128 mbytes will kill it flash in a little time and divice goes into thrash.
I understand Mikrotik routers uses not from a good life, for the same reason I want several uses for one device.
Of course it will kill the NAND flash after a while, it's only designed to run the OS and store configuration.

I don't use dude myself but it does run far better in a CHR on a cheap PC/server. That goes for FTP as well.

PC is also worth the money and takes up space in the rack. It’s better to pay for Mikrotik some money for the SSD option or buy an SD card. You will not deny that the presence of an SD card in the CCR1000 is a necessary option.

Posted: **Sat Apr 25, 2020 9:50 pm**

@Mikhalich

Could you use "Post reply" button instead of quotting previous posts all the time? Most readers are able to read them if they need it and there is no need to quote.

Posted: **Sun Apr 26, 2020 5:02 am**

For SOHO, a 100G switch with 24x 10G and 2x 100G would be very interesting.

You mean SO or also HO?

Posted: **Sun Apr 26, 2020 3:25 pm**

For SOHO, a 100G switch with 24x 10G and 2x 100G would be very interesting.
You mean SO or also HO?

I primarily mean SO, but the industry prefers the term SOHO...

Small Office / Home Office, also Department Office, Company Division / Branch, .... ie. a small organization/group, or unit of an org, with up to about 20 users, and of course also multiples of such autonomous units...

Posted: **Sun Apr 26, 2020 3:29 pm**

And then each of those users gets a 10G connection to their desktop system, and there is a 100G branch-to-branch connection to link them all the HQ?
Ok, I get it! I agree with you there must be a huge market for devices like that. Everyone in the SOHO business would want or need some.

Posted: **Sun Apr 26, 2020 3:36 pm**

I agree with you there must be a huge market for devices like that. Everyone in the SOHO business would want or need some.

What @vortex seems to be implying is that such devices should come with SOHO price tag - and I fully agree at this point. Absolute number (and currency symbol) varies from market to market, for me acceptable price is 2 to 3 digit number expressed in Euro ... however @vortex so far quoted numbers which are one order of magnitude higher ... which rule them out of SOHO segment at least where I live ... most certainly from H, but from S as well.

Posted: **Sun Apr 26, 2020 4:16 pm**

I can't imagine any SOHO needing 10G, much less 100G.

Posted: **Sun Apr 26, 2020 4:38 pm**

I can't imagine any SOHO needing 10G, much less 100G.

InHouse Distributed High Performance Computing (HPC) using Accelarator Cards for GPGPU computing via OpenCL/CUDA on true workstations (and servers), Games Dev, SciFi-Movie Dev, PCIe3/4 usage for realtime system backups to NAS system with NVMe SSDs, also crypto research (for example code cracking), ...
The said 2x 100G ports on the switch are intended to attach 2 HPC servers to the switch, so that the switch can serve the 20 clients with 10G each over its 10G ports...
There is and will always be demand for HPC. Of course these are special applications, simply HPC applications, not the usual Average Joe applications

Posted: **Sun Apr 26, 2020 4:46 pm**

I can't imagine any SOHO needing 10G, much less 100G.
InHouse Distributed High Performance Computing (HPC) using Accelarator Cards for GPGPU computing via OpenCL/CUDA on true workstations (and servers), Games Dev, SciFi-Movie Dev, PCIe3/4 usage for realtime system backups to NAS system with NVMe SSDs, also crypto research (for example code cracking)...
The said 2x 100G interfaces on the switch are intended to attach 2 HPC servers to the switch, so that the switch can serve the 20 clients with 10G each over its 10G ports...
There is and will always be demand for HPC. Of course these are special applications, simply HPC applications, not the usual Average Joe applications

That's fine. None of that is typically done in the home or small offices.

Posted: **Sun Apr 26, 2020 4:53 pm**

That's fine. None of that is typically done in the home or small offices.

But it is done in small closed research groups of up to that size. These are the core users, the horsepower of an org, excluding the foot folks like secretaries etc (1G suffices for these)

Posted: **Sun Apr 26, 2020 5:52 pm**

I can't imagine any SOHO needing 10G, much less 100G.

If people have 40G TB3, it is normal to want 40G Ethernet.

Maybe some professionals would step up to 100G.

Posted: **Sun Apr 26, 2020 5:58 pm**

What @vortex seems to be implying is that such devices should come with SOHO price tag - and I fully agree at this point. Absolute number (and currency symbol) varies from market to market, for me acceptable price is 2 to 3 digit number expressed in Euro ... however @vortex so far quoted numbers which are one order of magnitude higher ... which rule them out of SOHO segment at least where I live ... most certainly from H, but from S as well.

It was not me who called a 4k-11k router SOHO.

Actually, the price mentioned was 1k, which could be acceptable for professional or business use.

Posted: **Sun Apr 26, 2020 7:48 pm**

It was not me who called a 4k-11k router SOHO.

No, you didn't do it directly. However, you're very vocal at claiming 40Gbps is a minimum for contemporary SOHO[*] ... I'm just extrapolating that over current gear prices.

[*] I'm not talking about special uses such as set forward by @mutluit ... and even in HPC world most of the time super high bandwidth is not necessary, the uses he described demand low latency (which slightly correlates with high thtoughput, but not linearly) and for those uses there are specialized solutions (such as infiniband for communication between compute nodes or fibrechannel for attaching SAN). Those solutions come with price tags in the same order of magnitude as 40/100 Gbps ethernet.
Rather I'm talking about typical office ... which can as well be some engineers developing new products ... and in such offices, 40Gbps or 100Gbps is not a norm yet, 1Gbps mostly does it, perhaps small fraction of devices need 10Gbps.

Posted: **Sun Apr 26, 2020 7:50 pm**

That's fine. None of that is typically done in the home or small offices.
But it is done in small closed research groups of up to that size.

Research groups can't be considered typical SOHO groups. Hence they don't use SOHO gear ... even if it's the same devices, when they use them they are instruments

Posted: **Sun Apr 26, 2020 8:10 pm**

It was not me who called a 4k-11k router SOHO.

No, you didn't do it directly. However, you're very vocal at claiming 40Gbps is a minimum for contemporary SOHO[*] ... I'm just extrapolating that over current gear prices.

[*] I'm not talking about special uses such as set forward by @mutluit ... and even in HPC world most of the time super high bandwidth is not necessary, the uses he described demand low latency (which slightly correlates with high thtoughput, but not linearly) and for those uses there are specialized solutions (such as infiniband for communication between compute nodes or fibrechannel for attaching SAN). Those solutions come with price tags in the same order of magnitude as 40/100 Gbps ethernet.
Rather I'm talking about typical office ... which can as well be some engineers developing new products ... and in such offices, 40Gbps or 100Gbps is not a norm yet, 1Gbps mostly does it, perhaps small fraction of devices need 10Gbps.

I never said 40G is a minimum for SOHO, just that it is normal if professional users want that at home.

I did not buy a 10G NAS because I had 20G TB2. Now I would not buy less than a 40G NAS.

Posted: **Sun Apr 26, 2020 8:29 pm**

@vortex: do us a favour and flatter yourself for not being a typical home or small office user. Stop claiming that your desires are typical SOHO requirements ... because they're not.
You can call yourself a very advanced user who wants performance which is currently meant for HPC facilities and data centres. There's nothing wrong with that

Posted: **Sun Apr 26, 2020 9:35 pm**

I did not say 40G is a typical SOHO requirement, just a normal one.

40G is not HPC level. It is Mac level.

Posted: **Sun Apr 26, 2020 9:38 pm**

I did not say 40G is a typical SOHO requirement, just a normal one.

40G is not HPC level. It is Mac level.

It absolutely is not.

Posted: **Sun Apr 26, 2020 9:40 pm**

It absolutely is not.

Macs have had 40G TB3 for years.

No lesser NAS will be accepted here.

Posted: **Sun Apr 26, 2020 9:44 pm**

It absolutely is not.
Macs have had 40G TB3 for years.

No lesser NAS will be accepted here.

https://media1.tenor.com/images/196b84e ... d=14058849

Posted: **Sun Apr 26, 2020 9:59 pm**

vortex, the "everyone needs 40G/100G at home" forum troll.

PC is also worth the money and takes up space in the rack. It’s better to pay for Mikrotik some money for the SSD option or buy an SD card. You will not deny that the presence of an SD card in the CCR1000 is a necessary option.

A bog-standard PC is far cheaper than a CCR. We have several CCR, no SD cards at all.

Posted: **Sun Apr 26, 2020 10:03 pm**

I NEVER said EVERYBODY needs 40G. Read properly.

Posted: **Sun Apr 26, 2020 10:25 pm**

40G is the new 10G.

When I was asking for home routers capable of 10G switching some years ago, it was professional level. Now not only is 10G cheap, some people even have 10G WAN at home.

Why would you buy a couple of 25G cards when they are not much cheaper than 40G? Workstations and NAS that can take them are not cheaper.

I did not buy a 10G NAS because it was slower than TB2. I will similarly avoid buying a 25G NAS because it is slower than TB3. If I really needed a serious NAS, I would want 40G (with NVMe slots of course).

Vortex ... please do stop treating us as kids. All the time you write ... MORE, MORE, MORE ... SOHO needs more ... I need MORE ... it's obvious what you write about and ask for no matter what words you don't use.

What Mac has network card @40Gb for years?

Posted: **Sun Apr 26, 2020 10:29 pm**

I said 40G TB3. Direct attached storage is popular on Macs.

I did not really need a NAS, so I avoided 10G.

Posted: **Sun Apr 26, 2020 10:35 pm**

And what is the implication of that? What does it mean for Mac's networking?

Posted: **Sun Apr 26, 2020 10:40 pm**

It was 10GbE time, I could have added it via TB1.

Now I use a PC, I could add 40GbE via TB3 (some cards also work with Macs).

Some people with 2012 MacPros were asking about 100G cards in 2018 (they exist, but the computer is not that fast).

I stopped caring about Macs when Apple only bothered making expensive toys. I think later MacPros can only do 40GbE via TB3, so Mac-level was quite accurate.

Now, allocating 1200 MHz more of unlicensed spectrum seems extremely excessive.

Technical agencies should not be run by lawyers.

Posted: **Tue Apr 28, 2020 9:30 pm**

vortex, please stop spamming

Posted: **Wed Apr 29, 2020 9:20 am**

I _really_ wish it was Marvell Octeon TX2, but this platform looks like Annapurna Labs ARM SoC's.

Possibly Mikrotik have combined Annapurna Labs ARM core's with Marvell Prestera Switch ASIC's, where the AL chip does the initial packet pass (slow path) then the fast-path is handled by the Marvell ASIC. This would allow them to scale port counts independently of the ARM core and would also allow them to scale the platform from low-cost designs (CCR2004) with just an Annapurna Labs ARM processor, to higher-cost designs (CCR2016) with a Annapurna Labs core and a Marvell Prestera switch ASIC.

we have a winner

https://mikrotik.com/product/ccr2004_1g_12s_2xs

CCR2004-1G-12Splus2XS-1588077298.jpg

Posted: **Wed Apr 29, 2020 9:36 am**

What is my prize ?

Posted: **Wed Apr 29, 2020 5:34 pm**

The 98PX1012 https://www.marvell.com/products/switch ... ra-px.html

Posted: **Wed Apr 29, 2020 9:16 pm**

It was 10GbE time, I could have added it via TB1.
Now I use a PC, I could add 40GbE via TB3 (some cards also work with Macs).
[...]

Following this bus speed logic I could say hurr durr 200G is basic b*tch gaming PC level because I can buy this totally consumer network card: https://store.mellanox.com/products/mel ... hs-r6.html and put in my totally consumer 4th gen Ryzen gaming PC.

Just because TB3 allows for 40G on paper doesn't mean there are actually QSFP+ interfaces for TB3 that realistically reach 40G speeds. Best I've seen was dual 10G because yeah I was looking for some op network card for laptop with TB3.

I built 10G backbone at home years ago (using CCR1009-PC) because I always was pursuing this cutting edge storage networks and I imagined it's gonna solve all my problems. Then I got two Intel X710-DA4 4x10G network cards for my both PCs to get this supper cutting edge storage network speeds.

And guess what. It's worth noting:

You don't use ROUTER for storage networks in the first place - latency introduced by routing is killing all the performance. All serious high performance storage networks are L2.
Actually iSCSI/NFS are kinda meh since ethernet is kinda meh and real high performance storage networks use FC or Infiniband anyways so whatever
I actually never managed to saturate single 10G link, not to mention 4 of them because I never managed to get fast enough, redundant RAID storage to saturate that. My 10 bay NAS was bottlenecking at around 600 MB/s read and 200-something write because it was RAID6.

In this thread we're talking about routers so in that case 10G/25G is already quite a good deal. The primary problem of CCR1009 for me is that it actually DOESN'T manage to reach full 10G speed on single TCP tunnel when you don't use jumbo frames and fasttrack so it was kinda meh.

But yeah 40G/100G affordable SOHO (by HO I actually mean there bearded enthusiast lab network) switch would be nice. That said at the end of the day you're pursuing all that speed and it's SOOO COOOL but at some point you realize that... what's even point of that if you have 120/20 mbps DSL internet which can be easily handled by hAP ac²... And when you want to download some file from your home server when you're somewhere else it takes forever to download because your DSL has terrible upload and faster internet is only available for big companies and costs kidney every month.

My company has virtualization cluster on vSphere/ESXi that consists of 8 virtualization hosts and two storage arrays. We use gigabit backbone. Those two storage arrays are connected with 2G bondings each. When I suggested to "maybe get at least one 10G backbone switch since we're moving to new office and building new servers farm anyways???" I received reply "and what's the point of that? Why would you need more than 1G to server if all computers have only 1G network cards anyways? And 90% of people don't even use cable just crappy overloaded wifi that bottlenecks at 40mbps. We don't use storage servers that often and even if 2 or 4 people will use storage server at once then 250mbps is more than enough".

Backups are automated and scheduled so they never run all at once but one after another.

They don't need to be fast since they're copied from incremental snapshots so there's no risk that data will get outdated and usually it's not really that big. It's not a problem if backup takes even several hours since VMs are backed up once every few days. We wouldn't have space for more frequent backups anyways since arrays are not that big.

Also we have dual 100/100 symmetric internet links so It won't really saturate anything either. And we're actually not even "small" company in meaning of SOHO definition. If my company would have spare funds they're 100x more likely to get higher-end Cisco 1G switches and routers than buy 10G or 100G MikroTik ones because "who needs 10G lul". Raw speed is rarely top priority. Especially above 10G.

---

All in all it's cool and swag and nice flex but at the end of the day nobody needs that. I'm totally gonna get that CCR2004 because it's cool but I'm not gonnay say I actually need that. It's just weird flex and I want to pair it with my two CCR317s, replacing CCR1009 xD

Posted: **Wed Apr 29, 2020 11:52 pm**

@lapsio: nicely written. Too bad @vortex won't get past the first paragraph ...

Posted: **Thu Apr 30, 2020 1:06 am**

You CAN add 40GbE with TB3. You just need a TB3 enclosure for a card.

I am a home user. I don't want a separate switch.

40G for LAN has nothing to do with WAN.

You can saturate 40G with NVMe.

What the average office does is not representative of power use.

A professional user who would spend 1k on a 100G card would not like AM4.

I would not pay for more than 40G now and would try to avoid small socket.

I don't have a network or computer lab.

I did want to buy a 12-core Xeon laptop in 2014 but it was not really mobile.

Posted: **Tue May 05, 2020 7:56 am**

So does this thing have a switch chip? Is that what the port extender is? Can the ports be put into switch groups for wire speed switching?

Posted: **Tue May 05, 2020 9:20 am**

No. The port extender is just that and does no local switching (as the 'bridging' numbers indicate).

Posted: **Tue May 05, 2020 1:01 pm**

It is a bit unclear to me how useful it is to have 170Gbps worth of ports connected via 2x25Gbps bottleneck, and without mutual switching capability.
With switching I could see that (the device could be used as a combined router/switch) and maybe it could even do "hardware routing" similar to an L3 switch.
But with overall performance limited to ~35Gbps I don't see the point of all those 10G and 25G ports...

Posted: **Tue May 05, 2020 2:17 pm**

Join the club. I'm not getting the idea behind this box either. It's woefully underpowered for the IO it has.

Posted: **Tue May 05, 2020 2:52 pm**

Maybe we should just consider it a replacement for the CCR1009 and put it in the same performance category, with a little more peak performance.
The SFP+ slots should be primarily populated with 1Gbit/s interfaces and the occasional 10 Gbit/s.
Of the 25Gbit/s interfaces only a single one should be in active use, the other one maybe for fallback.
Then it could work out OK...

Posted: **Tue May 05, 2020 2:55 pm**

It is a bit unclear to me how useful it is to have 170Gbps worth of ports connected via 2x25Gbps bottleneck, and without mutual switching capability.
With switching I could see that (the device could be used as a combined router/switch) and maybe it could even do "hardware routing" similar to an L3 switch.
But with overall performance limited to ~35Gbps I don't see the point of all those 10G and 25G ports...

Basically, i'd treat it as a router with a 25G Uplink (LAG) to the core switches and 12x 1G conntections, that can scale up to 10G.
If you need more throughput, you'll have to wait for the CCR2016 version. Assuming the throughput scales linearly, you'd have 140Gbit routing capacity @ 1518 Bytes - close enough to wirespeed.

Personally, I see it as an upgrade for my current CCR1016-12S-1S+

Posted: **Tue May 05, 2020 3:16 pm**

You're not supposed to run all of your ports full all of the time.

I also am not sure of the advantages of the PIPE over a switch chip. Hopefully someone from Mikrotik can explain.

Posted: **Tue May 05, 2020 3:34 pm**

If you need more throughput, you'll have to wait for the CCR2016 version. Assuming the throughput scales linearly, you'd have 140Gbit routing capacity @ 1518 Bytes - close enough to wirespeed.

Personally, I see it as an upgrade for my current CCR1016-12S-1S+

Yep, it does pretty much kill off the CCR1016-12S-1S+. But a CCR2016 version of this CCR2004 doesn't seem to be on the horizon, although it would make a lot more sense than this one. The only CCR2016 hinted at in the 7.0 beta 5 binaries is a CCR2016-1G-12xs-2xq, which with 12 25G and 2 100G ports is only slightly less underpowered than this one...

Posted: **Tue May 05, 2020 5:33 pm**

CCR2004-1G-12S+2XS is our router with the most powerful single-core performance so far. It provides incredible results in single tunnel (up to 3.4 Gbps) and BGP feed processing.

Maybe it is the answer.

Posted: **Tue May 05, 2020 5:38 pm**

It is a bit unclear to me how useful it is to have 170Gbps worth of ports connected via 2x25Gbps bottleneck, and without mutual switching capability.
With switching I could see that (the device could be used as a combined router/switch) and maybe it could even do "hardware routing" similar to an L3 switch.
But with overall performance limited to ~35Gbps I don't see the point of all those 10G and 25G ports...
Basically, i'd treat it as a router with a 25G Uplink (LAG) to the core switches and 12x 1G conntections, that can scale up to 10G.

That is what I meant. Fortunately, I have no need for such a device. In my networks, a device like the CCR1009 is plenty powerful enough.
(internet connections to homes and small/medium offices here rarely exceed 1Gbps and I use a separate switch behind the router)

Posted: **Tue May 05, 2020 6:13 pm**

It is a bit unclear to me how useful it is to have 170Gbps worth of ports connected via 2x25Gbps bottleneck, and without mutual switching capability.
With switching I could see that (the device could be used as a combined router/switch) and maybe it could even do "hardware routing" similar to an L3 switch.
But with overall performance limited to ~35Gbps I don't see the point of all those 10G and 25G ports...

ccr2004 is not a switch is a router

you don't need to move 10 gbps to justify the need of 10gbps interface, for example is enough the need to move 1.5gbps

in fact when you need to move 10gbps yo have to go for next speed to avoid bottlenecks

same case ported to 25gbps: 25gbps is useful to move 12gbps
same case ported to 40gbps: 40gbps is useful to move 27gbps
same case ported to 100gbps: 100gbps is useful to move 42gbps

the objective is to avoid interface to be the bottleneck

some times you don't expect to fill all access interfaces at the same time you just want to be ready to hold up an spike on traffic on some of them

in the same manner sometimes when you have 2 up-link interfaces do not necessarily you gonna full them simultaneously, in some case 2 up-link interfaces are useful to implement redundant topology without filling them at the same time

because that to do good purchases you have to read carefully the datasheet, check block diagrams, and performance figures

Posted: **Thu May 07, 2020 1:34 am**

Id take this in an outdoor box in a second, with less ports and DC POE.

Posted: **Thu May 07, 2020 11:35 pm**

I do have a use case personally. With WiFi6 (802.11ax), useful speeds will exceed 1 gigabit/sec. And if you have > gigabit fiber internet (dreaming is allowed) that's 2 ports in use.
Then a new NAS would likely also interface at > gigabit speeds. And hey, no separate switch needed which is a bonus if space is limited.

One question for the experts though, will this router also interface @2.5 and 5 gigabit?

Posted: **Thu May 07, 2020 11:46 pm**

One question for the experts though, will this router also interface @2.5 and 5 gigabit?

If you put an S+RJ10 in it, I see no reason why not. My CCR1036 did (when I used copper 10G).

Posted: **Thu May 07, 2020 11:48 pm**

I do have a use case personally. With WiFi6 (802.11ax), useful speeds will exceed 1 gigabit/sec. And if you have > gigabit fiber internet (dreaming is allowed) that's 2 ports in use.
Then a new NAS would likely also interface at > gigabit speeds. And hey, no separate switch needed which is a bonus if space is limited.

Due to the architecture of this thing, you better get your separate switch (like CRS326) when you want that high-performance LAN.
It is exactly for purposes like that where it would have been more useful when a switch chip had been used instead of that port extender.
I could understand why some users would want to use several of the SFP+ ports to connect fast equipment, and then maybe a 1Gbps switch for slower stuff, but the data path between the 10 Gbps ports will be really limited by everything going via the CPU and its 35Gbps max data rate.

Posted: **Fri May 08, 2020 12:49 am**

@pe1chl I had read your comments earlier in the thread and I strongly agree. So many ports are begging for a switch chip to be more useful since the port-IO vastly exceeds the routing capabilities. A bit of a wasted opportunity. A cut down model (read: cheaper, maybe passively cooled and smaller) with fewer ports would make more sense.
Alternatively, a router that has at least 2 SFP+ ports, one for a future proof WAN-uplink and one for a LAN-uplink to connect with a 10 gigabit switch would be a good solution too.
It can be paired with https://mikrotik.com/product/crs305_1g_4s_in in the study, living room, ... for silent operation.

@mbovenka exactly what I would need to connect the new breed of WiFi6 gear. Thanks!

Posted: **Fri May 08, 2020 7:01 pm**

Are these chips from 'Annapurna labs' going to have a Long Term Support? or Another 'TILE' Incident with No New Features / Development in Future.

Please share Roadmaps and Support Plans so the investment can be secure.

Cheers

Posted: **Fri May 08, 2020 7:20 pm**

Are these chips from 'Annapurna labs' going to have a Long Term Support? or Another 'TILE' Incident with No New Features / Development in Future.

They are an Amazon company, and Amazon uses their stuff (de Graviton & Graviton2) in AWS. I think we're pretty safe. Not to mention everybody and their dog uses the lower-end chips in things like NAS boxes and the like. They are a far cry from Tilera (for which MikroTik was pretty much the only outside customer I know of).

Posted: **Fri May 08, 2020 7:43 pm**

In principle it is totally unimportant what chips MikroTik use and how long they are supported.
It is THEIR task to release the RouterOS for the chip, and for us as users it does not matter at all whether the architecture is X86, MIPS, TILE, ARM, ARM64 or whatever.
Of course they should try to avoid the situation where there are so many architectures to worry about that it becomes a major burden during release builds, but the point of "supporting another architecture besides the one on which development started" has been passed long ago, and going from 7 to 8 architectures really should not matter much.
(with the Linux kernel being between the hardware and the higher level software, and that software all being written in architecture-independent languages anyway)

Besides, with every new architecture added the chance is reduced that another new chip family will be another new architecture

Posted: **Fri May 08, 2020 8:12 pm**

I would say yes and no.

No, it doesn't matter if it's MIPS, PPC, ARMv7, ARMv8 etc.
Yes, it does matter when the number of interfaces, power consumption, form-factor or performance is drastically changed [because of a SoC-change].

If your network rely on a device like that has a certain number of interfaces, performance and memory, and that model suddenly vanish because of an architecture-change, then you have a problem. That's why Cisco is so successful in enterprises. For example Catalyst 2960 has been around for over 15 years now in various flavors. The stuff on the inside has changed a lot but the basic features stay the same.

Posted: **Fri May 08, 2020 8:29 pm**

If your network rely on a device like that has a certain number of interfaces, performance and memory, and that model suddenly vanish because of an architecture-change, then you have a problem.

Oh but that happens without architecture-change! The CCR1009 line originally had 8 ethernet ports and 2 SFP slots. The first 4 ethernet ports were on a switch chip.
In the later CCR1009 models the switch chip was removed, every ethernet port had its own connection to the TILE, but as there now was one connection too few, the first ethernet port became a COMBO port with the second SFP.
That of course would cause problems for those relying on 10 ports, and also the removal of the ethernet chip can have impact.
(I have one installation where the CCR1009 is really only a router and the new design would be better, and another one where the CCR1009 is in a datacenter serving two servers and the 4-port switch is handy to build a fast local network that does not load the CPU)

So instead of making this an architecture- or chip based problem, it is more like a product specification problem. Probably Cisco have a more rigid (and therefore sometimes more restrictive) featureset of products. But they do serve a different market too.

Posted: **Fri May 08, 2020 9:08 pm**

I have been running the math and this device does work out somewhat. In my LAN, I only have a 10 gigabit server, a 2.5 gigabit wireless backhaul (which realistically can go up to 1.4 gigabit) and a 10 gigabit desktop. All the slower 1 gigabit legacy gear goes to a CRS326-24G-2SplusRM with would uplink with the router in this topic at 10 gigabit.

I cannot really see another cost effective way to future proof for more than 1 gigabit internet on one end and at least four 10 gigabit LAN interfaces.
If I drop the future proof internet requirement, I could just keep my current router and combine the CRS326 with this bad boy CRS309-1G-8S+IN.

Choices, choices...

Posted: **Fri May 08, 2020 10:07 pm**

They are an Amazon company, and Amazon uses their stuff (de Graviton & Graviton2) in AWS. I think we're pretty safe. Not to mention everybody and their dog uses the lower-end chips in things like NAS boxes and the like. They are a far cry from Tilera (for which MikroTik was pretty much the only outside customer I know of).

Amazon has a great team for In-House Development, which Mikrotik is not very good at. We can also see this with TILE series. Facebook developed and used it for many great things, but Mikrotik never got to that point, just generic usage as a router for a long time. Slow Development and No New Features.

In this era, We also need Very High PPS Handling with Multiple High Speed Physical Ports. Just my two cents. DDoS is a COMMON thing now.

Also, TILE restricted the quick development and new features (OLD Kernel) for x86 all this time!

Posted: **Fri May 08, 2020 11:49 pm**

Comletely agree with the above comment.
dDOS is a very common thing now and no RouterOS device is capable of handling more than a million pps even in a plain routing setup.
x86 is useless with the new and modern server hardware and CHR is simply a joke when it comes to performance.
We need performant Core Router hardware which can handle high pps or a proper v7.0 x86-64 version for us to build one.

Posted: **Sat May 09, 2020 4:03 am**

Got one of these today and it comes with 6.46.3, but thats all.. arm64 cant be downloaded any newer. lol.

Posted: **Sat May 09, 2020 12:04 pm**

So instead of making this an architecture- or chip based problem, it is more like a product specification problem. Probably Cisco have a more rigid (and therefore sometimes more restrictive) featureset of products. But they do serve a different market too.

That is true. Sometimes I suspect that Mikrotik finds a good SoC on the market, then creates the product of it, regardless of the usage or market research.

For example, the RB1100AHx4 is passive, has both 48V and dual AC power supplies and a great candidate for a lot of installations. But it has no SFP-cages at all for some reason - so it can't be used where it should be logical to use one.

Posted: **Sat May 09, 2020 12:21 pm**

Sometimes I suspect that Mikrotik finds a good SoC on the market, then creates the product of it, regardless of the usage or market research.

I agree with that! This is apparently how it works.
Of course, this is partly related to the market segment they try to operate in.
When your shiny new SoC has everything you need except USB port, you can choose between releasing the product without USB port or adding circuitry to provide USB (assuming the SoC has some other unused bus connection that could be used to attach a USB controller).
However, that will raise the cost of the final product and put off some prospective customers. On the other hand, other customers may be put off by the lack of USB.
So on many of the newer products we see useful features (we have become accustomed to) suddenly missing (like USB port, beeper, button, voltage/temp sensing, and of course the classic LCD touchscreen)

Posted: **Mon May 11, 2020 9:57 am**

Are these chips from 'Annapurna labs' going to have a Long Term Support? or Another 'TILE' Incident with No New Features / Development in Future.

Please share Roadmaps and Support Plans so the investment can be secure.

Cheers

Two points:

1. It's ARM based and has mainline kernel support.
2. It's $600 RRP - Keep that in perspective, it is exceptional value!

Posted: **Mon May 11, 2020 11:09 am**

2. It's $600 RRP - Keep that in perspective, it is exceptional value!

Of course keep in mind that in most cases you will have to add about the same money for SFP modules...

Posted: **Mon May 11, 2020 3:06 pm**

2. It's $600 RRP - Keep that in perspective, it is exceptional value!
Of course keep in mind that in most cases you will have to add about the same money for SFP modules...

Eh, depends. $24 for 10G, $59 for 25G

Posted: **Mon May 11, 2020 9:27 pm**

That is what I meant. Fortunately, I have no need for such a device. In my networks, a device like the CCR1009 is plenty powerful enough.
(internet connections to homes and small/medium offices here rarely exceed 1Gbps and I use a separate switch behind the router)

I think more people would want a couple 40G ports at home than 12x 10G.

Posted: **Tue May 12, 2020 3:37 am**

vortex.

Last call ... PLEASE DO STOP throwing everywhere comments that 10+ Gb ports are a common need at homes ....

Posted: **Tue May 12, 2020 3:49 am**

I said nothing about 10G+ being requested by most home users.

Posted: **Tue May 12, 2020 4:23 am**

I think more people would want a couple 40G ports at home than 12x 10G.

vortex ... please do not treat us as fools and please do not lie ... once more i see such a comment and you will be banned

Posted: **Sat Jul 25, 2020 6:46 pm**

I extrapolated these codes out:
Code: Select all
ccr2004-2g-2xc-2s+/r2-60	4 Core, 2x 1G, 2x 10G/25G SFP28, 2x 10G SFP+, 2.4Ghz WiFi, 802.11ay 60Ghz
It all seems possible, but not completely logical.
imagine a device - the ccr2004-2g-xxx - which is compact. as compact as a powerbox, maybe thicker. has decent fiberoptic options for high speed fixed backhaul.
2.4GHz wifi for easier management if it is mounted to a tower / pole.
and has 2 independent .11ay radios, each with 180 degree FOV - using 3 RF stages each, similarly to the current APx3.
because this is what this looks like.
and i would buy them on first sight.
I think you are on the right track here, and this could be a device in a square plastic enclosure that has connectors and a mounting bracket on the bottom to act as a Terragraph node.

I have no real idea, but it makes sense and the speculation is fun

this now has a way prettier name in 7.1b1: CCR2004-2G-2S+2XS-60ayD
(also a reference to a lovely addition to the crs3 series is in the image, but i don't want to spoil all the fun)

Posted: **Mon Jul 27, 2020 2:42 pm**

How do you find it, wanna see by my eyes.
?

Sent from my Moto Z3 Play using Tapatalk

Posted: **Mon Jul 27, 2020 10:49 pm**

(also a reference to a lovely addition to the crs3 series is in the image, but i don't want to spoil all the fun)

Found it.. Nice

Posted: **Tue Jul 28, 2020 12:05 am**

How do you find it, wanna see by my eyes.

npk-tools, and some traditional unix utils: find, strings, sort, grep.

Posted: **Tue Jul 28, 2020 12:30 am**

How do you find it, wanna see by my eyes.
npk-tools, and some traditional unix utils: find, strings, sort, grep.

I extracted find all the info that show on first page but not luck to find any reference crs3

Sent from my Moto Z3 Play using Tapatalk

Posted: **Tue Jul 28, 2020 10:15 am**

If I remember correctly it is in a file called 'net'.

Posted: **Fri Dec 04, 2020 5:33 pm**

No new findings in the new beta 3[emoji848][emoji848]?

Sent from my Moto Z3 Play using Tapatalk

Posted: **Wed Dec 23, 2020 1:47 am**

No new findings in the new beta 3[emoji848][emoji848]?

Sent from my Moto Z3 Play using Tapatalk

Maybe some! ->

ccr2004-16g-2s+
ccr2116-12g-4s+
CRS326-4C-20XG-2Q+
CRS418

Posted: **Wed Dec 23, 2020 1:49 am**

No new findings in the new beta 3[emoji848][emoji848]?

Sent from my Moto Z3 Play using Tapatalk
Maybe some! ->

ccr2004-2g-2xc-2s+
ccr2004-2g-2xc-2s+/r2
ccr2004-2g-2xc-2s+/r2-60
ccr2004-16g-2s+
ccr2004-1g-12s+2xs
ccr2016-1g-12xs-2xq
ccr2116-12g-4s+

Guess this are the same discovered previously
I think the last one was a typo correct?

Sent from my Moto Z3 Play using Tapatalk

Posted: **Wed Dec 23, 2020 2:12 am**

No new findings in the new beta 3[emoji848][emoji848]?

Sent from my Moto Z3 Play using Tapatalk
Maybe some! ->

ccr2004-2g-2xc-2s+
ccr2004-2g-2xc-2s+/r2
ccr2004-2g-2xc-2s+/r2-60
ccr2004-16g-2s+
ccr2004-1g-12s+2xs
ccr2016-1g-12xs-2xq
ccr2116-12g-4s+
Guess this are the same discovered previously
I think the last one was a typo correct?

Sent from my Moto Z3 Play using Tapatalk

some new ->
ccr2004-16g-2s+
ccr2116-12g-4s+
CRS326-4C-20XG-2Q+
CRS418

Posted: **Wed Dec 23, 2020 2:14 am**

Thanks

Sent from my Moto Z3 Play using Tapatalk

Posted: **Tue Mar 09, 2021 3:24 pm**

Some references to CRS5xx series here...

ali@bf-ali-ubuntu:~/routeros-software/7.1b4/mipsbe/_routeros-7.1beta4-mipsbe.npk.extracted/squashfs-root/lib/modules/5.6.3/drivers/net$ strings * | grep -i crs
is_crs_type
music: CRS212
is_crs_type
crs332
crs312
crs354
crs518
crs326hb
crs504
crs332
crs312
crs354
crs518
crs326hb
crs504

Posted: **Wed Jun 09, 2021 7:35 am**

And in RouterOS 7.1beta6

**** ARM64

crs520-4xs-16xq
crs419-hk
crs419w-hk
crs419-hk
crs419w-hk

ccr2004-16g-2s+
ccr2004-1g-12s+2xs
ccr2004-1g-2xs-pcie
ccr2016-1g-12xs-2xq
ccr2116-12g-4s+
ccr2004-2g-2xc-2s+
ccr2004-2g-2xc-2s+/r2
ccr2004-2g-2xc-2s+/r2-60
ccr2004-16g-2s+

**** ARM

crs520-4xs-16xq
pdx_crs309_sfpp
pdx_crs317_sfpp
pdx_crs326
pdx_crs319_eth
pdx_crs328_sfpp
pdx_crs305_gpio_mux
pdx_crs419_gpio_mux
pdx_crs319_gpio_mux
pdx_crs328_gpio_mux

ccr2004-16g-2s+
ccr2004-1g-12s+2xs
ccr2004-1g-2xs-pcie
ccr2016-1g-12xs-2xq
ccr2116-12g-4s+
ccr2004-2g-2xc-2s+
ccr2004-2g-2xc-2s+/r2
ccr2004-2g-2xc-2s+/r2-60

**** MIPSBE

crs332
crs312
crs354
crs518
crs326hb
crs504
crs510
crs332
crs312
crs354
crs518
crs326hb
crs504
crs510

I also found traces of Marvell Armada 7040 and Qualcomm IPQ5018 as well as 6018 chipset support, which do not align with any current products

Posted: **Wed Aug 04, 2021 3:40 pm**

I also found traces of Marvell Armada 7040 and Qualcomm IPQ5018 as well as 6018 chipset support, which do not align with any current products

Well it turns out the Armada 7040 based product is the RB5009.

We still have plenty more surprises to come !

Posted: **Wed Aug 04, 2021 3:47 pm**

The new ccr2004 is not based on that same chips?

Posted: **Wed Aug 04, 2021 7:22 pm**

The new ccr2004 is not based on that same chips?

No, it is an Annapurna Labs (Amazon) AL32400 clocked at 1.7Ghz. This is a 64bit ARM processor.

The same processor is used in the existing CCR2004-1G-12S+2XS as well.

Posted: **Wed Aug 04, 2021 7:41 pm**

The new ccr2004 is not based on that same chips?

No, it is an Annapurna Labs (Amazon) AL32400 clocked at 1.7Ghz. This is a 64bit ARM processor.

The same processor is used in the existing CCR2004-1G-12S+2XS as well.

Oh my bad I made a confusion

Posted: **Tue Aug 24, 2021 6:47 pm**

traces of CCR2116 also show up in the defconf scripts...

:if ($board->"model"~"CCR2116") do={

so it's near...

Posted: **Tue Aug 24, 2021 7:24 pm**

"Extreme Performances" ARM64 Series:
CCR-eOW-12x100G-36x25Gw
CCR-eOW-1x25Gw-2x10G
CCR-eOW-1Gw-1G

ARM64:
ccr2116-12g-4s+ (al64v3)
crs520-4xs-16xq (4x 25gbit + 16x 100gbit)

Posted: **Wed Aug 25, 2021 4:29 am**

Well they have been a long time coming, and they are still not here yet....

Posted: **Wed Aug 25, 2021 3:38 pm**

"Extreme Performances" ARM64 Series:
CCR-eOW-12x100G-36x25Gw
CCR-eOW-1x25Gw-2x10G
CCR-eOW-1Gw-1G

ARM64:
ccr2116-12g-4s+ (al64v3)
crs520-4xs-16xq (4x 25gbit + 16x 100gbit)

The top3 model number look like bullocks to me. The bottom two look like they could be legit

Posted: **Wed Aug 25, 2021 3:41 pm**

Ok, on "system.x3" file inside software the MikroTik staff leave an easter egg for ethernetOverWater and Gigawater port (pipes with water inside...)

Posted: **Wed Aug 25, 2021 10:48 pm**

You could say they are poisoning the well of our future model knowledge...

Posted: **Tue Nov 23, 2021 2:29 pm**

CCR2116 is coming
https://help.mikrotik.com/docs/pages/di ... Version=55

Posted: **Sat Nov 27, 2021 11:57 pm**

ccr2216-1g-12xs-2xq

Maybe ccr2016-1g-12xs-2xq is not coming, and new version with ALC12B00??

MikroTik

Just going to leave this here...

Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...

Re: Just going to leave this here...