Which use cases for CCR2004-1G-2XS-PCIe ?

RouterOS General
21

Hi,

I thought I’d put my input on this question as I now own one of these routers!

My business has been a Cisco house for almost 10 years, and I’m now slowly converting my networking infrastructure to MikroTik devices, starting with replacing the core router for my server room with this new router.

I’ve tested this router in several of my servers and a few desktops with varying results.

RESULTS
IBM x3650 M4: WORKS BUT I DO NOT RECOMMEND - rebooting the PCIe router causes the server to reboot too. Also other strange behavior- AND no option for boot delay.

QCT QuantaPlex Node: WORKS WITH LIMITATIONS - These QCT Nodes have no built in Ethernet ports, so if you use up the PCIe slot for the MikroTik you loose your PCIe option for ethernet/SFP ports. (In my case the passthrough won’t work on my nodes as they run Windows Hyper-V and Windows doesn’t recognize them) HOWEVER these nodes have optional Mezz network cards you could use instead - other issue, no option for boot delay.

Supermicro 6017/6027: WORKS without passthrough - sure, do this if you want. The older Supermicro servers don’t have a boot delay option either, so mileage may vary for passthrough.

Asus Z270 PC board: works, but no boot delay option.

At the end of the day I’m going to pick any server that’s not an IBM x3650 M4 with an empty PCIe 8x or better slot and use it. The majority of my servers don’t have a boot delay option so PCIe passthrough is out of the question entirely, however that was not a selling point for me and my systems all have an abundance of connectivity options so it’s not a deal breaker either. The simple fact I can gain a single RU by ditching my Cisco router and putting a router into a PCIe slot that also supports 25Gb speeds is what makes this product so appealing to me.

The only downside when testing in a desktop was noise, after a few hours the fan on the MikroTik would gradually ramp higher and higher until it was a maximum speed, even with the router shutdown. For me it wont be a problem as this will be installed in a server, in a server room, where noise is just a fact, and I probably won’t even hear it among the servers.

CCR 2004 PCIe 10/10 would recommend

22

Is a boot delay required for the BIOS to see the card and then present it to the OS? or can Linux detect the card later when it comes online? (maybe as hotplug)
I remember the days when BIOS would not properly recognize certain disk devices, but still it was no problem mounting them later in Linux, one just could not boot from them.
Is it the same here? (maybe you want to netboot the server, that would be impossible of course)

Also it seems the card can reset the server, but it also would be interesting to know if the server will reset the card and if that can be inhibited.
I.e. when you reboot the server OS, will the RouterOS remain running or will it get restarted too?
When the server is ‘shut down’ in its remote management (ILO, DRAC, IMM) is it still possible to keep the card running (on ‘standby power’) or will it power down with the server?

23

As far as I know, the router needs to boot before the server in order for the PCIe virtual interfaces to be recognized in the *nix system. I haven’t extensively tested this yet as my servers are mostly Windows with the odd TrueNAS box, under which these interfaces DID appear, however they did not appear to be working as intended, presumably because they were not present at system boot? ** I do plan to test this further when I have my new SFP modules next week.

I remember the days when BIOS would not properly recognize certain disk devices, but still it was no problem mounting them later in Linux, one just could not boot from them.
Is it the same here? (maybe you want to netboot the server, that would be impossible of course)

Mounting them later is on my to-test list.

Also it seems the card can reset the server, but it also would be interesting to know if the server will reset the card and if that can be inhibited.
I.e. when you reboot the server OS, will the RouterOS remain running or will it get restarted too?
When the server is ‘shut down’ in its remote management (ILO, DRAC, IMM) is it still possible to keep the card running (on ‘standby power’) or will it power down with the server?

In my testing the router always shuts down with the server (I haven’t actually checked logs to see if it does a graceful shutdown or not). I imagine with the right server hardware and appropriate PCIe device power settings in BIOS that rebooting/shutting down it may be possible to keep the router up. ** I haven’t dug into that yet, but it’s also on my to-test list.

24

Ok good luck with the experiments, looking forward to see your findings.
Of course the use case would be to have such a card in a colocated server, connected to the internet only via a fast link on one of the fiber modules, and then plug a short cable between the ethernet port and the dedicated ILO/DRAC/IMM port on the server, then being to remotely access the server management via a VPN running on the card.

25

That’s an interesting scenario. That’s got me thinking about how to implement the management of the router on it’s MGMT port into my topology.

Using VPN/MGMT to remotely enter the MGMT LAN might be a useful design…but right now I have a completely segregated physical network for Server IMM (with it’s own unique challenges and topology).

26

Of course the solution for a home network or on-premises server would be different from a pure colocation server where you just have rack space and a raw internet connection.

27

Some updates.

Device rescan
It seems BSD does not have an implementation of PCI rescan that many linux distros have.

Boot delay/device delay
I haven’t found any of my servers having support for PCI device power on delay, or system startup delay.

Passthrough
Windows never sees the passthrough ports, and BSD doesn’t either.

Power
This device always reboots with the system (although I haven’t tested that since installing it for production, as the system it’s in was shut down for the first time in a year in order to install the card…and that system will likely stay up for the next year+)…hence its installation location.

Installation
So I’ve put this into a 2U SAN in my server room, and after about 10-15 hours of heavy configuration/testing it’s ready for production. I chose a server that has redundant power, backup power connected and alternate power sources so it shouldn’t have any issues keeping the router online. This system also has one of the best track records in my server room which was a big factor, having only been shut down a handful of times in 6 years.

This device has zero default configuration aside from basic interface setup for the passthrough emulation, so firewall setup was where most of my time was spent… but that wasn’t so bad, with my 14 years of experience running a web services business and the extremely helpful people here on this forum!

FYI - my use case in case anyone missed it - is as the core router in a server room that has a full height rack that’s quickly running out of space, and it handles traffic to dozens of servers, several switches and handles some VPN tunneling for access to servers remotely. So far it’s taking it like a champ and I haven’t lost any downtime, with the exception of about 6 hours of email due to misconfiguration.

28

i think the idea of introduce delay on server boot is to give time for ccr to boot before the OS boot

maybe enabling extended memory testing or something like that in bios can help

29

Yes I understand the reason for boot delay, and I tried extended memory testing but it didn’t change anything.

30

Try to add for delay also boot from network…

31

That will probably not work because boot from network is done as the last step, well after PCI initialisation.

32

Network boot did not change anything with the host systems recognizing this card on Linux machines.

33

Hmm…

Really like the concept :slight_smile:
What if: this card is placed in a ‘mining rizer’ and powered by a pico PSU.. agreed you don’t have the network interface pass-through to the OS.
But if you want to have a router with 2 sfp28 interfaces; without switching, because that’s done by another device…

The question is… Will the card power up in a riser…
Does anyone tried this already… I’ve did some searching but there is (for now) no evidence :slight_smile:

73’s

34

i am interested in that too
hopefully Mikrotik does not artificially limit this possibility

35

Hi!

Does anybody around here know if it possible to use this device with Synology rs822+?

Thanks in advance!

36

Sorry but I dont find any useful documentation on the net.
Do I need a working OS on the host machine to use the CCR?
I would like to use the CCR and the routeros in it, not using it a passthrough card.

Or I can have a box with any OS installed and just power it up and then enter on the winbox ?

Really sorry to have revived two posts but I an without answers.

37

Not much is known about this card… I also asked once if it has to be in a PCIe bus or if you could plug it into an extender card and supply only power.
No answer.

38

I own one of these, but it´s currently not in use. My use case would have been an OpenBSD machine, but due to lack of support (either from OpenBSD or from ROS side), I couldn´t use it. First OpenBSD compatibility was announced, then it disappeared from the MT materials. It works OK with Linux, but not with Win or not with ESXi.
http://forum.mikrotik.com/t/ccr2004-1g-2xs-pcie-and-freebsd/159742/1

As for using it offline, just with a dummy PCIexpress adapter card: I tried it once, since it has pretty decent HW. Could be useful as a replacement for an RB5009, though there is no USB port.
My experiment ended with blowing up an elco on the PCIe adapter board and also blowing the SMD fuse on the CCR. Then I decided it´s a waste of time, though I believe the culprit was the cheap PCIe board and the card might have worked. I could repair my card though by replacing the blown fuse and I tested it with Linux.

http://forum.mikrotik.com/t/ccr2004-1g-2xs-pcie-blown-fuse-would-need-specification/161016/1

39

Indeed I was once considering to use it in ESXi servers, with a short UTP from the RJ45 to the iLO port and internet feed via the SFP.
But no ESXi support (of course we are now phasing out ESXi so that could be replaced with Linux) and also no clarity if it could be powered and running when the main server chassis is “off” (via iLO).
So in the end I did not to forward with that.
It would have been nice because a 1HE rackspace in a hosting environment is usually cheaper and this could integrate server and router in a single 1HE.
But it apparently is not that easy, I also read about server crashes/reboots when the CCR was rebooted (e.g. for upgrade).

40

I believe this card could have been a hit, if MT gave it some more love:

  • Reset button accessible from the outside (not bare contacts)
  • External backup powering, or power by a supercap?
  • Some pins to control the PCs power
  • Maybe a serial port
  • Utopistic, but maybe KVM functionality?
  • Stable software support…

Could have made such Chinese cards unnecessary:
https://wiki.sipeed.com/hardware/en/kvm/NanoKVM_PCIe/quick_start.html