Community discussions

MikroTik App
 
stipett
just joined
Topic Author
Posts: 10
Joined: Wed Oct 17, 2007 7:13 pm

Core Router Hardware

Wed Oct 17, 2007 7:44 pm

Hi!

We are planning on deploing several Core Routers with Mikrotik in ournetwork.

Can anyone recommend a hardware mix(even better with link to the equipment, but name and number will do fine) that are tested and working good? Anyone using/tried dell-servers for this?

I would be happy if someone could give me the list of gear to put together to create a good Core Router. Price is not the mather, i just nett to be assured that it will work.

It needs several NICs for our deployment.
 
User avatar
BrianHiggins
Forum Veteran
Forum Veteran
Posts: 702
Joined: Mon Jan 16, 2006 6:07 am
Location: Norwalk, CT
Contact:

Re: Core Router Hardware

Wed Oct 17, 2007 8:03 pm

we've used dell optiplex 170l workstations before with huge success. haven't tried on any other dell's yet
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Thu Oct 18, 2007 6:16 am

We have several Dell PowerEdge 860's in service in both core router and firewall roles. Currently they're all running v3.x to because the bootable drive controllers are SATA. The onboard Broadcom Gigabit NICs work great with ROS. They have 2 slots that are PCI-E, but you can request a PCI-X riser if you need it. You can easily get ten Gigabit interfaces with 2 quad cards.

We're using Intel EXPI9404PT PCI-E quad cards in our latest builds.

Addonics makes a CF-SATA adapter with a bracket that fits nicely in the slim cd-rom bay.
http://www.addonics.com/products/flash_ ... sahdcf.asp

We also have several HP DL140's and the work well also (we have both 2.x and 3.x on these).

We've load tested the PE 860's at several Gigabits throughput with a Xeon Dual-Core CPU. That was across three bonded Gigabit interfaces running traffic between several VLANs.

We're in the middle of a new build out now; I'll post some pictures of our MikroTiks when everything is online. One of the guys took the ugly plastic Dell bezel and gave them a red fading to black paint job more appropriate to a firewall.
 
stipett
just joined
Topic Author
Posts: 10
Joined: Wed Oct 17, 2007 7:13 pm

Re: Core Router Hardware

Tue Jan 08, 2008 1:01 am

Hi!

Did you finish your new build? If so, how did it go and what harware did you use.

Stig
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Wed Jan 09, 2008 12:52 am

Yes, the build is finished, although we're still working on some migrations and making the racks look nice.

We have two firewall/routers, both Dell PowerEdge 860's with Dual-core Xeon CPUs. The one serves our busines network, the other services our wired and wireless guest network running hotspot. There's a third identical cold spare mounted below. Move the flash card and network cables, then hit power button. If the flash is bad, we have an extra flash with the same version of ROS to dump a backup or export onto from either of the primarys.

I'll post some pics soon.

Our streaming went also live on the 1st, there's an HP DL140 on it built as a transparent bridge. Today it hit 200Mbs+ outgoing / 20mbs incoming on the outside bonded interfaces for several hours. It's still 2.x; so we're watching it closely as it only sees a single CPU.
 
User avatar
gmsmstr
Trainer
Trainer
Posts: 982
Joined: Fri Jun 04, 2004 2:22 am
Location: St. Louis, MO
Contact:

Re: Core Router Hardware

Wed Jan 09, 2008 1:19 am

http://www.mikrotikrouter.com is something to look at.

Several things to consider in this type of hardware

Reliability of equipment.
Size (sometimes this is important)
troubleshooting components
PCI bus speeds (some computers and system boards share 1 or 2 buss, this might not necessarily be able to handle the traffic you have)
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Wed Jan 09, 2008 1:40 am

We looked at those as well as an OEM unit that looks almost identical. It looks like a very nice piece for many applications. In our case, space is not a big issue as we have plenty of rack space where we are deploying them. In a NEMA box or a cramped rack at a POP they'd be an ideal choice.

At our pricing levels, we can also put the PE860's together for considerablly less than the cost of the PowerRouter. This is including 24x7x4 hardware support from Dell, the CF adaptor and a quad gig Intel NIC.

We looked at the specs on the PE860s and put them through considerable testing before picking them. We use the PE860's for other applications and have hundreds of them deployed. We took several from our lab and beat on them to be sure they would perform as expected in this application.
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Wed Jan 09, 2008 4:04 am

Some install pictures.
You do not have the required permissions to view the files attached to this post.
 
User avatar
gmsmstr
Trainer
Trainer
Posts: 982
Joined: Fri Jun 04, 2004 2:22 am
Location: St. Louis, MO
Contact:

Re: Core Router Hardware

Wed Jan 09, 2008 4:02 pm

Look nice. On thing, all the ethernets are in the back! Thats the hardest thing to fix too!
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Wed Jan 09, 2008 4:22 pm

In this install all of the racks are prewired with a 48port Cat6 patch panel to central wire management racks. All switching is located in the wire management racks. The patch panel is mounted at the very top, rear of the rack; so the rear Ethernet is suited for this type of install. You can almost see it in the top picture behind the modem sitting on the top unit. This room has 10 racks for equipment and then 4 wider racks with additional wire management space that houses the main patch panels, fiber patch panels to IDFs, switching, other routers, etc..
 
keefe007
Member Candidate
Member Candidate
Posts: 125
Joined: Sun Jun 25, 2006 3:01 am

Re: Core Router Hardware

Fri Jan 11, 2008 5:25 am

We looked at those as well as an OEM unit that looks almost identical. It looks like a very nice piece for many applications. In our case, space is not a big issue as we have plenty of rack space where we are deploying them. In a NEMA box or a cramped rack at a POP they'd be an ideal choice.

At our pricing levels, we can also put the PE860's together for considerablly less than the cost of the PowerRouter. This is including 24x7x4 hardware support from Dell, the CF adaptor and a quad gig Intel NIC.

We looked at the specs on the PE860s and put them through considerable testing before picking them. We use the PE860's for other applications and have hundreds of them deployed. We took several from our lab and beat on them to be sure they would perform as expected in this application.
Which OEM unit were you looking at?

Keefe
 
stipett
just joined
Topic Author
Posts: 10
Joined: Wed Oct 17, 2007 7:13 pm

Re: Core Router Hardware

Fri Jan 11, 2008 6:14 am

Hi and thanks for updating this thread :-)

Ive ordered a couple PE860 and a couple other servers that are a little bigger, like R200 that we are gonna do some testing on.

Ive also been looking at http://www.commell.com.tw/ and find their product somewhat interesting. Have any of you done some testing of these products and can state their opinion about them? I see they have som nice MBs without fan and cooling and that might be well suited in many installations since the mechanik parts are not there to fail.

regards,
Stig
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Fri Jan 11, 2008 6:18 am

Check out the NA-1801 under network appliances:
http://www.axiomtek.com

Here's a few good sources for rackmount stuff:

http://www.win-enterprises.com
http://www.advantech.com
http://www.logicsupply.com
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Fri Jan 11, 2008 6:29 am

Hi and thanks for updating this thread :-)

Ive ordered a couple PE860 and a couple other servers that are a little bigger, like R200 that we are gonna do some testing on.

Ive also been looking at http://www.commell.com.tw/ and find their product somewhat interesting. Have any of you done some testing of these products and can state their opinion about them? I see they have som nice MBs without fan and cooling and that might be well suited in many installations since the mechanik parts are not there to fail.

regards,
Stig

The R200 is replacing the PE860 and should be fairly close; we should have our sample from Dell next week. I think he said the biggest change is quad core and larger ram support.

I've looked at the Commell stuff; but not tested any personally. A friend of mine built an under seat mp3 player for his truck and besides running hot he's never had a problem.
 
stipett
just joined
Topic Author
Posts: 10
Joined: Wed Oct 17, 2007 7:13 pm

Re: Core Router Hardware

Fri Jan 11, 2008 6:41 am

Do you have any good online shops in USA where we can buy Quad-cards? I guess they will be alot cheeper there.

Stig
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Fri Jan 11, 2008 7:12 am

Do you have any good online shops in USA where we can buy Quad-cards? I guess they will be alot cheeper there.

Stig
We've been ordering Intel EXPI9404PT PCI-E from Dell with the PE860's.
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Sat Jan 12, 2008 12:32 am

This graph is from an HP DL140 that's serves our streaming.
It went live the 1st of the year.
You do not have the required permissions to view the files attached to this post.
 
User avatar
omega-00
Forum Guru
Forum Guru
Posts: 1167
Joined: Sat Jun 06, 2009 4:54 am
Location: Australia
Contact:

Re: Core Router Hardware

Mon Mar 24, 2008 11:52 am

These actually work out to be very similar in price to the powerrouters. (the Dell PowerEdge860)
 
stipett
just joined
Topic Author
Posts: 10
Joined: Wed Oct 17, 2007 7:13 pm

Re: Core Router Hardware

Thu Mar 27, 2008 3:43 pm

Hi!

We have now installed the system but we cant get the Quad Card up running.

We have this card:
Its an Intel Pro 1000 VT Quad Port GB NIC
(Intel PCI express GX750)
CHIP: WE-MIDCOM 7412-35 PM750LF1
SN: MY-0YT674-12402-819-00WR

Anyone know what driver will work or will this card not work at all with Mikrotic?

Regards,
Stig
 
User avatar
omega-00
Forum Guru
Forum Guru
Posts: 1167
Joined: Sat Jun 06, 2009 4:54 am
Location: Australia
Contact:

Re: Core Router Hardware

Fri Mar 28, 2008 8:03 pm

http://www.mikrotik.com/testdocs/ros/3. ... t.main.php -- there should be a list of drivers there but the link doesn't work. Perhaps in responding a mikrotik rep can fix this too?
 
stipett
just joined
Topic Author
Posts: 10
Joined: Wed Oct 17, 2007 7:13 pm

Re: Core Router Hardware

Fri Mar 28, 2008 11:36 pm

Hmm, and the 1000 VT is not in the list....

CRAP...

Is it hard to get a driver for the 1000VT than or should I just get another one? It was very expensive..

Regards,
Stig
 
User avatar
omega-00
Forum Guru
Forum Guru
Posts: 1167
Joined: Sat Jun 06, 2009 4:54 am
Location: Australia
Contact:

Re: Core Router Hardware

Sat Mar 29, 2008 2:43 am

If it doesn't already work with the system then if mikrotik were to add support for the device, it would have to be in the next software update.
Try finding a card that has working linux drivers first of all, then you may be able to ask for it to be included in newer releases.
 
stipett
just joined
Topic Author
Posts: 10
Joined: Wed Oct 17, 2007 7:13 pm

Re: Core Router Hardware

Sat Mar 29, 2008 12:14 pm

Thanks for you info.

This 1000VT card have good Linux drivers that are available from Intel Download Support.

Hopefully ill see this in the next release, if not ill try get one that are supported.

Regards,
Stig
 
User avatar
omega-00
Forum Guru
Forum Guru
Posts: 1167
Joined: Sat Jun 06, 2009 4:54 am
Location: Australia
Contact:

Re: Core Router Hardware

Fri Apr 04, 2008 11:12 am

Just to backup what has been stated earlier, the company I work for recently purchased a Dell PowerEdge860 for using with mikrotik.

Ended up just getting the 2 x 1gb port cards from the dell site and the PCI-E riser.
1gb ram + 80gb sata drive, then I bought a seperate 500gb drive to use as a cache.

Works fabulously.
 
User avatar
BrianHiggins
Forum Veteran
Forum Veteran
Posts: 702
Joined: Mon Jan 16, 2006 6:07 am
Location: Norwalk, CT
Contact:

Re: Core Router Hardware

Wed May 07, 2008 7:51 pm

This graph is from an HP DL140 that's serves our streaming.
It went live the 1st of the year.

Do you have any updated graphs? I'm curious to see just how much traffic you're pushing.
 
User avatar
gustkiller
Member
Member
Posts: 419
Joined: Sat Jan 07, 2006 5:15 am
Location: Brazil
Contact:

Re: Core Router Hardware

Thu May 08, 2008 2:36 am

This graph is from an HP DL140 that's serves our streaming.
It went live the 1st of the year.

Do you have any updated graphs? I'm curious to see just how much traffic you're pushing.
and cpu graphics! and whats you are doing with the router? only routing or firewall, queues and so?
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Fri May 09, 2008 4:30 am

This graph is from an HP DL140 that's serves our streaming.
It went live the 1st of the year.

Do you have any updated graphs? I'm curious to see just how much traffic you're pushing.

Lately during peak hours it's been pushing over 500Mbs out the pipe. It's still 2.x; so CPU usage is a bit high on the single CPU.
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Fri May 09, 2008 4:38 am

This graph is from an HP DL140 that's serves our streaming.
It went live the 1st of the year.

Do you have any updated graphs? I'm curious to see just how much traffic you're pushing.
and cpu graphics! and whats you are doing with the router? only routing or firewall, queues and so?
I'll post current graphs soon (we've been swamped with projects lately). This system is configured as a transparent bridge. There are two gig NICs bonded coming in, two gig nics bonded going out with a third nic for management. It sits between a pair of layer3 dell gigabit switches; one is configured as the edge router, the other is a core switch. That core switch feeds two Dell blade chassis. Each Blade chassis is fed with three gigabit links from the core switch. The edge is fed with a 10gigabit fiber connection currently throttled to 2Gbs by the ISP. The platform hosts internet streaming for radio stations.
 
User avatar
gustkiller
Member
Member
Posts: 419
Joined: Sat Jan 07, 2006 5:15 am
Location: Brazil
Contact:

Re: Core Router Hardware

Fri May 09, 2008 6:00 am

thanks! and send the interface tab from winbox ( for we see the pps) :-D
 
mmmpils
just joined
Posts: 18
Joined: Fri May 09, 2008 11:21 am

Re: Core Router Hardware

Fri May 09, 2008 2:51 pm

Thanks for you info.

This 1000VT card have good Linux drivers that are available from Intel Download Support.

Hopefully ill see this in the next release, if not ill try get one that are supported.

Regards,
Stig
Hmm.

anyone know if there has been a release from MT witch support 1000VT PCI-E??

The Release note sees: "*) added support for Intel Gigabit PCI-Express cards;". Do someone here know exactly witch card they have added??


Lasse
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Fri May 09, 2008 11:07 pm

Here are the current stats from the DL140.
You do not have the required permissions to view the files attached to this post.
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Fri May 09, 2008 11:14 pm

Here are the resources from the DL140. There were about 34,000 connections at the time.
You do not have the required permissions to view the files attached to this post.
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Fri May 09, 2008 11:16 pm

Here are the interfaces from the DL140
You do not have the required permissions to view the files attached to this post.
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Fri May 09, 2008 11:21 pm

Here are the yearly interface and CPU graphs.
You do not have the required permissions to view the files attached to this post.
 
User avatar
gustkiller
Member
Member
Posts: 419
Joined: Sat Jan 07, 2006 5:15 am
Location: Brazil
Contact:

Re: Core Router Hardware

Sat May 10, 2008 11:21 pm

you have a well-behaved network :) only 45k pps for 400Mbps + troughput.
:D
here with only 120mbits we got about 25 k :)

are you with conntrack activated , any queue or firewall filter rules activated?

thanks!
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Sun May 11, 2008 5:41 am

you have a well-behaved network :) only 45k pps for 400Mbps + troughput.
:D
here with only 120mbits we got about 25 k :)

are you with conntrack activated , any queue or firewall filter rules activated?

thanks!
No queues, connection tracking is enabled; there are 46 firewall filters.
 
User avatar
BrianHiggins
Forum Veteran
Forum Veteran
Posts: 702
Joined: Mon Jan 16, 2006 6:07 am
Location: Norwalk, CT
Contact:

Re: Core Router Hardware

Wed Nov 26, 2008 4:31 pm

I'm curious to hear how this is working for you now, your last post showed significant growth patterns, by now I'm sure you've had to make some changes to accomidate that growth...
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Wed Nov 26, 2008 4:55 pm

We've since upgraded the connection to 2Gbs and it will soon be 3Gbs. We're currently averaging 1.1Gbs (2.2Gbs+ throughput) during the normal peak of the day. We did upgrade the hardware and move to 10Gbs PCIe network adapters to allow for the expansion. We're currently running on a pre-release ROS; I'll be doing a write-up soon.

That original hardware was able to handle the 1Gbs connection (2Gbs throughput), but started having issues (related to the bonding) as we went over 1Gbs.
 
User avatar
BrianHiggins
Forum Veteran
Forum Veteran
Posts: 702
Joined: Mon Jan 16, 2006 6:07 am
Location: Norwalk, CT
Contact:

Re: Core Router Hardware

Tue Dec 02, 2008 4:56 pm

are you using the same Dell PE840 chassis as in the picture?

what model NIC did you install?
 
mknnoc
Trainer
Trainer
Posts: 229
Joined: Thu Feb 28, 2008 6:40 am
Location: cambodia

Re: Core Router Hardware

Thu Dec 11, 2008 12:20 pm

Very interesting topic that i am looking for!

i never install ROS on any servers like DELL or HP because most of the time i did it. it cannot recognize the HDD(SATA). so, can anyone let me know what to change to let ROS recognize R200 or PE860 HDD(SATA)?
 
jucomadmin
just joined
Posts: 6
Joined: Thu Dec 11, 2008 3:39 pm
Location: Riga, Latvia
Contact:

Re: Core Router Hardware

Thu Dec 11, 2008 4:11 pm

Hello!
We have a core-router, running on MikroTik v3.14; it uses Border Gateway Protocol and no firewall; we have three Gb links connected to this router and about 200-300 Mbps of traffic on each interface.
Hardware configuration:
Mother Board - Intel Server Board http://www.intel.com/products/server/mo ... erview.htm
CPU - Pentium D 3 GHz

CPU load is approximately 70-80 percent.

Will we see the difference if we change CPU to Intel Dual Core processor ~3 GHz?
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Thu Dec 11, 2008 9:33 pm

Very interesting topic that i am looking for!

i never install ROS on any servers like DELL or HP because most of the time i did it. it cannot recognize the HDD(SATA). so, can anyone let me know what to change to let ROS recognize R200 or PE860 HDD(SATA)?
We have typically pre-installed ROS on the flash drives with different system. You also need to make sure that the SATA controller is set to "native" or "legacy" if there is an option.
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Thu Dec 11, 2008 9:38 pm

Hello!
CPU - Pentium D 3 GHz
CPU load is approximately 70-80 percent.
Will we see the difference if we change CPU to Intel Dual Core processor ~3 GHz?
Which Pentium D 3Ghz to which dual core (or Core2 as they are differnt)? There are many versions of each with variations of power usage, cache and bus speed. A faster bus speed and more cache would probably have a bigger impact than more mhz. Are you planning on trying it with multi-cpu on or single?
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Thu Dec 11, 2008 9:42 pm

are you using the same Dell PE840 chassis as in the picture?

what model NIC did you install?
We've upgraded to a Dell PowerEdge 2950 with two Intel based 10G adapters. I will do a complete writeup with pictures when I have some spare time.
 
mknnoc
Trainer
Trainer
Posts: 229
Joined: Thu Feb 28, 2008 6:40 am
Location: cambodia

Re: Core Router Hardware

Fri Dec 12, 2008 6:05 am

We have typically pre-installed ROS on the flash drives with different system. You also need to make sure that the SATA controller is set to "native" or "legacy" if there is an option.
I installed successfully with HP DL320 G5p. the system is running in Quad-core Intel Xeon processor. First installed without any configuration, i see the CPU hits up to ~20%. and the server takes time or idle when restart or shutdown.

I think it may have problem with the CPU. but i cannot find option in BIOS to make it act as single core.

my ROS version is 3.17.

any suggestion!!
 
jucomadmin
just joined
Posts: 6
Joined: Thu Dec 11, 2008 3:39 pm
Location: Riga, Latvia
Contact:

Re: Core Router Hardware

Fri Dec 12, 2008 1:04 pm

Which Pentium D 3Ghz to which dual core (or Core2 as they are differnt)? There are many versions of each with variations of power usage, cache and bus speed. A faster bus speed and more cache would probably have a bigger impact than more mhz. Are you planning on trying it with multi-cpu on or single?
Actually, I don't know, what is better, to install one CPU with multiple cores or two single-core CPU? For example, in both cases BUS speed and cache size are similar.
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Fri Dec 12, 2008 7:00 pm

This graph is from an HP DL140 that's serves our streaming.
It went live the 1st of the year.

Do you have any updated graphs? I'm curious to see just how much traffic you're pushing.
As this seems to be a very hot topic..
Here is a graph from the Dell PowerEdge 2950 with 10G adapters.

This is still pre release 3.17 so the graph itself is correct, but the numbers below are wrong. We have a window scheduled next week to apply the release version of 3.17 that lists adding support for 10G graphing.

CPU max has been 10% during peaks with multi-cpu enabled; which so far has been stable.
You do not have the required permissions to view the files attached to this post.
 
BelWave
Member Candidate
Member Candidate
Posts: 184
Joined: Wed Aug 04, 2004 12:49 am

Re: Core Router Hardware

Sun Dec 14, 2008 3:53 am

This graph is from an HP DL140 that's serves our streaming.
It went live the 1st of the year.

Do you have any updated graphs? I'm curious to see just how much traffic you're pushing.
As this seems to be a very hot topic..
Here is a graph from the Dell PowerEdge 2950 with 10G adapters.

This is still pre release 3.17 so the graph itself is correct, but the numbers below are wrong. We have a window scheduled next week to apply the release version of 3.17 that lists adding support for 10G graphing.

CPU max has been 10% during peaks with multi-cpu enabled; which so far has been stable.

Very impressive dankerr.

I think one of the first PC based MikroTik routers we deployed was a Dell SC400 with a P4 CPU. Worked well, but since then we've been building our core routers with Intel hardware because in many cases we require a 4U shallow depth case.

This discussion regarding Dell and HP boxes has peaked my interest in them again. We are in the process of bringing up a new data center/co-lo and a standard HP or Dell server case will work fine. I'm curious how MikroTik handles their license if we decided to load the ROS on Dell RAID drives. I like the idea of having ROS loaded on redundant mirrored drives in the event a drive fails.

Doesn't MikroTik key the license on the HDD? What happens if one of the drives in a RAID fails and you replace it? Does the MikroTik license get lost or otherwise render the router "dead" if you swap out a drive within the RAID? Or is the license keyed on the RAID as a whole?

Thanks,

Brad
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Sun Dec 14, 2008 7:34 pm


I'm curious how MikroTik handles their license if we decided to load the ROS on Dell RAID drives. I like the idea of having ROS loaded on redundant mirrored drives in the event a drive fails.

Doesn't MikroTik key the license on the HDD? What happens if one of the drives in a RAID fails and you replace it? Does the MikroTik license get lost or otherwise render the router "dead" if you swap out a drive within the RAID? Or is the license keyed on the RAID as a whole?

Thanks,

Brad
First thing, ROS doesn't support the RAID hardware. Even if it did, I would rather use a flash drive of some kind. A flash drive eliminates several moving parts, and a heat source.
 
BelWave
Member Candidate
Member Candidate
Posts: 184
Joined: Wed Aug 04, 2004 12:49 am

Re: Core Router Hardware

Sun Dec 14, 2008 8:05 pm


I'm curious how MikroTik handles their license if we decided to load the ROS on Dell RAID drives. I like the idea of having ROS loaded on redundant mirrored drives in the event a drive fails.

Doesn't MikroTik key the license on the HDD? What happens if one of the drives in a RAID fails and you replace it? Does the MikroTik license get lost or otherwise render the router "dead" if you swap out a drive within the RAID? Or is the license keyed on the RAID as a whole?

Thanks,

Brad
First thing, ROS doesn't support the RAID hardware. Even if it did, I would rather use a flash drive of some kind. A flash drive eliminates several moving parts, and a heat source.
I'm not concerned with heat or size of the router in this case. I mentioned earlier this is for a data/co-lo center.

We've had flash drives fail in the past leaving the router hard down. With redundant HDDs it doesn't matter if there are additional moving parts. If a drive fails you simply remove it and slide in a new one. Router never misses a beat.

If that's true regading MT not supporting RAID then that is a shame. Drops MikroTik down a notch in the redundancy category.

Brad
 
BelWave
Member Candidate
Member Candidate
Posts: 184
Joined: Wed Aug 04, 2004 12:49 am

Re: Core Router Hardware

Mon Dec 15, 2008 7:06 pm

Regarding the loading ROS on a RAID setup...how would the OS know it's on a RAID? In the case of HP and Dell servers aren't their RAIDs hardware based? The RAID hardware simply ties the HDDs together and the OS sees the array as one HDD.

Correct?

What is the risk of MikroTik allowing RAID support? A HDD pulled from a RAID array isn't going to be able to boot up on its own in a different machine.

I think this is worth a try for the peace of mind knowing that a single HDD failure won't take a core router offline.

Brad
 
sandman
newbie
Posts: 37
Joined: Sat Apr 07, 2007 2:13 am

Re: Core Router Hardware

Mon Dec 15, 2008 11:32 pm

Even though they are hardware based RAID, they still require drivers to communcate with the RAID hardware. That's why you have to load RAID driver at the very beginning of Windows setup, if you plan to install to a hardware based RAID array.

Those drivers would need to be loaded and supported by Mikrotik RouterOS, which I believe they are not. It would be a nice feature, but with so much RAID hardware out there, might be a lot of work....
 
BelWave
Member Candidate
Member Candidate
Posts: 184
Joined: Wed Aug 04, 2004 12:49 am

Re: Core Router Hardware

Tue Dec 16, 2008 1:32 am

Aren't there motherboards with RAID built-in? If this is the case wouldn't the motherboard BIOS allow you to setup the HDDs in the array? Then I would assume once you boot off of the CDROM MikroTik would "see" the RAID array as a target install drive no different than a single IDE DoM or STATA DoM.

Thoughts?


Brad
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Tue Dec 16, 2008 3:43 am

Aren't there motherboards with RAID built-in? If this is the case wouldn't the motherboard BIOS allow you to setup the HDDs in the array? Then I would assume once you boot off of the CDROM MikroTik would "see" the RAID array as a target install drive no different than a single IDE DoM or STATA DoM.

Thoughts?


Brad
All of the "built-in" controllers I've worked with still would require drivers and software to work correctly; as they are really software emulation. That software emulation eats into your CPU resources.

And why complicate a router with RAID? The more parts you have, the more things there are to break. Drivers aside, adding a RAID device into the picture gets you into maintaining controller BIOS and drive BIOS updates. Recently there was a major firmware update affecting most current Dell PERC controllers. I really would not have wanted to update the RAID firmware on my 10G router.
 
BelWave
Member Candidate
Member Candidate
Posts: 184
Joined: Wed Aug 04, 2004 12:49 am

Re: Core Router Hardware

Tue Dec 16, 2008 6:48 am

I guess you've never had a DoM/flash drive fail on you. We have had this happen once maybe twice in the past few years with MikroTik and have also had a flash drive fail on a Packeteer device. In all cases the failed flash HDD brought the device down.

By default MT has Interface graphing storing to disk every 5min. This has to considerably eat into the service life of any flash drive while a couple RAID drives could handle this amount of read/writing better. If a drive were to fail (spinning - moving parts and all...) it could simply be swapped out without interrupting service.

If you really have a problem with conventional HHDs (moving parts) then go with two or more SATA flash drives in a RAID and get the best of both worlds.

I spent a little time on the phone today with our Dell rep. Explained what we were trying to do and they didn't see any reason why their built-in RAID controller wouldn't work with MikroTik. With the 30 day return policy and no cash out of pocket I'm thinking why not give it a try. Another attraction to the Dell R900 is the 7 PCIe slots. That means we could have 42 or more Interfaces using the 6 port NICs we've been testing.

I can see updating a RAID driver during the initial installation, but why would you ever update a RAID driver if it was operating properly?

Brad
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 26322
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: Core Router Hardware

Tue Dec 16, 2008 8:41 am

you do NEED a driver to install anything on a RAID system, even if it's hardware RAID, just as others said. As there are too many drivers out there, and RouterOS never supported any external drivers - we can't build them all into the installation. The gain is too small.
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Tue Dec 16, 2008 5:12 pm


I can see updating a RAID driver during the initial installation, but why would you ever update a RAID driver if it was operating properly?

Brad

Have you applied the recent rash of Dell PERC firmware and driver updates to your systems? Without them, your PERC controllers will not properly detect or report a failed drive.

There are also too many RAID cards to include. In our fleet I can think of at least six variations of the Dell PERC 6. Let alone others from the PERC line. Start factoring in options from HP, IBM, OEM LSI, OEM Adaptec, OEM Intel and the list grows.

One of the reasons we use ROS, is it's simplicty. We treat it as a network device; just as we do a Cisco router or switch. As you start adding the complexity of RAID, you loose that simplicity.
 
BelWave
Member Candidate
Member Candidate
Posts: 184
Joined: Wed Aug 04, 2004 12:49 am

Re: Core Router Hardware

Tue Dec 16, 2008 5:40 pm

Actually we're largely running HP/Compaq servers because our server admin prefers them over Dell.

The last Dell we have running has at one time had a HDD fail and we were able to identify it and replace it without service interruption. It's a pretty old box now (PowerEdge 2950?) I think we bought it back in 2001. Can't believe we've had it that long, but it hasn't required much if any handholding other than the usual Windows updates.

I personally prefer Dell, but don't have anything against HP. The Dell R900 just looks like a nice fit given the 7 PCIe slots.

We've had HP HDDs fail as well without any interruption in service. Walked by and noticed a red light on a drive, pulled it out and slipped a new one in place. Done. Problem solved. Just was thinking that would be a nice thing to be able to do with a core MT router as well.

Maybe our flash drive failures were an anomaly. Have you never had a flash die before?
 
User avatar
dankerr
Frequent Visitor
Frequent Visitor
Posts: 77
Joined: Wed Sep 12, 2007 10:56 pm
Location: Atlanta, GA USA

Re: Core Router Hardware

Tue Dec 16, 2008 6:02 pm

Actually we're largely running HP/Compaq servers because our server admin prefers them over Dell.

The last Dell we have running has at one time had a HDD fail and we were able to identify it and replace it without service interruption. It's a pretty old box now (PowerEdge 2950?) I think we bought it back in 2001. Can't believe we've had it that long, but it hasn't required much if any handholding other than the usual Windows updates.

I personally prefer Dell, but don't have anything against HP. The Dell R900 just looks like a nice fit given the 7 PCIe slots.

We've had HP HDDs fail as well without any interruption in service. Walked by and noticed a red light on a drive, pulled it out and slipped a new one in place. Done. Problem solved. Just was thinking that would be a nice thing to be able to do with a core MT router as well.

Maybe our flash drive failures were an anomaly. Have you never had a flash die before?
A Dell from 2001 would be a 2550 or possibly a 2650.

We've had flash fail after years of service or an "outside influence" such as a lightning strike. It's not a regular occurance. HDD failures however are and we expect them. That's why at the very least you run RAID1 on a server. We have several that are failed now (both on HP and Dell) that will get replaced tomorrow when the replacements arrive. With several hundred servers; you expect that.

With a router; it's just as simple to replace the bad flash and dump the configuration back on. We typically have a licensed spare or two at the ready for new builds anyway. I'd rather not need to wait for the RAID to rebuild or have degraded performance while the array rebuilds.

For any of our critical routers (Cisco included) we also have extra hardware ready. After a failure; we drop-in the replacement and then deal with getting the failed hardware fixed or replaced. Even with 4hr support on an Dell server chassis or a Cisco router; swapping to new hardware is faster.
 
User avatar
omega-00
Forum Guru
Forum Guru
Posts: 1167
Joined: Sat Jun 06, 2009 4:54 am
Location: Australia
Contact:

Re: Core Router Hardware

Mon Mar 16, 2009 2:19 pm

posted this in a thread of my own but i believe it is relevant to this thread also
---
As I see it, the best mikrotik hardware I could hope for would have:

2ghz or higher CPU
2gb ram - optionally 4gb
2 x SATA drive bays
2 x Internally mounted USB ports (3g modem + optional mikrotik USB drive)
2 x externally mounted USB ports (for UPS connection and/or 3g modem and/or USB drive)
2 - 4 onboard gb ethernet ports
2 x PCI-E slots for:
- 1 x 4 port gb ethernet card or other interface card
- 1 x Remote access card

Edit: possibly a CF slot as well.

The remote access card is the one I find most useful on current servers as if for any reason I break a config or even upgrade a device improperly and lose access to it, I have a backup method of regaining access to my machine and getting it back online with no hands on support. Not so much of a problem if the router you're using is in the data center with you, but having this extra level of accessability helps me sleep better. (more often too!)
 
boardman
Member Candidate
Member Candidate
Posts: 258
Joined: Fri May 28, 2004 11:10 pm
Location: Mexico
Contact:

Re: Core Router Hardware

Wed Jun 17, 2009 12:57 am

Actually we're largely running HP/Compaq servers because our server admin prefers them over Dell.

The last Dell we have running has at one time had a HDD fail and we were able to identify it and replace it without service interruption. It's a pretty old box now (PowerEdge 2950?) I think we bought it back in 2001. Can't believe we've had it that long, but it hasn't required much if any handholding other than the usual Windows updates.

I personally prefer Dell, but don't have anything against HP. The Dell R900 just looks like a nice fit given the 7 PCIe slots.

We've had HP HDDs fail as well without any interruption in service. Walked by and noticed a red light on a drive, pulled it out and slipped a new one in place. Done. Problem solved. Just was thinking that would be a nice thing to be able to do with a core MT router as well.

Maybe our flash drive failures were an anomaly. Have you never had a flash die before?
A Dell from 2001 would be a 2550 or possibly a 2650.

We've had flash fail after years of service or an "outside influence" such as a lightning strike. It's not a regular occurance. HDD failures however are and we expect them. That's why at the very least you run RAID1 on a server. We have several that are failed now (both on HP and Dell) that will get replaced tomorrow when the replacements arrive. With several hundred servers; you expect that.

With a router; it's just as simple to replace the bad flash and dump the configuration back on. We typically have a licensed spare or two at the ready for new builds anyway. I'd rather not need to wait for the RAID to rebuild or have degraded performance while the array rebuilds.

For any of our critical routers (Cisco included) we also have extra hardware ready. After a failure; we drop-in the replacement and then deal with getting the failed hardware fixed or replaced. Even with 4hr support on an Dell server chassis or a Cisco router; swapping to new hardware is faster.

Hi

When you replace the flash into another server, hardware failure on the first server, what happens to the MAC address of the new hardware? Does it works just like that, or you have to manually edit the configuration and fix the MAC issues?

Regards

JB
 
andrejolas
just joined
Posts: 1
Joined: Tue Jun 29, 2010 10:51 am

Re: Core Router Hardware

Tue Jun 29, 2010 11:58 am

Hi guys,
I am starting with RouterOS and tried installation RouterOS 3.30 4.10 and 5.0 on HP DL140 G1 you have mentioned before. The installation always end up without any error message, but after restart the system hangs with message "system loading".
Can you please give me any clues what should I do?
Thanks

Andrej
 
User avatar
omega-00
Forum Guru
Forum Guru
Posts: 1167
Joined: Sat Jun 06, 2009 4:54 am
Location: Australia
Contact:

Re: Core Router Hardware

Fri Sep 17, 2010 8:29 am

Hey dankerr

Any further updates on your 10G setup? Interested to see what sort of traffic levels you're doing nowdays.
Going to be rolling out a setup in the ~300Mbps range shortly so will hopefully be posting some of my own :-P

Who is online

Users browsing this forum: anav, Batterio and 92 guests