This is a common requirement, and is in many cases easier - but not with Internet connections. Load balanced lines and load balanced Internet connections are two very different things.
Consider this diagram:
ISP
|
Main Router (Company)
| |
| |
Branch Office (Company)
There, there is a single ISP providing service to the company, with load balanced / redundant private circuits between the main router and branch office are inside the company’s network and under the company’s IP space. This means that everybody can talk to each other. The load balancing solution is fairly simple, since it can be accomplished just with route costs, and QoS by setting up interface-attached queue trees on the interfaces on either side.
It is much more complicated in a situation where you have two ISP links (assuming you are not peering with them with BGP):
ISP-A ISP-B
| |
| |
Main Router
Because then, the two links that you need to load balance with are NOT under the control of the company, and have different address spaces given by the ISP (or perhaps different ISPs). Most networks have routing rules that, if a reply packet comes from the wrong address or different route, it should be dropped. That means that you end up needing to create extra rules to make sure that when the router sends a reply to a packet sent by ISP-A, that the ISP-A interface is used to send the reply and not the ISP-B interface, and that keeps a session on the same interface rather than sending some packets from one interface and some from the other. That’s where the complexity comes in. Good documents on PCC setup take care of this part, but don’t address the QoS aspect, which becomes much more complicated (as you will see) when you have PCC.
If you get your own IP space from ARIN or whatever authority it falls under, and you get two redundant BGP peer connections, most of this complexity vanishes. However, that kind of connection is not generally available for the same price point as a regular DIA, cable, or PPPoE internet service.
It is separate, but you actually have to pull it together. I’ll explain how you would probably do the QoS part, once you get the PCC working.
Queueing upload is fairly easy to do - the trick is that MikroTik needs to be told how fast your upload connection is. You can make a few queue trees connected to the upload interfaces to the two ISPs, and those queue trees would need a max-limit of a speed just below what that ISP provides you with. It is very easy to handle just upload.
There are two problems with queueing download. Normally you would want to queue download on the other side of the slow link by having a router there as well, and queuing it as upload (that is what you would do in the case of redundant links), but here that is not a choice. So, the two problems are:
- You can only queue download after it has gone across the congested link. This is imperfect by definition, it is not guaranteed. What this means is, if you have a 10Mbps download, and you set your download queuing to 9.5 or something (you have to limit to just below your actual download), the rate of packets coming in the connection can still exceed 10Mbps because your queue is not taking effect until after. The TCP windowing mechanism will ensure that TCP streams slow down to your 9.5Mbps queueing; however UDP is another matter. Certain UDP streams like Torrent traffic fire packets at the user like an Uzi. This overwhelms the congested link at the ISP end, before it even hits your router and before your router has a chance to drop it. Unlike your careful QoS setup, the ISP would drop all packets equally, and drop voice packets, torrent packets, whatever is necessary in order to rate limit you to your purchased 10Mbps. Still, at least queueing download on your side is somewhat effective with TCP streams and other normal traffic, and so it is worth doing for that reason.
- You have to queue download to just below the maximum rates for the internet connection it comes in on - meaning, you need two sets of queues, one for each download link, and match packets coming in on that link with the appropriate queues.
While with upload queuing, you can use interface-attached queue trees, for download you would need to use a global-attached queue tree and mark the packets appropriately not only based on their content but also based on what connection they came in on so that they will be associated with the correct queue.
It is much simpler on upload. On upload you can stamp any packet (with a ‘packet mark’), by creating mangle rules based on criteria, for instance you might use the following marks:
voice-up
importantdata-up
no-mark <-- default, used for normal data
Then, interface-based queue trees will take care of assigning the traffic shaping and QoS allocations to the two ISP connections. You would create two queue trees.
ISP A tree (limit to just below ISP A upload rate) (parent = WAN interface that connects to ISP, not global)
L voice (match packet mark voice-up)
L importantdata (match packet mark importantdata-up)
L normal (match packet mark no-mark)
ISP B tree (limit to just below ISP B upload rate) (parent = WAN interface that connects to ISP, not global)
L voice (match packet mark voice-up)
L importantdata (match packet mark importantdata-up)
L normal (match packet mark no-mark)
Download is possible, but much more difficult. You will need to have a series of packet marks for instance:
ispa-voice-down
ispa-importantdata-down
ispa-normaldata-down (here you can't use no-mark!)
ispb-voice-down
ispb-importantdata-down
ispb-normaldata-down (here you can't use no-mark!)
Then those will need to be attached to a single queue tree with parent=global, with the following structure:
Download Queueing - parent (global attached)
L ISP A tree (limit to just below ISP A download rate) (child of 'download queueing')
L voice (match packet mark ispa-voice-down)
L important (match packet mark ispa-importantdata-down)
L normal (match packet mark ispa-normaldata-down)
L ISP B tree (limit to just below ISP B download rate) (child of 'download queueing')
L voice (match packet mark ispb-voice-down)
L important (match packet mark ispb-importantdata-down)
L normal (match packet mark ispb-normaldata-down)
Creating the queues is fairly easy, the difficult part is going to be constructing the mangle rules in such a way that identifies not only what type of data the packet is (voice, important data, normal data, whatever other categories you want), but also identifies what ISP interface the packet arrived through. If you accidentally mark a voice packet from ISP B as ‘ispa-voice-down’ then the whole thing falls apart because the MikroTik starts thinking the voice packets from ISP-B are counting towards the download rate of your ISP-A tree and not the download rate of the ISP-B tree. If some download packets aren’t marked at all (ex. if you forget a rule, or there is a logic error in your rule design) then the queue tree can think it has free capacity when it actually doesn’t, and again the QoS falls apart. You also have to make sure that you don’t mark an upload packet with a mark you are using for download (ex. marking a voice packet going to ispa as ispa-voice-down instead of voice-up). In that case, it would not be counted towards the upload queue and would incorrectly be counted on the download queue, again causing QoS to not work properly.
As to how to get there, I would recommend first trying out QoS on its own, as I said. There are two scripts that I might suggest you try out, since one would be my recommended approach for upload and the other would be my recommended approach for download:
My own script, showing interface-attached HTB (queue tree), which is the easiest and best way for you to handle upload traffic for the two connections: http://forum.mikrotik.com/t/fasttrack-friendly-qos-script/102401/1
Greg Sowell’s script, which uses global-attached HTB (queue tree), which is the only way you could handle download traffic from two providers: http://gregsowell.com/?p=4665
Greg Sowell’s script does some other things differently - he doesn’t use DSCP at all or packet priority (IP precedence) and directly marks packets based on the criteria. With mine, you would apply a DSCP tag to the packet and then the priority and the packet mark would be derived from that. In your case, you use either approach, as long as you are consistent. if your packets are passing through more than one router, the DSCP approach might make more sense; if that is your only router, marking the packets based on the criteria might be more efficient.
Keep in mind that in your case you will need to disable fasttrack entirely to do QoS with two connections. That means disabling the fasttrack-connection rule in IP->Firewall.