I've been struggling all weekend to figure out how to make our routers shape traffic to suit our new AT&T Switched Ethernet circuits. Our network is a hub and spoke configuration with one central site (yep!) 5 remote locations and 1 circuit sending all traffic out that's limited to 1000M, we VLAN tag all traffic.

Here's our network:
Uplink Site: 1GB CIR - MCOE_VLAN
Central Site: 1GB CIR - LAN (10.96.80.0/21)
Remote 1: 100MB CIR - BKS_VLAN (10.96.88.0/21)
Remote 2: 100MB CIR - HV_VLAN (10.96.96.0/21)
Remote 3: 100MB CIR - MA_VLAN (10.96.104.0/21)
Remote 4: 100MB CIR - WT_VLAN (10.96.112.0/21)
Remote 5: 1GB CIR - WH_VLAN (10.96.120.0/21)

I've got one simple queue at Central which I believe is causing all my problems:
add dst=MCOE_VLAN max-limit=1G/1G name=TOTAL priority=1/1 queue=default/default \ target=""

MCOE_VLAN is the VLAN to the uplink site. I mainly use the GUI and the target in the GUI for the TOTAL queue is set to 0.0.0.0/0 which is capturing all traffic right? Should I setup multiple targets for each remote subnet?

Once I figure out the TOTAL queue, I need to setup a queue for each remote site. Sites 1-4 need to be set to 100MB and site 5 will be 1GB. How would I go about accomplishing this task?

Use queue tree instead of simple queue.
There you can nest the speeds just as you like.

drop the dst from your queue and target your whole subnet going out.
10.96.80.0/17

ie

So I've got one site up and running with a queue tree, but when I attempted to get site 2 up it ruined traffic on both remote networks. As soon as I took site 2 off the new ASE network, site 1 was fine. I don't so much have a queue tree as three separate queues with their VLAN interfaces set as their parent.

Here's the config for the Central Site: https://docs.google.com/document/d/1nQv ... sp=sharing

Here's the config for the working Remote 1: https://docs.google.com/document/d/10Ct ... sp=sharing

Thanks in advance for any insight into where I've messed this up.

I have checked your configs and have 1 question. You have only a few queues. unless you are planning to have more queues why are you using mangle and queue trees. I understand the benefits when you processing large number of queues but your setup appears very simple and perform as well with a simple view point. If you plan to expand in future adding more rules etc then I understand.

Dgnevans, The person who replied before you suggested I use Queue Tree's at which point I began to research them and test that route. I will need 6 total queues, 1 for each remote site and one for the uplink to the MCOE_VLAN.

I'm really not well versed yet in Mikrotik configuration to be perfectly honest. All my local resources use Cisco equipment and have no idea how to help setup traffic shaping on our equipment. AT&T is of no help because they test the fiber and see traffic flowing between their equipment just fine. So it's clearly not their problem.

Above is what I would use. if you use this you only need to put this on the router connected directly to the internet. You would not need to mangle rules. If you do use this you will need to change the order of the rules moving the WAN rule to the bottom. You may want to edit your pcq-upload-default and pcq-download-default and adjust the total-limit (total-limit = expected simultaneous number of users per queue * limit (50 KiB)) Calculate this carefully with the number of users you have potential of being connected as this will use a lot of RAM.Lets say PCQ-TOTAL-LIMIT=x
RAM required = x*(2000Byte+200Byte) (2000Byte buffer for 1 packet. 200 Byte service data for 1 packet) eg Ram required = LIMIT * USERS = 50KiB*80=4000KiB*2.2 = 8800 Kib = <8.8 MB Ram required if you have 80 people concurrently online at one time. If you only have a few customers then this is not going to affect much but some of the larger ISP's will be taking this into account. It is covered under this page. viewtopic.php?t=14269 If you look at https://mum.mikrotik.com//presentations/US08/janism.pdf There is more detail.

Dgnevans, Thanks so much for this. I implemented the queues late Friday evening and everything looked fine as far as my existing connections were concerned. However when I attempted to switch HV_LAN over to the new ASE circuit the network lost packets like crazy. Here's what I remember, ping test wise, in hopes that it might give you a clue as to what might be happening.

Ping Results:
HV -> 8.8.4.4 = 0% Loss
HV -> LAN = 0% Loss
LAN -> HV = 60% Loss
LAN -> BKS = 60% Loss
HV -> BKS = 100% Loss
HV -> any other VLAN = 0% Loss

As soon as I switched HV back over to the old AT&T OPT-E-Man circuit the network went right back to normal. All ping results went back to 0% Loss.

Do I need to get in touch with AT&T and ask them to look at what's happening to traffic during this time? Thanks for sticking with me...I'm getting pretty desperate.

As this is working with your old link this looks more likely to be a problem with AT&T new link. I suggest you start there before we look at making any changes to your config. What is the maximum throughput you have to the end points for intervlan routing. If is 1 gig between all sites and the central router you can add this to the top of your simple queues so intervlan routing is not shaped.

Thanks, I'll give them a call and hobble down that rabbit hole. The maximum throughput to BKS, HV, MA and WT is only 100MB so I need to shape traffic.

I'll be back in touch soon, hopefully.

AT&T looked through all the configurations and found nothing on their end. They only operate on Layer 2, however once the tech had me switch HV_LAN over to the ASE network to look at packet traffic it worked without any loss to that network or the other 2 existing networks already switched over.

Now as soon as I added MA_VLAN it acted the exact same way. The tech again saw no packet loss and even gave a physical port an ip address on the VLAN 103 that I could ping without any problems. Odd as it may sound, during a router reboot on MA_VLAN all traffic at other sites went back to normal. So wouldn't that mean the issues reside on those remote routers?

Here's the config file from the remote 3 site (MA_VLAN): https://docs.google.com/document/d/1cpk ... sp=sharing

I'll just try switching over periodically and see if traffic flows smoothly.

It seems there is an issue with the remote sites. as It only happens when you add MA_VLAN router to the network. Are you sure you do not have a duplicate adress on the network or a loop somewhere from MA_VLAN.

That's exactly what AT&T thought. If it were a duplicate address it would have to be one of the addresses I set for the new VLAN interfaces. Plus the issues seem to randomly resolve themselves over time. Nothing changed address wise after HV_VLAN gave me the same issues when I turned that network up on ASE. I might have tweaked a queue, but that's it. Then after a weekend it suddenly decides to start working after AT&T pokes around in their switches.

If it were a loop I'd see it on the MA_VLAN network while it was still connected to the old OPT-E-Man. I just googled layer 3 loop and found out a bit about Routing Loops. Could that be happening here?

Are you running any bridges on your routers. if so try disabling rstp on the bridge. this could cause some issues. Secondly during the times of the issues are you seeing anything abnormal reported in the logs?

No bridges, I'm trying to keep it simple. Nothing shows up in the logs during those times except my changes to the route rules and some dhcp, info entries for assigned/deassigned addresses.

confirm this is not happening with all your other routers connected using at&t links just when you connect MA_VLAN. did you copy the config from 1 router to the other. could there be a possible duplicate mac address on your network. it would not show up on the routers. to check view each of the interfaces linking the sites together.

DGNEVANS you're a genius! I had multiple duplicate MAC addresses configured on my physical interfaces. I had set these up so long ago that I must have forgotten that I'd copied over 1 base config for all of them. I've reset MAC address on those conflicts and traffic is flowing smooth as silk.

THANK YOU SOOOOO MUCH FOR STICKING WITH ME AND FIGURING THIS OUT. I love this community.

No Problem we have all done this once before. For future when copying a config from one router to another try using the export method. Glad its working.