Page 1 of 1

Flow Control, should I use it?

Posted: Thu Dec 10, 2015 1:47 am
by jmay
One of my routers was giving an error "fcs error on link". I have an Airfiber plugged into that port and I found something that suggested turning on flow control would fix the problem since flow control was enabled on the Airfiber. This indeed made the error go away, but then I started wondering if I should use flow control on every thing. I've been reading pros and cons and now I'm confused. Should I use it or not? I have many wireless links every where and am wondering if there could be a benefit.

Re: Flow Control, should I use it?

Posted: Thu Dec 10, 2015 11:51 pm
by jmay
Anyone...

Anyone...

Bueller...

Re: Flow Control, should I use it?

Posted: Fri Dec 11, 2015 12:08 am
by IntrusDave
FCS and Flow Control should not be be related. FCS (Frame Check Sequence) tells you that you are getting CRC errors on the link. Flow control should only be needed if you are running at near 100% link capacity.

That said, FCS errors are common caused by misbehaving hardware. often it could be as simple as a half-duplex link, or outdated firmware in a networked device. In more extreme situations you can look into the hardware itself. Even a cheap or damaged cat5 cable can cause framing errors. Hell, I've had them because a cable was run too close to light fixture.

Re: Flow Control, should I use it?

Posted: Fri Dec 11, 2015 1:55 am
by jmay
Interesting. Turning on flow control did make the error go away tho and I can reproduce it by turning it back off. This is a link that runs close to 600 megs at times, and minimally runs 300 megs.

Re: Flow Control, should I use it?

Posted: Fri Dec 11, 2015 2:14 am
by IntrusDave
interesting. The link should be able to run at a full gigabit without errors. What is the physical connection and what is on the other side of that connection? How many patch panels/devices/etc?

Re: Flow Control, should I use it?

Posted: Fri Dec 11, 2015 2:23 am
by IntrusDave
After re-reading your first post, and some googling - it does look like this is a common with the AirFibers. Almost every one has been solved by setting the flow control to auto or on. It's likely that the AirFiber is enabling it on it's end no matter what. So, looks like you found your solution on your own.

Re: Flow Control, should I use it?

Posted: Fri Dec 11, 2015 10:43 pm
by jmay
The airfiber does have an option to turn it off, but its on by default. Ubiquiti says its best to leave it on, but I find many posts like yours suggesting a network is better off without it. I guess I'll leave it on for airfibers, and off for everything else.

Re: Flow Control, should I use it?

Posted: Fri Dec 11, 2015 10:45 pm
by IntrusDave
Honestly, the only time that you **REALLY** need it off, is with VoIP. it really screws with the packets and makes tons of jitter.

Re: Flow Control, should I use it?

Posted: Sat May 07, 2016 2:07 am
by WirelessRudy
Honestly, the only time that you **REALLY** need it off, is with VoIP. it really screws with the packets and makes tons of jitter.
So nobody can really switch it on? I'd presume every traffic nowadays has VoIP packages in it as well (if you are a internet provider)?

We have tcp throughput issues and are looking for solutions. We thought to have it found since all our network (full MT, hundreds of units) have it 'off' by default.
For a moment we thought this might be the solution, but we DO have clients using VoIP.......... so where are we now?

Re: Flow Control, should I use it?

Posted: Sat May 07, 2016 7:58 am
by IntrusDave
More bandwidth? Without knowing all the details of your network, links, load, traffic, etc - there is really no way to answer that. If you have over-sold your bandwidth, you don't have a lot of choice. You can try traffic shaping, throttling, etc.. Flow Control is going going to help with the physical links are maxed out. It stops traffic indiscriminately. So NOT a good option when dealing with VoIP.

What you will need to do is analyze your traffic, figure out where it's all being used and then figure out if you have to buy more bandwidth, or if you need to do some QoS.

Re: Flow Control, should I use it?

Posted: Sat May 07, 2016 12:39 pm
by WirelessRudy
More bandwidth? Without knowing all the details of your network, links, load, traffic, etc - there is really no way to answer that. If you have over-sold your bandwidth, you don't have a lot of choice. You can try traffic shaping, throttling, etc.. Flow Control is going going to help with the physical links are maxed out. It stops traffic indiscriminately. So NOT a good option when dealing with VoIP.

What you will need to do is analyze your traffic, figure out where it's all being used and then figure out if you have to buy more bandwidth, or if you need to do some QoS.
Bandwidth can't be the problem. We have a 300/300Mb symmetric line and rarely see combined client traffic coming close to 200...
On my PC's we do at times tests and by opening several download streams, preferably on 2 or 3 PC's we get up to max. 295Mbps download traffic.
Upload with www.speedtest.net usually shows close to 300Mbps..

If there are bottlenecks, it's in the network. But like any other provider we have to oversubscribe, even at the level of an P2MP network. I don't know if there are any operators that keep the possible max clients usages within levels the AP can handle (100Mb connected AP could only server 10 x 10Mb contracted clients? Bad, and thus very rare business model.)

Every operator sells more bandwidth than he could ever deliver if all his clients would demand their contracted speed at the same time. That counts for all network infrastructures. If it is internet, the electricity company's, the national road infrastructure or the domestic water company.

I know the national adsl provider has some 10 million clients and they are all sold 10Mbps. How much would that be if all would use that at the same time? Not a single network can deal with that. So we all work with oversubscribing our network at some levels. We all have to make educated best guesses at which points the first priority lies to upgrade which bottleneck to produce better overall capacity again and again.

So bottleneck's are part of every network. And thus mixed capacity links are part of a network.

We have as said 300/300Mb capacity coming to us via a Cisco router that connects by gigabit to our CCR. From here we have gigabit Ethernet going to several back-haul radio's. Some of these maintain a 80Hz ac link and some only a 20Mhz 'n' link.
Than on the other side we see some radio's connected with 100M to the next radio (AP or just a new link) where others have yet again gigabit connections to the next routers.

In general I would say that each client requested packages that comes in from the 300Mb symettric would 'see' some gigabit links, some 100M links, some highspeed 'ac' radio backhaul, some medium speed 20Mhz 'n' backhaul towards an AP that usually connects to 'n' (some still single chain) clients where these CPE's than connect with 100M to client's devices.

Since download is the bulk of traffic I should start looking from where the traffic come in?

1st capacity is 300Mb ('pipe' from provider to Cisco with 300Mb)
2nd: 1000M cable to CCR. Presume here no traffic flow control needed?
3rd: New 1000M cable to 'ac' radio with 80Mhz channel (we could 'pump' some 250-300Mb tcp over it. It's actually 2 links with OSPF simulated duplex)
4th: That radio link capable of 250-300Mbps
5th: Other radio wich connects with 1000M to another CCR.
6th: This CCR routes traffic to 4 new bachauls, all of the radio's connect with 1000M to this CCR.
6tha: This CCR also through a switch supplies 3 x AP that serves some 75 clients. All connections are Gig but radio links are much weaker. Conn. rate in generall 80Mb.
7th: New backhauls to new remote locations. Some have 40Mhz 'n' links set to 270Mbps conn. rate, other have 130Mbps conn. rate 20Mhz channel.
8th etc. More links, more AP's

Some clients in our network are only reached after the package travels over 5 wireless links (including the AP to CPE link) and had seen at least 15 Mikrotik routers in our network before it has reached clients Wifi router or PC.....

A small minority of clients would use VoIP.
70 of our clients now connect with PPPoE and within a month this should be 100%
The PPPoE server in our gateway creates single queues for each user depending to its assigned contract. Highest speeds is 50Mb download, bulk is 20Mb and 40% only 10Mb.

What would be your best strategy if we talk about flow control?

Re: Flow Control, should I use it?

Posted: Sat May 07, 2016 4:37 pm
by pukkita
In my experience, and IMHO, using traffic flow control is like disabling auto negotiation, i.e. a "patch" to "solve" (read, alleviate) the true problem.

You shouldn't need enabling traffic flow for starters. If it alleviates a problem, go fix the true problem.

The two typical scenarios I see people resorting to it are:

Backpressure.
If it's backpressure, go upgrade  the relevant network devices or  re-engineer that part of the network.

FCS errors:
AirFibers and certain UBNT more "pro" equipment are known to cause FCS errors.

Putting it into single terms, power radiated by the antenna gets injected back into the ether cable/port.

Try to reduce Tx power specially if your install is not optimal in terms of nearby obstacles, and specially reduce it to the safest minimum margin leaving enough SNR; less power is always better.

Use top notch "armored" cables and RJ45 connectors, swap for known-to-be-working ones, change POE units, avoid sources of EMI (AC motors, FM stations...), replace the unit itself...

In summary: fix the problem, not the symptom. If there are errors, there's no magic to make them dissapear, but to eliminate the origin of the interference.

Re: Flow Control, should I use it?

Posted: Sat May 07, 2016 9:18 pm
by WirelessRudy
@pukkita: We don't have an issue with errors. We see an occasional fcs error but usually that has something to do with poor cabling indeed and lately we found our installers tied an ftp cable to a 'cattle wire' sending thousands of volts (low amp) that induced into our network cable!!!!
But new to me is radio energy that might penetrate into the Ethernet connectors / cable. Is that really something that can happen? I guess so. ...
We are as much as possible using radios in metal shielded boxes and even shield antennas as much as possible against unwanted radio wave (from unwanted directions.)
Then we use for all our AP's and Backhaul radio's ftp cable and are in the process of upgrading the earth connections wherever possible. (Very hard at times since even the soils here are powder dry in the summer so even houses lack proper earth at times. We run now also a short earth cable direct from the radio toward the first next device. Plus earth to the tower that in itself also has earth connections.)

Reducing power levels is a pending task since up to some weeks ago we usually had default power settings everywhere.
In an attempt to reduce overall noise and thus interference this is our latest project (We have 50+ AP working radios in a 12km radius. Plus some 20 or the competition means each channel is used everywhere more then once.)

But anyway, we still have many complaints from people showing us that www.speedtest.net gives about half the speeds that we say we deliver to them. Running tcp bandwidth tests from within the CPE's towards our gateway always shows the full speeds assigned by the client's queue.

In the last half year while upgrading back-hauls into the latest NetMetals from MT with the best (Jirious!) antennas we can get and wherever possible we use gigabit Ethernet and giga switches and routers we see the average bandwidth usage going up, we measure real improvements in both latency and throughput with the MT bandwidth tests while at the same time we are getting more and more complaints about slow internet......
Hence we are looking for solutions/reasons why tcp toward client itself is often much worse than what we can put to its CPE?
Hence we now are thinking traffic flow might help us? But we see very mixed opinions about this..... so I don't know.

Re: Flow Control, should I use it?

Posted: Sat May 07, 2016 9:50 pm
by n21roadie
A very interesting topic! We have a mix of 100Mbit WiFi routers, 100Mb and Gb AP', switches 100 and 1Gb and PTP's Gb..., Flow control certainly makes sense unless you are using Gb equipment all the way from customers WiFi router...,

At present we are testing flow control set to "Auto" on a section of the network which has a mix of 100-1Gb equipment

http://wiki.mikrotik.com/wiki/Manual:Interface/Ethernet
....auto is the same as on except when auto-negotiation=yes flow control status is resolved by taking into account what other end advertises.....
Also using a custom queue type - pfifo, Queue Size =10 packets for pppoe server

Perception of a fast network is very important regardless of what package they have sign up for, so to this we use short term traffic bursts, most customers generally don't check speeds until they notice a connection has slowed down and we are trying to ensure they don't have a reason to complain!

Re: Flow Control, should I use it?

Posted: Sun May 08, 2016 12:44 pm
by pukkita
@WirelessRudy:

Yes, Earthing in Spain or any other country with very hot weather can be a challenge...

The best is not trusting anything already installed (unless you measure it and tests ok on a dry, hot summer day), and to install you own grounding, you can source everything needed from a Electricity supplier: special conductivity salts, and the ground rods or plates best suited for the terrain: http://www.dielectroindustrial.es/syste ... 202011.pdf

Regarding speedtest, do you have your own servers? Most servers are deployed by other (W)ISPs, so measuring using those servers will give you rather mixed results as most as you said, oversubscribe.

Are you sure yours is a backpressure problem and not a TCP connection, or fragmentation issue??

Do you use simple queues at the PPPoE AC to limit speeds?
Perception of a fast network is very important regardless of what package they have sign up for, so to this we use short term traffic bursts, most customers generally don't check speeds until they notice a connection has slowed down and we are trying to ensure they don't have a reason to complain!
I cannot agree more... I have found one of the  #1 factors to provide that "snappy" browsing experience lies on a very fast DNS cache, and making all your users to actually use it.

Re: Flow Control, should I use it?

Posted: Sun May 08, 2016 5:07 pm
by WirelessRudy
@WirelessRudy:

Yes, Earthing in Spain or any other country with very hot weather can be a challenge...
The best is not trusting anything already installed (unless you measure it and tests ok on a dry, hot summer day), and to install you own grounding, you can source everything needed from a Electricity supplier: special conductivity salts, and the ground rods or plates best suited for the terrain: http://www.dielectroindustrial.es/syste ... 202011.pdf
Thanks for the link, very interesting! Did you take the effort to find something for me in Spanish or are you Spanish? I am not, although live here in Alicante region for 16 years now.... My languages are Dutch, English, Spanish, German.... in that order..... but reading the document in Spanish is not a problem...

Regarding speedtest, do you have your own servers? Most servers are deployed by other (W)ISPs, so measuring using those servers will give you rather mixed results as most as you said, oversubscribe.
Do you refer to the http://www.speedtest.net servers? No, we don't have our own. I am thinking of it now. But my assistant thinks its a bad idea since we are only behind a 300/300Mbit line from our provider and such server would be open to everybody else in the world! (Although servers are picked after a ping test. I'd presume this will guarantee the rest of the world will not pick our server?)
I am on the other hand thinking it might be a good idea so our clients test directly to our server and thus give best results.....
Are you sure yours is a backpressure problem and not a TCP connection, or fragmentation issue??
Not sure what you mean with "back pressure" problem. To be honest, with so many variable that can be set at different places in our network and in the routers, most of them are in use for years, I am without directions.
In fact, we had an 'audit' done by some IT engineer last year and he pointed to some things we could do better but overall but we never got a good help out of that. Our network was/is too complicated for just a simple general advice.... :(
Do you use simple queues at the PPPoE AC to limit speeds?
Yes, in our CCR1016-12G that is connected to provider's Cisco we have the PPPPoE server assigning simple queues to the authenticated clients.
We have an Intel i7 3.2Ghz with 8Gb running a CHR virtual environment which in return has the Dude running and user manager to manage the clients.
In that user manager we have some profiles for the clients;
Rate limits capture.JPG
There are no priorities set for users. Because we never reach the maximum of our assigned bandwidth priorities have no use in this router (?)
The difference in what client's connections are and the burst we give them is quite big and can be for some long time. I've done this since I don't care if people make a 5 min heavy download at great speed but am not waiting people starting downloads at the highest speeds lasting for hours..... but maybe this whole setup need review after the many years we basically used this same principle (I only increased the speeds and prolonged the times of the burst together with the data thresholds...)
Perception of a fast network is very important regardless of what package they have sign up for, so to this we use short term traffic bursts, most customers generally don't check speeds until they notice a connection has slowed down and we are trying to ensure they don't have a reason to complain!
I cannot agree more... I have found one of the  #1 factors to provide that "snappy" browsing experience lies on a very fast DNS cache, and making all your users to actually use it.
I agree. But what is the best way to achieve?
Once, long time ago, I started having all clients pointing to the our gateway that would run a dns cache and resolved to the servers of my 'at the time' provider.
Than one day we found 'OpenDNS' servers actually performing much better than provider's dns (Telefonica) so we changed that in the gateway.
Than one day we ran in some issues with http://www.google.com/es becoming unreachable this way. It was solved by disabling the MT cache (in the RB1100AHx2 at the time). The forum was with several discussions about this issue.

Since then we actually started to point each CPE and every other router directly to the OpenDNS servers so request just passed the gateway router.
In the last year we tried the 'dst-nat re-route to itself' with the dns cache server in the gateway again but again after a week or so we ran into dns issues at the clients. Many pages became unresponsive so we disabled the dns cache in MT again and replaced the forewarding to the google servers. (A special test program revealed that these for us are faster than OpenDNS or any other dns server.)

So this is the situation now:
add action=dst-nat chain=dstnat comment="re-route dns requests to Google DNS" dst-port=53 protocol=udp to-addresses=8.8.8.8 to-ports=53
add action=dst-nat chain=dstnat comment="re-route dns requests to Google DNS" dst-port=53 protocol=tcp to-addresses=8.8.8.8 to-ports=53
Maybe its better to have CPE's point to the google servers directly but that takes a change in all CPE's.

Re: Flow Control, should I use it?

Posted: Mon May 09, 2016 12:53 pm
by pukkita
Thanks for the link, very interesting! Did you take the effort to find something for me in Spanish or are you Spanish? I am not, although live here in Alicante region for 16 years now.... My languages are Dutch, English, Spanish, German.... in that order..... but reading the document in Spanish is not a problem...
Main reason was to ease the local sourcing of the components, I knew you're located in Alicante. Yes I am Spanish.

Regarding your network, pointing out possible enhancements is just one part while tuning a production network... then comes actual enhancements deployment, measurements, testing and reasoning out results towards further improvements or pointing out bottlenecks and "weak rings in the chain"... it's not too unusual to unleash hidden flaws or problems while doing so.

Wireless Networks in production have a "organic" component of sorts, it is usually needed to closely watch and monitor them for an extended time, and slowly introduce changes one at a time to be able to tell, and measure the before and after difference in order to thoroughly audit it; sometimes fixing a bottleneck reveals another one upstream or downstream, thus changing all the game.

So knowing and pointing out possible enhancements is just half of the task... the other half (in value, not in time needed) being actually implementing it.

Regarding DNS cache I meant deploying a recursive resolver DNS cache locally. Google services can change rather quick, so TTL is something to specially watch out for while caching their records.

The difference of using google DNS or provider DNS vs a local, fast recursive cache difference can be of several orders of magnitude; and this is what all customers routinely use of the network starts with everytime:
Captura de pantalla 2016-05-09 a la(s) 12.07.40.png
(Cache DNS POPx = RouterOS DNS using the local recursive DNS Cache)

DNS requests are resolved by the local cache in 1/8th of the time using external DNS would take. This means slicing out 1,750ms of all requests, and more important: you're optimizing this traffic, which is amongst the most critical for a smooth, snappy network User Experience, if not the most important.

Redirection is usually the most easy way of making (forcing) everybody to use the cache, but it may not work for all your customers depending on how they are resolving DNS, usually a minority will need their ip bypassed as their resolver will complain or refuse to work when redirected.

Re: Flow Control, should I use it?

Posted: Mon May 09, 2016 7:00 pm
by ZeroByte
Great discussion going on here.

One tidbit I'd like to add is that if you're going back and forth between 100Mbps and 1Gbps several times in a path, this can cause issues if the equipment isn't well aware of it - suppose a router can transmit across a gigabit interface to some device via a switch, but the next device is a 100Mbps-eth device.

Suppose you have this:

(R1) ---gige---> [switch] -----faste---> {bridge} --faste---> [switch] ---gige--> (R2)

R1 and R2 see it as a gigabit link. Even if you queue at 100Mbps limits, there can be issues because the traffic still crosses the wire at gigabit speeds, and in small timescales, you may exceed the line rate of the 100M device links... and if the {bridge} happens to be experiencing congestion/interference/lower TX/RX rates - the problem can get worse.

You would be better off using 100Mbps links from the routers to the switches in this case so that the routers can't barf packets at the switches faster than they can clear the 100Mbps links.

I think this whole situation is referred to as "bufferbloat" (but I could be wrong on that).

Re: Flow Control, should I use it?

Posted: Tue May 10, 2016 1:50 am
by WirelessRudy
Great discussion going on here.
So keep it going! :D
One tidbit I'd like to add is that if you're going back and forth between 100Mbps and 1Gbps several times in a path, this can cause issues if the equipment isn't well aware of it - suppose a router can transmit across a gigabit interface to some device via a switch, but the next device is a 100Mbps-eth device.

Suppose you have this:

(R1) ---gige---> [switch] -----faste---> {bridge} --faste---> [switch] ---gige--> (R2)

R1 and R2 see it as a gigabit link. Even if you queue at 100Mbps limits, there can be issues because the traffic still crosses the wire at gigabit speeds, and in small timescales, you may exceed the line rate of the 100M device links... and if the {bridge} happens to be experiencing congestion/interference/lower TX/RX rates - the problem can get worse.

You would be better off using 100Mbps links from the routers to the switches in this case so that the routers can't barf packets at the switches faster than they can clear the 100Mbps links.
So, this would mean looking from your gateway 'into' your network once Gigabit is passed and 100M is reached further down the 'pipe' you'd better not use gigabit again?

And what about;

(R1) ---gige---> [router/radio] -----Wifi (200Mb rate)---> {router/radio} --gige---> [router] ---gige--> [router/radio] ---->(Wifi (100Mbrate)--->[Router/radio]--->gige--->R2)

Q1; Are units configured as router inherently different than a switch of configured as bridge (or switch funcion) in this respect?

Q2; If the 'route' of the package consist of both gige, wireless and in the end 100M. How we look at the situation now?

Q3. In this more complicated mixed topology, how would the use of 'flow control' be of any help?
(I see some supporters to the use as well as negative advices in this forum and external. So I'm left a bit in the dark.)

Re: Flow Control, should I use it?

Posted: Tue May 10, 2016 2:07 am
by WirelessRudy
Regarding your network, pointing out possible enhancements is just one part while tuning a production network... then comes actual enhancements deployment, measurements, testing and reasoning out results towards further improvements or pointing out bottlenecks and "weak rings in the chain"... it's not too unusual to unleash hidden flaws or problems while doing so.
And in upgrading the network to answer to the growing demand from customers/keeping pace or head with competition we can and will offer more speeds to the customer. Hence a network that always worked fine with 4Mb's assigned to clients that occasionally used that (apart from the usual lice using P2P) we now see people bringing back their smart TV's to start watching streaming video on demand since he, the provider says he delivers 15Mb now so why not 'eat' that.... So yes, any flaws will pop up... :o
Wireless Networks in production have a "organic" component of sorts, it is usually needed to closely watch and monitor them for an extended time, and slowly introduce changes one at a time to be able to tell, and measure the before and after difference in order to thoroughly audit it; sometimes fixing a bottleneck reveals another one upstream or downstream, thus changing all the game.
I agree. I'ts alive, it's alive. :D
But yes, fix one thing just to run into something new.....
Regarding DNS cache I meant deploying a recursive resolver DNS cache locally. Google services can change rather quick, so TTL is something to specially watch out for while caching their records.

The difference of using google DNS or provider DNS vs a local, fast recursive cache difference can be of several orders of magnitude; and this is what all customers routinely use of the network starts with everytime:

DNS requests are resolved by the local cache in 1/8th of the time using external DNS would take. This means slicing out 1,750ms of all requests, and more important: you're optimizing this traffic, which is amongst the most critical for a smooth, snappy network User Experience, if not the most important.

Redirection is usually the most easy way of making (forcing) everybody to use the cache, but it may not work for all your customers depending on how they are resolving DNS, usually a minority will need their ip bypassed as their resolver will complain or refuse to work when redirected.
Well, just switched the gateway redirector on again. See for how long the dns cache will do good to us.

In relation to this; How about same local cache dns in some nodes further down in our network? In some major node that interconnect several backhauls towards AP's we have some excess rb1000, 1100AH, 2011 etc setup for doing the OSPF and VPLS/MPLS routing and failover. These could take the extra task? Would it enhance the resolving? Measurably? (These caches will have to point towards the main router, well the redirect will take care of that...)

Re: Flow Control, should I use it?

Posted: Tue May 10, 2016 5:58 am
by changeip
Since then we actually started to point each CPE and every other router directly to the OpenDNS servers so request just passed the gateway router.
In the last year we tried the 'dst-nat re-route to itself' with the dns cache server in the gateway again but again after a week or so we ran into dns issues at the clients. Many pages became unresponsive so we disabled the dns cache in MT again and replaced the forewarding to the google servers. (A special test program revealed that these for us are faster than OpenDNS or any other dns server.)
Dont do this. Run your own DNS resolvers inside your own network. They query the root and aren't leaching another providers DNS infrastructure. I used to be a dns provider and it sucks when everyone sucks your resources for no return.

If you are really fancy you run anycast DNS inside your network. Advertise a single address in multiple places using ospf (100.100.100.100 is a great IP to use) and it will automatically serve it from the closest location.

PS - use BIND or something similar, do not use MT resolvers built into the routers.

Re: Flow Control, should I use it?

Posted: Tue May 10, 2016 9:41 am
by pukkita
totally subscribe the DNS Anycast idea...

@Wirelessrudy: I meant a dedicated recursive resolver cache, using ROS would be better than using an external one, however ROS DNS cache has its limits: it cannot resolve more than 100 concurrent requests, and albeit performing great (see the graph), isn't on par with a tuned, dedicated cache.

I have stayed away from BIND (just like sendmail) since the late nineties, my cache of choice is dnscache from Daniel J Bernstein (qmail author). But there are several modern implementation of lightweight, modern features DNS resolvers like MaraDNS, or PowerDNS recursor that seem to perform great.

I just did a test recently and... I'm staying with dnscache. Needs some patches for nowadays DNS, and some tender, but is the fastest and more lightweight implementation I've tested.

MaraDNS has even an OpenWRT package, I think for djbdns too, you could use a metarouter instance f you feel like testing... otherwise a regular PC with FreeBSD or Linux will do.

Re: Flow Control, should I use it?

Posted: Tue May 10, 2016 11:21 am
by WirelessRudy
Well, I switched the 'redirect to my own' rule on yesterday to be woken up this morning with several clients complaining their internet is down. After some checks found their resolving stopped working. (Google presenting the 'unvalid certificate' message???, other pages simply didn't open. Even when given the IP address????).
So switched it back off and gone were the client's issues.
Many other clients including our own office systems weren't effected....? How come?

Redirect rules we'd used;
/ip dns
set allow-remote-requests=yes cache-max-ttl=1d cache-size=2048KiB max-udp-packet-size=4096 query-server-timeout=2s query-total-timeout=10s servers=8.8.8.8,8.8.4.4,208.67.222.222,208.67.220.220

/ip firewall nat
add action=redirect chain=dstnat comment="re-route dns requests to Google DNS" dst-port=53 protocol=udp to-ports=53
add action=redirect chain=dstnat comment="re-route dns requests to Google DNS" dst-port=53 protocol=tcp to-ports=53
Anything wrong in this?
Or could it be this is the ROS DNS issue? No more than 100 concurrent requests?
But why would it not work for clients some hobs away where I can open as many pages i'd like without any problems?

Re: Flow Control, should I use it?

Posted: Tue May 10, 2016 11:35 am
by WirelessRudy
Just opened a new tread on the dns discussion; http://forum.mikrotik.com/viewtopic.php ... 01#p537093
We were drifting off topic.

Here I'd like to talk more about the 'flow control' and or mtu settings and or mixed 100M/1000M ethernet in our network

Re: Flow Control, should I use it?

Posted: Tue May 10, 2016 11:45 am
by WirelessRudy
@zerobyte;

Any input on the mixed 100M/1000M theme?
Is it indeed a better idea that once the route from gateway to client has seen the first 100M cable it is not wise to use 1000M in the network further down to the client?

But what is the client itself installs gigabit in his LAN?
Or it the 'mixture' issue only an issue if we use switched or bridged interfaces in a route?

And what about the transfer from Ethernet to wireless?
I can imagine traffic that comes from an internet server at full speed (initially 300Mb from my provider) running over gigabit cables to my CCR and then to a NetMetal will see its first bottleneck in a 200 conn. rate wireless link. After that again some gigabit cables and another CCR and more gigabit cable to the next NetMetal that serves a 100Mb link. Finally further down the road we will even come accross a 100M cable and the P2MP network will only deliver some 10, 20, 40?? Mbs to the client?
All in all, at the first CCR, my gateway, the PPPoE server assigned a single queue for that specific client that limits the speed to 10, 20 or 30Mbps.......

So how is the package behaviour in respect of 'mixed speed' environment now?

Re: Flow Control, should I use it?

Posted: Tue May 10, 2016 12:18 pm
by WirelessRudy
Also opened another tread in relation to our issue but off tread of this one; http://forum.mikrotik.com/viewtopic.php?f=2&t=108202

Re: Flow Control, should I use it?

Posted: Tue May 10, 2016 12:44 pm
by juanvi
Flow Control, should I use it?

IMHO: NO. Now we have VoIP and great and more modern QoS systems thath can hadle this type of traffic the right way.

extract from wiki: https://en.wikipedia.org/wiki/Ethernet_flow_control

It is an old mechanism, it was the FIRST mechanism! : "The first flow control mechanism, the PAUSE frame, was defined by the Institute of Electrical and Electronics Engineers (IEEE) task force that defined full duplex Ethernet link segments. The IEEE standard 802.3x was issued in 1997"

The original motivation for the pause frame was to handle network interface controllers (NICs) that did not have enough buffering to handle full-speed reception.

Ethernet Flow control disturbs the Ethernet class of service (defined in IEEE 802.1p), as the data of all priorities are stopped to clear the existing buffers which might also consist of low priority data.Ethernet Flow control disturbs the Ethernet class of service (defined in IEEE 802.1p), as the data of all priorities are stopped to clear the existing buffers which might also consist of low priority data.

Re: Flow Control, should I use it?

Posted: Tue May 10, 2016 2:19 pm
by WirelessRudy
Flow Control, should I use it?

IMHO: NO. Now we have VoIP and great and more modern QoS systems thath can hadle this type of traffic the right way.

extract from wiki: https://en.wikipedia.org/wiki/Ethernet_flow_control

It is an old mechanism, it was the FIRST mechanism! : "The first flow control mechanism, the PAUSE frame, was defined by the Institute of Electrical and Electronics Engineers (IEEE) task force that defined full duplex Ethernet link segments. The IEEE standard 802.3x was issued in 1997"

The original motivation for the pause frame was to handle network interface controllers (NICs) that did not have enough buffering to handle full-speed reception.

Ethernet Flow control disturbs the Ethernet class of service (defined in IEEE 802.1p), as the data of all priorities are stopped to clear the existing buffers which might also consist of low priority data.Ethernet Flow control disturbs the Ethernet class of service (defined in IEEE 802.1p), as the data of all priorities are stopped to clear the existing buffers which might also consist of low priority data.
Ok, with your post and reading up a bit more it seems the camp of 'No, don't use it!' is winning.......

Regarding QoS; We only have simple queues (by pppoe server) for clients. No queue tree. The simple queues only limit the client's assigned bandwidth. No priority is set.
We have no queue tree.

We don't use priority (yet) and no (more, we did in the past) Queue tree since the never use the provider's 'pipe' up to its full capacity anyway.
In regard to the 'down the Xtrmas tree' into our network, all wired links and routers are capable of handling a multitude of traffic we ever will send to our clients. (In theory max. 300Mb aggregated clients traffic) so we'd presumed we don't need queue tree for incoming traffic?

The latter might see some discussion, after all, at some point the traffic will hit a bottle neck, usually the AP.
Should we set queue tree in each AP?
Is that not putting too much strain on the cpu of an OmniTik? (We have faster AP's but several are still OmniTiks)
To set a queue tree on an AP we need to know the capacity of an AP. How to measure in heavens name to capacity of an AP?
We can run a sigle bandwidth test from a client towards a router behind AP (so we download traffic from that remote unit into a device behind client's CPE) and get some max. speed.
Then we can do that for 2, 3 or more CPE's at the same time. The total will run up but each client gets less now.
This total is always more than what a single client can achieve.
2, 3 or even 4 clients always get some more than for instance 10 or 15 clients at the same time.
So what is the best procedure to find out the max bandwidth throughput for an AP?

Let's say we can achieve 50Mbps max towards some clients at the same time. The 'rule of thumb' was alwayst use 70-80% of that for the mother queue to have some spare and for margin.
So we set 40Mbps for an AP? Is that not very little for an AP that might serve up to 30 clients that in total would be allowed (15 x 20 + 15 x 6 = 390Mbps!)

Re: Flow Control, should I use it?

Posted: Tue May 10, 2016 4:45 pm
by ZeroByte
@zerobyte;

Any input on the mixed 100M/1000M theme?
Is it indeed a better idea that once the route from gateway to client has seen the first 100M cable it is not wise to use 1000M in the network further down to the client?

But what is the client itself installs gigabit in his LAN?
Or it the 'mixture' issue only an issue if we use switched or bridged interfaces in a route?

And what about the transfer from Ethernet to wireless?
I can imagine traffic that comes from an internet server at full speed (initially 300Mb from my provider) running over gigabit cables to my CCR and then to a NetMetal will see its first bottleneck in a 200 conn. rate wireless link. After that again some gigabit cables and another CCR and more gigabit cable to the next NetMetal that serves a 100Mb link. Finally further down the road we will even come accross a 100M cable and the P2MP network will only deliver some 10, 20, 40?? Mbs to the client?
All in all, at the first CCR, my gateway, the PPPoE server assigned a single queue for that specific client that limits the speed to 10, 20 or 30Mbps.......

So how is the package behaviour in respect of 'mixed speed' environment now?

Think of this device:
220px-Wooden_hourglass_3.jpg
... and all should be clear - Note that the sand falls quite freely though the lower large portion of the hourglass. It doesn't really matter how fast the network is beyond the bottleneck - if your network can push too much traffic up against a bottleneck then you get this effect. If the sand were only allowed to fall into the upper half of the hourglass at the same rate that it drains through the neck, then there wouldn't be a pile of sand in the top half.

There's always going to be a bottleneck, but it's how well the bottleneck handles traffic and how well the devices feeding data into the bottleneck handle this that can make the difference between smooth performance and jittery performance.

In general, a traffic shaper will give much better results than a hard limitation that simply fifo queues a few Kilobytes of data. The reason I said "router" in my example is that most routers have a lot of qos and traffic management tools at their disposal, and many switches/bridges have only limited tools to this effect. For instance, Ubiquiti gear will prioritize certain traffic based on DSCP values - if the router is classifying and marking important traffic with high-priority DSCP values, then the wireless gear will behave better during congestion than it would if there were no QoS at all.

If you were to implement a queue tree on the R1 router which shapes the traffic to fit well through the 100Mbps links (I usually shape just a tad slower than the physical bottleneck) then you're going to have more control over performance than if a simple store-and-forward switch receives frames too fast to write them out the 100M interface.

Obviously, there's no one-size-fits-all rule of thumb here - suppose you have a gig-e link into a switch and there are 4 possible 100Mbps paths that traffic could take beyond the switch - you can't just shape to 100M and call it a day.

Re: Flow Control, should I use it?

Posted: Tue May 10, 2016 6:04 pm
by WirelessRudy
@zerobyte;

Any input on the mixed 100M/1000M theme?
Is it indeed a better idea that once the route from gateway to client has seen the first 100M cable it is not wise to use 1000M in the network further down to the client?

But what is the client itself installs gigabit in his LAN?
Or it the 'mixture' issue only an issue if we use switched or bridged interfaces in a route?

And what about the transfer from Ethernet to wireless?
I can imagine traffic that comes from an internet server at full speed (initially 300Mb from my provider) running over gigabit cables to my CCR and then to a NetMetal will see its first bottleneck in a 200 conn. rate wireless link. After that again some gigabit cables and another CCR and more gigabit cable to the next NetMetal that serves a 100Mb link. Finally further down the road we will even come accross a 100M cable and the P2MP network will only deliver some 10, 20, 40?? Mbs to the client?
All in all, at the first CCR, my gateway, the PPPoE server assigned a single queue for that specific client that limits the speed to 10, 20 or 30Mbps.......

So how is the package behaviour in respect of 'mixed speed' environment now?

Think of this device:
220px-Wooden_hourglass_3.jpg
... and all should be clear - Note that the sand falls quite freely though the lower large portion of the hourglass. It doesn't really matter how fast the network is beyond the bottleneck - if your network can push too much traffic up against a bottleneck then you get this effect. If the sand were only allowed to fall into the upper half of the hourglass at the same rate that it drains through the neck, then there wouldn't be a pile of sand in the top half.

There's always going to be a bottleneck, but it's how well the bottleneck handles traffic and how well the devices feeding data into the bottleneck handle this that can make the difference between smooth performance and jittery performance.

In general, a traffic shaper will give much better results than a hard limitation that simply fifo queues a few Kilobytes of data. The reason I said "router" in my example is that most routers have a lot of qos and traffic management tools at their disposal, and many switches/bridges have only limited tools to this effect. For instance, Ubiquiti gear will prioritize certain traffic based on DSCP values - if the router is classifying and marking important traffic with high-priority DSCP values, then the wireless gear will behave better during congestion than it would if there were no QoS at all.

If you were to implement a queue tree on the R1 router which shapes the traffic to fit well through the 100Mbps links (I usually shape just a tad slower than the physical bottleneck) then you're going to have more control over performance than if a simple store-and-forward switch receives frames too fast to write them out the 100M interface.

Obviously, there's no one-size-fits-all rule of thumb here - suppose you have a gig-e link into a switch and there are 4 possible 100Mbps paths that traffic could take beyond the switch - you can't just shape to 100M and call it a day.
Yeah ok, but does it hurt if behind the bottle neck (that could well be a 100M cable) we find another stretch of the path formed by a gigabit cable?
From your analogy I would think it doesn't matter. But through some posts in this forum I got a bit the impression it does .....

Re: Flow Control, should I use it?

Posted: Tue May 10, 2016 6:11 pm
by ZeroByte
In general, no, if you have a 100M device going through a switch which forwards along a 1G interface, there's no way for your device to overflow the capacity of the 1G interface so it doesn't hurt to go from smaller to larger, even with "dumb" devices like workgroup switches, media converters, etc.

Just realize that networks aren't uniderectional - so in the return direction, things are going from fast to slow.....

Anyway, if a link looks like this:

----======------ then you're never going to overflow the buffers at the 100/1g boundaries because it's 100M at the ends...

if it looks like this:

====-----===== then you could possibly run into problems.

Re: Flow Control, should I use it?

Posted: Tue May 10, 2016 9:12 pm
by WirelessRudy
In general, no, if you have a 100M device going through a switch which forwards along a 1G interface, there's no way for your device to overflow the capacity of the 1G interface so it doesn't hurt to go from smaller to larger, even with "dumb" devices like workgroup switches, media converters, etc.

Just realize that networks aren't uniderectional - so in the return direction, things are going from fast to slow.....

Anyway, if a link looks like this:

----======------ then you're never going to overflow the buffers at the 100/1g boundaries because it's 100M at the ends...

if it looks like this:

====-----===== then you could possibly run into problems.
OK, clear as a bone.. but what about the mix with wireless?
Is ======/\/\/\/\===== poor when the /\/\/\/\ is a wirless link with 130Mb conn. rate and simplex?

Re: Flow Control, should I use it?

Posted: Tue May 10, 2016 10:17 pm
by ZeroByte
Nice wireless ASCII art. :)

Worst case, try setting the interfaces to 100M (if the wireless link is at or below 100M) - otherwise, I don't think there's a lot on the physical link that you can/must do. Any tweaking you investigate should come in the form of queues/traffic shaping.

Re: Flow Control, should I use it?

Posted: Wed May 11, 2016 12:34 am
by WirelessRudy
Nice wireless ASCII art. :)
8)
Worst case, try setting the interfaces to 100M (if the wireless link is at or below 100M) - otherwise, I don't think there's a lot on the physical link that you can/must do. Any tweaking you investigate should come in the form of queues/traffic shaping.
Ok, conclusion sort of is;

a. Seen in download direction (= to client = usually 80% of all traffic) each section of the link should go down in speed, not up. Behind a 50Mb wireless link it serves no purpose to use gigabit cable but also doesn't hurt.

b. Hardware flow control is not recommended. It disturbs tcp speed regulating and voip sessions. If the rest of the network runs fine, don't use it.

Can you agree on that?

Next discussion. MTU and indeed QoS.

MTU is in another tread http://forum.mikrotik.com/viewtopic.php ... &hilit=mtu

QoS; Lets start another tread: http://forum.mikrotik.com/posting.php?mode=post&f=2

Anymore input from others on the 'flow control' theme is still appreciated though. :)

Re: Flow Control, should I use it?

Posted: Tue May 17, 2016 8:22 pm
by ericsj
I know this is a bit late to the conversation but the Linux world solved this using the fq_codel and cake tools. I've used them for a while and it's really nice how it handles variations in flow, backups further in the net etc. automatically. It makes QOS in a no-brainer.

Is cake missing from RouterOS?

Re: Flow Control, should I use it?

Posted: Tue Jun 21, 2016 11:18 pm
by amt
Chk