Community discussions

MikroTik App
 
User avatar
homerwsmith
Member Candidate
Member Candidate
Topic Author
Posts: 152
Joined: Fri Dec 02, 2011 3:01 am
Location: Ithaca, NY
Contact:

Speedtest firewalling and redirecting

Wed Nov 30, 2016 9:21 am

HI Folks,

This is a highly sensitive subject, so please try to help if you can independent of your views on the matter.

I wish to redirect speedtest.net to a local speedtest at our noc.

We are a small WISP with a 100meg Time Warner light fiber line that we use to feed the boonies.

We do not limit how much bandwidth an end user gets, they get what they get, which is a lot for some
and little for others. This is will change in the near future. However sometimes the net slows down, sometimes our pipes saturate, sometimes both slow down like during recent elections.

Our customers are not very bright, they use speedtest.net to measure their speeds and do not understand that speedtest.net has
nothing to do with their local link, particularly when their local link IS faster than the net as a whole.

I have provided okla's mini speedtest on our our internal links, so I can redirect the customer to test them, to determine if the problem is with us or with the net as a whole which I can not do anything about.

I wish to know how to block access to speedtest entirely or to redirect it, but it seems to use a very complicated set of IP numbers and
now maybe proprietary protocols.

I am also concerned that when a user picks a local server to test against, it picks one at a local competitor who has set up a server for people to test against. Results from that server are generally poor during the say compared to a larger ISP in Syracuse for example.
I have always thought that by downloading from that closest server we are testing their upload, and by uploading to the server we are testing their download. When they are full during the day it makes us look awful.

However in looking at the actual tcpdump of the process, the tests are actually done between 10 or more different widely separated servers and thus are not connected directly to the competitors network at all while the testing takes place. So I don't really understand how speedtest.net picks its many servers for each test, nor why the speeds should vary so greatly according to which testing server I test to on speedtests home page as the tests don't go to that network at all anyhow. What am I missing?

Anyhow if anyone is wiling to share some insight into blocking or redirecting requests to speedtest.net and the many other speedtesting sites I would be obliged. I know firewaalling pretty well, but not the exact structure of speedtest.net The redirect might lead customers to a web page that would instruct them on how the system works, and allow them to then
click on any testing service they wish or their local link, once they understand the difference.

Thanks Homer Smith
CEO LIghtlink Internet
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1048
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: Speedtest firewalling and redirecting

Wed Nov 30, 2016 3:46 pm

re: I wish to redirect speedtest.net to a local speedtest at our noc.

Bad idea - very very bad idea

It sounds like you are selling customers one thing but tricking them into something else - almost fraud.

What you can do is setup a local speedtest server using software you can download from speedtest.net and run your own speedtest server for your local customers.

North Idaho Tom Jones
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1048
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: Speedtest firewalling and redirecting

Wed Nov 30, 2016 3:51 pm

Your customers are paying you for a certain speed to/from the Internet

They are not paying for a certain speed to/from you and then 1/100th the speed to the Internet.

Customers should be able to use any and all speedtest servers they want and verify you are delivering to your customers what you are selling them.

If you can not deliver to all of your active customers what they are paying for - then it is time for you to upgrade your network or get rid of some customers.
 
User avatar
omega-00
Forum Guru
Forum Guru
Posts: 1167
Joined: Sat Jun 06, 2009 4:54 am
Location: Australia
Contact:

Re: Speedtest firewalling and redirecting

Wed Nov 30, 2016 4:34 pm

I think you're kidding if you don't think the large ISPs don't do this already Tom. Large L7 DPI systems have specific profiles specifically for things like speedtests and BitTorrent to affect traffic speeds (not to mention forcibly cache 'uncachable' content).

With that said, Homer asked for input without bringing personal viewpoints into it.

Homer: yes it is possible but you should be aware that it could break speedtests if one server is updated and the requests are being made to files that don't exist on your local server.

Alternatively you can use L7 filters to identify and prioritize traffic to speedtest servers - here's an example of an overmatching L7 filter that you could use to identify the IP addresses of speedtest servers:
/ip firewall layer7-protocol
add comment="Always TCP: No fixed port" name=speedtest-servers regexp="^.*(get|GET).+speedtest.*\$"
/ip firewall filter 
add chain=forward layer7-protocol=speedtest-servers protocol=tcp action=add-dst-to-address-list address-list=speedtest-servers
Also as a rule I would try and pay for your own ookla speedtest server (they can facilitate logging of all tests done which helps your support team) and encourage users to:
A) test while cabled into their router
B) test 3 times to get an average.

Otherwise you could do something similar to redirect users to a help page the first time they try to reach speedtest.net - then allow them to continue through after acknowledging the message presented.
brightwifi.com | mikrotik-routeros.com | MTCNA,MTCWE.MTCTCE | Give karma where due
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1048
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: Speedtest firewalling and redirecting

Wed Nov 30, 2016 6:13 pm

omega-00

I am not trying to attack, I am trying to warn and suggest a solution that will make customers happy with the service.

I've been in the ISP industry for more than 20+ years. I have been in the same environment where my customers have saturated my primary Internet feed.

Re-directing traffic intended to go to/from an outside Internet location to one of your local servers with the intent of providing false information is against the law in almost every country including here in the United States.

It would be a better policy to do any or all of the following:
- upgrade
- explain to customers and make account policies and expected throughput public information (make sure customers understand what they are paying for and what they can expect)
- and yes - setup a local speedtest server so that customers can test to their local ISP while still permitting customers to be able to connect to any legal Internet server anywhere they want.

ISP companies that do not provide the bandwidths they are selling their customers are getting the attention of the FCC. That is why the FCC is pushing Net neutrality regulation.

Re-directing Internet traffic to a local server to provide misleading information is one of the newest and hottest topics that just recently and heavily discussed at ICANN in India just a few weeks ago. They are suggesting secure methods to help prevent hi-jacking and re-directing Internet sites (such as what started this set of posts).

North Idaho Tom Jones
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1048
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: Speedtest firewalling and redirecting

Wed Nov 30, 2016 6:24 pm

FYI:
My director often attends FCC meetings
My director is a member of ICANN and was at the last meeting in India
We are going to be participating in a multi-year on-going official government sponsored program where they actively monitor bandwidths and auto-compare those measured bandwidths to the account speeds customers are paying for. (the servers only measure bandwidths and do not capture any from/to destination traffic).

Note - also ... With the new highly recommended secure DNS being pushed by ICANN, customers will automatically get warnings when an Internet located server is DNS hi-jacked and/or redirected to a different server.

North Idaho Tom Jones
 
msatter
Forum Guru
Forum Guru
Posts: 1703
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: Speedtest firewalling and redirecting

Wed Nov 30, 2016 11:21 pm

Where I live DNSSEC has reached a high implementation grade however the browsers don't use this information to inform the users.

Secure DNS is something I recognize not immediatly and I see a link yo DNS crypt or ate you referring to DNSSEC?
One RB4011 (cooled) and a RB760iGS (hEX S) in series. The 4011 Does PPPoE/IKEv2.
The cooler: viewtopic.php?f=3&t=138613&start=300#p799879
Running:
RouterOS 6.47 / Winbox 3.24 / MikroTik APP 1.3.14
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1048
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: Speedtest firewalling and redirecting

Thu Dec 01, 2016 1:48 am

Where I live DNSSEC has reached a high implementation grade however the browsers don't use this information to inform the users.

Secure DNS is something I recognize not immediatly and I see a link yo DNS crypt or ate you referring to DNSSEC?

I am not sure which it is called - secure dns or dnssec

I would have to review the on-line meeting notes for the last ICANN that was in India
 
belsamber
just joined
Posts: 1
Joined: Thu Dec 01, 2016 5:02 am

Re: Speedtest firewalling and redirecting

Thu Dec 01, 2016 5:23 am

As others have said, I think redirecting all of speedtest.net will be a world of pain, but it's free to run a speedtest.net server, and you end up with a URL like lightlink.speedtest.net that you can pass to your customers.

Combine that with some education - we publish articles for our customers on topics like this. Our support team refer customers to the article when needed. Saves support time, and published articles feel "more formal."

I'm always in favour of setting expectations correctly and providing tools to prove your point rather than trying to fake results or make the network solve what is fundamentally a people problem.
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1048
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: Speedtest firewalling and redirecting

Thu Dec 01, 2016 5:45 am

Re: Alternatively you can use L7 filters to identify and prioritize traffic to speedtest servers - here's an example of an overmatching L7 filter that you could use to identify the IP addresses of speedtest servers:

The more I think about what you suggest ....

This looks like a classic example of deliberately changing priorities and packet flow with the designed and sole intent to mislead customers and what they are paying you for !!!

How would you like it if your upstream provider did this to you. Where you measure you have a 100 meg up/down speedtest - but in real life your upstream ISP network is totally saturated and you could only download in real life at less than 500k ??? I don't know about you - but if my upstream internet provider did this to me I would file a huge law suit and demand millions back because of the damages they deliberately created to me. my business, my reputation and my customer damages as well.

Is it worth it to cheat and mislead what you are selling to your customers and to furthermore use a software configuration designed to fool/trick the tester that they are getting something which they will never get in real life?

VW did something like this with a software program designed and intended to skew the testing results. All of the lawsuits might bankrupt them.

North Idaho Tom Jones
 
User avatar
homerwsmith
Member Candidate
Member Candidate
Topic Author
Posts: 152
Joined: Fri Dec 02, 2011 3:01 am
Location: Ithaca, NY
Contact:

Re: Speedtest firewalling and redirecting

Mon Feb 13, 2017 10:35 pm

I know this is old but it is still important to me.

I can't guarantee what I can't conttrol.

AFAIK Bandwidth is always measured and offered from point A, the customer's home, to point B, perhaps the ISP's border router.

One can not promise speeds from customers to the 'internet' because the internet is not a defined place. Speeds change
depending on where you go. That's why even speedtest has many different sites to test TO from your home location. And they all
test differently and sometimes some test really bad which in our case is most always the 'closest' one. This creates a tech support storm where in customers take the slow test at its word and start complaining.

We sell local loops, just like Verizon sells DSL, always have and always will. We offer a minimum best effort of 5x1.5 wireless local loop to the nearest core router, but since our connections are not capped, they often get way more. In our printed material we offer that SD Netflix should work without interruption on normal nights between 7pm and 11pm. If that does not obtain we try to fix it.

We do offer on our home page multiple local speedtest servers to help customers understand where in the network they are, and to help
determine where a bottle neck may be along a chain of local routers back to our core.

Be sides the fact we sell local loops and not guarantees to the 'internet', I got concerned because speedtest.net picks as a closest server a competitor of ours whose up and down stream pipes are FULL and reported speeds to our customers were uniformly slow. At the time I thought that the testing was done directly to the speed test server at the remote location only, and thus all I was testing was the speed between the
remote location and ourselves. That's really the only way I could explain the large discrepancies in reports from different locations.

I still don't know how speed test works, but it seems to be controlled by a remote server, but actually somehow manages to coral a lot of transfers from a lot of remote sites at the same time, thus giving an average 'across the net'. If that's the truth then i still don't understand why different remote sites give wildly different readings for up and down.

In any case I don't like speedtest.net, somehow it seems to be a scam to make customers not like their own ISP, and since they do not offer a clear statement of HOW they test their speeds, the lack of transparency is to me is a non starter.

Homer W Smith CEO
Lightlink Internet
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1048
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: Speedtest firewalling and redirecting

Tue Feb 14, 2017 8:36 pm

Homer

FYI - I just ran some speedtests to your speedtest sites. I thought you might want to see what I am getting to your speedtest servers.

Here are my results to your speedtest servers:
speedtestfv.lightlink.com (Fairview) 13.35 down & 7.57 up
speedtestch.lightlink.com (Conn Hill) Could not connect
speedtestax.lightlink.com (South Hill Business Campus) 2.92 down & 1.74 up
speedtestrp.lightlink.com (Roy Parks) 9.91 down & 4.07 up
speedtest.net (Internet) 961.55 down & 945.87 up
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1048
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: Speedtest firewalling and redirecting

Tue Feb 14, 2017 9:03 pm

Homer

One thing you might want to consider... A caching Squid proxy server.

Back in the mid-late1990s, I had a 56k Frame-Relay Internet connection to my house. I built up a Squid caching Proxy server. It would cache Internet web pages and deliver locally from the proxy server if the remote Internet content had been recently accessed. Thus the first person to pull up a video-clip would get it at 56k, then a second computer could then access the same web page at 100 meg. It worked pretty good and it even reduced network traffic on my WAN because the cache would start filling up Internet web pages to my local server.

Around 2004 when I had a 45 meg Internet connection, I again built up another Squid caching proxy server (bigger/faster server) for my ISP customers. Many of the customers on my fiber network were able to pull up Internet web page traffic at nearly 1-gig (when the data was already locally cached because somebody else had recently accessed the remote web sites).

The idea of a caching proxy server may be worth taking a look at. All you need is the physical server, all the software is free/open-source.

There are two basic ways to build a caching proxy server.
- 1 , an optional setting in the customers web browser to use it
- 2 , a forced setting where everybody uses it no matter what their browser setting are (unless they are pointing to another proxy server somewhere else)

North Idaho Tom Jones
 
User avatar
pukkita
Trainer
Trainer
Posts: 3037
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: Speedtest firewalling and redirecting

Tue Feb 14, 2017 9:52 pm

Interesting discussion, couldn't resist to stop by and chime in :)

I'm with TomjNorthIdaho, cheating your customers is never a good idea.

Install your local speedtest server, and educate your customers as has been already said, you cannot control speeds to all servers around the world. Some (W)ISPs will cap bw to their speedtest server from the "outside", or de-prioritize it, so having your own is always a good idea.

Having a speedtest server won't be a solution for serious lack of bandwidth though, people could be non tech-savvy, but they aren't fools; if they aren't happy with your service, they will fly away, no matter if speedtest reports great speed.

Apply QoS policies to make your traffic as smooth as possible. QoS is not a substitute for real bandwitdh however, though it can "pack" more customers for a given BW.

In my experience customers rarely do speedtests if they feel browsing is "snappy" (and QoS helps a lot in that regard).

There always be those that cannot sleep unless they have their line fully saturated 24/7, if you don't have the bw resources to keep them... you'd better let them go, or charge accordingly.

IMHO, most traffic nowadays, more than 60% at least from my experience around here, is HTTPs (uncacheable), so caching has a limited usefulness; it can be useful anyway to cache OS updates, etc (with some tinkering, not so sure if everything has moved to HTTPS).

This requires a powerful server with heaps of RAM and fast HDDs, which may not be the highest cost of deploying caching, but... maintenance.

BW nowadays is one of the cheapest variables; so IMHO, unless you have transit limitations, investing on it is the best expense that can be done.
Simplicity is the Ultimate Sophistication - Da Vinci
Getting the most out of this forum
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1048
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: Speedtest firewalling and redirecting

Tue Feb 14, 2017 10:25 pm

[quote="pukkita"]Interesting discussion, couldn't resist to stop by and chime in :)
... HTTPs (uncacheable), so caching has a limited usefulness; it can be useful anyway to cache OS updates, etc (with some tinkering, not so sure if everything has moved to HTTPS).
...
This requires a powerful server with heaps of RAM and fast HDDs, which may not be the highest cost of deploying caching, but... maintenance.
...

re: HTTPs and proxy
I am not sure - but I think it Squid is configurable to work with both HTTP and HTTPS traffic.
And yes - the initial Squid server does require some tinkering to get it working optimal. (download sizes (min & max) to qualify for local cache buffering, percentage of download to quality for cache buffering, expire time & . . .) However the default settings are often enough to get it running.
When I ran mine, I made it a user selectable setting option a customer could use if they wanted. Then if there was a problem, a customer could always disable it.

re: QOS
QOS is not a total solution to saturated or slower than needed network feed - but it does help.
Perhaps both QOS and a caching proxy at the same time, could help customer throughput together.

QOS is pretty straight forward to configure.
Squid Proxy may take some thought to configure beyond the default setting. If I were to try it again, I would try a fast Xeon multi-core server with at 64 to 256 Gig of RAM with some 4-TB SAS drives, running a LTS Ubuntu.
The good thing is, after QOS & Squid either-one/or-both configured correctly, it is pretty much hand-off and just let it run.

North Idaho Tom Jones
 
User avatar
pukkita
Trainer
Trainer
Posts: 3037
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: Speedtest firewalling and redirecting

Wed Feb 15, 2017 11:30 am

re: HTTPs and proxy
I am not sure - but I think it Squid is configurable to work with both HTTP and HTTPS traffic.
Sorry I didn't express myself clearly, I refer on "transparent proxying" scenarios, HTTPs is not cacheable: would pop up a warning on the customer browser as someone is trying to impersonate the HTTPS site.

To use it, customers need to be aware of it, and install proper certificate, which depending on scenario it may not be the best solution.
Simplicity is the Ultimate Sophistication - Da Vinci
Getting the most out of this forum
 
enlace101
just joined
Posts: 6
Joined: Thu Jan 03, 2019 8:03 pm

Re: Speedtest firewalling and redirecting

Sun Jun 28, 2020 10:44 pm

What is the opinion of using a bequant server?
I am testing it now.
the quality increases considerably.
I am afraid it is a false effect.
take a look and tell me if it really works
thank you

Who is online

Users browsing this forum: anav, bpwl, Danielmhmdi, Google [Bot], msatter, pe1chl and 54 guests