Community discussions

MikroTik App
 
adminbas
just joined
Topic Author
Posts: 10
Joined: Thu Aug 10, 2017 8:54 am

fetch via 301 redirect

Fri Mar 11, 2022 9:03 pm

Hi scripting guys
I need to create a script that will download a file from source that presented as 301 redirect in DNS
Data.mydomain.com —A ->301-> https://mysite/file.rsc
Fetch data.mydomain.com gives an error to download file.rsc due to 301 redirect usage
Any ideas what to do?
 
User avatar
smyers119
Member Candidate
Member Candidate
Posts: 232
Joined: Sat Feb 27, 2021 8:16 pm
Location: USA

Re: fetch via 301 redirect

Sat Mar 12, 2022 2:39 am

A 301 is a permanent redirect, why wouldn't you just point to the new url?
 
adminbas
just joined
Topic Author
Posts: 10
Joined: Thu Aug 10, 2017 8:54 am

Re: fetch via 301 redirect

Sat Mar 12, 2022 12:03 pm

A 301 is a permanent redirect, why wouldn't you just point to the new url?
cause its a script to download script in a case of global updates on many routers. I can change 301 redirect to any hosting I'll get at that time
I dont know where download part will be at that time
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1041
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: fetch via 301 redirect

Sat Mar 12, 2022 12:58 pm

Hi scripting guys
I need to create a script that will download a file from source that presented as 301 redirect in DNS
Data.mydomain.com —A ->301-> https://mysite/file.rsc
Fetch data.mydomain.com gives an error to download file.rsc due to 301 redirect usage
Any ideas what to do?

First of all DNS redirect is not a standard and can only be achieved using specialized services (ala hacked/extended DNS server). You need to find out exactly how the DNS response is constructed in order to check if it can be parsed by the client side (ie Mikrotik). Secondly as someone stated before, HTTP 301 is normally used for permanent redirects thus you should be able to parse the redirect manually and then use that result in your script. If you claim it might change from time to time it's not a permanent redirect.

Check this service as an example: https://cp.enom.com/kb/kb/kb_0352-creat ... direct.htm
Last edited by Larsa on Sat Mar 12, 2022 11:47 pm, edited 2 times in total.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: fetch via 301 redirect

Sat Mar 12, 2022 7:38 pm

If you control data.mydomain.com, then just make it return the right file content instead of redirect, it should be the easiest solution. Fetch command doesn't seem to have option to follow redirects. It's probably possible to solve it in ROS, but ... yikes.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3250
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: fetch via 301 redirect

Sat Mar 12, 2022 7:45 pm

I need to create a script that will download a file from source that presented as 301 redirect in DNS
Data.mydomain.com —A ->301-> https://mysite/file.rsc
Fetch data.mydomain.com gives an error to download file.rsc due to 301 redirect usage
Any ideas what to do?

The HTTP specs suggests a client (e.g. /tool/fetch) MAY, e.g. not MUST, follow a HTTP 301 redirect,
6.4.2. 301 Moved Permanently
The server SHOULD generate a Location header field in the response containing a preferred URI reference for the new permanent URI. The user agent MAY use the Location field value for automatic redirection. The server's response payload usually contains a short hypertext note with a hyperlink to the new URI(s). The user agent MAY use the Location field value for automatic redirection. The server's response payload usually contains a short hypertext note with a hyperlink to the new URI(s).

While /tool/fetch doesn't return the actual 301 error code, or any HTTP error codes, which makes /tool/fetch even more limited. It does return the body of the requests, which should have the "Location:" in the data returned even after a 301 "error". So theoretically, you can parse the data returned from first call to /tool/fetch, e.g. using :find / :pick to find the text "Location:" and then the URL. Again here RouterOS script doesn't have built-in string parsing either, so you need write that (or search the forum for examples / "library"). e.g. like @rextended's example linked above: viewtopic.php?p=878643#p878643

SO while I think its possible to do with a script, but you may be better offer working to get someone to fix the webserver/URL paths to avoid needing what would be a rather complex RouterOS script.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1041
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: fetch via 301 redirect

Sat Mar 12, 2022 8:22 pm

You need to consider that OP referred to DNS redirection which a complete different matter compared to HTTP redirect.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: fetch via 301 redirect

Sat Mar 12, 2022 9:11 pm

And does such DNS redirection even exist? I'm not aware of anything like that.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1041
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: fetch via 301 redirect

Sat Mar 12, 2022 11:37 pm

As I explained earlier in this thread it doesn't exist as a standard but you may find specialised extensions (hacks) in some of the DNS server providers. Like this one for example: https://cp.enom.com/kb/kb/kb_0352-creat ... direct.htm

EDIT:
@adminbas; what DNS service provider are you using?
Last edited by Larsa on Sun Mar 13, 2022 12:00 am, edited 2 times in total.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: fetch via 301 redirect

Sat Mar 12, 2022 11:52 pm

Problem is, it doesn't really have anything to do with DNS. It looks like some internal mechanism, where DNS server knows about external web server, points given hostnames to it, and web server does standard http redirection. So the only difference is that you don't need own web server for doing redirections, you get one from DNS provider.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1041
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: fetch via 301 redirect

Sat Mar 12, 2022 11:53 pm

Spot on and as it’s a hack the behaviour might differ between the different providers.
 
User avatar
smyers119
Member Candidate
Member Candidate
Posts: 232
Joined: Sat Feb 27, 2021 8:16 pm
Location: USA

Re: fetch via 301 redirect

Sun Mar 13, 2022 12:42 am

A 301 is a permanent redirect, why wouldn't you just point to the new url?
cause its a script to download script in a case of global updates on many routers. I can change 301 redirect to any hosting I'll get at that time
I dont know where download part will be at that time
Again a 301 is for perminent redirects. you'll want to use a 302. when you fetch a website with 302/301, are you able to parse thew response headers? That will have the new location.

I still don't understand your use case though. There has got to be a better way to do what you want to do, whatever your trying to do doesn't jive.
 
snehasharma
just joined
Posts: 4
Joined: Sat Jul 01, 2023 4:52 pm
Location: india
Contact:

Re: fetch via 301 redirect

Mon Jul 17, 2023 9:28 pm

Hi scripting guys
I need to create a script that will download a file from source that presented as 301 redirect in DNS
Data.mydomain.com —A ->301-> https://mysite/file.rsc
Fetch data.mydomain.com gives an error to download file.rsc due to 301 redirect usage
Any ideas what to do?
Yes you can follow the redirect using the requests library in Python that fetches the file from the redirected URL.
[*]First of all you have to install the requests library.
Here's an example script.
import requests

source_url = 'http://data.mydomain.com'  # The original URL
response = requests.get(source_url, allow_redirects=True)

if response.status_code == 200:
    final_url = response.url  # The final URL after following redirects
    file_name = final_url.split('/')[-1]  # Extracting the file name from the URL
    with open(file_name, 'wb') as file:
        file.write(response.content)
        print(f"File '{file_name}' downloaded successfully.")
else:
    print("Error: Failed to download the file.")
I hope it help you.

Who is online

Users browsing this forum: No registered users and 17 guests