fetch via 301 redirect

RouterOS Scripting
1

Hi scripting guys
I need to create a script that will download a file from source that presented as 301 redirect in DNS
Data.mydomain.com —A ->301-> https://mysite/file.rsc
Fetch data.mydomain.com gives an error to download file.rsc due to 301 redirect usage
Any ideas what to do?

2

A 301 is a permanent redirect, why wouldn’t you just point to the new url?

3

cause its a script to download script in a case of global updates on many routers. I can change 301 redirect to any hosting I’ll get at that time
I dont know where download part will be at that time

4

First of all DNS redirect is not a standard and can only be achieved using specialized services (ala hacked/extended DNS server). You need to find out exactly how the DNS response is constructed in order to check if it can be parsed by the client side (ie Mikrotik). Secondly as someone stated before, HTTP 301 is normally used for permanent redirects thus you should be able to parse the redirect manually and then use that result in your script. If you claim it might change from time to time it’s not a permanent redirect.

Check this service as an example: https://cp.enom.com/kb/kb/kb_0352-create-301-redirect.htm

5

If you control data.mydomain.com, then just make it return the right file content instead of redirect, it should be the easiest solution. Fetch command doesn’t seem to have option to follow redirects. It’s probably possible to solve it in ROS, but … yikes.

6

The HTTP specs suggests a client (e.g. /tool/fetch) MAY, e.g. not MUST, follow a HTTP 301 redirect,

6.4.2. 301 Moved Permanently
The server SHOULD generate a Location header field in the response containing a preferred URI reference for the new permanent URI. The user agent MAY use the Location field value for automatic redirection. The server’s response payload usually contains a short hypertext note with a hyperlink to the new URI(s). The user agent MAY use the Location field value for automatic redirection. The server’s response payload usually contains a short hypertext note with a hyperlink to the new URI(s).

While /tool/fetch doesn’t return the actual 301 error code, or any HTTP error codes, which makes /tool/fetch even more limited. It does return the body of the requests, which should have the “Location:” in the data returned even after a 301 “error”. So theoretically, you can parse the data returned from first call to /tool/fetch, e.g. using :find / :pick to find the text “Location:” and then the URL. Again here RouterOS script doesn’t have built-in string parsing either, so you need write that (or search the forum for examples / “library”). e.g. like @rextended’s example linked above: http://forum.mikrotik.com/t/fetch-capable-of-following-redirects/151723/7

SO while I think its possible to do with a script, but you may be better offer working to get someone to fix the webserver/URL paths to avoid needing what would be a rather complex RouterOS script.

7

You need to consider that OP referred to DNS redirection which a complete different matter compared to HTTP redirect.

8

And does such DNS redirection even exist? I’m not aware of anything like that.

9

As I explained earlier in this thread it doesn’t exist as a standard but you may find specialised extensions (hacks) in some of the DNS server providers. Like this one for example: https://cp.enom.com/kb/kb/kb_0352-create-301-redirect.htm

EDIT:
@adminbas; what DNS service provider are you using?

10

Problem is, it doesn’t really have anything to do with DNS. It looks like some internal mechanism, where DNS server knows about external web server, points given hostnames to it, and web server does standard http redirection. So the only difference is that you don’t need own web server for doing redirections, you get one from DNS provider.

11

Spot on and as it’s a hack the behaviour might differ between the different providers.

12

Again a 301 is for perminent redirects. you’ll want to use a 302. when you fetch a website with 302/301, are you able to parse thew response headers? That will have the new location.

I still don’t understand your use case though. There has got to be a better way to do what you want to do, whatever your trying to do doesn’t jive.

13

Yes you can follow the redirect using the requests library in Python that fetches the file from the redirected URL.
[*]First of all you have to install the requests library.
Here’s an example script.

import requests

source_url = 'http://data.mydomain.com'  # The original URL
response = requests.get(source_url, allow_redirects=True)

if response.status_code == 200:
    final_url = response.url  # The final URL after following redirects
    file_name = final_url.split('/')[-1]  # Extracting the file name from the URL
    with open(file_name, 'wb') as file:
        file.write(response.content)
        print(f"File '{file_name}' downloaded successfully.")
else:
    print("Error: Failed to download the file.")

I hope it help you.