Community discussions

MikroTik App
 
eguun
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 82
Joined: Fri Apr 10, 2020 10:18 pm

fetch capable of following redirects

Wed Sep 08, 2021 8:11 pm

Dear community,

To my surprise, the standard fetch tool in routerOS has no option to deal with following redirect.

So I decided to enhance it by scripting a function that would loop x times the fetch, each loop with the url pointing towards the redirect provided by the previous iteration.

My issue is that I don't think the scripting language enables me to do such basic task: it seems that the fetch command throws an error when facing a 301 (or 302).

So I can't consume what if the 302 pointing towards in order to loop again towards that new target.
- if I run fetch without error handling, the script stops at the fetch command
- if I run fetch with error handing, then the script resumes, but I get no output from the fetch action, and consequently can't assess where to redirect to

Would you have any advise how to overcome this?

Sample script below that shows a redirect on mikrotik.com website.
I would be keen to get a script POC that manages to extract where the redirect points towards
:do {:local output [/tool fetch url="http://mikrotik.com/" as-value output="user" mode="http"];} on-error={
	:put "error handling";
	:put $output; # shows nothing when error is triggered
}
 
eguun
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 82
Joined: Fri Apr 10, 2020 10:18 pm

Re: fetch capable of following redirects

Thu Sep 09, 2021 3:26 pm

I have looked around,

The core issue is that when inside the "on-error" error catching, then there is no way to get the error message to parse it.

This seems like a recurring issue raised by several forum threads already:
How to retrieve error details inside the on-error clause: viewtopic.php?t=160076
Any way to get the error-string in `on-error=`: viewtopic.php?t=108445
Error handling?: viewtopic.php?t=113197

Any hints on how I can get around this?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11968
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: fetch capable of following redirects

Thu Sep 09, 2021 5:22 pm

I'm able to catch the error.
Last edited by rextended on Thu Sep 09, 2021 5:29 pm, edited 4 times in total.
 
eguun
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 82
Joined: Fri Apr 10, 2020 10:18 pm

Re: fetch capable of following redirects

Thu Sep 09, 2021 5:24 pm

Thanks,

can you please post some sample code to display the error message and or store that error message to a variable?

Thanks
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11968
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: fetch capable of following redirects

Thu Sep 09, 2021 5:30 pm

From what I see till now, 99% of sript programmers are not programmers.
All must be programmed.
Just the handle of external tools like fetch can justify the use of the "on-error"

1) How to retrieve error details inside the on-error clause: viewtopic.php?t=160076
2) Any way to get the error-string in `on-error=`: viewtopic.php?t=108445
3) Error handling?: viewtopic.php?t=113197
1) This is the only true question
2) This is an erratic way to do something, not a question of error or not
3) The problem is the script code: is not done well, is not that way than the thing must done, is why the script give errors...


One example over all from 3):
# wrong coding
:do {/interface bridge add name=loopback; } on-error={:put "loopback exists"}
# translating: create loopback, if you got an error because already exist, ignore it
# on short: create it, hoping not already exist

# right coding
/interface bridge
:if ([:len [:find where name="loopback"]] = 0) do={add name=loopback}
# translating: if the result of the count of interface named loopback are zero, create the interface
# on short: if not exist, create it
 
eguun
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 82
Joined: Fri Apr 10, 2020 10:18 pm

Re: fetch capable of following redirects

Thu Sep 09, 2021 5:51 pm

Thanks for the contribution.

Not having access to the error message that triggered the on-error only gets you so far.

But on our topic at hand: the error message contains the redirect url.
One cannot just guess it, or am I missing the bigger picture?
How can one get that redirect url?

What would be the "right coding" to do it?
I'm a bit out of option.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11968
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: fetch capable of following redirects  [SOLVED]

Thu Sep 09, 2021 5:56 pm

search tag # rextended manage fetch errors

I write one post a time, leave me the time!!!! :lol: :lol: :lol:

Is not for suspance, I need some time to write and test....

This is the code:
/file remove [find where name="testfetch.txt"]
{
    :local jobid [:execute file=testfetch.txt script="/tool fetch url=http://mikrotik.com"]
    :put "Waiting the end of process for file testfetch.txt to be ready, max 20 seconds..."
    :global Gltesec 0
    :while (([:len [/sys script job find where .id=$jobid]] = 1) && ($Gltesec < 20)) do={
        :set Gltesec ($Gltesec + 1)
        :delay 1s
        :put "waiting... $Gltesec"
    }
    :put "Done. Elapsed Seconds: $Gltesec\r\n"
    :if ([:len [/file find where name="testfetch.txt"]] = 1) do={
        :local filecontent [/file get [/file find where name="testfetch.txt"] contents]
        :put "Result of Fetch:\r\n****************************\r\n$filecontent\r\n****************************"
    } else={
        :put "File not created."
    }
}

This is the result:

terminal code

Waiting the end of process for file testfetch.txt to be ready, max 20 seconds...
waiting... 1
waiting... 2
Done. Elapsed Seconds: 2

Result of Fetch:
****************************
  status: connecting

  status: failed

failure: closing connection: <302 Found "https://mikrotik.com/"> 159.148.147.196:80 (4)
                                                                                       
****************************
the result is a multiline file between the ****************************

with :find and :pick inside the string "closing connection: <302 Found "https://mikrotik.com/"> 159.148.147.196:80 (4)" can be isolated

the part containing the redirect "https://mikrotik.com/"

you can recall again the test code and you discover another redirect to "https://www.mikrotik.com/" :lol: :lol: :lol:

End.
Last edited by rextended on Sat Sep 11, 2021 2:48 pm, edited 4 times in total.
 
eguun
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 82
Joined: Fri Apr 10, 2020 10:18 pm

Re: fetch capable of following redirects

Thu Sep 09, 2021 6:02 pm

Thanks

It's very a interesting approach to isolate the code execution into a separate job, and monitor that job progress!

I'll get my head around this and do some testing: I understand that within that job, the code is outputing the fetch to a file.
I was keen to limit such disk access to preserve the SSD life.
So I'll try to get that value in a variable (such as a global variable).

But it's a very cool approach, many thanks!
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11968
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: fetch capable of following redirects

Thu Sep 09, 2021 6:06 pm

Do not worry about disk writes, on flash the write go already n ramdisk, not wearing at all,
about NAND, are years (from 2009) than I use user-manager on nand, and no problem,
and user-manager write continuosly on nand....
 
eguun
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 82
Joined: Fri Apr 10, 2020 10:18 pm

Re: fetch capable of following redirects

Thu Sep 09, 2021 6:24 pm

I respect your faith in SSD ... but a small voice in my head will push me to attempt to get this running without disk access :D
 
eguun
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 82
Joined: Fri Apr 10, 2020 10:18 pm

Re: fetch capable of following redirects

Thu Sep 09, 2021 9:24 pm

After a bit of (unsuccessful) testing, there doesn't seem to be possible to get fetch error message without using disk read/write.

rextended your solution is still the best workaround, and I'll mark this as the solution.
But it still remain damn painful to not have access on a simple variable of the error message, like most programming language offer by standard.

Who is online

Users browsing this forum: Ahrefs [Bot], dinosgb, sergejs and 90 guests