Community discussions

MikroTik App
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

The maximum size of a read/written file.

Mon Jun 19, 2023 6:15 pm

I might have misunderstood, but in MikroTik there is a limitation on the size of the read and written files, which is 4 kB. This means that even a log file cannot be read by a script if it exceeds this size. Am I correct in understanding that one needs to somehow concatenate files when reading and split them when writing? Or is there another way to work with large files?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12014
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: The maximum size of a read/written file.

Mon Jun 19, 2023 6:19 pm

The limit reading the file from /file is 4095 Bytes, but by fetch is 64512 Bytes
But with my method, on supported hosts, the read is ulimited.
Is time you read and study my posts..........................
viewtopic.php?f=9&t=177530
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Mon Jun 19, 2023 11:58 pm

The limit reading the file from /file is 4095 Bytes, but by fetch is 64512 Bytes
But with my method, on supported hosts, the read is ulimited.
Is time you read and study my posts..........................
viewtopic.php?f=9&t=177530
I continue to attribute our misunderstanding not to your unwillingness to listen, but to the poor translation, even though I'm using ChatGPT. I was not interested in reading from other websites. I was interested in writing a single large file, including when reading SMS messages from a modem. Even if you have only 25 SMS messages on a SIM card, they may not fit into one file under these circumstances. And if you have SMS messages stored in the modem's memory, where there can be a couple of hundred messages, they will definitely not fit into one file. Therefore, I was wondering if it's possible to write any file with any extension, such as .log or .abirval, as long as there is no need to split the file into parts. Splitting the file into parts is not a problem, and even concatenating the read content into one variable. I did that on the first day. But I wanted to write some logs, including having an array of read SMS messages.
MikroTik allows you to write a large-sized file if you copy it from a computer. However, it cannot be read without prior splitting. Moreover, the presence of an SSD slot on the router looks quite odd. Why is it there if MikroTik cannot read large files? I was hoping that there would be some recording format, some extension, or a way to write the necessary files onto such a disk.
This is the option I came up with on the first day when I started working on reading SMS. Files are read in a similar way. So there is no problem with splitting and reading. The inconvenience lies in the misunderstanding of MikroTik's policy. It's like wings on a chicken, they exist, but it can't fly over the fence.
:local output ( [ /interface lte at-chat lte1 wait=yes input="AT+CMGL=4" as-value ]->"output" );
:local lenOutput ([:len $output]);

:local condition 1;
:local i 0;
:local j 0;

:while ($condition=1) do={
    :local nameFile ("sms".$j.".txt")
    /file print file=$nameFile;
    :delay 3s;
    /file set $nameFile contents=[:pick $output $i ($i+4000)];
    :set i ($i + 4000);
    :if ($i < $lenOutput) do={
        :set j ($j +1)
        :set condition 1;
    } else={:set condition 0;}
}
 
msatter
Forum Guru
Forum Guru
Posts: 2912
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: The maximum size of a read/written file.

Tue Jun 20, 2023 12:21 am

You can read large files by not using a script to read the file but put the script in the file in a form of a function and import the RSC file.

Exporting large files is possible and rename it then to an RSC file so they can be imported again.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12014
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: The maximum size of a read/written file.

Tue Jun 20, 2023 10:14 am

I continue to attribute our misunderstanding not to your unwillingness to listen, but to the poor translation
Exactly, if I wasn't interested, I wouldn't even have answered you.
But what is your native language?
 
pe1chl
Forum Guru
Forum Guru
Posts: 10248
Joined: Mon Jun 08, 2015 12:09 pm

Re: The maximum size of a read/written file.

Tue Jun 20, 2023 12:22 pm

The issue is not "a policy", but the limit on the length of a string variable.
Because the command to read from a file can only read the entire file in one read and the result will have to fit in a variable, there is this limitation.

It would be easy to solve, by providing separate open/read/close commands where read would read some specified data (#bytes, "a line") and can be called in a loop after open to process all data in the file record-by-record.
Unfortunately RouterOS does not provide that. It has been in the feature suggestions for a long time.

Now, when you want to read "the log", it can be done in a different way because you can get records one-by-one from the log memory buffer.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12014
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: The maximum size of a read/written file.

Tue Jun 20, 2023 12:36 pm

In the end it's just RouterOS, not "eXtendedOS".
Either you use Containers so you do what you like, or you accept that in the end it's a Router, not one Synology or one of the many "SIM Manager" programs....
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Tue Jun 20, 2023 2:41 pm

You can read large files by not using a script to read the file but put the script in the file in a form of a function and import the RSC file.

Exporting large files is possible and rename it then to an RSC file so they can be imported again.
Can you give a couple of links to this topic to read more and understand it?
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Tue Jun 20, 2023 3:14 pm

In the end it's just RouterOS, not "eXtendedOS".
Either you use Containers so you do what you like, or you accept that in the end it's a Router, not one Synology or one of the many "SIM Manager" programs....
I understand that RouterOS is designed specifically for routers (and possibly single-board computers). I don't have excessive demands for it. However, if the manufacturer claims to support certain hardware, it would be desirable to have full support rather than limited functionality. Especially since we are not only talking about reading SMS messages.
Exactly, if I wasn't interested, I wouldn't even have answered you.
But what is your native language?
I think it's not important, even though I live in one country, my native language is different. Overall, I've gotten used to the fact that Google has been doing a poor job of translating my native language in the past couple of years. Even AliExpress started providing bad translations in my native language. But I'm not blaming my country or Google. Maybe there are fewer specialists who know my language now. And there are fewer people who need my native language. There used to be more of us, but times change and everything evolves.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12014
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: The maximum size of a read/written file.

Tue Jun 20, 2023 3:34 pm

[…]
 
msatter
Forum Guru
Forum Guru
Posts: 2912
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: The maximum size of a read/written file.

Tue Jun 20, 2023 5:19 pm

This was the startingpoint: viewtopic.php?p=819118
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12014
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: The maximum size of a read/written file.

Tue Jun 20, 2023 6:11 pm

Let's sort it out...

1) Read an internal file to the RouterBOARD.
a) Reading the contents of any file into /files and putting it into a variable is limited to 4095Bytes.
b) Reading the contents of any file into /files and auto-sharing it as a file in ftp and putting it into a variable, is limited to 64512Bytes.

2) Read an external file to the RouterBOARD.
a) Reading the content of any file via http,https and ftp and putting it in a variable is limited to 64512Bytes.
b) It is possible to read longer files, one piece at a time, with my method on http and https only.

3) Read content (from a file) at most large 65452Bytes (is limited to 30000Bytes if the field is filled on WinBox textarea field).
a) It is possible to read/write from a particular area of RouterOS, as if it were a reserved area,
but inevitably the data is read by doing an "import" and written by doing an "export".
This zone is the "Startup Note" zone.
If you don't plan to use it, it provides easy import and export, which is not even mixed with schedulers or scripts,
so that you only export/import what you want, so you don't export other schedulers and scripts to the file as well.
Example of one export/import:

example big file export code

# jun/20/2023 17:05:40 by RouterOS 6.48.6
# software id = DEKE-RIJB
#
# model = SXT G-5HPacD r2
# serial number = DEADBEEFC1AX
/system note
set note="12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678\r\
    \n12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678\r\
[…] 651 identical lines removed
    \n12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678\r\
    \n12345678901234567890123456789012345678901234567890\r\
    \n" show-at-login=no
For read, just import the file (no matter if have or not .rsc extension) and you can have the correct content with :local varname [/sys note get note]

So, in conclusion for READ section, 65452Bytes can contain without problem 256 SMS.
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Tue Jun 20, 2023 8:12 pm

I guess all (or most) these DyadyaGenya topics will not be necessary if sms tool is improved - viewtopic.php?t=191963, it was my first topic on this forum, I should probably need to add link to it into feature request topic.
Last edited by optio on Wed Jun 21, 2023 6:41 pm, edited 1 time in total.
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Wed Jun 21, 2023 4:49 pm

b) Reading the contents of any file into /files and auto-sharing it as a file in ftp and putting it into a variable, is limited to 64512Bytes.
Very strange. I don't understand that case, but I can read a file that I may have copied from the computer to the Mikrotik and which has a size of 6.4 KiB. I do like this:
:local sms [ /file get [ find name="sms.txt" ] contents ]
:put $sms
But nothing is printed. If I reduce the file size to 4KiB, then everything works out.
Or is this case related to point "a"?
2) Read an external file to the RouterBOARD.
Not interested at the moment
3) Read content (from a file) at most large 65452Bytes (is limited to 30000Bytes if the field is filled on WinBox textarea field).
and

example big file export code

/system note
set note="..."
and
For read, just import the file (no matter if have or not .rsc extension) and you can have the correct content with :local varname [/sys note get note]
I didn't understand this example. Let's say I did this:
/system note set note  ( [ /interface lte at-chat lte1 wait=yes input="AT+CMGL=4" as-value ]->"output" );
How can I write this to the "sms.txt" file?
And how then to read this file?
Still interested in how many such files can be created and where are they stored? Can I create such files on an ssd drive? Because I would like to write not only SMS, but also to make a safety net for IoT records.
Last edited by DyadyaGenya on Wed Jun 21, 2023 5:03 pm, edited 1 time in total.
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Wed Jun 21, 2023 4:56 pm

I guess all (or most) these DyadyaGenya topics will not be unnecessary if sms tool is improved - viewtopic.php?t=191963, it was my first topic on this forum, I should probably need to add link to it into feature request topic.
I subscribe to your every word. This is exactly what I was talking about when comparing chicken wings. Yes, chickens are good birds. There are many benefits from it: meat, eggs, feathers, the opportunity to show care. But Mikrotik claims to be an eagle. Indeed, in many areas he behaves like an eagle. But there are moments when he turns into a chicken that does not fly higher than the fence, although there are wings.
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Wed Jun 21, 2023 5:13 pm

This was the startingpoint: viewtopic.php?p=819118
I think that it is more correct to continue in this topic, and not in the one where the link is.

I tried doing something like this:
:local sms [/system script get "sms.txt" source]; 
:put $sms
But it gives an error: "no such item"
 
pe1chl
Forum Guru
Forum Guru
Posts: 10248
Joined: Mon Jun 08, 2015 12:09 pm

Re: The maximum size of a read/written file.

Wed Jun 21, 2023 6:15 pm

Probably when you ask MikroTik for SMS support, you will get the reply "nobody uses SMS anymore, we all use Telegram".
It is similar to the request to support DSL (through other people's hardware) that is waved away with "everyone has fiber today".
They seem to see the local Latvian situation as leading for the entire world...
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3506
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: The maximum size of a read/written file.

Wed Jun 21, 2023 6:32 pm

I'd say Mikrotik is more a magpie (good luck, bad luck, or a pest...depending on context) than eagle – but liked the analogy.

@rextended feel free to correct me :). But "/file x set content=$mytest" is limited to 4k chars. Part of this is because RouterOS variables are really UNIX shell variable – which used to be limited to 4K but now 32K or large in modern Linux. Mikrotik allows up to 32K someplaces, but not in /file – so RouterOS is a chicken when it comes to /file.

The only way around this is some "scripting trickery" since commands that right don't go through /file with variables aren't limited in size – @rextended has some suggestions, all complex because of the scripting limitations – so don't shoot the pigeon with the message. But I think it the /file operations that enforces this artificial 4K limit, so no way around that part. At some point you'll run into the "real" limit of 32K however.

And, yes, it's fair question now that Mikrotik sells many LTE devices why they don't parse PDUs as part of /tool/sms.
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Wed Jun 21, 2023 6:44 pm

And, yes, it's fair question now that Mikrotik sells many LTE devices why they don't parse PDUs as part of /tool/sms.
Yes, sadly. There are advanced features like parsing commands from message body, but not basics like Unicode encoding and segmented messages support.
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3506
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: The maximum size of a read/written file.

Wed Jun 21, 2023 7:14 pm

Well, changing the 4K file limit from content= be a start... @DyadyaGenya and @rextended have come a long way here... to run into yet another limitation & another set of complex scripting to workaround.
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Wed Jun 21, 2023 7:31 pm

Well, changing the 4K file limit from content= be a start... @DyadyaGenya and @rextended have come a long way here... to run into yet another limitation & another set of complex scripting to workaround.
Personally I will go with approach: In loop where records are appended to variable for file save, before append record to that variable I will first check size of that variable and if that length of that variable + length of record variable exceeds 4K then I will write current content value into file segment, then assign current record variable into that variable. File segment counter needs to be raised each time when file is saved. At the end you will have files like: sms.txt.1, sms.txt.2, sms.txt.3... which can be merged on other system where are downloaded.
Last edited by optio on Wed Jun 21, 2023 7:33 pm, edited 1 time in total.
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Wed Jun 21, 2023 7:32 pm

...
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Wed Jun 21, 2023 9:12 pm

Personally I will go with approach: In loop where records are appended to variable for file save, before append record to that variable I will first check size of that variable and if that length of that variable + length of record variable exceeds 4K then I will write current content value into file segment, then assign current record variable into that variable. File segment counter needs to be raised each time when file is saved. At the end you will have files like: sms.txt.1, sms.txt.2, sms.txt.3... which can be merged on other system where are downloaded.
Why consider the volume of a variable, if everything is well written to a file without it. Have you seen my example above? There is exactly the same principle as you described, only without checking the volume of the variable.
I do not argue, the example is not very well designed, but it was one of the first scripts that I wrote and I published it here just so that the @rextended understood what I was talking about.
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Wed Jun 21, 2023 9:28 pm

Why consider the volume of a variable, if everything is well written to a file without it. Have you seen my example above? There is exactly the same principle as you described, only without checking the volume of the variable.
Well that is not exactly, that part is crucial if you don't want to reach 4k limit. Topic is The maximum size of a read/written file. which is actually limit of variable that contains content of a file.
I do not argue, the example is not very well designed, but it was one of the first scripts that I wrote and I published it here just so that the @rextended understood what I was talking about.
I saw your script example, yes it contains loop for writing file segments, but it is lacking logic which I mentioned.
 
msatter
Forum Guru
Forum Guru
Posts: 2912
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: The maximum size of a read/written file.

Wed Jun 21, 2023 9:36 pm

This was the startingpoint: viewtopic.php?p=819118
I think that it is more correct to continue in this topic, and not in the one where the link is.

I tried doing something like this:
:local sms [/system script get "sms.txt" source]; 
:put $sms
But it gives an error: "no such item"

execute "/system script get \"sms.txt\" source" file="sms.txt"
I don't do scripting anymore but this should write to the file. If you come up with the correct commands it should write to the file sms.txt
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Wed Jun 21, 2023 10:25 pm

@DyadyaGenya try with this
:local output ( [ /interface lte at-chat lte1 wait=yes input="AT+CMGL=4" as-value ]->"output" )
:local lenOutput [:len $output]
:local splitSize 4095
:local offset 0
:local part 1

:while ($offset < $lenOutput) do={
  :local nameFile ("sms".$part.".txt")

  :if (($offset + $splitSize) > $lenOutput) do={
    /file add name=$nameFile contents=[:pick $output $offset $lenOutput]
    :set offset $lenOutput
  } else={
    /file add name=$nameFile contents=[:pick $output $offset ($offset + $splitSize)]
    :set offset ($offset + $splitSize)
  }

  :set part ($part + 1)
}
Last edited by optio on Thu Jun 22, 2023 7:32 pm, edited 2 times in total.
 
msatter
Forum Guru
Forum Guru
Posts: 2912
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 12:42 am

@optio Why still self limit to 4096 bytes!? Many people here walked into this limitation for many years and some found little gems in RouterOS which works around problems. One of those is ":execute"

Lets keep it simple, this will store the output of a command or script to a file like it was display in you screen.
:execute "/ip/route/export" file="test"
It adds the default extension "txt" to the filename, here is that "test"

You can find more about :execute: https://wiki.mikrotik.com/wiki/Manual:S ... l_commands
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 1:17 am

@optio Why still self limit to 4096 bytes!? Many people here walked into this limitation for many years and some found little gems in RouterOS which works around problems. One of those is ":execute"

Lets keep it simple, this will store the output of a command or script to a file like it was display in you screen.
:execute "/ip/route/export" file="test"
It adds the default extension "txt" to the filename, here is that "test"

You can find more about :execute: https://wiki.mikrotik.com/wiki/Manual:S ... l_commands
Ok, from DyadyaGenya script example correct answer will be then
:execute ":put ([/interface lte at-chat lte1 wait=yes input=\"AT+CMGL=4\" as-value ]->\"output\")" file=sms.txt
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 1:54 am

But what is your native language?
This may answer https://forummikrotik.ru/viewtopic.php?p=89403#p89403 :)
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 3506
Joined: Sun May 01, 2016 7:12 pm
Location: California

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 4:56 am

But what is your native language?
This may answer https://forummikrotik.ru/viewtopic.php?p=89403#p89403 :)
LOL. About the same...
Всё написано выше. Имеющий глаза - увидит
(Google tells me, that means "Everything is written above. Those who have eyes will see.")
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12014
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 10:34 am

But what is your native language?
This may answer https://forummikrotik.ru/viewtopic.php?p=89403#p89403 :)
What does it have to do with it, I wanted him to specify it, if he wanted...
If you look closely you'll see that the Uncle Женя lives in Chernihiv, and was born on January 17.. :lol: (Женя жарт... ;) )
I don't put the year, if he wants he can put it there... :lol:
 
msatter
Forum Guru
Forum Guru
Posts: 2912
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 2:12 pm

@optio Why still self limit to 4096 bytes!? Many people here walked into this limitation for many years and some found little gems in RouterOS which works around problems. One of those is ":execute"

Lets keep it simple, this will store the output of a command or script to a file like it was display in you screen.
:execute "/ip/route/export" file="test"
It adds the default extension "txt" to the filename, here is that "test"

You can find more about :execute: https://wiki.mikrotik.com/wiki/Manual:S ... l_commands
Ok, from DyadyaGenya script example correct answer will be then
:execute ":put ([/interface lte at-chat lte1 wait=yes input=\"AT+CMGL=4\" as-value ]->\"output\")" file=sms.txt


No, I wrote "It adds the default extension "txt" to the filename, here is that "test" ".
:execute ":put ([/interface lte at-chat lte1 wait=yes input=\"AT+CMGL=4\" as-value ]->\"output\")" file="sms"
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 3:48 pm

:execute ":put ([/interface lte at-chat lte1 wait=yes input=\"AT+CMGL=4\" as-value ]->\"output\")" file="sms"
Works also with .txt extension in file param, then it will not append .txt to filename.
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 3:58 pm

No, I wrote "It adds the default extension "txt" to the filename, here is that "test" ".
:execute ":put ([/interface lte at-chat lte1 wait=yes input=\"AT+CMGL=4\" as-value ]->\"output\")" file="sms"
In fact, both options work. And if you tell me how to read the received files, then this will be an almost ideal solution. I write "almost" because for some reason (maybe it's something wrong with me) after rebooting the router, the file disappears. It has to be copied to another location or renamed.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10248
Joined: Mon Jun 08, 2015 12:09 pm

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 4:18 pm

That depends on the type of router. Unfortunately, MikroTik has missed the chance of defining standard paths of persistent/volatile files.
Now, you will have to check yourself and use a path suitable for your router type and configuration.
 
msatter
Forum Guru
Forum Guru
Posts: 2912
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 4:37 pm

:execute ":put ([/interface lte at-chat lte1 wait=yes input=\"AT+CMGL=4\" as-value ]->\"output\")" file="sms"
Works also with .txt extension in file param, then it will not append .txt to filename.
Not overhere on ROS 7.10
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 4:43 pm

Not overhere on ROS 7.10
I'm also on 7.10
> /system/routerboard/print 
  routerboard: yes
  model: D53G-5HacD2HnD
  serial-number: XXXXXXXX
  firmware-type: ipq4000L
  factory-firmware: 7.1beta5
  current-firmware: 7.10
  upgrade-firmware: 7.10
> :execute ":put ([/interface lte at-chat lte1 wait=yes input=\"AT+CMGL=4\" as-value ]->\"output\")" file="sms.txt" 
sms_file.png
You do not have the required permissions to view the files attached to this post.
 
msatter
Forum Guru
Forum Guru
Posts: 2912
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 4:50 pm

If you want to restore the stored data or transfer to a differen router then selective path exporting and exporting is the simple way.

I don't know how thatworks with LTE or SMS because I don't have that.

In this posting is mentioned that extension does not need to be RSC.

viewtopic.php?p=1009229#p1008826
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12014
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 4:57 pm

*** see my next post ***
Last edited by rextended on Thu Jun 22, 2023 5:05 pm, edited 2 times in total.
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 5:00 pm

In fact, both options work. And if you tell me how to read the received files, then this will be an almost ideal solution.
Already explained above viewtopic.php?t=197190#p1008826 and system note can be used as workaround. You can also provide file over http and use chunked download with fetch tool, also you have post with script by rextended viewtopic.php?f=9&t=177530
I write "almost" because for some reason (maybe it's something wrong with me) after rebooting the router, the file disappears. It has to be copied to another location or renamed.
you can use flash of usb storage for persistence, depends on device, to save it on flash use
file="flash/sms"
 
msatter
Forum Guru
Forum Guru
Posts: 2912
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 5:00 pm

Not overhere on ROS 7.10
I'm also on 7.10
> /system/routerboard/print 
  routerboard: yes
  model: D53G-5HacD2HnD
  serial-number: XXXXXXXX
  firmware-type: ipq4000L
  factory-firmware: 7.1beta5
  current-firmware: 7.10
  upgrade-firmware: 7.10
> :execute ":put ([/interface lte at-chat lte1 wait=yes input=\"AT+CMGL=4\" as-value ]->\"output\")" file="sms.txt" 
sms_file.png
I checked again and you are correct. I mixed up that by having then a extra extension to the filename "sms.file" --> "sys.file.txt"
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 5:03 pm

I checked again and you are correct. I mixed up that by having then a extra extension to the filename "sms.file" --> "sys.file.txt"
Yes, for other extensions is appended, but for txt not.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12014
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 5:05 pm

This work for sure, but the file is created after some seconds, or you wait ":execute" to finish it's job, or you do not see instantly the file...
Also the flash fix is easy...
{
    :local filename "readallsmspdu.txt"
    :if ([:len [/file find where name="flash" and type="disk"]]=1) do={:set filename "flash/$filename"}
:local jobID [:execute ":put ([/interface lte at-chat ([find]->0) input=\"AT+CMGF=0;+CMGL=4\" wait=yes as-value ]->\"output\")" file=$filename]
    :while ([:len [/system script job find where .id=$jobID]] > 0) do={
        :delay 1s
    }
}
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 5:06 pm

What does it have to do with it, I wanted him to specify it, if he wanted...
If you look closely you'll see that the Uncle Женя lives in Chernihiv, and was born on January 17.. :lol: (Женя жарт... ;) )
I don't put the year, if he wants he can put it there... :lol:
You didn't guess my native language correctly ))) I appreciate your tact, even if it wasn't fully demonstrated. As I mentioned earlier, I don't consider nationality, language, age, or other characteristics to be important. People should be judged by their actions, not by their language or age.
By the way, regarding the language you attempted to make a joke in, it's a perfect demonstration of stereotypical thinking and contemporary narratives. It's understandable and forgivable that you know how to use a search engine but draw superficial conclusions from the very first results. However, there are people who would consider such a joke offensive because there are still people in Chernihiv whose ancestors were massacred by Ukrainians, including infants. To be more specific, the city of Chernihiv itself was massacred by Ukrainians three times. Other cities in modern Ukraine were also massacred, such as Putivl. But Ukrainians didn't just target the Left Bank of the Dnieper. They killed Poles not only in the past but also in more recent history. Read about the Volhynian massacre, where about 100,000 Poles were killed regardless of age, gender, or health condition. I would really hope that all of this is in the past, and one way to ensure that is by not discussing such matters, as they will inevitably lead to conflicts.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12014
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 5:11 pm

Do you really think I give a damn if you're Russian, Polish or Ukrainian?

I just keep seeing a Brother.
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 5:32 pm

@DyadyaGenya try with this
I tried it. Firstly, the pros. It seems visually faster and creates two files at once. As for the cons, there is more code and more "extra movements." But extra movements are a matter of personal preference.
But what is your native language?
This may answer https://forummikrotik.ru/viewtopic.php?p=89403#p89403 :)
I hope that your little investigation brought you moral satisfaction and perhaps even physical satisfaction.
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 5:40 pm

Do you really think I give a damn if you're Russian, Polish or Ukrainian?
That's why I mentioned tact. I don't know how the translator will render it, so let me explain it in different words. Don't feel compelled to voice everything you know.

By the way, about those words. The translator translated this phrase in three different ways in the three languages mentioned here ))) And in one of them, it has a very negative meaning ))))
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 5:50 pm

you can use flash of usb storage for persistence, depends on device, to save it on flash use
file="flash/sms"
Naturally, I tried that option right away. I saved the file to my SSD drive. It doesn't matter, after rebooting, it still disappears.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12014
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 6:13 pm

Do you really think I give a damn if you're Russian, Polish or Ukrainian?
in one of them, it has a very negative meaning ))))
The sentence, on purpose, was meant to be negative, to give the appropriate emphasis to the concept: "I just keep seeing a Brother.".
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 6:19 pm

I hope that your little investigation brought you moral satisfaction and perhaps even physical satisfaction.
No, just curiosity was raised when you complaining about understanding google/chat gpt translations and unwillingness to reveal you natural language. I thought maybe you speak Njerep but translation of you nick guided me to russian forum and thats about it.
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 6:23 pm


in one of them, it has a very negative meaning ))))
The sentence, on purpose, was meant to be negative, to give the appropriate emphasis to the concept: "I just keep seeing a Brother.".
Everything is fine, I understand simple things without a translator. And I understand where the translator is likely to make mistakes, so I constantly talk about the issue of understanding and don't blame the person for inaccuracies.
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 6:37 pm

This work for sure, but the file is created after some seconds, or you wait ":execute" to finish it's job, or you do not see instantly the file...
I still don't understand how to read such a file. You're talking about exporting, and as I understood, using notes, but I still couldn't accomplish it.
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Thu Jun 22, 2023 6:43 pm

I still don't understand how to read such a file. You're talking about exporting, and as I understood, using notes, but I still couldn't accomplish it.
It is hard to understand what you want to accomplish with all this. First if you you want to save PDU to file then it seems you want to be read on other system, but you want also read PDU from file, what case is that? You use external system to modify that file and again to be read inside ROS, why just not process PDU on that system? If external system is not involved here, why using files anyway? You want to create some kind of SMS inbox with ROS script so that you can have in it more that 25 messages? If that is the case, I suggest to create something in container and download over ftp PDU file in some interval and create scheduler that saves that saves PDU in file and erases messages from SIM.
If persist to achieve this with files inside ROS script, then you will need to go with segmented solution as in script viewtopic.php?t=197190#p1009097. To read that in other script you will need to create loop that will rise segment number, create filename with same pattern as are written and append to variable content of that file segment. Inside loop you can check if file for segment exists
:if ([find where name=<filename>]) do={...}
and break loop if not exists.
Here are both scripts to do that:

Write script
:local output ( [ /interface lte at-chat lte1 wait=yes input="AT+CMGL=4" as-value ]->"output" )
:local lenOutput [:len $output]
:local splitSize 4095
:local offset 0
:local part 1

/file
:while ($offset < $lenOutput) do={
  :local nameFile ("sms".$part.".txt")
  :local end

  :if (($offset + $splitSize) > $lenOutput) do={
    :set end $lenOutput
  } else={
    :set end ($offset + $splitSize)
    :set part ($part + 1)
  }

  add name=$nameFile contents=[:pick $output $offset $end]
  :set offset $end
}

Read script
:local part 1
:local input ""
:local done false

/file
:while (!$done) do={
  :local nameFile ("sms".$part.".txt")

  :if ([find where name=$nameFile]) do={
    :set input ($input.[get $nameFile contents])
    :set part ($part + 1)
  } else={
    :set done true
  }
}

:put $input
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Fri Jun 23, 2023 5:53 pm

It is hard to understand what you want to accomplish with all this.
It's probably my fault that I didn't describe in detail what I want. I have several objectives: 1. Occasionally save the contents of PDU to a file. 2. Occasionally save processed SMS to another file. 3. Sometimes save data from smart home devices to a third file. 4. Occasionally save incoming Telegram messages to multiple files. These messages can be related to modem operations as well as IoT device operations. 5. Accordingly, depending on the situation, it is necessary to periodically append data to one of these files.

By the way, the option with "execute" for writing suits me just fine. But I don't understand if it allows appending and how to read from it
 
pe1chl
Forum Guru
Forum Guru
Posts: 10248
Joined: Mon Jun 08, 2015 12:09 pm

Re: The maximum size of a read/written file.

Fri Jun 23, 2023 5:59 pm

One way of "appending" to a file that may be suitable for your purpose is to use /log to log the message at some severity, and define a logging action and corresponding logging destination (for that severity and "script" topic) to write these messages to a file.
Of course it will put date/time on the messages but you may even want that.
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Fri Jun 23, 2023 6:00 pm

By the way, the option with "execute" for writing suits me just fine. But I don't understand if it allows appending and how to read from it
It was explained in above posts why you can't and that you can only load chunked/segmented data into script variable. If you need to load large file from external system you can provide it over http and fetch it chunked or split file into segments and upload to device.
My opinion for features you need, it is better to create software solution for PDU processing on some other system, not ROS, from ROS you can just provide PDU data file (for this you can use :execute and save all data into single file) over ftp that can be fetched from other system.
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Fri Jun 23, 2023 7:13 pm

If you need to load large file from external system you can provide it over http and fetch it chunked or split file into segments and upload to device.
I don't understand what you mean by an external system. Are you referring to my mention of IoT devices? These devices send brief information in small messages. Since it's not possible to create SQL databases on MikroTik, I wanted to save this data in CSV or JSON format. This doesn't require transmitting large amounts of information over HTTP.
(for this you can use :execute and save all data into single file) over ftp that can be fetched from other system.
The data will definitely be transmitted and stored on another system, which will be the main system. However, I hope to create a sufficiently resilient system with multiple safeguards. And I hope that one of those safeguards will be MikroTik. In this regard, I recall the Italian Arduino and its Chinese clones. They can easily write and read files of large sizes, although they are not full-fledged systems compared to the MikroTik processor.
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Fri Jun 23, 2023 7:17 pm

One way of "appending" to a file that may be suitable for your purpose is to use /log to log the message at some severity, and define a logging action and corresponding logging destination (for that severity and "script" topic) to write these messages to a file.
Of course it will put date/time on the messages but you may even want that.
I have already inquired about log files, but no one has explained to me specifically what can be done with them and how. And yes, if we are talking about SMS, I would like to record all the data, including the timestamp. The same applies if I am saving data from IoT devices.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12014
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: The maximum size of a read/written file.

Fri Jun 23, 2023 7:31 pm

Log facility already has it's own file management system.
Simply go on system / logging / action menu and manage the logs from that.
/system logging action
add disk-file-count=10 disk-file-name=atlog disk-lines-per-file=1000 name=atlog target=disk
/system logging
add action=atlog topics=gsm,info,!debug
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Fri Jun 23, 2023 9:40 pm

The data will definitely be transmitted and stored on another system, which will be the main system. However, I hope to create a sufficiently resilient system with multiple safeguards. And I hope that one of those safeguards will be MikroTik.
Safeguards regarding networking is MikroTik device domain, but for some advanced approach of data processing which will be provided to main system, I will suggest to run/create some software/scripts in container which is also on same device but you have more flexibility/ability to process data there and deliver to main system, if you have such device that can run containers.
 
msatter
Forum Guru
Forum Guru
Posts: 2912
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: The maximum size of a read/written file.

Sat Jun 24, 2023 1:14 am

You can read BIG files when you put the script for reading the data in the same file.

Using a variable as call to the function, allows to import data as it was used with addresslists. Every variable is a new call of the function and so it will be repeated till there are no more vatiables left.

It will need, on generating the file that the function is inserted first and the data is is put after the variable. You can even have multiple data fields in, by using $1, $2, $3, $.... in the function.

:execute allows to put serveral different stuff in one file. The file is then self contained and does not need a external script to run, it uses the import command in ROS.
Last edited by msatter on Sat Jun 24, 2023 1:22 pm, edited 1 time in total.
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Sat Jun 24, 2023 12:56 pm

:execute allows to put serveral different stuff in one file. The file is then self contained and does need a external script to run, it uses the import command in ROS.
Then it is not data file, it is a script with data variable. Also if data need to be imported from other system then there needs to be generated with script syntax so that can be loaded, just masochism...
 
msatter
Forum Guru
Forum Guru
Posts: 2912
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: The maximum size of a read/written file.

Sat Jun 24, 2023 1:27 pm

Indeed it needs a script to be generate the data file. On import you just import the file that contains a helper to process the data and put it in the correct place in ROS.

That's makes it ideal for distribution of data to many routers.
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Sat Jun 24, 2023 2:07 pm

That's makes it ideal for distribution of data to many routers.
Indeed if all routers are on ROS...
 
msatter
Forum Guru
Forum Guru
Posts: 2912
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: The maximum size of a read/written file.

Sat Jun 24, 2023 6:22 pm

I meant for Asus routers. Wrong forum I see now. Sorry. ;-)
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Sat Jun 24, 2023 6:36 pm

I meant for Asus routers. Wrong forum I see now. Sorry. ;-)
I mentioned it because maybe not everybody has all routers from same brand or when you want just to process same data on other system, doesn't need to be a router system. I will personally rather then process such data in container than additionally convert it into script syntax just for ROS. Ofc. depends on purpose, in some cases maybe it will not be possible to do something from containers, eg. to process large JSON or CSV and create some rules on ROS from it, but for PDU processing which is context of this topic it doesn't matter if processing is done in container with untouched content.
 
DyadyaGenya
Member Candidate
Member Candidate
Topic Author
Posts: 220
Joined: Mon May 08, 2023 10:34 pm

Re: The maximum size of a read/written file.

Sun Jun 25, 2023 10:41 am

Safeguards regarding networking is MikroTik device domain, but for some advanced approach of data processing which will be provided to main system, I will suggest to run/create some software/scripts in container which is also on same device but you have more flexibility/ability to process data there and deliver to main system, if you have such device that can run containers.
I have a vague understanding of what a container is. Could you provide a couple of links where, in your opinion, it is explained in beginner-friendly language?
 
pe1chl
Forum Guru
Forum Guru
Posts: 10248
Joined: Mon Jun 08, 2015 12:09 pm

Re: The maximum size of a read/written file.

Sun Jun 25, 2023 11:50 am

Well, container is only a solution for a very limited subset of the products anyway.
Often it is impossible due to either unsupported processor, not enough RAM, or not enough disk storage (and impossible to add it).

I think it would be reasonable to support "unlimited files" via a "open / read or write records in a loop / close" construct instead of trying to
do everything in a single statement. Or at least support to read a file into an array (one line per array item) instead of a single variable,
and to support write in append mode.
 
optio
Long time Member
Long time Member
Posts: 675
Joined: Mon Dec 26, 2022 2:57 pm

Re: The maximum size of a read/written file.

Sun Jun 25, 2023 1:23 pm

I have a vague understanding of what a container is. Could you provide a couple of links where, in your opinion, it is explained in beginner-friendly language?
https://help.mikrotik.com/docs/display/ROS/Container
https://www.docker.com/resources/what-container/
In general you have additional Linux system on your disposal in ROS. It is supported only on devices with arm, arm64 and x86 architectures.
 
flydvorkin
just joined
Posts: 15
Joined: Mon Mar 11, 2019 12:59 pm

Re: The maximum size of a read/written file.

Sat Jul 01, 2023 12:58 pm

Here are some of my thoughts on bypassing the 4k limit of file operations:

1. Write big data into one file
The above is the correct way to write large files:
:execute script file=filename
where script must print or return data, which "execute" will write to file.
I'll add a few of my thoughts to this:
a) If we want to write data from string variable - it must be remembered that a variable can contain data no larger than 64kb (variable limit, but at least this limit is larger)
b) If we use script=":put \"$data\"" in "ececute" - content must be very simple text string, without special chars, control chars, and other chars which requires to escaped.
The way to output original data with any chars without damage - pass variable into "execute" scope through global environment:
:global tempData $data;
:execute script={ :global tempData; :put $tempData; :set tempData; } file=filename
c) This "execute" method also has an unrecoverable drawback.
Whatever data is written to the file - a newline "\r\n" will always be added at the end of the file.
If we use "return" instead "put" in "execute" script - also "newline" added, but "\n", and in doing so, we lose the ability to remove the global variable at the end of this script. Based on this, it is more preferable to use "put".


2. Read big data from one file to variable
May be use:
:local fileContent ( [ /tool fetch url=( "ftp://127.0.0.1/" . $fileName ) user="login" password="password" port=21 mode=ftp as-value output=user ]->"data" );
via local FTP
But this method involves additional steps and checks.
a) Need save ftp-server "disabled" state and current port
b) Enable ftp-server if than disabled
c) Script need contain plain-text login and password of system user with full-rights. Another way - create temp user in script
d) Read file into variable by "tool fetch" with current port
e) Remove temp user if created
f) Disable the server if it was disabled before this operation (use saved state)
Also remeber - we have a 64kb variable limit.
But if file more than 64kb - it is possible to read at least the first 64kb by this method.
 
User avatar
miku
just joined
Posts: 15
Joined: Fri Feb 23, 2024 3:57 pm
Location: Poland

Re: The maximum size of a read/written file.

Mon Mar 04, 2024 1:50 pm

Maybe my idea is stupid :roll: , but it works for some applications, for example writing the result of the /console/inspect command to a file. You can use the system logging.

I have defined a new action for logging (/system/logging)
name=diskDebug
type=disk
file-name=flash/debug
lines-per-file=100000
file-count=5

Then a new logging rule
topics=debug,script
action=diskDebug

Inside the script, just insert the line:
:log debug message=";$yourVariable"

Note that the message begins with a semicolon so that you can easily separate the lines later (in Notepad++ or Excel...). Of course, you can use any other character for this purpose. If you have an external drive, e.g. a USB stick, you can save files to it instead of /flash. In this case, the file size is limited (probably) only by the disk size.

Who is online

Users browsing this forum: No registered users and 8 guests