Inconsistent results with JSON serialize, deserialize and numbers

xrlls · Thu Dec 28, 2023 11:43 am

A word of warning with the new serialize and deserialize commands, each of them do different conversion, so the process is not reversible!

I have been working a bit with the new serialise and deserialize commands and JSON containing various types of numbers, and the results are wildly inconsistent.

An example JSON containing various types of numbers, and numbers as strings:

{
    "1": "7",
    "2": 7,
    "3": "7.1",
    "4": 7.1,
    "5": 7.14,
    "6": "7.14",
    "8": "7.1234567"
}

First deserialized to an array, and then serialized to JSON:

global a "{\"1\":\"7\",\"2\":7,\"3\":\"7.1\",\"4\":7.1,\"5\":7.14,\"6\":\"7.14\",\"8\":\"7.1234567\"}"    
global b [deserialize $a from=json ]                                                                  
put $b                                                                                                
1=7;2=7;3=7.0.0.1;4=7.1;5=7.14;6=7.0.0.14;8=00:00:07.123456700
global c [serialize $b to=json ]                                                                      
put $c                                                                                                
{"1":7,"2":7,"3":"7.0.0.1","4":7.100000,"5":7.140000,"6":"7.0.0.14","8":"1970-01-01 00:00:07"}

And the result:

{
    "1": 7,
    "2": 7,
    "3": "7.0.0.1",
    "4": 7.1,
    "5": 7.14,
    "6": "7.0.0.14",
    "8": "1970-01-01 00:00:07"
}

Also both strings and number are converted to numbers when serializing, however not the same numbers:

global d 
set ($d->"1") "7"
set ($d->"2") 7   
global e [serialize $d to=json ]
put $d
1=7;2=7
put $e                          
{"1":7.000000,"2":7}

This shows that a number contained in a string is always converted to a number with 6 decimals, while the same number as a number is converted to an integer.

I have created a support ticket on this also: SUP-138863

rextended · Thu Dec 28, 2023 11:54 am

Since RouterOS doesn't support floating point numbers, it can't import them from JSON either.
And speaking of the "dot", if there is a dot the data tries to be interpreted as an IP, so 7.1 is 7.0.0.1 and 7.14 is 7.0.0.14 as an IP...

The true problem is that if the value is quoted "" must be treated as string, and not converted to number, IP or time

On my code I added also missing null, boolean, array and object examples.

:global a "{\"0\":null,\"0q\":\"null\",\"1\":7,\"1q\":\"7\",\"2\":7.1,\"2q\":\"7.1\",\"3\":false,\"3q\":\"false\",\"4\":[\"A\",7.1,\"7.1\"],\"5\":{\"X\":7.1,\"Xq\":\"7.1\"}}"
:global b [:deserialize $a from=json]
:foreach x,y in=$b do={
   :put "$x = $[:tostr $y] as $[:typeof $y]"
}

json code

{
     "0":null,
    "0q":"null",
     "1":7,
    "1q":"7",
     "2":7.1,
    "2q":"7.1",
     "3":false,
    "3q":"false",
     "4":["A",7.1,"7.1"],
     "5":{
           "X":7.1,
          "Xq":"7.1"
         }
}

results code

0 =  as nil
0q = null as str
1 = 7 as num
1q = 7 as num
2 = 7.1 as str
2q = 7.0.0.1 as ip
3 = false as bool
3q = false as bool
4 = A;7.1;7.0.0.1 as array
5 = X=7.1;Xq=7.0.0.1 as array

Also inside the converted arrays is present the same problem.

Amm0 · Thu Dec 28, 2023 4:56 pm

There are no floats in RouterOS script, so that there "lost in translation" is to be expected IMO — there is no way to store the floating point number. So it becomes a string, and thus a string if later serialized back to JSON. This is expected behavior IMO since "serialized" has no way to know the variables was originally a floating point. So str type make sense here:

:put [:typeof ([:deserialize "{\"x\":1.1}" from=json ]->"x")] 
str

I think it's RouterOS "ip" and "ip-prefix" variable types that make things look wrong... RouterOS script allows pre-CIDR IPv4 addressing so 1.1 are 1.1.1 are both valid IP addresses so if used without quotes they becomes an "ip" type in script — not a floating point. This is just how RouterOS script work, and since JSON does NOT have an IP address type, it comes a string. And essentially ":tostr" is performed on the IP type before serializing into JSON. So this make sense...in the context of how RouterOS script types work that is

:put [:serialize to=json 1.1.1]                                 
"1.1.0.1"

@rextended, there is the "time" type too, which is handled correctly IMO:

:put [:serialize to=json [:timestamp]]     
"2023-12-28 14:33:04"

but that is also asymmetric, similar to floats (but in reverse), since JSON does NOT have a time type. So "serialized" make the time a ISO-like date as a JSON string. Also seems right to me.

Perhaps more simply, the RouterOS types and JSON types don't perfectly align — so something has to happen.
e.g. RouterOS does not have floats & JSON does not have ip address nor time types.
So...when type mismatch happens... it becomes a string in either serialize/deserialize direction is the basic rule.

xrlls · Thu Dec 28, 2023 5:04 pm

I agree on the quotes must be treated as string! However I can see where they are coming from, as the current functionality was probably a work around to allow writing floats to JSON, such as voltage, power consumption or some other measure that is typically handled by strings by ROS. It just comes with a lot of downsides.

Anyway, I’ll wait and see what support has to say.

Amm0 · Thu Dec 28, 2023 5:20 pm

Yeah the lack of float comes up more generally.

But you're correct there is no way to generate a floating point type in the JSON that's serialized, even with more scripting. Right now, the other end receiving the JSON have to deal with either /100 the integer or convert a string to float. But... depending on the backend server, some services (like a database and/or have schema that know some JSON field should be a float) will also just convert a float in a string since the world of JSON is messy beyond RouterOS. So a small script to convert temp/etc into a RouterOS string storing floating point, before serialize, may work... depending on what using the JSON.

rextended · Thu Dec 28, 2023 5:36 pm

@rextended, there is the "time" type too, which is handled correctly IMO:

time type not exist on json, and...

The true problem is that if the value is quoted "" must be treated as string, and not converted to number, IP or time

but also (non-existant on json) id, ip, ip6, ip-prefix, ip6-prefix, nothing, lookup (, code, function, op, apireq, backreference, cmd, exclamation, iterator...)

you do not catch the point:

if the value is quoted "" (inside the json) must be treated as string

Example for store decimals:

example for simple division code

{
:local a 5
:local b -3
:local p 2 ; # precision, how many decimals

:local m 1
:for counter=tmp from=1 to=$p step=1 do={:set m ($m * 10)}
:local r "$($a * $m / $b)"
:set r "$[:pick $r 0 ([:len $r] - $p)].$[:pick $r ([:len $r] - $p) [:len $r]]"
:put "$a / $b = $r"
:put [:serialize to=json {"result"=$r}]
}

result code

5 / -3 = -1.66
{"result":-1.660000}

Max :serialize decimals are 6 and approximated, for example -1.6666666666 is serialized to -1.666667