Community discussions

MUM Europe 2020
 
AmrSubZero
just joined
Topic Author
Posts: 9
Joined: Tue Dec 02, 2014 3:29 pm
Location: Cairo/Egypt
Contact:

Which Charset does Mikrotik use for displaying Arabic and how to convert a string to UTF-8

Thu Jul 07, 2016 9:28 am

I'm using the Mikrotik Java API by GideonLeGrange.

But i have problem with displaying the Arabic words in my android project, it's displaying like diamond Question marks "���", i've searched the whole internet to know what kind of problem is this, tried to convert from "ISO 8859-1", "WINDOWS-1256" and "WINDOWS-1252" to "UTF-8" but all failed, the "ISO 8859-1" displayed "question marks" (not inside a the black diamonds) and i assume that it's a Unicode problem (and i have no idea how to solve it), also the windows 1256 & 1252 displayed nothing (empty string) but it displays numbers if there was.

Here's some examples that i used in Java (Android):
String str = new String(originalStr.getBytes("ISO-8859-1"), "UTF-8"); // returns question marks but not inside the black square

String str = new String(originalStr.getBytes("WINDOWS-1256"), "UTF-8"); // returns empty string

String str = new String(originalStr.getBytes("WINDOWS-1252"), "UTF-8"); // also returns empty string
I think that i need to know which charset does Mikrotik use to display arabic words, which i should convert from, to "UTF-8" because my android project charset is "UTF-8"
I'm truly confused and don't know what to do, so guys can you help with that?

NOTE : in PHP i used iconv and converted from "windows-1252" to "utf-8" and the arabic displayed correctly, but what about java (Android)?
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 24315
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: Which Charset does Mikrotik use for displaying Arabic and how to convert a string to UTF-8

Thu Jul 07, 2016 9:39 am

RouterOS does not suppport UTF8 or actually any non standard characters. Basically you can only rely on a-z 0-9
No answer to your question? How to write posts
 
AmrSubZero
just joined
Topic Author
Posts: 9
Joined: Tue Dec 02, 2014 3:29 pm
Location: Cairo/Egypt
Contact:

Re: Which Charset does Mikrotik use for displaying Arabic and how to convert a string to UTF-8

Thu Jul 07, 2016 9:54 am

I think that's not a practical solution in my case, if Winbox uses the charset "windows-1256" for arabic support, i need to understand how Mikrotik send the data to be displayed, does it send in Bytes? my problem is that i've no idea what the data type is! and then i could convert it to utf-8 like i did in PHP using iconv.
 
User avatar
boen_robot
Forum Guru
Forum Guru
Posts: 2411
Joined: Thu Aug 31, 2006 4:43 pm
Location: europe://Bulgaria/Plovdiv

Re: Which Charset does Mikrotik use for displaying Arabic and how to convert a string to UTF-8

Thu Jul 07, 2016 4:46 pm

The API sends and receives the raw bytes, and Winbox displays the bytes using the Windows ANSI charset, which is different for every locale. If you've successfully converted windows-1252 to UTF-8 with iconv, then you should be able to do the same with Java.

BUT, the problem with Java then becomes where and how you output the result. If you were to write the UTF-8 output to a text file, and read it with a UTF-8 aware text editor, you should see the correct characters.

Outputting the text on a Windows terminal however would use Windows' "OEM code page" charset, which is NOT the same as the "windows-*" charset. And outputting it in the GUI... depends on the GUI toolkit, and I'm not personally familiar with Android's rules. If UTF-8 doesn't work, then my next guess would be UTF-16LE or UTF-16BE.
PEAR2_Net_RouterOS(1.0.0b6) - My API client in PHP
(Rate my posts? If you want... no pressure...)
 
legrang
just joined
Posts: 22
Joined: Wed Nov 03, 2010 4:05 pm
Location: South Africa
Contact:

Re: Which Charset does Mikrotik use for displaying Arabic and how to convert a string to UTF-8

Sat Oct 08, 2016 8:08 pm

@AmrSubZero I think I have a fix for this. If you want to test the fix, please pull the current master branch from github and build and try that version.

As an experiment, I've changed the internal encoding for transmission between the mikrotik-java API and RouterOS to be UTF-8 and it seems to solve the problem. I'll release version 3.0.3 shortly to make this change available.

Thanks to @boen_robot - your comments above pointed me in the right direction, which lead to this StackOverflow article http://stackoverflow.com/questions/5729 ... g-to-utf-8
A Java String is internally always encoded in UTF-16 - but you really should think about it like this: an encoding is a way to translate between Strings and bytes.
So if you have an encoding problem, by the time you have String, it's too late to fix. You need to fix the place where you create that String from a file, DB or network connection.
Gideon le Grange
RouterOS Java API: https://github.com/GideonLeGrange/mikrotik-java
 
User avatar
boen_robot
Forum Guru
Forum Guru
Posts: 2411
Joined: Thu Aug 31, 2006 4:43 pm
Location: europe://Bulgaria/Plovdiv

Re: Which Charset does Mikrotik use for displaying Arabic and how to convert a string to UTF-8

Sat Oct 08, 2016 8:49 pm

@legrang I'd highly recommend you make the charset configurable, even if it does default to UTF-8. Some people may want to use an ANSI charset (e.g. "windows-1252", as AmrSubZero, or "windows-1251", as Cyrillic using users, etc.), so that the contents can still be read from Winbox, as well as their app.
PEAR2_Net_RouterOS(1.0.0b6) - My API client in PHP
(Rate my posts? If you want... no pressure...)

Who is online

Users browsing this forum: No registered users and 15 guests