Comments on: WTF-16 http://blog.mozilla.com/dmandelin/2008/02/14/wtf-16/ Just another Blog.mozilla.com weblog Wed, 25 Jan 2012 23:29:35 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: blassey http://blog.mozilla.com/dmandelin/2008/02/14/wtf-16/comment-page-1/#comment-50 blassey Mon, 18 Feb 2008 21:01:35 +0000 http://blog.mozilla.com/dmandelin/2008/02/14/wtf-16/#comment-50 This is just a guess, but I would assume that the cost of the conversions in processing time, heap size and code size would be more damaging to performance on mobile devices than the reduction of memory use. Perhaps what is really needed is an abstraction layer that only converts the string when necessary. For instance, if you follow the start up on windows of xulrunner, command line arguments are passed in as UTF-16 and immediately converted to UTF-8. Various string manipulations are performed and then various system APIs are called, but only after the UFT-8 strings are converted back to UTF-16. The memory for the UTF-8 strings isn't freed until shutdown. This is just a guess, but I would assume that the cost of the conversions in processing time, heap size and code size would be more damaging to performance on mobile devices than the reduction of memory use.

Perhaps what is really needed is an abstraction layer that only converts the string when necessary. For instance, if you follow the start up on windows of xulrunner, command line arguments are passed in as UTF-16 and immediately converted to UTF-8. Various string manipulations are performed and then various system APIs are called, but only after the UFT-8 strings are converted back to UTF-16. The memory for the UTF-8 strings isn’t freed until shutdown.

]]> By: dmandelin http://blog.mozilla.com/dmandelin/2008/02/14/wtf-16/comment-page-1/#comment-48 dmandelin Fri, 15 Feb 2008 21:47:25 +0000 http://blog.mozilla.com/dmandelin/2008/02/14/wtf-16/#comment-48 Matthew: Good points. IMO, ease of use for programmers really is the biggest reason to think about decreasing the number encodings. Based on what I hear, it sounds like we would need some more help to make such a big change happen for Mozilla 2. Kim: That's the only problem I know of with UCS-2. I'm not sure how important Linear B and Phoenician support really are anyway, but it just sort of bugs me that UCS-2 isn't truly a Unicode encoding any more. Matthew: Good points. IMO, ease of use for programmers really is the biggest reason to think about decreasing the number encodings. Based on what I hear, it sounds like we would need some more help to make such a big change happen for Mozilla 2.

Kim: That’s the only problem I know of with UCS-2. I’m not sure how important Linear B and Phoenician support really are anyway, but it just sort of bugs me that UCS-2 isn’t truly a Unicode encoding any more.

]]> By: Kim Sullivan http://blog.mozilla.com/dmandelin/2008/02/14/wtf-16/comment-page-1/#comment-47 Kim Sullivan Fri, 15 Feb 2008 10:48:59 +0000 http://blog.mozilla.com/dmandelin/2008/02/14/wtf-16/#comment-47 UCS-2 is mostly the same as UTF16 - UTF16 just has the advantage of supporting surrogate pairs. Why exactly do you revile UCS-2, while UTF-16 seems to be considered a viable alternative? (I agree that not supporting U+10000 through U+10FFFF is a serious drawback, just wanted to know if there was something else to it). UCS-2 is mostly the same as UTF16 – UTF16 just has the advantage of supporting surrogate pairs. Why exactly do you revile UCS-2, while UTF-16 seems to be considered a viable alternative?

(I agree that not supporting U+10000 through U+10FFFF is a serious drawback, just wanted to know if there was something else to it).

]]> By: Matthew Gertner http://blog.mozilla.com/dmandelin/2008/02/14/wtf-16/comment-page-1/#comment-46 Matthew Gertner Fri, 15 Feb 2008 09:44:48 +0000 http://blog.mozilla.com/dmandelin/2008/02/14/wtf-16/#comment-46 Although this will never be the primary consideration, I wouldn't discount the value of a more consistent API for developers. The Mozilla platform is large, complex and daunting as it is. It's pretty baffling for the newbie Mozilla coder to figure out why certain APIs take UTF-8 and other UTF-16, seemingly at random. For people designing new APIs for Mozilla, there is a constant decision of whether to use one ("but then I'll have to convert to use so-and-so API") or the other ("but then I'll have to convert for some other API"). Conversion code isn't that slow, perhaps, but it does muck up and obfuscate code. As I say, not the most important factor, perhaps, but all things being equal (and it sounds like they are quite emphatically not equal and that UTF-8 is superior), harmonizing on a single Unicode encoding would be a great step. Given all the big changes planned for Mozilla 2, I'd be disappointed if this one didn't make the cut. Although this will never be the primary consideration, I wouldn’t discount the value of a more consistent API for developers. The Mozilla platform is large, complex and daunting as it is. It’s pretty baffling for the newbie Mozilla coder to figure out why certain APIs take UTF-8 and other UTF-16, seemingly at random. For people designing new APIs for Mozilla, there is a constant decision of whether to use one (“but then I’ll have to convert to use so-and-so API”) or the other (“but then I’ll have to convert for some other API”). Conversion code isn’t that slow, perhaps, but it does muck up and obfuscate code.

As I say, not the most important factor, perhaps, but all things being equal (and it sounds like they are quite emphatically not equal and that UTF-8 is superior), harmonizing on a single Unicode encoding would be a great step. Given all the big changes planned for Mozilla 2, I’d be disappointed if this one didn’t make the cut.

]]> By: Robert O'Callahan http://blog.mozilla.com/dmandelin/2008/02/14/wtf-16/comment-page-1/#comment-39 Robert O'Callahan Thu, 14 Feb 2008 23:47:44 +0000 http://blog.mozilla.com/dmandelin/2008/02/14/wtf-16/#comment-39 CJK characters are 3 bytes each in UTF-8. That's why the memory usage reduction is surprising. The Windows and Mac text APIs are not a problem for UTF-8. We already have a string copy in the textrun construction path that we could use to convert from UTF-8 to UTF-16 as well. CJK characters are 3 bytes each in UTF-8. That’s why the memory usage reduction is surprising.

The Windows and Mac text APIs are not a problem for UTF-8. We already have a string copy in the textrun construction path that we could use to convert from UTF-8 to UTF-16 as well.

]]>