UTF-8 bug

[sergyn]

In message list, russian letter Р displayed as ? in Browser when navigation from page to page. Seems AJAX make something wrong?

[till]

Can you provide a patch for your messages file? I sure cannot. :)

[thomasb]

Remove the following lines from rep_specialchars_output() should help:

if ($enctype!='html')
  $str = str_replace(chr(160), ' ', $str);

Suggested by Yoshikazu Tsuji

[thomasb]

Fixed in [74ae8849]

[yskzt@…]

I searched web sites.

• The chr(160) in UTF-8 is incorrect. (​http://en.wikipedia.org/wiki/UTF-8)
• In UTF-8, no-break space(nbsp) is 0xC2 0xA0. (​http://www.fileformat.info/info/unicode/char/00a0/index.htm)

So, how about this?

--- main.inc_   2007-09-03 16:10:32.000000000 +0900 (rev 774)
+++ main.inc    2007-09-12 01:05:31.000000000 +0900 (changed)
@@ -1103,6 +1103,17 @@
   return $str;
   }

+function mb_str_replace($search_str, $replace_str, $str)
+  {
+  $current_pos = 0;
+  while (($found_pos = mb_strpos($str, $search_str, $current_pos)) !==
false)
+    {
+    $str = mb_substr($str, 0, $found_pos).$replace_str.mb_substr($str,
$found_pos + mb_strlen($search_str));
+    $current_pos = $found_pos + strlen($replace_str);
+    }
+
+  return $str;
+  }

 /**
  * Replacing specials characters to a specific encoding type
@@ -1123,7 +1134,12 @@

   // convert nbsps back to normal spaces if not html
   if ($enctype!='html')
-    $str = str_replace(chr(160), ' ', $str);
+    {
+    if ($OUTPUT->get_charset()=='UTF-8')
+      $str = mb_str_replace(chr(194).chr(160), ' ', $str);
+    else
+      $str = str_replace(chr(160), ' ', $str);
+    }

   // encode for plaintext
   if ($enctype=='text')

[yskzt@…]

I noticed that the mb_str_replace function can be replace to mb_ereg_replace function. mb_str_replace fuction was the function I added and bad naming.

--- main.inc_   2007-09-03 16:10:32.000000000 +0900 (rev 774)
+++ main.inc    2007-09-12 01:05:31.000000000 +0900 (changed)
@@ -1103,6 +1103,17 @@
   return $str;
   }

 /**
  * Replacing specials characters to a specific encoding type
@@ -1123,7 +1134,12 @@

   // convert nbsps back to normal spaces if not html
   if ($enctype!='html')
-    $str = str_replace(chr(160), ' ', $str);
+    {
+    if ($OUTPUT->get_charset()=='UTF-8')
+      $str = mb_ereg_replace(chr(194).chr(160), ' ', $str);
+    else
+      $str = str_replace(chr(160), ' ', $str);
+    }

   // encode for plaintext
   if ($enctype=='text')

[thomasb]

We simply leave nbsps untouched as any other character. There's no need for a complicated fix like this.