Serialization of header data with non ascii characters fails on unicode postgresql database

[NetCompany]

Some non-compliant messages contain latin1 characters in their headers. These messages generate the following error when loading my inbox:

[Sat Oct 13 15:05:21 2007] [error] [client 213.84.186.140] PHP Notice:  DB Error: unknown error Query: INSERT INTO messages (user_id, del, cache_key, created, idx, uid,
 subject, "from", "to", cc, date, size, headers, structure) VALUES ('1', 0, 'INBOX.msg', now(), '20021', '102068', 'RE: Re: [Fwd: MAGISTER-SYNC: geen records opgehaald 
uit magister!]', 'Ren\xc3\xa9 Klein <rene.klein@hml.nl>', 'Kasper Schoonman <kasper@netcompany.nl>', '', '2007-10-11 13:55:35', 17378, 'O:14:"iilBasicHeader":25:{s:2:"i
d";s:5:"20021";s:3:"uid";s:6:"102068";s:7:"subject";s:66:"RE: Re: [Fwd: MAGISTER-SYNC: geen records opgehaald uit mag in /usr/share/roundcube/program/include/bugs.inc o
n line 80, referer: http://netmail.nu/

I think the problem is in rcube_imap.inc around line 2142. Here the header data is serialized with the PHP serialize() function. If this header data is not ASCII (or UTF-8) postgresql with a unicode database will complain about mall-formed UTF-8 sequences.

I do not directly see what the right solution will be. Either the field has to change from text to bytea type. Or the data has to be encapsulated using base64 or maybe even UTF-8.

[seansan]

Review in 0.1.1

[alec]

Confirmed. The problem isn't only with headers and structure columns. Also subject, from and to must be taken into consideration. Bytea type is not a solution for such problem.

[till]

I thought MDB2 handled all that? Or is the data itself not UTF-8?

[alec]

Replying to till:

I thought MDB2 handled all that? Or is the data itself not UTF-8?

the data

[dano]

I see lot of errors like this:

MDB2 Error: unknown error (-1): _doQuery: [Error message: Could not execute statement][Last executed query: EXECUTE mdb2_statement_pgsql_blabla ('1', 'INBOX.msg', '2', '3456', 'subject', 'Bádnáme  <email@example.com>', 'User <user@example.com>', '', 33772, 'O:14:"iilBasicHeader":31:{s:2:"id";s:2:"47";s:3:"uid";s:4:"3456";s:7:"subject";s:11:"subject";s:1:"from";s:65:""=?windows-1250?B?S+FsbeFuIEguIFNpbHZpYQ==?= " <mail@example.com>";s:2:"to";s:44:""User" <user@example.com>";s:2:"cc";N;s:7:"replyto";N;s:11:"in_reply_to";N;s:4:"date";s:60:"Fri, 31 Oct 2008 22:27:11 +0100 (St?edn? Evropa (b?<9E>n? ?as))";s:9:"messageID";s:28:"<490B782F.00001F.02752@KHSZ>";s:4:"size";s:5:"33772";s:8:"encoding";N;s:7:"charset";s:10:"iso-8859-1";s:5:"ctype";s:17:"Multipart/related";s:5:"flags";a:3:{i:0;s:8:"Answered";i:1;s:7:"Deleted";i:2;s:4:"Seen";}s:9:"timestamp";i:1225484831;s:1:"f";N;s:12:"internaldate";s:20:"31-Oct-2008 22:27:24";s:10:"references";N;s:8:"priority";i:3;s:6:"mdn_to";N;s:8:"mdn_sent";b:0;s:
8:"is_draft";b:0;s:4:"seen";b:1;s:7:"deleted";b:1;s:6:"recent";b:0;s:8:"answered";b:1;s:9:"forwarded";b:0;s:4:"junk";b:0;s:7:"flagged";b:0;s:9:"structure";N;}', NULL)]
[Native message: ERROR:  invalid byte sequence for encoding "UTF8": 0xf8
HINT:  This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".]

[01-Nov-2008 12:46:19 +0100] DB Error: MDB2 Error: unknown error Query: _doQuery: [Error message: Could not execute statement] [Last executed query: EXECUTE mdb2_statement_pgsql_blabla ('1', 'INBOX.msg', '2', '3456', 'subject', 'Bádnáme <email@example.com>', 'User <user@example.com>', '', 33772, 'O:14:"iilBasicHeader":31:{s:2:"id";s:2:"47";s:3:"uid";s:4:"3663";s:7:"subject";s:11:"subject";s:4:"from";s:65:""=?windows-1250?B?S+FsbeFuIEguIFNpbHZpYQ==?= " <email@example.com>";s:2:"to";s:44:""User" in /path/rc/program/include/rcube_mdb2.php on line 540 (GET /rc/?_task=mail&_mbox=INBOX&_refresh=1)

is this related to this bug?

[alec]

Fixed using utf8_encode/utf8_decode in [ac622998].