get_headers function (rcube_imap class) use of message caching

[iznogoud]

Listing several mailboxes at once (see code below), I've seen that the get_headers method returns wrong headers if the headers of the same message id (or uid) was already asked using the same method on a different mailbox (i.e. same id (or uid) on different mailboxes).

My solution was to add :

unset ($this->icache['message']);

as the first line of the get_headers method (in the program/include/rcube_imap.php file)

Code sample :

$message_list = $rcmail->imap->message_index($folder, 'Date:*', 'ASC');
foreach ($message_list as $message_id) {
$my_header = $rcmail->imap->get_headers($message_id, $folder, false, false);
/* $my_header->timestamp or $my_header->subject would be the same
// as previously if the same $message_id of a different $folder
// has already been returned
}

[alec]

get_headers() and get_structure() are not designed for use in loops. This is not good for performance. However, when talking 'mailbox' you mean other account not folder, right? Maybe we should clear internal cache on disconnect?

[iznogoud]

Replying to alec:

get_headers() and get_structure() are not designed for use in loops. This is not good for performance.

I can understand that, but as I need to parse all an account's folders for "old" messages (that's the purpose of the plugin I wrote : cleaning "old" messages, at least Sent and Trash folders), I haven't found an other solution.

However, when talking 'mailbox' you mean other account not folder, right? Maybe we should clear internal cache on disconnect?

No, I mean folder (all the folders of the same account). I've tried to clear the internal cache when changing folder (even closing and re-opening the IMAP connection), but it didn't help.

[alec]

Replying to iznogoud:

No, I mean folder (all the folders of the same account). I've tried to clear the internal cache when changing folder (even closing and re-opening the IMAP connection), but it didn't help.

There shouldn't be two messages with the same UID in different folders. The issue must be in other place. BTW, currently there's no code which will clean the cache on reconnect.

[iznogoud]

Replying to alec:

There shouldn't be two messages with the same UID in different folders.

That was a big surprise for me to discover that I've got the same UID for two different messages in two different folders using message_index() on different folders.
I've even found messages with different id but sharing the same uid

See this log:

[28-Jan-2011 15:24:00 +0100]: Folder INBOX 6 messages
[28-Jan-2011 15:24:00 +0100]: Folder INBOX id 1 uid 1 timestamp : 1073400526
[28-Jan-2011 15:24:00 +0100]: Folder INBOX id 2 uid 9 timestamp : 1157564645
[28-Jan-2011 15:24:00 +0100]: Folder INBOX id 3 uid 26 timestamp : 1260997715
[28-Jan-2011 15:24:00 +0100]: Folder INBOX id 4 uid 27 timestamp : 1274695058
[28-Jan-2011 15:24:00 +0100]: Folder Trash 62 messages
[28-Jan-2011 15:24:00 +0100]: Folder Trash id 1 uid 1 timestamp : 1073400526
[28-Jan-2011 15:24:00 +0100]: Folder Trash id 9 uid 9 timestamp : 1157564645

The folder's message # was obtained using :

$rcmail->imap->messagecount($folder_name);

and The message uid:

$msg_uid = $rcmail->imap->get_uid($message_id,$folder_name);

The timestamp:

$my_header->timestamp

[alec]

Do you have enabled caching? What IMAP server?

[alec]

Please, provide more info and imap_debug log for those operations above. Works for me.

[iznogoud]

Caching :

main.inc.php:$rcmail_config['enable_caching'] = false;

IMAP server : Dovecot

The log above isn't enough ? When you write "Works for me", do you mean that you don't have similar uid for messages in different mailboxes (because if you don't have similar uid for different message, you can't reproduce the real problem.....) ?

As I've deleted a lots of messages from my mailboxes (that was the aim of the plugin I wrote...), I don't see this problem anymore.... that's a pity because it IS real, and this means that the bug is still present.
I'll try my best to reproduce it, and send you an imap_debug file.

[alec]

I've requested an imap_debug because we need to make sure the UIDs are not unique in server responses. If so, it's a server issue.

[iznogoud]

Replying to alec:

I've requested an imap_debug because we need to make sure the UIDs are not unique in server responses. If so, it's a server issue.

If I'm able to get duplicated uid again, I'll send you and imap_debug.

PS: Even if that's a server issue, as soon as I added

unset ($this->icache['message']);

as the first line of the get_headers method (in the program/include/rcube_imap.php file),

$rcmail->imap->get_headers($message_id, $folder, false, false);

returned the right message headers (of the right folder, and not the one of the previously listed folder), allowing my plugin to work flawlessly even with a potentially buggy IMAP server.

[alec]

Check with attached patch.

[iznogoud]

Replying to alec:

Check with attached patch.

Thank you, I'll try as soon as I'll be able to get duplicated uid again.

[alec]

BTW, what Dovecot version?

[iznogoud]

A pretty old version : 1.0rc15

If I succeed having duplicate uid again, I'll try to upgrade after testing your patch.

[iznogoud]

imap_debug with some duplicate uid

[iznogoud]

Plugin log, highlighting duplicate UIDs

[iznogoud]

Hello,

As I'm able to trace duplicate UIDs again, here are two new files :

imap.zip contains a zipped imap_debug
foldermaintenance is a list of duplicated UIDs found by my plugin.

Feel free to ask any more infos.

BR (and thanks againg for your help)

[alec]

Question is, did you test my patch?

[iznogoud]

Sorry I forgot to mention : yes I'vre tried it before sending the log files, I haven't seen any difference.

[iznogoud]

Hello.. Still no news on this issue ?

BR

[alec]

I'll work on this if you provide an account so I'll be able to connect to it from my box.

[iznogoud]

I'm not sure to fully understand :

• Do you need to connect to your own IMAP server through my Roundcube implementation ?
• Or as a regular user (my IMAP server) ?

[alec]

I want to connect to your server from my Roundcube instance. Of course the account must contain some duplicated-uid messages.

[iznogoud]

As of today the only account I know containing duplicated-uid messages is mine (Roundcube is still in "evaluation mode", expected to replace OpenWebMail? as the standard webmail).

I'll try to find a solution (mailbox copy or other account).