Support RFC5819 (LIST-STATUS)

[brandond]

RFC 5819 adds support for extended information (total and unread messages, in the most common case) to be returned in the LIST and LSUB commands. This removes the need to LIST/LSUB and then subsequently retrieve the STATUS of each mailbox.

​http://www.ietf.org/rfc/rfc5819.txt

[brandond]

I wasn't aware that the current IMAP engine doesn't make use of the IMAP4rev1 STATUS command at all - for things like unread message count it actually does a SEARCH ALL UNSEEN and then returns the size of the result list.

I suppose I could open another ticket for Roundcube to use STATUS where possible; perhaps that could be added at the same time as LIST-EXTENDED support. The results from the LIST-EXTENDED versions of LIST/LSUB could be cached; countUnseen (and any other inefficient routines) could be modified to use STATUS instead of SEARCH when cached information is not available.

[brandond]

Add support for IMAP4rev1 STATUS command; use STATUS instead of SEARCH ALL UNSEEN for countUnseen

[brandond]

The patch I just attached adds support for the IMAP4rev1 STATUS command, and (if available) uses it to find unread message counts. On my test system, this executes in about half the time as the current SEARCH method.

We could save a bit of additional work if we used the LIST-EXTENDED extension to precache the data in _listMailboxes, and then used this instead of calling STATUS on each folder. I'll take a look at doing that, although I'm not sure at this point how to go about caching, and whether or not it should be done unconditionally.

[alec]

Another related extension is ESEARCH (RFC4731). RFC3501 says: "Unlike the LIST command, the STATUS command is not guaranteed to be fast in its response. Under certain circumstances, it can be quite slow." What server are you using? I'd like to know what would be the performance of "SEARCH RETURN (COUNT) UNSEEN" in comparision with STATUS and with basic SEARCH.

[brandond]

Yeah, SELECT is not guaranteed to be fast, but neither is SEARCH. I'm using Dovecot, which does optimize flag searches pretty well, but I believe that STATUS is even further optimized - Timo is co-author of RFC5819, and recognizes that retrieving STATUS for a large number of folders is common client behavior; see section 2.

I think most of the savings over the current implementation are just on the wire - I have a number of folders with thousands of unread messages, and just receiving and parsing the UID list is time-consuming.

I'll see if I can get some timing data on STATUS vs ESEARCH, but it should be noted that ESEARCH requires an additional round trip to the server to SELECT the folder before SEARCHing. For Roundcube this means we verify that the folder was selected successfully, parse and store the STATUS response fields, etc, all even before the SEARCH can begin - whereas STATUS results are returned and parsed immediately.

[brandond]

I asked Timo for his input on STATUS versus SEARCH, he says:

The reason why STATUS is mentioned to be possibly slow is to discourage clients from doing a STATUS to all mailboxes.
STATUS is definitely faster than SELECT+SEARCH with all IMAP servers.

[alec]

Thanks for the info, we'll probably use STATUS then, but we should check for LIST-EXTENDED capability, because if this isn't supported there's no sense to call STATUS and then SELECT+SEARCH.
Still ESEARCH would be useful when when skip_deleted=true or STATUS doesn't return UNSEEN (for any reason).

[brandond]

I think I may have confused the issue by getting into STATUS; STATUS is IMAP4rev1 core while LIST-EXTENDED and ESEARCH are IMAP4rev1 extensions. As I see it:

• LIST-EXTENDED could be used to populate icache for EXISTS/UNSEEN counts during LIST/LSUB for later use by some cases of messagecount.
• STATUS could be used for servers that don't support LIST-EXTENDED or when data is not in icache
• SELECT+ESEARCH could be used for more complicated scenarios (skip_deleted, etc)
• SELECT+SEARCH or SELECT+THREAD is fallback for older servers or more complicated cases

Does that sound right?

[alec]

Right. My mistake in last comment. I've miss that UNSEEN is standard (RFC3501) STATUS response, so checking for LIST-EXTENDED before using STATUS is not needed. There is a place in RFC3501 saying that UNSEEN is for "number of the first message without the \Seen flag set" (see 7.1), but STATUS command response is described as number of messages. This is what I've miss.

[alec]

Patch with small mods added in [710e2748]. I'll work on ESEARCH support.

[alec]

ESEARCH support added in [659cf14c].

[alec]

I've added LIST-STATUS support to rcube_imap_generic class in [6a16031d]. Using it will require a lot more changes. Currently I don't have a test IMAP server supporting LIST-STATUS.

[brandond]

I guess I'm unclear what the difference is between LIST-STATUS and LIST-EXTENDED. I suppose I should re-read the RFC.

Anyway, I've got a server set up with Dovecot 2.0.5 and an account for you. I'll e-mail you privately with login information. I also set up a shared public namespace, as I seem to recall that was something that was awaiting further testing as well.

[brandond]

DNS has been fixed. whitestar.uoregon.edu - see e-mail for credentials.

[brandond]

Changes so far are looking really great! Two comments:

• The message count column in the folder management view is still using SELECT instead of STATUS:

  --- program/steps/settings/manage_folders.inc.old       2010-10-26 11:18:47.000000000 -0700
  +++ program/steps/settings/manage_folders.inc   2010-10-26 11:18:06.000000000 -0700
  @@ -299,7 +299,7 @@
       $table->add_row(array('id' => 'rcmrow'.$idx, 'class' => join(' ', $classes)));
       
       $table->add('name', Q($display_folder));
  -    $table->add('msgcount', ($folder['virtual'] ? '' : $IMAP->messagecount($folder['id'], 'ALL', false, false)));
  +    $table->add('msgcount', ($folder['virtual'] ? '' : $IMAP->messagecount($folder['id'], 'ALL', false, true)));
       $table->add('subscribed', $checkbox_subscribe->show(($subscribed ? $folder_utf8 : ''),
         array('value' => $folder_utf8, 'disabled' => ($protected || $noselect) ? 'disabled' : '')));
       if ($threading_supported) {
  

• Use of LIST (\Subscribed) is great - I'm guessing next step is to add flag to enable use of RETURN and then cache result instead of doing explicit STATUS later? This is the final thing necessary to do away with 90% of folder iteration round-trips. ie:

  A0003 LIST (SUBSCRIBED) "" "*" RETURN (STATUS (MESSAGES UNSEEN))
  * LIST (\Subscribed) "/" "Trash"
  * STATUS "Trash" (MESSAGES 4 UNSEEN 0)
  * LIST (\Subscribed) "/" "Sent"
  * STATUS "Sent" (MESSAGES 6 UNSEEN 0)
  * LIST (\Subscribed) "/" "Junk"
  * STATUS "Junk" (MESSAGES 0 UNSEEN 0)
  * LIST (\Subscribed) "/" "Drafts"
  * STATUS "Drafts" (MESSAGES 0 UNSEEN 0)
  

Sorry if I'm bothering you, I'm just really excited to see this come together!

[alec]

Replying to brandond:

• The message count column in the folder management view is still using SELECT instead of STATUS:
  

Fixed in [36ed9d7e].

• Use of LIST (\Subscribed) is great - I'm guessing next step is to add flag to enable use of RETURN and then cache result instead of doing explicit STATUS later?
  

Right.