RoundCube and performance

[Emil Wojak]

Hi!

Since RoundCube behaves rather poorly on my machine, I decided to find out what takes it so long.
At first I thought it might be network related, but a bit of tcpdump sniffing at the server side showed it's not - a single HTTP request for fetching messages list is processed for about 4-5 seconds by the server.

It's not the IMAP server that's slowing down either - Dovecot responds nearly instantly even for sorting requests in mailboxes filled with about 2000 messages (Wireshark screenshot with packet times attached).

I did some profiling using Xdebug, which highlighted some bottlenecks in the code. Results attached - best viewed with ​KCacheGrind (Linux, KDE) or ​WinCacheGrind (Windows).

While being profiled, the script executed about 5 times longer, so the whole transaction took about 22 seconds. But still the proportions of timings are very informative.

I think here is some place for improvements:

1. rcube_imap->_list_headers sorts messages even though IMAP server has sorting capabilities.
2. rcmail_js_message_list generates the JavaScript code utilising some highly complicated and inefficient quoting routines that occupy CPU most of the time (see rep_specialchars_output).
3. RoundCube could use the capabilities advertised in the IMAP greeting response if the server does provide them. That would save one request to the server, although this doesn't have that much impact on speed, especially on local installations.

The tests were taken on a Celeron 366 MHz machine which is rather of the antique kind, nevertheless RC could be faster just to defer high loads, regardless of the server speed. My RC was configured to list 40 messages per page.

Hope this will at least inspire someone to think of optimisation.

[Emil Wojak]

Xdebug v2 output zipped

[Emil Wojak]

Screenshot of server-side traffic displayed in Wireshark

[thomasb]

1) RoundCube does use the IMAP sort capability on this line in rcube_imap.inc

if ($this->get_capability('sort') && ($msg_index = iil_C_Sort($this->conn, $mailbox, ...)

This returns a sorted list of message IDs where we can take a subset for the current page and request the complete headers for this subset. Unfortunately some IMAP server do not return the headers in the same order as requested. That's the reason why RoundCube sorts those headers again. But sorting 40 headers is not what slows down an application.

2) To produce valid HTML output we need to quote each header value. Also mail headers can be specified in any charset and RoundCube has to convert them to UTF-8 first. rep_specialchars_output() might look complicated but all it does is a strtr() using a (statically) cached translation table. Of course we're open for suggestions to improve this routine.

3) It does (at least for sorting) but if I missed something from the IMAP specification, please let me know.

[Emil Wojak]

Thanks for your reply.

1. I agree, that sorting 40 messages should not be an issue, nevertheless the time it takes to sort the messages highly depends on the overall number of messages in a folder, thus even sorting 10 messages from a folder with 3000 messages takes about 8 seconds. I also noticed, that the response to FETCH request is ordered by SNs, which is generally not the order in which headers were requested.
   So RC calls the sort_headers() routine, which is a wrapper for uasort() with the compare_seqnums() callback function, which in turn searches the big $sequence_numbers array linearly and does that twice for a call! Note, that compare_seqnums() is called multiple times by uasort() (185 in my test case). That, I think, does slow down the application.
   The comment from sort_headers() states, that $headers has UIDs instead od SNs as keys, but why is that so, if the response actually contains SNs? (See ​IMAP extensions s.3 and ​RFC3501 s.7.5.2)
   IMHO this should be as simple as storing the response in an associative array with SN as the key, and then fetching headers from that array by SNs in the same order as requested. No explicit sorting necessary here. Is there some architectural obstacle?
2. Well, it just looked scary at first glance :P But still it seems that quoting is hidden behind too many levels of abstraction or too generic. Please just look at the attached profiling output​, particularly at rcmail_js_message_list(), and notice, that in fact rep_specialchars_output() does much more than just calling strstr(), espacially when called by JQ().
3. I realize that, but what I meant was using the capabilities advertised in the server greeting, whereas requesting the capabilities by the client again is redundant. As you can see in the attached screenshot​, RC asked for capabilities despite they had been already sent in the greeting message. Please have a look at ​RFC3501 s.7.1.

[bewst]

Upping priority as performance issues bite me too, especially in very large mailboxes.

[bewst]

Upping priority again, but I have new data :). I have a mailbox with tens of thousands of emails -- it's my "everything in history" folder. When I visit it in roundcube, Firefox hangs completely and apache's CPU usage on my server shoots to 100%. The symptoms are similar to #1399715 except I doubt the cache setting would help me since:

1. the IMAP server is the same machine as the webserver
2. I don't visit this folder very often

This is really important to me, as searching that folder is something I very commonly want to do with webmail (letting the server side do the searching of course!)

[flapphead]

I have the same problem with a large inbox. it appears as though roundcube fetches the ENTIRE inbox, then sorts it, then spits out a slice of it.

I have courier and the latest version of roundcube. When I fetch a subset of my inbox, roundcube makes a request to imapd, imapd THRASHES my disk, then apache uses the 100% of the cpu to, presumably, sort and slice.

However, when i make a request from telnet like "fetch 2000:2200", imapd responds instantaneously.

[seansan]

Review performance in 0.1.5

[jolt]

I am getting the same poor performance with large mailboxes. HTTPD sits at 100% CPU utilization for upwards of a minute on a 250 Opteron. Granted there are about 100K emails in there, but I only want to see the most recent ones - not load the whole mailbox.

[roundcube@…]

This is just another confirmation... I am using RoundCube under ISPConfig, and after the initial login screen Apache eats 99.x% CPU. I'm running this on an older machine (P3 866MHz w/256MB), so it eventually times out with this error message:

Fatal error: Maximum execution time of 300 seconds exceeded in /home/admispconfig/ispconfig/web/roundcubemail/program/lib/imap.inc on line 211

That is on a dev box with two test users, zero emails, and no other processes using up CPU.

I consider this to be a fatal bug. I have been using Squirrelmail (another PHP IMAP client) for years, and it is peppy and responsive on very weak hardware even with huge folders holding thousands of emails. (The bottleneck is not my Courier-IMAP server.)

I'm an experienced developer with optimization experience, so I'd be happy to help you dig into this (and provide you with an SSH login to my dev box). I volunteer to spend a few hours helping to resolve this issue if a RoundCube core developer will work with me in a realtime, pair-programming fashion (on the phone, or IRC/Gaim/Skype/etc). (I'll dig deepr on my own, too, but it will be a while before I have time to jump into another codebase without help getting up-to-speed.)

I'm running v. 0.1.1 as packaged for ISPConfig here:
​http://ispconfig.bb-hosting.org/uk/downloads.htm

The screenshots look amazing and I'd love to offer RoundCube to my customers. But at this point it's a no go.... :( In a few days hopefully I'll have time to try 1.5 and dig a little deeper.

[roundcube@…]

Update: Bug Found

I noticed I was getting the 99.9% CPU crash when running the installer script, at Stage 3 when I tried to test the logins. (Your installer script totally rocks, btw.)

I looked at the file and line number in the error message, and eventually found a bug in the login loop in imap.inc, line 303. It winds up looping forever, resulting in the CPU usage:

When I add a loop max-iteration counter, then my problem goes away (although it fails to log in with my username and password):

#dereks

$loop_count=0;

do {

$loop_count += 1; # added by me

$line = iil_ReadReply($conn->fp);

if ($line === false) {

break;

}

# } while (!iil_StartsWith($line, "a001 "));

} while ((!iil_StartsWith($line, "a001 ")) && $loop_count < 1000);

So, it appears there is a bug logging in to Courier-IMAP? Again, I'm happy to grant access to my dev server to help resolve this issue.

Thanks,
Derek

P.S.> After looking at the code, I don't understand why you guys are parsing the IMAP login response manually from a socket fp. Why not use the PHP built-in IMAP functions for that?

​http://www.php.net/imap

It seems like you are re-inventing the wheel (and introducing infinite-loop bugs :) ).

P.P.S.> Several projects use the file extension "filename.inc.php" instead of just ".inc" because it offers better security. If somebody's Apache is misconfigured to allow HTTP file downloads for .inc files, they could see source code and database passwords and stuff. But if the file is named .inc.php, then the PHP include file will just be "run" by Apache, like it usually is when it's included from some other .php file, and it just produces no output. (Also, text editors know how to syntax-highlight the .inc.php file correctly based on file extension.) So I recommend using .inc.php for include files instead of .inc.

[roundcube@…]

Ok... my bug was found. But I think my bug was not the same as the other users here.

I just did a fresh install of ISPConfig on Ubuntu 7.10. That includes a fresh install of Courier-IMAP.

In ISPConfig, I forgot to do this, to make ISPConfig work with Courier-IMAP:

-- Enable Maildir under Management -> Server -> Settings -> EMail in the ISPConfig web interface.

I went and did that, and now RoundCube works! The interface is great... nice colors, nice layout, nice AJAX... great work guys! CPU usage still seems a little heavy (just by myself I'm causing spikes up to 7% or 8% CPU usage per page hit... that seems very high for single user on an idle box.)

I'm not sure what that ISPConfig variable does... I thought it just affected the procmail-based delivery to save in Maildir format for Courier-IMAP, but apparantly it affects Courier-IMAP login as well.

I still consider this an infinite loop "bug" in RoundCube, because it should display an error message, not loop forever and kill the server's CPU.

Thanks again for your great work on Roundcube... and to the author of Iloha before you... OSS rul3z!

[alec]

Point 3. fixed in [ca4c087f] also other things like sorting was improved some time ago. Because this request is very old I'm closing it. Please, open a new request with new conclusions if needed.