Opened 6 years ago

Closed 4 years ago

Last modified 2 years ago

#1484660 closed Feature Patches (fixed)

Attachment Excessive Memory Use Error

Reported by: JohnDoh Owned by: till
Priority: 5 Milestone: 0.4-beta
Component: PHP backend Version: git-master
Severity: major Keywords: pear mail mime encode memory optimize
Cc: zdali, r.i.k@…, mihai@…, suporte@…, SKaero@…, m.duelli@…, rudolfs.osins@…, osos@…, wavexx@…, roundcube@…, eric@…, lists@…, john@…, dennylin93@…

Description

Hi,

I know tickets have been created about this before but I cant find the exact one and many of them seem to be lost in some kind of "dupicate of" hell. I thought it was probably easier to just start a new one. I applogies if I am repeating others informaiton but I cant find the previous tickets.

The amount of memory required to send an email with attachments seems to massivly out way the size of the attachments giving an error like:

"Fatal error: Allowed memory size of blah bytes exhausted (tried to allocate blah bytes)"

in the error log.

(thrown by the quotedata() function in program/lib/Net/SMTP.php)

Examples:
required more than 64mb to send 7mb attachment or 25mb to send 5.5

More people are now reporting this on the forum (http://roundcubeforum.net/forum/index.php?topic=1811.0)

I know that the attachment size limits (which I think only apply to individual files, not the combined size) and the php memory limits can be altered but i dont think this counts as a solution when the difference in requirements is so great

This still occurs in SVN890

Thanks and sorry again if I am repeating stuff but I cant track down the previos tickets which I know exist about this exact issue.

Attachments (3)

patch.txt (105.6 KB) - added by duelli 4 years ago.
Updated the patch with omitted SVN versions
mem.patch (18.0 KB) - added by alec 4 years ago.
forwarding_trace.txt.gz (65.5 KB) - added by roundreport 4 years ago.

Download all attachments as: .zip

Change History (63)

comment:1 Changed 6 years ago by thomasb

  • Component changed from Client to PHP backend

Well the message body is passed as a string and is altered by Net/SMTP.php. This means that PHP copies the whole string in memory. The only solution is to re-write the PEAR classes to work with streams instead of string variables. Any volunteers out there?

comment:2 Changed 6 years ago by zdali

I'll foolishly put my hand up for this. I already have a working prototype for this.

comment:3 Changed 6 years ago by seansan

  • Cc zdali added
  • Milestone set to later

Marked as later until we have a solution

comment:4 Changed 6 years ago by till

  • Milestone changed from later to 0.1-stable

My settings:

memory_limit = 256M
max_execution_time = -1

IMO not a RoundCube issue, but more subject to fine-tuning your configuration. My setting is not too high either (at least, IMHO) since the current memory_limit setting PHP is shipped with is 128M(B).

comment:5 Changed 6 years ago by till

  • Owner set to till
  • Status changed from new to assigned

comment:6 Changed 6 years ago by oleole

This is a major issue for shared hosting, VPS hosting and other types of hosting servers.

The solution to raise the memory limit to 256MB is unrealistic and just plainly wrong. Hosting servers are very limited in memory and 256MB could be the entire memory of a whole VPS server.

Disabling the maximum execution time is just plain stupid. That option is there to avoid lockups, deadlocks and on a production server is a must-have. If you disable it, you risk of bringing the whole server down with a few astray infinite loops.

This is definitely an issue, since roundcube is getting ready to be included in major hosting control panels like cPanel. Shooting our selfs in the foot with this will limit roundcube's user base.

comment:7 Changed 6 years ago by kaouete

Hello,

I hope this problem will not be fixed like till proposed.

By looking at the forum post the bug is refering to, it is obvious there is a problem in the code ...

Thanks :)

comment:8 Changed 6 years ago by till

  • Milestone changed from 0.1-stable to 0.1.1

I installed a fresh PHP5 on my sandbox over the weekend, the current value is already 128 MB. Can anyone try with this value?

Not to be rude here, but if 256 MB is unrealistic for you guys, then what exactly is realistic here? 16 MB? 32 MB?

I am not closing this issue, however, I am postponing it to 0.1.1 for review.

comment:9 Changed 6 years ago by jbarbuc

I have a patch for this issue that uses file streams as suggested above.

With this patch I was able to send a 10MB sized file with a 5MB memory limit,
but I have no plans on keeping the limit so low.

See http://public.lanl.gov/jbarbuc/patches/roundcube/ for information
and http://public.lanl.gov/jbarbuc/patches/roundcube/roundcubemail-0.1.attach.patch
for the patch.

comment:10 Changed 6 years ago by till

Traced this problem with a bit of help from the Mail_Mime maintainer, I think we should "free" the $MAIL_MIME contents ASAP, before we hand them off to smtp_mail (which wraps Net_SMTP). This would free a lot of memory.

For reference:
http://trac.roundcube.net/browser/trunk/roundcubemail/program/steps/mail/func.inc#L1388

comment:11 Changed 6 years ago by till

  • Milestone changed from 0.1.1 to 0.1.2

But (of course) we can't delete it there, because after rcmail_deliver_message, it's also used to build the message for "sent".

Putting this on 0.1.2. Also had no chance to check this "streaming" patch yet.

comment:12 Changed 6 years ago by till

We can't apply the patch because you heavily patch external libs to use RoundCube function calls, etc..

Let me know if you would like to work on this, email me - till'at'roundcube'dot'net.

comment:13 Changed 6 years ago by thomasb

  • Milestone changed from 0.2-beta to 0.4-beta

comment:14 Changed 6 years ago by Atmos4

How about adding a third option to send mail ("Native"), that uses a pipe to sendmail to send the mail? Code is pretty simple:

<?php
$fd = popen("/usr/sbin/sendmail -t","w");
fputs($fd, "To: Someone Else <user@domain.tld>\r\n");
fputs($fd, "From: RoundCube User <rcuser@otherdomain.tld>\r\n");
fputs($fd, "Subject: Test message from my web site\n");
fputs($fd, "X-Mailer: RoundCubeMail $version\r\n");
fputs($fd, "\r\n");
/* now read from tempfile containing message body to pipe in chunks of eg. 4k */
pclose($fd); 
?>

Of course one could also do a native SMTP session in similar style, by opening a socket to the smtp server and doing basically the same. I might write sample code for that, SMTP protocol is rather simple. The socket functions however require more error handling than the pipe to sendmail, but it's portable across systems.

-- Felix

comment:15 Changed 5 years ago by floeff

  • Cc floeff@… added

Any news on this one? I have the same issue here with 0.2 and trunk, sending 5 MB takes 40 MB of RAM... Anyone got a working patch for 0.2-beta or for trunk?

comment:16 Changed 5 years ago by Erika

  • Cc r.i.k@… added
  • Keywords pear mail mime encode memory optimize added

You can patch the PEAR Mail_Mime package in order to optimize the memory used...

See my patch at : http://pear.php.net/bugs/bug.php?id=15913

comment:17 Changed 5 years ago by wizputer

There seems to be an issue in PHP5 where the default copy-on-write functionality is broken when explicitly setting arguments to be passed-by-reference. Instead it copies the variable when it is passed.

For example, using RCube v0.2.2 in rcube_smtp.inc line 56, the smtp_mail function, by not explicitly setting $body to be passed-by-reference, it cuts down on memory usage by 1 factor.

This was tested with PHP 5.2.6 and RCube 0.2.2. Sending an email with no body/subject and with a 14.7MB attachment. Before the change peak memory usage was 100.6MB. After changing the declaration of the smtp_mail function it is 80.7MB.

And of course for some reason memory_limit needs to be set 2x those peak values otherwise the out of memory exception gets thrown.

comment:18 Changed 5 years ago by mihailim

  • Cc mihai@… added

comment:19 Changed 5 years ago by netbit

  • Cc suporte@… added

comment:20 Changed 4 years ago by SKaero

  • Cc SKaero@… added

comment:21 Changed 4 years ago by duelli

  • Cc m.duelli@… added
  • Severity changed from normal to major

Increasing memory_limit is not an alternative for many users.

The patch mentioned above does not mitigate this problem.

This prevents RoundCube from reaching a broader audience.

comment:22 Changed 4 years ago by rudolfs

  • Cc rudolfs.osins@… added

comment:23 Changed 4 years ago by osos

  • Cc osos@… added

Just wondering...

How does Squirrelmail and other php webmails handle attachements?

comment:24 Changed 4 years ago by sftf

Confirm this problem on Debian lenny with roundcube (0.2.2-1~bpo50+1) [backports].
We need 20MB attachments for corporate our users.
Sending empty mail with 20MB attachment fail with

[25-12-2009 17:04:03] PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 55578943 bytes) in /usr/share/php/Net/SMTP.php on line 836
in /var/log/roundcube/errors

This forced me set memory_limit = 512M in /etc/php5/apache2/php.ini !!!
This is VERY bad news for me.

comment:25 Changed 4 years ago by duelli

What makes this issue even worse in my opinion is that it also affects forwarding of emails whose size cannot be limited by the upload limit by an administrator.

comment:26 Changed 4 years ago by till

Hey guys,

for starters, PHP runs top to bottom - so whenever something finished executing (the page rendered), the memory is free'd. It's not that bad to have a huge memory limit. And Debian is ultra conservative with the setting.

As for the bug - it's related to one of the libs we use. I believe Alec is working on a fix, so please have some patience. If you want to know how Squirrelmail does it, feel free to read the source and let us know. =)

Also, always feel free to supply a patch. We are open to contributions!

Till

comment:27 Changed 4 years ago by roundreport

  • Cc wavexx@… added

comment:28 Changed 4 years ago by alec

Svn versions of both Net_SMTP and Mail_Mime packages supports already big messages via files or file handles. So, implementing this in Roundcube should be quite simple now, but I'll have probably no time in this month for doing this.

comment:29 follow-up: Changed 4 years ago by duelli

The Patch I attached updates Mail_mime to version 1.6.0RC1 (Net_SMTP was already up to date as far as I can see).

Now, I can use $rcmail_config['smtp_server'] = '%h'; with

php_value       upload_max_filesize     10M
php_value       post_max_size           10M
php_value       memory_limit            150M

Possible I am still missing something (otherwise Alec would have done this quick patch by himself). Sending mail works as expected with this patch!

Using top shows me that the corresponding apache process will allocate up to 18% of my 512 MB RAM which is about 110 MB when sending a 10M file as an attachment.

I had to increase the memory_limit to 150 MB to not get caught by the out of memory exception.

For the brave and curious of you: What are your experiences with this patch?

comment:30 Changed 4 years ago by alec

Without Roundcube code changes this will work as before. And as I said you need Net_SMTP from PEAR's svn.

comment:31 Changed 4 years ago by till

http://svn.php.net/viewvc/pear/packages/Net_SMTP/trunk
http://svn.php.net/viewvc/pear/packages/Mail_Mime/trunk

Check out with:

svn co http://svn.php.net/repository/pear/packages/Mail_Mime/trunk
svn co http://svn.php.net/repository/pear/packages/Net_SMTP/trunk

Changed 4 years ago by duelli

Updated the patch with omitted SVN versions

comment:32 in reply to: ↑ 29 ; follow-up: Changed 4 years ago by duelli

Replying to duelli:
Same memory requirements also with newest patch.

comment:33 in reply to: ↑ 32 Changed 4 years ago by till

Replying to duelli:

Replying to duelli:
Same memory requirements also with newest patch.

duelli, please read alec's comment and please stop adding noise here.

It's getting a bit confusing. ;-)

comment:34 Changed 4 years ago by duelli

Sorry for confusing you. I am just trying to push the development of this bug a litte since I consider it very important!

The latest version of the attached patch contains most recent SVN changes to Net_SMTP and Mail_mime. Obviously, further changes to RoundCube are needed.

Looking forward for your changes :-)

comment:35 Changed 4 years ago by alec

I've added a patch. Requires Net_SMTP and Mail_mime from PEAR's svn (today's versions). It works only with smtp_server set. This probably is not complete, but I've made a couple of tests and looks working. I was able to send a message with 10MB attachment using only 11MB of memory (peak).

comment:36 Changed 4 years ago by duelli

I tested your patch mem.patch in combination with

The patch applied smoothly.

I limited memory_limit to 64M and increased the upload_limit to 20M.

When sending a 20M file, I can see a peak memory usage of around 22M. So this works great from the memory point of view.

Possibly, there is a trade-off with run time, currently. The corresponding apache process was at 100% CPU for about 3 minutes.

Nevertheless, this is great news! Thank you very much.

comment:37 Changed 4 years ago by Lazlo

  • Cc roundcube@… added
  • Type changed from Bugs to Patches

Changed 4 years ago by alec

comment:38 Changed 4 years ago by alec

Patch updated, requires current trunk.

comment:39 Changed 4 years ago by duelli

The new patch also applied smoothly with a few automatically solved hunks. No further changes are required anymore as new SMTP and Mail_mime are now in SVN trunk.

We have used this patch for about 3 days now without problems.

Has anybody else conducted performance measurements regarding RAM and/or CPU?

comment:40 Changed 4 years ago by lacri

  • Cc eric@… added

comment:41 Changed 4 years ago by iezzip

  • Cc lists@… added

comment:42 Changed 4 years ago by jfeeney

  • Cc john@… added

I had the same problem and now it's resolved. Thanks very much to Alec for the patch, and Duelli for pushing the issue! :)

With the patch and the supplementary files Duelli linked to above, I was able to send a 14 MB attachment with the following settings:

memory_limit = 64M
upload_max_filesize = 20M

In terms of CPU/RAM usage, I see the CPU briefly hits 98% during send (send takes less than 5 seconds) whereas RAM usage peaks at only 3.5% (on a 512MB VPS).

Again, thanks to all involved, this fix is a godsend.

John

comment:43 follow-up: Changed 4 years ago by roundreport

I'm willing to test the fix, but can someone tell if, with the last fix, the memory_limit still needs to be as high or higher than upload_max_filesize? (meaning that at least one copy is hold in memory?).

I'm still using an old svn/hacked version containing the fix from jbarbuc (the "streaming" patch), which scales very well when many users are uploading large files. Holding a single copy in ram would still kill the ability to send arbitrarily sized attachments or scale with users.

comment:44 in reply to: ↑ 43 ; follow-up: Changed 4 years ago by alec

Replying to roundreport:

I'm willing to test the fix, but can someone tell if, with the last fix, the memory_limit still needs to be as high or higher than upload_max_filesize? (meaning that at least one copy is hold in memory?).

No, you should be able to send mail with 100MB attachment and memory_limit=16MB.

comment:45 in reply to: ↑ 44 Changed 4 years ago by roundreport

Replying to alec:

Replying to roundreport:

I'm willing to test the fix, but can someone tell if, with the last fix, the memory_limit still needs to be as high or higher than upload_max_filesize? (meaning that at least one copy is hold in memory?).

No, you should be able to send mail with 100MB attachment and memory_limit=16MB.

Ok, I was successful in sending a 40mb attachment with a 32mb memory limit with 2 users concurrently. Perfect.

comment:46 Changed 4 years ago by roundreport

I tried on another host with php-cgi, but with less success.

memory limit: 32mb
attachment size: 4.4mb

[06-Feb-2010 20:03:37] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 5880349 bytes) in program/lib/Net/SMTP.php on line 890

I was able to send the attachment by raising the memory_limit to 64mb.

With 64mb of memory_limit, I still cannot send a 34mb attachment:

[06-Feb-2010 20:33:43] PHP Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 46357565 bytes) in program/lib/Mail/mimePart.php on line 463

smtp_server is obviously set. Source is from SVN r3256/svn.

comment:47 follow-up: Changed 4 years ago by alec

@roundreport: with patch applied? There are memory calculations in the code. It looks like in your case Roundcube is using old method.

// Check if we have enough memory to handle the message in it 
// It's faster than using files, so we'll do this if we only can 
if (is_array($_SESSION['compose']['attachments']) && $CONFIG['smtp_server'] 
  && ($mem_limit = parse_bytes(ini_get('memory_limit')))) 
{ 
  $memory = function_exists('memory_get_usage') ? memory_get_usage() : 16*1024*1024; // safe value: 16MB 

  foreach ($_SESSION['compose']['attachments'] as $id => $attachment) 
    $memory += $attachment['size']; 
 
  // Yeah, Net_SMTP needs up to 12x more memory, 1.33 is for base64 
  if ($memory * 1.33 * 12 > $mem_limit) 
    $MAIL_MIME->setParam('delay_file_io', true); 
}

comment:48 in reply to: ↑ 47 Changed 4 years ago by roundreport

Replying to alec:

@roundreport: with patch applied? There are memory calculations in the code. It looks like in your case Roundcube is using old method.

I didn't realize the patch wasn't already in SVN, sorry for the noise.

After applying the patch I can confirm it works as advertised. I was able to send a 100mb attachment with a 32mb memory_limit. I can see the CPU usage skyrockets while sending the message, but as far as memory is concerned, it's perfect.

comment:49 Changed 4 years ago by alec

  • Resolution set to fixed
  • Status changed from assigned to closed

Patch applied in [91790e41].

comment:50 Changed 4 years ago by roundreport

  • Resolution fixed deleted
  • Status changed from closed to reopened

Hi again. Just wanted to note that after a few days of testing, I had some users complaining about failing attachments. Indeed, it seems that Net_SMTP needs more than 12x the amount of memory. I was able to resolve the issue by raising the estimation to 15x (by trial and error) in the following code:

Yeah, Net_SMTP needs up to 12x more memory, 1.33 is for base64
if ($memory * 1.33 * 12 > $mem_limit)

$MAIL_MIME->setParam('delay_file_io', true);

Anyone experienced the same issue? Apparently, this happened for attachments approaching the size of the free available memory.

comment:51 follow-up: Changed 4 years ago by duelli

I have never seen such problems for my users.

I guess it will help if you present the relevant system parameters like RAM, upload limits, etc. and information on your RoundCube installation. Are there any other modifications to your current installation?

comment:52 in reply to: ↑ 51 ; follow-up: Changed 4 years ago by roundreport

I guess it will help if you present the relevant system parameters like RAM, upload limits, etc. and information on your RoundCube installation. Are there any other modifications to your current installation?

Of course: source is still SVN r3256/svn + patch applied. No custom modifications.
Running php-fcgi, with recommended.ini, with the exception of memory_limit set to 32MB.

In a specific case, the user was trying to forward an attachment of ~8mb to someone, but the POST would fail. I tried increasing the limit to 64MB with no luck:

[11-Feb-2010 20:11:00] PHP Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 19250989 bytes) in /srv/www/webmail/program/lib/Net/SMTP.php on line 890

The following are users trying to send attachments with my usual memory_limit of 32MB:

[10-Feb-2010 10:47:45] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 9101735 bytes) in /srv/www/webmail/program/lib/Mail/mime.php on line 1029
[10-Feb-2010 10:49:54] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 9101797 bytes) in /srv/www/webmail/program/lib/Mail/mime.php on line 1029
[10-Feb-2010 10:53:31] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 7393147 bytes) in /srv/www/webmail/program/lib/Net/SMTP.php on line 890
[10-Feb-2010 10:55:36] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 7453235 bytes) in /srv/www/webmail/program/lib/Net/SMTP.php on line 890

After tweaking $memory * 1.33 * 15 I was able to forward the first attachment with 32MB of memory_limit, but I couldn't verify if it would fix all other cases. I didn't see any error reports in the log as of now.

comment:53 in reply to: ↑ 52 Changed 4 years ago by till

Replying to roundreport:

I guess it will help if you present the relevant system parameters like RAM, upload limits, etc. and information on your RoundCube installation. Are there any other modifications to your current installation?

Of course: source is still SVN r3256/svn + patch applied. No custom modifications.
Running php-fcgi, with recommended.ini, with the exception of memory_limit set to 32MB.

In a specific case, the user was trying to forward an attachment of ~8mb to someone, but the POST would fail. I tried increasing the limit to 64MB with no luck:

[11-Feb-2010 20:11:00] PHP Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 19250989 bytes) in /srv/www/webmail/program/lib/Net/SMTP.php on line 890

The following are users trying to send attachments with my usual memory_limit of 32MB:

[10-Feb-2010 10:47:45] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 9101735 bytes) in /srv/www/webmail/program/lib/Mail/mime.php on line 1029
[10-Feb-2010 10:49:54] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 9101797 bytes) in /srv/www/webmail/program/lib/Mail/mime.php on line 1029
[10-Feb-2010 10:53:31] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 7393147 bytes) in /srv/www/webmail/program/lib/Net/SMTP.php on line 890
[10-Feb-2010 10:55:36] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 7453235 bytes) in /srv/www/webmail/program/lib/Net/SMTP.php on line 890

After tweaking $memory * 1.33 * 15 I was able to forward the first attachment with 32MB of memory_limit, but I couldn't verify if it would fix all other cases. I didn't see any error reports in the log as of now.

Is the interesting bit "forward"? I don't remember testing this. I assume it would use the same or similar routines, but maybe there's another callgraph.

comment:54 follow-up: Changed 4 years ago by alec

Works for me with default factor (12). See 8MB * 1.33 * 12 = 127MB, it's more than memory_limit, so file-based methods should be used. When forwarding 9.5MB attachment I've got:

mail/compose [6.2 MB/13 MB]: 1.5267 sec
mail/send [6.6 MB/7.5 MB]: 8.0048 sec

comment:55 in reply to: ↑ 54 Changed 4 years ago by till

Replying to alec:

Works for me with default factor (12). See 8MB * 1.33 * 12 = 127MB, it's more than memory_limit, so file-based methods should be used. When forwarding 9.5MB attachment I've got:

mail/compose [6.2 MB/13 MB]: 1.5267 sec
mail/send [6.6 MB/7.5 MB]: 8.0048 sec

I can confirm that it works also.

@roundreport: Maybe you can install xdebug so you get a full stack trace with the error. Also enable profiling (and use WinCachegrind?, KCacheGrind or WebCacheGrind?) and see what you can find - e.g. where the memory hog is located.

Changed 4 years ago by roundreport

comment:56 Changed 4 years ago by roundreport

As suggested, I've dug into the issue a little more with xdebug.
My config is unchanged, the factor is back to 12 and memory_limit is 32MB.

Two interesting finds:

  • Attaching a 4.2mb attachment works as expected.
  • Forwarding an email with a 4.2mb attachment fails.
  • Twiddling the multiplier up to 15 fixes forwarding.

I'm attaching a xdebug machine-readable trace of the forwarding failure.

comment:57 Changed 4 years ago by alec

  • Resolution set to fixed
  • Status changed from reopened to closed

I've found it. Fixed in [3b1426a1].

comment:58 Changed 4 years ago by dennylin93

  • Cc dennylin93@… added

comment:60 Changed 4 years ago by floeff

  • Cc floeff@… removed

comment:61 Changed 2 years ago by mytux

it seems the problem is back.
I'm running round cube for a long time, and never had that issue with versions 4, 5 and 6.
I updated to version 7.2 today, and I cannot send mails with big attachments anymore.
I tried to update to 8 beta, but it's the same.
I get those messages in the errors file
[13-Mar-2012 14:07:53] PHP Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 10146831 bytes) in /usr/share/php/PEAR.php on line 252

Note: See TracTickets for help on using tickets.