Memory runs out on new server

Memory runs out on new server

by Paul Nijbakker -
Number of replies: 33
Hi all,

We moved to Moodle 1.8.2 and in the process moved to a new server that is twice as powerful (twice as much RAM (4Gb), twice as many processors, faster disks) as the previous production server.

Nonetheless, we get from time to time an error that says "Fatal error: Out of memory (allocated 9437184) (tried to allocate 1048577 bytes) in ...xxx.php on line xxx", we did not get this error on the old server, or even on the very slow temporary server where the environment was parked while we installed the new production server. There must be something wrong with one or more memory settings, but which one(s)?

Present settings are in php.ini
max_execution_time = 300
max_input_time = 600
memory_limit = 128M

in Apache httpd.conf
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15

Moodle.docs mentioned also MaxClients, but we have no such setting in httpd.conf and I haven't been able to find it anywhere else either.

I would be very grateful if anyone could point me in the right direction as to what could cause this memory problem on a brand new server.

adTHANKsvance

Rgrds,
Paul.
Average of ratings: -
In reply to Paul Nijbakker

Re: Memory runs out on new server

by Pieterjan Heyse -
As it seems to me, your php.ini parameter memory_limit is not being read (or being overridden). Can you tell us the memory_limit parameter when you go to the admin -> server -> phpinfo ?
In reply to Pieterjan Heyse

Re: Memory runs out on new server

by Paul Nijbakker -
Yes, it is 128M

And the error is serious since it totally stops Moodle from working, sometimes, like right at this moment, I am staring at a white screen. It does not even show the error message. At the same time activity in the server is practically at zero (httpd.exe is at 850+ Mb and MySqld-nt.exe at 300+ Mb, but that leaves some 3 Gb of RAM and neither is registering any processor time (System idle process is at 99).
Stopping and starting the server will make the problem go away temporarily, but that is not a solution. Help sad

Rgrds,
Paul.
In reply to Pieterjan Heyse

Re: Memory runs out on new server

by Paul Nijbakker -
The Apache error log shows a whole bunch of these:

[Wed Sep 05 13:06:06 2007] [error] [client xxx.xx.xxx.x] PHP Fatal error: Out of memory (allocated 262144) (tried to allocate 393216 bytes) in ...\\xxx.php on line xxx, referer: xxx

and then a bunch of these:

[Wed Sep 05 13:11:15 2007] [error] [client xxx.xx.xxx.xx] PHP Fatal error: out of dynamic memory in yy_create_buffer() in Unknown on line 0

And then this

[Wed Sep 05 13:11:16 2007] [notice] Parent: child process exited with status 2147483655 -- Restarting.

So it has something to do with child processes, whatever that means. thoughtful
In reply to Paul Nijbakker

Re: Memory runs out on new server

by Pieterjan Heyse -
Hmm it seems like all the settings are ok, but apache is crashing due to php not being able to get enough memory. Apache spawns child processes to serve cliënts, but they seem to be having troubles on allocating memory. Maybe you should try to increase the number of apache threads?

Can you post us the apache config Section 1 : global Environment about the number of clients and apache forking behaviour?

these parameters:
KeepAlive
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
StartServers 15
MinSpareServers 5
MaxSpareServers 20
MaxClients 150
MaxRequestsPerChild 5000
</IfModule>

Dit you install any kind of code cacher (APC, eaccelerator, ..) or weird apache module?

Just trying to help, but I think I'm running on the edge of my knowledge here...
In reply to Pieterjan Heyse

Re: Memory runs out on new server

by Paul Nijbakker -
Hi Pieterjan,

Thanks for responding. Can you tell me where (in what folder, in which file) I should look to find this Section 1 : global Environment about the number of clients and apache forking behaviour?

I have been unable to find a MaxClients setting in any of the Apache files I have checked.

To my knowledge we have no PHP accelerator of any kind installed.

Here is another piece of our error log:

[Wed Sep 05 10:24:25 2007] [error] [client xxx.xx.xx.xxx] PHP Fatal error: Out of memory (allocated 19922944) (tried to allocate 393216 bytes) in ...\\xxx.php on line xxx, referer: xxx
Error in my_thread_global_end(): 253 threads didn't exit
zend_mm_heap corrupted
[Wed Sep 05 10:24:30 2007] [notice] Parent: child process exited with status 1 -- Restarting.
[Wed Sep 05 10:24:30 2007] [notice] Apache/2.2.4 (Win32) PHP/5.2.3 configured -- resuming normal operations
[Wed Sep 05 10:24:30 2007] [notice] Server built: Jan 9 2007 23:17:20
[Wed Sep 05 10:24:30 2007] [notice] Parent: Created child process 2832
PHP Warning: Module 'curl' already loaded in Unknown on line 0
PHP Warning: Module 'mysql' already loaded in Unknown on line 0
PHP Warning: Module 'gd' already loaded in Unknown on line 0
PHP Warning: Module 'mbstring' already loaded in Unknown on line 0
[Wed Sep 05 10:24:31 2007] [notice] Child 2832: Child process is running
[Wed Sep 05 10:24:31 2007] [notice] Child 2832: Acquired the start mutex.
[Wed Sep 05 10:24:31 2007] [notice] Child 2832: Starting 250 worker threads.
[Wed Sep 05 10:24:31 2007] [notice] Child 2832: Starting thread to listen on port 80.
[Wed Sep 05 10:24:31 2007] [warn] (OS 64)The specified network name is no longer available.
etc.

Rgrds,
Paul.
In reply to Paul Nijbakker

Re: Memory runs out on new server

by Patrick Craig -

Paul,

Look for 'Section 1: Global Environment' in the Apache httpd.conf file. Mine is located at C:\Apache2\conf\httpd.conf.

I have been experiencing precisely the same issue (white screen and all) with Moodle 1.8.2+ (2007021520). I had not had this issue until I moved up to this version from 1.7.

I am using Apache 2.0.59/PHP 5.2.1/MySQL 5.0.37 on a Dell PE1750 with 4 GB RAM and no PHP accelerator.

My PHP Resource Limits are:

------------------------------------

max_execution_time = 30

max_input_time = 60

memory_limit = 512M (I just increased this from 128M to see what would happen)

--------------------------------------------

And my Section 1: Global Environment (Commented lines removed)

------------------------------------------------------------------------------------

### Section 1: Global Environment

ServerRoot "C:/Apache2"

PHPIniDir "C:/php/"

PidFile logs/httpd.pid

Timeout 300

KeepAlive On

MaxKeepAliveRequests 100

KeepAliveTimeout 15

<IfModule mpm_winnt.c>

ThreadsPerChild 250

MaxRequestsPerChild 0

</IfModule>

Listen 80

LoadModule php5_module "C:/php/php5apache2.dll"

LoadModule access_module modules/mod_access.so

LoadModule actions_module modules/mod_actions.so

LoadModule alias_module modules/mod_alias.so

LoadModule asis_module modules/mod_asis.so

LoadModule auth_module modules/mod_auth.so

LoadModule autoindex_module modules/mod_autoindex.so

LoadModule cgi_module modules/mod_cgi.so

LoadModule dir_module modules/mod_dir.so

LoadModule env_module modules/mod_env.so

LoadModule imap_module modules/mod_imap.so

LoadModule include_module modules/mod_include.so

LoadModule isapi_module modules/mod_isapi.so

LoadModule log_config_module modules/mod_log_config.so

LoadModule mime_module modules/mod_mime.so

LoadModule negotiation_module modules/mod_negotiation.so

LoadModule setenvif_module modules/mod_setenvif.so

LoadModule userdir_module modules/mod_userdir.so

------------------------------------------------------------------------------------

Patrick

In reply to Patrick Craig

Re: Memory runs out on new server

by Paul Nijbakker -
Hi Patrick,

Our httpd.conf file does not mention any Section 1: Global Environment at all, but we have roughly the same settings in the file:

ThreadsPerChild 250
MaxRequestsPerChild 0

ServerRoot "E:/Server/Apache2.2"

Listen :80

LoadModule actions_module modules/mod_actions.so
LoadModule alias_module modules/mod_alias.so
LoadModule asis_module modules/mod_asis.so
LoadModule auth_basic_module modules/mod_auth_basic.so

LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authn_file_module modules/mod_authn_file.so

LoadModule authz_default_module modules/mod_authz_default.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule autoindex_module modules/mod_autoindex.so

LoadModule cgi_module modules/mod_cgi.so

LoadModule dir_module modules/mod_dir.so
LoadModule env_module modules/mod_env.so

LoadModule imagemap_module modules/mod_imagemap.so
LoadModule include_module modules/mod_include.so

LoadModule isapi_module modules/mod_isapi.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule mime_module modules/mod_mime.so

LoadModule negotiation_module modules/mod_negotiation.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule setenvif_module modules/mod_setenvif.so

LoadModule userdir_module modules/mod_userdir.so

Timeout 300

KeepAlive On

MaxKeepAliveRequests 100

KeepAliveTimeout 15

DocumentRoot "xxx/htdocs"

Rgrds,
Paul.


In reply to Paul Nijbakker

Re: Memory runs out on new server

by Janne Mikkonen -
Picture of Core developers
It means that memory_limit 128M isn't enough.
In reply to Janne Mikkonen

Re: Memory runs out on new server

by Paul Nijbakker -
Hi Janne,

We have increased it to 512M, but isn't that a stop gap solution? On the old server the memory_limit was only 32M and we did not have problems with that installation (Moodle 1.6).

Rgrds,
Paul.
In reply to Paul Nijbakker

Re: Memory runs out on new server

by Paul Nijbakker -
During a site backup on the new server this morning the server crashed with the eror message:

”Error: Database connection failed.

It is possible that the database is overloaded or otherwise not running properly.

The site administrator should also check that the database details have been correctly specified in config.php”

config.php is normal. The extra load from the backup process caused the crash. Isn't it paradoxical that we get memory problems when we move to a new sever with twice as much memory. And we had originally set the memory limit at 128M, i.e. over three times the minimum limit suggested for Moodle 1.8 (40M).

I hope someone can come up with a solution. My own idea is that there is something not right with our httpd.conf, since it does not mention any MaxClients setting. I wonder how MAxClients can be set too high when we don't even have it!surprise

Rgrds,
Paul.


In reply to Paul Nijbakker

Re: Memory runs out on new server

by Samuli Karevaara -
The default value for MaxClients in Apache is 256 (http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients), which with Moodle is too much for 4 Gb.

You are currently probably allowing more Apache child processes than MySQL connections, which results in the db connection error, as with Moodle every Apache child process will connect to the database. This is the default case, as MySQL allows only 100 connections by default (http://dev.mysql.com/doc/refman/5.0/en/too-many-connections.html).

Raise your MySQL max_connections to 170 and lower the Apache MaxClients to 150 and see what happens. If the httpd processes are hogging memory, then try even lower numbers.
In reply to Samuli Karevaara

Re: Memory runs out on new server

by Paul Nijbakker -
Hi Samuli,

MySql max_connections was at 1210 and we did not have any MaxClients setting in httpd.conf. We will try your suggestion and see what happens.

Thanks for your reply.
Rgrds,
Paul.
In reply to Samuli Karevaara

Re: Memory runs out on new server

by Paul Nijbakker -
Hi again,

Well, the result was immediate. The server would not restart and the crash is complete sad. We are going back to previous imperfect settings, hoping for more suggestions.

Rgrds,
Paul.
In reply to Paul Nijbakker

Re: Memory runs out on new server

by Samuli Karevaara -
sad However, it's very unlikely that lowering that MaxClients is responsible for that. Apache won't start if there is an error in the configuration file. I'd say that is the case with the server not restarting.
In reply to Samuli Karevaara

Re: Memory runs out on new server

by Paul Nijbakker -
Hi Samuli,

Thanks for your reaction. We had no MaxClients setting in httpd.conf. All I did was add the line MaxClients 150 to httpd.conf (as suggested by Iñaki), but Apache did not accept that. (I had also changed max_connections in my.ini to 170, but that did not seem to cause any problems)

Do I have to write MaxClients in any specific location in httpd.conf, or surrounded by particular codes?

Rgrds,
Paul.
In reply to Paul Nijbakker

Re: Memory runs out on new server

by Samuli Karevaara -
We are running Apache as prefork MPM, which is a non-threaded multi-processing module, as some parts of PHP are said to be non-thread-safe. Therefore we have the MaxClients as this:
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>
In reply to Samuli Karevaara

Re: Memory runs out on new server

by Paul Nijbakker -
Hi Samuli,

I will try this, tomorrow morning, when traffic is low. I'll let you know how it went.

Rgrds,
Paul.
In reply to Samuli Karevaara

Re: Memory runs out on new server

by Paul Nijbakker -
Moi Samuli,

Thanks for the code. This time the server restarted smoothly (after having been stuck for most of the night). I am just wondering: we still have these settings elsewhere in our httpd.conf
ThreadsPerChild 250
MaxRequestsPerChild 0

Should I put those between codes too? e.g.
<IfModule mpm_winnt.c>
ThreadsPerChild 250
MaxRequestsPerChild 0
</IfModule>

Does it hurt to have MaxRequestsPerChild twice in the httpd.conf file? Could you perhaps send me a copy of your conf file, so I can compare it to ours?

Much appreciated,
Rgrds,
Paul.
In reply to Samuli Karevaara

Re: Memory runs out on new server

by Paul Nijbakker -
Hi Samuli,

With the new settings the server runs out of memory only once every 24 hours. We intend to set them even lower.

Rgrds,
Paul.
In reply to Paul Nijbakker

Re: Memory runs out on new server

by Paul Nijbakker -
However, we do get more complaints from users being unable to log in to Moodle. Does setting MaxClients lower result in the fact that the server can handle fewer users? We have hundreds of users online any given school day!

Httpd.exe was consuming up to a Gb of RAM before I last reduced the settings (And it made me wonder, why does httpd.exe need so much memory at a point when almost nobody is using Moodle? Why doesn't the memory use go down when there is little traffic?).

Rgrds,
Paul.
In reply to Paul Nijbakker

Re: Memory runs out on new server

by Samuli Karevaara -
Basically yes. But if you set the KeepAliveTimeout to 2 (seconds), then it means something like "this many clicks in a two-second time-frame". So you can have MaxClients at 150 and hundreds of users online, if they are not frantically clicking, but pausing to read from time to time smile But if the KeepAliveTimeout is at 15 (seconds), then it roughly means that you can have 7.5 times less users online than with the 2 second setting.

It seems that you might be asking too much from a Windows box running Apache. I've had bad experiences on the Apache performance on Windows. IIS might run smoother, if Linux is (still) not a viable option for you.

One rather radical thing that you might try is to set Apache to restart at 3:30 am or something... (Just Apache, not the whole server! It will clear the memory overflows and result in a break of a few seconds only. Just make sure it starts!)
In reply to Samuli Karevaara

Re: Memory runs out on new server

by Paul Nijbakker -
Hi Samuli,

I have thought about this restart issue, but I would not know how to arrange that. As far as I am concerned we could have adopted Linux, but we (still) do not have anyone with that expertise on our staff.

What baffles me most, though, is that, even though we had other problems, when we were running Moodle 1.6 in a much slower server with a much lower memory setting, but high timeout etc. We never had any of these memory errors. Can memory usage have exploded so much between Moodle 1.6 and 1.8 or is it so that we simply cannot get our server settings right?

Rgrds,
Paul.
In reply to Paul Nijbakker

Re: Memory runs out on new server

by Samuli Karevaara -
BTW, do you have the KeepAliveTimeout at 2 seconds? It should clear the queue a bit faster. Though it seems like your Apache is leaking memory...

Apache can be restarted from the command line, then just set it as a scheduled task at the server.
In reply to Samuli Karevaara

Re: Memory runs out on new server

by Paul Nijbakker -
Hi Samuli,

Yes, we did set KeepAliveTimeout to 2 seconds and the memory build-up for httpd.exe has slowed considerably, but it is not gone.
We are considering having the server restart at night as you suggested earlier, if out IT staff can arrange it.

Rgrds,
Paul.
In reply to Paul Nijbakker

Re: Memory runs out on new server

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers
If Apache doesn't start, it logs the reason in the error logs. Have a look at them.

Saludos. Iñaki.
In reply to Paul Nijbakker

Re: Memory runs out on new server

by Martín Langhoff -
Search around this forums for maxclients, and my explanations on how to calculate the right value for it. Your main problem is that MaxClients is set way too high.
In reply to Martín Langhoff

Re: Memory runs out on new server

by Paul Nijbakker -
Hi Martín,

Could you please, tell me in which folder and which file I can find this MaxClients setting, so that I can change it. It is not in httpd.conf, http-config.h, or httpd-default.conf.

Max connections in MySql is set at 1,210.

Rgrds,
Paul.
In reply to Paul Nijbakker

Re: Memory runs out on new server

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Peer reviewers Picture of Plugin developers

If you don't find it, then Apache is using the default value (and I don't know what it is). Just add a line like

MaxClients NN

(where 'NN' is the number of clients) in httpd.conf and reload Apache.

Saludos. Iñaki.

In reply to Paul Nijbakker

Re: Memory runs out on new server

by Jon Pinckney -

We too are also having this problem.

We have upgraded from 1.5 to 1.8 big leap, and now having the same problems you are having.

Previously everything was running fine on performance but now our classes are struggling to run.

At any one time we may have upto 155 users online,

Login time is incredibly slow.

Enrolment into a course also just as slow.

Once onto a course everything seems fine.

We are receiving the same memory errors  "Fatal error: Out of memory (allocated 9437184) (tried to allocate 1048577 bytes) in ...xxx.php on line xxx" or users are presented with a blank screen.

Ive gone thru all the suggestions on here so far, with no joy. Even after re-starting the apache server, and allowing 30 students login the thing crawled.

What is on the horizon ? Should i go back a version? Maybe to 1.6?

Any thoughts?

Rgds

Jon

In reply to Jon Pinckney

Re: Memory runs out on new server

by Jon Pinckney -

I suppose I better add my server specs etc..

Dual Xeon 2.8 Ghz with 4gb ram on a Windows Server 2003 box.

Running Apache 2.2.3, Php 5.2.0 and mysql 5.0.27

In reply to Jon Pinckney

Re: Memory runs out on new server

by Mark Hilliard -
Jon,

I happened to see this thread while I was searching the performance forum and thought I would share our experience with you...

I would be willing to bet that the problems you see being described here are classic examples of a very well known "memory leak" problem that occurs when using Windows 200x as a server with PHP and Apache Web server.

This selection of software has a very poor record on the whole for supporting larger, more active sites.

Do a Google search on Window 2000 or 2003, PHP, Apache 2.x and the phrase "memory leak" and you will find hundreds of people complaining all the way back (3-4 years or more) about this exact type of problem. (e.g. Apache server under windows with PHP will have its child processes (the workers Apache web server starts begin to eat memory and never release it until the server becomes completely unresponsive)

Typically when you see the HTTPD.exe or Apache.exe child web process consume about 400 to 1000mb of RAM the Moodle server will freeze - you can watch this ugly memory process grow hourly in the Windows Task Mgr if you want....

My team tried about 15 combinations of Win 2003 server, PHP 5.x and Apache 2.x and they ALL did the same thing - ie. leak memory hourly under a Moodle load. We talked with sites all over the US and Europe and most people were doing ugly "hacks" like limiting the MaxRequestPerchild and doing a nightly or daily re-start of apache to keep things going. We looked hard and to my knowledge no one has ever solved this issue. (see the ApacheLounge website which focuses on Windows and Apache support )

Some people gave up on Apache and used IIS with some success. Others like us went back to some type of linux OS which works well with Apache/PHP and releases memory properly and has no "memory leak"

Good luck!

Mark H.
Minnesota
In reply to Jon Pinckney

Re: Memory runs out on new server

by Myles Carrick -
hi Jon,

Are you running the very latest MOODLE_18_STABLE from CVS? There have been a number of improvements here in the past few weeks... a few merges back everything sped up considerably for us (although we weren't experiencing such severe issues).

The "Out of memory..." message is not related to aggregate memory usage, but rather to a specific script's requirements.
What memory settings are in your php.ini?

Are you running a PHP accelerator?

Be sure to check the installation page (http://docs.moodle.org/en/Installing_Moodle#Check_PHP_settings) in the docs - this has been updated with some Moodle 1.8 stuff
(e.g. recommended 128M+memory settings, etc).

Cheers,
MC
In reply to Paul Nijbakker

Re: Memory runs out on new server

by BamCandy ! -
Hi,

I had this problem as well, and only after upgrading Moodle to version 1.8.2 and after securing web access to XAMPP, phpMyadmin etc. I changed all the memory limits, max users/connections etc, but it didn't fix it. The only solution I found was to stop using XAMPP, I changed to WAMP and this fixed the problem.

Hope this helps..

BC