Apache Memory Leak

Apache Memory Leak

by Brant Peterson -
Number of replies: 27

I have the apache memory leak issue.  Here is my setup:

WIndows 2003
Apache 2.2.3
PHP 5.1.6
MySQL 5
1.1 GHz
2GB RAM

I have tried several combinations of versions of Apache and PHP.  I have two questions:

  1. If the HTTPD.exe service starts at 14,000K should it release back to 14,000K when working properly?  Mine releases back to around 30,000K-40,000K.
  2. Does this issue occur with Linux as the operating system?
Average of ratings: -
In reply to Brant Peterson

Re: Apache Memory Leak

by Mark Hilliard -
Brant,
We have run Debian Linux, Ubuntu Linux and other x's and have NEVER in last 4 years seen the memory leak that shows up with some window 2003/Apache/PHP combo's. We tried W23k,Apache and PHP for 3 months last summer (mostly newest versions of PHP/Apache, but some older too). We had some many memory and performance issues that we eventually gave it up.

We have are seeing better luck with Window23K, IIS6 (FastCGI) and new PHP, but we will not use it for production yet...

On your other questions... It's hard to compare servers and typical RAM use because it all depends on the number of end users and what they are doing... I can't really confirm you have the mem leak bug from what you write above... just confirm a few facts

First, it was not at all uncommon for the Apache process our our windows 2003 server (Httpd.exe) to take 15 to 90MB of RAM under light or typical use. When it was "stable" the apache process would fluctuate from 50 to 250MB RAM and take and release RAM to any level between these numbers (based on OUR users), but with the bad combinations of Apache/PHP and the memory leak we would see the RAM start at 40-80MB and then just keep climbing. Some RAM would occasionally be released, but often NOT completely, and eventually RAM use increased to sometimes 500 - 800MB RAM! As you would expect the server became progressively less responsive and soon would no longer server Moodle pages.

If this is what is happening to you (especially under light/medium load) then you probably have the real mem leak. There are combinations of Apache/PHP that work (I know PHP 5.2.1 is well regarded), but you have to get the right version of Apache to work with it. I know I read about 2-3 combo's of Win23K, Apache and PHP that major sites claim are stable. I didn't save the links I found, but if you search the Windows forum well (PHP, Apache, window 2003) I am sure you will find them. Look for bigger sites (colleges, Universities, etc) with a few years under their belts. Ken Wilson has a stable windows server using IIS, PHP and MySQL.

Mark
Average of ratings: -
In reply to Mark Hilliard

Re: Apache Memory Leak

by Brant Peterson -

After further testing, it looks like my httpd.exe process grows by about 1000-4000k each time I reload a page.  As long as there aren't too many users on at once the server can handle this with a reset of apache every 10 minutes. 

I obviously don't have this right yet.

Average of ratings: -
In reply to Brant Peterson

Re: Apache Memory Leak

by Greg Hayden -
apache has memory leaks on windows. I'm going to recommend a switch to linux.
Average of ratings: -
In reply to Brant Peterson

Re: Apache Memory Leak

by Mark Hilliard -
Brant!
That's not good... Normally even on a badly configured server you might only have to reboot a couple times a day, re-starting every 10 minutes is really not acceptable for production server.

If you can't find stable pairings of Apache and PHP in the windows forum, then you might want to try using IIS6 with FastCGI if you need to stay with windows.

If you haven't found it in the Moodle forums, Ken Wilson (UK) had an Apache, Windows 2003, PHP setup that works very well for him at a larger school, Here's the link : http://moodle.org/mod/forum/discuss.php?d=46135&parent=374604 )

Give this a try before you give up. I respect Ken's work and would trust his setup based on his experience and success. If you can't get his setup to work for you then I would try Linux.

My suggestion, if possible would be to try LAMP (Linux, Apache, MySQL and PHP). We have had very, very good luck with Ubuntu and Debian Linux and have never run into the memory issues seen with windows, not to mention better security, stability and of course access the latest fixes and development sooner.

Mark H.
MPls, MN
Average of ratings: -
In reply to Mark Hilliard

Re: Apache Memory Leak

by Ryan Smith -
The latest versions of all of the WAMP software are working fine here.
Apache 2.2.8
MySQL 5.0.51a
PHP 5.2.5
eAccelerator 0.9.5.2

Our production and dev server have each been up for 34 days with no Apache restarts. Our production site serves a little over 1000 logins a day.

You said:

"...not to mention better security, stability and of course access the latest fixes and development sooner."

This is very misleading. Windows Server is a very stable and secure platform. On my development server I have access to the same latest fixes and developments that you do via the CVS.
Average of ratings: -
In reply to Ryan Smith

Re: Apache Memory Leak

by Mark Hilliard -
Ryan,
Thanks for sharing your successful WAMP! Hopefully this benefits others as well!

I really don't have any type of operating system preference (we use Debian, Windows 2003, Ubuntu, MySQL and MSSQL) for many different major development projects at our company, but I just spent 3 months this summer running over 12 combinations of Apache 2.x and PHP 5.x and we hit the memory leak problem on everyone. I am glad YOURS config works, unfortunately many more DON'T. I even got Martin Langhoff to confirm this problem. It has been a known, recurring problem for 2-3 years that is often hidden by reboot strategies.
I respectfully do say that with 90 of Moodle on Linux, you will see much more activity, development and work done on LInux first.

On stability, our last linux server ran for 3.5 years w/o a restart.

Mark
Average of ratings: -
In reply to Mark Hilliard

Re: Apache Memory Leak

by Chris Potter -

I've seen quite a few posts about this, and I wondered if there was a place we could try to get a list of versions that don't work versus versions that do work with Windows. I know this would benefit the Windows community. Sometimes, as I'm sure others can relate, you don't have a "choice" of linux or Windows, you're just stuck with what you've got. The best thing we can do is give the Windows users some hope... smile

On that note... I'll start:

I have the memory leak on:

Windows Server 2003 x64
PHP 5.2.2
Apache 2.2.4
MySQL 5.0.41

Average of ratings: -
In reply to Chris Potter

Re: Apache Memory Leak

by Paul Nijbakker -
We have the leak on:

Windows Server 2003
PHP 5.0.45
Apache 2.2.4
MySQL 5.2.3

With keep_alive_timeout set to 2 seconds and a server restart every morning we can manage with our present user load.

Rgrds,
Paul.
Average of ratings: -
In reply to Paul Nijbakker

Re: Apache Memory Leak

by Chris Potter -

What kind of user load do you have per day? We're at about 300-400 per day with about 3000 activity views.

Average of ratings: -
In reply to Chris Potter

Re: Apache Memory Leak

by Paul Nijbakker -
Hi Chris,

I do not have exact figures, but on account of running a dozen or so degree programmes entirely online (apart from our regular Moodle supported education) we have many hundreds of users online every day and many thousands of views (our Apache access log is so big that I cannot open it anymore), but we try to avoid having more than 50 simultaneous queries for the same page (for example, when giving instruction to large student groups we organise staggered access to, for example, a quiz page).

Rgrds,
Paul.
Average of ratings: -
In reply to Chris Potter

Re: Apache Memory Leak

by Neil S -
Hi Chris, how are things with our southern neighbors smile

I'd be willing to assist getting you going on Linux if you wish. You can contact me directly if that may be a future direction you might take. I'm sure we can work something out.

It seems like it should be possible to determine exactly where the memory leak is coming from and perhaps work around it by configuration. Perhaps not, but it would be interesting to figure out.

I'm wondering if anyone has tried lighttpd in place of apache? FastCGI isn't an option yet with this setup on windows (afaik), and thus I don't think accelerators would work; I'm just wondering if it will lead to more stability even if it is slower...
http://trac.lighttpd.net/trac/wiki/TutorialLighttpdAndPHP

Perhaps I'll need to set up a windows box. I don't have any right now, everything is linux...

Take care,
NS


Average of ratings: -
In reply to Neil S

Re: Apache Memory Leak

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I have some experience of running lighttpd with eAccelerator and PHP-CGI. So far, this seems rock solid and very fast. It's quite a new installation so I won't know for certain for a while but it looks good.

Don't use Windows. No no no tongueout
Average of ratings: -
In reply to Chris Potter

Re: Apache Memory Leak

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I think the Apache memory leak situation is well known on Windows. My understanding is that you have to get just the right combination of versions (of everything) to solve it.

Sounds like too much trouble to me, when you could just use Linux.
Average of ratings: -
In reply to Howard Miller

Re: Apache Memory Leak

by Matt Gibson -
I'm just trying out this solution and will report back. Certainly seems like a much better option than automated restarts.
Average of ratings: -
In reply to Matt Gibson

Re: Apache Memory Leak

by Brant Peterson -

I have already tried that solution.  I have the following settings:

ThreadsPerChild 200
MaxRequestsPerChild  0
MaxMemFree 16
#MaxClients 200 maxclients causes an error on start
Timeout 200
KeepAlive On
MaxKeepAliveRequests 200
KeepAliveTimeout 2
EnableSendfile Off
EnableMMAP Off
Win32DisableAcceptEx

I still have the memory leak problems.  I also seem to get an error when I use MaxClients.

We are purchasing a better server and I am going to try and go back to a different configuration that I have gotten from a school that has not had problems.  When I get that up and running and tested, I will post the versions and settings if they work.

Average of ratings: -
In reply to Matt Gibson

Re: Apache Memory Leak

by Matt Gibson -
You beat me to it, Brant. Same as before - memory wasted, sometimes within a few hours it locks up if usage is heavy.
Average of ratings: -
In reply to Chris Potter

Re: Apache Memory Leak

by Ben Steeples -
Looks like we potentially have the memory leak too:

Windows Server 2003
PHP 5.1.6
Apache 2.2.2
MSSQL 2000 (FreeTDS 5.1.7.7)

Memory increases ~500k per visit. I suspect it gets too high and bombs out, but I won't be able to check whether our config changes have improved anything, until the morning.
Average of ratings: -
In reply to Ben Steeples

Re: Apache Memory Leak

by Ken Wilson -
Hi Ben

Just to add to the discussion... At work we've been using the Apache build from Apache Lounge (http://www.apachelounge.com/download/) since we started using Moodle. Never had any memory leaks or any problems, and I know other colleges in our region have not had any problems also. Recently, this post suggests that others may have found this also to be the case.

Hope this helps!

Ken
Average of ratings: -
In reply to Ken Wilson

Re: Apache Memory Leak

by Matt Gibson -
I've just dropped the apache lounge version into my XAMPP folder, replacing the original one. Instructions are attached to MDLSITE-385. So far, resting memory usage after a restart is down to about 20MB from 60-70 before. I'll see how it performs after a few days.
Average of ratings: -
In reply to Matt Gibson

Re: Apache Memory Leak

by Matt Gibson -
Success, I think. No more leaking so far. I'll be away on holiday for the next week so I can't check again until next Sunday, but I recommend giving this a try.
Average of ratings: -
In reply to Matt Gibson

Re: Apache Memory Leak

by Matt Gibson -
Trying to check this more, I'm using process explorer to monitor memory usage, but I'm not sure what is what. Leakage has certainly slowed, but I'm not sure if its stopped. I currently have:
  • Private bytes
  • Virtual size
  • Working set
Which one should I be keeping my eye on? I know that usage is likely to grow a bit as my large eaccelerator SHM cache fills up, but I'm not sure where that would be.
Average of ratings: -
In reply to Matt Gibson

Re: Apache Memory Leak

by Eloy Lafuente (stronk7) -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Peer reviewers Picture of Plugin developers Picture of Testers
Great research Matt! cool

anyway, 2-3 days ago I've updated the Win32 packages to use XAMPP Lite 1.6.6a - with Apache 2.2.8 + PHP 5.2.5 + MySQL 5.0.51a .

My initial (and limited!) testing says that memory stays stable arround 21-23 MB.

It would be helpful to receive some feedback for people using it intensively to see if that's a "stable" combination.

If we confirm that problems continue... we should start thinking about how to rebuild those packages with alternative solutions such as yours.

TIA and ciao smile

P.S: For reference, in the Tracker, this is MDLSITE-385
Average of ratings: -
In reply to Eloy Lafuente (stronk7)

Re: Apache Memory Leak

by Matt Gibson -
Hmm...

I am noticing slight leakage with the apachelounge solution, although not half as bad as it used to be. Seems like there may be problems with the above XAMPP build.
Average of ratings: -
In reply to Ken Wilson

Re: Apache Memory Leak

by Doris Johnson -

Hi, Ken

Are you still using a windows box. If so, what programs and versions? I have a windows box (production server) that has the memory problem. I am using windows 2003 server, apache 2.2.10, php 5.2.6 and mysql 5.0.51A. Thanks for any help.

Average of ratings: -
In reply to Chris Potter

Re: Apache Memory Leak

by Doris Johnson -
I know this is an old post, but what are you using to diagnose the memory leak? Are you using the performance program in windows or do you have other programs to measure this problem.
Average of ratings: -
In reply to Doris Johnson

Re: Apache Memory Leak

by Mark Hilliard -
Hi Doris,
I stumbled across your post, hope you got an answer...What we always used for checking memory leak on windows was to open the Task Manager on the Web Server.

Click on the Processes Tab. If you are running Apache sort the list by alpha and find the process called HTTPD.EXE (or something very close) You can watch this process grow/shrink. If you have the leak the mem attached to this will grow/grow till your server dies from swapping out.

By now there should be versions of PHP/Apache that are solid w/o the leaks. Look at earlier threads here.

Mark Hilliard
Mpls MN
Average of ratings: -