Windows server: APC bug(?) related to MaxRequestsPerChild - crashes Apache

Windows server: APC bug(?) related to MaxRequestsPerChild - crashes Apache

by kael teh -
Number of replies: 7
All, thank you for any insight in advance.

I have APC installed and it is causing Apache to shutdown once MaxRequestsPerChild is reached. I came to this conclusion after load testing and checking the error log.

Under load testing, with a low MaxRequestsPerChild (e.g., 100-250) i get the errors below. However, if i set MaxRequestsPerCHild to no limit (0), i do not get the errors. I do not want to leave this setting to 'no limit' though.

Has anyone come across this situation before and found an explanation or fix? I would like to keep using APC but if this bug is a valid one it will be a showstopper.

ANy help and input much appreciated.

Environment
php 5.1.6
mysql 5.0.24a
win 2003 server 
apache 2.2.3 (win32)

httpd.conf (part)
ThreadsPerChild 150
MaxRequestsPerChild  0
MaxKeepAliveRequests 250
KeepAlive On
KeepAliveTimeout 2
EnableMMAP off
EnableSendfile off
Win32DisableAcceptEx

APC settings
; apc
extension  = php_apc.dll
apc.enabled  = 1
apc.shm_segments  = 4
apc.shm_size  = 128
apc.optimization  = 0
apc.num_files_hint  = 1000
apc.ttl  = 0
apc.gc_ttl   = 3600
apc.cache_by_default = On
apc.slam_defense = 15
apc.file_update_protection = 2
apc.enable_cli = 0
apc.stat  = 1
apc.report_autofilter = 1
apc.user_ttl = 600 ; seconds

Load Test
JMeter - x threads hitting phpinfo.php, mysqlinfo.php, and login page.

error.log
[notice] Apache/2.2.3 (Win32) PHP/5.1.6 configured -- resuming normal operations
[notice] Server built: Sep  6 2006 12:52:38
[notice] Parent: Created child process 7556
[notice] Disabled use of AcceptEx() WinSock2 API
[notice] Child 7556: Child process is running
[notice] Child 7556: Acquired the start mutex.
[notice] Child 7556: Starting 150 worker threads.
[notice] Child 7556: Listening on port 80.
[notice] Child 7556: Process exiting because it reached MaxRequestsPerChild. Signaling the parent to restart a new child process.
[notice] Parent: Received restart signal -- Restarting the server.
[apc-error] apc_shm_detach: shmdt failed: No error
[notice] Child 7556: Released the start mutex
[notice] Child 7556: Waiting for 150 worker threads to exit.
[notice] Child 7556: All worker threads have exited.
[Mon Sep 25 16:37:19 2006] [notice] Child 7556: Child process is exiting



Average of ratings: -
In reply to kael teh

Re: Windows server: APC bug(?) related to MaxRequestsPerChild - crashes Apache

by Anthony Borrow -
Picture of Core developers Picture of Plugin developers Picture of Testers
I'm glad to see someone else who likes to run with the latest versions. I just wante to check and see which version of APC you are experiencing problems with.
Average of ratings: -
In reply to Anthony Borrow

Re: Windows server: APC bug(?) related to MaxRequestsPerChild - crashes Apache

by kael teh -
I am currently using  version  3.0.12p2. I am going to test earlier versions and report back.

i guess there's always eAccelerator smile
Average of ratings: -
In reply to kael teh

Re: Windows server: APC bug(?) related to MaxRequestsPerChild - crashes Apache

by Anthony Borrow -
Picture of Core developers Picture of Plugin developers Picture of Testers
I've been happy with APC 3.0.10 but I am running on a LAMP server (Linux 2.6.15-26-386, Apache 2.0.55, MySQL 5.0.22, PHP 5.1.2). We have an average of 250,000 hits per day and averaging about 1.5G of traffic per day. I occasionally get some fragmentation; however, the server rolls the logs once a week and restarts and that has kept the fragmentation to a minimum.
Average of ratings: -
In reply to kael teh

Re: Windows server: APC bug(?) related to MaxRequestsPerChild - crashes Apache

by Gary Anderson -
I have had a similar problem of apache crashing 2 to 5 times a day and will try setting MaxRequestsPerChild to 0 (I now have it at 10000) and see if the crashes go away.
For me it is not a show-stopper as I just have the apache process set to restart on failure with no delay.  Users probably never notice a problem as it is just seconds before things are up again.  Still, I was waiting for the next XAMPP distribution that includes an update to php (I use 5.1.4) and a matching APC.dll to see what works.
Without APC, things were much slower and had problems when 100 students were on at once, so for now, it seems to be worth it.
I'll post back here to report if the problem goes away with the new experimental seting of MaxRequestsPerChild.  And then we can look for a longer term fix.
Average of ratings: -
In reply to Gary Anderson

Re: Windows server: APC bug(?) related to MaxRequestsPerChild - crashes Apache

by kael teh -
Gary,

 For me it is not a show-stopper as I just have the apache process set to restart on failure with no delay

Can you let me know how you accomplished this? In the 'Recovery' tab of the Apache service properties, i've tried using either the 'restart the service' option and 'Run program' option with no success.

I don't mind setting a high value for MaxRequestsPerChild, it's just that i'm not comfortable knowing that there is a potential for abuse, i.e., what if somebody decides to exploit this?

Thanks smile
Average of ratings: -
In reply to kael teh

Re: Windows server: APC bug(?) related to MaxRequestsPerChild - crashes Apache

by Ken Wilson -

Hi Kael

From this document (http://httpd.apache.org/docs/2.2/platform/windows.html), the recommended default for apache 2 in a Windows environment is to have MaxRequestsPerChild set to 0.

AFAIK, with Windows, Apache only needs the ThreadsPerChild setting changed to adjust for your potential maximum load in case of an exploit on your server - as its multithreading and doesn't prefork. So leave the MaxRequestsPerChild to the default as in the Windows docs - the behaviour is different to *nix apache.

Given this, the log entries you're seeing are to be expected. From the apc_shm.c code, the error is indicating that there is a problem freeing up the shared memory - acceptable given that you've guven apache a limit on the requests. So it's really a notice, as it says "No error" at the end. Does this make sense?

A few more thoughts:
- as of 3.0.11, the apc.slam_defense setting is obsoleted.
- you'll find better behaviour by setting the shm_segments to 1 and setting the shm_size to 512 (the algorithm seems to fragment memory less wink). That is a very big setting for a php opcode cache - is it above 80% utilised?
- set apc.stat = 0 and you'll also get better performance (unless you're a developer and changing Moodle code often)

Hope this helps!

Ken

Average of ratings: -
In reply to Ken Wilson

Re: Windows server: APC bug(?) related to MaxRequestsPerChild - crashes Apache

by kael teh -
Ken, thanks for you reply.

Lol yeah i've read the docs and was experimenting with tuning apache, but came across a few articles that specifically mention to change the MaxRequestsPerChild setting, with good reason too, so i did just for experimentation.  I will probably revert this back to 0.

I'm not too sure if i want to adjust ThreadsPerChild to be the potential maximum - through repeated load tests i am able to realize lower average times and higher throughputs using a more conservative figure. has anyone experienced this as well? I will explore this further.

The apc_shm.c code notice makes sense. Thank you smile

and for the rest, well i was just playing around with the APC settings, just to figure out the different effects. I will probably leave it at 1 segment @ 256 - I ran a quick simulation with 6 users and the cache was hardly used.

i had to set apc.stat = 1 to resolve the blank pages problem (even on prod. server).

Thanks everyone for the help/discussions.


Average of ratings: -