CPU & connections max'ed - PHP 5.3.1 problem?

CPU & connections max'ed - PHP 5.3.1 problem?

by Worth Bishop -
Number of replies: 11
Wondering whether there might be a problem running Moodle 1.97+ against PHP 5.3+?

I've recently migrated a Moodle site running against PostgreSQL from a
single server, running FreeBSD, to a webserver and a database server (spec's below).

After upgrading to beefier hardware & current releases of software (notably PHP 5.3.1) we had several instances where Apache stopped responding. The error_log showed:

[Thu Feb 04 13:38:36 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting.

We upped the MaxClients setting from 150 to 225. Since then, we are seeing episodes of high CPU usage (>98%) which effectively prevents Apache from responding again.

We haven't been able to trace this condition to any specific user activity. It doesn't appear to be a gradually building phenomenon, as the machine can hum along nicely for hours, then suddenly spike and stay spiked. We have observed that it seems to happen pretty reliably when 12 - 15 users are taking quizzes simultaneously, but it happens at other times as well.

This did not happen under our previous configuration, running PHP 5.2.6. However, after migrating to the two new servers, we upgraded PHP on the old server (now sparingly used by another Moodle instance) to 5.3.1 and have experienced one occasion when Apache stopped responding there as well.

Searching the forums, bugtracker & web has shown others with somewhat similar problems reported (see references below) but no definitive solutions. I'm wondering whether their problems are related to this version of PHP?

The error_log is logging other PHP errors (below) which appear to be only warnings. During automatic backups, tens of thousands of:

"PHP Deprecated: Function ereg() is deprecated in /htdocs/moodle/lib/moodlelib.php on line 4738"

and

"PHP Deprecated: Function ereg_replace() is deprecated..."

errors have been logged in the space of a few minutes, bloating the logfiles.

Before I roll back to an earlier version of PHP (probably PHP 5.2.12) - has anyone any better suggestions?

Thanks!
Error Messages:


[Thu Feb 04 01:20:04 2010] [error] [client #.#.#.#] PHP Warning: strftime(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/New_York' for 'EST/-5.0/no DST' instead in /htdocs/moodle/lib/moodlelib.php on line 1255
[Thu Feb 04 01:20:04 2010] [error] [client #.#.#.#] PHP Warning: strftime(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/New_York' for 'EST/-5.0/no DST' instead in /htdocs/moodle/lib/moodlelib.php on line 1256
[Thu Feb 04 01:20:04 2010] [error] [client #.#.#.#] PHP Warning: date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/New_York' for 'EST/-5.0/no DST' instead in /htdocs/moodle/lib/statslib.php on line 912
[Thu Feb 04 01:20:04 2010] [error] [client #.#.#.#] PHP Warning: strtotime(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/New_York' for 'EST/-5.0/no DST' instead in /htdocs/moodle/lib/statslib.php on line 912
[Thu Feb 04 01:20:04 2010] [error] [client #.#.#.#] PHP Warning: date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/New_York' for 'EST/-5.0/no DST' instead in /htdocs/moodle/lib/statslib.php on line 912
[Thu Feb 04 01:20:04 2010] [error] [client #.#.#.#] PHP Warning: strtotime(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/New_York' for 'EST/-5.0/no DST' instead in /htdocs/moodle/lib/statslib.php on line 912
[Thu Feb 04 01:20:04 2010] [error] [client #.#.#.#] PHP Warning: date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/New_York' for 'EST/-5.0/no DST' instead in /htdocs/moodle/lib/statslib.php on line 912
[Thu Feb 04 01:20:04 2010] [error] [client #.#.#.#] PHP Warning: strtotime(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/New_York' for 'EST/-5.0/no DST' instead in /htdocs/moodle/lib/statslib.php on line 912


Configuration:

- two virtual machines, both running CentOS 5.4, Linux version
2.6.18-14.10.1.el5 both with 3 Gb RAM and two dual-core
Intel processors allocated to each. A SAN is providing 200 Gb storage for each.

- the web server is running Apache 2.2.14 & PHP 5.31 with eAccelerator v0.9.6-rc1 and Suhosin v0.9.29.

- the database server is running PostgreSQL 8.4.1, with pg_hba.conf set up
to trust the webserver on port 5432.

- both Apache & PostgreSQL are set to accept 225 max connections, otherwise
the conf's are pretty much default.

- web server is running OpenSSL for secure login, but serving general html
pages without https.

- tcp_keepalive_time in both is default 7200 seconds (which, as I read in
various posts, etc., shouldn't really matter anyway, but...)

- we are running against an external LDAP server, using OpenSSL for logins only.

Forum & bug tracker references:

Massive server problem - Nick Oliver Jan 12, 2010
http://moodle.org/mod/forum/discuss.php?d=141359

Moodle installed with xampp- High CPU usage for 50 concurrent users -Sayak Sarkar Jan 8, 2010
http://moodle.org/mod/forum/discuss.php?d=141108

Current Server = 5 users max... need a new server? - Josh Jones Jan 8, 2010
http://moodle.org/mod/forum/discuss.php?d=141053

Hanging while loading profiles - Tim Hart - Dec 25, 2009
http://moodle.org/mod/forum/discuss.php?d=140505

Apache stops processing - Renato Ardigo - Dec 9, 2009
http://moodle.org/mod/forum/discuss.php?d=139563

convert_urls_into_links() is converting inside image tags
http://tracker.moodle.org/browse/MDL-21168

convert_urls_into_links() causing slowdown/timeouts
http://tracker.moodle.org/browse/MDL-21296


Average of ratings: -
In reply to Worth Bishop

Re: CPU & connections max'ed - PHP 5.3.1 problem?

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Have you given it a try without the accelerator? I have had some very variable experiences with those.

Also how many users and how many simultaneous connections are we talking about (roughly)?

Have you got something monitoring your boxes? I usually use munin - it's dead easy to install and you get lots of pretty graphs. The pretty graphs often can show up the cause of a problem much ore quickly than staring at logs.
Average of ratings: -
In reply to Howard Miller

Re: CPU & connections max'ed - PHP 5.3.1 problem?

by Worth Bishop -
Thanks for the reply, Howard.

I initially tried it without the accelerator, but performance was disappointing. I'd been using e-accelerator before migrating to the new machines with good results, so reinstalled it.

Re: users - a few hundred user accounts; maybe 25-30 users actively using the system within the same 5 minute period (taking a quiz - as they'd done on the old server with perhaps a little slow down, but no show stoppers).

Max connections is a little tougher to say - when the system is running well, netstat on the webserver will show 40-50 sockets in varying states (ESTABLISHED, TIME-WAIT, LAST-ACK, etc) connected via the webserver port and a like number connected to the database machine.

When the problem hits, netstat shows vastly more sockets, the CPU pegs, connecting via web becomes virtually impossible (ssh extremely slow to connect & respond) and the "Server has reached MaxClients" gets logged.

I'll give munin a try...

FWIW, I had been running a copy of MediaWiki on the old server, along with the Moodle instance. After updating PHP-5.26 to PHP-5.3.1, MediaWiki stopped working and threw PHP errors. From what I've seen Googling, the only options there are to upgrade MediaWiki or revert to an earlier version of PHP...

Thanks again for your efforts!


Average of ratings: -
In reply to Worth Bishop

Re: CPU & connections max'ed - PHP 5.3.1 problem?

by Geoffrey Rowland -
Picture of Plugin developers

Hi Worth

I had a similar issue with an upgrade of CentOS 5 to PHP 5.3.x (from the Remi rpm repository, usually very reliable for latest PHP, MySQL etc for RHEL/CentOS).

I 'fixed' the issue by rapidly reverting to PHP 5.2.x. Currently, things are working well for us on:

PHP 5.2.10, MySQL 5.1.37 Moodle 1.9.7 (Also, PostgreSQL 8.2.15 behind our Mahara 1.2.x)

Amongst other issues, there does seem to be a specific problem with eAccelerator with some (early?) PHP 5.3.x releases. For example, from the release notes, recent XAMPP bundles with PHP 5.3.x have eAccelerator disabled.

Hope that helps

Geoff

Average of ratings: -
In reply to Worth Bishop

Re: CPU & connections max'ed - PHP 5.3.1 problem?

by Brenda Honn -

We are having similar performance problems - consistently with quizes but at other times also. We are using a host that just (March 4th) upgraded and are currently running the following.

MySql 5.0.89
PHP 5.3.1
Apache 2.2.14
Moodle 1.9.7

We would be very interested in any solutions found.

Average of ratings: -
In reply to Worth Bishop

Re: CPU & connections max'ed - PHP 5.3.1 problem? - SOLVED

by Worth Bishop -
Wanted to close the book on this:

For reasons unrelated to these problems, we ended up moving this site to a different machine. The machine itself was similar in all material respects to the previous machine. Running under VMWare, we set up two servers, one for the webserver and one for the database server.

The main difference in the new configuration was that instead of CentOS, we installed FreeBSD 8.0. All the other components of the stack - PHP, PostgreSql, Moodle, etc were configured pretty much identically to the previous installation. We also used the same Apache webserver version as previously.

When we started Apache the first time, we logged this error:

warn] (2)No such file or directory:
Failed to enable the ‘httpready’ Accept Filter

After curing this problem, (a single line added to the loader.conf, permanent fix) Apache ran flawlessly and all our CPU-related problems have disappeared. We no longer have issues when students are taking quizzes, our CPU utilization rates are comfortably low and life is good.

The Apache "Accept Filter Directive" did the trick:

"This directive enables operating system specific optimizations for a listening socket by the Protocol type. The basic premise is for the kernel to not send a socket to the server process until either data is received or an entire HTTP Request is buffered. Only FreeBSD's Accept Filters and Linux's more primitive TCP_DEFER_ACCEPT are currently supported."

Put another way:

"The utility of accf_http is such that a server will not have to context switch several times before performing the initial parsing of the request. This effectively reduces the amount of required CPU utilization to handle incoming requests by keeping active processes in preforking servers such as Apache low and reducing the size of the file descriptor set that needs to be managed by interfaces such as select(), poll() or kevent() based servers."

As the Apache installation on the previous CentOS-based server was installed by someone else, I don't know whether the Linux flavor of this directive was implemented. I mention it here for what it's worth, should someone else have similar issues and care to give it a try.

Meanwhile, FreeBSD is working very well for us!



Average of ratings: -
In reply to Worth Bishop

Re: CPU & connections max'ed - PHP 5.3.1 problem? - SOLVED

by Paul Hodgson -
We were having problems with this and even though we tried increasing limits and so on, we traced the problem to UPDATE mdl_stats_daily running and not "letting go" as it was supposed to.

We took the radical approach of turning off statistics gathering and we've had a stable system ever since.
Average of ratings: -
In reply to Paul Hodgson

Re: CPU & connections max'ed - PHP 5.3.1 problem? - SOLVED

by HJWUCGA INC. -
Paul,

You can always run the stats by itself outside of the cron itself.

That's what we do in our productions server. The stats do lock up the system and is quite db intensive.


Average of ratings: -
In reply to HJWUCGA INC.

Re: CPU & connections max'ed - PHP 5.3.1 problem? - SOLVED

by Noveck Gowandan -

How did you accomplish this?
My logic would be to isolate the stats script in cron.php into a separate script and then call that script from the Server Cron.

Was isolating the stats processing code difficult?

Regards,
Noveck

Average of ratings: -