Tuning FCGI and Apache for Moodle

Tuning FCGI and Apache for Moodle

by Stephen Bourget -
Number of replies: 12
Picture of Core developers Picture of Plugin developers Picture of Testers

Hi Moodlers,

I'm running Moodle 3.5.X on a WAMP stack and encountering a weird issue with PHP7 & FCGI.
I currently have apache 2.4 running on Windows server 2008r2 and configured with FCGI with the following settings:

LoadModule fcgid_module modules/mod_fcgid.so

<IfModule fcgid_module>
    FcgidInitialEnv PATH "C:/Progra~1/PHP/;C:/WINDOWS/system32;C:/WINDOWS;C:/WINDOWS/System32/Wbem;"
    FcgidInitialEnv SystemRoot "C:/Windows"
    FcgidInitialEnv SystemDrive "C:"
    FcgidInitialEnv TEMP "C:\Windows\Temp"
    FcgidInitialEnv TMP "C:\Windows\Temp"
    FcgidInitialEnv windir "C:/WINDOWS"

    FcgidIOTimeout 1200
    FcgidConnectTimeout 1200
        FcgidBusyScanInterval 1200
        FcgidBusyTimeout 1200
        FcgidErrorScanInterval 1200
        FcgidIdleScanInterval 1200
        FcgidIdleTimeout 1200
        FcgidProcessLifeTime 3600
        FcgidZombieScanInterval 1200

    FcgidMaxRequestsPerProcess 1000
    FcgidMaxProcesses 500
    FcgidMaxRequestLen 813107200
    # Location php.ini:
    FcgidInitialEnv PHPRC "C:/Progra~1/PHP/"
    FcgidInitialEnv PHP_FCGI_MAX_REQUESTS 5000

    <Files ~ "\.php$">
        Options Indexes FollowSymLinks ExecCGI
        AddHandler fcgid-script .php
        FcgidWrapper "C:/Progra~1/PHP/php-cgi.exe" .php
    </Files>
</IfModule>

Intermittently Moodle stops responding to requests and the logs are full of errors like:

[Thu May 16 10:03:50.071867 2019] [core:error] [pid 14044:tid 8284] [client 172.17.115.24:54804] End of script output before headers: index.php
[Thu May 16 10:03:50.071867 2019] [fcgid:warn] [pid 14044:tid 7640] (OS 109)The pipe has been ended.  : [client 10.0.77.1:52951] mod_fcgid: get overlap result error, referer: https://moodleserver_URL/mod/assign/view.php?id=88

Googling the error hasn't been that helpful, and I'm assuming the server not responding is due to some FCGI timeout (As I have 100 php-cgi.exe processes running) and other virtual hosts on that server are responding and windows task manager is showing that there are both CPU and RAM resources to spare.

Is anyone else running Apache 2.4 and PHP 7 using FCGI on windows?  If so are you willing to share your FCGI config?

Average of ratings: -
In reply to Stephen Bourget

Re: Tuning FCGI and Apache for Moodle

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
My understanding (and, as I like to say, I don't know anything about Windows) is that running Apache on Windows in production is a bad plan. Much better to stick with IIS.
Average of ratings: -
In reply to Howard Miller

Re: Re: Tuning FCGI and Apache for Moodle

by Stephen Bourget -
Picture of Core developers Picture of Plugin developers Picture of Testers
Unfortunately switching to IIS is not really an option at the moment, since I have some other open source applications running on the same machine using virtual hosts (they use .htaccess files for mod_rewrite).

I'd be happy with some pointers on configuring mod_fcgi for Moodle since it seems to be the recommended way for running Apache with PHP, but there really aren't any current docs with best practices.
Average of ratings: -
In reply to Stephen Bourget

Re: Re: Re: Tuning FCGI and Apache for Moodle

by Luis de Vasconcelos -
Apache on Windows can be quite a pain to troubleshoot... You can do the same thing that .htaccess does in Apache using IIS web.config files. And the URL Rewrite Module in Wndows is the equivalent to the Apache mod_rewrite.

Maybe https://docs.microsoft.com/en-us/iis/application-frameworks/install-and-configure-php-applications-on-iis/translate-htaccess-content-to-iis-webconfig or https://blogs.msdn.microsoft.com/azureossds/2015/04/23/converting-apache-htaccess-rules-to-iis-web-config-using-iis-manager-for-azure-websites/ can help you?
Average of ratings: -
In reply to Stephen Bourget

Re: Re: Re: Tuning FCGI and Apache for Moodle

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
May also a non-option for you. Nevertheless since you want to stick to Apache and run Open Source other than Moodle too, why not fully Open Source, I mean the LAMP stack? As a bonus, you will get better community support here. (Not because they boycott Microsoft, no, most of the supporters and developers run either Linux or some other Unix version.)
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Re: Re: Re: Tuning FCGI and Apache for Moodle

by Stephen Bourget -
Picture of Core developers Picture of Plugin developers Picture of Testers
While I appreciate the responses and the suggestions to migrate the server to Linux or replace Apache with IIS, none of these are really viable in the middle of an academic term due to the downtime it would create. (We have discussed moving the server to CentOS, but that will require a few days of downtime and would not be possible until mid summer at best.)

Our Apache server has been remarkably stable over the last few years, I've only started having issues since moving to mod_fcgid from mod_php with the migration to PHP7. This makes me suspicious that it is not configured appropriately for hosting Moodle. For people who are using Linux and Apache, how are you running PHP? Are you using mod_fcgid? If so, how do you have it configured?
Average of ratings: -
In reply to Stephen Bourget

Re: Re: Re: Re: Re: Tuning FCGI and Apache for Moodle

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
You're really not going to appreciate me saying, "well move back to mod_php then", are you? wink
Average of ratings: -
In reply to Howard Miller

Re: Re: Re: Re: Re: Re: Tuning FCGI and Apache for Moodle

by Stephen Bourget -
Picture of Core developers Picture of Plugin developers Picture of Testers
I would, but on windows it's even less stable than mod_fcgid with PHP7. (It worked wonderful with PHP 5.6 but of course Moodle 3.5 doesn't support that) sad 
I did try running it on PHP7 but I had to restart Apache every few hours due to some sort of memory leak.  PHP 7.1 might be better, but I haven't had a chance to test all of the other things running on that server for compatibility yet.
The irony is I've got 4 different servers running Apache and mod_fcgi on windows and the other 3 don't have the issue (same version of apache / php).  Of course two of the others aren't running Moodle and the last is only running moodle in a test / dev environment.

I can always post to the apachelounge.com forums, but they wouldn't know how to tune it for Moodle
Average of ratings: -
In reply to Stephen Bourget

Re: Re: Re: Re: Re: Re: Re: Tuning FCGI and Apache for Moodle

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Honestly - and I've read what you've said about and sympathise with your situation - your one-hit-and-forget-about-it solution is Windows+IIS or Linux+Apache.

I've said it before and I'll no doubt say it again. Use the configurations everybody else uses or you'll be on your own. I strongly suspect, this is a case of you being on your own.
Average of ratings: Useful (1)
In reply to Stephen Bourget

Re: Re: Re: Re: Re: Tuning FCGI and Apache for Moodle

by Ken Task -
Picture of Particularly helpful Moodlers

Chiming in on what's italized below ... and something to consider (BTW, have done what am proposing below more than once).   Maybe reference it later ...

"We have discussed moving the server to CentOS, but that will require a few days of downtime and would not be possible until mid summer at best"

You wouldn't need to wait until mid summer to do the heavy lifting!!!
A CentOS 7 server (or setup ... dedicated DB server - private IP ... dedicated server for filedir of moodledata - private IP ... single web front end used as a 'stealth'.   Key word ... 'stealth'.

Setup the stealth CentOS 7 apache however you'd like to run that ... mod, php-fmp,fastcgi ... whatever.

Clone production server to it ... and/or the dedicated DB server for stealth .. the dedicated server for filedir of moodledata.  Restore it and still using the FQDN of the production server.

All the above could be accomplished without using browser from a workstation but when you get ready for browser use, on your workstation add the mapping to your local host file .... **newCentOS7IP** SAMEFQDN SAMESHORTNAME.

On the stealth one could reset all courses ... no users are in them ... no teachers are assigned ... all student work removed ... just shell courses ready for teachers to be assigned and students enrolled.

On a temporary basis one would have to run the stealth with self-signed cert.
And it's no biggy to setup a valid cert right after changing internal/external DNS
to point the FQDN from old iP to new IP.

And that would be the appropriate time to mess with Mail + authentications ... depending upon how y'all do that.

A 'stealth' setup could be done in a day or a week or over a few weeks ... no rush.
You could actually get some baselines on performance of the entire setup since it's only dev's using the setup ... would have something to compare when checking performance later! smile

Ok ... realize this doesn't address the current issues ... but felt if I didn't mention/briefly describe now, I'd forget to .... happens when one grows older! :\

My 2 cents.

'SoS', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: Re: Re: Re: Re: Re: Tuning FCGI and Apache for Moodle

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I agree... with planning this could be done in the time it takes to copy your database over. Certainly hours not days.
Average of ratings: -
In reply to Stephen Bourget

Re: Tuning FCGI and Apache for Moodle

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

If you are familiar with Linux and considered that option, then I said nothing! ;)

But then _caution_: I didn't suggest you to migrate tomorrow. Such a major change needs planning. The earliest I can think of it the summer break (if you are in the northern hemisphere).
Saw that Ken gave a longer answer.

The main point, I don't say Windows make a bad server for Moodle. But you see the problem. You have two Moodle servers doing fine on Windows and one not. But your debugging capabilities are limited. Or, to put it another way, your moves are dictated by Windows, not the vast choice Linux offers. (Sorry, I can not answer your particular question, I migrated to Linux decades ago! Even my desktop is Linux.)
Average of ratings: -
In reply to Stephen Bourget

Re: Tuning FCGI and Apache for Moodle

by Simon Lewis -

Hello,

Did you get anywhere with your fine tuning? We're also running on a windows server/php 7.2 and have a fcgid occasional issue where it runs out of slots, or the process slots aren't deleted (mod_fcgid: can't apply process slot ), so I'm just curious to know if anything you did helped.


thanks,

Simon

Average of ratings: -