Unexplained proxy_fcgi:error messages in server logs

Unexplained proxy_fcgi:error messages in server logs

by Stuart Anderson -
Number of replies: 6

Hi,

I have moodle running in CentOS Linux release 7.9.2009.  Since early March we have started seeing proxy_fcgi errors appearing in our log files. At their worst these were showing: 

[proxy_fcgi:error] (104) Connection reset by peer: [client xx.xx.xx.xx:63353] AH01075: Error dispatching request to :, referer: https://stack.fse.manchester.ac.uk/moodle/course/view.php?id=84

and some functionality around browsing and adding enrolments was affected in the front end.


This week we have see a few of the following:

[Tue Mar 08 22:16:27.750609 2022] [proxy_fcgi:error] [pid 1662] (70008)Partial results are valid but processing is incomplete: [client xx.xx.xx.xx:56859] AH01075: Error dispatching request to :, referer: https://stack.fse.manchester.ac.uk/moodle/mod/quiz/attempt.php?attempt=302668&cmid=783&page=2

[Tue Mar 08 23:48:57.867353 2022] [proxy_fcgi:error] [pid 1720] (70008)Partial results are valid but processing is incomplete: [client xx.xx.xx.xx:3778] AH01075: Error dispatching request to :, referer: https://stack.fse.manchester.ac.uk/moodle/mod/quiz/attempt.php?attempt=302571&cmid=2007&scrollpos=4650

[Wed Mar 09 02:27:00.835661 2022] [proxy_fcgi:error] [pid 1718] (70007)The timeout specified has expired: [client xx.xx.xx.xx:6288] AH01075: Error dispatching request to :, referer: https://online.manchester.ac.uk/

The "partial results" and "timeout" messages haven't resulted in any complaints from students/staff, but there has only been a few of these types of error.


I'm looking for some advise for the reason for these and how we can prevent them.  It doesn’t seem to be at all right that we are seeing:

Error dispatching request to :, referer ...

Where it is sending a request to? Maybe that was causing some of the problems we have been seeing.

Advice and suggestions greatly appreciated.

Best wishes,

Stuart

Average of ratings: -
In reply to Stuart Anderson

Re: Unexplained proxy_fcgi:error messages in server logs

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Is there a proxy in the mix? Are the timeouts and so on set sensibly?
Average of ratings: -
In reply to Howard Miller

Re: Unexplained proxy_fcgi:error messages in server logs

by Stuart Anderson -
Thanks. We have no proxy and timeouts are set at defaults (and thus can be changed). I presume you are going to recommend increasing the timeouts?
Average of ratings: -
In reply to Stuart Anderson

Re: Unexplained proxy_fcgi:error messages in server logs

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Only in relation to a proxy. That's what those errors sounded like. Can you explain your server setup in a bit of detail?
Average of ratings: -
In reply to Howard Miller

Re: Unexplained proxy_fcgi:error messages in server logs

by Stuart Anderson -
Thanks. The magagment of our service is shared between myself and some dedicated IT server admins. I'll do my best to describe the setup, but for more specific detail, tell me the kind of thing it is helpful to know and I'll get the info from our dedicated IT server admins.

Our moodle service has been setup purely to provide a platform for running mathematical STACK question based quizzes. We use LTI to connect our main VLE platform (Blackboard) to moodle to allow students to take STACK based quizzes. Once students are trasferred from Blackboard to moodle via LTI, they are effectively the same as any other moodle user.  That detail isn't that important, but as a result of this we have a 3 server setup. All servers are running CentOS Linux release 7.9.2009 and are more-or-less vanilla out of the box installs with no special configuration (yet). Our servers are all running on Virtual Machines:

  • Moodle - running on apache with fast-cgi enabled.
  • Maria DB - no DB tuning (yet).
  • Maxima - a dedicated server running the maxima tool for evaluating and processing STACK question input.

Our memory/processor config is:

  • Moodle: CPU cores: 8 x Intel(R) Xeon(R) CPU @ 2.60GHz; Physical Memory: 16GB
  • Maria DB: CPU cores: 2 x Intel(R) Xeon(R) CPU @ 2.60GHz; Physical Memory: 10GB
  • Maxima: CPU cores: 2 x Intel(R) Xeon(R) CPU @ 2.60GHz; Physical Memory: 16GB

The only other point to mention is that as far as we know, these errors have only recently started appearing.  There were no relevant recent changes made to the server immediately beforehand.

Thanks and best wishes,

Stuart

Average of ratings: -
In reply to Stuart Anderson

Re: Unexplained proxy_fcgi:error messages in server logs

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

I know that this discussion is old. I came here from your recent discussion Behaviour of FAILURES_BEFORE_NOTIFY in quiz autosave in the Quiz forum. Were you able to resolve this issue?

I haven't come across the same error. But a web search points to the Apache mod_reqtimeout. It is active by default. Its config file, reqtimeout.conf, has this:

<IfModule reqtimeout_module>
# mod_reqtimeout limits the time waiting on the client to prevent an
# attacker from causing a denial of service by opening many connections
# but not sending requests. This file tries to give a sensible default
# configuration, but it may be necessary to tune the timeout values to
# the actual situation. Note that it is also possible to configure
# mod_reqtimeout per virtual host.

# Wait max 20 seconds for the first byte of the request line+headers
# From then, require a minimum data rate of 500 bytes/s, but don't
# wait longer than 40 seconds in total.
# Note: Lower timeouts may make sense on non-ssl virtual hosts but can
# cause problem with ssl enabled virtual hosts: This timeout includes
# the time a browser may need to fetch the CRL for the certificate. If
# the CRL server is not reachable, it may take more than 10 seconds
# until the browser gives up.
RequestReadTimeout header=20-40,minrate=500

# Wait max 10 seconds for the first byte of the request body (if any)
# From then, require a minimum data rate of 500 bytes/s
RequestReadTimeout body=10,minrate=500

</IfModule>

The solutions vary from completely disabling the module to changing the configuration, for example to:
RequestReadTimeout header=10-40,MinRate=500 body=40,MinRate=500

Source: proxy_fcgi:error (70008)Partial results are valid but processing is incomplete. AH01075

Average of ratings: -
In reply to Stuart Anderson

Re: Unexplained proxy_fcgi:error messages in server logs

by Stuart Anderson -
I can now follow up on this with what I think was the cause of my error. I did the confirm this in our production environment, but all indicators from work on my test server is that this issue was a result of the PHP opchache not being configured with the required settings for moodle. We had opcache installed, but only with defaults. The correct opcache settings are detailed here:
https://docs.moodle.org/311/en/OPcache

When relying on opcache defaults, seg fault erros bubble up and present as proxy_fcgi errors else where when errors in cacheing start to occur. This was a major problem during our moodle upgrade in our test environment where changes to the code base were out of step with that was cached and a right mess ensued.

Hope this hopes another.
Average of ratings: -