Moodle in English: Sudden Proxy Error, no recent system changes

Sudden Proxy Error, no recent system changes

by Tiffany Morgan - Tuesday, May 14, 2019, 4:49 AM

Hi all

We are using Moodle 3.5.1+ (Build: 20180831)

Starting approximately a week ago, teachers began reporting the following error when trying to make small text changes on pages:

Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request POST /course/modedit.php.

Reason: Error reading from remote server

This message is received after waiting approximately 2 minutes., Last week I was able to replicate this error across all courses, regardless of course size. Today it seems to be limited to certain courses. I still get a similar error when trying to restore a course.

Since then we have confirmed the firewall logs are not showing anything being restricted, there have been no network or server changes. We have also tried changing the following

Search for the post_max_size setting to 200M
Do the same for upload_max_filesize (make the numbers the same)
Search for the max_execution_time setting and up this to 300

I've also updated the Maximum upload size via the Moodle admin to 200M

Any recommendations as to a possible cause and/or additional items to test?

Thanks!

Re: Sudden Proxy Error, no recent system changes

by Tiffany Morgan - Wednesday, May 15, 2019, 4:11 AM

I've done some more testing:

Increased PHP memory limit from 128MB to 256MB with no change.
I've been able to successfully edit a plain text page (no html) within a course which previously produced an error.
I also am able to get a page to produce a proxy error and then not produce a proxy error by removing code pointing to the css file. I've tried this across multiple courses each using completely different css files with the same result. These css files have been in place an unchanged for 3+ years

Any recommendations as to a possible cause and/or additional items to test?

Thanks!

Average of ratings: -

Re: Sudden Proxy Error, no recent system changes

by Jerry Lau - Thursday, May 16, 2019, 3:42 AM

try exactly all the steps you have done outside your regular network, on a different machine and browser and let us know what happens

Average of ratings: -

Re: Sudden Proxy Error, no recent system changes

by Jerry Lau - Thursday, May 16, 2019, 3:45 AM

did you IT department update their antivirus?

Average of ratings: -

Re: Sudden Proxy Error, no recent system changes

by Tiffany Morgan - Friday, May 17, 2019, 12:20 AM

Thanks Jerry, appreciate the response. I tried on a different network, device, and browser with the same error message results. No changes to antivirus.

Average of ratings: -

Re: Sudden Proxy Error, no recent system changes

by Jeff White - Friday, May 17, 2019, 1:46 AM

With pages that load successfully do you get any messages in browser debug mode? Is it saying it cant read the css, js, etc files? Also do you have the https traffic offloaded as http to your web nodes?

Average of ratings: -

Re: Sudden Proxy Error, no recent system changes

by Tiffany Morgan - Friday, May 17, 2019, 3:30 AM

Hi Jeff

We offload https at the proxy. The moodle server only sees HTTP. I've turned on moodle debugging. No messages there. I turned on Moodle performance messages. Here is one example of a display

0.36 seconds: Load Time

31.5MB: Memory Used

340: Files Included

164/53: DB Read/Write

Using Chrome dev tools I did receive two error messages. Two indicated mixed content (calling http image on the http site). Fixed that and those error messages went away, but the proxy error remained.

Average of ratings: -

Re: Sudden Proxy Error, no recent system changes

by Jeff White - Friday, May 17, 2019, 4:21 AM

Do you have the ability to copy your entire prod moodle env to a stage/dev environment that emulates your prod? I would try to see if the issue is consistent across environments.

If you have load balancing, I would turn off all but 1 of the web nodes and try to reproduce the error with multiple attempts on each web nodes with all the others off. That would help to see if maybe 1 of the web nodes is misconfigured or being flaky. As you might guess it would be best to do this on your dev/stage over prod as this will impair performance or you knock out your system while playing around in it.

Average of ratings: -

Re: Re: Sudden Proxy Error, no recent system changes

by Tiffany Morgan - Friday, May 17, 2019, 10:13 PM

Hi all Despite my previous inquiries to IT, I just today learned there has in fact been recent updates to the system, specifically to a firewall. When that was turned off for testing the error immediately disappeared. I appreciate the help and apologize for taking up moodlers time. Thanks again.

Average of ratings:Useful (2)