PHP-FPM crashes - log file errors

PHP-FPM crashes - log file errors

by Susan Mangan -
Number of replies: 10

Administering Moodle for 10+ years and need some help troubleshooting php-fpm crashes

Version 3.2.x 

Been getting php-fpm crash reports for a while and they have been slowly increasing - thinking this is likely due to specific application usage.

Crash reports indicate pwd of /mnt/moodle/moodle/grade/report

Above crash reports tie into the following php-fmp error log:

[21-Nov-2017 14:12:45 America/Vancouver]

PHP Warning: session_start(): read returned less bytes than requested in /mnt/moodle/moodle/lib/classes/session/handler.php on line 42

[21-Nov-2017 14:12:45 America/Vancouver]

PHP Warning: session_start(): Failed to decode session object. Session has been destroyed in /mnt/moodle/moodle/lib/classes/session/handler.php on line 42

Average of ratings: -
In reply to Susan Mangan

Re: PHP-FPM crashes - log file errors

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers

I cannot help you.  However, I wonder why you run php-fpm?  I am installing an new moodle and wonder whether I should install this php-fpm.  I'm working on moodle 3.4.

Average of ratings: -
In reply to Rick Jerz

Re: PHP-FPM crashes - log file errors

by Susan Mangan -

Hi Rick,

PHP-FPM was installed by one of our linux guys a number of years back.  At the time he wanted the latest and greatest and not vanilla php.

I can't really comment on whether or not it's worth it as I don't work on the back end too much of the time.  Usually just for upgrades and troubleshooting since I am more familiar with application-specific configuration.

What I do remember mostly about this particular setup is that it did not work with slash arguments turned on.  We had to make a manual adjustment in our httpd config for the proxypassmatch. (Thanks to Matteo for help with that one years back!!!)

Susan

Average of ratings: -
In reply to Susan Mangan

Re: PHP-FPM crashes - log file errors

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Susan,
for the record, MDL-51554 and https://github.com/scara/moodle/commit/3b1a3abcf9dd2faff5da8b8591c00b6412af0f88 wink.

BTW, that message comes from here, https://github.com/php/php-src/blob/811dfaa57b914630e1805c96f3fed83ecc97cc45/ext/session/mod_files.c#L529.
Are you still running PHP sessions based on files or e.g. memcached?
Did you notice an high pressure on RAM? What is the memory setting in your PHP_FPM conf and by means of what directive, php_admin_value vs php_value?
Is the recent change in usage due to more traffic or e.g. having deployed new plug-ins?
Any other change that happened in the mean time before the increase of such errors?

HTH,
Matteo

P.S.: probably time to plan an update of both OS and PHP versions, including accessing PHP via mod_fastcgi w/ some more RAM too wink.

Average of ratings: -
In reply to Matteo Scaramuccia

Re: PHP-FPM crashes - log file errors

by Susan Mangan -

Hi Matteo, thanks for the reply, my responses below:

 

Are you still running PHP sessions based on files or e.g. memcached?

- We have a dedicated memached server that we implemented when we built out our gluster farm but we are not using it for sessions.  Still files sessions.


Did you notice an high pressure on RAM? What is the memory setting in your PHP_FPM conf and by means of what directive, php_admin_value vs php_value?

- I don't see any spike in memory at the time of the crashes.  Just CPU.  I'll have to check the conf file for the value...

 

Is the recent change in usage due to more traffic or e.g. having deployed new plug-ins?

Any other change that happened in the mean time before the increase of such errors?

- We tried running unoconv in September after upgrading to 3.2 in May but there were so many php errors coming through after that and based on forum discussions it seemed as though a lot of others were also having issues with it so we decided not to proceed with it at this time.

- We went full HTTPS in June of this year and installed Content Security Policy to be able to monitor ssl violations but have since disabled it.

- Our actual usage has gone up slightly (number of users and courses on the system) but nothing dramatic.

- Our current linux admin seems to think the php crashes started happening just over a year ago which would have been right after we upgraded from 2.9 to 3.0

- We though it might have something to do with this bug that is still occurring on our current version 3.2x

- https://tracker.moodle.org/browse/MDL-46539

- https://tracker.moodle.org/browse/MDL-56223

 

P.S.: probably time to plan an update of both OS and PHP versions, including accessing PHP via mod_fastcgi w/ some more RAM too

- I should get our linux admin to comment on this.  As far as I am aware he updates software regularly.  We are currently running php 5.6

 

Average of ratings: -
In reply to Susan Mangan

Re: PHP-FPM crashes - log file errors

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers
I should get our linux admin to comment on this. As far as I am aware he updates software regularly. We are currently running php 5.6

I mean no regular updates on the same major but:

  • PHP 7.x: 7.1 at least
  • CentOS 7
  • mod_fastcgi

Waiting for the memory value to start thinking at what could be the root cause (frankly speaking, almost no idea as per the info provided here and what known from the past),
Matteo

Average of ratings: -
In reply to Matteo Scaramuccia

Re: PHP-FPM crashes - log file errors

by Susan Mangan -

Oh sorry for the delay Matteo, you are always such a great help.

PHP Memory for our webservers is set to 512M per web node (we have 3).

The application is set to use 2048M. The application value was set before we went live with our load balanced farm so I'm wondering if that should be brought back down.  I can look into ...

We are still load balancing with F5 and using dedicated memcached server for application cache (not sessions) and gluster software. 

Performance has been really good since we implemented this set up 2 years ago. 

Oh.. and we still use MySQL windows fail over cluster.

Can I send you a couple of crash reports?  I'm not sure if I should be uploading that detailed information here. 

Our linux admin thought the problem could be linked to oracle.  We use the External Database plug-in for our automated course and enrolment set up so we have the oracle client installed to communicate with MySQL.  This was configured years ago.  I think I will disable this temporarily to see if there is any correlation.

 

I mean no regular updates on the same major but:

  • PHP 7.x: 7.1 at least
  • CentOS 7
  • mod_fastcgi
Average of ratings: -
In reply to Susan Mangan

Re: PHP-FPM crashes - log file errors

by Ken Task -
Picture of Particularly helpful Moodlers

Don't run php-fpm but did have a little experience with it on a server where I was helping troubleshoot moodle issues ... that's my qualfications for this response. :\

Crash .. is that like a core dump?   If so, those are typically attributed to memory segmentation faults.   The php-fpm log file might give you a hint/clue, but it might not be the actual reason for the crash/core dump.  Note the php-fpm log file shows 'warnings' and not critical 'show stoppers'.   Moodle does check sessions often and might even refresh a sessions file in /pathto/moodledata/sessions/ for a user (teacher or student).  The actual error you see might mean the crash happened when moodle was attempting to refresh a sessions file.

I take it that turning on moodle debugging has not rendered anything or anything useful? (that might be true as the routine doesn't get a chance to render anything at all).

This might have to do with the config of php-fpm itself ... IF I re-call correctly there is a separate php-fpm config file ... from what you've shared so far, we know it's linux ... but other system details are left up to the imagination of responders. ;)

There is, if I re-call, a memory limit setting as well as others that might be related.

'spirit of sharing', Ken


Average of ratings: -
In reply to Susan Mangan

Re: PHP-FPM crashes - log file errors

by Ken Task -
Picture of Particularly helpful Moodlers

Follow up to response ... 3.2.x introduced unoconv (a python script) to converting  of uploaded docs submitted by students to PDF's.   Required the installation of LibreOffice that unoconv used for document conversions to PDF so grader to annotate submissions.

All that does eat up memory resources of a server.

You running unoconv/libreoffice etc. that sort of setup?

'spirit of sharing', Ken



Average of ratings: -
In reply to Ken Task

Re: PHP-FPM crashes - log file errors

by Susan Mangan -

Yes!  We did install unoconv and tried to get it working but we just got too many errors cropping up in the logs so decided to put it on hold for a while until we had more time to really look into it.

We never reach max memory.  CPU spikes occasionally from time to time  but doesn't seem like there is a correlation with when crashes happen and CPU spikes.

i do have php error logging on and I posted the errors that appear to coincide with the crashes (dumps) In original post.

i also discovered an old bug that still exists so maybe that is the culprit.  hundreds of Calendar events being logged by the second via cron. I thought upgrading would have remedied that but apparently not.  I've posted in the tracker .. so will see if that might be cause. https://tracker.moodle.org/plugins/servlet/mobile#issue/MDL-56223

thanks for your feedback as always .. trying to figure these things out can be draining at times ... 

 

Average of ratings: -