Problems crashing on the server

Problems crashing on the server

by Franky Vallori -
Number of replies: 7

The version of moodle is 3.1.11.

The PHP version is 7.0.10.

The version of MariaDb is 5.5.44

Redhat Linux Server 7

About 40,000 users.

Courses about 3000.


Since a few weeks, after updating to PHP 7.0, we have encountered some crash problems, due either to the PHP server or the MariaDB Server.

Often, the server can not handle connections to the DB.


The server is very powerful: 128 Gb RAM, 24 CUP.


Do you have any parameter configuration recommendations for PHP and MYSQL servers?
Example timeout or maximum connections or other.


Thank you,
Francesco

Average of ratings: -
In reply to Franky Vallori

Re: Problems crashing on the server

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

I saw your previous status here https://moodle.org/mod/forum/discuss.php?d=361560. So you upgraded PHP to 7.0.10 (from 5.4.16) and kept the Moodle version 3.1 with minor upgrade from 3.1.8 to 3.1.11? Nothing else changed?

By all account, you should expect an improvement in performance - not a regression. You need to monitor the server carefully during peak hours and find the bottle neck. Just tuning parameters without seeing their results will make things worse.

You run a big site. It is a single server, right? Makes debugging easier. All caching mechanisms are properly set, right?

Edit: I see, you have localized the problem to the database server. What are the errors you get?

In reply to Visvanath Ratnaweera

Re: Problems crashing on the server

by Franky Vallori -

Hi,

I'm monitoring the server carefully during peak hours.

It's a single server. 

Makes debugging easier?
My configuration is on attach

All caching mechanisms are properly set, right?
My setting is default of "Cache administration"

What are the errors you get?
Page with "Error reading from database"


Thanks,
Francesco

Attachment cache_store.jpg
Attachment debug.jpg
In reply to Franky Vallori

Re: Problems crashing on the server

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
So, what is the monitoring system you use? (You will find a discussion thread with many possible monitoring tools in the forum documentation.) You need to use a system, where you can interpret the result. As Ken already wrote, a remote diagnosis is not possible.

The Moodle reports you attached show at least there is no misconfiguration. One thing I noticed, no memcache?

"Error reading from database" could only mean your database is overloaded. That should be the first place to investigate. See Ken's prev. post.
In reply to Franky Vallori

Re: Problems crashing on the server

by Ken Task -
Picture of Particularly helpful Moodlers

Don't think anyone could provide accurate settings for your server ... especially if it's an all in one ... DB server on same server as webservice and moodle code.

There are tools that will help with the DB ... install mysqltuner.   It's a perl script that will provide recommendations for tweaks to the DB ... such things as addressing 'MySQL server has gone away' - usually max_allowed_packet

https://dev.mysql.com/doc/refman/8.0/en/packet-too-large.html

The above link to MySQL version 8 but the same page and information is also for 5.6,etc.

Also: https://mariadb.com/kb/en/library/server-system-variables/

MariaDB is now the 'darling database' for true RedHat Enterprise.   Supposed to be a drop in replacement for MySQL so the docs concerning mysqlserver should/might apply to MariaDB.

You could be reaching usage such that a dedicated DB server would be best.  Have read that DB performs best when it can use 80% of the memory.  But on a standalone box that's hard to achieve because it's a balancing act between web service and DB server.  MySQLTuner will help you.  In it's report/info display it will show the number of dropped connections ... which usually means one need to tweak 'max_connections' value for DB upwards ... default is 151 in MySQL and I bet you've already tweaked that upwards.

While you have a large number of users and courses, students basicially interact with courses/moodle asynchronously .. a few students at a time .... not like a K12 entity using Moodle in a F-2-F setting.

Then there are also some things in Moodle ... which if on ... will degrade performance.  There should be a link to server performance checks.    Running autobackups?   Running cron frequently enough?

Time outs for php ... defaults for PHP usually have to be tweaked upwards ... or it wouldn't hurt to tweak upwards ... a little at a time.   Try NOT to guess.

Example: time for a script to run ... default is 30 seconds but if you've got some heavy processing going on then tweak that upwards ... 30 -> 60 ... etc. ... again, TRY NOT to guess.

Those sorts of things require some monitoring ... to begin with, simple top command during prime time might provide you some info and point you in a direction.   Server running any SWAP space?   What's using SWAP?  MariaDB?   'man top' for options to display of top info so you can see what's using SWAP.

Hope that gives you some info and a place to start. smile

'spirit of sharing', Ken

Average of ratings: Useful (1)
In reply to Franky Vallori

Re: Problems crashing on the server

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

How *exactly* do you define "crash"?

In reply to Howard Miller

Re: Problems crashing on the server

by Albert Ramsbottom -
I was going to ask the same question? What is a "Crash" and what errors are you getting in Apache? How often does this happen?


And yes it is easier to debug with a single server


Albert