Moodle Performance Issues (Load Balancer / Cluster) (Willing to Pay for Help)

Moodle Performance Issues (Load Balancer / Cluster) (Willing to Pay for Help)

by Amber Lynn -
Number of replies: 15

So first I'd like to describe the setup we have to make sure you guys can give us some guidance possibly. I have inherited a Moodle installation that was originally run on a single server with a local data cache on the same box.   There are about 100k accounts in the system and we add between 100 and 200 users per week. This client has a education program they sale which teaches people how to pass the USPS exam and get hired for the postal service. So here is the new setup on the system...


Two Identical Machines

QuadCore E5620 @ 2.4Ghz (per core)

24 Gigs of Ram

500 Gig Local Storage

Ubuntu 14.04 Trusty

Server version: Apache/2.4.7 (Ubuntu)

MySQL Galera (MariaDB Cluster with 3 Nodes)


Amazon AWS Barracude Load Balancer (Content Caching Enabled)

ALL HTTP and MySQL traffic are sent through this balancer as we have 3 nodes on the MySQL not just the two servers which handle Moodle HTTP.

So as of right now all traffic is routed to the load balancer, it then determines which server node is least busy and routes that student to the appropriate node to work.  Also the Moodle install are using the load balancer to access MySQL database.

The two largest issues are are having are: 

1) Moodle generates around 50,000 of these errors a day: "mysql connection unexpectedly terminated"

2) Quiz time loads very slowly because questions with images require Moodle to make DB request and generate image every single request and are not properly cached (or so it seems)

As far as we can tell the Moodle cache store is working as there are around 500MB of cache data generated on both servers and resetting it will regenerate the data.  However we are using the default application cache which uses the moodle application data area.  I did find something interesting in running the cache tester which seems to show alot of "misses" but I'm not entirely sure... see attached picture.

Overall the system loads are very very low in terms of resources and usage.  The servers should be able to handle 4 times the current traffic load especially if caching is working properly.

Any Help / Pointers Welcome!


Attachment 2017-02-15_10-53-21.jpg
Attachment 2017-02-15_10-54-18.jpg
Average of ratings: Useful (1)
In reply to Amber Lynn

Re: Moodle Performance Issues (Load Balancer / Cluster) (Willing to Pay for Help)

by Luis de Vasconcelos -

I presume the "Two Identical Machines" are your two web nodes (both running Apache). Is that correct? If so, where is your Moodledata folder?

Are you using some kind of NAS for the Moodledata folder?

Average of ratings: -
In reply to Luis de Vasconcelos

Re: Moodle Performance Issues (Load Balancer / Cluster) (Willing to Pay for Help)

by Amber Lynn -

  They are dedicated machines that run Web and MySQL instances on them.  The services are load balanced through a barracuda in the AWS.  Moodle data is stored on the local machine that runs the webserver.

Average of ratings: Useful (1)
In reply to Amber Lynn

Re: Moodle Performance Issues (Load Balancer / Cluster) (Willing to Pay for Help)

by Luis de Vasconcelos -

Sorry, I'm struggling to get the picture! Is one of the machines running MySQL and the other machine running the Web server (Apache I presume)?

And is the Moodledata on that Web server node (i.e. shared with Apache0?

Average of ratings: -
In reply to Amber Lynn

Re: Moodle Performance Issues (Load Balancer / Cluster) (Willing to Pay for Help)

by Amber Lynn -

  Each Server runs MySQL and Apache.  The MySQL instances are all clustered together via MariaDB Galera 3.  The Apache instances just use the local storage for websites/data.  All DB request are routed through the Load Balancer for HA, and just to be clear, we have tried to route them just locally as well and the speed is the same.

  After going through some more things it seems the real issue is when the quiz is loading. Once you click "Start Quiz" and it loads the next page.  It seems the "Image Attachments" that are used for each question each take 2 to 3 seconds to load.  Since its a 30+ question quiz... it takes along time to load the page.

 We have traced it back into moodle code and it seems that Question Attachments are run through the "PluginFile" php script which, I'm assuming validates the users access and then renders the image back to the client.  This seems like a huge waste as it does not seem to use any cache and does a full DB/Check image render.


Average of ratings: -
In reply to Amber Lynn

Re: Moodle Performance Issues (Load Balancer / Cluster) (Willing to Pay for Help)

by Christos Savva -
Maybe I am wrong but moodledata should be shared between the Web Server nodes otherwise you will have unexpected problems.
Average of ratings: -
In reply to Amber Lynn

Re: Moodle Performance Issues (Load Balancer / Cluster) (Willing to Pay for Help)

by Luis de Vasconcelos -

Your screenshots suggest that you aren't using Memcache(d) or MongoDB for your session management. Considering that you have multiple server nodes in your setup, how have you setup your user session management?

Are you storing the user session information in your Moodle database?
(In other words, is the "Use database for session information" option under Site administration / Server / Session handling on or off?)


Average of ratings: -
In reply to Luis de Vasconcelos

Re: Moodle Performance Issues (Load Balancer / Cluster) (Willing to Pay for Help)

by Amber Lynn -

  No, they are just using the default moodle session cache which is in the local application directory.  A centralized cache wasn't necessary because the load balancer will route the same user to the same node for the duration of the visit.  Its not round robin.  As far as I know it is not being stored in the database as we did not want the overhead there.

Average of ratings: Useful (1)
In reply to Amber Lynn

Re: Moodle Performance Issues (Load Balancer / Cluster) (Willing to Pay for Help)

by Luis de Vasconcelos -

One more thing....

Look at the settings in your Moodle config.php file:

  1. What is the $CFG->cachedir setting pointing to? Is it a folder on shared storage?
  2. And what do the $CFG->tempdir and $CFG->localcachedir settings point to? Are they pointing to local folders defined on each of your server nodes?
  3. How have the $CFG->session_handler_class and $CFG->session_file_save_path settings been configured?


Average of ratings: -
In reply to Luis de Vasconcelos

Re: Moodle Performance Issues (Load Balancer / Cluster) (Willing to Pay for Help)

by Amber Lynn -

I have removed a few lines but here is what is in the config...  all of the settings your asking about are not used and are using whatever the defaults are in the local machine application directory from what I have read.  Given that it is all stored locally I would not think its causing the issue since the disk speed should be much faster.


<?php  /// Moodle Configuration File

unset($CFG);  // Ignore this line

global $CFG;  // This is necessary here for PHPUnit execution

$CFG = new stdClass();


$CFG->dbtype    = 'mysql';

$CFG->dbname    = 'moodle';

$CFG->dbpersist =  true;

$CFG->prefix    = 'mdl_';


$CFG->wwwroot   = 'http://'.$_SERVER['HTTP_HOST'].'/exam';

$CFG->dirroot   = dirname(__FILE__);

$CFG->dataroot  = $CFG->dirroot.'/../../jssmoodle_data';

$CFG->admin     = 'admin';


$CFG->directorypermissions = 00777;  // try 02777 on a server in Safe Mode


Average of ratings: -
In reply to Amber Lynn

Re: Moodle Performance Issues (Load Balancer / Cluster) (Willing to Pay for Help)

by Luis de Vasconcelos -

Interesting... What version of Moodle are you running?

And what do you mean by "Given that it is all stored locally..."? What is stored locally?

Average of ratings: -
In reply to Amber Lynn

Re: Moodle Performance Issues (Load Balancer / Cluster) (Willing to Pay for Help)

by Amber Lynn -

What I mean is that the "dataroot" for Moodle is stored locally and the default caching features should be running at a maximum speed.

The files are stored on the local machine on SSD 6Gbps drives. So speed shouldn't be an issue in terms of accessing it...  The database is also stored on local SSD and the overall system loads are still below 1% average and 10% during peak times.



Average of ratings: -
In reply to Amber Lynn

Re: Moodle Performance Issues (Load Balancer / Cluster) (Willing to Pay for Help)

by Amber Lynn -

  It might be worth noting that images loaded for questions that come from the "pluginfile.php" take an extremely long time to load...  especially if there are 20 plus images for a quiz. View the below link for example of the file... any suggestions or ideas for this? Our only idea is to move the images to an external web directory and put direct links to them as it seems to cause alot of database/webserver load having to serve these files to every person on each quiz... and I'm not sure why the system doesn't just cache them and serve them instantly.


URL From Pluginfile

/exam/pluginfile.php/56159/question/questiontext/740457/3/29632/Form_13_Mail_Stamp_Requisition.jpg


Average of ratings: Useful (1)
In reply to Amber Lynn

Re: Moodle Performance Issues (Load Balancer / Cluster) (Willing to Pay for Help)

by Amber Lynn -

We have OPCache installed according to: php -v


xxx@cdn66-171-xxx:/home/admin# php -v

PHP 5.6.30-1+deb.sury.org~trusty+1 (cli)

Copyright (c) 1997-2016 The PHP Group

Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies

    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies


Average of ratings: -
In reply to Amber Lynn

Re: Moodle Performance Issues (Load Balancer / Cluster) (Willing to Pay for Help)

by Jorge Dávila López -

Amber,

I guess you have an issue with MySQL. Appears there are a large quantity of records for retrieving and the query can not obtain them before the time configured for the execution of the sentence expires.

If you can connect to the database using an SQL Client (MySQL Workbench) execute some sql sentences and see what is happeninng ...

About the caching test, "misses" are the files which does not are in the cache and the system must retrieves it from the source (the physical location of the files).

Jorge.

Average of ratings: Useful (1)