We have been facing a technical emergency over the the last couple of days where users have not been able to take a mobile moodle test in offline mode and there is severe impact to the server causing '503 Service Unavailable' situations. Impact seems to be limited to the users who are taking the moodle mobile test (50 users). Other users who are accessing other activities remain unimpacted. The issue then also impacts our other instances on the same server.
* The Production Server server has 2 CPUs and 5 GB RAM. OS is Linux 3.12.18-clouder0 #1 SMP. These are not getting breached and usage is around 60%
* There are 4 instances of Moodle LMS on this server (Apache, PHP, MySql), each with its own server software and MySQL database. Most moodle instances are version 3.5. Our language of default are English. We also use Indian language packs where the project so requires. Of the 4 instances, 2 instances have very little traffic while the 3rd has traffic of approx 40 (maybe a concurrence of 10-15). If is the 4th one which is having technical issues.
* The impacted instance uses Moodle 3.5 (Build: 20180517) PHP Version 7.1.33. It was installed using download from moodle.org.
* We have conducted these moodle mobile tests earlier too across the locations, however always in the online mode.
* The impacted instance is Gram Shakti (GS) - with the largest user base. It has over 1000+ with around 500 active users. Almost 100% Users in the field use the following Moodle Apps - Moodle Classic, Moodle versions from Playstore from 2019 and 2020. Moodle Classic and Moodle 2019 have minor customisations limited to customer logo branding and colours. Courseware is accessed in various Indian Langauges and are focused towards Financial Inclusion. Server concurrency may be very low as almost 100% of them are accessing courseware through mobile phones, where they download the course scorms and go through them offline.
Issue origin on 8-Jul, 2020
* The issue occurred when we conducted a moodle mobile test using the offline access option. It looks like the mobile app interacting webservice calls are bombing due to which the server is getting overwhelmed.
* However, it is not overwhelming the resources as CPU and RAM are well within limits.
* From the error logs, the webservices are facing enormous amount of executions.
* Do check the findings are shared in the hosting service provider's trouble ticket response.
Simulation on 9-Jul
* We tried simulating the response with around 25-30 users across Indian states with similar experience. Moodle mobile quiz ONLINE went through well however, similar error logs got generated when we used moodle mobile quiz OFFLINE.
The interaction with the server hosting provider, their reading and analysis is provided in the attached file. Happy to provide other details as may be required for analysing this issue.
Do assist urgently.