Moodle creating Excessive I/Os

Moodle creating Excessive I/Os

by Greg Merriman -
Number of replies: 3

I hope that somebody may be able to help me as I have come up against a brick wall with my hosting support.

My hosting company is SiteGround and I have cloud hosting with 2 CPU Cores, CentOS, 4GB RAM, 50GB SSD, cPanel, Cloud Storage. I am hosting our main website with a link to our moodle installation

I am running a relatively small moodle installation and have been experiencing 503 errors across all sites on my server. I have contacted support and they have provided me with some information but not enough to troubleshoot accurately. Something is creating peaks which brings everything crashing down. Reloading the page will usually resolve the issue within about 30 seconds.

The information provided to me by SiteGround support is:

I have reviewed the server logs and I found the following error message there: 

Code:
[Mon Feb 18 05:27:40.320660 2019] [:error] [pid 29057] Execute of /home/ntcie/public_html/portal/index.php stopped because of load 128.07
[Mon Feb 18 05:27:40.659089 2019] [:error] [pid 30928] Execute of /home/ntcie/public_html/portal/my/index.php stopped because of load 128.07
[Mon Feb 18 05:27:54.354306 2019] [:error] [pid 30968] Execute of /home/ntcie/public_html/portal/index.php stopped because of load 68.53
[Mon Feb 18 05:27:55.760910 2019] [:error] [pid 30928] Execute of /home/ntcie/public_html/portal/my/index.php stopped because of load 68.53
[Mon Feb 18 05:27:55.761096 2019] [:error] [pid 30928] Execute of /home/ntcie/public_html/portal/index.php stopped because of load 68.53
[Mon Feb 18 05:27:57.530200 2019] [:error] [pid 30482] Execute of /home/ntcie/public_html/portal/my/index.php stopped because of load 37.15
[Mon Feb 18 05:27:57.530396 2019] [:error] [pid 30482] Execute of /home/ntcie/public_html/portal/index.php stopped because of load 37.15

That means your Cloud Server gets overloaded because of too many IOPs which actually are read and write requests. I can confirm that your Cloud Server is stable but the normal IOPs usage is around 5000 IOPs. I am providing you with the IOPs usage for your Cloud Server below:
Code:
2019-02-18 05:27:34.625002-06 | IOPs-Read: 56619 | IOPs-Write: 2302
2019-02-18 04:54:31.197769-06 | IOPs-Read: 23527 | IOPs-Write: 1813
2019-02-18 04:53:34.832969-06 | IOPs-Read: 79953 | IOPs-Write: 860
2019-02-18 04:52:33.579889-06 | IOPs-Read: 85229 | IOPs-Write: 412
2019-02-18 04:51:30.372856-06 | IOPs-Read: 12152 | IOPs-Write: 1799
2019-02-18 04:05:33.503622-06 | IOPs-Read: 21450 | IOPs-Write: 1983

Unfortunately, upgrading your Cloud Server's CPU cores and/or RAM wont resolve the issue.


Could anybody point me in the right direction to resolve this?


Average of ratings: -
In reply to Greg Merriman

Re: Moodle creating Excessive I/Os

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

Difficult to imagine:
- A small Moodle installation is too much for a Linux server with 2 vCPU, 4GB RAM and SSD? How many users were engaged as this happened? Doing what?

- The overloading happens in index.php and my/index.php? What are the line numbers? The debug report seems to be something Siteground specific. What is the output of the Moodle debug https://docs.moodle.org/en/Debugging ?

- What are IOPs?

- Loads of 128? On a dual core machine? Impossible, unless you measure the load on the virualization machine and it has 256 cores! (A dual core is 100% busy with a load of 2 in rough terms.)

Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Moodle creating Excessive I/Os

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Remember that Load Average is the number of processes in the queue. It's well within possibility to have a load average of 128+ on a box with two nodes if you forkbomb it.

IOPs is the IO operations Per Second. It's a standard metric for measuring disk usage.

Average of ratings: -
In reply to Greg Merriman

Re: Moodle creating Excessive I/Os

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

You need to find out what is actually causing the IOPs.

I would advise checking:

  • that your memory is not damaged
  • your SWAP usage
  • what your DB server is doing (check the processlist on MySQL for example)

But really you need to find out what your server is doing. Moodle is not necessarily the cause. It could be a symptom. Someone could have hacked your server and be using it for mining bitcoin, sending e-mails, or something completely unrelated.

Andrew

Average of ratings: Useful (1)