AWS 99.9% cpu

AWS 99.9% cpu

by Andrew M -
Number of replies: 4

Hi all,

I am running a small Moodle site via AWS. Occasionally the server shoots up to 99.9% cpu, hangs for a couple of hours and crashes all the processes. Moodle then attempts to restart without success, leaving the website inaccessible. 

I am running Moodle 4.0.4 on a docker container, with Nginx Proxy Manager for SSL certificates. There is nothing relevant on Moodle or docker logs apart from the server restarting itself after the crash.


The following shows CPU usage over the last 2 months.

Average of ratings: -
In reply to Andrew M

Re: AWS 99.9% cpu

by Brett Dalton -
Picture of Moodle HQ Picture of Particularly helpful Moodlers
I would suggest looking at your database slow transaction logs and connection pools. Web server request logs are also worth looking at. There could be an integrated service hammering the API, or even a particular user with problematic data.
Average of ratings: -
In reply to Andrew M

Re: AWS 99.9% cpu

by heli g -

Just on the off chance this is relevant here, do you have the Assignment's Annotate PDF enabled? What document converter do you use? We use Unoconv and have to monitor and restart it regularly because it takes down the server.

Average of ratings: -
In reply to heli g

Re: AWS 99.9% cpu

by seaghan moriarty -
Thank you Heli Mamatheka

My CPU was high each night (The task was apparently not obvious in 'scheduled tasks')
I confirmed in 'system paths' the path to Ghostscript and unoconv

When the processes subsequently ran, yes the CPU was high, for about 4 hours
but no server crash AND after this process completed - no more high CPU

HTH anyone with similar problems and thanks again Heli!
If Unoconv is still problematic, I have one moodle using LibreLambda (outsources the processing to an AWS process) which has worked perfectly for a couple of years now. Read: https://www.elearningworld.org/moodle-pdf-conversion-plugins-good-bad-ugly/ for good perspective.
Average of ratings: Useful (1)
In reply to seaghan moriarty

Re: AWS 99.9% cpu

by heli g -
I am so glad to be of assistance! Unfortunately I am not on AWS and stuck with unoconv for now.
My workaround:
  • Disable the scheduled task Prepare submissions for annotation \assignfeedback_editpdf\task\convert_submissions (the first one on the list).
  • Disable Annotate PDF feedback in Assignment defaults.
  • Obsessively monitor unoconv status, top and on munin: mysql connections (Memory, CPU usage, and mysql threads connected, respectively) and restart regularly.
Now Assignment documents are only converted on viewing the assignment and the Teacher has to choose to have the feature enabled (last I tested, the scheduled task attempts to convert everything regardless of whether Annotate PDF is enabled in the assignment or not - so it has to be disabled).
Average of ratings: Useful (1)