Improving the performance of the moodle database

Improving the performance of the moodle database

by Juan Luis Maestre -
Number of replies: 8

Hello, I am improving the performance of my moodle site database (~ 20k users, ~ 3k courses, ...): moodle 3.7.3 with mysql-5.6 database.

I've verified that there are too many writes in the mdl_logstore_standard_log table. There are some records that I don't want to store in that table (web service calls, ...)

Is there any way to filter what is written in that table? by type of record or something similar...

Thanks

Average of ratings: -
In reply to Juan Luis Maestre

Re: Improving the performance of the moodle database

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

"I've verified that there are too many writes in the mdl_logstore_standard_log table"

How? More details please?

In reply to Howard Miller

Re: Improving the performance of the moodle database

by Juan Luis Maestre -
Yes of course. We are currently migrating data from our old platform to moodle and we have a process running 3 times a day that calls moodle web services many times. I monitor the database IOPS using AWS Cloudwatch and RDS Performance Insights and during this process the number of database write operations is greatly increased.

I have observed that part of the contribution are the WriteOps in the mdl_logstore_standard_log table.

The information that is stored in this table, with respect to WebService Calls for example, does not provide any information that is useful to us.

In order to reduce these WriteOps and improve the DB performance we want to limit what type of information is written in that table. The objective is to reduce the WriteOps, not to erase the information once it has been written.

Thanks
In reply to Juan Luis Maestre

Re: Improving the performance of the moodle database

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
I have never heard of Moodle logging being a performance issue, are you sure it is a problem?
In reply to Juan Luis Maestre

Re: Improving the performance of the moodle database

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Like Marcus I am not convinced.

However, you can write your own logstore plugin.
Average of ratings: Useful (1)
In reply to Howard Miller

Re: Improving the performance of the moodle database

by Juan Luis Maestre -
Ok, I'll check. You can see in the attached figure the effect that I want to mitigate.

In https://docs.moodle.org/dev/Event_2: "The logging system is an event observer, listening to all events...." Can this be changed? Or should I write my own logstore plugin to only listen to certain events?


Attachment Capture1.png
In reply to Juan Luis Maestre

Re: Improving the performance of the moodle database

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Ok - but this is entirely without context. What is the rest of the database activity contributing to load? How *big* is your log table? Do you have years and years of data? Do you need it? Have you checked the schema? Is there a missing index? Is your database properly (or at all) tuned? and so on.
In reply to Howard Miller

Re: Improving the performance of the moodle database

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
To follow on from Howards comment, log tables can get big quickly as you might guess form the amount of data it records. If this is the issue, and I have no idea if it is, then truncating the log might be a good idea.
Average of ratings: Useful (1)
In reply to Marcus Green

Re: Improving the performance of the moodle database

by Juan Luis Maestre -
Marcus, Howard thank you very much for your help and comments

First of all tell you that the platform works really well and fast. Maybe everything is due to our obsession with optimizing the performance of our moodle database. Maybe performance is not the most appropriate term, and excuse me if I explained it wrong:

In our case, by using AWS we are limited to the number of IOPS we provision to the database. To really be a cost-effective solution we need to reduce the number of IOPS to the minimum possible, but maintaining the latency and response time at acceptable values.

In the past we observed that "at rest" (without activity or users logged in moodle) moodle required around 130-140 IOPS (around 95% were writing operations). The cron is a like a vampire (running every minute), but we really got the improvement using a lock_factory based on memcached, which allowed us to establish our baseline at 3-4 IOPS, something more acceptable.

Working in that sense we are trying to reduce the database load to the minimum possible, offloading to the database of writes that do not provide us with useful information. In this task we find an increase in specific moments of writing operations in the mdl_logstore_standard_log table that we want to reduce.

In any case, I explained myself badly, but the solution goes through as you propose to use a custom logstore plugin. Our development team will work in that direction.

Again, thanks for your help.