Moodle 3.9 - reduce the size of mdl_log table

Moodle 3.9 - reduce the size of mdl_log table

by Verónica V. -
Number of replies: 19

Hello community, I am writing this time to ask you how to reduce the size of the mdl_log table. I have an environment (Moodle) that I have been updating for several years and this table has grown too much. This is a problem every time I have to update the version.

If someone knows something about this I appreciate your help.

Best regards.
Verónica.

Average of ratings: -
In reply to Verónica V.

Re: Moodle 3.9 - reduce the size of mdl_log table

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
As I recall, you can adjust how long you want Moodle to keep your logs. This is found in Administration, Plugins, Logging, Standard Log. The default is "Never delete logs." But if you change this to something like "90 days," the mdl_log table (i.e., mdl_logstore_standard_log) will not grow large. Of course, if you don't keep all the data, if you every want to run special statistics, your Moodle might not be able to do so.
In reply to Rick Jerz

Re: Moodle 3.9 - reduce the size of mdl_log table

by Verónica V. -

Hi Rick, thank you very much for your answer, but I would like to keep the records until 2017 and the rest delete it. But it won't let me, the maximum is 1000 days, can that be changed?

On the other hand, I activate it and what happens, it just takes care of deleting the records in the mdl_log table?

Again thank you very much.
Best regards.
Verónica.

In reply to Verónica V.

Re: Moodle 3.9 - reduce the size of mdl_log table

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
The actual maximum is "Never delete logs." So, the jump is from around 2.75 years to infinity. You cannot easily change this. You might be able to create a Tracker request that this setting be changed to a specific date, like 12/31/2017, or maybe to a table size, like 500GB. But the size of this table depends on how you and your students use your Moodle.

I should have told you to back up your Moodle before making this change. You probably keep regular backups, right?
In reply to Rick Jerz

Re: Moodle 3.9 - reduce the size of mdl_log table

by Verónica V. -
Hi Rick, thank you very much again. And yes I do perform backups periodically. Please could you tell me what it would be like to create a Tracker request with the date, is it done from the code?
Greetings.
Verónica.
In reply to Verónica V.

Re: Moodle 3.9 - reduce the size of mdl_log table

by Verónica V. -
Sorry Rick, I mixed up the text. Firstly, how would it be to create a Tracker request, and secondly, can this change be achieved by modifying the code?
Thanks again.
Verónica.
In reply to Verónica V.

Re: Moodle 3.9 - reduce the size of mdl_log table

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
You should see Tracker in the main menu above. I forget the rules, but I think you have to set up an account (free) and maybe do something else to verify that you are a human. Then, once you are in Tracker, at the top, there is a Create button. From there, you pick "improvement."

Tracker is not hard to use, and anyone can post. Tracker is the best system that I have ever encountered dealing with software bugs and improvements. The key is to be thorough. A suggestion like "add 2017 to logs" will not cut it. Spend a little time in Tracker to get a sense of what it does.
In reply to Rick Jerz

Re: Moodle 3.9 - reduce the size of mdl_log table

by Ken Task -
Picture of Particularly helpful Moodlers
@Rick ... if OP has phpmyadmin or similar tool, OP could do as desired, right?

timecreated field is an epoch time stamp.
Epoch timestamp: 1483272000
is
Date and time (GMT): Sunday, January 1, 2017 12:00:00 PM

If one queried that table for timecreated less than 1483272000 could remove those records.
If one queried that table for timecreated greater than 1483272000 those you'd want to keep.

Have I done it?   No.  K12 entities usually don't care about the 'who done it' table after the close of an academic year and grades have been submitted, so my job is easy ... truncate the table.

'SoS', Ken

In reply to Ken Task

Re: Moodle 3.9 - reduce the size of mdl_log table

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers
Actually Ken, I thought about directly editing that mdl_logstore table, but I wasn't sure if any other tables are directly related to it (i.e., there are dependent relationships). I seldom directly edit Moodle tables, so I am uncomfortable telling others to do it. Leon made this post a while ago, and he would know better than I.
In reply to Rick Jerz

Re: Moodle 3.9 - reduce the size of mdl_log table

by Verónica V. -
Hi Rick and Ken, that doubt is the one I have, generate inconsistency in the tables by some relationship between them.
Anyway, thank you very much for all the suggestions, I will take them for testing.
Regards.
Verónica.
In reply to Verónica V.

Re: Moodle 3.9 - reduce the size of mdl_log table

by Ken Task -
Picture of Particularly helpful Moodlers
In reading the post Rick linked to:

"The risk of a TRUNCATE TABLE is to your records-keeping and data retention requirements."

@Veronica ... what are those requirements for you site?

Have any hidden courses that are not active?

I am able to do more on a K12 site due to usage and requirements.
Quick summary:

After classes are over for an academic session and grades have been posted in their official grade book (not moodle's), teachers reset their course ... and make sure the option to remove the student role and the students work in the reset.
I check using the who done it that all teachers used reset on their courses.   If they didn't, I do it. :|

Reset removes student accounts, their work, their history/records in that course.
Since site uses LDAP, they eventually remove graduating students from their LDAP - I don't have to do an LDAP sync as the student accounts will expire by themselves ... lack of use.

Check things using moosh commands and do cull orphaned files.

Like many things in moodle, there is a beginning ... and if there were just a few more things added to allow finer tuning or selection, would be great.   But, alas ... not yet ... so that's where things like direct DB manipulation and command line utilities like moosh come into play - using those very carefully, of course.

And, if one does such things, a backup is required for safety net. smile

'SoS', Ken

In reply to Ken Task

Re: Moodle 3.9 - reduce the size of mdl_log table

by Gregor McNish -
Picture of Particularly helpful Moodlers
we do a yearly site backup (going back now to 2016)-- those backups have never delete logs set (since they don't get too many new entries).
Our production site only keeps logs for a year.
So this way, we can always look back to the year something happened, but our production logs don't grow to ridiculous size, which is helpful for our IT backup procedures..
Average of ratings: Useful (1)
In reply to Gregor McNish

Re: Moodle 3.9 - reduce the size of mdl_log table

by Verónica V. -

Thank you very much Gregor for your contribution!

Regards.
Verónica.

In reply to Rick Jerz

Re: Moodle 3.9 - reduce the size of mdl_log table

by Verónica V. -

Hi Rick, I changed the "logstore_standard | loglifetime" option to 1000 days, 2 days have passed and the table has not reduced its size. Do I have to configure something in particular about the cron? Because it works correctly.

What could have happened? or what would I be missing to configure?

I wait your answer.
From already thank you very much.
Greetings.
Verónica.

In reply to Verónica V.

Re: Moodle 3.9 - reduce the size of mdl_log table

by Leon Stringer -
Picture of Core developers Picture of Particularly helpful Moodlers

... 2 days have passed and the table has not reduced its size.

How are you checking this? The scheduled task that removes old logs only runs for 5 minutes every day. If there are many old logs then it can take many days to reduce this to 1000 days.

What database software are you using? For MySQL or MariaDB you can see the date of the oldest log entry with the SQL query SELECT FROM_UNIXTIME(MIN(timecreated)) FROM mdl_logstore_standard_log, for example:

MariaDB [moodle]> SELECT FROM_UNIXTIME(MIN(timecreated)) FROM mdl_logstore_standard_log;
+---------------------------------+
| FROM_UNIXTIME(MIN(timecreated)) |
+---------------------------------+
| 2019-12-11 13:16:17             |
+---------------------------------+

This should change to a more recent date every day if the task is working.

I changed the "logstore_standard | loglifetime" option to 1000 days

This setting specifically affects the newer mdl_logstore_standard_log table (introduced in Moodle 2.7). If it's the old mdl_log table you want to manage you must change the loglifetime setting for the legacy log store.

In reply to Leon Stringer

Re: Moodle 3.9 - reduce the size of mdl_log table

by Verónica V. -

Hello Leon, thank you very, very much for your reply. I'll answer you and detail the situation:

I have two Moodle:

  • Moodle 1 (version 3.9.11): it has been in use since version 2.4.1 (2014 or so) with several updates in between until December 2023 and with ‘logstore_standard | loglifetime’ in never deleted.

MariaDB [ingreso]> SELECT FROM_UNIXTIME(MIN(timecreated)) FROM mdl_logstore_standard_log;  
+---------------------------------+
| FROM_UNIXTIME(MIN(timecreated)) |
+---------------------------------+
| 2019-06-12 18:59:06             |
+---------------------------------+
1 row in set (0.022 sec)

MariaDB [ingreso]> select count(*) from mdl_logstore_standard_log;  
+----------+
| count(*) |
+----------+
|  2469142 |
+----------+
1 row in set (3.100 sec)

rw-rw---- 1 mysql mysql 494927872 ene  2 17:20 mdl_logstore_standard_log.ibd

  • Moodle 2 (version 4.1.7): is a copy of Moodle 1 but updated to this version and is the one currently in use. On Wednesday 10 April ‘logstore_standard | loglifetime’ was changed to 1000 days.

MariaDB [ingreso]> SELECT FROM_UNIXTIME(MIN(timecreated)) FROM mdl_logstore_standard_log;
+---------------------------------+
| FROM_UNIXTIME(MIN(timecreated)) |
+---------------------------------+
| 2021-07-30 04:57:47             |
+---------------------------------+
1 row in set (0,001 sec)

MariaDB [ingreso]> select count(*) from  mdl_logstore_standard_log;
+----------+
| count(*) |
+----------+
|  1598573 |
+----------+
1 row in set (9,762 sec)

-rw-rw---- 1 mysql mysql 692060160 abr 25 18:45 mdl_logstore_standard_log.ibd

As far as I can see, the date is fine, the change made in ‘logstore_standard | loglifetime’ is working, but what about the size? Why has it grown so much despite having a deletion in between?

Best regards.
Verónica.

In reply to Verónica V.

Re: Moodle 3.9 - reduce the size of mdl_log table

by Leon Stringer -
Picture of Core developers Picture of Particularly helpful Moodlers

With database software typically if you add 10 rows then delete 10 rows the storage still keeps the original space and reuses this later if needed.

For MariaDB you can recover this space on demand with OPTIMIZE TABLE. For a production site this can cause performance issues while it's running so you would need to plan a maintenance window. So you would run:

MariaDB [ingreso]> OPTIMIZE TABLE mdl_logstore_standard_log;

I would suggest placing your site in maintenance mode while this running. I can't say how long it will take as it depends on your server's performance (maybe 10 minutes?).

Average of ratings: Useful (1)
In reply to Verónica V.

Re: Moodle 3.9 - reduce the size of mdl_log table

by Ken Task -
Picture of Particularly helpful Moodlers

Suggest installing MySQLtuner.pl - a pearl script - and run it with superuser (root for DB server) credentials.

https://github.com/major/MySQLTuner-perl

It will look at DB stats you don' t see in moodle and make recommendations for fine tuning your DB server.   One of the items it detects is tables that are in need of optimizing and even shows the command to do so.

Moodle is backended by DB and the better your DB config, the better for moodle.   I run tuner about once a week on busy sites.   Not unusual for a table to be in need of optimizing if teachers have been doing clean ups of courses, etc.

'SoS', Ken

In reply to Verónica V.

Re: Moodle 3.9 - reduce the size of mdl_log table

by Mark Sharp -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers
There is a tracker item (MDL-49936) which proposed adding annual increments (2, 3). Sounds like you want 7 years? That's a lot!

You could manually change the database setting, but you'd have to be aware that resaving the log settings in Moodle adminstration will reset that.

You could write a database procedure to delete records after a set period.

Or you could write your own plugin that does deletion independent of the core logstore plugin, then you can set whatever retention you want. I do this so I can have different retention periods for different sorts of logs.
In reply to Mark Sharp

Re: Moodle 3.9 - reduce the size of mdl_log table

by Verónica V. -

Thank you very much Mark for your reply!

How do you write your own plugin?

I'm going to look in detail at the tracker item, but it occurs to me to modify the code to add 2500 days in the dropdown list, for example What do you think?

Regards.
Verónica.