mdl_logstore_standard_log 35 million rows

mdl_logstore_standard_log 35 million rows

by Danny Wahl -
Number of replies: 6

We've recently upgraded to 2.7.2+ from 2.6.X+ and have started experiencing occasional database connectivity issues.  In investigating I found that the new Standard log store has 'never delete logs' by default (guess I missed that one) and as a result that table has over 35,045,968 rows (at time of this writing).

Knowing that mdl_log used to cause some issues for other users when it would hit 7-9 million lines I wanted to investigate pruning it, but by setting a log limit.  I set it to 35 days but then I discovered that the earliest entry is September 19 (when we did the upgrade)- only 31 days ago.

Is the new log format really logging that much more info (mdl_log has 56,000 entries for 35 days)- and what are the performance implications?

Average of ratings: -
In reply to Danny Wahl

Re: mdl_logstore_standard_log 35 million rows

by Danny Wahl -
just for the sake of trying it I set the standard log to keep only 5 days of logs and executed schedule_task again- this time it took 250 seconds to run (compared to only a few the first time)

but there still seem to be 35M rows in the table.  will it not trim older data or something on a value change?
Average of ratings: -
In reply to Danny Wahl

Re: mdl_logstore_standard_log 35 million rows

by Danny Wahl -

I came back to work this morning and there's now 1.9 million rows in the table, so obviously the schedule_task command doesn't do what I think it does.

but still: 1.9 million entries for 5 days? wow!

Average of ratings: -
In reply to Danny Wahl

Re: mdl_logstore_standard_log 35 million rows

by James McLean -

What is the contents of the rows? While I know nothing about the new logging system, is it possible there is a process that's run away and is generating all the logs? Perhaps there is some kind of attack going on?

Average of ratings: -
In reply to James McLean

Re: mdl_logstore_standard_log 35 million rows

by Danny Wahl -

the content all seem to be legitimate- but not knowing what's supposed to be there I can't really speak to it one way or the other, it's a lot of "updated calendar event", "viewed quiz", etc... just much much more verbose than 2.6- which I guess is kind of the point.

Average of ratings: -
In reply to Danny Wahl

Re: mdl_logstore_standard_log 35 million rows

by sam marshall -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Bugs aside, I don't think the number of logged entries should have dramatically increased, as in general Moodle HQ have mainly just converted existing log calls to use the new system rather than adding new logging.

By the way, at the OU we will be setting our systems to use the legacy log table because we have a number of integrations with third-party systems that depend on it. (Obviously we'll turn standard logging off.) If you're having problems with the new log table, you could also consider switching to legacy logging for the time being?

Most likely we won't change to use the new log system until we have selected or implemented a log output plugin that doesn't use the database. There's actually a great opportunity for performance gain compared to 2.6 and previous versions, if you log somewhere that's quick to write to and doesn't limit scalability in the way database writes do, e.g. the local filesystem. I'm not sure if there are any high-performance log plugins available out-of-the-box though.

--sam

Average of ratings: -