SQL Performance on lesson course restoration

SQL Performance on lesson course restoration

by Luis Alberto Asturias Núñez -
Number of replies: 6
Hello all.

We are managing a moodle with mdl_lesson_branch table composed by 12685665 rows.

The system is working fine, but when we import a course or copy courses with lesson pages, performance is poor and it takes more than 30 minutes.

We reviewed the code and the cause is the execution of below code from mod/lesson/backup/moodle2/restore_lesson_stepslib.php:
// Replay the upgrade step 2015032700.
// Delete any orphaned lesson_branch record.
if ($DB->get_dbfamily() === 'mysql') {
$sql = "DELETE {lesson_branch}
FROM {lesson_branch}
LEFT JOIN {lesson_pages}
ON {lesson_branch}.pageid = {lesson_pages}.id
WHERE {lesson_pages}.id IS NULL";
} else {
$sql = "DELETE FROM {lesson_branch}
WHERE NOT EXISTS (
SELECT 'x' FROM {lesson_pages}
WHERE {lesson_branch}.pageid = {lesson_pages}.id)";
}

$DB->execute($sql);

The sentence takes about 60 seconds in our database.

Does anyone know if this SQL request can be safely improved or cleaned from the code.
We are considering deleting this query and planning a nightly execution to avoid any problems with orphaned lesson branch records.

Many thanks in advance.

Average of ratings: Useful (2)
In reply to Luis Alberto Asturias Núñez

Re: SQL Performance on lesson course restoration

by Fabio Serenelli -
Hello,
My understanding is that the purpose of the code pasted is to find and remove orphan outlines which are part of the Moodle Lesson activities.
Please, does anybody know if by commenting / hiding this snippet of code we could unwillingly block other processes, beyond Moodle Lessons?
Thanks and regards,
fabio
Average of ratings: Useful (1)
In reply to Luis Alberto Asturias Núñez

Re: SQL Performance on lesson course restoration

by Andre Farrugia -

Hi, Luis 

We are interested in understanding this issue. We currently experience long backup/restore time only for courses based on long/highly structured Moodle Lessons. The duration of the operation does not depend on the weight of the course or single activity (a few MB!) but their complexity.

 I hope we can get some insight from the community as this issue is impacting our efficiency in our day-to-day operations!


In reply to Andre Farrugia

backup/restore time: from 30min to 30sec...

by Fabio Serenelli -
The question is... if by hiding this code backup/restore time is reduced from 30min to 30sec... Why is that piece of code there? Maybe it's needed for other stuff.
I guess it would it be more efficient to schedule it as a nightly task as Luis has suggested.
In reply to Luis Alberto Asturias Núñez

Re: SQL Performance on lesson course restoration

by Fabio Serenelli -
Hello,
the impact of this issue increased significantly during our tests for upgrading Moodle from 3.9 to 3.11.
This is unfortunately a blocker, as at the moment, until we don't know if by removing this code we affect other processes, we can't upgrade the LMS.
Thanks for any suggestion.
Fabio
In reply to Luis Alberto Asturias Núñez

Re: SQL Performance on lesson course restoration

by Joseph Rézeau -
Picture of Core developers Picture of Plugin developers Picture of Testers Picture of Translators

Hello Lujis,

I understand that you are experiencing performance issues during course import or copying in Moodle, specifically related to the execution of an SQL query that deletes orphaned `lesson_branch` records. You have shared the code snippet responsible for this process from the `restore_lesson_stepslib.php` file.

To address your concern, you are considering removing the SQL query and instead planning to execute it on a nightly basis to avoid any problems with orphaned `lesson_branch` records. Before proceeding with such a modification, it's essential to thoroughly evaluate the potential impact and explore possible improvements to the SQL query.

Upon reviewing the code snippet you shared, it appears that the query is designed to delete orphaned `lesson_branch` records by joining the `lesson_branch` and `lesson_pages` tables. This query checks for records in `lesson_branch` where the corresponding `pageid` does not exist in `lesson_pages`.

To improve the performance of this query, you can consider the following suggestions:

1. Indexing: Ensure that appropriate indexes are present on the relevant columns (`pageid` in `lesson_branch` and `id` in `lesson_pages`). Indexing can significantly improve the performance of join operations.

2. Query Optimization: Depending on your database engine, you can analyze the query execution plan and identify any areas that can be optimized. Tools like EXPLAIN (in MySQL) or equivalent features in other database systems can provide insights into query execution.

3. Batch Processing: Instead of running the query for each individual record, you can explore batching the deletion process. By deleting orphaned `lesson_branch` records in smaller chunks or batches, you can potentially reduce the impact on performance.

4. Data Cleanup Strategy: If you still face performance issues even after optimizing the query, you can consider alternative data cleanup strategies. For example, you could schedule periodic cleanup tasks during low-traffic periods to manage orphaned records more efficiently.

However, please exercise caution when modifying code in the Moodle core. Making changes without a thorough understanding of the implications can lead to unforeseen issues or compatibility problems with future Moodle updates.

I would recommend reaching out to the Moodle community or forum for further assistance. Developers and experienced Moodle users can provide more specific insights and advice based on their expertise and knowledge of the Moodle codebase.

Best regards,

Chat GPT

In reply to Joseph Rézeau

Re: SQL Performance on lesson course restoration

by Luis Alberto Asturias Núñez -
Many thanks for your comment.
Data cleanup strategy is the solution we have decided, to avoid any problem related to orphaned lesson_branch records.
By now, no orphaned records were detected, but it will reduce the risk.