[Sharing Cart] Restore not releasing memory it consumed

[Sharing Cart] Restore not releasing memory it consumed

by Visvanath Ratnaweera -
Number of replies: 9
Picture of Particularly helpful Moodlers Picture of Translators

There was a second incident today noon. The DBMS committed 116 GB RAM, which it does not have, so swapped heavily. The load average went to 47 in a (physical) machine with 24 CPU threads. Stated just past 11:30 was finished by 12:00. The monitoring graphs look like somebody has restarted the server (haven't restarted even the DB server).

I was ready this time, Show origin of SQL calls was set to Show full stack trace and the slow queries were written to a file. The part between 11:30 and 12:00 is attached.

You'll see that three databases are involved, all are Moodle DBs. The first one, xyzfmoodle, is the suspected one. I have logs of a teacher who was transferring quizzed through the sharing cart. She has been doing it since 11:19 dozens of times. According to Moodle logs none has taken more than 1 second each. So could be a red herring.

More likely are these tasks running now:

and the Ad hoc tasts show:

How can I find out where they come from and why still unfinished although the DB was released? Or, had the DB dropped everything and restarted?

Notice that the xyzfmoodle is the database once crashed with 3 million duplicate questions. After much cleaning it has close to 600'000 question instances now.

Average of ratings: -
In reply to Visvanath Ratnaweera

Hunting slow queries in the quiz sub-system, incident 2

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Nothing int that slow log is relevant to quiz. It us fully of queries that should be lightening fast (e.g. get a course by id) that are very slow - so that is a sympton of the DB already being very overloaded, not the cauase.

If there are mulitple restore tasks currently running, then that would put a lot of load on the database.
Average of ratings: -
In reply to Tim Hunt

Hunting slow queries in the quiz sub-system, incident 2

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Tim

You wrote:
> It us fully of queries that should be lightening fast (e.g. get a course by id) that are very slow -

Thanks for that hint. Yes, I could double check by running those queries manually. Sorry for pointing fingers to the Quiz end, that is the place our site was exposed once.

> so that is a symptom of the DB already being very overloaded, not the cause.

Well, the sessions have not started yet, it is still the New Year vacation. My monitoring confirms low usage. I should look in to corrupt data. 

> If there are multiple restore tasks currently running, then that would put a lot of load on the database.

That could be. How about table locks hampering each other?

In fact, I had a similar thought. The activity copying across the courses using sharing cart, that happened between 11:14 and 11:28. There were some 30 of them. Is it possible the task system put them in a queue and then run all at once?

Either way, I will do more digging and come back tomorrow.
Average of ratings: -
In reply to Visvanath Ratnaweera

Hunting slow queries in the quiz sub-system, incident 2

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
The promised update: Had a look with a DB "guru" and found killed queries. So the slow queries are not the culprits but victims. PHP started requesting large amount of memory in Apache until the main MariaDB process was restarted. It all points to PHP building huge tables, most probably because the DB produces huge results. Investigating what they are. I could spare memory from the DB and give them to Apache but waiting to provocate more incidents to come closer to those huge data sets.

P.S. I know, the subject line is not fitting. Again waiting for the true cause before renaming and probably moving the discussion to Hardware and performance forum.
Average of ratings: -
In reply to Visvanath Ratnaweera

[Sharing Cart] Restore not releasing memory it consumed

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators

Our teachers are used to duplicate courses by copying course sections one after the other. They contain, among other activities, complete tests (Quizzes). One such action seems to the cause of the original problem. 

The web server logs look like

6 January 2026, 11:14:05 AM [USER] User: [USER] Sharing Cart block_sharing_cart: backup section User with id 68 has backed up a section with id 53637 in course with id 3859

6 January 2026, 11:15:03 AM [USER] Course: [COURSE] Sharing Cart block_sharing_cart: restored section User with id 68 has restored a section with id 63355 in course with id 4581 that takes 0 seconds

6 January 2026, 11:15:03 AM [USER] Deleted forum (id '367508') Sharing Cart block_sharing_cart: restored course module User with id 68 has restored a course module with id 367508 in the course with id 4581 that takes 0 seconds

6 January 2026, 11:16:19 AM [COURSE] Course: [COURSE] Sharing Cart block_sharing_cart: restored section User with id 68 has restored a section with id 63356 in course with id 4581 that takes 1 seconds

6 January 2026, 11:16:19 AM [USER] Forum: [FORUM] Sharing Cart block_sharing_cart: restored course module User with id 68 has restored a course module with id 367509 in the course with id 4581 that takes 1 seconds

(160 such lines in total within about 20 minutes)

 Although they say the transactions were finished, the committed memory jumped by 50 GB(!) during the same period making even the swap got exhausted and the OS started hunting processes, almost all killed processes were mariadb. That left Moodle with the unfinished tasks in the OP https://moodle.org/mod/forum/discuss.php?d=471223#p1893188.

Is this a bug in Sharing Cart?

Moodle 4.5.8+, mysql Ver 15.1 Distrib 10.11.14-MariaDB, PHP 8.1.34 (I know, support ended just 10 days ago. This is a heavy installation that needs a lot of planning before shifting. I have a staging server running PHP 8.3 to compare, if necessary)

Sharing Cart 5.0, release 6 2025092900

extramemorylimit is 512 MB (the default)

Edit: Found something suspicious: https://github.com/donhinkelman/moodle-block_sharing_cart/issues/257. Still would like to hear from the Sharing Cart side and how I could debug further.

Average of ratings: -
In reply to Visvanath Ratnaweera

[Sharing Cart] Restore not releasing memory it consumed

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Ping!

Sorry, I posted this first in an unrelated discussion (Hunting slow queries in the quiz sub-system). Got it split.

So the issue is, two teachers at most working during an hour or so managed to make the memory consumption jump by 50 GB within 20 minutes! Sharing cart is the suspect - See its Serious Memory issue #257. As massive as it is, it is not proven whether our site has this. Has anybody come across such behaviour? How can I dig deeper?
Average of ratings: -
In reply to Visvanath Ratnaweera

[Sharing Cart] Restore not releasing memory it consumed

by Mark Sharp -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers

We experienced the same thing with sharing cart last week too (though mostly with restore tasks). The processes seemed to be long dead, but Moodle kept waiting. Searching the server logs we found memory errors for the PIDs

image.png

After 24 hours we found the tasks in Moodle being cleared, however, we needed to change the frequency of running the `\core\task\task_lock_cleanup_task` task from once a day to every hour.

In our case it seemed that the blocker tasks came from just one or two people, though I've not been able to identify what the root cause is.

Usually these tasks take < 2 secs.

We know there were issues with badges being duplicated and included when backing up an activity cf https://moodle.atlassian.net/browse/MDL-78495 and Badges are included in backup · Issue #269 · donhinkelman/moodle-block_sharing_cart. I suspect Questions or sections might also be a possible issue: Moodle in English: [Bug Report] Section/Quiz duplication takes 5+ minutes and blocks entire site | Moodle.org

Average of ratings: Useful (1)
In reply to Mark Sharp

[Sharing Cart] Restore not releasing memory it consumed

by Mark Sharp -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers
Just found this tracker item (MDL-81511) which could be causing this issue. An infinite loop could easily bust memory, and can explain why it's not just big items that are failing.
 
The summary is that if the backup has failed then the async restore goes into a loop.
 
There is a proposed fix, but it hasn't been worked on for a long time, so perhaps needs more work.
 
I tested the fix a little on my dev, and it seems to stop the looping (though it will never fix a broken backup).
 
Before on dev:
 
block_sharing_cart\task\asynchronous_restore_task,Xdebug has detected a possible infinite loop, and aborted your script with a stack depth of '512' frames
Backtrace:
* line 121 of /backup/util/settings/backup_setting.class.php: call to md5()
* line 49 of /backup/util/helper/backup_general_helper.class.php: call to backup_setting->calculate_checksum()
* line 241 of /backup/util/plan/base_task.class.php: call to backup_general_helper::array_checksum_recursive()
 
 
Average of ratings: Useful (1)
In reply to Mark Sharp

[Sharing Cart] Restore not releasing memory it consumed

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Thank you very much!

The current state of the Sharing Cart is unacceptable. We deleted it in the production server - keeping it in our staging server to revisit some time.
Average of ratings: -