We have recently been experiencing performance issues when attempting to install or update plugins on our Moodle 3.2.3 setup. We run a load balanced setup of 3 web servers (Apache) and 1 database server (MariaDB).

Originally we were able to copy the new/updated plugin files to each app server and run the upgrade via the UI. The performance of this has degraded significantly over the last few months; at first it was slow, then it took the app server that it was running offline, now it takes our entire Moodle site down.

I originally believed this was related to using the database for sessions (we saw an increase in database locks during plugin installs/upgrades), so I tried using the CLI to run the upgrade script (sudo -u apache /usr/bin/php admin/cli/upgrade.php). I believed that this would avoid the locking problem, as no user session would be created, however the script ran for over an hour and did not actually change anything. There was no debug output to the console, no errors in the Apache/PHP log and no entries in mdl_upgrade_log. This was run at a time when there would be very little user activity on Moodle.

This presents a significant problem for us, as it limits our ability to develop and deploy in a agile way (it's not practical to scheduled downtime every time we want to make minor updates to plugins). The most logical explanation is that the volume of data in the database is causing these performance issues (~67GB), however I suspect there are significantly larger Moodle installations out there.

We are planning on moving to Redis for session storage, however I am no longer confident that this will resolve the upgrade issue (we are still going ahead with this, in accordance with the recommendation to avoid using the database for larger sites).

Has anyone else experienced issues like this? Does anyone have any ideas what might be causing this? I would be concerned if the cause were the amount of data we hold, as this could cause us even more problems down the line. The documentation also indicates that the upgrade script should complete quite quickly (https://docs.moodle.org/32/en/Administration_via_command_line#Upgrading).

Many thanks in advance!