For a while now, our mysqld.err
has a large number of entries for Aborted Connection (Unknown Error).
These occur frequently, often 2 - 10 minutes apart 24/7. Doing some
basic research into the mysql Aborted Connection error, a different
message would display if the problem was, for example, having too small a
packet size set in my.cnf. Looking at the error log, these entries started appearing immediately after an upgrade from 3.4 to our current version of 3.6.4 but we haven't noticed until recently.
I'm struggling to identify a cause. One idea I have is that mysql's memory buffer is full 100% of the time. With its reserved memory at capacity, and swap at capacity, old connections are terminated before cleanly closed to make space for new connections.
Another memory related issue we have is with the Automated Backups task. Every day at the same time, this task runs and results in a large memory spike. Sometimes there isn't enough free memory and the mysqld service crashes and auto-restarts into crash recovery. Our site will be inaccessible for only a few minutes. A prior sys admin configured this task to run hourly. Realistically, how frequently should this be run? I'm thinking about changing it to once or twice a day. This task could also be failing because we have a very large number of courses (probably in the tens of thousands) still on the site going back several years. No course clean up has ever been attempted. Could the Automated Backups task be iterating through too many courses at this point? If so, what are the best methods for bulk course removal, if any?
Our configuration is as follows:
- Moodle and MariaDB are on the same RHEL system.
- Moodle 3.6.4+ (Build: 20190524)
- MariaDB 10.1.12
- PHP 7.0.6