Moodle in English: Problems with my.cnf for moodle 8.2

Good afternoon Colleagues!

Please help me solve the problem with the database (MySQL). To optimize the database, I read a lot of information and made some settings. But during exams, the base falls off. Through HTOP I check the load of resources, they are not even half loaded. That is, it's not about the hardware. Maybe in the MySQL settings I am specifying something incorrectly or something needs to be added. I checked it through mysqltunner, I'll attach the screen. I will be glad to any recommendations. Thanks a lot!

Server characteristics

Xeon Е 5 44 core

RAM 388 Gb

RAID 1 + 0 2Tb

>> MySQLTuner 1.7.17 - Major Hayden <major@mhtx.net>

>> Bug reports, feature requests, and downloads at http://mysqltuner.com/

>> Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script

Please enter your MySQL administrative login: root

Please enter your MySQL administrative password: [OK] Currently running supported MySQL version 8.0.22

[OK] Operating on 64-bit architecture

-------- Log file Recommendations ------------------------------------------------------------------

[OK] Log file /var/log/mysqld.log exists

[--] Log file: /var/log/mysqld.log(76K)

[OK] Log file /var/log/mysqld.log is readable.

[OK] Log file /var/log/mysqld.log is not empty

[OK] Log file /var/log/mysqld.log is smaller than 32 Mb

[!!] /var/log/mysqld.log contains 392 warning(s).

[!!] /var/log/mysqld.log contains 3 error(s).

[--] 51 start(s) detected in /var/log/mysqld.log

[--] 1) 2021-11-06T04:40:09.288422Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.22' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community S erver - GPL.

[--] 2) 2021-11-06T04:40:06.020181Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock

[--] 3) 2021-10-05T21:43:29.340890Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.22' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community S erver - GPL.

[--] 4) 2021-10-05T21:43:25.095241Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock

[--] 5) 2021-10-02T18:21:30.627873Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.22' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community S erver - GPL.

[--] 6) 2021-10-02T18:21:26.296969Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock

[--] 7) 2021-08-25T04:06:19.322149Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.22' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community S erver - GPL.

[--] 8) 2021-08-25T04:06:19.102800Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock

[--] 9) 2021-08-24T10:17:31.573919Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.22' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community S erver - GPL.

[--] 10) 2021-08-24T10:17:25.728312Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock

[--] 24 shutdown(s) detected in /var/log/mysqld.log

[--] 1) 2021-11-06T04:35:54.131217Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.22) MySQL Community Server - GPL.

[--] 2) 2021-10-02T18:17:13.580562Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.22) MySQL Community Server - GPL.

[--] 3) 2021-08-25T04:04:04.606815Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.22) MySQL Community Server - GPL.

[--] 4) 2021-08-24T10:10:32.009034Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.22) MySQL Community Server - GPL.

[--] 5) 2020-11-13T05:44:15.210013Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.21) MySQL Community Server - GPL.

[--] 6) 2020-11-13T04:47:29.773677Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.21) MySQL Community Server - GPL.

[--] 7) 2020-11-13T04:44:59.052589Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.21) MySQL Community Server - GPL.

[--] 8) 2020-11-10T04:22:07.398753Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.21) MySQL Community Server - GPL.

[--] 9) 2020-11-10T04:16:49.896608Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.21) MySQL Community Server - GPL.

[--] 10) 2020-10-12T12:18:57.040227Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.21) MySQL Community Server - GPL.

-------- Storage Engine Statistics -----------------------------------------------------------------

[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA

[--] Data in InnoDB tables: 27.2G (Tables: 480)

[OK] Total fragmented tables: 0

-------- Analysis Performance Metrics --------------------------------------------------------------

[--] innodb_stats_on_metadata: OFF

[OK] No stat updates during querying INFORMATION_SCHEMA.

-------- Security Recommendations ------------------------------------------------------------------

[--] Skipped due to unsupported feature for MySQL 8

-------- CVE Security Recommendations --------------------------------------------------------------

[--] Skipped due to --cvefile option undefined

-------- Performance Metrics -----------------------------------------------------------------------

[--] Up for: 11d 1h 24m 12s (222M q [232.473 qps], 5M conn, TX: 302G, RX: 35G)

[--] Reads / Writes: 95% / 5%

[--] Binary logging is enabled (GTID MODE: OFF)

[--] Physical Memory : 377.4G

[--] Max MySQL memory : 9697.3G

[--] Other process memory: 0B

[--] Total buffers: 32.0G global + 64.0G per thread (151 max threads)

[--] P_S Max memory usage: 72B

[--] Galera GCache Max memory usage: 0B

[!!] Maximum reached memory usage: 9761.3G (2586.40% of installed RAM)

[!!] Maximum possible memory usage: 9697.3G (2569.44% of installed RAM)

[!!] Overall possible memory usage with other process exceeded memory

[OK] Slow queries: 0% (0/222M)

[!!] Highest connection usage: 100% (152/151)

[OK] Aborted connections: 0.00% (9/5319844)

[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance

[--] Query cache have been removed in MySQL 8

[OK] Sorts requiring temporary tables: 0% (1 temp sorts / 19M sorts)

[!!] Joins performed without indexes: 1501245

[OK] Temporary tables created on disk: 0% (93 on disk / 17M total)

[OK] Thread cache hit rate: 99% (20K created / 5M connections)

[OK] Table cache hit rate: 20% (4K open / 19K opened)

[OK] Open file limit used: 0% (3/10K)

[OK] Table locks acquired immediately: 100% (1M immediate / 1M locks)

[OK] Binlog cache memory access: 99.89% (5787399 Memory / 5793532 Total)

-------- Performance schema ------------------------------------------------------------------------

[--] Memory used by P_S: 72B

[--] Sys schema is installed.

-------- ThreadPool Metrics ------------------------------------------------------------------------

[--] ThreadPool stat is disabled.

-------- MyISAM Metrics ----------------------------------------------------------------------------

[--] MyISAM Metrics are disabled on last MySQL versions.

-------- InnoDB Metrics ----------------------------------------------------------------------------

[--] InnoDB is enabled.

[--] InnoDB Thread Concurrency: 0

[OK] InnoDB File per table is activated

[OK] InnoDB buffer pool / data size: 32.0G/27.2G

[OK] Ratio InnoDB log file size / InnoDB Buffer pool size: 4.0G * 2/32.0G should be equal to 25%

[OK] InnoDB buffer pool instances: 32

[--] Number of InnoDB Buffer Pool Chunk : 256 for 32 Buffer Pool Instance(s)

[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances

[OK] InnoDB Read buffer efficiency: 100.00% (19848984731 hits/ 19849345050 total)

[!!] InnoDB Write Log efficiency: 69.23% (48885526 hits/ 70609141 total)

[OK] InnoDB log waits: 0.00% (0 waits / 21723615 writes)

-------- AriaDB Metrics ----------------------------------------------------------------------------

[--] AriaDB is disabled.

-------- TokuDB Metrics ----------------------------------------------------------------------------

[--] TokuDB is disabled.

-------- XtraDB Metrics ----------------------------------------------------------------------------

[--] XtraDB is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------

[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------

[--] Galera Synchronous replication: NO

[--] No replication slave(s) for this server.

[--] Binlog format: ROW

[--] XA support enabled: ON

[--] Semi synchronous replication Master: Not Activated

[--] Semi synchronous replication Slave: Not Activated

[--] This is a standalone server

-------- Recommendations ---------------------------------------------------------------------------

General recommendations:

Control warning line(s) into /var/log/mysqld.log file

Control error line(s) into /var/log/mysqld.log file

Reduce your overall MySQL memory footprint for system stability

Dedicate this server to your database for highest performance.

Reduce or eliminate persistent connections to reduce connection usage

Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1

Adjust your join queries to always utilize indexes

Variables to adjust:

*** MySQL's maximum memory usage is dangerously high ***

*** Add RAM before increasing MySQL buffer variables ***

max_connections (> 151)

wait_timeout (< 28800)

interactive_timeout (< 28800)

join_buffer_size (> 64.0G, or always use indexes with JOINs)

You have mail in /var/spool/mail/root

Average of ratings: -

Re: Problems with my.cnf for moodle 8.2

by Howard Miller - Thursday, 18 November 2021, 4:08 PM

First thing.... why do you think MySQL is causing your performance problems?

Just an aside... the small amount of time I have spent with MySQL 8 seems to show that it is a LOT slower than 5.7. But it's not very scientific.

Average of ratings: -

Re: Problems with my.cnf for moodle 8.2

by Alex Rowe - Tuesday, 23 November 2021, 5:12 AM

The main things that jump out are these lines:

[--] Physical Memory     : 377.4G
[!!] Maximum reached memory usage: 9761.3G (2586.40% of installed RAM)
[!!] Maximum possible memory usage: 9697.3G (2569.44% of installed RAM)
[--] Total buffers: 32.0G global + 64.0G per thread (151 max threads)

You have a lot of RAM for this server, is it all just for MySQL or are you hosting Moodle on it as well?

The physical memory is 377GB, InnoDB is set to 32GB (good based on your 27GB database size), but your per thread rates are set to 64GB? It's showing with 151 connections that you could consume 9TB of RAM.

With MySQL, larger number aren't always going to be faster so you need to know exactly what you're changing and what you expect to happen.

For your database size, the InnoDB buffer pool instances are too many, InnoDB log file size is too large and join_buffer_size is ridiculously large at 64GB (bring that down to 4MB or lower).

You also need to monitor these values in MySQL to work out if they need to go up/down or are set correctly.

Check some of my other posts for some other information on MySQL settings.

You may also not be seeing SQL issues, it may be disk related, caching, network etc. With specifics, more information can be provided.

Average of ratings: -

Re: Problems with my.cnf for moodle 8.2

by Ken Task - Tuesday, 23 November 2021, 7:59 AM

In addition to what Alex said ... this one stands out:

max_connections (> 151)

151 connections is defaults and max and the recommendations from tuner says to increase that. To what depends, but from memory + other things seen in tuner report, would say it's a big server and you have many students/teachers using.

Many courses giving a quiz for example will use a lot of DB resources/connections.

Guess ... set max_connections to 451 ... restart DB server. Wait a day ... run tuner again. The odd number (the 1), BTW .... mysql always reserves 1 connection for root user.

'SoS', Ken

Average of ratings: -

Re: Problems with my.cnf for moodle 8.2

by Alex Rowe - Tuesday, 23 November 2021, 8:46 AM

It's best to check the max_used_connections status variable to see what you normally hit.

If it's the same as max_connections, then you've hit a limit and need to raise it. If it's less, then maybe think about lowering it.

MySQL also has buffers per thread, so if you go too high, like 451, you need to make sure you have the memory for "thread buffers X max_connections", on top of your InnoDB pool and log files (and your OS level RAM requirements).

Average of ratings: Useful (1)

Re: Problems with my.cnf for moodle 8.2

by Ken Task - Tuesday, 23 November 2021, 10:17 AM

Good point, Alex!

tuner did say:
[!!] Highest connection usage: 100% (152/151)

Users would see an error and have to wait before doing X for a connection to become available. Have users been seeing such an error?

Just in case:

mysql –u root –p SHOW STATUS LIKE 'max_used_connections';

Another in tuner report:

[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance

That applies even if config.php has 'localhost'. So skip_networking also advised.

Might also see what other max's ...

show status like 'max_%';

'SoS', Ken

Average of ratings: Useful (1)

Hardware and performance

Problems with my.cnf for moodle 8.2

Problems with my.cnf for moodle 8.2

Re: Problems with my.cnf for moodle 8.2

Re: Problems with my.cnf for moodle 8.2

Re: Problems with my.cnf for moodle 8.2

Re: Problems with my.cnf for moodle 8.2

Re: Problems with my.cnf for moodle 8.2

Empowering educators to improve our world

Moodle

Support

Get Involved

Contributions

Downloads

Tracker

Development

Empowering educators to improve our world