MySQL Performance for Moodle

MySQL Performance for Moodle

by Jonathan V -
Number of replies: 10

Hello,

I am a MySQL performance engineer and I am new to moodle.

I have recently reviewed a large moodle installation with X thousand users and I wanted to show what I discovered as well as ask more seasoned moodle engineers if this is a good idea.

Firstly, moodle is largely read based. The writes in MySQL need to be optimised by in the context of not locking the reads. Reads to Writes are about 50 to 1.

Query cache appears to be a good idea for moodle.

The table mdl_files table seems to be a very big bottleneck when you log all queries hitting the database.

I recommend adding a partition to reduce contention

alter table mdl_files drop primary key, drop key mdl_file_pat_uix, 
add primary key (id,contextid), 
add unique key mdl_file_pat_uix(pathnamehash, contextid) 
partition by hash(contextid) partitions 10;

I also found some indexes that may speed up the login pages:

alter table moodle_live.mdl_scorm_scoes_track drop index mdl_scorscoetrac_use_ix, add index mdl_scorscoetrac_use_ix(userid,scoid,attempt,element), add index idx_userid2(userid,scormid,attempt);
alter table moodle_live.mdl_cache_flags drop index mdl_cachflag_fla_ix, add index mdl_cachflag_fla_ix(flagtype,expiry);

Also, logstore table seems to be quite big, but not read very often. 
However, courseid seems to be a good key to partition it on if you would like to.

Please review and let me know if you think that these are helpful.




Average of ratings: Useful (2)
In reply to Jonathan V

Re: MySQL Performance for Moodle

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers

I am nowhere near a MySQL expert.  It appears that these changes that you recommend should probably be done at "core" level, since they are altering the tables.  Right?

I am going to share with you my my.cnf file that I am currently using with my Moodle.  I am always looking for ways to improve it, and any suggestions that you have would be welcomed.  I have tried to document this file well, how my comments in it do not bother you.  I am using MySQL version 5.6.34, php 5.6.29, and running on Apache 2.4.23.  This is all within a CentOS VPS.  I have a small moodle, about 150 students in total.  I also spread the due dates on assignments in order to spread the load on my server.

In reply to Rick Jerz

Re: MySQL Performance for Moodle

by Jonathan V -

Thank you for your reply, Rick.

I think the indexes I noticed could be added to core. It would depend if it is moodle core or an addition from a developer working for the school. Probably the former.

Regarding partitioning, it would be nice if that would be added up front to be honest, but its up to moodle itself to decide.


The place I am at the moment has in fact has a lot of issues with submissions. I made a recommendation to the my.cnf, but please note that it has not been implemented yet.

I would say though, that I have gone over it very carefully and tuned it to their use case:


innodb_buffer_pool_size = 12G ### How much innodb data to store in memory. Higher = faster performance

innodb_file_per_table = 1 ### Each innodb table is its own file on disk. 

innodb_flush_log_at_trx_commit = 2 ### Flushes to disk in batches instead of per change

innodb_io_capacity = 800 ### Min IO speed expected to write to disk. 

read_buffer_size = 2M ### Helps with reading temp tables faster, bulk inserts and nested queries

read_rnd_buffer_size = 8M ### Helps with multi range queries and ORDER Bys. 

sort_buffer_size = 8M ### Helps with ORDER BY queries, which we have a lot of.

table_open_cache = 4096 ### Cache data about open tables. Saves reopening tables.

thread_cache_size = 128 ### How many threads to cache. Can help reduce thread count.

thread_stack = 256k ### How much cache each thread can have. Higher = more complicated results are cached

tmp_table_size = 64M ### Max size a tmp table can be.

innodb_flush_method = O_DIRECT ## Innodb flushes to table files directly

innodb_log_file_size = 512M ##

innodb_log_files_in_group = 3 ##

innodb_read_io_threads = 8 ## innodb background threads for reads. Helps with reads

join_buffer_size = 2M ## Helps when a join doesn’t have an index or an index wont be more helpful than a full table scan

max_heap_table_size = 64M ## Max size a tmp table can be in memory

query_cache_limit = 2M ##

query_cache_min_res_unit = 2048 ##

query_cache_size = 196M ##

query_cache_type = 1 ## Caches complete query results. Data gets invalidated when underlying tables are changed.

table_open_cache_instances = 8 ## Divides the table cache to reduce contention

table-definition-cache = 4096 ## Stores table definitions - .frm files

key_buffer_size = 16M # As there are no MyISAM tables, this variable helps with MyISAM temp tables.

back_log = 1500 # When MySQL gets a lot of connections in a short time, it will keep them in a back log

innodb_purge_threads = 2 # May reduce locks for when updates/deletes are run often (like sessions table)

innodb_sort_buffer_size = 2M # Helps with index changes and create table statements

#innodb_page_cleaners = 4 # Will help with start up and shutdown in 5.7


Average of ratings: Useful (1)
In reply to Jonathan V

Re: MySQL Performance for Moodle

by Jonathan V -

In addition, some OS settings I recommend tweaking are:

transparent huge pages disabled
vm.swapiness = 1
ioschedular = deadline
kernel.sched_autogroup_enabled = 0
kernel.sched_migration_cost_ns= 5000000


In reply to Jonathan V

Re: MySQL Performance for Moodle

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I have to be honest and say that I had never even heard of partitions before. 

It may well be worth raising a ticket in tracker.moodle.org. However, Moodle does need to work with multiple database servers. I don't know if this is an issue. 

In reply to Howard Miller

Re: MySQL Performance for Moodle

by Jonathan V -

I just tried submitting it on their tracker, but was not allowed as I have not 'searched for other issues on the matter'.

But partitioning is common among relational databases: MySQL and Postgres have it.

In reply to Jonathan V

Re: MySQL Performance for Moodle

by Tomasz Muras -
Picture of Core developers Picture of Plugin developers Picture of Plugins guardians Picture of Translators

Hi Jonathan,

Thanks for sharing your findings.

For mdl_cache_flags issue, please add a comment to  MDL-54035 (you can also vote for this issue) - maybe we will finally move on with it.

Partitioning mdl_files is an interesting idea - but will it really help with the performance? Have you done have some tests before/after partitioning it? Or maybe you have a set of SQL queries that you saw in your logs and you think they will benefit from the partition?


There is also another issue with mdl_files table. Adding a resource, which is a single file creates to records in the database. The record for current directory "." and the record about the file itself. The record with the directory information is pretty much useless - it contains no data. I think it was done this way for convenience - so resource with single file is implemented the same way as resource containing many files / directories.

However - if your Moodle has 100s of thousands file resources and most of them are just singe file, then you are generating a lot of unnecessary entries in the DB. I saw real-life installations with 8 million records - and about 4 million of them being just this "directory" entry, which could be removed. 4 million records less in a table is a lot smile.

In reply to Tomasz Muras

Re: MySQL Performance for Moodle

by Jonathan V -

Hello Tomasz,


Thank you for replying. 

Regarding partitioning in general, I would say, it does help solve this sort of problem. The trick is to find the right key to partition on or you may make the situation worse in some instances.

Regarding if I tested partitioning on this table, the answer would be not yet, but we hope to do so in 2-3 weeks.

Our current mdl_files has slightly under 4million. If there is something we can do to reduce that, I would be most grateful. Should I check for all filenames where they equal "." and remove them?


Best Regards

Jonathan Levin


In reply to Jonathan V

Re: MySQL Performance for Moodle

by Tomasz Muras -
Picture of Core developers Picture of Plugin developers Picture of Plugins guardians Picture of Translators

You can not just drop them at the moment. I'm pretty sure that the current implementation expects those records to be there - a patch to Moodle is needed here.

In reply to Tomasz Muras

Re: MySQL Performance for Moodle

by Jonathan V -

Apparently, the high usage of the mdl_files table is to do with a pdf conversation plugin in moodle.

May not apply to everyone.

In reply to Jonathan V

Re: MySQL Performance for Moodle

by Tavi Al -
I have a Moodle installation that I intend to use ,for 600 concurrent users and my Server specs are 16GB Ram and 8 processor cores. I would appreciate it if you can look at my my.cnf settings below and make recommendations on how best to optimise the system. I have not gone above 300 concurrent users taking a quiz at the same time and I have Database failure warnings.

Thanks much! 


interactive_timeout=600

key_cache_block_size=4096

max_heap_table_size=132M

max_join_size=1000000000

max_allowed_packet=268435456

open_files_limit=10000

query_cache_size=200M

thread_cache_size=100

tmp_table_size=532M

wait_timeout=3800

max_user_connections=2000

myisam_recover_options=FORCE

innodb_file_per_table=1

innodb_flush_log_at_trx_commit=0

innodb_purge_threads=1

innodb_support_xa=0

innodb_thread_concurrency=40

table_open_cache=328

innodb_buffer_pool_size=200000000

innodb_file_format = barracuda

innodb_large_prefix

innodb_file_per_table = 1