Don't think anyone could provide accurate settings for your server ... especially if it's an all in one ... DB server on same server as webservice and moodle code.
There are tools that will help with the DB ... install mysqltuner. It's a perl script that will provide recommendations for tweaks to the DB ... such things as addressing 'MySQL server has gone away' - usually max_allowed_packet
https://dev.mysql.com/doc/refman/8.0/en/packet-too-large.html
The above link to MySQL version 8 but the same page and information is also for 5.6,etc.
Also: https://mariadb.com/kb/en/library/server-system-variables/
MariaDB is now the 'darling database' for true RedHat Enterprise. Supposed to be a drop in replacement for MySQL so the docs concerning mysqlserver should/might apply to MariaDB.
You could be reaching usage such that a dedicated DB server would be best. Have read that DB performs best when it can use 80% of the memory. But on a standalone box that's hard to achieve because it's a balancing act between web service and DB server. MySQLTuner will help you. In it's report/info display it will show the number of dropped connections ... which usually means one need to tweak 'max_connections' value for DB upwards ... default is 151 in MySQL and I bet you've already tweaked that upwards.
While you have a large number of users and courses, students basicially interact with courses/moodle asynchronously .. a few students at a time .... not like a K12 entity using Moodle in a F-2-F setting.
Then there are also some things in Moodle ... which if on ... will degrade performance. There should be a link to server performance checks. Running autobackups? Running cron frequently enough?
Time outs for php ... defaults for PHP usually have to be tweaked upwards ... or it wouldn't hurt to tweak upwards ... a little at a time. Try NOT to guess.
Example: time for a script to run ... default is 30 seconds but if you've got some heavy processing going on then tweak that upwards ... 30 -> 60 ... etc. ... again, TRY NOT to guess.
Those sorts of things require some monitoring ... to begin with, simple top command during prime time might provide you some info and point you in a direction. Server running any SWAP space? What's using SWAP? MariaDB? 'man top' for options to display of top info so you can see what's using SWAP.
Hope that gives you some info and a place to start.
'spirit of sharing', Ken