Hi Alex and all
I had a first-hand look off-list. And the problem was... Well, there was no problem, just everything was default: Linux 4.15.0-112-generic, Apache/2.4.18 (Ubuntu) with MPM prefork, missing APACHE_LOCK_DIR and many other similar parameters, PHP 7.4.8 on Apache, no external cache (other than OpCache and MUC), etc.
Basically increasing the MaxRequestWorkers to 512 and making the Innodb buffer pool bigger than the DB did the trick. The busiest exam they had so far, 257 candidates sitting for a programming assessment of 30 min duration which was open only for 35 min, ran without a single complaint. All finished the exam. Must say that all the candidates were accessing the exam through mobile broadband. So the network was only lightly loaded.
I had a first-hand look off-list. And the problem was... Well, there was no problem, just everything was default: Linux 4.15.0-112-generic, Apache/2.4.18 (Ubuntu) with MPM prefork, missing APACHE_LOCK_DIR and many other similar parameters, PHP 7.4.8 on Apache, no external cache (other than OpCache and MUC), etc.
Basically increasing the MaxRequestWorkers to 512 and making the Innodb buffer pool bigger than the DB did the trick. The busiest exam they had so far, 257 candidates sitting for a programming assessment of 30 min duration which was open only for 35 min, ran without a single complaint. All finished the exam. Must say that all the candidates were accessing the exam through mobile broadband. So the network was only lightly loaded.
The number of processes and memory utilization graphs are shown below.
- The full length of the graph is 60 min. The exam opening at 25 min in to it (measurement no. 250)
- No. of processes: Apache is in blue, MySQL in red (only 2 procs constatntly), total no, of all processes in green
- Memory consumption: in GB, same colour coding
- The full length of the graph is 60 min. The exam opening at 25 min in to it (measurement no. 250)
- No. of processes: Apache is in blue, MySQL in red (only 2 procs constatntly), total no, of all processes in green
- Memory consumption: in GB, same colour coding
The information was gathered through standard Unix commands like ps and from tools in the sysstat package. The graphs were created quick-and-dirty with the JpGraph https://jpgraph.net/ package.
What surprises me is the average size of an Apache process. It varied from 50 to 75 MB over time. Anybody knows Why? BTW, we are not too far from our forbidden "10-20 concurrent users per GB" rule. 
P.S. Only later I came to know that their Virtual Programming Lab https://vpl.dis.ulpgc.es/ server runs on the same machine. No idea how it impacted the measurements.