Hi Khaled! How are you?
The first thing that stands out in your scenario is the 404 error, which indicates that the requested file is not being found. The issue that needs to be investigated is how the Moodle data is being accessed. Are they local on each server, or are you accessing them remotely via the network? This information is crucial because, in a cluster setup where servers are distributed, if the Moodle files (PHP and images) are not being shared among all of them, this can cause a significant impact on performance.
When you have multiple servers in a cluster and the access to the files is concentrated in one location (for example, a single server or a shared network area), Moodle suffers from slowness. The processor needs to wait for the network's response to continue processing requests, which results in overload and decreased performance, especially when there are many concurrent users.
So, where are the Moodle installation files located?
And for MoodleData, there is the Alternative File System plugin that I built to move MoodleData to external servers.
Another important point is the images, CSS, and JS of Moodle. Since they are managed by PHP, each request makes the server execute processes to generate and deliver these files, which overloads the infrastructure. I have a guide on how to improve Moodle's performance using caching, which could be helpful for your case. Take a look at Comprehensive Guide to Configuring BunnyCDN in Moodle. This can reduce the load on the server and help maintain performance even with many concurrent users.
Eduardo Kraus
Teacher and programmer