I am experiencing the following problem and I would like to get your opinions and/or suggestions.
I have setup Moodle in a cluster environment using EC2(Web servers), RDS (MariaDB), EFS(File System) and ElasticCache(Memcached).
Under normal operations Moodle might have ~23k concurrent connections (new and existing) to the Memcached servers.
The problem is when I am calling Moodle WebServices from a program we have developed to bulk create and enroll users to courses. Whenever this program runs Moodle opens ~80k concurrent connections to Memecached servers which hits the limit they have in AWS.
This causes the error:
[27-Sep-2018 10:16:31 Europe/Athens] PHP Warning: Memcached::setMulti(): failed to set key in /moodle/cache/stores/memcached/lib.php on line 442
And then DB connections are starting to piling up, and they are suddenly increased from ~20 DB connections to ~400 DB connections.
Then Moodle becomes unresponsive, I get 504 errors and the page does not load.
Few minutes after I stop the program which creates users, Memcached connections and DB connections drop and Moodle is back to normal.
These are the list of functions I am using, are those function so expensive to execute? What is the relation of these function to memcached?
Thanks for your time