Hello Moodlers,
I am experiencing the following problem and I would like to get your opinions and/or suggestions.
I have setup Moodle in a cluster environment using EC2(Web servers), RDS (MariaDB), EFS(File System) and ElasticCache(Memcached).
Under normal operations Moodle might have ~23k concurrent connections (new and existing) to the Memcached servers.
The problem is when I am calling Moodle WebServices from a program we have developed to bulk create and enroll users to courses. Whenever this program runs Moodle opens ~80k concurrent connections to Memecached servers which hits the limit they have in AWS.
This causes the error:
[27-Sep-2018 10:16:31 Europe/Athens] PHP Warning: Memcached::setMulti(): failed to set key in /moodle/cache/stores/memcached/lib.php on line 442
And then DB connections are starting to piling up, and they are suddenly increased from ~20 DB connections to ~400 DB connections.
Then Moodle becomes unresponsive, I get 504 errors and the page does not load.
Few minutes after I stop the program which creates users, Memcached connections and DB connections drop and Moodle is back to normal.
These are the list of functions I am using, are those function so expensive to execute? What is the relation of these function to memcached?
"core_cohort_add_cohort_members";
"core_cohort_delete_cohort_members";
"local_wsgetusercohorts";
"core_course_create_courses";
"core_course_get_courses";
"core_course_get_courses_by_field";
"core_enrol_get_enrolled_users";
"local_wsgetusercourses";
"core_user_create_users";
"core_user_get_users";
"core_user_get_users_by_field";
"core_user_update_users";
"enrol_manual_enrol_users";
"enrol_manual_unenrol_users";
"core_group_add_group_members";
"core_group_get_course_groups";
Thanks for your time
Christos