Hi,
I am fairly certain this topic has been discussed over and over again. Forgive me for sounding repetitive. I have been, you could say, thrown into the deep end in getting our University LMS to accommodate online exams. We are currently hosting on Google Cloud and this is pretty much our architecture (they are all in the same region):
- 1 Load Balancer to distribute traffic to Compute Engine
- Compute Engine launches the VMs hosting Moodle code (16GB vcpu and 64GB RAM) scaled between 1 and 3 instances
- Filestore (1 TB) for Moodledata automounted on the auto-scaled VMs
- Cloud SQL for our PostgreSQL Instance (8GB vcpu and 32GB RAM)
At first, the platform was used to allow students to access assignments which they would download and submit at a later time. This worked out well. Problems began when we introduced real time quizes. We started with 500 people which the system managed well. When we scaled out to 2000 users, the numbers of requests overwhelmed the database which eventually crashed. We were introduced to the dreaded error:
db=moodledb,user=postgres FATAL: remaining connection slots are reserved for non-replication superuser connections"
After reading on a number of forums, I decided to place a pgBouncer instance between the VMs and PostreSQL. I did not perform any customizations other than adding the database connection details. Now, instead of the message above, I get multiple messages like this:
ERROR accept() failed: Too many open files
closing because: client close request (age=160s)
closing because: client close request (age=90s)
WARNING C-0x55c61cc57780: (nodb)/(nouser)...pooler error: no more connections allowed (max_client_conn)
closing because: no more connections allowed (max_client_conn) (age=0s)
I am stuck and we have been off all day. I doubt we can afford to go be off another day. I will be grateful for any advice on what I can do. If I need to provision more resources. More RAM or something else. If there are any configurations I can look out for? For a bit more context, I happen to have the exam timetable and the largest real-time quiz has 2300 people. Although this does not account for other students who will be browsing at that time. Thanks everyone. Looking forward to your responses.
