Separate Admin server

Separate Admin server

by Ron Meske -
Number of replies: 19
Picture of Particularly helpful Moodlers
Hello,

I have a client with an existing single server (AWS EC2 m5xlarge) setup where the client has addressed growth by just increasing the EC2 instance size.  Yet there are still occasions where the server slows down.  Reviewing their setup and logs, I have found the following:

  1. Database has close to 60,000 users.  Most of those no longer access the system, but they still need to retain them for reporting purposes for an extended period of time.  Cleanup of this is being discussed
  2. There are a number of schedule reports that are run and some manually which do impact the performance of the server for users.  Some reports are taking several minutes to run.
  3. When the nightly backup runs that creates a backup of the current database and the moodledata folder, there is also a significant slow down though not as impactful to users.
  4. Basic review of usage show that in a 1 hour timeframe, there is a range of 100-200 users active.
  5. 95% of courses are SCORM either locally installed or connected with an LTI connector.
  6. Moodle 3.9, php 7.2, Mysql5.7 (PHP will be upgrade to 7.4 and MySQL 8 very soon, Moodle is being discussed)
  7. Monitoring of the CPU usage shows that during the scheduled reports and the manual reports, the CPU is hitting 90-100% for several minutes, sometimes as long as 15 minutes.
  8. The cache hits seem to be fine for PHP files.  I have not yet looked at any database caching. 
To address the high usage by the reporting, is the following configuration possible and could it help minimize impacts on the "students" that currently occur?

  1. Separate DB server
  2. "Student" facing web server with local moodledata folder
  3. "Admin" facing web server with it's own moodledata folder

If the Admin server is running scheduled tasks and is only accessed by admin staff for reporting, does the moodledata folder need to be in sync with the "Student" version?


Average of ratings: -
In reply to Ron Meske

Re: Separate Admin server

by Ken Task -
Picture of Particularly helpful Moodlers
First, qualification for this response ... none.   Don't host with AWS, but have had 'experience' with Moodles and hosting with mutltiple providers.

"AWS EC2 m5xlarge" .... "just increasing the EC2 instance size"

Mind translating the above into real numbers?
Resource I found ...
4 CPU's, 16 Gig Memory
Storage     Value
EBS Optimized     True
Max Bandwidth (Mbps) on (EBS)     4750
Max Throughput (MB/s) on EBS     593.75
Max I/O Operations/second (IOPS)     18750
Baseline Bandwidth (Mbps) on (EBS)     1150
Baseline Throughput (MB/s) on EBS     143.75
Baseline I/O Operations/second (IOPS)     6000

above, if true and accurate, gives no indication of 'size' (like space) ... just bandwidth/throughput.

Have they/you ever used any tool to check 'health' of the DB
and/or usage of DB compared to usage of Web service?  From what you've described, currently, the web service is on the same machine as DB server/DB for moodle ... correct?  Can you install MySQLTuner - a perl script?

Am also assuming Linux - what flavor and do you have ssh access to it?

Backing up moodledata ... really the only subdirectory that is critical is filedir/
Might be best to use rsync - progressive.

Backing up code done only before and after an update or upgrade.
Using git?

Backup of DB (assuming MySQL - you've not said ... typical is mysqldump

Not sure what you mean by 'Student' version.  Nor do what you've ask .. breaking apart admin stuff from student/teacher stuff. :|


'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: Separate Admin server

by Ron Meske -
Picture of Particularly helpful Moodlers
Thank you Ken for your reply.

1. You did find the correct specs for the server.

2. The storage/space is 350 GB SSD, of that about 100GB is free.

3. The OS is Unbuntu 18 currently. It will be upgraded to version 22 and at the same time PHP 7.4 and MySQL 8.

4. The client has a custom plugin that is currently creating the backups and placing in S3 (Block storage). The entire server drive is also backed each night using the AWS Snapshot. Backup policy is also on the agenda to be reviewed.

5. The Moodledata folder is 220GB in size, about 90% of that is in the filedir.

6. The code is in a git repository, currently is also being backed up

7. The database is 8.3 GB in size and yes it is MySQL and is being backed up via mysqldump from their custom backup plugin.

I have not done any benchmarking or used any database tuning tools yet or checked the health of the DB. For testing purposes I have created a clone of their existing server and do have SSH access to install MySQLTuner. I will add that to my list for this week.

I am currently just in the fact finding phase and starting to explore potential solutions.

Because their reporting, both manual and scheduled, seems to be the main source of slow response for the students, I am wondering if a potential solution is setting up a multi-server system, might work and control costs better.

So a separate MySQL server (AWS terms: RDS or Aurora).
One web server that is only accessed by students to access their courses and progress.
One web server that is only accessed by admins to run reports and possibly handle the task scheduler.

Normally you see 2 web servers using a load balancer to manage load on the web server. This requires a shared or synced Moodledata folder, session cache, and I believe some other items. In this case, a user would always be directed to the same server based on their role. The exact way this will be handled I still need to work out.

Before I spend a lot of time exploring this, I wanted to see if this is even possible without a shared Moodledata folder. It seems plausible, since the reports are pulling from the database and not the moodledata folder. It looks like the scheduled tasks, or at least the majority, also only interact with the database.

Of course, benchmarking will need to be done to determine if there is any improvement and what the cost impact will be.

Ron
Average of ratings: -
In reply to Ron Meske

Re: Separate Admin server

by Brett Dalton -
Picture of Moodle HQ Picture of Particularly helpful Moodlers
I would look to investigate using RDS native replication and run your mysql dumps against the replica. This totally removes that load from anything student facing. Depending on how you are running your reporting it should also be possible to run it against this db instance. I've not implemented this myself but I know its been done by multiple Moodle institutions. I would not be running your MySQLDumps to the moodledata directory, thats is always going to compound the performance issue and can be a security issues, esp if server file respositary is enable or your web server is misconfigured. Use a separate, secure volume

For reference it is also possible to move the object storage to S3 which then makes sharing the storage between moodle instances much easier. there is a plugin written by Catalyst you might be interested in looking at. https://moodle.org/plugins/tool_objectfs
Average of ratings: -
In reply to Ron Meske

Re: Separate Admin server

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

First of all, no qualifications directly related to AWS, never been to the Amazonas nor speak their language. I'm just guessing from first principles.

Q1 about adding a separate server for the database. Generally, throwing resources show any improvement only if the piece it supports has been the weak point, the bottleneck, of the chain. From what you describe, the database is probably the current bottleneck. So it will help, provided that the new database is faster!

Q2 and 3, adding a web server, is a more involved. You are not replacing the old web server, instead add it to the old in tandem. With that you enter web clusters, which come with quite an overhead, for example you need a load balancer. Moodle imposes more restrictions around its moodledata/. Some directories _have_ to be shared. See details under https://docs.moodle.org/dev/Server_clustering_improvements_proposal.

Not only clusters have a steep learning curve, there is a considerable overhead too. So, don't be disappointed if the duo does only 150% of a single. Except fro that a cluster does what you want, separate the back end from the front end.

Aren't there cheaper solutions? How dynamic need those back end be? Are 24 h too long? If not, the back end processes may run during the night and only presents the results during the day.
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Separate Admin server

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
A correction to "a cluster does what you want, separate the back end from the front end". That is not what a web cluster usually does. It distributes the load amoung many web servers. But people can also use it as a means of partitioning. For example, there are people here, who run cron on one node, that node not participating in the pool.

All in all, IMHO you are aiming at a luxury solutions. There may be, must be, simpler/cheaper solutions. In fact I'm facing a similar problem right now. See assignfeedback_editpdf strikes again, this time in an upgrade 3.11 > 4.1.
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Separate Admin server

by Ron Meske -
Picture of Particularly helpful Moodlers
Your comment, "For example, there are people here, who run cron on one node, that node not participating in the pool." describes what I am asking about implementing.

In addition, I just need to know if the moodledata folder on the node running the cron job needs to be in sync with the other "nodes". Based on task descriptions it looks like there are some tasks that do impact data stored in the moodledata folder.

I would like to also use that "node" for admins to run reports.
Average of ratings: -
In reply to Ron Meske

Re: Separate Admin server

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
A node in the jargon is one server in a server cluster. I thought you wanted to avoid that complexity. Anyway, judge for yourself: https://docs.moodle.org/dev/Server_clustering_improvements_proposal. (It is the same link I gave you earlier.)

That document may answer some of your other questions.

Please note that this forum is about Moodle. Clustering is an extended topic, you get little support in these forums. There was a break-away group sometime in the past. I think our cluster specialists went there.
Average of ratings: -
In reply to Ron Meske

Re: Separate Admin server

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
It's all one database. You can't separate the admin bits from the student bits. And trying to have two databases synced together is likely to result in something on the spectrum....

madness <==> abject failure
Average of ratings: -
In reply to Howard Miller

Re: Separate Admin server

by Ron Meske -
Picture of Particularly helpful Moodlers
Let me try this again.

I am not looking to create a true cluster nor have two databases. I really don't think that level of complexity is needed at this time.

Let's say my setup is:
- DB server running MySQL
- Web server running Apache, PHP, MySQL client

That setup has been tuned and works well. The load on the database is light when students are using it. The high usage comes from the scheduled tasks.

It seems that the majority of the scheduled tasks do not need access to the Moodledata folder, but there may be something going on behind the scenes that I am not aware of yet.

I would like to know, if I setup a second web server, can that web server that also connects to the same database and have it dedicated to running the schedule tasks and disable the cron job on the first server?

Assuming that will work, I would then identify a way to have those running reports access the second server without needing a load balancer to run their reports.
Average of ratings: -
In reply to Ron Meske

Re: Separate Admin server

by Brett Dalton -
Picture of Moodle HQ Picture of Particularly helpful Moodlers
it wont help because the load leading to slow downs is on your database server, not the web server. The web server is doing basically nothing when running reports except for waiting for the DB.
Average of ratings: Useful (1)
In reply to Brett Dalton

Re: Separate Admin server

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
While that is a very natural conclusion to reach, in fact it is more complex than that.

The reason for having separate admin servers is to do with how web servers like Apache work, with a fixed number of worker threads that are avaiable for process requests. Most requests, hopefully, come in and get processesd in a fraciton of a second, so you want your Apache setup optimised to churn through those. When you get a long-running request like a slow report, even though the webserver is waiting for the DB and not consuming CPU, this request is tying up a slot for a long period of time, and that is bad. It only takes a few long-running requests to gum up the server, and block the flow of normal requests. Having a separate 'admin' server to handle known slow requests is one way to avoid this.

However, to do this effectively, you must first have got Moodle running in a clustered setup with multiple web servers. Then, the other thing you need is an easy way to filter out slow requests from the normal ones, so your load-balancer can send the right traffic to the right servers.

In the past, when the Open University ran our own servers, we did have separate 'normal' and 'admin' servers, and we could do that based on IP address of the user, becuase we are a distance learning university, and so all our students have off-campus IPs, and staff who did slow operations were all on our campus network. But then, Covid happened breaking that assumption about where different users were based (IP-address-wise) and also we moved our setup to AWS, so we don't do this any more.

The other thing to try to do is to move really slow operations away from the web UI, and use scheduled or ad-hoc tasks instead (like was done for Moodle backups).
Average of ratings: Useful (2)
In reply to Tim Hunt

Re: Separate Admin server

by Ron Meske -
Picture of Particularly helpful Moodlers
Hi Tim,

Thank you for sharing your experience. Yes there are several long running operations, both in scheduled tasks and admin running reports. with both the web server and the db server on the same machine, I am seeing long spikes in processer usages above 95% for several minutes. So instead of just bumping up the AWS server size I have been exploring various other ways to address the problem.

It is the reports that are taxing the server. I asked for a two day period of no reports and though memory and cpu utilization is high, the server is still responsive. When testing just the reports being run on a cloned test server configured the same, the server is less responsive. So moving the reports to scheduled tasks makes a lot of sense. I just need to get buy-in from those users doing the reporting and getting them to change from adhoc reporting to a scheduled approach.

Has anyone looked at dynamically changing the innodb buffer size for individual reports to see how it impacts the server? Does Moodle support that?
Average of ratings: -
In reply to Ron Meske

Re: Separate Admin server

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
It all depends on how current the reports need to be. Say, if 24 h delay is OK, then you can clone the site daily and run the reports in the clone.
Average of ratings: -
In reply to Ron Meske

Re: Separate Admin server

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Really if you are at the point where one server is struggling, you should switch to a clustered setup.

The AWS-approved way to set that up is this reference architecture: https://docs.aws.amazon.com/architecture-diagrams/latest/moodle-learning-management-system-on-aws/moodle-learning-management-system-on-aws.html?did=wp_card&trk=wp_card, which is actually pretty close to how we run our very large Moodle site. Of course, that is a lot of AWS stuff to wrap your head around (All in the scope of the 3-day 'Architecting with AWS' training course, but I am not sure I would recommend paying for that - we got some free places on it.)

Certainly with a cluster in AWS, it is worth looking at their hosted Aurora Database and Redis hosting for the Moodle database and cache, rather than maintaining your own storage.

However, I can see why you might want a quick fix for now - I am just afraid I can't suggest one.
Average of ratings: Useful (1)
In reply to Ron Meske

Re: Separate Admin server

by Brett Dalton -
Picture of Moodle HQ Picture of Particularly helpful Moodlers
@ron I'd recommend to not try and dynamically change the buffer pool sizes, the operation can't happen until all current transactions are completed and new transactions cant start until the resize is complete. This will mean it will effectively halt everything for a short period during that operation. it will generally only be short but not ideal. Unless you are super memory constrained on your database its better to have it set to an appropriate size in the first place. This is also another advantage of using a read replica database as this can be configured differently.

@Tim, web workers are very rarely an issue in this type of case. database pool connections frequently are. Data base connection pool sizes tend to be in the 100s of connections (Mysql default connection_pool size is 100), Web servers on the other hand tend to be in the 1000s. /etc/httpd/conf/httpd.conf default max connections per child process is 4000 and defaulot max_clients is 256 from memory. Contention is almost always at the database layer, not the web server, its only through use of decent caching that you can scale web servers faster than database servers.
Average of ratings: Useful (1)
In reply to Brett Dalton

Re: Separate Admin server

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Agreed. DB is the bottleneck in scaling Moodle.

When tuning a Moodle site, it is important to get everything consistent: e.g. Apache max_clients needs to be right given the memory use of one Moodle request and the RAM available on your server - the default is probably not best; and DB max_connections probably needs to be number of web servers x max_clients per server. And so on.

However, it was the case that for many years, the Open University ran our server with admin traffic separated from student traffic, because that made a significant difference for us. It seemed worth sharing that in the context of this thread. (At the point this made a difference we had 11 web servers, and ~60,000 users logging in each day, making 2 million+ page-views.)
Average of ratings: -
In reply to Tim Hunt

Re: Separate Admin server

by Ron Meske -
Picture of Particularly helpful Moodlers
Thank you everyone for your insights and advice.  I have not found the need for setting up a cluster of multiple web servers yet.  The closest I have come is 1 web server and 1 db server.  Pre-Aurora I used an RDS server running MySQL and when Aurora came out I did switch to that but at the time Moodle didn't support Mariadb which meant remembering to modify some code, an if statement, with each update.  

I have found that running separate web and db servers allows fine tuning the server for a specific purpose and you can get much better performance with lower cost servers than having one big one.  This may be the answer for this Moodle installation as well, but wanted to see what other options there may be that others have done.

@Tim, as I plan for the future need of moving into using multiple web servers, are there any pointers on AWS configuration for the EC2's and disk I/O with ESF?  
Average of ratings: Useful (1)
In reply to Ron Meske

Re: Separate Admin server

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Well, the only real pointer I had was the reference architecture I linked to.

A few other tips:

If your Moodle site has a lot of files, then Catalyst's objectfs plugin, which lets most of the uploaded files be stored in S3, rather than EFS, will probably save you money. However, we setup our site without that first, and got everything working, and then added objectfs later.

With Aurora Redis, it turns out that the resoruce constraints are tricky and not clearnly documented. In particular, network traffic is quoted as a 'burst' bandwidth. What this means if that you can only just that much bandwith for a small amout of time. If you exceed that, you get throttled! SO, if with high traffic, your site works for about 20 minutes, then grinds to a halt, this might be the cause (and you need to bigger Redis instance).

Anyway, as you say, just separating the web server and the DB is a good first step in moving away from everything-on-one-server. And Aurora DB-as-a-service is probably a good way to go down that route.
Average of ratings: -