How does actual "very big size" instalaltion deal with big crowded forums for bulk mail emission ? As moodle cron needs to check email setup of each user before sending, is there a possible performance trap in having hudred of thousand users subscribing in a single forum ?
How does moodle.org manage that in general purpose forums ?
Thanks
How big is 'very big'?
There are two issues here that jump to mind -
1. If you send that much mail will you hit any bulk mail limits that may apply to you. Worse, will you look like a spammer?
2. How much load will this cause cron to take up?
You can always run cron on another 'front end'. Set it up like a load-balanced system. However, one front-end is not web accessible, it is just used to run the cron processes. Obviously, this could be a major re-design of your system and introduces other issues (e.g. cache performance)
Say f.e. 30.000 users in a course which can be typical for wide audience MOOC applications... and just sending a course news there.
I have a practical case of this situation with 8000 registered users in a knowledge review volume, and load for advising all members is clearly noticeable (receiving mail sent notification as admin for a big while when processing). This is just for one single course !, say we play hundred of them !
I have a plan to reduce drastically such load by customizing a forum to only send to "discussion subscribers". This would keep most discussions on an interative web mode (read and answer) and let people just cherry pick some message they would like to track by mail.
I was just asking myself how moodle.org xas proceeding with general developer/support forums which are actually crowded...
i found also this trick of adding a special cluster for cron processing, although it will also load common database engine.
Yep - if you use a cluster with a 'spare' machine for cron then you clearly need to keep an eye on the database performance. At these sort of levels it becomes very difficult to generalise. I usually come back to, do what you think is best, measure everything you can think of and monitor closely.
However, it would be interesting to know how moodle.org forums have been configured.
At the OU, we use a separate server in our cluster for running cron. Serving web pages is one sort of thing, and running a long batch process is another, so there are two main advantages to a separate server:
Tim,
Is there still a good case for running cron on a separate server in 2.7 now that 2.7 has Scheduled Tasks?
This is a good question. I think there is now the option to have multiple servers all running cron. Those might be your web servers, or might be some other servers.
It still seems plausible to me that really big jobs (E.g. forum emails, enrolment processing) are better on a separate server where you can configure PHP settings (memory limit, time-out, ...) differenty, but that is just a guess.
I don't know how much scope there is in the scheduled tasks system to control which tasks run on which servers. It might be possible to add that if it is not there already.