Hi all,
I am currently working in Uzbekistan as a freelance consultant in a countrywide project for distance learning for primary school teachers. We are going to use Moodle as our LMS. We are now in the process of planning the infrastructure. We will have a central Distance Education Development Center (DEDC) in Tashkent and Regional Coordination Units (RCU) in the 14 regions of Usbekistan. I really would appreciate getting some ideas how best to set up the infrastructure.
Course creation will be done in the RCUs and in the DEDC. So we will have centrally created courses for all regions and individually created courses by the regions. Thats why I suppose we should install an individual Moodle instance for each region. Yet, administrative capacities in the regions still have to be developed. There is this question whether to have an individual instance in each of the regions, or to have all instances installed in Tashkent at the DEDC.
What kind of advantages, disadvantages do this 2 options have?
In terms of scalability we will follow a step by step approach. Until mid of next year we will be in a pilot phase. There will be only between 50 and 80 users up to th end of this year. Then the no. of users will increase to about 500 with an estimated no. of concurrent users of 100 to 200.
We will probably start with a single server. Yet the layout of the system has to take into account that after the pilot phase the no. of users will potentiallly increase to something like 200.000 (estimated concurrent users ~2000 - 3000). So we have to plan for for a load balanced server cluster.
It is already planned to procure 15 servers based on win 2003 Server all with dual core CPU and 4GB of RAM. I will try to get more RAM for the servers.
All other details still need to be figured out.
Please give me your ideas how best to deal with this, I need to get the basic structure clear.
Thanks for all ideas I am hopefully going to receive,
Manfred
Hi Manfred.
I am actually on same questionning than yours for a french national wide project.
May I argue with you on centralized/distributed discussion.
Centralized hosting center may have great benefits in maintaining, updating, replicating and having a better control of failures and service continuity that physically distributed organisation. A centralized data center will not avoid administration (funcional and pedagogical) to be delegated to your RCU's scope.
The question of building a very huge system monolithic with 100.000 users or dispatching those 100.000 users on regional subvolumes was argued also. There should probably a way to have an higher factorization of maintenance, using a single code base with switched configuration files. This was evocated in some technical discussions on forums, with a light but not weak problem of updating multiple database volume (on each "virtual moodle") when a module has been upgraded or installed.
I would rely on a centralized hosting service that can afford having technical qualified and appointed crew to take care of it, rather a widely distributed architecture that will need a ressource running (or flying) from here to there to solve problems, with a probable heterogeneization of the local environments.
Other question is why using Windows 2003 servers that will cost you a great amount of perfomance and much more care in security survey than Linux mounted, without GUI (no need for clusters) ? (Of course I do not have your context, and am arguing "theoretically")...
I would second Valery's question. Why use Windows 2003 for this cluster? You will see much better performance under Linux for this work load. This would be way more advantageous than getting more RAM for each node.
Thanks for your comments Valery,
I would rather go for Linux as well, but responsible government staff already came up with an procurement list even before I entered the project. bueraucracy is a real issue in Uzbekistan and something which is being fixed in written and approved by some administrative staff is difficult to change. I will try.
Can you explain a bit deeper how a centralized infrastructure based on Linux OS, on a loadbalanced cluster would look like? I myself am not that much experienced in this huge installations. I am still waiting for a system developer to join the team.
Thanks,
Manfred
I would rather go for Linux as well, but responsible government staff already came up with an procurement list even before I entered the project. bueraucracy is a real issue in Uzbekistan and something which is being fixed in written and approved by some administrative staff is difficult to change. I will try.
Can you explain a bit deeper how a centralized infrastructure based on Linux OS, on a loadbalanced cluster would look like? I myself am not that much experienced in this huge installations. I am still waiting for a system developer to join the team.
Thanks,
Manfred
'bueraucracy is a real issue'
This may be a tautology in many countries, Manfred...
. I was suspecting something like that.
In your case, (as in mine) the real question and first architectural question would be :
is the platforme a huge monolithic monster, using a clustered load splitting architecture, all clusters virtually able to provide any user, or is the system regionalized, that is, the regional service is providden by an host which is the exact clone (application code, base and startup settings, module versions, agreed adds-on) than the other one. This may impact the topology of the physical infrastucture a lot.
In the first case, you will have to solve how regional entities will identify their own local space. Will the internal administrative model of Moodle allow administration delegation enough, sufficiant insulation of region spaces if required, or conversely sufficient factorization of commands and settings ? There will be in that case "one big Moodle" for all, running on we could say that three parts system : the front-end apache/php with moodle application, a physical backend filer for a huge moodledata, and the data backend as a clusterized MySQL.
I have not yet experimented the real field situation. I need still time to read all docs for these loaded systems. It's mostly my work in the 2 next weeks.
In the second case, you must deploy N independant moodles, that theretically could be setup on N separated servers. Economically, there will be some resistances to provide you so many machines. So local moodles should be ventilated over the available processors. Say there is a big region with a lot of people and potentially a lot of users, and there are four country regions that do have much less. 2 servers could hold the system with four virtual moodles on the latter and only one on the former.
In all cases, there will be an absolute need to develop some tools for code synchronisation. A module cannot (or should not) be updated manually on 10 machines, and 35 moodles. A checkup should not be executed on the system if changes are not immediately replicated on all clones. These may be scripts using rsync or NFS mounting and xcopying strategy, with probably a tool for checking code integrity over all clusters.
You should read as I will all load balancing related threads in forums, along with load balancing docs. I'm interested on what you'll find either...
Small things about load balancing :
If the regional hardware structure is retained, load balancing should not be a startup issue, un less some regional bases have a really important amount of users.
What I know till now is that frontend balancing will do need checking the session consistence issue. This is transparent with sessionless high load web sites or cookie driven session, what is the case in Moodle. A network level load balancer will route queries to roundrobin IPs or load detector based algorithm.
There should be no detectable issue when load is low, but probability of access collision on tables in the DB system, or on files in moodledata (the only 2 locations where there is mainly something written in the system), will raise.
I'm not really sure Moodle is full safe in transactionnal locking, probably not, and even less in third-party modules.
To be continued...