A friend of me told me about the Moodle project, and I was just checking the source code, but I was really shocked by the code base. Maybe I'm not looking at the right parts of the code, but the bits I've seen, look a little bit messy to me to be quite honest.
It seems to me there is not a domain model at all. Presentational logic (like HTML), data store logic (SQL queries) and domain logic are all mixed together. Am I really missing something, or is Domain Driven Design not a paradigm in use by this community?
I agree that Moodle is a mess in regards to the organization of the project coding. The people who work on it are very giving and very responsive. Maybe you can help get it more in line with a supportable code base. Version 2.0 of Moodle is currently being developed as major step forward. Take a look at that. I use Moodle and I am very happy that people have donated their time and effort for this application.
First of all I'd like to second you I appreciate the time and effort people devote to this project and all other open source projects.
I took a look at the code on http://cvs.moodle.org/moodle and I assume this is the latest code base.
First of all I'd split all classes into their own file. If you take a look at the following url for example, you see one large file containing a whole bunch of classes:
http://cvs.moodle.org/moodle/admin/roles/lib.php?view=markup
If speed is an argument for putting all the classes into one large file, I'd use a compiler to pack the different classes into one larger file which can be included by one single include;
capability_table_base.php
<?php
abstract class capability_table_base { ... }
?>
<?php capability_table_base.php
// @requires 'moodle/admin/roles/capability_table_base.php'
class explain_capability_table extends capability_table_base { ... }
?>
The compiler should create a dependency graph for determining in which order the files should be included. This could be a quite simple tool written in either Java or PHP.
This would be the first step in improving the ease of understanding the code. What do you guys think of splitting the large files into smaller ones which are easier to read and understand?
In a second phase I'd create repositories for the data store, for the sake of separating the domain logic and the way the data is persisted.
Instead of:
$role = $DB->get_record('role', array('id' => $this->roleid));
I'd create a repository;
<?php
$role = RoleRepositoryFactory::getInstance()->getRoleById($this->roleId);
?>
This hides the way the role is stored and retrieved. The factory can use a DatabaseRoleRepository, or maybe an XmlRoleRepository or maybe even an SoapRoleRepository for retrieving the roles from a different service.
The RoleRepositoryFactory gets populated by calling:
<?php
RoleRepositoryFactory::setInstance(new DatabaseRoleRepository());
?>
Let me know what you think of these conceptual improvements.
I took a look at the code on http://cvs.moodle.org/moodle and I assume this is the latest code base.
First of all I'd split all classes into their own file. If you take a look at the following url for example, you see one large file containing a whole bunch of classes:
http://cvs.moodle.org/moodle/admin/roles/lib.php?view=markup
If speed is an argument for putting all the classes into one large file, I'd use a compiler to pack the different classes into one larger file which can be included by one single include;
capability_table_base.php
<?php
abstract class capability_table_base { ... }
?>
<?php capability_table_base.php
// @requires 'moodle/admin/roles/capability_table_base.php'
class explain_capability_table extends capability_table_base { ... }
?>
The compiler should create a dependency graph for determining in which order the files should be included. This could be a quite simple tool written in either Java or PHP.
This would be the first step in improving the ease of understanding the code. What do you guys think of splitting the large files into smaller ones which are easier to read and understand?
In a second phase I'd create repositories for the data store, for the sake of separating the domain logic and the way the data is persisted.
Instead of:
$role = $DB->get_record('role', array('id' => $this->roleid));
I'd create a repository;
<?php
$role = RoleRepositoryFactory::getInstance()->getRoleById($this->roleId);
?>
This hides the way the role is stored and retrieved. The factory can use a DatabaseRoleRepository, or maybe an XmlRoleRepository or maybe even an SoapRoleRepository for retrieving the roles from a different service.
The RoleRepositoryFactory gets populated by calling:
<?php
RoleRepositoryFactory::setInstance(new DatabaseRoleRepository());
?>
Let me know what you think of these conceptual improvements.
I had similar thoughts the first time I saw the Moodle code-base. But, rather surprisingly, it does not seem to matter.
That is a bit annoying for me, because I am quite interested in software architecture, and it would be more fun to have a cool design to play around with.
However, Moodle has always focussed on building tools that are designed to help teachers teach, and that are easy for anyone to install and hack on. I think Martin D got the priorities right. Of course, that is less fun for the developers, but I get plenty of satisfaction from working on something that teachers and students all around the world use.
You should also note that most of what Moodle is doing is not rocket science, in software development terms. It is just taking bits of text from the user. Sticking them in a relational database, and later displaying them as HTML. Also, Moodle is not one application. It is a wiki and a forum and a quiz, and these modules should be, and are, relatively independent. For almost all of these simple applications, transaction script is a perfectly good architecture, and one that is much easier to get started with.
That is a bit annoying for me, because I am quite interested in software architecture, and it would be more fun to have a cool design to play around with.
However, Moodle has always focussed on building tools that are designed to help teachers teach, and that are easy for anyone to install and hack on. I think Martin D got the priorities right. Of course, that is less fun for the developers, but I get plenty of satisfaction from working on something that teachers and students all around the world use.
You should also note that most of what Moodle is doing is not rocket science, in software development terms. It is just taking bits of text from the user. Sticking them in a relational database, and later displaying them as HTML. Also, Moodle is not one application. It is a wiki and a forum and a quiz, and these modules should be, and are, relatively independent. For almost all of these simple applications, transaction script is a perfectly good architecture, and one that is much easier to get started with.
I second tim on this one.
But tim, even though your explanation was bang on, wont it be more easier for anybody to identify, understand and play around if atleast the html and sql queries alone are seperated from core php code (here am leaving out the php code refactoring which is not necessary as you said) ?
But tim, even though your explanation was bang on, wont it be more easier for anybody to identify, understand and play around if atleast the html and sql queries alone are seperated from core php code (here am leaving out the php code refactoring which is not necessary as you said) ?
'hacking' is fine, but large institutions like Universities worldwide use Moodle, and 'hacking' is not the term I'd like to use in a conversation when I talk to my not-so-open-source clients. Imho, nice and clean code should always be in pursuit.
To be quite honest, I don't know enough of the Moodle project to decide to use either Transaction Script or a full Domain Model. For some parts of the code Transaction Scripts could be a better choice, and for other parts maybe a Domain Model.
The bottomline of my post was; there is no separation of presentation logic, domain logic and data store logic. That, together with very large hard to read files, is, to my mind the main issue of the code base.
I mean, I don't see a Model View Controller framework. Some plugins try to use some sort of MVC functionality by using a plain switch statement, others don't. Security needs to be checked within every file, HTML is mixed with database statements.
Do you agree with me, this is a major problem, and if so, what can be done to resolve this issue and what should be on top of the priority list? And how can I contribute?
To be quite honest, I don't know enough of the Moodle project to decide to use either Transaction Script or a full Domain Model. For some parts of the code Transaction Scripts could be a better choice, and for other parts maybe a Domain Model.
The bottomline of my post was; there is no separation of presentation logic, domain logic and data store logic. That, together with very large hard to read files, is, to my mind the main issue of the code base.
I mean, I don't see a Model View Controller framework. Some plugins try to use some sort of MVC functionality by using a plain switch statement, others don't. Security needs to be checked within every file, HTML is mixed with database statements.
Do you agree with me, this is a major problem, and if so, what can be done to resolve this issue and what should be on top of the priority list? And how can I contribute?
Dear Jorgen,
Yes, when i mean "if atleast the html and sql queries alone are seperated from core php code", you can see it was leading to the MVC pattern.
Regarding security,knowing about moodle for some time i dont think there is a compromise in security because of this architecture.
Yes, when i mean "if atleast the html and sql queries alone are seperated from core php code", you can see it was leading to the MVC pattern.
Regarding security,knowing about moodle for some time i dont think there is a compromise in security because of this architecture.
We're one of the largest institutions in the world using Moodle and although I might not speak that way to the vice-chancellor (um, who can read this if he happens to google it... oops) I have no problems using words like 'hacking'... just as long as they are combined with words like 'testing' and 'performance improvement' etc etc. 
Basically (puts on stupid accent) the Moodle codebase is a god darn mess and ain't no amount of theory gonna change that.
Any changes have to be at least somewhat incremental - a change that suggests 'let's throw it all away and rewrite the whole thing' won't work. However much some of us might like to do that, it wouldn't be helpful within the Moodle project at this particular stage.
There's a bunch of experienced developers in and around the core team, some of whom even have degrees and such and might be able to follow the latest buzzwords - hadn't heard 'domain-driven design' before, but I assume it's just yet another word about separating user interface from back-end code, probably with an attached detailed theory which is all so much guff which you might as well totally ignore around that basic (correct) principle. Anyhow most of those developers have a reasonable understanding of what code ought to be like, and how much Moodle isn't. These are the people making decisions about how to make Moodle better without breaking the whole project, which is why Moodle 2 is significantly improved architecture-wise compared to 1.9. Those decisions and designs are often really, really difficult - certainly much harder than 'if you were building a system from scratch to do what Moodle does, how would you do it' which is easy (*cough* Java *cough*).
I'm not really one to talk about this personally because I think reinventing the wheel is great and throwing away and rewriting entire projects sounds like fun. But with a major open-source project that is not always a helpful attitude and right now it certainly isn't the time for any such thing - Moodle 2 is limping gradually toward a hopeful release date and definitely any additional architecture changes are not appropriate. Maybe Moodle 3, though...
For a really quick summary - yes I think everyone knows the code is a mess, but the solutions are not nearly as easy as they might seem. And this doesn't always cause critical problems. I'm sure there are many parts of, say, Windows source code that are a horrific mess - in fact I've probably seen some of them, don't recall now - and it sells millions every week or whatever, and is generally quite reliable...
--sam
Basically (puts on stupid accent) the Moodle codebase is a god darn mess and ain't no amount of theory gonna change that.
Any changes have to be at least somewhat incremental - a change that suggests 'let's throw it all away and rewrite the whole thing' won't work. However much some of us might like to do that, it wouldn't be helpful within the Moodle project at this particular stage.
There's a bunch of experienced developers in and around the core team, some of whom even have degrees and such and might be able to follow the latest buzzwords - hadn't heard 'domain-driven design' before, but I assume it's just yet another word about separating user interface from back-end code, probably with an attached detailed theory which is all so much guff which you might as well totally ignore around that basic (correct) principle.
I'm not really one to talk about this personally because I think reinventing the wheel is great and throwing away and rewriting entire projects sounds like fun.
For a really quick summary - yes I think everyone knows the code is a mess, but the solutions are not nearly as easy as they might seem. And this doesn't always cause critical problems. I'm sure there are many parts of, say, Windows source code that are a horrific mess - in fact I've probably seen some of them, don't recall now - and it sells millions every week or whatever, and is generally quite reliable...
--sam
Thank you for your reply. Combining the word 'testing' with 'hacking' requires the code is able to be tested. I'm quite positive testing the major parts of the code is impossible. Writing good unit tests for the majority of the modules is impossible.
Domain driven design is focussing on what the application should be able to do, and less about focussing on the user interface or the specific datastore in use. This funcionality is expressed in a domain model which can be communicated to others easily, and can be tested easily.
Maybe I have not expressed myself clearly enough, but I was not suggesting to rewrite the entire code or throw away large parts of the code. I understand Moodle has a large user base and has a legacy design which can not be broken.
I was suggesting to introduce patterns like Repository for separating the datastore from the code. Instead of mixing domain logic and sql queries, I'd rather like to see the code refactored using that (and other) design pattern(s). By doing so, improving the code graduately.
So refactoring is in my opinion the key to improving the code, but for me the key question is: does this community feel the urge to separate the different layers (i.e. the domain logic and datastore logic)?
Domain driven design is focussing on what the application should be able to do, and less about focussing on the user interface or the specific datastore in use. This funcionality is expressed in a domain model which can be communicated to others easily, and can be tested easily.
Maybe I have not expressed myself clearly enough, but I was not suggesting to rewrite the entire code or throw away large parts of the code. I understand Moodle has a large user base and has a legacy design which can not be broken.
I was suggesting to introduce patterns like Repository for separating the datastore from the code. Instead of mixing domain logic and sql queries, I'd rather like to see the code refactored using that (and other) design pattern(s). By doing so, improving the code graduately.
So refactoring is in my opinion the key to improving the code, but for me the key question is: does this community feel the urge to separate the different layers (i.e. the domain logic and datastore logic)?
Oh, okay!
Well, I would not like to see SQL queries separated from business logic. Moodle attempted to do that already (get_record, update_record etc instead of direct database access) and it was disastrous for performance. I don't think there is any need for a three-stage separation. For each area I think we need a back-end (which handles business logic including sql) and a front-end (UI). And also this is PHP, so we can't expect to be pure about either part of that...
About testing, I did not mean unit testing (which exists in Moodle but is not very extensive nor reliable - large parts of code are not possible to test, but new code generally is - albeit it takes way too much effort to write tests, we really need an SQLite-based test system as was discussed earlier). I meant 'person' testing, including community use. For example my impression is that Moodle core is, generally, slightly more reliable than parts we have written here at the OU - in other words when something breaks, here, it's usually because we broke it. Not always but more often than not. We frequently run tests here with several full-time testers and they don't very often find bugs in core code.
As far as I'm aware, most parts of code that are being rewritten are done in a more separated manner (front-end UI scripts + separate back-end libraries), and many of the architectural changes are meant to aid better development along these lines; $OUTPUT allows to separate UI scripts into 'decide what to display' and 'actually display it' layers, for example, while $DB makes it easier to support unit testing. The code is still a mishmash because not all parts of code are included in that 'being rewritten' group. I think trying to rewrite all the bits that suck would be quite an effort and would also need significant co-ordination work. (Also bear in mind that anyone can write code, but it takes an expert to review it... Moodle.com only really has a few such experts and they have other things to do as well.) However that doesn't mean it would not be worthwhile.
Maybe a general 'improvement drive' should be considered for 2.1 or 2.2 or ongoing, in which specific areas can be targeted for a list of standard refactoring improvements (make full use of current recommended APIs; separate back-end and front-end; implement comprehensive unit tests; etc)?
--sam
Well, I would not like to see SQL queries separated from business logic. Moodle attempted to do that already (get_record, update_record etc instead of direct database access) and it was disastrous for performance. I don't think there is any need for a three-stage separation. For each area I think we need a back-end (which handles business logic including sql) and a front-end (UI). And also this is PHP, so we can't expect to be pure about either part of that...
About testing, I did not mean unit testing (which exists in Moodle but is not very extensive nor reliable - large parts of code are not possible to test, but new code generally is - albeit it takes way too much effort to write tests, we really need an SQLite-based test system as was discussed earlier). I meant 'person' testing, including community use. For example my impression is that Moodle core is, generally, slightly more reliable than parts we have written here at the OU - in other words when something breaks, here, it's usually because we broke it. Not always but more often than not. We frequently run tests here with several full-time testers and they don't very often find bugs in core code.
As far as I'm aware, most parts of code that are being rewritten are done in a more separated manner (front-end UI scripts + separate back-end libraries), and many of the architectural changes are meant to aid better development along these lines; $OUTPUT allows to separate UI scripts into 'decide what to display' and 'actually display it' layers, for example, while $DB makes it easier to support unit testing. The code is still a mishmash because not all parts of code are included in that 'being rewritten' group.
Maybe a general 'improvement drive' should be considered for 2.1 or 2.2 or ongoing, in which specific areas can be targeted for a list of standard refactoring improvements (make full use of current recommended APIs; separate back-end and front-end; implement comprehensive unit tests; etc)?
--sam
I think everyone with a solid software engineering background is a bit taken aback the first time they look at the moodle codebase.
I understand it's based on a PHP project started in 2000, so I wasn't expecting a real ORM layer or an attempt at MVC or anything, and I'm fine with procedural style, but even so there are some obvious shortcomings here.
My biggest issues are the long, 500+ lines of if ... else blocks (beautiful formatted I might add) which slowly and with great complexity and many nested if..elses build an output string, looking up data and running various queries along the way. To see an example of this madness take a look at the code that builds SCORM table of contents. (there are many other examples).
I think at the very minimum the core devs should develop some basic standards even if they continue with the procedural coding style. CRUD is a pretty well explored domain now, there are clear best practices.
At a minimum I suggest:
1)
gather data -> display data
rather than:
gather some data -> add it to some inline html -> gather some more data (over and over...)
2)
Picking a sane max # of nested blocks (refactor and functionalize), I suggest 3.
3)
Picking a sane max # of LOC per function / file. (refactor and functionalize)
If implemented, I think this will make the code easier to maintain and extend.
I understand it's based on a PHP project started in 2000, so I wasn't expecting a real ORM layer or an attempt at MVC or anything, and I'm fine with procedural style, but even so there are some obvious shortcomings here.
My biggest issues are the long, 500+ lines of if ... else blocks (beautiful formatted I might add) which slowly and with great complexity and many nested if..elses build an output string, looking up data and running various queries along the way. To see an example of this madness take a look at the code that builds SCORM table of contents. (there are many other examples).
I think at the very minimum the core devs should develop some basic standards even if they continue with the procedural coding style. CRUD is a pretty well explored domain now, there are clear best practices.
At a minimum I suggest:
1)
gather data -> display data
rather than:
gather some data -> add it to some inline html -> gather some more data (over and over...)
2)
Picking a sane max # of nested blocks (refactor and functionalize), I suggest 3.
3)
Picking a sane max # of LOC per function / file. (refactor and functionalize)
If implemented, I think this will make the code easier to maintain and extend.
Well done. You have spotted the other key point: that at lot of the design decisions in Moodle date back to before OO PHP even existed.
Moodle 2.0 is the first release where we an assume PHP 5+, and so it is the first release were serious use of PHP classes/objects is sane. We are also taking the change in major version number as an excuse to break some of the APIs, and so introduce more objects.
The problem with transaction script is that it is really easy to make a bloody mess that still works. Therefore, people do. However, that does not stop you from writing good transaction script code, and I have also seen some horrible messes made with OO code in my time.
If you read (perhaps between the lines of) the Moodle coding guidelines, you can see that a simple module code should look like:
view.php:
* gather data
* is there an action in the request? If so, process and redirect.
* else display view.
That code should be written in terms of functions in a lib.php file that contains:
* domain logic functions.
* database access functions.
Of course, more complex modules may have more that one script, and break the library file into more than one bit.
But, that is not where we are now with Moodle, so what is a practical way to sort out the mess that is currently there, without having to rewrite everything at once? Well, with any luck, I found a way to let us gradually separate out all the presentation code one module at a time: Development:Theme_engines_for_Moodle? - in a way that lets themes change the HTML that is generated without hacking other code. The core renderer is implemented in Moodle 2.0. However, there are not yet renderers for individual modules, apart from Workshop, which David Mudrak did.
(By the way, no one is forcing you to use the word hacking with clients who will not understand it, but the work has a long tradition in the Free Software community - actually from before Free Software was invented - and I will continue to use it here.)
Also, if you want to see the kind of code I write when left to my own devices, start browsing from http://github.com/timhunt/Moodle-Question-Engine-2/blob/new_qe/question/engine/lib.php and Development:Question_Engine_2. That probably does count as a domain model, and that part of Moodle needs it. In due course, the Quiz will use this new code. The quiz will remain transaction script, using the question engine as a library.
Moodle 2.0 is the first release where we an assume PHP 5+, and so it is the first release were serious use of PHP classes/objects is sane. We are also taking the change in major version number as an excuse to break some of the APIs, and so introduce more objects.
The problem with transaction script is that it is really easy to make a bloody mess that still works. Therefore, people do. However, that does not stop you from writing good transaction script code, and I have also seen some horrible messes made with OO code in my time.
If you read (perhaps between the lines of) the Moodle coding guidelines, you can see that a simple module code should look like:
view.php:
* gather data
* is there an action in the request? If so, process and redirect.
* else display view.
That code should be written in terms of functions in a lib.php file that contains:
* domain logic functions.
* database access functions.
Of course, more complex modules may have more that one script, and break the library file into more than one bit.
But, that is not where we are now with Moodle, so what is a practical way to sort out the mess that is currently there, without having to rewrite everything at once? Well, with any luck, I found a way to let us gradually separate out all the presentation code one module at a time: Development:Theme_engines_for_Moodle? - in a way that lets themes change the HTML that is generated without hacking other code. The core renderer is implemented in Moodle 2.0. However, there are not yet renderers for individual modules, apart from Workshop, which David Mudrak did.
(By the way, no one is forcing you to use the word hacking with clients who will not understand it, but the work has a long tradition in the Free Software community - actually from before Free Software was invented - and I will continue to use it here.)
Also, if you want to see the kind of code I write when left to my own devices, start browsing from http://github.com/timhunt/Moodle-Question-Engine-2/blob/new_qe/question/engine/lib.php and Development:Question_Engine_2. That probably does count as a domain model, and that part of Moodle needs it. In due course, the Quiz will use this new code. The quiz will remain transaction script, using the question engine as a library.
Dan hijacks this post to encourage Kyle to submit a patch in the tracker to tidy up those if statements in SCORM 
(or any other area of SCORM!!!!!)
Thanks for the encouragement . I'm currently working my way through the moodle dev course and will seriously consider cleaning it up once I have a better understanding of how moodle and the dev process (and the SCORM integration) works.
hehe - getting into the code and submitting patches can be a good way to understand how the dev process works! - just create a bug in the tracker, attach the changes you are suggesting - then (hopefully) someone will see the bug/review your suggestions and either apply them or suggest improvements etc.
(A lot of time could be spent tidying up code - especially in SCORM!)
(A lot of time could be spent tidying up code - especially in SCORM!)
Dan:
Please don't create bugs (there are more than enough); but do create bug reports in the tracker.... just create a bug in the tracker...
For a description of the dev process, follow the 'overview of the Moodle development process' link near the top of Development:Developer_documentation.
Jorgen, do you have some examples of code you have contributed to active projects so we can get look at how you would implement some of these ideas?
To be quite honest: no. I have never contributed to an existing open source project. The only thing I've contributed ever is my W3C Range implementation for Internet Explorer written in javascript.
http://jorgenhorstink.nl/test/javascript/range/final/class.range.js
I'll make a simple example which shows how I implement a domain model and different repositories for data storage and an application service layer which controls security and manages the transaction handling.
I'll keep you up to date...
http://jorgenhorstink.nl/test/javascript/range/final/class.range.js
I'll make a simple example which shows how I implement a domain model and different repositories for data storage and an application service layer which controls security and manages the transaction handling.
I'll keep you up to date...
These sort of question 'No support' and 'why doesn't use project XYZ use often comes from people that have no experience building large projects or don't understand the history of a existing projects.
Projects like Moodle often started when in a time when/where we didn't had these buzzwords and frankly it was and still often is not needed to use some new cool framework/buzzword method.
No offense Jorgen, but I have seen very often people come in to a open source project telling 'us' why we didn't use some new fancy buzzword method because it would have been so much better.
In fact, most if not all Open Source projects with a current active userbase will do just fine using there own methods of accomplishing things, look in google for Samba and thread usage, or check the TYPO3 mailing list for similar discussions.
cheers,
Ries
Projects like Moodle often started when in a time when/where we didn't had these buzzwords and frankly it was and still often is not needed to use some new cool framework/buzzword method.
No offense Jorgen, but I have seen very often people come in to a open source project telling 'us' why we didn't use some new fancy buzzword method because it would have been so much better.
In fact, most if not all Open Source projects with a current active userbase will do just fine using there own methods of accomplishing things, look in google for Samba and thread usage, or check the TYPO3 mailing list for similar discussions.
cheers,
Ries
I'm not offended that quickly...
If you've read my replies, you know I fully understand how complex it is to maintain existing projects. I was suggesting an iterative improvement of the code base, and it was not my intention to say: 'well guys, this sucks, you'd better start all over again'. At least, that was not what I meant, and if some of you read my post like that I'm sorry for not expressing myself clearly enough.
I'm not trying to introduce buzzwords. I just like to read insights of other people in the field. I like to read books and improve my knowledge about software architecture. Buzzswords have a negative connotation as if one would only use those techniques because they are 'hot'. Patterns and concepts are used to be able to communicate thoughts about the architecture of software.
Those 'buzzwords' I introduced, like Repository, can significantly improve the readibility and maintainability of the code to my mind, and if I'm right I gave a couple of reasons why such a pattern could be helpful. I only heard one argument based on reason: this community tried to introduce get_record etc, but it significantly slowed the performance. Well, that's an argument not to use a n-tier architecture. If speed is more important then readability and maintainability. To my mind hardware is always cheaper then fix hard-to-find bugs, but that's just a matter of taste.
I was just wondering why this active userbase does what it does. And how it does what it does. And I was trying to make people ask themself the question: is there a problem with the current code base, and if so, what can we do to improve it.
If you've read my replies, you know I fully understand how complex it is to maintain existing projects. I was suggesting an iterative improvement of the code base, and it was not my intention to say: 'well guys, this sucks, you'd better start all over again'. At least, that was not what I meant, and if some of you read my post like that I'm sorry for not expressing myself clearly enough.
I'm not trying to introduce buzzwords. I just like to read insights of other people in the field. I like to read books and improve my knowledge about software architecture. Buzzswords have a negative connotation as if one would only use those techniques because they are 'hot'. Patterns and concepts are used to be able to communicate thoughts about the architecture of software.
Those 'buzzwords' I introduced, like Repository, can significantly improve the readibility and maintainability of the code to my mind, and if I'm right I gave a couple of reasons why such a pattern could be helpful. I only heard one argument based on reason: this community tried to introduce get_record etc, but it significantly slowed the performance. Well, that's an argument not to use a n-tier architecture. If speed is more important then readability and maintainability. To my mind hardware is always cheaper then fix hard-to-find bugs, but that's just a matter of taste.
I was just wondering why this active userbase does what it does. And how it does what it does. And I was trying to make people ask themself the question: is there a problem with the current code base, and if so, what can we do to improve it.
Hardware cheapness is not always the solution. I'm not an expert on the hardware we buy here but I think it's basically fairly high-end commodity servers. I think our database server has 16 cores or something and 64GB RAM, but we'll be upgrading it fairly soon. Anyhow, I don't know the price when this stuff was new, but probably several thousand, let's say £5,000. Even when you consider we need a pair for failover, and four webservers too, and we do have to upgrade rather frequently as our usage increases so you might consider this a yearly cost... it's still not that steep - less than the total employment cost of one developer.
(Disclaimer - this is not my area, I don't know if there are any confidentiality or competition or whatever other reasons why I wouldn't be able to give out the real prices, so I want to emphasise, the number above is not the real price we paid for hardware. Entirely made up.)
Anyhow, what I'm trying to say here is, we get pretty much the fastest affordable servers, and a large Moodle installation still stresses them. (The main bottleneck is DB server; you can have as many webservers as you like, these will scale infinitely, but depending on the database system in use, it can be a challenge to scale database performance short of just buying a faster server.)
Moore's Law is great - every year processors get faster. But at the moment, every year our courses are making better and more extensive use of their websites...
Anything that reduced Moodle performance significantly would be a big problem for systems like ours.
In the case of get_record etc being problematic - by the way I should mention these are absolutely fine for simple use when you know you only need one record - this occurs when they would need to be done repeatedly in order to carry out a task. Let's take a course with 4,000 students (we have a number with more than that). Some of the cases where performance was a concern are where it was doing things like making say 12,000 queries (4,000 * N, say 3) in order to carry out operations - the equivalent of a fully-separated 'load student information' call. The typical pattern would be, do a database query which returns a list of users who meet some criteria or other, then do another query for each one of those users to check something - that type of thing.
Each one of those queries is very easy for the database to answer, but each one costs a network round-trip and whatever per-query initialisation the database does. Changing those into a single query has sometimes improved performance on that request by a factor of 100. Even if the scaling could be achieved, you are going to have to spend a lot on hardware to make up for two-order-of-magnitude speed decreases. At a time when budgets are tightening everywhere, I don't think any institution's going to be impressed if you suggest they might need a few million on hardware each year because that makes the code neater.
This isn't a theoretical example; there were a number of cases where this happened relating to number of students on a course, number of courses in the system, number of activities in a course, etc etc etc. I think our #1 record was something to do with the calendar view in an older Moodle version, where depending on the user, I think we managed to achieve over 100,000 database queries (and a few minutes of real time) to render a single web page. Anything that could possibly increase the number of queries proportional to any of those numbers above (students, courses, etc) is really, really bad news - and this is hard to avoid in a 'pure' separated system which hasn't been designed from the start to work that way.
Basically the community does need to improve the 'architecture' or lack of it in moodle, and the separation between various levels - some of which the changes in Moodle 2 do achieve. But we/they could also benefit from improving performance (if anyone has any new ideas to achieve that; again, there are some already in moodle 2) - and definitely we need to avoid making it any worse than at present.
--sam
(Disclaimer - this is not my area, I don't know if there are any confidentiality or competition or whatever other reasons why I wouldn't be able to give out the real prices, so I want to emphasise, the number above is not the real price we paid for hardware. Entirely made up.)
Anyhow, what I'm trying to say here is, we get pretty much the fastest affordable servers, and a large Moodle installation still stresses them. (The main bottleneck is DB server; you can have as many webservers as you like, these will scale infinitely, but depending on the database system in use, it can be a challenge to scale database performance short of just buying a faster server.)
Moore's Law is great - every year processors get faster. But at the moment, every year our courses are making better and more extensive use of their websites...
Anything that reduced Moodle performance significantly would be a big problem for systems like ours.
In the case of get_record etc being problematic - by the way I should mention these are absolutely fine for simple use when you know you only need one record - this occurs when they would need to be done repeatedly in order to carry out a task. Let's take a course with 4,000 students (we have a number with more than that). Some of the cases where performance was a concern are where it was doing things like making say 12,000 queries (4,000 * N, say 3) in order to carry out operations - the equivalent of a fully-separated 'load student information' call. The typical pattern would be, do a database query which returns a list of users who meet some criteria or other, then do another query for each one of those users to check something - that type of thing.
Each one of those queries is very easy for the database to answer, but each one costs a network round-trip and whatever per-query initialisation the database does. Changing those into a single query has sometimes improved performance on that request by a factor of 100. Even if the scaling could be achieved, you are going to have to spend a lot on hardware to make up for two-order-of-magnitude speed decreases. At a time when budgets are tightening everywhere, I don't think any institution's going to be impressed if you suggest they might need a few million on hardware each year because that makes the code neater.
This isn't a theoretical example; there were a number of cases where this happened relating to number of students on a course, number of courses in the system, number of activities in a course, etc etc etc. I think our #1 record was something to do with the calendar view in an older Moodle version, where depending on the user, I think we managed to achieve over 100,000 database queries (and a few minutes of real time) to render a single web page. Anything that could possibly increase the number of queries proportional to any of those numbers above (students, courses, etc) is really, really bad news - and this is hard to avoid in a 'pure' separated system which hasn't been designed from the start to work that way.
Basically the community does need to improve the 'architecture' or lack of it in moodle, and the separation between various levels - some of which the changes in Moodle 2 do achieve. But we/they could also benefit from improving performance (if anyone has any new ideas to achieve that; again, there are some already in moodle 2) - and definitely we need to avoid making it any worse than at present.
--sam
Jorgen,
it does what it does because of some reasons:
1) it worked already for a long time, no need to fix something that isn't broken
2) hardware isn't always cheaper, specially in open source projects where a fix to software that can be patched is cheaper then buying a Dell quad core XYZ server. it's not for nothing that companies start to virtualize more and more because of hardware maintenance cost. (check what amazon, google and MS is doing in that area...)
3) A lot of open source projects are database driven, what I mean with this that a lot of info can get obtained by the querying the DB. If for example extension/moodle adds field XYZ then you get this field back in some query and/or in a get_record API function.
4) DDD might slow down development because if you need an additional function in the core system of a open source project you need to go through the core developers explaining why you want it for your unique field, for example:
$roles = RoleRepositoryFactory::getInstance()->getRolesByFieldXYZ($this->myFieldXYZId);
(XYZ is a fiedl created by some module not part of core)
In that case you need to add that function getRolesByFieldXYZ to the core of the project, which is odd...
This would lead to having dependency injection as a must for DDD style open source projects (a MAYOR slowdown in PHP), in java much more common where DI is a stabilized solution.
I agree that it increases readability but what I see is that people strugly more with what to get back from a API (array or object and if so, what info does it contain). DDD Also slows down development because you need to define/create much more The FLOW3 people already building a DDD system for over more then 2 or 3 years, and the system doesn't do anything unless you tell it to do something. Go figure how much time it will take to make any other open source project in DDD correctl!
I am not against it for sure, but the consequences of doing any project in DDD rather then 'bare bone' must be well understood before starting, also remember.. PHP is a scripting language!
About speed, what sam is saying is completely true.
As a side note, looking at the type of workload that Moodle gives to the database I think that a Moodle sort of system would benefit using a OODB rather then using a RDBM.
Ries
PS: Check FLOW3 for template system/DDD/DI options, you might like that...
it does what it does because of some reasons:
1) it worked already for a long time, no need to fix something that isn't broken
2) hardware isn't always cheaper, specially in open source projects where a fix to software that can be patched is cheaper then buying a Dell quad core XYZ server. it's not for nothing that companies start to virtualize more and more because of hardware maintenance cost. (check what amazon, google and MS is doing in that area...)
3) A lot of open source projects are database driven, what I mean with this that a lot of info can get obtained by the querying the DB. If for example extension/moodle adds field XYZ then you get this field back in some query and/or in a get_record API function.
4) DDD might slow down development because if you need an additional function in the core system of a open source project you need to go through the core developers explaining why you want it for your unique field, for example:
$roles = RoleRepositoryFactory::getInstance()->getRolesByFieldXYZ($this->myFieldXYZId);
(XYZ is a fiedl created by some module not part of core)
In that case you need to add that function getRolesByFieldXYZ to the core of the project, which is odd...
This would lead to having dependency injection as a must for DDD style open source projects (a MAYOR slowdown in PHP), in java much more common where DI is a stabilized solution.
I agree that it increases readability but what I see is that people strugly more with what to get back from a API (array or object and if so, what info does it contain). DDD Also slows down development because you need to define/create much more The FLOW3 people already building a DDD system for over more then 2 or 3 years, and the system doesn't do anything unless you tell it to do something. Go figure how much time it will take to make any other open source project in DDD correctl!
I am not against it for sure, but the consequences of doing any project in DDD rather then 'bare bone' must be well understood before starting, also remember.. PHP is a scripting language!
About speed, what sam is saying is completely true.
As a side note, looking at the type of workload that Moodle gives to the database I think that a Moodle sort of system would benefit using a OODB rather then using a RDBM.
Ries
PS: Check FLOW3 for template system/DDD/DI options, you might like that...
Jorgen, you have initiated a debate that is worth having. It is always worth stepping back from a project and checking out if things can be done differently.