We (the Open University) are considering - this isn't definite yet - doing a rewrite of the forum system to:
- Optimise performance and clean up the back-end code (and database structures if necessary).
- Simplify options e.g. make tracking always-on unless disabled at site level.
- Improve the discussion interface with full AJAX implementation so you can reply on the same page etc., and replace current view modes with one AJAX-based view mode that combines the best features of flat and nested view.
- Add in a few of the custom features we already have (or which our users require) in OU forums which would be of general benefit to other users. For example we have a feature which shows teachers which students have looked at a particular discussion/post.
Since we will need it for 1.9, we might release a version for 1.9 (as an optional plugin), however it would obviously be for 2.x that we might try to get it into the core Moodle system.
Proposal PDF (written for internal use so may seem a bit irrelevant in places)
Note that this proposal is not a design document, it's sort of more like an idea document. Also, I have some more ideas that aren't in there like pluggable forum types... this is really early stuff that I put together in a hurry, an actual design will come later.
There is a link to an experimental forum system in the PDF. This is not the proposed system or anything and isn't within Moodle, so don't worry that it looks totally different to Moodle (also don't worry if it crashes . It's a totally separate forum I wrote some time ago that demonstrates some of the AJAX behaviour and the proposed view mode. Oh and it's running on my home DSL so be gentle and don't try to load-test it please
Anyhow - the point of all this post is - if you have any comments on the proposed 'discuss' system, please read (or at least skim) the PDF and then post here. Just to be clear, at this point I don't have any agreement with Moodle HQ to include this in standard Moodle, so if you hate the very idea with a burning fury, there is no need to panic just yet.
PS for those waiting for conditional activities development I'm supposed to be doing - no I haven't forgotten, that will happen soon, before any code is done for this new forum system
I really like the AJAX view mode you've described... (similar to Digg?). As well as the Expand all button, an Expand thread button is useful for displaying all the parent and/or child messages of a post quickly.
Actually, I wonder whether it would make sense to leave the forum as a simple module, serving the news and basic posts, and have your discuss (I would prefer a noun form: discussion) module replacing Forum+ and other enhanced forum variants. This would be parallel to having choice, feedback, and quiz modules. It would also give more flexibility in its development without having to worry about Moodle core.
I don't think duplication of behaviour (with different interfaces) is really a great thing so I wouldn't, in the long term, be in favour of having two forums. You get a code-bloat aspect if we add new things without removing old ones, which is a huge maintenance problem - for 'problem' read 'disaster' - so if this did go into core, I'm pretty sure the Moodle HQ developers would be keen to remove the old forum as soon as possible!
Also in terms of interface I'd like to make it kind of Forum- if possible! I.e. if I can I'd like to try to keep it simpler or at least no more complex than the current forum, especially for users with ordinary 'student' permissions. So it hopefully won't be the 'advanced' option even if there are a few more features. Not sure if it will be possible to live up to this, but we'll see...
The problems of existing forum integration with other parts of moodle core are part of the reason behind developing it as a separate module; for example, one way to imagine progress would be:
1) we develop it completely independent of forum as a 1.9 contributed module.
2) we port it to 2.x, moodle HQ like it, and it gets incorporated into moodle 2.x - but at this stage it is just another module ie forum still exists.
3) in moodle 2.y (y>x) other developers including Moodle HQ contribute further tweaks and improvements until the new module is considered completely ready. an upgrade converts existing 'forum' instances to the new module and changes the hooks for things like course news to use the new one. The old forum code is removed, but might remain available as a contrib extension for any who still prefer it.
I'm not saying they would do that but it's possible that this kind of transition process could be provided if necessary. Of course another possibility is that it just stays as contrib for both 1.9 and 2.x until such time as it's considered ready for core. And yet another possibility is that Martin hates it and it stays contrib forever.
I would encourage you to pursue your idea of forum types as plugins from the beginning. That would allow us to have simple and more complex forums independent, letting users decide what they need for a given application, just like we select type of assignment nowadays. There is no way for one forum module to suit all.
This is an excellent proposal. I particularly like the uber-cool purple backgrounds to the section headings on the pdf document .
I think that it would be good to release it for 1.9 -- that way you'd get community feedback before 2.0 is released. And "discussion" is probably a better name than "discuss".
This is an excellent proposal. It's good to see improvements to the core capabilities of Moodle being addressed. Next stop the Assignment module?
Frankly, I did not have the resources to take Martin up on his offer (see http://tracker.moodle.org/browse/MDL-14993) but before building new, wouldn't it make a great deal more sense to specifically examine use of an existing php forum and then share the results of that study with the community?
On the other hand, if OU has already done a cost/benefit analysis it would certainly weigh in as far as community support if it was shared......
I think taking something that is already working well and integrating it with Moodle is a good concept, but the practicalities are quite different.
At a low level, we re-use a range of open-source libraries in Moodle (just check out the lib directory for examples) - but to implement a "tool" such as a discussion forum becomes quite difficult (as I found with my experience and the Blog module!)
Most good discussion forum packages that I've seen are written as stand-alone forums - they handle authentication/user profiles/file uploads/spam filters/glossary filters using their own internal architecture. To replace/integrate all of these things to use the "Moodle Way" is a HUGE effort. You start with lots of duplication in code too. For example - error handling, - Moodle has it's own error handling routines that prints errors to the screen with customisable CSS - any new forum system will use it's own with completely different CSS styles and naming...and we all like consistency, so we'd want to make the errors from the "forum" look the same as standard Moodle errors right? - more work......what about File uploads? - well, we'd like to use the new file API right?....what about filters? - well, all the normal Moodle filters should be used.... What about database routines? - it needs to run on Oracle/Ms Sql/Postgres etc etc... This work would go on and on, until the code base looks absolutely nothing like the original code, and would most likely be a lot cleaner/easy to read/more secure/more scalable if it had been written from scratch!
With all this customisation to the external forum software, when the "external software" is updated....It would be a crazy job to try and integrate any of those changes into Moodle...
From my limited experience - my vote would be to stay away from trying to integrate another open-source forum with Moodle, but keep our own/rewrite our own.
Of course, to write a forum system from scratch you have to deal with authentication and database interfaces and rendering HTML forms and so on, and as Dan says, Moodle already is a platform that provides all these facilities, so actually writing the forum bit is pretty easy - and if you want it to it integrate slickly with the rest of Moodle's look and feel, building it from scratch will give the best results.
There is another reason why a generic forum system does not fit with Moodle. That is, Moodle is an educational tool. This means that you have students and teachers, and to some extent, teachers need to be able to manage the discussion process. Also, the concept of grading contributions is strange in a generic forum system, but in Moodle, sometimes teachers want to grade student's contributions and have that appear in the gradebook. Of course other forum systems have moderators and ratings, but I suspect that a system built from the ground up to work in an educational context will be subtly different. (A similar argument answers the question 'why does Moodle have its own wiki system when you could run wikipedia?')
I think you're dead right -- look at the mess that the wiki is in; 'ewiki', 'Nwiki' both try to meld existing code into the Moodle system and end up in a mess. It's no surprise that the wiki is being rewritten for 2.0.
I actually haven't thought much about the pluggable type idea - I threw that in because it seemed sensible. I think it will emerge mostly because one part of this plan would be to improve code quality by highly separating the back-end from front-end; if we do that then it will be easier to write different front-ends. (Maybe?) When I do an actual design I will obviously think about this more, based on ideas in this thread etc... it needs to implement at least the Moodle standard types, so I would assume that if I work out a plugin architecture that handles that, it's a good start.
I agree with others that trying to integrate an existing forum into Moodle would be (no offence to the person who suggested it) a terrible idea for three reasons:
- Integrating an existing forum with Moodle is going to be very difficult because it would have to meet all Moodle's infrastructure and user interface requirements - that would likely be much more work than creating one from scratch. Most PHP forums already include about half of what Moodle does (user management, database management, HTML editor, smilies, etc). Ripping all this out - in a way that we could still upgrade with new versions of the external forum program, and for any security fixes it requires - would be an absolute nightmare and probably several times more work than writing a new forum from scratch.
- Existing forums often have too many features. A forum in Moodle needs to be suitable for student use, meaning that it shouldn't be more complicated than the existing Moodle forum. Also, if there is any prospect of changing forums, the new one should work in a similar way to the current one (same terminology etc) even if it's a bit different interface-wise.
- Most existing PHP forums aren't very good either! (This is not a great argument as obviously we could find the one that was, but.)
That kind of 'er, there's a link and you don't have to log on again' integration works perfectly well, and won't take many months of effort to integrate and maintain. Just doesn't give a consistent/integrated experience.
PS The standard Moodle 1.9 wiki is built on (munged from) an existing wiki. That's only one of the reasons why it is such a horrible mess, but it's an important one
Actually, I hope not because the current assignment does not have any variations. The individual assignment types are de facto separate activities and you cannot switch between them as you used to be able to in version 1.6. I do hope that the different types of Forum are not extracted to independent entities as in the case of Assignment.
it's great that you are interested in taking on this. I suspect that you can do a lot more refactoring in-place than you suspect
One of the tricks of writing a new implementation from scratch is that when planning that, we tend to forget all the little features and special cases that make up a finished product. So it definitely looks easier to write it from scratch, because a lot of the complexity is forgotten (only to be re-added later). In other words -- it's common to underestimate a rewrite (ask the mozilla folks about that!).
Using a good scm helps when you're taking on a bit refactor -- in fact, you can get quite aggressive in the changes you make, and roll forward in short steps, with commits at each stage. It's a different programming style -- a lot of fun, and I think it often leads to much better code.
Candidly, I do agree with your comment about estimation. I won't comment any more about that here because there might be OU folk reading. Suffice to say that I do genuinely believe a rewrite will take the same or less time than trying to make incremental changes, and with much better results - however the former part (same time) might possibly be because if we don't redo it from scratch I will probably farm it out to someone else on my team rather than doing it personally, which means it will take at least twice as long...
it's better to write completely fresh code
You do have to consider that when you do that, you replace debugged code that's been in place for ages with fresh code -- which means a whole lotta new bugs.
I know my new fresh code looks great when I first write it, but after a year you'll see lots of little patches were needed to fix a problem there, and an oversight here. Sometimes I think I can write perfect code, but it's rare for me to get that drunk (MoodleMoot Mojitos?)
Good code ages well. New code is like new wine, even if it has the promise in it of being good, it'll take a while until it's actually good.
And old code doesn't rust. In particular code like mod/forum, that's been getting active maintenance, might look messy but has a lot of work put in it. Yes, it'll benefit from streamlining here and there, but if you throw it out, you'll be losing a lot of well debugged code that supports many corner cases you weren't thinking of.
Just think of all the conditionals around the different forum modes, groups and groupings modes and the various cases of membership to various group(ings). All that case matrix, combine it with the message digest option and ASCII vs HTML email. Throw in per-course and per-user language packs. And you're just getting started in the logic for posting emails in forum_cron(). None of that complexity is discussed in the (otherwise pretty good) spec PDF for example.
The situation is different from where we were with mod wiki, where I have to agree a full on rewrite was needed. (Which rewrite? Well, that's a different question ).
For some areas I am definitely planning to lift the code straight from current forum with minimal changes. (Backup/restore, which also wasn't on my PDF.) And for other areas I'm sure I will do the same with smaller chunks of code, functions, etc. Finally for other areas, I will be following through the code of the existing one to make sure I get things right like conditions.
Good news though is that since this (assuming it goes ahead) will be developed as an independent module, there's no need for Moodle to take a risk on including it until the code has already been running in our installation for a while and hopefully also tested by other people from contrib. (Our QA staff and then students should do a good job of testing the basic elements, but for the parts we don't use, testing elsewhere would be a big plus.)
I agree that bugginess is not a major problem of the current forum (there are a few bugs: the way 're' is in a random language depending on who replies is definitely a bug, for instance). But it has other problems and the code is generally a bit manky - I think starting from scratch is definitely going to be an easier way to work.
Using a good scm helps when you're taking on a bit refactor -- in fact, you can get quite aggressive in the changes you make, and roll forward in short steps, with commits at each stage. It's a different programming style -- a lot of fun, and I think it often leads to much better code.
Martin -- can you expand on this? What do you mean by 'scm' and 'bit refactor' ?
A good example was the accesslib refactoring. I would not have been able to do that without git or a similar scm (perhaps mercurial).
Forums alone are interesting, but also forums could be applied to other activities, some examples:
- Book/lesson: To allow students/teachers to post questions on each page basis,
- Database: Each record can be commented now, but does not provide the functionality of forums, first there is no mail or tracking
- Assignment: As it has been discussed many times, it should be interesting that after a given date, all the students could see and maybe comment on it, using a forum?
- wiki pages....
Note 2.x. I agree this might not be a good idea for 2.0. (Although that partly depends on how long 2.0 is delayed.)
If we get this released in contrib for 1.9, then there will be plenty of time for people to look at it and make decisions about including in a later version. At this stage, I'm just hoping for some kind of opinion from Martin D etc. that, if it turns out good , this could be included into core at some point.
One thing still on my wish-list would be the ability to save a draft of a post, so user can return and post later.
* the ability to subscribe to threads, instead of the whole forum.
* the addition of a 'References:' header to the mail out of the forum posts, that holds (ideally) the unique identifiers of all the parent posts (to build the thread tree) or if that is too cpu intensive, at least the unique identifier of the parent post.
That would allow people like me to track the threads where I've participated more easily, and specially to easily highlight messages that are replies to my own postings. Which is a big plus if you are a PHM here at moodle.org and try to help people in dozens of threads across several forums .
I'm currently trying to get something similar to the second request by fetching the RSS feeds of the forums, munging them into a NNTP article (with a unique ID built from the URL of the post, I'm yet to find a way to get the parent URL of the post, since this is not available in the RSS feed) and then inyecting them into a news server at work. Then I use Gnus to read the resulting news feeds, which allows me to do what I ask for above. Quite a bit of a setup (with some custom perl code) , isn't it? I'd rather have the forum code to all the work for me (and do it right, instead of the hackish patch I'm currently using )
course-name forum-name post-subject
course-name forum-name truncated-post-subject
I have hacked all our Moodle instances in every version we use to allow the 3rd form, so we can identify which post comes from where (each of our courses, a few hundred of them, has news and participant forums, so it was quite confusing).
As part of the hack, I added a field with short forum name on forum's config page, and that short form (if defined) is used for mail subjects. In case of courses, course id could be used as alternative to short name.
Add to that an option to select one of few styles of mail subject, like
even better a template for subject, sth like
Note that those settings should be at site level IMHO.
It would be difficult (impossible?) to implement NNTP using PHP, and also would be a lot of work anyway Of course it could still be done e.g. in Java with connection to Moodle's database... Anyhow, seriously this is not going to happen, but it would be pretty cool for those who prefer using a client. Maybe one day, huh?
As for threading headers, In-Reply-To and References [parent post only] both appear to be sent by the current forum already? I just checked.
I actually haven't thought much about the pluggable type idea - I threw that in because it seemed sensible.
Another reason for pluggable forum types is a possible Poll-Forum Type. This is a combined Choice/Forum that allows students to:
answer a poll question
give a reason for their answer
later view the poll and reasons of the class/other participants
It is very powerful educationally because it structures data in a closed format (quantitative), but then allows open-ended explanations and expansions (qualitative). It also shifts nicely between individual thinking to group comparison and group commenting.
If you want to make such a forum type, it would put your name in a permanent listing in heaven. If not, I hope your plugin system will allow somewhat easier coding.
It's a plan for a bored day, and haven't had one in years, so I don't know when it'll happen but I do like network protocol programming, and Jon Udell showed -- back in '99 -- fairly good examples of how to do it in his Practical internet groupware book, a favourite of mine.
(this is getting offtopic though!)
One additional thing not mentioned yet (or I missed it) - the option to be emailed immediately when someone posts a reply to a message you have written. I find that the bug tracker, Slashdot and other systems with this feature are far more involving than moodle.org's forums, where I've often forgotten about posting a message and have only noticed a reply weeks or even months later.
the forum post delay can be skipped when posting to 'News' forum. For normal forum discussions, the delay is a fantastic flame-retardant feature, and a big reason why moodle.org is the least flamewar-laden discussion forum/mailinglist that I participate in
How does that work? If you have immediate feedback, it's really easy to get into staccato replies, 2-scorpions-in-a-sock fights over a semicolon.
In moodle, however, you have 30 minutes to edit your post so you can hit post, and 5 minutes later go "hmmm, maybe that was a bit too much", come back and edit it before it's sent. Also, with those 30 minutes, a tight conversation still has a 60-minute "full roundtrip" between my message, and receiving a reply via email. Plenty of time to cool off
There are some interesting writings on this - Clay Shirky wrote a bit about these dynamics - http://shirky.com/writings/group_enemy.html and http://www.shirky.com/writings/group_user.html . A little known fact is that the notoriously flamewar-y debian-devel mailing list has successfully been playing with automatic hidden delays in posting new messages in threads that are too active.
So... you knew it was coming: it's feature, not a bug.
I think you're dead right about the delay, but the point about a specific email about a reply to one's post still holds - I haven't the time to wade through all the forum digests looking for the right ones, so it would be good to be notified about it, even with a 60 minute time-shift.
Fascinating links, BTW - I'll be reading those on the train later.
Quite right. You may be interested to hear that I have a special "didactic" use for that 30 min. delay. In my English language classes on moodle I have gradually built a fairly large "Glossary database" of common mistakes made by my students. When they write and post a message to the course forum, I advise them to re-read their message carefully and to scan it for any hyperlinked words to the Grammar/Vocabulary glossaries. Whenever they spot a mistake, the Glossary popup window gives an explanation, and they can edit their message and correct their mistake within the 30 min. delay.
How do I add words/expressions to the Grammar & Vocabulary? Let's take for instance the infamous "I am agree" mistake (based on the French syntax of the equivalent expression). In my Grammar glossary I have created an entry for "I agree", and within that entry, have added "I am agree", "I'm agree", etc. as Keywords, so any of those mistakes in a forum message is automatically hyperlinked to the "I agree" entry...
However, the suggestion of subscribing only to responses to your own posts, rather than to all new forum posts, is an interesting one. I've got that option turned on for, er, I think, Engadget; it works extremely poorly there but I think that's because their comments are fundamentally broken in almost every possible respect. (Except the fade-out rating system. That's kind of cool.)
A related, perhaps simpler possibility would be subscribing to a single discussion (or discussions) rather than the whole forum.
But I wonder if these aren't more relevant in casual web-forums than in a forum used for teaching? I.e. perhaps not needed...
We are keen to get our students to come up with original thought and not participate in a "me too" or "that's what I was going to say" type discussion. Now I realise that we can set up a forum so that students have to post before they can view other's postings but unfortunately the 30 minute edit period in this case works against us as it enables students to post (possibly evn just an single word!), read everyone else's post and then edit their own post to come up with something that is, well frankly not their original work.
Is there a universal change that can be made for the 30 minute thing or is it just part of how Moodle manages these postings?
Were you planning, or is it possible, to implement AJAX ratings?
For example, it would be great if ratings could be saved to the server as they were selected, rather than having to press the "Save my ratings" button, then getting redirected to a "Ratings Saved" screen, then back to the forum.
[That said, I'm not sure why we don't use forum ratings here. I think they can surely lead to some fantastic opportunities for name-calling, virtual bullying, and arguments! We should totally turn them on everywhere. And come up with amusing names for the scale points.]
Just to reiterate, this is an internal proposal as well as this public one; I'm hoping to do this, but it hasn't been approved by our management yet. So don't anyone get too excited about the prospect.
PS don't get me wrong, despite the second paragraph - or perhaps genuinely due to those reasons - I really do like forum ratings...
It is also helpful if you are following posts by a particular author.
Just by the way, the existing (in 2.0 HEAD builds) completion system lets you automatically 'tick' a forum if the user makes N posts, i.e. if you want users to make 5 posts, you can set it to track this for you. It is then possible to see a list of all students who have/haven't done this. Students can also see that the forum becomes 'ticked', meaning they have contributed sufficiently.
However, there is of course nothing to stop students making blank or silly posts, and this system provides no way to automatically assign a grade (probably just as well, given the blank/silly post thing), so it may not solve your problem. This is intended more as a rough indicator of participation, mainly for the student's benefit, than for any assessment. Just mentioning it.
Typically, we will not have but one post per person within a single discussion (I still think of these as threads, not moodle language). Therefore it doesn't help to be able to view a list of individual contributions. The scope is too limited.
Just mentioning it as an idea that would be useful.
I would like to add the request for the forum post sort order to be controlled by an admin setting. As far as I can tell it is hard coded to always sort posts by the last reply. This is not always desirable.
For example, on my site I use the "News Forum" on the front page to post timely news. I want to sort from the newest post to the oldest post. I show the three newest posts on the front page; however, the users can see all the front page news posts by clicking a menu choice. If a user comments on an older news post it automatically gets put back at the top on the front page. This is not the result I would like... to be adding old news back at the top simply because someone commented on the post. For this reason, I have to turn off commenting on this forum. Also if I go back to make to edit an older post, it gets sorted back to the top again.
There has been several threads in the forums on the sort issue. So if you are going to rewrite the code I would like for your team to consider this feature.
Possible sort options could be:
- Chronological by initial post date (ascending or descending)
- Chronological by reply date (ascending or descending)
- Alphabetical sort on the topic (A-Z or Z-A)
Does anybody really need the 'ascending' date options, or a Z-A sort on topics? Seems to me those options don't have much use. (Fewer options = better software!)
The question is whether this should be a fixed property of a forum, a view option for the user (thru popup), or both, with the property being the default sorting.
>>We already have - maybe this is an OU custom change - the ability to let users sort by any column,<<
My suggestion was to allow the forum sort set by the admin or teacher in the course settings based on how the forum is to be used.
>>Does anybody really need the 'ascending' date options, or a Z-A sort on topics? Seems to me those options don't have much use<<
The most usefull forum sort option would be a date sort on the original post date with the newest at the top. This would allow the admin or teacher to configure the news forum posts to act somewhat like blog posts. This is nice when you are showing X number of news forum posts on the course page. You don't want an old news item moved back to the front page just because someone comented on it.
As far as a alpha sort on the subject, I have used this in some forum software to fix the order of posts. I have done this by prefixxing the subject with something like "1-", "2-" excetera. I did this to make a forum look more like a FAQ and I wanted to have the questions in a particular order. Maybe I am just trying to make a forum do too much
Thanks for your considerations!
We will need to have our 'read by' feature which currently is basically for tutors (ie restricted by capability due to privacy implications), I guess we could support a 'count' feature that everyone can see - that would be a nice interface actually. (I.e. display the count - if you have the permission it also becomes a link you can click on to see the actual names.)
Note that this information is not truly available per-post; it is really per-discussion (as people normally view the whole discussion at once, not a single post). Also, it expires after a fixed time (default 14 days).
It would be possible to implement this by storing a separate number like you thought, but I agree with Martin's suggestion. Since one of my proposals is removing all the current options that exist to turn read tracking off [except at server level if admin really needs to save performance], this would fit fairly naturally. There is a question about retrieval performance, but I think that can be resolved, and overall I like it better than storing yet another piece of data about people's reading habits. (I.e. viewing a forum page already does at least two database writes, best not to add more without a really good reason.)
In addition, I think the number 'how many different people have read this discussion' rather than 'how many times have people read this discussion' is probably more useful in general.