Forum reply by e-mail

Forum reply by e-mail

by Andrew Lyons -
Number of replies: 23
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi all,

One of the projects I've worked on recently is the ability to reply to forum posts by e-mail, and I'm happy to announce that a prototype of the functionality is now available. This request was originally raised in MDL-3714.

I'm still working on documentation, and some of the language strings will need adjusting, but the prototype is otherwise fully functional.

To access the prototype site, you will need to create your own account at http://prototype.moodle.net/verp, where instructions on how to use the prototype system are provided, along with some suggestions on other things to try.

The system I've conceived allows for much more than just e-mail to forum replies. There is an extensive API which be used to create a message handler for almost anything within Moodle. As an example, the above prototype site also includes an e-mail to private files feature. The full details can be found in the parent issue, MDL-47194.

The initial developer documentation is available at https://docs.moodle.org/dev/VERP, with API documentation available available at http://prototype.moodle.net/verp/apidocs/classes/core.verp.handler.html.

There is one caveat that I have discovered - when using Google Mail, and replying to an e-mail whose Sender is an address that is one of your verified e-mail aliases, Google ignores the "Reply-To" header. I'm trying to find a solution for this issue, but without success so far.

Andrew

Average of ratings: Useful (7)
In reply to Andrew Lyons

Re: Forum reply by e-mail

by Joseph Thibault -
Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Andrew, fantastic work. A few questions:

1. will their be a "reply above this line" or something similar to prevent discussion posts from including previous messages or email signatures?

2. at some point will we nix the confirmation email from Moodle letting users know that their email has been accepted?

Average of ratings: -
In reply to Joseph Thibault

Re: Forum reply by e-mail

by Mark Andrews -

Hi Andrew,

Great to be able to test this out! - worked a treat.

Would be helpful for the documentation has some kind of process diagram that shows what notification the user gets when posting to a forum or uploading a document. 

  1. So when I reply to a forum post by email I get two back; a 'process' confirmation and then the post - could these possibly be rolled into one for the poster? - this would make the whole thing more elegant. 
  2. When a message fails (due to using a non-registered email account) it would be good for that message to give possible explanations to the user as to why it might have failed.

I also agree with Joseph's comment 1 - that would be great and also signpost to the user that they can reply via email.


All good stuff though.


cheers

Mark


Average of ratings: Useful (1)
In reply to Mark Andrews

Re: Forum reply by e-mail

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Mark,

Yes, I agree - documentation is definitely on the list. I've been getting the code to a working point and debugging so far. The API docs are pretty complete now, but I would like to start putting some diagrams together too.

1) I'm not sure at this stage... the confirmation e-mail is generated by the higher-level API, but I may move it such that the handler sends the message. Even then though, it is very difficult for us to combine the two into a single message at this stage. That's because the outbound messages are sent by the forum cron. We store a single flag for whether a message has been sent -- in the forum_post table. Each post knows whether it has been processed for e-mail. We do store a message_read flag too, but I think that is only respected if the user has read tracking enabled. As a result, we don't know whether to send the user the e-mail in all situations. It's something I'll look into, but I need to get the rest of the API sorted first.

2) Yup, again I agree. The current language strings are placeholders and we'll work on those before release.

Andrew

Average of ratings: -
In reply to Andrew Lyons

Re: Forum reply by e-mail

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Perhaps we just need a user preference in messaging somewhere to turn the confirmations on and off, with a link in the footer of all emails to turn them off quickly.

When you turn them off you'd get one last email containing the description of what happened and a link to turn them on again.

Average of ratings: -
In reply to Martin Dougiamas

Re: Forum reply by e-mail

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Maybe that's the best solution,

One of the reasons I added the e-mail was as a confirmation of the changes. If someone did manage to obtain the token for a reply, then they could send as many replies as they like. If the user was not subscribed to the forum/discussion, then they would not be aware that they were posting e-mails.

As another countenance to this, I wonder whether we should set a validity period - perhaps of 1 week or so?

Andrew

Average of ratings: -
In reply to Andrew Lyons

Re: Forum reply by e-mail

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

I'm currently modifying things such that:

  • there is now a default address validity which can be changed by an administrator on a per-handler basis;
  • the e-mail reply address is included in the footer of outbound forum e-mails;
  • the footer of all success e-mails include a note to inform users that they can turn off the message in their messaging preferences.

Note, because the confirmation message is sent using the message API, it was already possible to disable this message. Huzzah for free wins smile

Andrew

Average of ratings: -
In reply to Joseph Thibault

Re: Forum reply by e-mail

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Joseph,

Glad you like it smile

1) I'd really like to get that in, but it's hard. Every e-mail client seems to do slightly different things when it comes to replies. The "Reply above this line" idea is probably the easiest to implement, and it's something I'll try and do if I'm allocated any further time on this before release. I don't see it as a deal-breaker though. It's improvement that could theoretically be added to the Forum handler at a later stage if required, but I would really like to have it in there.

2) I've replied to this query in my reply to Mark below.

All the best,

Andrew

Average of ratings: -
In reply to Andrew Lyons

Re: Forum reply by e-mail

by Danny Wahl -

I'd like to also express a desire for the ability to at least try to strip out replies - from a usability perspective.

I have to use both SpiceWorks and JIRA and SpiceWorks has no 'filter' for previous messages- it means that every message contains the entire dialog + new content creating horribly long and near incomprehensible threads of emails with the majority being duplicated content.  Compare that to JIRA or github issues where you only see what's new both in the email and in the discussion online.

As far as I'm concerned, for the sake of usability of students and teachers, without there being a 'filter' to strip quoted material we would in all likelihood end up turning off this feature.

I hope I don't sound like I'm beating you up for your work- this is great!  I can't begin to tell you how many messages get set to our mail server because people intrinsically try to reply to a forum post via email and no-reply discards it.

Average of ratings: Useful (1)
In reply to Danny Wahl

Re: Forum reply by e-mail

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Danny,

As I've mentioned I'd really like to do this. The problem is working out where to split things. It's not an easy task, and I haven't yet found a PHP library to do so to lighten the load.

I wasn't aware that Jira did this yet, but I know that GitHub does not. What GitHub actually does is to include the entire response but collapse it. As an example, see my reply-by-email comment in https://github.com/yui/yui3/issues/1336#issuecomment-28349568. Gmail also collapses the reply section of the response, and perhaps this is something that we should look at too.

There have been a couple of suggestions on how we could remove the response though:

  1. work out where the reply ends and the original begins, and chop, however:
    1. different mail clients behave very differently when it comes to quoting replies;
    2. different languages could potentially make this harder; and
    3. people who bottom post (as e-mail was originally designed) will lose their replies.
  2. add a 'reply above this line' type comment, however:
    1. again, this is language dependent; and
    2. we may encounter issues with HTML vs. Plain text.
  3. add a non-language dependent marker (e.g. ========================), however:
    1. this probably requires a description too;
    2. it prevents people using the same marker in their reply; and
    3. we may encounter issues with HTML vs. Plain text.

If you have another suggestion, we can look into it. I've put down this project for a few days to give people a chance to give feedback, and I'll be picking it back up again later this week so I'll try a few different approaches then.

Thanks for the feedback,

Andrew

Average of ratings: -
In reply to Andrew Lyons

Re: Forum reply by e-mail

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
I think everyone would like to see some intelligent parsing of the reply to remove quoting.

If anyone can find robust code or an algorithm anywhere for this (in any language) please it to Andrew here or in MDL-3174.
Average of ratings: -
In reply to Andrew Lyons

Re: Forum reply by e-mail

by Danny Wahl -

Hello Andrew, I suppose I was being a bit semantic, you are correct github just hides it- but  whether we hide or strip it will still require the same type of parser.  I would post this in the tracker but I'm currently unable to access JIRA (it's a China thing- not a moodle thing):

here's the email_reply_parser that github uses: https://github.com/github/email_reply_parser

and here's a php port that someone made: https://github.com/willdurand/EmailReplyParser

Average of ratings: -
In reply to Danny Wahl

Re: Forum reply by e-mail

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Danny,

Thanks for the link to the github library and it's port. I'm not sure whether it will suit our requirements for a couple of reasons:

  • it's only designed for plain text e-mail, and not HTML mail; and
  • looking at the code, it inspects the text and looks for different strings which are frequently used (e.g. Re: foo'). It currently only supports three languages from what I can tell.

We could pop it into the code pretty easily (there's a handy space to do so), but I'm concerned by the number of caveats, and the lack of language support. One of the known issues is that it does not support Gmail for example.


Actually, it is more than just a semantic difference and it does make a difference to how we do things:

  • If we cut and remove content at the appropriate location, then it is gone. We don't store it in the database and it is not possible to recover it if it has been cut in the wrong place; conversely
  • if we collapse, and we choose to do so in the wrong location, then it is recoverable and nothing is permanently lost.

There's also a difference based on the point at which we make the cut:

  • If we make the cut when we receive the e-mail then we store less in the database, but at the expense of a permanent change.
  • If we make the cut at display time, then we have a recoverable content and we can change algorithms to retrospectively improve the cut point. However we store more in the database and we add more processing at display time.

Ideally I'd prefer to do the latter, but I don't think it will necessarily be easy. I imagine we'd have to introduce a new text format (in addition to HTML, Plain, and Markdown) - e.g. Reply. We'd then filter those differently:

  • When displayed on screen we would put the hidden content behind a collapse span; and
  • when sent by e-mail, we would cut.

Andrew

Average of ratings: -
In reply to Andrew Lyons

Re: Forum reply by e-mail

by Jacqui Sciancalepore -

Hi Andrew

So glad to see a prototype of this.

It worked quite well at first, however I've been testing it with a colleague and we are still waiting on her replies to come through. It been over 15 minutes, and she hasn't received any emails confirming or denying her post, not sure if this has to do with having a bigpond account.

I agree with the others that the confirmation emails do tend to crowd up the inbox, the possibility of being able to turn this off would be a benefit.

Otherwise great function. When would be expect this to be live and available? 

Regards


Jacqui

Average of ratings: Useful (1)
In reply to Jacqui Sciancalepore

Re: Forum reply by e-mail

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Jacqui,

Thanks for trying out the new functionality and sorry to hear you've hit some issues.

I can see your colleagues e-mails waiting for a pickup, but they're being rejected because there's something wrong with the e-mail address that she's used. I was making some changes on the prototype server yesterday around the time you made this post which may have broken replies for some users and I think this is what's happened. Sorry about that - I was fixing some bugs and inadvertently made some breaking changes. This won't happen in normal use, but the prototype server is a demonstration system and sometimes things get broken.

I've stopped playing around now, and if she replies to a new e-mail, then it should work. It won't be possible to reply to the old e-mails any more I'm afraid.

Andrew

Average of ratings: -
In reply to Andrew Lyons

Re: Forum reply by e-mail

by Jacqui Sciancalepore -

Thinking about this function a little more. I can see there are options at the bottom of the emails to unsubscribe from forums, but what about subscribing from a single discussion topic. I can see that when logged in, the option is there. Is it possible to modify the unsubscribe section?

Average of ratings: Useful (1)
In reply to Jacqui Sciancalepore

Re: Forum reply by e-mail

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Jacqui,

Thanks for pointing that out. I've created MDL-47390 to address that and I'll make sure that it makes it into Moodle 2.8,

Andrew

Average of ratings: -
In reply to Andrew Lyons

Re: Forum reply by e-mail

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

Having read this discussion, I now understand the line at the top of each email asking people to not include the post in their reply.

The problem with this line is that it then becomes the preview of every Moodle email making it an extra step to see what the email is about.  I used to be able to just scroll down my email list and quickly see if there are some posts that fall in an area where I can help or where I am looking for help.  Now, I actually have to open each email to see the first couple of lines of the email.


Average of ratings: Useful (2)
In reply to Emma Richardson

Re: Forum reply by e-mail

by Daniel Neis Araujo -
Picture of Core developers Picture of Plugin developers Picture of Translators

Hello,


I agree with Emma, this message could be moved to bottom of email.


Kind regards,

Daniel

Average of ratings: -
In reply to Andrew Lyons

Re: Forum reply by e-mail

by Brendan Heywood -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

There seems to be a long tail of formatting tweaks to the emails and I can see differences of opinions on how they should look and what they should contain. ie I'd personally prefer to strip out most of the redundant wrapping and make it streamlined more like github or google doc emails.

But rather than argue about it, why not just move forum_make_mail_text, forum_make_mail_html and similar into a renderer so a theme can override it? Why shouldn't emails be first class citizens 

mod_assign_renderer->email_text(...)
mod_assign_renderer->email_html(...)
mod_assign_renderer->email_digest_text(...)
mod_assign_renderer->email_digest_html(...)

Maybe even new layouts to wrap all emails regardless of type across a site

/theme/xyz/layout/htmlemail.php
/theme/xyz/layout/textemail.php


Average of ratings: -