Need help configuring forum's "Reply to post" feature.

Re: Need help configuring forum's "Reply to post" feature.

by Mukarram Syed -
Number of replies: 9

I desperately need help with this feature. 

Here is my current situation:

  • Running Ubuntu 14.04, apache, using postfix to send emails
  • Latest Moodle 2.8.3+
  • Setup all the configurations required for incoming email (Used gmail IMAP)
  • Private file area working fine without any issues. 
  • Forum email notifications just don't seem to work at all. No matter what I do, the emails don't have a reply-to in the headers. I have checked to see if the server is stripping it by stopping the postfix queue and the original headers don't contain the reply-to email address. 
  • Followed all suggestions here, added a catchall setting for postfix. No go.


I am just receiving emails with the no-reply email address I had defined in the setting site-administration->plugins->message-output->email. There is no reply to email in the header even when seeing the original source from the gmail or outlook interface. I have also enable "Always send email from the no-reply address?"

What am I missing here???


In reply to Mukarram Syed

Re: Need help configuring forum's "Reply to post" feature.

by Andréa Oliveira -

You are configuring with encryption or no encryption?


In reply to Andréa Oliveira

Re: Need help configuring forum's "Reply to post" feature.

by Mukarram Syed -

UPDATE: After a bit of reviewing the code in /mod/forum/lib.php and testing stuff out, I have a few findings below:

Question for Andrew: Why did we move the reply-to email variable at the bottom of email body and not in the email header in the file /mod/forum/lib.php?

After spending 6 hours of troubleshooting time, I just couldn't figure out why the reply-to email was not coming through. 

I did a couple of things:

Hardcoded $replyaddress variable on line: https://github.com/moodle/moodle/blob/MOODLE_28_STABLE/mod/forum/lib.php#L762  It seemed to work. I added a dummy plus address format email and it seems to work fine too. The reply to was coming through in the notifications.  But the moment I remove the hard-coded email, the reply-to is complete missing in the notifications. 

Then after a bit of research and according to some specs online, the reply-to should be present in the header. 

I added this line: $userfrom->customheaders[] = 'Reply-To: ' . $replyaddress; (immediately after line 763) and it all seems to work just fine! The reply-to is coming through the way it is supposed to. The reply by email is working. The posts are being posted correclty.(Tested using 4 different users and 4 different email addresses by different providers).

I wonder why this is happening and what could be the reason behind this bizarre issue. Is it postfix? Do I need to configure it to pickup reply-to from the body of the email since its not within the header? Remember, Postfix is what is sending these emails. I have gmail configured for IMAP in Incoming Mail Configuration. 

HELP! Even though what I did works perfectly for me, I don't want to modify the core code?

Kind Regards,

Mukarram

In reply to Mukarram Syed

Re: Need help configuring forum's "Reply to post" feature.

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Mukarram,

Firstly my apologies for the delay in my reply. I had written a reply yesterday morning, but inadvertently lost it, and I haven't had a chance to re-write because of weekly integration.

So first things first, the replyto variable in the body is no longer used. It was originally used, but moved recently in MDL-48212. The replyto variable is passed to the message headers by way of the eventdata.

By passing this into the eventdata, it makes it's way to:

  1. lib/messagelib.php, where it is passed to:
  2. send_message and then to:
  3. \core\message\manager::send_message where it is passed to:
  4. anager::send_message_to_processors where it is passed to each message output processor, including:
  5. email where the variable is:
  6. picked up and then passed to:
  7. email_to_user: where it's eventually:
  8. added to the headers.

So with this information in mind, the first thing to check is whether the replyaddress variable is actually set. We can do this by adding some debugging to find out whether our $replyaddress variable was set by adding the following debugging just just after replyaddress is first set:

    if ($userto->canpost[$discussion->id] && array_key_exists($post->id, $messageinboundhandlers)) {
        $messageinboundgenerator->set_data($post->id, $messageinboundhandlers[$post->id]);
        $replyaddress = $messageinboundgenerator->generate($userto->id);
    }

mtrace("Setting the reply address for {$userto->id} / {$post->id} to {$replyaddress}");
var_dump($replyaddress);

    $a = new stdClass();

If this reply address is not set, then we can add further debugging to the point at which the data key is generated:

    // Save the Inbound Message datakey here to reduce DB queries later.
    $messageinboundgenerator->set_data($pid);
    $messageinboundhandlers[$pid] = $messageinboundgenerator->fetch_data_key();

mtrace("Setting the data key for post {$pid} to {$messageinboundgenerator[$pid]}");

    // Caching subscribed users of each forum.

If this is not set, then we'll need to dig deeper into the data key generation to work out why it's not and which condition has not been met.

If both of these are set, then we'll need to take things through the above logic all the way to where the headers are actually applied in lib/moodlelib.php.

One place that things could be going wrong (though it is unlikely) is in the character encoding phase of email_to_user.

If you could have a look at the above, it may help track things down.

Cheers,

Andrew

In reply to Andrew Lyons

Re: Need help configuring forum's "Reply to post" feature.

by Mukarram Syed -

Hi Andrew,

Thanks a ton for your detailed reply. 

I had done debugging for step 1:

  1. The replyaddress is not being generated; and/or 

The replyaddress is being generated and is printed out in the cron job, when I do vardump for this variable.


Looks like there is an issues somewhere here:

By passing this into the eventdata, it makes it's way to:

  1. lib/messagelib.php, where it is passed to:
  2. send_message and then to:
  3. \core\message\manager::send_message where it is passed to:
  4. anager::send_message_to_processors where it is passed to each message output processor, including:
  5. email where the variable is:
  6. picked up and then passed to:
  7. email_to_user: where it's eventually:
  8. added to the headers.


Remember, If I set the reply-to address to the $userfrom->customheaders[], everything works. Somehow, the replytoaddress from $eventdata is not being added to the header by the Message API.


I am going to debug this over the weekend with the steps you mentioned and post my findings on here.


Just a quick recap: I have not done any core modification, no plugins are activated currently, clean upgrade from 2.7.5 to 2.8.3, ubuntu 14.04, apache2, postfix for mails.


Kind Regards,

Syed




In reply to Mukarram Syed

Re: Need help configuring forum's "Reply to post" feature.

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

I'd recommend starting by var_dumping the $replyto variable just before it's passed to email_to_user() at https://github.com/moodle/moodle/blob/v2.8.3/message/output/email/message_output_email.php#L95.

In reply to Andrew Lyons

Re: Need help configuring forum's "Reply to post" feature.

by Miguel Da Silva -

Hej hej, I've done this and the variable $replyto is set correctly. I
added the lines below just after the "}" character at line 94 of
message/output/email/message_output_email.php and then executed cron
manually.

mtrace("replyto - message/output/email/message_output_email.php -
{$replyto}");
var_dump($replyto);
The output is just like what I was expecting:

replyto - message/output/email/message_output_email.php -
moodleuser+AAAAAAAAAAIAAAAAAAAlewAAAAAAAmuHmgUDHHinY952DegC@domain.com
string(79)
"moodleuser+AAAAAAAAAAIAAAAAAAAlewAAAAAAAmuHmgUDHHinY952DegC@domain.com"

Moodle is configured such that the IMAP account is
moodleuser@domain.com, that's why "moodleuser" appears on the
messages. Nevertheless, the only way to get the feature working is
doing the same thing Mukarram Syed did, i.e., adding the follwoing
line to mod/forum/lib.php:

$userfrom->customheaders[] = 'Reply-To: ' . $replyaddress;
On the other hand, I'm checking the Message API code to be sure whether
the following statement actually makes Moodle add the "Reply-To"
string to the mail header (such statement goes here in the code
https://github.com/moodle/moodle/blob/v2.8.3/mod/forum/lib.php#L788):

$eventdata->replyto             = $replyaddress;
I'll be sending some news soon.

Greetings.

In reply to Miguel Da Silva

Re: Need help configuring forum's "Reply to post" feature.

by Miguel Da Silva -

So, after doing some work related to debugging this issue I got some novelties. It seems to be the fuction validateAddress() is getting tricked when it's going to validate an address generated in order to handle answers from e-mail clients.

The following two addresses was generated by Moodle and validateAddress() returned false:

moodlereceiveduser+AAAAAAAAAAIAAAAAAAA3UgAAAAAAAmuTC2W9OVpmOlmIEH/I@domain.com
moodlereceiveduser+AAAAAAAAAAIAAAAAAAAlewAAAAAAAmuTUOkwZ9YI2PtfxWOj@domain.com

I changed them in order to post this message, but I guess it is enough to give an example of addresses involved in this issue; moodlereceiveduser is the IMAP user in the local domain running the site.

Well, tracking it down a little bit made me check some code in lib/moodlelib.php; here is where the "Reply-To" header option is to be included in the outgoing message. In this section of the code the function addReplyTo() will be called and its jobs is to invoke addAnAddress() in a proper way.

The later one is responsible to execute validateAddress() and then problems arise. sad

Soon, more news...

Greetings.

In reply to Miguel Da Silva

Re: Need help configuring forum's "Reply to post" feature.

by Miguel Da Silva -

I got it!

After checking some more pieces of code I found the «localpart» of a email address can not be longer than 64 characters, and since the IMAP account I set up is 18 characters long, the final reply-to address is 67 characters long; that's why validateAddress() returns false.

On the other hand, here there are a couple of comments regarding the regular expression used to validate an e-mail address. On the notes presented in that link it's also possible to conclude the address length was the root of the problem.

I also found this comment by Andrew Nicols regarding the maximum length for the localpart.

If I am not wrong, the «data» portion of the localpart will be 48 characters long and 1 more character is added (the "+" sign), so, the dedicated email used to retrieve posts sent by mail can not be longer than 15 characters. As I tried to get a more descriptive email as possible I used 18 characters.

As a final observation, I didn't find anything regarding this "limit" in the documentation of the feature (please let me know if I'm wrong) and it seems that it'd be better to get it as part of the instructions.

If I have more novelties about it I'll add a new comment to this post, but for now I can say the feature is working in my environment (Moodle 2.8.3).

Greetings.


In reply to Miguel Da Silva

Re: Need help configuring forum's "Reply to post" feature.

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Thanks for tracking this down Miguel!

That most certainly does explain it, and I'm sorry that it didn't occur to me sooner! This is indeed documented in the developer documentation at https://docs.moodle.org/dev/Inbound_message_API#Summary, but it appears that this did not make it to the user documentation.

The maximum length of the localpart of an e-mail address (including subadress) as defined in RFC 5232, is 64 characters. The current maximum length of the subaddress component is 48 characters, with a further character used for the subaddress separator. This leaves space for 15 characters in the address component.

I'll speak to Mary and Helen about the best way of adding this information, and I've raised MDL-49707 to add checking to the administration setting too.

I'm so pleased to hear that everything is now working in your environment and thank you for all of your hard work in tracking this issue down. We should be able to address this in core so that others will not be affected.

Best wishes,

Andrew