Incoming mail reply-to forums not working with @gmail

Incoming mail reply-to forums not working with @gmail

by No name -
Number of replies: 8

Hi,

Using Moodle 2.8.2 I have successfully set up incoming mail for forums using a test mail account. Hooray! ... 

oh wait...

But then, when I tried to make it work with the real mail account I noticed that the "reply-to" field is missing in email header. It's weird beause both email accounts (test one and real one) use gmail servers. Google Apps email it's working but if I use a standard @gmail account the "reply-to" field dissapears.

I know the incoming mail processing is working because "Email to Private files" handler works.

Let me show you the emails headers:

  1. email header using a google apps account:

    [...]
To: "User name" <myemail@mydomain.com>
Subject: FORUM: Subject
X-PHP-Originating-Script: 10000:class.phpmailer.php
Date: Tue, 13 Jan 2015 18:32:03 +0000
From: "Administrator" <email@googleappsdomain.com>
Reply-To: account+AAAAAAAAAAIAOOOOOssswAAAAAAKcXuXXhCKp3tuf65g@googleappsdomain.com
[...]

  2. email header using a standard @gmail account:

    [...]
To: "User name" <myemail@mydomain.com>
Subject: FORUM: Subject
X-PHP-Originating-Script: 10000:class.phpmailer.php
Date: Tue, 13 Jan 2015 18:32:03 +0000
From: "Administrator" <email@gmail.com>
[...]

I don't know what's happening. If I use the google apps account it works and when I just change the username and password for the gmail account (everything else it's ok), then the reply-to field dissappears. And finally, if I configure again the google apps, it works again!!!

Any help?

I'm going crazy... 

Thank you in advance.

Average of ratings: -
In reply to No name

Re: Incoming mail reply-to forums not working with @gmail

by No name -

Any thoughts? sad

Average of ratings: -
In reply to No name

Re: Incoming mail reply-to forums not working with @gmail

by Mukarram Syed -

Hi Emilio, 

I have the same issues and scourged the internet to find a solution with no success.

For me, I just don't seem to receive reply-to headers at all. I am using gmail imap setting in the incoming mail configuration and the "email to private files" is working perfectly.

For some reason, forum notifications that are being sent just don't contain the reply-to header. My setup includes, Ubuntu14.04, apache, postfix for emailing, moodle 2.8.3.

I have testing sending emails from the forum to users with gmails, outlook.com, my company exchange account, and even yahoo, but with no success. There is no reply to in the email header and replying by email feature for the forum doesn't work at all.

PLEASE HELP SOMEONE.

sad

Average of ratings: -
In reply to Mukarram Syed

Re: Incoming mail reply-to forums not working with @gmail

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Muhammed,

Please ensure that you check the following:

  1. SIte administration -> Server -> Incoming mail configuration -> Mail settings
    1. Enable incoming mail processing should be checked
    2. Mailbox namedshould be filled (e.g. mygmailusername)
    3. Email domain should be filled (e.g. gmail.com)
    4. Incoming mail domain should be filled (e.g. imap.gmail.com)
    5. Username should be filled (e.g. mygmailusername)
    6. Password should be filled (e.g. SuperSecureEmailPassword)
  2. Navigate to Site adminisgration -> Server -> Incoming mail configuration -> Message handlers
    1. Edit Reply to forum posts
    2. Ensure that Enabled is checked

That's that that should be required. If all of these steps are already checked, please reply and we can look further.

Andrew

Average of ratings: -
In reply to Andrew Lyons

Re: Incoming mail reply-to forums not working with @gmail

by Mukarram Syed -

Hi Andrew,

I confirm all the above are checked. 

I also confirm that post to private file area is working with all these setting. Users are able to send attachments to their unique email addresses and they are getting posted successfully.

The forum notification emails don't have a reply-to header at all. 

Here is my current situation:

  • Running Ubuntu 14.04, apache, using postfix to send emails
  • Latest Moodle 2.8.3+ (Note: Upgraded from a fresh install of 2.7.5)
  • Setup all the configurations required for incoming email (Used gmail IMAP)
  • Private file area working fine without any issues. 
  • Forum email notifications just don't seem to work at all. No matter what I do, the emails don't have a reply-to in the headers. I have checked to see if the server is stripping it by stopping the postfix queue and the original headers don't contain the reply-to email address. 
  • Followed all suggestions here, added a catchall setting for postfix. No go.
Average of ratings: -
In reply to Mukarram Syed

Re: Incoming mail reply-to forums not working with @gmail

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Mukarram,
do you have something as below in your Postfix configuration?

header_checks = regexp:/etc/path/to/header_checks

If yes, double check /etc/path/to/header_checks to look for a regexp matching Reply-To on some circumstances.

HTH,
Matteo

Average of ratings: -
In reply to Andrew Lyons

Re: Incoming mail reply-to forums not working with @gmail

by Mukarram Syed -

UPDATE: After a bit of reviewing the code in /mod/forum/lib.php and testing stuff out, I have a few findings below:

Question for Andrew: Why did we move the reply-to email at the bottom of email body and not in the header?

After spending 6 hours of troubleshooting time, I just couldn't figure out why the reply-to email was not coming through. 

I did a couple of things:

Hardcoded $replyaddress variable on line: https://github.com/moodle/moodle/blob/MOODLE_28_STABLE/mod/forum/lib.php#L762  It seemed to work. I added a dummy plus address format email and it seems to work fine too. The reply to was coming through in the notifications.

But the moment I remove the hard-coded email, the reply-to is complete missing in the notifications. 

Then i realized, according to some specs online, the reply-to should be present in the header. 

I added this line: $userfrom->customheaders[] = 'Reply-To: ' . $replyaddress; (immediately after line 763) and it all seems to work just fine! The reply-to is coming through the way it is supposed to. The reply by email is working. The posts are being posted correclty.(Tested using 4 different users and 4 different email addresses by different providers).

I wonder why this is happening and what could be the reason behind this bizarre issue. Is it postfix? Do I need to configure it to pickup reply-to from the body of the email and not within the header? Remember, Postfix is what is sending these emails. I have gmail configured for IMAP in Incoming Mail Configuration.

HELP! Even though what I did works perfectly for me, I don't want to modify the core code?

Kind Regards,

Mukarram



Average of ratings: -
In reply to Mukarram Syed

Re: Incoming mail reply-to forums not working with @gmail

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Mukarram

We have only moved the text to the bottom of the email. The text does not perform any function other than information to the user.

The reply-to address is indeed in the header. See https://github.com/moodle/moodle/blob/MOODLE_28_STABLE/mod/forum/lib.php#L788 - that sets the Reply-To in the userfrom field of the eventdata.

The eventdata->replyto is eventually passed to message/output/email/message_output_email.php where it's then picked up and passed to email_to_user() as a parameter.  It's then added to the headers. If you have some charset conversion issues, it could be being nullified in https://github.com/moodle/moodle/blob/MOODLE_28_STABLE/lib/moodlelib.php#L5961 but I find that unlikely.

This could be going wrong in two possible places that I can think of:

  1. The replyaddress is not being generated; and/or
  2. The message API is not adding the reply address to the headers.

To check 1, add some form of debugging call (print_object / var_dump) of $replyaddress to just after the address has been generated. I'd recommend line 764 in the 2.8 link you sent.

To check 2 is much harder, so let's see what we're getting from 1 first.

If 1 is still null, then I suspect that the reply address is not being generated for some reason.  That could be:

  1. The user cannot post to this discussion (check group memberships, capabilities, forum type, etc);
  2. The data key for the message handler has not been set.

To check 2, looking at https://github.com/moodle/moodle/blob/MOODLE_28_STABLE/mod/forum/lib.php#L549, ensure that the data key was generated by adding debugging there:

mtrace("Generating data key for {$pid}: {$messageinbounderhandlers[$pid]}");

If you aren't getting a data key for this post, then there's some kind of issue in the data key generation and we need to look there.

I think that's probably enough to get things started. Once we work out where things are failing, we can work out where to look next.

Andrew

Average of ratings: -
In reply to Andrew Lyons

Re: Incoming mail reply-to forums not working with @gmail

by Mukarram Syed -

Hi Andrew,

Thank you for your detailed reply. 

I had done debugging for step 1:

  1. The replyaddress is not being generated; and/or

The replyaddress is being generated and is printed out in the cron job. 


It looks like there is an issue with reply-to address not being set as an header through eventdata object. I saw your reply on this other post about this: https://moodle.org/mod/forum/discuss.php?d=277594

If I set the reply-to address to the $userfrom->customheaders[], everything works. Somehow, the replytoaddress from $eventdata is not being added to the header by the Message API.


I am going to debug this over the weekend with the steps you mentioned and post my findings on this post: https://moodle.org/mod/forum/discuss.php?d=277594 . Since its the same issue on two different posts, lets move our conversation to the official post mentioned "https://moodle.org/mod/forum/discuss.php?d=277594" in the docs for the sake of others finding this conversation.


Kind Regards,

Syed





Average of ratings: -