SMTP through O365 fails

SMTP through O365 fails

by Nate Swetland -
Number of replies: 10

I am pulling my hair out on a new installation of Moodle 3.8 on BlueHost.

I installed Moodle 3.5 or 3.6 using BlueHosts's one click install and it went very smooth for the most part.  Then, I tried setting it up to use my Office 365 tenant for SMTP outgoing email.  

I have manually updated to Moodle 3.8 just a day or two ago.

My Outgoing Email settings are:

SMTP Host: smtp.office365.com:587
SMTP Security: TLS
SMTP Auth Type: LOGIN
SMTP Username: person@mydomain.com
SMTP password: (person's password)
SMTP Session Limit: 1
no reply address: person@mydomain.com
allowed email domains:  mydomain.com, mydomain.onmicrosoft.com

Email via information: Always

Everything else blank/default
Does something above look incorrect?   Auth Type? Host:Port?



It gives me errors about not connecting to O365, timeouts, phpmail() errors, etc.

I contact BlueHost and they gave me the runaround since our MX records arent through them, but they told me that they dont block outgoing mail stuff.


I am at my wit's end.  Does someone have a working O365 SMTP outgoing mail configuration they can share with me?


Thanks!!!

Average of ratings: -
In reply to Nate Swetland

Re: SMTP through O365 fails

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Looks fine to me...

Can we get the *actual* errors please? If you haven't done so already, enable 'Debug email sending'. You might also want to look at...

https://docs.moodle.org/38/en/Debugging#More_tools_for_debugging_outgoing_mail_.28SMTP.29

BlueHost may not block email... but does whatever OS you are using have a firewall that may be blocking outgoing mail?

I did a Google search and found some other posts complaining about the same thing (not with Moodle, but it doesn't matter). At least one of them ended up moving hosts
Average of ratings: -
In reply to Howard Miller

Re: SMTP through O365 fails

by Nate Swetland -
So a few things happen. When I do a test mail to myself, it will just time out and not show any error messages...

In my webserver's php logs (they are really annoying and delayed and not very helpful) the last entry I have is:
[03-Dec-2019 16:57:13 America/New_York] Default exception handler: Tried to send you an email but failed! Debug:
Error code: auth_emailnoemail
* line 494 of /lib/setuplib.php: moodle_exception thrown
* line 142 of /auth/email/auth.php: call to print_error()
* line 99 of /auth/email/auth.php: call to auth_plugin_email->user_signup_with_confirmation()
* line 89 of /login/signup.php: call to auth_plugin_email->user_signup()


In my moodle logs it says:
Failed to send an email from the user with id '2' to the user with id '-99' due to the following error: "SMTP connect() failed. https://github.com/PHPMailer/PHPMailer/wiki/Troubleshooting".


I have email debugging enabled.. a few times I will get the pink block where it will show me messages, but most of the time it just cranks away and then times out to a blank screen.

By "whatever os you are using have a firewall" do you mean my OS or the webserver OS? Ive tried it from multiple computers, work, home, etc..
my BlueHost is on linux and from what the unhelpful rep told me was that their sendmail was functioning correctly.


I am mostly convinced that it's BlueHost and that the tech I was on chat with wanted to just blame it on "not using their MX records" instead of helping me resolve it but I have used BlueHost for quite some time and we have had various wordpress installs that had working email so I want to do more troubleshooting before I migrate away from that host.
Average of ratings: -
In reply to Nate Swetland

Re: SMTP through O365 fails

by Ken Task -
Picture of Particularly helpful Moodlers

From the terminal of your server via ssh logged on as any user that can use CLI tools for DNS (assuming they are installed on your server):

dig your.fq.dn

That should return a response of DNS servers with an ANSWER SECTION
showing something like:

your.fq.dn 7200 IN A an.ip.add.ress

above is what is called an 'A record'.  The number is the TTL (time to live) value ... how long the A record info will live on internet before the need to refresh the lookup.
IN A indicates the A record and 'an.ip.add.ress' above is your server's IPv4 address.

See it's reverse entry (in-addr or PTR [pointer] record):

dig -x 'youripaddress'

;; ANSWER SECTION:
nnn.nnn.nnn.nnn.in-addr.arpa. 900 IN    PTR yourserverfqdn

The n's above are numbers and shown in reverse order.
Using example above:

ress.add.ip.an.in-addr.arpa. 900 IN PTR yourserverfqdn

Now some other records:

dig yourserverfqdn -t NS

query for name servers for your domain.  Hopefully you will see an 'Authoratative' response.

dig yourserverfqdn -t MX

and finally other TXT records which come into play these days
for anti-spam, etc.

dig yourserverfqdn -t TXT

See any SPF or DMARC or DKIM records for above?

Minamumly there should be an SPF record for the IPv4 address of your Moodle server.

If not, mail servers on the receiving end might reject messsages.

BTW, providers don't want to be known as the 'AOL' of old (spammers haven) so many of them actually list their own IP addresses in SpamHause.  Customer can clear or request clearance via url/form SpamHause provides in postmaster bounded message.

and a though/re-action to:

"wordpress installs that had working email" ... that's past tense right?   How long ago and was the WordPress an open relay (but you didn't know it)?

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: SMTP through O365 fails

by Nate Swetland -
Ah, these are some helpful tips. I did some DNS changes to add my moodle host to my SPF. I havent updated my DMARC or DKIM but it still seems to just be having the same problem. the other stuff looks mostly fine. Some of our other tools that send email dont really need a lot of extra config to get them to work, so it just leads me to believe that the host is the problem.


I think this might be the kick in the pants we need to migrate away from BlueHost. They have been getting worse and worse over time so this is a good of an excuse as any to migrate.

I am going to keep messing around with this but hopefully a new host will not be so much trouble.
Average of ratings: -
In reply to Nate Swetland

Re: SMTP through O365 fails

by Ken Task -
Picture of Particularly helpful Moodlers

In looking again at original posting ... user id 2 is the user that initially installed the moodle ... like admin ... what email address do you have for that user?

I have found best to use the moodle servers FQDN for that address.

Example: moodle server FQDN is elearning.somenet.net ... for the admin user I would use admin@elearning.somenet.net

That admin user might actually have an account on that server but it's never used to accept mail ... just send.

That would be one of the many items checked by spam checkers.

The other thing I do ... setup either postfix or sendmail as a sending only server.   All mail leaving the moodle then goes through postfix or sendmail ... whose apps keeps logs on the moodle server itself.   Postfix or Sendmail (just the sending part) could be set up with a 'smarthost' config - which basically is any mail not bound to root or local accounts on the moodle server, goes to 'smarthost' for relay/sorting.

I get to see bounces in logs of postfix or sendmail.

The SMTP host ... the true relay to all other email addresses, I more than likely do NOT have full control over ... your o365 ... my Google ... only logins/password/ports and transport can I set in Moodle or Postfix/Sendmail.

I see Bluehost docs say port 26 ... not 25 .. amazing what one digit off will do ... not do! smile

Begin soap box ...

And a comment about hosting ... is it better the 'devils' you know, than the 'devils' you don't?

Hosting providers are not known to have *fully disclosing* capabilities ... and many have a lot of promises ... good marketing ... but technically have difficulty delivering.

Even VPS's limit the customer ... and the general thought on VPS's is account holder can do all things ... well, guess what, not 100% true when it comes to DB (devils you don't know).   When it becomes a contest for control, I'd rather have superuser creds for DB and have to put up with annoyance in EMail.   But that's me ... we all choose our own poisons ... and then learn our own antidotes!

--- end soap box!

'SoS', Ken



Average of ratings: -
In reply to Ken Task

Re: SMTP through O365 fails

by Nate Swetland -
Greatly appreciate all the help so far.

I believe I have made it past the SMTP connection/authentication error but I am running into some Send-As errors it seems. I wound up migrating away from GoDaddy/BlueHost to NameCheap as they are much more responsive and easier to use. They opened port 587 for me without a huge argument like I had with BlueHost...

Anyways...
I feel like I am not fully understanding this no-reply and domain section...

  • My website is set up as a "university.ACME.com" but I dont have any university.acme.com emails and I don't have any desire to.
  • I have a service email account I am using for testing that is "helpdesk@ACME.com"
  • In my no-reply address, I have tried "helpdesk@ACME.com" and no-reply@ACME.com" and blank/default of "no-reply@university.ACME.com"
  • Under Allowed email domains I have tried BLANK/EMPTY, and then I also attempted some combinations of "ACME.com, ACME.onmicrosoft.com, *.ACME.com and university.acme.com"
  • I have tried the "Email via information" as always and never.



when I go to send an email to me@acme.com it fails. It gives me an error about sendasdeniedexception.mapiexceptionsendasdenied

One thing that stands out to me is the line:
2020-01-01 21:08:32    CLIENT -> SERVER: Message-ID: <5e0d0a4de5a8c0.64424587@university.ACME.com>

Why is it trying to use a "university.ACME.com" to send my email?  Can this be corrected?
Why is it using a randomly generated email?    Can this be corrected?

I am starting to see a lot of posts that Send-As through PHP SMTP on Office365 basically doesnt work...

I have an SPF record for both university.ACME.com and ACME.com that include the IP address of my NameCheap shared hosting


Here is the full error message with a little bit of identity scrubbing on it.

2020-01-01 21:08:29    Connection: opening to smtp.office365.com:587, timeout=300, options=array()
2020-01-01 21:08:30    Connection: opened
2020-01-01 21:08:30    SERVER -> CLIENT: 220 BYAPR08CA0028.outlook.office365.com Microsoft ESMTP MAIL Service ready at Wed, 1 Jan 2020 21:08:29 +0000
2020-01-01 21:08:30    CLIENT -> SERVER: EHLO university.ACME.com (changed)
2020-01-01 21:08:30    SERVER -> CLIENT: 250-BYAPR08CA0028.outlook.office365.com Hello [MY.WEB.HOST.IP] (changed)
                                         250-SIZE 157286400
                                         250-PIPELINING
                                         250-DSN
                                         250-ENHANCEDSTATUSCODES
                                         250-STARTTLS
                                         250-8BITMIME
                                         250-BINARYMIME
                                         250-CHUNKING
                                         250 SMTPUTF8
2020-01-01 21:08:30    CLIENT -> SERVER: STARTTLS
2020-01-01 21:08:30    SERVER -> CLIENT: 220 2.0.0 SMTP server ready
2020-01-01 21:08:30    CLIENT -> SERVER: EHLO university.ACME.com (changed)
2020-01-01 21:08:30    SERVER -> CLIENT: 250-BYAPR08CA0028.outlook.office365.com Hello [MY.WEEB.HOST.IP]
                                         250-SIZE 157286400
                                         250-PIPELINING
                                         250-DSN
                                         250-ENHANCEDSTATUSCODES
                                         250-AUTH LOGIN XOAUTH2
                                         250-8BITMIME
                                         250-BINARYMIME
                                         250-CHUNKING
                                         250 SMTPUTF8
2020-01-01 21:08:30    CLIENT -> SERVER: AUTH LOGIN
2020-01-01 21:08:30    SERVER -> CLIENT: 334
2020-01-01 21:08:30    CLIENT -> SERVER:
2020-01-01 21:08:30    SERVER -> CLIENT: 334
2020-01-01 21:08:30    CLIENT -> SERVER:
2020-01-01 21:08:31    SERVER -> CLIENT: 235 2.7.0 Authentication successful
2020-01-01 21:08:31    CLIENT -> SERVER: MAIL FROM:
2020-01-01 21:08:31    SERVER -> CLIENT: 250 2.1.0 Sender OK
2020-01-01 21:08:31    CLIENT -> SERVER: RCPT TO:
2020-01-01 21:08:31    SERVER -> CLIENT: 250 2.1.5 Recipient OK
2020-01-01 21:08:31    CLIENT -> SERVER: DATA
2020-01-01 21:08:32    SERVER -> CLIENT: 354 Start mail input; end with .
2020-01-01 21:08:32    CLIENT -> SERVER: Date: Wed, 1 Jan 2020 16:08:29 -0500
2020-01-01 21:08:32    CLIENT -> SERVER: To: ME@ACME.com (changed)
2020-01-01 21:08:32    CLIENT -> SERVER: From: "System Administrator"
2020-01-01 21:08:32    CLIENT -> SERVER: Reply-To: "System Administrator"
2020-01-01 21:08:32    CLIENT -> SERVER: Subject: ACME University ACME University: test message
2020-01-01 21:08:32    CLIENT -> SERVER: Message-ID: <5e0d0a4de5a8c0.64424587@university.ACME.com>
2020-01-01 21:08:32    CLIENT -> SERVER: X-Mailer: PHPMailer 6.0.7 (https://github.com/PHPMailer/PHPMailer)
2020-01-01 21:08:32    CLIENT -> SERVER: MIME-Version: 1.0
2020-01-01 21:08:32    CLIENT -> SERVER: Content-Type: text/plain; charset=UTF-8
2020-01-01 21:08:32    CLIENT -> SERVER:
2020-01-01 21:08:32    CLIENT -> SERVER:
2020-01-01 21:08:32    CLIENT -> SERVER: This is a test message to confirm that you have successfully configured
2020-01-01 21:08:32    CLIENT -> SERVER: your site's outgoing mail.
2020-01-01 21:08:32    CLIENT -> SERVER:
2020-01-01 21:08:32    CLIENT -> SERVER: .
2020-01-01 21:08:32    SERVER -> CLIENT: 554 5.2.0 STOREDRV.Submission.Exception:SendAsDeniedException.MapiExceptionSendAsDenied; Failed to process message due to a permanent exception with message Cannot submit message. 0.35250:0A000D85, 1.36674:0A000000, 1.61250:00000000, 1.45378:02000000, 1.44866:55210000, 1.36674:0E000000, 1.61250:00000000, 1.45378:5A210000, 1.44866:0A020000, 16.55847:3E100000, 17.43559:0000000004020000000000000000000000000000, 20.52176:140F2A8A0A00101043050000, 20.50032:140F2A8A7A17000000000000, 0.35180:48050000, 255.23226:0A00
2020-01-01 21:08:32    SMTP ERROR: DATA END command failed: 554 5.2.0 STOREDRV.Submission.Exception:SendAsDeniedException.MapiExceptionSendAsDenied; Failed to process message due to a permanent exception with message Cannot submit message. 0.35250:0A000D85, 1.36674:0A000000, 1.61250:00000000, 1.45378:02000000, 1.44866:55210000, 1.36674:0E000000, 1.61250:00000000, 1.45378:5A210000, 1.44866:0A020000, 16.55847:3E100000, 17.43559:0000000004020000000000000000000000000000, 20.52176:140F2A8A0A00101043050000, 20.50032:140F2A8A7A17000000000000, 0.35180:48050000, 255.23226:0A00
2020-01-01 21:08:32    SMTP Error: data not accepted.



Average of ratings: -
In reply to Nate Swetland

Re: SMTP through O365 fails

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

You wrote:
> One thing that stands out to me is the line:
> 2020-01-01 21:08:32 CLIENT -> SERVER: Message-ID: <5e0d0a4de5a8c0.64424587@university.ACME.com>
>
> Why is it trying to use a "university.ACME.com" to send my email? Can this be corrected?
> Why is it using a randomly generated email? Can this be corrected?

Although the Message-ID has the format of an e-mail address, it is what its name says, just a (globally) unique identifier. So not the reason for rejecting the e-mail message.

> 2020-01-01 21:08:32 CLIENT -> SERVER: From: "System Administrator"

I guess the this forum editor has purged the e-mail address at the end of the line. I guess it to be one of:
> In my no-reply address, I have tried "helpdesk@ACME.com" and no-reply@ACME.com" and blank/default of "no-reply@university.ACME.com"

That is where the From: "on behalf of" address comes in. (BTW, the defaut no-reply@ has no hyphen.)

> STOREDRV.Submission.Exception:SendAsDeniedException.MapiExceptionSendAsDenied; Failed to process message due to a permanent exception with message Cannot submit message.

The server, in this case smtp.office365.com, refused to send e-mails on behalf of @*ACME.com. I have nothing to do with Microsoft, I assume they expect the sender to have "their" domain. May be others can confirm. Failing that, you can try to configure Thunderbird to send e-mails through this server with any random From address.

Edit: Only after replying I see that there were previous posts. So the answer above may be off the mark.
Average of ratings: -
In reply to Nate Swetland

Re: SMTP through O365 fails

by Kent Graves -

I've been having the same issue for the past couple of days using my companies O365 domain email.  My debug errors were identical to yours.  I finally stumbled upon a solution on Stackoverflow.  

https://stackoverflow.com/questions/54784431/unable-to-send-smtp-mails-using-office365-settings

Apparently, O365/Outlook does not allow you to send an email with a "From:" address that is different than the username you authenticated with.  Changing the "No-reply" address to be identical to my username got it working.

This isn't ideal...because I don't want system emails having my own email address attached to them but it's a step closer.  At least it's working.

Edit:  

Ok, I found a workaround. 

1. Create a new O365 group named "No Reply". 
2. Give it the group email address of "no-reply@mydomain.com". 
3. Edit the group and under "delivery management" configure it to only accept messages from itself. (I did this just because this account will never be emailing itself. So it rejects all other incoming mail.) . 
4. Next, under group delegation, give the authenticated SMTP user you configured moodle with the "Send As" permission. (This lets that user send email as the no-reply@mydomain email address.)
5. Once that is saved, set your no-reply address in moodle to the address you just created. Then it should work.
Average of ratings: Useful (2)
In reply to Kent Graves

Re: SMTP through O365 fails

by Nate Swetland -

This is really helpful, thanks.

I finally got it to the point of getting past the connection/authentication but was getting those send-as errors.  Strangely, changing the email address for my admin account resolved that (but I think that was specifically because I was "testing email" using that account and it was trying to send from moodleadmin@whatever.com.

I must have read that same stackoverflow article about 5 times myself haha.

I was considering making a service account just for moodle but I like your suggestion of a o365 group with send-as permission and I will try that since that wont use up a license (not the end of the world but still).

I really appreciate your suggestion!


Average of ratings: -