SMTP connect error with Gmail after upgrade to 3.7

SMTP connect error with Gmail after upgrade to 3.7

by Emma Richardson -
Number of replies: 26
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

This has now happened on both my sites.

Upgrade to 3.7.

Using GMAIL relay for SMTP.  This has worked perfectly for over a year now.  

Testing email settings shows succesful.  My SMTP session limit is set to 200 - this fixed a previous issue with Gmail shutting down due to to many sessions being opened.

However, when sending a forum post to a large number of people, after about 120 emails go out, I get a smtp connect failed for all the other emails.  This is a huge problem.  If I change to phpmail, my emails all get marked as spam.  I did change to phpmail on one of the sites however and that did fix the error.  On my second site though, I really do not need the spam issue as we will not be able to contact people that we need to so I need to keep using Gmail if possible.

Does anyone have any idea of what I can do?  Where I can start?  I have searched through various server logs but yet to find any indication there.  

One site is on PHP 7.3 and one on 7.1.  The only change has been the upgrade from 3.6 to 3.7.

Average of ratings: -
In reply to Emma Richardson

Re: SMTP connect error with Gmail after upgrade to 3.7

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Emma

The answer is simple: Either you are dependent (on Google) or you are independent. You can't change Google, if it says no, then it is no!

You become independent by either a) running your own SMTP, the "phpmail" you mentioned, or b) sending mail via the SMTP server of your institution, or c) by taking the service of a commercial (authenticated) SMTP relay. As you have already experienced they don't have the bulk mail problem, if they complain you can always raise the limit. The (unavoidable) down side is, you have to get your mailer white listed, starting with a SPF record for your SMTP server.
Average of ratings: -
In reply to Emma Richardson

Re: SMTP connect error with Gmail after upgrade to 3.7

by Ken Task -
Picture of Particularly helpful Moodlers

Check the setup of the gmail account being used for relay.

Is there some sort of security alert?

Some info etc. that *might* help:

https://www.hostinger.com/tutorials/how-to-use-free-google-smtp-server

Free Google SMTP Sending Limits

Google limits outgoing emails to 100 per day. If the limit is reached, you won’t be able to send emails for the next 24 hours until it resets. More information about Google email limits can be found here.

https://support.google.com/a/answer/166852?hl=en

https://support.google.com/a/answer/2956491#sendinglimitsforrelay
link above has other info for sendmail and postfix at bottom.

Alt more testing/different ... the PHPMailer that comes with Moodle code doesn't include some specific example scripts ... one of those is a specific for gmail script.

Just did this on a Moodle 3.7 and tested ... successfully ... but not a forum list/posting.

cd /path/to/moodle37code/
git clone https://github.com/PHPMailer/PHPMailer.git mtest
cd mtest/examples
nano gmail.phps
cp gmail.phps gmail.php
php ./gmail.php

Note for above:
https://stackoverflow.com/questions/48128618/how-to-use-phpmailer-without-composer
added require lines for src/...
to script to get it to work locally on server

https://support.google.com/accounts/answer/6010255
*Less Secure Apps had to be allowed.

And just in case ... after testing have moved the mtest folder out of code to another location on server so folks reading this won't be tempted! smile

Good Luck!  Working with Google ... always an adventure!

'SoS', Ken


Average of ratings: -
In reply to Emma Richardson

Re: SMTP connect error with Gmail after upgrade to 3.7

by Ken Task -
Picture of Particularly helpful Moodlers

Visvanath does make a very valid point.

Much depends upon where servers are hosted.  Hosting providers really don't want to gain reputation as the home of a 'spam king' so they even list all the IP addresses of their servers in a blackhole list.   You may not see that, unless one installs postfix on same server and set it up to be sending only ... it acts as the first relay.  Advantage of that, bounced messages will be returned or non-delivered/blocked messages sent to root user on that system, and installing alpine on that server one can check root's mail via command line to read info on the bounces/blocks.

I also install a thing called logwatch on servers.  It runs once a day and sends to root user (localhost) a daily report ... in the daily, a section on email.

Now if you have control of DNS (domain for the moodle server), it helps to have at least an SPF record for the moodle server.  Other things like DMarc and DKIM would be best but those require a lot more config ... both DNS and on the server itself.

Also, some providers actually recommend certain smtp relay services.  Rackspace, for example, recommends 2 ... one of which is SendGrid.  How about your hosting provider?

It's not only Google but anything related to sending email ... 'always an adventure'! sad

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: SMTP connect error with Gmail after upgrade to 3.7

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers
I actually found out that what is going on is that forum post emailing has changed in 3.7 to the effect that they are breaking it down in multiple sessions which is in effect making google shut down because of multiple sessions running - so my 200 emails per session setting is not longer working.

With the Google relay, it will actually send more than 200. The issue is the simultaneous sessions. So now, I am tasked with setting up sendmail or postfix, etc. but need to make sure that it will not be spammed. I think I will try and set it up to use Gmail as the smtp server and hopefully will be able to get the settings as such that it will keep the connection alive and be able to send all the emails in one setting. If anyone has experience with this and wants to share a config file, that would be awesome!!
big grin
Average of ratings: Useful (1)
In reply to Emma Richardson

Re: SMTP connect error with Gmail after upgrade to 3.7

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers
Oh, and we self-host. Our domain mail is through Google Apps so it complicates a few things.
Average of ratings: -
In reply to Emma Richardson

Re: SMTP connect error with Gmail after upgrade to 3.7

by Jon Witts -
Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Hi Emma,

We self-host Moodle and run all of our emails through Google Suite for Education. Our solution was to set up a self-hosted postfix linux server that acts as an internal SMTP relay. It is locked down to only allow relaying from defined internal hosts and it then forwards all of it's mail onto the Google SMTP connectors...

In turn; if Google has email for any of our self-hosted servers (not Moodle - we have not turned on incoming email on Moodle), Google sends the emails back into our internal postfix box which forwards the emails to the relevant internal server...
Average of ratings: -
In reply to Jon Witts

Re: SMTP connect error with Gmail after upgrade to 3.7

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Jon

It is a simple choice you've made between:
A. Moodle -> internal SMTP relay -> Google SMTP to Internet
B. Moodle -> internal SMTP relay -> Own SMTP to Internet

In case A, the path you took, e-mails travel: Moodle -> internal SMTP relay -> Google SMTP to Internet -> recipient's MX. Recipient's MX is Google mailbox server for Gmail, your school's mailbox for your students.

In case B, the e-mails will travel Moodle -> internal SMTP relay -> Own SMTP to Internet -> recipient's MX. Recipient's MX is still Google mailbox server for Gmail, your school's mailbox for your students.

So the number of hops remains the same, the only difference is in hop 3: It is Google SMTP in A, your school's SMTP in B.
Average of ratings: Useful (1)
In reply to Jon Witts

Re: SMTP connect error with Gmail after upgrade to 3.7

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

Yes, that is what I effectively did too.  

Average of ratings: -
In reply to Emma Richardson

Re: SMTP connect error with Gmail after upgrade to 3.7

by Ken Task -
Picture of Particularly helpful Moodlers

At the bottom of resource shared earlier ...

https://support.google.com/a/answer/2956491#sendinglimitsforrelay

There are links to sendmail and postfix setups.

"need to make sure that it will not be spammed" ...

See in the next posting ...

"Our domain mail is through Google Apps so it complicates a few things"

To make sure not spam even if domain mail through google apps, DNS is a factor.  What DNS server is authoritative for the domains involved?

dig yourFQDN4MoodleServer -t TXT

Real example:

sosoftexas.org DNS WORLDNIC.COM

sos.sosoftexas.org has Moodle servers running versions 3.4 -> 3.7

sos.sosoftexas.org - which is a subdomain of sosoftexas.org

dig sos.sosoftexas.org -t TXT

will show an SPF record

Yet for top level domain ... sosoftexas.org ... mail is on google:

dig sosoftexas.org -t MX

If you'd share your domains with me via PM here in Moodle can check some things out for ya! smile

Am using postfix not set up to receive mail ... only send.

'SoS', Ken


Average of ratings: -
In reply to Emma Richardson

Re: SMTP connect error with Gmail after upgrade to 3.7

by Ken Task -
Picture of Particularly helpful Moodlers

"found out that what is going on is that forum post emailing has changed in 3.7 to the effect that they are breaking it down in multiple sessions"

How many users are subscribed to your forum(s)?

That in the release notes somewhere? or a setting somewhere? or hard coded? sad

'SoS', Ken

Average of ratings: -
In reply to Emma Richardson

Re: SMTP connect error with Gmail after upgrade to 3.7

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Emma

Good point about the parallel SMTP connections in Moodle 3.7. Haven't experienced that, I am still at 3.5. I thought I've seen suggestions on moodle.org exactly to this effect, to break up mail load in to sessions, since the MXs on the receiving end (hotmail.com, yahoo.com. etc) do not like a big load of mails in one go. Apparently Gmail behaves differently.

I run all three combinations of SMTP with Moodle. If you want to go Postfix on localhost, it is unbelievably simple. On Debian flavours of Linux, get do 'apt-get install postfix', tell the installtion script that yours is a site on the Internet. Then go through /etc/postfix/main.cf. Likely you need to focus on myorigin, myhostname and mydestination only. Restart the service. That is all on the Postfix side.

Getting the mail accepted by everybody is a longer story. The first step is to add the IP address of your serve to the TXT record of your domain in the DNS. See https://en.wikipedia.org/wiki/Sender_Policy_Framework#Implementation. Then carefully follow the SPAM warnings of the MXs. Improving the SPF record and/or progressing to DKIMhttps://en.wikipedia.org/wiki/DomainKeys_Identified_Mail, and further as necessary.
Average of ratings: Useful (1)
In reply to Visvanath Ratnaweera

Re: SMTP connect error with Gmail after upgrade to 3.7

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers
Ken, I actually put in a tracker and then had a great discussion with Andrew Nichols about it after he closed it. It is hard coded - the idea being to make things easier on the server but with unfortunate consequences for some! I have 800 subscribed to that forum. https://tracker.moodle.org/browse/MDL-66569 if you want to read the discussion.

I went ahead and set up Postfix using gmail-relay as the smtp relay and it works!! You are right, Visvanath, it was super simple. Had it up and running in under 20 minutes.

Thanks to you both for the support. It is always great to know someone else is helping to work through this stuff with you!
Average of ratings: Useful (1)
In reply to Emma Richardson

Re: SMTP connect error with Gmail after upgrade to 3.7

by Ken Task -
Picture of Particularly helpful Moodlers

Thanks for reply and info.   Interesting ... unless I've missed it, docs suggest using an SMTP server best, but not sure much is described in the setup of an MTA (Postfix or Sendmail).

On a side note ... in investigating your issue ... used Alpine ... a text based MTA which can be installed via package manager on most Linux based systems (CentOS one has to use epel).

Installation of Alpine allows testing outside of Moodle.

In alpine

Set SMTP server to
smtp.gmail.com:587/tls/user=youruserid@gmail.com

You can then compose an EMail to any of your Moodle users and test directly with the advantage of PostFix/Alpine handling bounces, etc.

The one thing that was annoying ... having to enter password for the gmail.com (or whatever Google addy using) all the time.

Fix:

Let's say you are root user.
From /root ... touch .pine-passfile
That creates a hidden file where alpine/pine will check for password
Put your password in that file.
nano .pine-passfile
paste your password ... yes, it's in clear text.  Risk am willing to take.

Next time you compose a message and send, you will not be prompted
to enter the password.

Another advantage ... emailing a user(s) with attachements larger than allowed in Moodle.

And a biggy ... smtp via Google ... message gets DKIM and DMARC without having to manually set that up.

Plus ... if one took the time to setup an address book in Alpine of teachers, one has an alt way of communicating with them outside of Moodle.

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: SMTP connect error with Gmail after upgrade to 3.7

by Sergio Rabellino -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Just two little things:
  • IMHO Alpine is not an MTA, to get moodle working with a local mail server you need to setup either postfix (simpler) or sendmail. Obviously, you can use Alpine to check your credential against remote mail server, as you suggest.
  • using an MTA like postfix/sendmail in chain with moodle not only solve this specific mail problems, but helps in guarantee the emails got delivered correctly, even if in a specific time (when your posts to forum should be sent), the google mail server (or any other mail server you use as a relay) cannot be contacted for any reason.

Hope this helps to clarify.
Average of ratings: Useful (2)
In reply to Sergio Rabellino

Re: SMTP connect error with Gmail after upgrade to 3.7

by Ken Task -
Picture of Particularly helpful Moodlers

@Sergio

Alpine ... unfortunate choice of terms ... yes, you are correct ... it is not an MTA - really a mail client.

Yes ... PostFix/Sendmail advantage of a mailque that will attempt to deliver later if upstream anything having issues.

Is also potential curse ... when PostFix/SendMail tries to send again a burst of resources used to do so happens.

If there are forced subscribed forums ... such as on front page ... I would hope it would be one way fourms ... Admin -> All users  smile

Have seen a Moodle with 7000+ users subscribed to a forum start to run as slow as one could imagine.   One might need to learn how to 'flush' ques ... even remove outgoing message ques.

Hey ... @Sergio ... wonder if you'd 'volunteer' to respond to:

https://moodle.org/mod/forum/discuss.php?d=390383#p1573693

All I can think of is 'Wow!". smile

'SoS', Ken

Average of ratings: -
In reply to Sergio Rabellino

Re: SMTP connect error with Gmail after upgrade to 3.7

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers
Yes, that was very similar to the post I used to set it up. I have actually found that using smtp-relay.gmail.com avoids the message number limit. I think you have to have a Google Apps account to use that though.
Average of ratings: -
In reply to Sergio Rabellino

Re: SMTP connect error with Gmail after upgrade to 3.7

by Arnoldo Diaz -

Dear Sergio:

Thanks a lot for sharing this information. Using the link you shared I'm able to send mail from a Postfix server though a G Suite user account. The idea is to use the Postfix server to be used as an intermediare server between Moodle and Gmail. However, I cannot send mail from our Moodle server using the intermediate (Postfx) server. Checking the mail.log information of the Postfix server I get the following error: 

connect from (our moodle server domain name)

SSL_accept error from..(our moodle server domain name)

lost connection after STARTTLS from (our moodle server domain name)

disconnect from  (our moodle server domain name)

The /etc/postfix/main.cf has been configured using the parameters shown in the link you shared.

Could you share your Moodle's configuration of "Site Admin -> Server -> Outgoing mail configuration"?

Any idea or suggestion will be very much appreciated.

Note: I am using Moodle 3.9
Average of ratings: -
In reply to Emma Richardson

Re: SMTP connect error with Gmail after upgrade to 3.7

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Emma

Delighted to hear that my babble inspired you to find your solution even though it was not my solution.
wink

I was not talking of a _relay_, just straight Postfix then to the MXs. In that solution all the Moodle configuration parameters, smtphosts | smtpsecure | smtpauthtype | smtpusername | smtppass, are empty or none. Then Moodle default values come in to play, which makes the mails to be delivered to the local SMTP, i.e. Postfix. Ref main.cf:mydestination. That's it!

In the case of mail relay, in addtion to the above you add the relay server to main.cf:relayhost and add the authentication parameters to smtp_sasl_*. (You'll find the long version in the link Sergio has given.) Did you really go that path?
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: SMTP connect error with Gmail after upgrade to 3.7

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers
Yes, I did. It gave me the benefit of a quick fix using the same path as previously, just with an interim step which actually gives me the added benefit of another step with detailed logs and actually makes it work! Using the same path (via gmail) stops the emails being flagged as spam which is important as my subscribers are teachers and the majority have never checked their spam folder in their lives and will just start flooding us with "why don't I get my notices/registration/certificate emails" any more questions.
I know it would be best to set up more dedicated email server for this purpose but it gets tricky when your domain email is already hosted at gmail. I wasn't sure how postfix would handle the gmail sessions but it seems to have kept the single connection alive while sending and the emails all went out.
Average of ratings: -
In reply to Emma Richardson

Re: SMTP connect error with Gmail after upgrade to 3.7

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Postfix SMTP delivering the mail to a mail relay, all in 20 minutes? Must have been some memorable 20 minutes!

No, you don't have to defend your solution, if it works for you without conditions. Yes, the local logs are a huge plus. That is what I meant by "independent", imagine begging Google for your logs. (They won't look at you.) Or asking the users for the explanation they get from their mail provider. (They haven't looked their spam folder all their lives. Ha, ha.)

The sad part is, your school have sold you and, worse, your kids to Google. You can't do much about it. This side of the pond is not much better. The Lord of the data here is called Microsoft. But that is OT. See the sub-thread starting at the end of this post https://moodle.org/mod/forum/discuss.php?d=390316#p1573552.

Back to the subject, to https://tracker.moodle.org/browse/MDL-66569 specifically. I understand the decision of the developers. I had many encounters with MXs complaining about too many mails in one connection. They force the sender to pause and resend, thereby purposely throttling the mail flow. But Moodle has to cater for large sites too. How would they send 10,000 copies of a single forum post? Breaking the flow in to multiple parallel connections is a big acheivement. But there can't be a Moodle-only solution, sending mail has to be delegated to a serious mail server. Mail servers are on their own incredibly fast. There are always reports on their speed records. We are far away from all that, our problem is the receiving end.
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: SMTP connect error with Gmail after upgrade to 3.7

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers
I have always stated that Google is evil - lol! That being said, I have a Google home device in almost every room in my house and love Android Auto in my car. I think you just have to realize that all that convenience comes at a cost and hopefully the kids are being taught that - here it is Google or Amazon (or Microsoft who is now making a play for the K12 education arena which Google has exploded into.) But it is hard to compete with free and so convenient so not sure if Microsoft will make a serious comeback or not...
Average of ratings: -
In reply to Emma Richardson

Re: SMTP connect error with Gmail after upgrade to 3.7

by Farid Lahdiri -

Hi Emma,

I am new to this.  I am helping a non profit organization to setup moodle.
I was wondering if you can expand a bit more on how you got this to work. 

I am running moodle 3.8.2 / Ubuntu18 /Droplet / Digital ocean.   I configured outbound email on moodle to use the smtp.gmail.com with the org's gmail account but  it did not work.
 
I then installed postfix on ubuntu and had to enable a 2-step verification with an app password on the google account. I am able to send email out from my ubuntu server (using the mail command)

In the moodle GUI -- I replaced  the SMTP hosts entry: smtp.gmail.com:587  by localhost (to point to my postfix MTA) 

Site Admin --> Server --> Email --> Outgoing Mail config --> 
                   SMTP hosts: localhost

That did not work. 

Can you share what you did to get it to work?

Thanks in advance.


Average of ratings: -
In reply to Farid Lahdiri

Re: SMTP connect error with Gmail after upgrade to 3.7

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hallo Farid

I am not Emma. Your problem is also not of hers https://moodle.org/mod/forum/discuss.php?d=390369#p1573723, a similar one, but not the same. Belongs to a new discussion.

You wrote:
> I am running moodle 3.8.2 / Ubuntu18 /Droplet / Digital ocean.

Never used a Digital Ocean (DO) "Droplet". Seen discussions by people using DO and assume it behaves identical to any Ubuntu Linux Server.

> I configured outbound email on moodle to use the smtp.gmail.com with the org's gmail account but it did not work.

If that is the path you want to go, then trying something else won't solve the problem. How "it did not work"? What did you do exactly? What was the result after each step? Ref https://docs.moodle.org/38/en/Email_setup_gmail.

> I then installed postfix on ubuntu

This is a new route.

> and had to enable a 2-step verification with an app password on the google account.

This is not connected to Postfix, right? You just want to use a 2FA service provided by Google?

> I am able to send email out from my ubuntu server (using the mail command)

That is a good start. So your Postfix is accepting mail from the local machine and relaying them. Analyse the full mail headers of such a mail at receiving end to see whether it has gone through something Gmail. (Do not use a Gmail account as a recipient.)

> In the moodle GUI -- I replaced the SMTP hosts entry: smtp.gmail.com:587 by localhost (to point to my postfix MTA)

In the local "sendmail" (Postfix) method, you leave the Moodle SMTP setting at their default. See the screenshot.

> Site Admin --> Server --> Email --> Outgoing Mail config --> SMTP hosts: localhost

See the attached screen-shot.

> That did not work.

Watch '# tail -f /var/log/mail.log' as Moodle is delivering a e-mail.
Average of ratings: -
In reply to Farid Lahdiri

Re: SMTP connect error with Gmail after upgrade to 3.7

by Félix Edmundo Martínez Retama -

Most of documentated procedures about how to configure GMail as SMTP are correct, except by the fact about which password to use,  which is not necesarly the same password you use to login in gmail. Follow the regular procedure but consider this:

To configure Gmail as SMTP , you need configure Google 2 step verification (Two factor authentication) which means you will need to add an extra security layer to your login account, either as message text or by using  google authenticator message, check this link about how to do it: 

https://support.google.com/accounts/answer/185839?co=GENIE.Platform%3DDesktop&hl=en

But that´s noth enought, this is a requirement for Google SMTP server, however moodle cannot work this way, since it requires  to login on your behalf , but it won´t work with the password, instead, you need to create an Application password in Google. Go to Google account-->security-->App passwords. You´ll be required to login again,  In Select App drop-down list, select Mail, in Select device choose Windows Computer, finally click Generate. You´ll get a 16 digit password. Just copy this one.


Now return to moodle an enter this 16 digit password in the field SMTP password and thats all. Test your configuration sending a test message, it should work.

I hope this helps!


Attachment Generate App password in Google.png
Average of ratings: Useful (1)
In reply to Félix Edmundo Martínez Retama

Re: SMTP connect error with Gmail after upgrade to 3.7

by Walter Byrd -

I am not sure I understand. Isn't the app password something for your phone? What does that have to do with SMTP?

Average of ratings: -