Security and privacy

Incoming mail configuration with Google accounts detected as insecure

 
Picture of Iñigo Zendegi Urzelai
Incoming mail configuration with Google accounts detected as insecure
 

Hi there,

We've been using the Moodle's Incoming Mail feature with GMail mailboxes for years, but recently Google has begun rejecting the connection from Moodle for been insecure.

I've read the documentation about it, have already enabled the "Allow less secure apps" setting and I guess it will solve the problem, but I guess it shouldn't detect that connection as insecure, should it? Is there maybe something to be done in Moodle to make it compliant with the security rules?

We're using Moodle 3.1.10 and this is our configuration:

Incoming Mail Configuration

 
Average of ratings: -
Picture of Iñaki Arenaza
Re: Incoming mail configuration with Google accounts detected as insecure
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

Hi Iñigo,

Google considers anything that doesn't use Oauth + HTTPS to access Gmail as a "less secure app". I guess the idean behind this is that the old POP / IMAP protocols don't 2FA (support two factor authentication). But I'd also say this is a a bullying tactic from Google to move people from old email apps (Outlook, Thunderbird, others) into their products, so they can control how you consume you Gmail mail (this is probably related to the their upcoming AMP move; see https://www.blog.google/products/g-suite/bringing-power-amp-gmail/).

So, IMHO, this is partly enhanced security (using 2FA) and partly FUD (using IMAP+SSL should be as good as using HTTPS).

Saludos. Iñaki.

 
Average of ratings: Useful (3)
Picture of Iñigo Zendegi Urzelai
Er: Re: Incoming mail configuration with Google accounts detected as insecure
 

Hi Iñaki,

Certanly there is some FUD from Google in this issue, because this same configuration is actually working in some instances and it makes troubles in others, that's the most confusing thing about it.

Anyway, if this can only be definitely solved using OAuth, maybe the Incoming Mail Configuration feature should be changed to use the OAuth 2 service?

Thanks.

 
Average of ratings: -
Picture of Iñaki Arenaza
Re: Er: Re: Incoming mail configuration with Google accounts detected as insecure
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

Hi Iñigo,

changing Incoming Mail configuration feature to use the OAuth2 service would make it work with Gmail and Outlook.com (they support it, according to https://msdn.microsoft.com/en-us/library/dn440163.aspx), but would leave pretty much any other IMAP service that doesn't support XOAUTH2[1] authentication unusable (and there quite a few of them).

In addition to that, in Gmail's case, OAuth2 authentication is only offered/required if the mailbox doesn't have two factor authentication enabled (2FA). If it is, the IMAP service simply doesn't offer the authentication option. Using 2FA from an application that doesn't have the user interacting with it at login time is rather complicated.

That's why, again in Gmail's case, they offer the option to create static application passwords. These passwords are special, in the sense that they are not subject to 2FA, and can be used as a valid password. They are auto-generated by Gmail, and you have to copy them manually to your application. I'm telling all this for those that are using a Gmail mailbox and don't want to go all the trouble to configure an OAuth2 service and configure XOAUTH2 in their moodle site (assuming they have their moodle site patched to include such functionality). It's less secure than using XOAUTH2, but it can be more secure than enabling the "less secure applications" options in Gmail, and using static passwords (the static application passwords auto generated by Google are longer than most people's passwords, and thus harder to guess).

So to summarise, we would need to support both XOAUTH2 and the traditional authentication methods. The good news is that Horde (the library Moodle uses to access IMAP mailboxes) already includes support for XOAUTH2. With a few hours of coding and fighting with Google machinery to create a client API key and secret, I've developed a patch that works in my test environment, using a Google Suite[2] account/mailbox.

In order to use it with a Gmail mailbox, you need to create a OAuth2 Google Service (following the instructions given at https://docs.moodle.org/34/en/OAuth_2_services), with two changes to the default values created by Moodle (see attached image):

Gmail XOAUTH2 service

In addition to that, you must use a system account with that OAuth2 Service, as it is intented to be used without any kind of user interaction. The easiest way is to use the account owning the mailbox:


Gmail XOAUTH2 System Account

Once you have all that in place, you can patch your Moodle site (the attached patch applies cleanly to Moodle 3.4 current as of today, but probably applies without too much trouble to other versions). Once you have patched it, if you go to the "Incoming Mail Configuration" page, you will find two new configuration settings (see attached image):

Incoming mail XOAUTH2 config

  • Use XOAUTH2: enable to use XOAUTH2 instead of the traditional authentication methods
  • OAuth2 Service: select the OAuth2 Service that you created above, to be used for XOAUTH2

If everything is correctly setup, you won't even need to configure the real password for the IMAP user, as it's not used by XOAUTH2 at all (but you still need to put something in there, as otherwise the Horde IMAP library complains loudly!).

You can now manually run the "Incoming email pickup" scheduled task to see if everything works as expected.

XOAUTH2 patch should work with other email providers, as long as you create a valid OAuth2 Service entry for that provider, following the provider documentation (you might need a custom OAuth2 service in many cases).

Saludos.

Iñaki.

[1] That's the name of particular authentication method they use for both IMAP and SMTP.

[2] Former Google Apps for Business/Education.


 
Average of ratings: Useful (3)
Picture of Iñaki Arenaza
Re: Er: Re: Incoming mail configuration with Google accounts detected as insecure
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

Iñigo, I forgot to add that if you test this and it works for you, let me know so I can create a bug to include the new functionality in core.

Saludos.

Iñaki.

 
Average of ratings: Useful (1)
Picture of Iñigo Zendegi Urzelai
Er: Re: Er: Re: Incoming mail configuration with Google accounts detected as insecure
 

Great work Iñaki, thanks!

Exactly this is what I thought it should be done in Moodle in order to remain working with other IMAP services, I would love to see this patch integrated in Moodle!

I'll test the patch on a Moodle 3.4.1 instance and will give you feedback.


 
Average of ratings: -
Picture of Iñigo Zendegi Urzelai
Er: Re: Er: Re: Incoming mail configuration with Google accounts detected as insecure
 
Hi Iñaki,

I've applied your patch on a Moodle 3.4.1, tested it with the "Emailing_private_files_as_an_attachment" function and it works like a charm (*), so here it is my +1 to propose this function to be included in the core functionality.


(*): I've tried to do so with "Reply_to_posts_via_email" function but found an unrelated issue and haven't tried it yet, but I guess it would also work as both use the incoming mail function in the same way.

 
Average of ratings: -
Picture of Iñigo Zendegi Urzelai
Er: Re: Er: Re: Incoming mail configuration with Google accounts detected as insecure
 

Hi,

Just noting two things:

- I've solved the issue using the reply to posts via email and tested Iñaki's patch with that functionality without any issues.

- Some time after setting the system account for the Gmail XOAUTH2 service (a week or so), Moodle sends an email warning about the token being expired (the string is oauthrefreshtokenexpired), but this functionality keeps working after that.

 
Average of ratings: Useful (1)
Picture of Iñaki Arenaza
Re: Er: Re: Er: Re: Incoming mail configuration with Google accounts detected as insecure
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

Hi Iñigo,

I've tried to reproduce your issue, but it's taken a bit long (I've been side-tracked by other issues, unfortunately)

I've had a look at the code that sends the message you mentioned, and its sent from one of the scheduled tasks. There's a task that is in charge of refreshing "system account" tokens, configured to run every half past hour (at minute 30).

As long as that task is run, the refresh and access tokens should be refreshed and shouldn't expire. Of course, a temporary problem might make the refresh process fail, and you'd get the message. I'd be more worried if you got such a message every half past hour (or the task scheduled time, if you changed it).

Given that refresh tokens (the ones used to obtain refreshed access tokens) have long(ish) expiration times (in the order of several days, even some weeks), getting this error from time to time shouldn't be a problem. It should recover automatically next time the task is run (I'm assuming you keep running the task once an hour, or every few hours), as long as the problem preventing the tokens from being refreshed persists (e.g., network connectivity with your OAuth2 provider, the associated OAuth2 account being revoked or the OAuth2 permissions being revoked, etc.).

Saludos.

Iñaki.

 
Average of ratings: -
Picture of Iñigo Zendegi Urzelai
Er: Re: Er: Re: Er: Re: Incoming mail configuration with Google accounts detected as insecure
 

Hi Iñaki,

You're right, even after receiving that email about the token being expired the incoming mail setting is still working fine, so I think there is no technical issue to fix, maybe just changing the oauthrefreshtokenexpired string may be enough; after reading what you said, I suggest something like this:

Current message:

The refresh token for one of the OAuth services {$a->issuer} on your site {$a->siteurl} has expired. This will limit the functionality of any plugins that use this service. To fix this issue, visit the OAuth 2 Services configuration page and click on the "Connect system account" icon in the table row for this service. Be sure to login using the same service account for the OAuth system each time.

Proposal:

The refresh token for one of the OAuth services {$a->issuer} on your site {$a->siteurl} has expired. This shouldn't be a problem as long as you receive this email it occassionally and you have the "Refresh OAuth tokens for service accounts" scheduled task running with the default frecuency. Otherwise, this will limit the functionality of any plugins that use this service. To fix this issue, visit the OAuth 2 Services configuration page and click on the "Connect system account" icon in the table row for this service. Be sure to login using the same service account for the OAuth system each time.


 
Average of ratings: -
Picture of Matteo Scaramuccia
Re: Er: Re: Incoming mail configuration with Google accounts detected as insecure
Core developersParticularly helpful MoodlersPlugin developers

Hi Iñaki,
did you already file an issue into the Tracker?

It would be nice to use the new authentication method even when using GMail for SMTP (i.e. outgoing).
IMHO both features should be a must have for 3.5.

Worth noting that PHPMailer has its own implementation: https://github.com/PHPMailer/PHPMailer/wiki/Using-Gmail-with-XOAUTH2 and https://github.com/PHPMailer/PHPMailer/wiki/Gmail-XOAUTH2-Using-Google-API-Client for the drawbacks.

TIA,
Matteo

 
Average of ratings: -
Picture of Matteo Scaramuccia
Re: Incoming mail configuration with Google accounts detected as insecure
Core developersParticularly helpful MoodlersPlugin developers

Hi Iñaki,
I've just added it: MDL-61921.

It would be nice if you could contribute on it since you've already coded more than a working patch for the incoming messages with a nice Moodle-ish solution smile.
You already deserve the credit for it!

TIA,
Matteo

 
Average of ratings: Useful (1)