Moodle 3.8 Oauth2 was working now getting error/moodle/Could not upgrade oauth token

Moodle 3.8 Oauth2 was working now getting error/moodle/Could not upgrade oauth token

โดย Colin Matheson -
Number of replies: 7

Upgraded a few weeks ago  and everything was working fine.

Now emails every hour saying "The refresh token for one of the OAuth services Google on your site https://moodle.carmelunified.org/moodle has expired. This will limit the functionality of any plugins that use this service. To fix this issue, visit the OAuth 2 Services configuration page and click on the "Connect system account" icon in the table row for this service. Be sure to login using the same service account for the OAuth system each time."

Using Google OAuth. I go to the services page and try to reauthorize and it gives me the error/moodle/Could not upgrade oauth token


การประเมินโดยเฉลี่ย: -
In reply to Colin Matheson

Re: Moodle 3.8 Oauth2 was working now getting error/moodle/Could not upgrade oauth token

โดย Ken Task -
รูปภาพของParticularly helpful Moodlers

Sounds like a scheduled task ... given email every X minutes.

Please see:
https://docs.moodle.org/38/en/OAuth_2_services#Refreshing_Access_Token

https://yoursite/admin/tool/task/scheduledtasks.php

Find oath2

Refresh OAuth tokens for service accounts \core\oauth2\refresh_system_tokens_task
If you have the path to php-cli set in your system there should be a 'Run Now' link.

Try the 'Run Now' link.

OR

Via ssh and command line:
cd /code/admin/tool/task/cli/
then
php schedule_task.php --execute="\core\oauth2\refresh_system_tokens_task"

'SoS', Ken

การประเมินโดยเฉลี่ย:Useful (1)
In reply to Colin Matheson

Re: Moodle 3.8 Oauth2 was working now getting error/moodle/Could not upgrade oauth token

โดย Leon Stringer -
รูปภาพของCore developers รูปภาพของParticularly helpful Moodlers

In this reply you say "users can't login using google Oauth", is this the same problem? Because the hourly email is from the Refresh OAuth tokens for service accounts scheduled task as Ken says. But these service accounts are to access shared services (e.g. a Google Drive repository) and isn't related to Moodle user logins with OAuth.

So is the site using OAuth authentication for Moodle users with Google as the issuer?

And is the site using a service, such as a Google Drive repository, through a Moodle service account? Presumably this isn't working?

If the site isn't using a such a service, could a service account have been inadvertently enabled in Moodle with the subsequent hourly refresh requests missing something causing Google to temporarily block OAuth access?


การประเมินโดยเฉลี่ย: -
In reply to Leon Stringer

Re: Moodle 3.8 Oauth2 was working now getting error/moodle/Could not upgrade oauth token

โดย Colin Matheson -
Yes the email is from the scheduled task. We do use Google Oauth2 as our main authentication for users (and also for Drive repository access) . When I get the error email and I go to the Oauth page and try to manually refresh the token it will error out with this message error/moodle/Could not upgrade oauth token. I then try again 5-10 times and it eventually is successful. Then I don' tget the email for 2-12 hours and it comes back.
We don't have a lot of users active right now but they are starting to come back and when the error is occurring Oauth login doesn't work.
I wonder if there is a setting in the database config that is making the error occur on some occasions.
Intermittent errors that appear and then go away are so tricky to troubleshoot.

I found the cli interface for the task and ran it (it was in this folder)
moodle/admin/tool/task/cli
But since the system is working at this moment it didn't return any interesting errors.
I looked in the Apache error logs for when it wasn't working and found this line, but not sure how to use that to investigate the source of the problem.
Default exception handler: error/Could not upgrade oauth token Debug: \nError code: Could not upgrade oauth token\n$a contents: \n* line 571 of /lib/oauthlib.php: moodle_exception thrown\n* line 479 of /lib/oauthlib.php: call to oauth2_client->upgrade_token()\n* line 905 of /lib/classes/oauth2/api.php: call to oauth2_client->is_logged_in()\n* line 175 of /admin/tool/oauth2/issuers.php: call to core\\oauth2\\api::connect_system_account()\n, referer: https://accounts.google.com/signin/oauth/consent?authuser=3&part=AJi8hAPE8DCgo9XGGJLiLV7zmYm0ZiZyxKv8GdS1kqvzD97qdylCjPmZg3onWiNgiDcf_nySrM2NjVw-oAdri33RUD50BsKvqGUPPTYN9nsEleS2uE2UhxwRbq-BNzQXCQTZ_xCSGNgvii-OVaoaLntvESPJvYYN5c7nUFTHC5JVFsQkScBDlIT68dltIz1vBuK3f9EmSWAX2MaCh_KHvkLMh_eIw1rlMVXrpKghQtq00w3ZbO75kSIa4W-IAe43ZJ4BJ_dwwgLYgV6Ijx-XXmMfV4HCY6sEzS-pMMCW9_yaPp-IGlKOZ_vReXEMMnQzB0n823I5do8xraYJyYdber3o3NtaX1fl40gs5vKcI6nf1Mn28mg4fnP8isRMQEbrMFyA5P0mif7s9LrvRKjgo6qRjhjYaRtFNeXrej2RlwezoWzVcHBy659y0JInJBFG0WKkotxi-tFvDW2yLYCUBvqStHxWzvERkudko982QHUXENZV7O_i4N2O-IuxBw4F_mLeXBDHp8XsQ6-8eCRE-fuEBod_JyWANWBvvfPeL-jXQGlrUGiW43E2y6k52J138ZItKU6osvja8s5iRXgwmgh-PVcMpkJ2eNmzVqdXEuahTQHDiLGVtLiyKb3_1WMBIaFcLl4lj5j6&hl=en&as=S1426925996%3A1595707631901837
การประเมินโดยเฉลี่ย: -
In reply to Colin Matheson

Re: Moodle 3.8 Oauth2 was working now getting error/moodle/Could not upgrade oauth token

โดย Colin Matheson -
Our network admin believes the error was due to changes in the Google account we used for the moodle oauth service not having access to API. Not sure if that was a Google change or a change we made to our permissions. We now have given the service account the higher level of permissions needed and will see if that fixes the issue.
การประเมินโดยเฉลี่ย: -
In reply to Colin Matheson

Re: Moodle 3.8 Oauth2 was working now getting error/moodle/Could not upgrade oauth token

โดย Colin Matheson -
Well that Google account wasn't the problem. We also noticed some database write errors which might be getting in the way of the oauth handshake. It works for a bit and then doesn't for a bit. Perhaps it is a problem with our database setup on this new server that didn't appear until a week or so after upgrade?
การประเมินโดยเฉลี่ย: -
In reply to Colin Matheson

Re: Moodle 3.8 Oauth2 was working now getting error/moodle/Could not upgrade oauth token

โดย Ken Task -
รูปภาพของParticularly helpful Moodlers

' ... noticed some database write errors ...'

like what?  Can you be specific?

And this part of 'works for a bit' ... then doesn't ... sounds like issues with cron or scheduled task.

'SoS', Ken

การประเมินโดยเฉลี่ย: -
In reply to Colin Matheson

Re: Moodle 3.8 Oauth2 was working now getting error/moodle/Could not upgrade oauth token

โดย Leon Stringer -
รูปภาพของCore developers รูปภาพของParticularly helpful Moodlers

Colin: Did you make any headway with this? One of the problems with OAuth is that when problems arise it's hard to see what's happening under the hood. You could try adding some debugging to lib/classes/oauth2/client.php to display the response when there's an error (i.e. insert the six lines after line 255):

233     public function upgrade_refresh_token(system_account $systemaccount) {

255         if ($this->info['http_code'] !== 200) {
256             error_reporting(E_ALL);
257             error_log("[OAuth Debug] HTTP status: " . $this->info['http_code']);
258             if ($this->error) {
259                 error_log("[OAuth Debug] Curl error" . $this->error);
260             }
261             error_log("[OAuth Debug] Response: $response");
262             throw new moodle_exception('Could not upgrade oauth token');
263         }

This should send some debug output to the shell if the scheduled task is run at the CLI, or to the PHP error_log if run via the web interface. E.g. if I break my issuer and run the scheduled task I see:

# sudo -u apache php admin/tool/task/cli/schedule_task.php --execute=\\core\\oauth2\\refresh_system_tokens_task
Execute scheduled task: Refresh OAuth tokens for service accounts (core\oauth2\refresh_system_tokens_task)
[OAuth Debug] HTTP status: 400
[OAuth Debug] Response: {"error":"unsupported_grant_type","error_description":"Grant type \"refresh_token\" not supported"}

which shows what the issuer didn't like about the request. Fingers crossed the output for Google will be similarly helpful.

การประเมินโดยเฉลี่ย:Useful (2)