oauth2 Google Drive converter stopped converting files in the Moodle 3.3.7 AnnotatePDF function

oauth2 Google Drive converter stopped converting files in the Moodle 3.3.7 AnnotatePDF function

by Matthew Willis -
Number of replies: 8

Hi there!

A while back, I set-up our Moodle 3.3.7 site with Google Drive's document converter via the Google Cloud Platform and it worked pretty dandily until a few days ago where a teacher reported that the AnnotatePDF stopped converting the non-PDF files for them to annotate and grade. I haven't changed any configurations in Moodle or Google Cloud Platform that would indicate an issue. I've performed the below troubleshooting steps and still cannot get the documents to convert (primarily MS Office files):

  1. Ensured the service account is connected to the Google Cloud Platform in Moodle (/admin/tool/oauth2/issuers.php). This Moodle site only uses the Google OAuth2 service to convert MS Office files for the AnnotatePDF function, so no "Allow Login" functionality necessary. Moodle agrees that the Google Oauth 2 service is correctly configured and discovery is OK. Just to be sure, I reconnected the account to the Google account that the Drive API is configured:
    Note: I noticed that despite my Google oauth2 service being enabled as per the screenshot below, this does not show up in the Site Administration block under Site Administration > Server > Oauth 2 Services, where it did when I initially set up the converter.

  2. Ensured the client ID and shared secret in the Google oauth service page () match the Oauth2.0 client ID in the created Client ID for Web Application in the Google Cloud Platform (APIs & Services > Credentials > select the ClientID (I only have 1 entry on this page). I left these as-is when I created the service earlier in the year (and it worked then).

  3. Ensured the scopes mentioned in the above screenshot match those in Google Cloud platform.
  4. Ensured the Moodle site is authorised in the ClientID's settings > Authorized redirect URIs (https://my-moodle.site.com/admin/oauth2callback.php).
  5. Ghostscript path is valid in /admin/settings.php?section=systempaths and the passes the "test ghostscript path" function (mod/assign/feedback/editpdf/testgs.php).
  6. Ensured the Google Drive API is enabled in Google Cloud Platform. According to the Moodle documentation, this interaction between Moodle and google cloud doesn't require account verification to convert the files (according to here).
  7. Check that the Google Drive document converter is enabled and configured as the Oauth2 Service.
  8. Test the Google Drive document conversion (/files/converter/googledrive/test.php)
    This is where I saw an explicit issue with the converter. The test.php page returns "This document converter is configured properly", but the "Test document conversion" function generates an invalid PDF.

    In Firefox, it doesn't load the PDF in the browser (/files/converter/googledrive/test.php?sendpdf=1&sesskey=REDACTED_FOR_OBVIOUS_REASONS) and asks if you want to open it in a different viewer (my default viewer is Adobe Acrobat reader):

    In Chrome:

    In Acrobat:


    In an assignment in the AnnotatePDF feature (/mod/assign/view.php?id=somenumber&action=grader). Sections of the page redacted for student privacy.

I've re-read the set-up documentation to ensure the set-up is correct, and it is, but still cannot convert files in the AnnotatePDF page. What am I missing that can cause this issue? Luckily, I kept a copy of the Word and PDF files the "Test Document Converter" users and generates, which I've attached to this post as test materials.
Average of ratings: -
In reply to Matthew Willis

Re: oauth2 Google Drive converter stopped converting files in the Moodle 3.3.7 AnnotatePDF function

by Ken Task -
Picture of Particularly helpful Moodlers

Why is "login" not green?

Scopes included and
Scopes included in a login request for offline access
show openid profile email

Add to those: https://www.googleapis.com/auth/drive

'SoS', Ken


Average of ratings: -
In reply to Ken Task

Re: oauth2 Google Drive converter stopped converting files in the Moodle 3.3.7 AnnotatePDF function

by Matthew Willis -
Hi Ken,

My apologies for my belated response - I thought I had submitted a response last week!

The login status is not green as we are not using Oauth to log into the LMS - this is configured this way in other Moodle LMSs I'm an administrator of, and it doesn't affect the Moodle platform connecting to Google Drive API to convert the files to PDF. I've even connected this problematic Moodle to another Google Cloud Platform account that manages these other (functional) Moodle LMSs and connected it to this problematic one, and it continues to exhibit the same behaviour, which tells me there is nothing wrong with the configuration. I've looked at the php-errors.log files while troubleshooting again today, and they do not show any errors at those timestamps.
Average of ratings: -
In reply to Matthew Willis

Re: oauth2 Google Drive converter stopped converting files in the Moodle 3.3.7 AnnotatePDF function

by Matthew Willis -
Update: there is an error in the logs! The below line is triggered when using the Test Document Conversion function in files/converter/googledrive/test.php.

Here it is - how can I troubleshoot this further?
[10-Dec-2019 14:37:54 Australia/Sydney] PHP Warning: readfile(): Filename cannot be empty in D:\WebApps\MoodleLive\lib\filelib.php on line 1858.

this line refers to the readfile_allow_large function, and errors on the function $filesize = filesize($path), so the path that is passed to the function is incorrect someshow. Function code:
/**
* The readfile function can fail when files are larger than 2GB (even on 64-bit
* platforms). This wrapper uses readfile for small files and custom code for
* large ones.
*
* @param string $path Path to file
* @param int $filesize Size of file (if left out, will get it automatically)
* @return int|bool Size read (will always be $filesize) or false if failed
*/
function readfile_allow_large($path, $filesize = -1) {
// Automatically get size if not specified.
if ($filesize === -1) {
$filesize = filesize($path);
}
if ($filesize <= 2147483647) {
// If the file is up to 2^31 - 1, send it normally using readfile.
return readfile($path);
} else {
// For large files, read and output in 64KB chunks.
$handle = fopen($path, 'r');
if ($handle === false) {
return false;
}
$left = $filesize;
while ($left > 0) {
$size = min($left, 65536);
$buffer = fread($handle, $size);
if ($buffer === false) {
return false;
}
echo $buffer;
$left -= $size;
}
return $filesize;
}
}
Average of ratings: -
In reply to Matthew Willis

Re: oauth2 Google Drive converter stopped converting files in the Moodle 3.3.7 AnnotatePDF function

by Matthew Willis -
Bump. Still stuck at this error. I can't find any documentation that indicates the next troubleshooting step. We're continuing to experience this issue.
Average of ratings: -
In reply to Matthew Willis

Re: oauth2 Google Drive converter stopped converting files in the Moodle 3.3.7 AnnotatePDF function

by Ken Task -
Picture of Particularly helpful Moodlers
Scopes show openid profile email

**** Add to those the following URL: https://www.googleapis.com/auth/drive

and the only way I have been able to get it to work with doc conversions is to have a system account.

Even if you are not using Google Authentication.

And path to ghostscript OK? Have you tested conversion of a .doc or whatever to .pdf using ghostscript from command line?

https://www.ghostscript.com/doc/current/Use.htm

'SoS', Ken
Average of ratings: -
In reply to Ken Task

Re: oauth2 Google Drive converter stopped converting files in the Moodle 3.3.7 AnnotatePDF function

by Ken Task -
Picture of Particularly helpful Moodlers

and a follow up ...

https://docs.moodle.org/38/en/Google_Drive_converter

https://docs.moodle.org/38/en/OAuth_2_Google_service

and the section on app verification ...

where one sets up credentials
privacy policy url and terms of service url

Those pages I'd suggest making statically if your entity
doesn't already have them.

Google, like every service provider like them, ups the bar from time to time and really doesn't notify users (at least I've not found something to subscribe to that does).

Also, when users first access google via your Moodle they will or should be prompted with a permissions screen where the service account set up for this is given access to files ... wish they phrased that differently ... might warn users service account cannot be used to hack their stuff/edit files/upload files, etc.  Service account is sort of a 'pass through' and exist so there is some accountabiliy.

If your entity has also a Google for Schools domain might x-check about things in there admin of that might have to allow all users to do, etc.

'SoS', Ken


Average of ratings: -
In reply to Ken Task

Re: oauth2 Google Drive converter stopped converting files in the Moodle 3.3.7 AnnotatePDF function

by Matthew Willis -
Hi Ken,

Thanks for responding to my issue, but I'm afraid none of your suggestions worked! sad

I'll respond to both of your comments here to keep the responses concise.
-------------------------------------------------------------------------------------
Scopes show openid profile email

**** Add to those the following URL: https://www.googleapis.com/auth/drive

and the only way I have been able to get it to work with doc conversions is to have a system account.

Even if you are not using Google Authentication.

And path to ghostscript OK? Have you tested conversion of a .doc or whatever to .pdf using ghostscript from command line?

https://www.ghostscript.com/doc/current/Use.htm
-------------------------------------------------------------------------------------

The other Google Cloud Platform accounts do not have "https://www.googleapis.com/auth/drive" added to the list of "Scopes for Google APIs" on the "Oauth Consent Screen", they only have the default "Email", "Profile", and "OpenID" attributes and their relevant Moodle LMSs do not have issues with converting non-PDF files to PDF via the AnnotatePDF function, so I fail to understand why I need to add this scope to fix this issue.

I have connected a system account to the Google OAuth 2 services within the Moodle platform (as per the first screenshot in my OP with the connected system account's details redacted for privacy reasons. This was done in accordance with the two Moodle Docs you mentioned at the time the Google integration was set up (ages before I created the OP).

Ghostscript works and the path is valid. However, we do not want to use Ghostscript due to its glaringly horrendous performance while under stress (our Moodle platform uses the AnnotatePDF function frequently, and we noticed that Ghostscript was not able to keep up the pace to suit demand). Path to Ghostscript is irrelevant, as the whole point of using Google's API is to use Google Drive converter as an alternative to the Ghostscript service.
-------------------------------------------------------------------------------------
and a follow up ...

https://docs.moodle.org/38/en/Google_Drive_converter

https://docs.moodle.org/38/en/OAuth_2_Google_service

and the section on app verification ...

where one sets up credentials
privacy policy url and terms of service url

Those pages I'd suggest making statically if your entity
doesn't already have them.

Google, like every service provider like them, ups the bar from time to time and really doesn't notify users (at least I've not found something to subscribe to that does).

Also, when users first access google via your Moodle they will or should be prompted with a permissions screen where the service account set up for this is given access to files ... wish they phrased that differently ... might warn users service account cannot be used to hack their stuff/edit files/upload files, etc. Service account is sort of a 'pass through' and exist so there is some accountabiliy.

If your entity has also a Google for Schools domain might x-check about things in there admin of that might have to allow all users to do, etc.
-------------------------------------------------------------------------------------

I have followed the Google Drive converter and Google Oauth 2 setup as per the articles you refer to when setting up this account and other accounts (I am very familiar with them, especially over the last few months), but the issue still persists in this single Moodle LMS, despite even moving it to another Google Cloud Platform account that manages other Moodle LMSs of a similar version (all v3 and higher), and it continues to be the only Moodle LMS that experiences this issue converting files to PDF.

We don't have any privacy policy or ToS URLs configured, as the articles you mentioned does not require it. We do not have these configured in any other Google Cloud Platform account, so they are also not needed here. Therefore, the issue is not with my Google Cloud Platform configurations, but with the Moodle LMS itself, thus my delve into the PHP errors in my last post.

I have looked at the problematic account today for the first time since Christmas, and I have noticed it was called successfully 44 times, but I continue to experience the same non-converting issue mentioned in my OP, both with existing assignment submissions (test ones dating back to September 2019) and test attempts conducted today.
Average of ratings: -
In reply to Matthew Willis

Re: oauth2 Google Drive converter stopped converting files in the Moodle 3.3.7 AnnotatePDF function

by Ken Task -
Picture of Particularly helpful Moodlers

Your initial posting in this thread looks like errors that unoconv document conversion is throwing now with 3.5.10+

https://moodle.org/mod/forum/discuss.php?d=396281#p1597707

If you were to rename the downoaded 'document.pdf' to 'document.html' ... does the .html refer back to your site showing moodle front page without css formatting and showing not logged on?


In this posting the .docx and .pdf files attached
Didn't think the test actually converted any .docx, just shows what a converted PDF would look like.

do not want to use Ghostscript
Didn't think one could Annotate without it!

3.3.7 AnnotatePDF
Highest version of 3.3 is 3.3.9+.

Google Cloud Platform accounts
Confused .... did you move the entire sites to Google Cloud hosting  or does this refer to the Google accounts used with systems still on Windows?  Am thinking the latter.

In the Google Account you are configuring in Moodle ... API API's & Services
OAuth consent screen.

What does that look like?  OAuth rate limits?

https://docs.moodle.org/37/en/Google_Drive_converter
https://docs.moodle.org/35/en/Assignment_settings
File submissions
 (Check with your admin that Ghostscript and a document converter are enabled, if you can't annotate uploaded files.)

Feedback types
Annotate PDF appears if this setting is enabled by the Site administrator in the Manage assignment feedback plugins section of Site admin and will allow the teacher to annotate using comments,stamps and other features.

?????

'SoS', Ken


Average of ratings: -