Google Document Converter Slooooooooow.

Google Document Converter Slooooooooow.

by Scott Christy -
Number of replies: 24

Revisiting this topic a year later....

Anyone have any tips about how to speed up the Google document converter for the assignment grading screen? Document conversion works flawlessly in our Moodle 3.3 install but converting even a 1 page MS Word document with only text on it takes 5-10 seconds (or more). With how long it takes for each document to convert, it's actually quicker to download and print each assignment and grade them on paper instead.  This sort of defeats the purpose of collecting assignments online, but slow document conversion just makes the workflow too slow for the average busy teacher.

I'm wondering if anyone has any tips on how to improve the speed. I assume the bottleneck is on the Google side of things so I'm guessing there's nothing that can be done on the Moodle side?

What if documents were converted when the student submits their document for grading rather than when the teacher accesses the assignment to grade? That way the PDF would be ready to open immediately when the teacher needs to grade it.

Thanks in advance for anyone that has suggestions on this.


Average of ratings: -
In reply to Scott Christy

Re: Google Document Converter Slooooooooow.

by AL Rachels -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Hi Scott,

I am no expert, but so far as I know it doesn't matter whether you are using Google Drive or Unoconv to convert your documents, the conversion is activated by one of two methods. Conversion for a document starts right away if a teacher tries to grade right after something is submitted, or by a convert_submissions cron job that runs every 15 minutes.

In other words, if a teacher is grading submissions right after a student submits, there is going to be a noticeable delay due to the conversion process kicking in right away. If the teacher waits until cron runs, then the delay is much less and is just the amount of time it takes Moodle to find the converted document and send it to the teachers computer. On my server sitting right in my lab with me, this works out to about 2 to 3 seconds on average.

Once I learned this, my frustration factor went down a bunch because I started waiting until the next class period before starting to grade, unless there was a real need to grade, right now!

Average of ratings: Useful (2)
In reply to AL Rachels

Re: Google Document Converter Slooooooooow.

by Scott Christy -

Hmmmm.

I knew about the conversion running when the teacher tries to open the file. I did not know that it could also be based on the cron running. I'm wondering if our cron is not running as frequently as every 15 minutes. I have a feeling that our server team is running that cron on a much less frequent basis - like once every 24 hours or something like that. 

Thanks for the tip. Time to do some digging.

Average of ratings: -
In reply to Scott Christy

Re: Google Document Converter Slooooooooow.

by AL Rachels -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Hi Scott,

Cron frequency is something you should definitely check up on. The current recommendation for a Moodle site is for it to run every minute. Many different parts of Moodle then have their own cron times. In this case, the Prepare submissions for annotation is set to 15 minutes. If you go to Site administration > Server > Scheduled tasks, you can see all the different ones that have their own cron times.

Average of ratings: Useful (1)
In reply to AL Rachels

Re: Google Document Converter Slooooooooow.

by Scott Christy -
Ok, I think we've narrowed down where the issue is....


When I go into scheduled tasks I see:


So obviously the cron job for document conversions isn't running with my usual 5 minute runs. The good news is this is the only cron job not being executed.

So I clicked "RUN NOW" to see what would happen and I get this:



So I'm thinking it must be a permissions issue with the convert_submissions file. I went through my installation and the only file I can find that seems even remotely similar to the path listed on the task failed line is located at:

C:\inetpub\wwwroot\moodle\mod\assign\feedback\editpdf\classes\task\convert_submissions.php

(yes, it's a Windows server)

I changed the permissions to allow read, write and execute on that file, ran the scheduled task again, and it didn't solve it. Apparently that's not the right file? Or that's not the cause?

So my question(s) are:

  1. Am I correct that this is most likely a permissions error?
  2. Where is "assignfeedback_editpdf\task\convert_submissions" because I can't seem to find it?
  3. Or is there something else that is causing this?

Thanks!




Average of ratings: -
In reply to Scott Christy

Re: Google Document Converter Slooooooooow.

by Scott Christy -

I just ran the scheduled task with full debugging reporting on and got this. Anyone have experience with this problem that might have some better insight?

Execute scheduled task: Prepare submissions for annotation (assignfeedback_editpdf\task\convert_submissions)
... started 09:50:05. Current memory use 8.7MB.
Convert 1 submission attempt(s) for assignment 1550
... used 7 dbqueries
... used 0.22985792160034 seconds
Scheduled task failed: Prepare submissions for annotation (assignfeedback_editpdf\task\convert_submissions),error/nopermission
Backtrace:
* line 254 of \mod\assign\feedback\editpdf\classes\document_services.php: call to print_error()
* line 302 of \mod\assign\feedback\editpdf\classes\document_services.php: call to assignfeedback_editpdf\document_services::get_combined_document_for_attempt()
* line 97 of \mod\assign\feedback\editpdf\classes\task\convert_submissions.php: call to assignfeedback_editpdf\document_services::get_combined_pdf_for_attempt()
* line 141 of \lib\cronlib.php: call to assignfeedback_editpdf\task\convert_submissions->execute()
* line 249 of \lib\cronlib.php: call to cron_run_inner_scheduled_task()
* line 91 of \admin\tool\task\schedule_task.php: call to cron_run_single_task()


Average of ratings: -
In reply to Scott Christy

Re: Google Document Converter Slooooooooow.

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Could you solve your other permissions problem: "Invalid permissions detected when trying to create a directory..."
https://moodle.org/mod/forum/discuss.php?d=352794 ?
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Google Document Converter Slooooooooow.

by Scott Christy -

We solved that over a year or so ago.

Using UNOCONV to convert documents never worked, but when we switched to using the Google Document converter it converts documents just fine when you open an uploaded assignment in the grading screen. It just won't convert documents in the background as part of the usual processing of the scheduled tasks when the cron is run.

Average of ratings: Useful (1)
In reply to Scott Christy

Re: Google Document Converter Slooooooooow.

by AL Rachels -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
First off, I have never set up Unonconv on Windows, but thinking back to all the problems I had when setting it up on Centos and Ubuntu, all I can suggest is to go back over each step of the setup directions until you are sure you know what it is telling you to do and that you have actually done it. My main mistake was failing to set up a working "Listener" that worked.

  1. Most likely still a permission problem as the conversion is being done by Libre Office. Have to make sure it has the permissions needed to convert and save where needed.
  2. C:\inetpub\wwwroot\moodle\mod\assign\feedback\editpdf\classes\task\convert_submissions.php should be the one you are looking for.
  3. Not sure at this point. Listener, maybe?
Average of ratings: -
In reply to AL Rachels

Re: Google Document Converter Slooooooooow.

by Scott Christy -

We're not using UNOCONV for the document conversion. We're using only the Google Document converter option. We never had success getting the UNOCONV solution working and the Google Document option works perfectly...well, at least the "convert the document when the teacher opens the assignment" part of it.


Average of ratings: -
In reply to Scott Christy

Re: Google Document Converter Slooooooooow.

by AL Rachels -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Hi Scott,

Sorry, I got off track due to working on a getting a Moodle master branch setup to use Unoconv as Moodle 3.6 release is not too far off. Got to make sure the plugins I support have no issues with it.

So, looks like you just need to get the permissions straightened out to get the 15 minute cron job for conversions to work and do things in the background.


Average of ratings: -
In reply to AL Rachels

Re: Google Document Converter Slooooooooow.

by Scott Christy -

AL,

I'm pleased to hear this is one of your Moodle areas of work/expertise.  big grin Thanks for taking time to try to help me out on this.

So based on the information I provided in the threads above, do you think the permissions issue is with this file?

C:\inetpub\wwwroot\moodle\mod\assign\feedback\editpdf\classes\task\convert_submissions.php

If so, what should the permissions settings be? I know I mentioned this is on a Windows server but if you give me Linux settings I can figure it out.

Another consideration to me is the fact that the conversions work perfectly on their own. In other words, when you open an assignment to grade, the document converts fine (although slowly) and then opens in the grading screen. Does that process use the
C:\inetpub\wwwroot\moodle\mod\assign\feedback\editpdf\classes\task\convert_submissions.php
file as well? If so, how are the permissions between the scheduled tasks run and the "on the fly" conversion similar or different?


Again. much thanks for offering suggestions.

Average of ratings: -
In reply to Scott Christy

Re: Google Document Converter Slooooooooow.

by Scott Christy -

Any suggestions on the above questions? I'm meeting with our server guys later today and was hoping to have some additional insight to share with them.

Thanks.

Average of ratings: -
In reply to Scott Christy

Re: Google Document Converter Slooooooooow.

by AL Rachels -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Sorry for the delay. Was out of town for a funeral.

From your provided error printout, I would say yes, that the permissions problem is from the convert_submissions.php file. As to what the permissions settings should be, I have no idea as I have never tried to setup conversion for a windows server.

Average of ratings: -
In reply to AL Rachels

Re: Google Document Converter Slooooooooow.

by Scott Christy -

I finally have time to look at this issue again...

So I have the permissions for convert_submissions.php wide open and still it doesn't work.


Full debugging is showing this:

Execute scheduled task: Prepare submissions for annotation (assignfeedback_editpdf\task\convert_submissions)

... started 09:51:18. Current memory use 8.7MB.

Convert 1 submission attempt(s) for assignment 1550

... used 7 dbqueries

... used 0.13356685638428 seconds

Scheduled task failed: Prepare submissions for annotation (assignfeedback_editpdf\task\convert_submissions),error/nopermission

Backtrace:

* line 254 of \mod\assign\feedback\editpdf\classes\document_services.php: call to print_error()

* line 302 of \mod\assign\feedback\editpdf\classes\document_services.php: call to assignfeedback_editpdf\document_services::get_combined_document_for_attempt()

* line 97 of \mod\assign\feedback\editpdf\classes\task\convert_submissions.php: call to assignfeedback_editpdf\document_services::get_combined_pdf_for_attempt()

* line 141 of \lib\cronlib.php: call to assignfeedback_editpdf\task\convert_submissions->execute()

* line 249 of \lib\cronlib.php: call to cron_run_inner_scheduled_task()

* line 91 of \admin\tool\task\schedule_task.php: call to cron_run_single_task()


Any suggestions as to what to look at next?



Average of ratings: -
In reply to Scott Christy

Re: Google Document Converter Slooooooooow.

by AL Rachels -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Hi Scott,

Looks to me like you still are having permissions problems based on this line in your listing:

Scheduled task failed: Prepare submissions for annotation (assignfeedback_editpdf\task\convert_submissions),error/nopermission

I'm sorry that I don't have anything set up to use Google Document converter, so I am at a loss on how to proceed.

If anyone else has any ideas on what to try, please jump in.

Average of ratings: -
In reply to AL Rachels

Re: Google Document Converter Slooooooooow.

by Scott Christy -

Let's take another angle on this...

Could it be that the type of files being uploaded are causing this cron not to run? We're using Google Document Converter  but we're not using unoconv because we never were able to get it to work reliably.

As I understand it, Google Document converter can handle, rtf, doc, xls, docx, xlsx, ppt, pptx, pdf. If a student uploaded a png file for example, could that be enough to make the cron job fail?

Average of ratings: -
In reply to Scott Christy

Re: Google Document Converter Slooooooooow.

by Ken Task -
Picture of Particularly helpful Moodlers

Me thinks ... concentrate on the ;nopermissions' angle.

Don't think that is referring to the execution of the script (php file), but rather, the rendering of the pdf(s) for annotation ...

From my limited understanding ... If the assignment required submission of multiple files, Moodle, in the annotation interface, attempts to generate a multi-part pdf comprised of the submitted files.

So the no permissions might be really in moodledata/ and the directories used to create/house the PDF's.

Look in moodledata for related directories .... /temp/assignfeedback_editpdf as well as error logs.

Really a guess! :|

Version of Moodle?   Anything in tracker related?

'spirit of sharing', Ken




Average of ratings: -
In reply to Ken Task

Re: Google Document Converter Slooooooooow.

by Scott Christy -

If Moodle does the conversion ok the moment the teacher opens an assignment grading window wouldn't it be dropping the conversion into the same folder as when the cron job runs?

I'm wondering if the problem is a file type that Google doesn't know what to do with. I dug through the database to find the assignment that shows up in the error code and it contains a few submissions from that assignment that are .png files. Since I never was able to get UNOCONV to work on our installation I wonder if the cron is hanging up on that file since Google can't convert it and that's why it's failing to execute.

I could change the option on our Moodle to only allow the upload of file types that Google doc converter can deal with, but in the mean time already uploaded assignments with file uploads that Google cannot convert leads me to think that might be the issue.

I'll check the tracker again. Last time I looked I found some similar things but nothing that seemed to match what my installation has.

Average of ratings: -
In reply to Scott Christy

Re: Google Document Converter Slooooooooow.

by Scott Christy -
I figured out the problem.

Last night I tracked through the dbase tables to figure out what it was about assignment 1550 that was causing the problem. After tracking back through the class, the assignment submissions, etc, I finally tracked it back to the student in question. It was then that I discovered he had been removed from the class. Bottom line, it was this issue:


So I deleted all of the rows in the mdl_assignfeedback_editpdf_queue table and boom, the problem went away and the schedule task completes as it should.

I'll follow up with the solution in the tracker to do a more permanent fix later today when I have time.

Thank you everyone who gave me suggestions on this problem. I DO appreciate it.
Average of ratings: -
In reply to Scott Christy

Re: Google Document Converter Slooooooooow.

by AL Rachels -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Hi Scott,

Glad to hear you found the solution to your problem. Thanks for letting us know.

Average of ratings: -
In reply to AL Rachels

Re: Google Document Converter Slooooooooow.

by Scott Christy -

Well, I guess the problem is not fully solved after all.

The scheduled task is set to run every 10 minutes and it IS in fact running every 10 minutes as it should.

Documents are still loading very slowly in the assignment grading screen even though these are assignments that were uploaded well before the last scheduled task run when they should have been converted. It definitely seems that they are not converting in the background during the scheduled task execution. As an example, a simple 1 page MS Word document with only text in it is still taking 10-15 seconds to load. Documents that had been previously viewed load in no more than 1 to 2 seconds.

Does anyone have any idea why the scheduled task for document conversion is not in fact working in the background as it should? For document conversion we're using the Google Document Converter. We are not using unoconv because we never were able to get it to run correctly on our server.

Any suggestions would be very much appreciated.

Average of ratings: -
In reply to Scott Christy

Re: Google Document Converter Slooooooooow.

by Ben Kahn -

I don't have any solutions but am just adding that I am currently evaluating the Google Converter in my dev environment and seeing similar issues. I will post back here if I find any solutions.

Average of ratings: -
In reply to Ben Kahn

Re: Google Document Converter Slooooooooow.

by Scott Christy -

Wierd....

All of a sudden, a few days ago assignments started converting as they should in the background and open quickly when you go into the grading screen.

Not sure why it's all of a sudden working as it should. We'll just assume that "magic happens" and the problem worked itself out.

Average of ratings: -