Export huge question bank

Export huge question bank

by Ratko Bucic -
Number of replies: 12

Hi,

I have a huge question bank (11000+ questions inside).

Any attempt to export it result with error 500. Is there any way to workaround?

I guess one is to move group of questions to another bank and export, and so on... but then I will have multiple exports, and not one complete...

Ratko

Average of ratings: -
In reply to Ratko Bucic

Re: Export huge question bank

by Daniel Thies -
Picture of Core developers Picture of Plugin developers Picture of Testers
Have you tried increasing the memory limit on the server? Site administration -> Server -> Performance -> Extra PHP memory limit.

In reply to Daniel Thies

Re: Export huge question bank

by Ratko Bucic -

HI Daniel,

current settings for mem limit is 2G... I will try with 3 or 4G, but since it is shared hosting I assume hosting company will not be happy with that ;)


In reply to Ratko Bucic

Re: Export huge question bank

by Ratko Bucic -

Tried with 4G mem limit (that is max) and same...
Export reporting:
> Your export file should start to download shortly. If not, please click here.
then it crunching it in background, and after some time I get HTTP ERROR 500

Timeout (maxtimelimit) is also set to 0 and server php timeout is very high - 1800 sec.

In reply to Ratko Bucic

Re: Export huge question bank

by Daniel Thies -
Picture of Core developers Picture of Plugin developers Picture of Testers

Hi Radko,

Can you backup up the course instead of exporting the question bank? You should be able to uncheck all the options except question bank and then merge that into another course while restoring.

If that has the same problem, it might be helpful to know you Moodle version and more details on the server set up.

Daniel

In reply to Daniel Thies

Re: Export huge question bank

by Ratko Bucic -

Hi Daniel,

no, export/import works just fine. But exporting quiz course using backup feature doesn't include entire question bank, only questions included in specific course. We have setup with huge system question bank to keep it all on one place. So no questions inside separate course category.

I need this question bank export to use on another application, not on another Moodle smile 
As I said in my first post, workaround is to make temporary question bank, move some questions in that bank, export, and so on... with my testing, chunks of 500-1000 can be exported after a lot of waiting smile Questions are also full of images in description, so that make it bigger and require more resources to pull it from file storage.

I'm guessing this is probably related to server/php limits rather than Moodle one.

If that helps, environment:

- Moodle 2.9.8+ (Build: 20160915)

- MySQL: 5.6.32.78.1

- PHP Version 5.6.31

Server is shared hosting as I mentioned.

PS. Ratko not Radko ;)

In reply to Ratko Bucic

Re: Export huge question bank

by Daniel Thies -
Picture of Core developers Picture of Plugin developers Picture of Testers

Are you exporting the question bank to Moodle XML format or to a different question export formula? I believe that Blackboard exports and some other formats are compressed zip archives and that the zip library has  a 4G limit on it. This should not be the problem for XML format since it is uncompressed, but may be for others.

In reply to Daniel Thies

Re: Export huge question bank

by Ratko Bucic -

Tried export with all available formats. (GIFT, XML, XHTML... and Word export with additional plugin)
All formats behave same for full bank export.



In reply to Ratko Bucic

Re: Export huge question bank

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I imagine it is running out of memory in the middle of the process. If you temporarily set Debugging to Developer level, you will then get more information in the error message.

Average of ratings: Useful (1)
In reply to Tim Hunt

Re: Export huge question bank

by Ratko Bucic -

Yeah, it is live server, can't do that in the middle of week smile Maybe over weekend.


In reply to Ratko Bucic

Re: Export huge question bank

by Daniel Thies -
Picture of Core developers Picture of Plugin developers Picture of Testers

It should be safe to set

$CFG->debugusers = '2';

in config.php. This will display debug messages only to the admin account.

Average of ratings: Useful (1)
In reply to Daniel Thies

Re: Export huge question bank

by Ratko Bucic -


Well, not much help from that, since it doesn't get until that point to use moodle code to display debug message smile 
As I said, after some time site respond with error 500 - which is just message from web server, not from moodle.

However, in PHP error log I've found this error, related to export to XML (just after export died):

[07-Nov-2017 10:52:12] PHP Fatal error:  Out of memory (allocated 493355008) (tried to allocate 184045378 bytes) in /home6/pejoscie/public_html/question/format/xml/format.php on line 1131

Odd thing is that PHP memory_limit is set to 2GB, and it crashing under 500MB. 



Attachment capture_07112017_085910.jpg
In reply to Ratko Bucic

Re: Export huge question bank

by Eoin Campbell -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers

You could try backing up your database using mysqldump, then downloading it to a local PC which is also running Moodle, and importing it into that database instead. This should give you more control over the PHP environment, so that you can increase limits in the php.ini file.

I assume your 11,000 questions are categorised to some extent, so you should be able to export whole categories containing suitable numbers of questions without moving questions around much. 

Since you mentioned that questions contain images, and use on a different system, your probable only option for exporting is Moodle Question XML. Nothing else will keep all the data intact, I think.


Average of ratings: Useful (2)