General add-ons

 
 
Picture of Mark Johnson
Bulk course upload 2.0
Group Developers

I've just had a bash at updating the Bulk Course Creation tool (uploadcourse.php) for Moodle 2. I've got it working, and it should work with databases other than MySQL (it uses $DB now).

It's not very tidy,  and it's not packaged as a plugin, just drop it into /admin as before.  To be honest, I've done just enough to get it working for me and commented out the features I don't need and therefore haven't upgraded (teachers, templates and topics), so any contributions and improvements are welcome.  It's up on github: https://github.com/marxjohnson/moodle_admin-uploadcourse.php

 
Average of ratings:Useful (3)
Picture of saif nat
Re: Bulk course upload 2.0
 

thanks for you i will testing it

and i will give you my notes

 

thanks

 

saif

 
Average of ratings: -
Picture of Mark Johnson
Re: Bulk course upload 2.0
Group Developers

There was a bug in the default_category function, fix is now on github.

Just a quick note about performance: The old version of the script used native mysql functions to speed up processing (the comments at the top mention cutting down from ~30 minutes via the old ADODB API to ~1 minute.  All queires (that affect the way I use it) now use Moodle 2's DB API - I just tested a file creating about 650 courses and it processed in about 40 seconds.

 
Average of ratings: -
Picture of Jason Shelvy
Re: Bulk course upload 2.0
 

Hello, I have a question that maybe some one can help with. When I run the uploadcourse script I get the following error:

An error occurred creating category with name BIO, meaning a total 1 category(ies) failed

not sure what is creating this problem. This happen if a category is already created or not.

any help would be appreciated

 
Average of ratings: -
Picture of Mark Johnson
Re: Bulk course upload 2.0
Group Developers

Hi Jason,

I've implemented creation of categories. If the category does already exist, you currently need to use the ID, not the name.

Hope that helps,

Mark

 
Average of ratings: -
Picture of Jason Shelvy
Re: Bulk course upload 2.0
 

Thanks you Mark I will give that a try.

Jason

 
Average of ratings: -
Picture of steve brydges
Re: Bulk course upload 2.0
 

Hi Mark,

   I'm trying this for the first time on a brand new Moodle 2.1. The csv file looks like this:

category,format,fullname,idnumber,numsections,shortname,showgrades,showreports,sortorder,visible
Medical School,topics,MBCHB,a101,10,MBChB,0,0,1,1
Maths,topics,Integers,ma231,10,Integers,0,0,1,1
English,topics,Shakespeare,en231,10,Shakespeare,0,0,1,1

and it is failing like this:

Parsed 3 course(s) from CSV
An error occured creating category with name Medical School, meaning a total 1 category(ies) failed
Course with ShortName MBChB could not be created because parent category(ies) failed
An error occured creating category with name Maths, meaning a total 1 category(ies) failed
Course with ShortName Integers could not be created because parent category(ies) failed
An error occured creating category with name English, meaning a total 1 category(ies) failed
Course with ShortName Shakespeare could not be created because parent category(ies) failed
Created 0 course(s) out of 3
3 course(s) failed due to errors
3 new category(ies) failed due to errors
 
Do you have an example of a correctly formatted csv file?
 
 
Average of ratings: -
Picture of Mark Johnson
Re: Bulk course upload 2.0
Group Developers

Hi Steve,

With the current version of the script, the category must contain the ID, not the name, of the category the course should be under.  I think I minced my words above - as I recall (it was a while ago now) I haven't implemented creation of categories that don't already exist.  You need to create them first, then upload your file using the category IDs.

 
Average of ratings: -
Picture of Steve Anatai
Re: Bulk course upload 2.0
 

Hi Mark.

Thanks for writing this script - it is a much needed feature.

I am in the process of trying this out on our testserver at school.  Will let you know how it goes.

I recently wrote a similar script that bulk-creates categories (we make heavy use of SIMS class synchronisation and the course importer allows courses to be assigned to categories - the only snag is, they have to already exist!).  I didn't see an obvious way of just creating categories with your script, but please let me know if I've missed something.  I'm happy to make my code available on request.

All the best,

Steve

 
Average of ratings: -
Picture of Paul Fynn
Re: Bulk course upload 2.0
 

Looking forwards to hearing how this goes....... 180 courses to create sad

BTW - anyone know who chose the acid tablet as the logo for the smileys ?

 
Average of ratings: -
Just wondering . . .
Re: Bulk course upload 2.0
Group Particularly helpful Moodlers

This request is the top-equal all time issue on the tracker with 88 votes, dating back to January 2008.  The top one, ical import has been written and is looking good)  If someone votes for this, then it will be the top.

Top Voted issues on Tracker

Here is the course create issue is you want to vote: http://tracker.moodle.org/browse/MDL-13114

-Derek

 
Average of ratings: -
Picture of Paul Fynn
Re: Bulk course upload 2.0
 

125 courses created by hand.....another 60 to go sad

Can anybody report back on using Marks script with 2.0 or, preferably, 2.1?

Many thanks

Paul

 
Average of ratings: -
Picture of Bruno Morais
Re: Bulk course upload 2.0
 

Just used the script on a new 2.2 install using MySQL and php 5.3.8, worked swiftly enough to save me an afternoon of manual labour. Many thanks, Mark!

 Slight issue, though, not sure if it's fixable or not:

- Being a PT school, several course names had special characters. These got "ignored" during upload. I tried to code the csv file in UTF-8 and in every alternative to ANSI. To no avail.

 As a script/module it would be much more powerful without this handicap.

Still - awesome work

BM 

 
Average of ratings: -
Picture of Mark Johnson
Re: Bulk course upload 2.0
Group Developers

Hi Bruno,

What sort of special characters are you have trouble with? Accented letters etc?

I'm not sure why any of that would cause a problem, but I very rarely work with anything other than basic latin characters and punctuation so I'm far from an expert on the matter, I'm afraid.

 
Average of ratings: -
Picture of Bruno Morais
Re: Bulk course upload 2.0
 

Hi

 Yes, the problem was with accented letters, tilde and comma on c... the following characters, give or take:

á, à, é, í, ó, ã, ç 

As I said, your code didn't accept files coded in any other form than ASCI. If you (or any one) could turn it into UTF-8 friendly, it would suit the international community. Between Portugal, Spain, Romania, Turkey, etc.. I can think of so many countries with special characters needs :D

 Cheers and happy new year!

 
Average of ratings:Useful (1)
Picture of Graham Paull
Re: Bulk course upload 2.0
 

Bruno,

How did you get this working with MySQL ?

My CSV seems to fail at teacher1_account with the following error:

string(5) "Admin"

get_record() not available anymore

I have attached my CSV if you don't mind having a look.

Thanks,

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

For those that maybe interested, an admin tool plugin has been developed for 2.2+ that does bulk course upload operations in a similar vein to user upload.

This enables create/update/delete of courses, and the ability to nominate either a Moodle course backup, or existing course as a template.

https://github.com/piersharding/moodle-tool_uploadcourse

Cheers,

Piers Harding.

 
Average of ratings:Useful (1)
Picture of Sam Stegers
Re: Bulk course upload 2.0
Group Testers

Piers,

I am very much looking forward to give your tool a go. Somehow I have not been able to find a .csv template which I could use. Could you supply a .csv file where all available columns are present, so I know what to put where.

TIA,

 

Sam

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi Sam -

Please remember that you must be on 2.2 and above.  I have extended the documentation at https://github.com/piersharding/moodle-tool_uploadcourse with column names and a trvial example.

Cheers,

Piers Harding.

 
Average of ratings: -
Coney Island
Re: Bulk course upload 2.0
 

Piers,
Thank you for all your work on this! This is a very nice tool. Have you looked at integrating teacher enrollment along with the courseupload? The tool for pre-2.2 had that capability and now we are having to do a lot of data manipulation to get enrollments sorted by teacher for the user upload tool rather than simply enrolling the teachers with the course upload. Fortunately we are only dealing with 250 courses. I believe the enrollment module has changed significantly even between 2.2 and 2.3 so maybe it's not possible but it would really help for next time.
Thanks,
Erik

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi Erik - I think it would be a case of replicating too much functionality that already exists in the CSV user upload tool that is part of Moodle core - I understand that it is a pain having to juggle both tools, but I would prefer not to do that.

 

Cheers,

Piers Harding.

 
Average of ratings: -
Me
Re: Bulk course upload 2.0
 

Hi Piers,

I'm experimenting with your tool at the moment, as we have a large number of courses to set up (>1000), some with templates, others default. I'll probably do it in batches, so as not to overwhelm the system.

Could you please explain what the "teacher" and "teachers" fields are for, and what the format of them should be? I'm imagining that the "teacher" field is the role name, and the "teachers" is the users - should this be the database ID, login id, fullname...?

Many thanks in advance, and thank you for *hopefully* making my life so much easier!

Richard

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi Richard -

This was not working correctly, and I have revamped the code around it so you will need to get an updated version from https://github.com/piersharding/moodle-tool_uploadcourse

I have relabelled the field names to:

'teacher', 'editingteacher', 'student', 'manager', 'coursecreator', 'guest', 'user'

Which are the role name substitutions that you can do when you go and edit the course details eg: for manager you might want 'purse string holder' - or some such.

Please test on a small group of courses first - the effects you could possibly get from uploading course backups over the top of an existing course, or doing a template copy can vary wildly depending on how you setup those templates, or make those backups.  Make sure you backup before you go.

Let me know how you get on.

Cheers,

Piers Harding.

 
Average of ratings: -
Me
Re: Bulk course upload 2.0
 

Thanks Piers, I understand now. I guess I'll have to also employ the csv enrolment plugin to get the correct teachers on the courses.

Don't worry, I'll be taking a cautious approach! I'll report back here once I have some results...

Thanks again,

Richard

 
Average of ratings: -
Picture of Flavio Martins
Re: Bulk course upload 2.0
 

Hi, Piers

I've been trying to upload a csv file that has this format:

fullname;shortname;category;summary;format;showgrades;numsections
Moodle course 1;Course 1;Courses/Discipline 4/Classroom 1;a;topics;0;6
Moodle course 2;Course 2;Courses/Discipline 4/Classroom 1;a;topics;0;6
Moodle course 3;Course 3;Courses/Discipline 4/Classroom 3;a;topics;0;6

When I upload it, moodle reads it fine at first, but when I click on the "upload courses" button, moodle shows a page with only the table headers without lsting the results. And here comes the bizarre: Later checking the course list I found that only the second line of the csv file (first course) were added in moodle.

If I try to upload the csv file again, it adds the next course cause the previous one is already recorded...

What did I do to make it fail like that? It was supposed to add all courses in one way, right?

Thanks!!

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi Flavio  - can you try turning on Moodle debugging and see if that gives us some further information.  You are correct - it should process all your course creations sequentially, so it sounds like there is some kind of error happening after the first course is created.

 

Cheers,

Piers Harding.

 
Average of ratings: -
Picture of Damien Kempt
Re: Bulk course upload 2.0
 

Dear Piers,

 

I've been trying out your course upload plugin and it seems to work great.  I have one question, is it possible to assign courses to a sub-category with an upload.  I tried entering a sub-category in the category column in this format "Category Name/Sub-Category Name", but it creates the following error:


Invalid value for field category
Can not find data record in database table course_categories.


The category and sub-category definitely exist.  Do you have any suggestions?


Thank you.

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi Damien - there was no handling for sub categories, but I have since added this support, so you will need to download an update from https://github.com/piersharding/moodle-tool_uploadcourse

Cheers,

Piers Harding.

 
Average of ratings: -
Picture of David O'Brien
Re: Bulk course upload 2.0
 

Hi

I have a dev version of Moodle 2.3 Moodle 2.3dev (Build: 20120511) and have been trying to use the uploadcourse tool from https://github.com/piersharding/moodle-tool_uploadcourse

I have tried all formats of a csv but get a stack trace of:

Stack trace:
  • line 467 of /lib/setuplib.php: moodle_exception thrown
  • line 105 of /admin/tool/uploadcourse/index.php: call to print_error()

A csv I have tried with the above error is:

fullname,shortname,category,idnumber,summary
Computer Science 101,CS101,Masters,CS101,The first thing you will ever know

Can anybody shed any light on the cause of this issue so that I will know where to look further.

Regards

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi -

 

I've just tried with your file as described and it works for me.  Where that error is being thrown is where the generic CSV file uploader is executed.  Usual causes for errors are a mismatch of the column counts (more or less headings than columns), or something to do with the delimiter you have specified.  It could also be something to do with the file encoding.

Cheers,

Piers Harding.

 
Average of ratings: -
Picture of A Barger
Re: Bulk course upload 2.0
 

Hi Piers,

I am currently using your course upload tool with much success. I have one question.  We have a sub category that actually has a "/" in the name.  When I use the tool, it is treating the "/" as a character for a sub sub category.  Is there any way to have a slash in your name without it being delimited in this way?

Category: "Science & Technology"

Sub-Category: "Computer Science/Industrial Technology"

Thanks!

 
Average of ratings: -
Picture of A Barger
Re: Bulk course upload 2.0
 

I just decided to change the source code and alter this line $categories = explode('/', $course->category);

 

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi -

The '/' as a delimiter is kind of traditional for categories, as this is used in the display of hierarchies.  If we change this then it would be counter intuitive for others.  However, if we can think of a decent way to 'escape' '/' then I'll support it.

 

Cheers,

Piers Harding.

 
Average of ratings: -
Picture of Richard Oelmann
Re: Bulk course upload 2.0
Group Particularly helpful MoodlersGroup Testers

I faced this exact problem recently and to be honest I tool a completly different solution - I didn't want to change the code, neither did I want to change the delimiter - as Piers says the / makes sense.

My solution was to change the problem course titles for the upload and then manually change them back later - I only had about 3 or 4 to edit having uploaded about 650 trial courses with Piers' tool and felt that manually editing those 3 or 4 would be less problematic than code alterations or changing the category divider for the other 640+

I realise this may not be of help in all situations, but that was my fix - and it took under 5 minutes to achieve for my small number of courses that were an issue smile

HTH

Richard

 
Average of ratings: -
Picture of Matt Gleeson
Re: Bulk course upload 2.0
 

Hi Piers,

What about // to escape a forward slash, as in regex etc?

ie. in /campus//mode/online/Diploma of Graphic Design the // would be treated as a forward slash literal

What do you think?

-Matt.

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi Matt -

I would prefer taking this a step further - the defacto standard is to escape things with a backslash so this would be:

$x = 'campus\/mode/online/Diploma of Graphic Design';  // no leading slash

array(3) {

[0]=> string(11) "campus/mode"
[1]=> string(6) "online"
[2]=> string(25) "Diploma of Graphic Design"
}

I've put up a fix for this at: https://github.com/piersharding/moodle-tool_uploadcourse

Cheers,

Piers Harding.

 

 
Average of ratings: -
Picture of Matt Gleeson
Re: Bulk course upload 2.0
 

Oops, sorry, I don't know where my head was at - of course it's a backslash to escape, not a forward slash :P

Cool, thanks for the fix Piers, that's awesome! smile

-Matt.

 
Average of ratings: -
I Teach With Moodle
Re: Bulk course upload 2.0
Group Testers

Hi, 

I've not tested this but I'm pretty sure it would work fine (it's not pretty though). How about you swap your '/' with another obscure character in your CSV file, a character that you're pretty sure will never get used in any of your category names, and then upload the CSV file using the bulk upload tool. 

Once the data is in your database, you could try and run a UPDATE statement replacing that character with '/'   This is not a real solution but should help anyone with the same problem, especially if they have a lot of subcategories.

Cheers,

Fred 

 
Average of ratings: -
Picture of David O'Brien
Re: Bulk course upload 2.0
 

Hi Piers,

Thank you for the reply. My main problem was encoding. For anybody who needs help with that I found some good instructions:

(1) Open an Excel file where you have the info (.xls, .xlsx)
(2) In Excel, choose "CSV (Comma Delimited) (*.csv) as the file type and save as that type.
(3) In NOTEPAD (found under "Programs" and then Accessories in Start menu), open the saved .csv file in Notepad
(4) Then choose -> Save As..and at the bottom of the "save as" box, there is a select box labelled as "Encoding". Select UTF-8 (do NOT use ANSI or you lose all accents etc). After selecting UTF-8, then save the file to a slightly different file name from the original.

This file is in UTF-8 and retains all characters and accents and can be imported, for example, into MySQL and other database programs.

Otherwise a fantastic tool. Thank you for the work.

Regards

 
Average of ratings: -
Picture of Susan Mangan
Re: Bulk course upload 2.0
Group Particularly helpful Moodlers

Hi there,

I'm running v2.3

I also got this same error.  I followed your instructions and saved the .csv as UTF-8 but it still wouldn't go through sad 

I ended up just removing the summary column which contained the offending code.  I just wanted to post in case someone else has the same problem/error that isn't resolved.  You might have to search for the offending character(s) before your file will upload. 

 
Average of ratings: -
Picture of Noveck Gowandan
Re: Bulk course upload 2.0
 

Good day,

I'm just wondering, does this script also create subcategories if it does not exist?

For example, if the csv file contains a nested category

eg.

ParentCat/SubCat1/SubCat2

Would it create the subcategories if the names are int he CSV?

Much thanks for your help.

-noveck

 

 
Average of ratings: -
Picture of Paul Fynn
Re: Bulk course upload 2.0
 

Hi Piers,

Thanks for the tracker response.

I'm very appreciative of your work on the bulk upload, and find it bizarre that we are still awaiting a core Moodle solution.

Currently I'll be attempting to replicate just under 300 courses using your upload program. We saved a lot of time last semester with your previous version, but the copy over facility is very important to avoid the need for users to back up and restore.

Unfortunately all our users make their own choices of format and topic numbers, so ideally it would be better for us if there was an automatic pick up of format and weeks/topic number. (Alternatively would setting the maximum number of sections in th eupload file work ?)

Two difficulties I'm having at the moment

1) finding a comprehensive list of what the permissible content is for each field (format field, for example, seems to be a bit picky)

2) Currently the uploads seem to be loading as SCORM format, irrespective of the format settings .

Thanks for any tips you can offer,

best

Paul

 

 

 
Average of ratings: -
Picture of Paul Fynn
Re: Bulk course upload 2.0
 

As a quick update,whilst I can create empty courses with the right number of topics, any attempt to import content during, or seperately after the process, seems to result in the copied content appearing twice in topic 0 in the new or updated course.

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi -

If you have things like 'News Forum' automatically added by default on the creation of a new course, then this will give the appearance of duplication when you import a backup file (or copy from template course Id).

Perhaps look at your course format defaults and blocks?

Cheers,

Piers Harding.

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi Paul -

I've been testing the 'format' issue, and it appears to work fine for me, if you put the format in the CSV file or setting the default via the screen options after the initial load of the file.

The options for the format value are 'scorm', 'social', weeks', and 'topics'.  As a general rule, the input values for fields are what you find on the data entry form if you inspect the HTML element.

I've also checked - and as you say, the value set in the file, or as defaults is the format that the course is created with irrespective of what it would have been in the course backup that is loaded over the top.

There does not appear to be any obvious way of passing parameters to the restore controller to get it to pull these details from the backup file.  The only way it appears to work is when you run the restore controller in backup::TARGET_NEW_COURSE.  This adds a restore step that pulls all of the main course settings - which would not be ideal either.

We need another solution...

 

Cheers,

Piers Harding.

 

 
Average of ratings: -
Picture of Paul Fynn
Re: Bulk course upload 2.0
 

Hi Piers,

Many thanks - on reflection I'm wondering whether the duplication issue (it is content as well as news) occured after I used the 'overwrite...with defaults' option.

I'll do some more checking.

I'm hoping that I can install Tim Hunts SQL plugin, and that might give a way of 'querying out' the data on format type and number nof sections.

I'll post back here on progress on the main issues,

Many thanks again

Paul

 
Average of ratings: -
Picture of Paul Fynn
Re: Bulk course upload 2.0
 

Hi again,

having switched on the debugging, I'm now seeing this and wondering whether this implicates the upload file, template file or config.

Upload File contents:

fullname  shortname  idnumber summary format enrolmethod_1 role_1  startdate  numsections  maxbytes  lang  templatename
2012.2 HFG122  2012.2 HFG122 2012.1 HFG122 a topics manual student 23.07.2012 20 20971520 en 2012.9 HFG122
2012.2 PG610 Marketing 2012.2 PG610 2012.2 PG610 a topics manual student 23.07.2012 20 20971520 en 2012.9 610PG
2012.2 708 Independent  2012.2 708 2012.2 708 a topics manual student 23.07.2012 20 20971520 en 2012.9 708

Error report:

Upload courses results

Error: mdb->get_record() found more than one record!
  • line 1309 of /lib/dml/moodle_database.php: call to debugging()
  • line 1372 of /lib/dml/moodle_database.php: call to moodle_database->get_record_sql()
  • line 151 of /backup/moodle2/restore_qtype_plugin.class.php: call to moodle_database->get_field_sql()
  • line 131 of /backup/util/plan/restore_structure_step.class.php: call to restore_qtype_plugin->process_question_answer()
  • line 103 of /backup/util/helper/restore_structure_parser_processor.class.php: call to restore_structure_step->process()
  • line 125 of /backup/util/xml/parser/processors/grouped_parser_processor.class.php: call to restore_structure_parser_processor->dispatch_chunk()
  • line 91 of /backup/util/helper/restore_structure_parser_processor.class.php: call to grouped_parser_processor->postprocess_chunk()
  • line 148 of /backup/util/xml/parser/processors/simplified_parser_processor.class.php: call to restore_structure_parser_processor->postprocess_chunk()
  • line 92 of /backup/util/xml/parser/processors/progressive_parser_processor.class.php: call to simplified_parser_processor->process_chunk()
  • line 169 of /backup/util/xml/parser/progressive_parser.class.php: call to progressive_parser_processor->receive_chunk()
  • line 253 of /backup/util/xml/parser/progressive_parser.class.php: call to progressive_parser->publish()
  • line ? of unknownfile: call to progressive_parser->end_tag()
  • line 158 of /backup/util/xml/parser/progressive_parser.class.php: call to xml_parse()
  • line 137 of /backup/util/xml/parser/progressive_parser.class.php: call to progressive_parser->parse()
  • line 105 of /backup/util/plan/restore_structure_step.class.php: call to progressive_parser->process()
  • line 153 of /backup/util/plan/base_task.class.php: call to restore_structure_step->execute()
  • line 148 of /backup/util/plan/base_plan.class.php: call to base_task->execute()
  • line 157 of /backup/util/plan/restore_plan.class.php: call to base_plan->execute()
  • line 310 of /backup/controller/restore_controller.class.php: call to restore_plan->execute()
  • line 747 of /admin/tool/uploadcourse/index.php: call to restore_controller->execute_plan()
Error: mdb->get_record() found more than one record!
  • line 1309 of /lib/dml/moodle_database.php: call to debugging()
  • line 1372 of /lib/dml/moodle_database.php: call to moodle_database->get_record_sql()
  • line 151 of /backup/moodle2/restore_qtype_plugin.class.php: call to moodle_database->get_field_sql()
  • line 131 of /backup/util/plan/restore_structure_step.class.php: call to restore_qtype_plugin->process_question_answer()
  • line 103 of /backup/util/helper/restore_structure_parser_processor.class.php: call to restore_structure_step->process()
  • line 125 of /backup/util/xml/parser/processors/grouped_parser_processor.class.php: call to restore_structure_parser_processor->dispatch_chunk()
  • line 91 of /backup/util/helper/restore_structure_parser_processor.class.php: call to grouped_parser_processor->postprocess_chunk()
  • line 148 of /backup/util/xml/parser/processors/simplified_parser_processor.class.php: call to restore_structure_parser_processor->postprocess_chunk()
  • line 92 of /backup/util/xml/parser/processors/progressive_parser_processor.class.php: call to simplified_parser_processor->process_chunk()
  • line 169 of /backup/util/xml/parser/progressive_parser.class.php: call to progressive_parser_processor->receive_chunk()
  • line 253 of /backup/util/xml/parser/progressive_parser.class.php: call to progressive_parser->publish()
  • line ? of unknownfile: call to progressive_parser->end_tag()
  • line 158 of /backup/util/xml/parser/progressive_parser.class.php: call to xml_parse()
  • line 137 of /backup/util/xml/parser/progressive_parser.class.php: call to progressive_parser->parse()
  • line 105 of /backup/util/plan/restore_structure_step.class.php: call to progressive_parser->process()
  • line 153 of /backup/util/plan/base_task.class.php: call to restore_structure_step->execute()
  • line 148 of /backup/util/plan/base_plan.class.php: call to base_task->execute()
  • line 157 of /backup/util/plan/restore_plan.class.php: call to base_plan->execute()
  • line 310 of /backup/controller/restore_controller.class.php: call to restore_plan->execute()
  • line 747 of /admin/tool/uploadcourse/index.php: call to restore_controller->execute_plan()
  • StatusCSV lineIDFull nameShort nameCategoryID numberSummaryDelete
    New course 2 735 2012.2 HFG122 2012.2 HFG122   2012.1 HFG122 a  
    New course 3 736 2012.2 PG610 Marketing 2012.2 PG610   2012.2 PG610 a  
    New course 4 737 2012.2 708 Independent 2012.2 708   2012.2 708 a  

    Courses created: 3
    Errors: 0

 

 
Average of ratings: -
Picture of Paul Fynn
Re: Bulk course upload 2.0
 

Hi Piers,

We uploaded to 2.2.3 and the May 16th git version of the upload which on initial inspection appears to be working fine.

Will post a progress report here.

Many thanks for all your help

Paul

 
Average of ratings: -
Picture of Paul Fynn
Re: Bulk course upload 2.0
 

Update: The content is now posting appropriately to the required sections, but continues to duplicate content.

This looks like the pointer as to where the issue may be

Error: mdb->get_record() found more than one record!

  • line 1309 of /lib/dml/moodle_database.php: call to debugging()
 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Having a look at the code, this appears to be an issue with the Question Type restore function - it seems to be reporting that it is getting duplicate values for answers (if I'm looking at the same Moodle version as you).  Are you by any chance, restoring the same file twice - you can specify the backup multiple times -  templatename (course shortname), the backup file uploaded in the gui, and a specific backup file in the CSV file.  If there is overlap with these then they will clash - the uploader works through all of these options sequentially for each course, so in theory it is possible to restore 3 different (or the same) courses over a course in the same run.

Cheers,

Piers Harding.

 
Average of ratings: -
Picture of Paul Fynn
Re: Bulk course upload 2.0
 

Hi Piers, as the sun goes down in Switzerland, it illuminates, I presume, NZ ?? Good to catch you on the crossover.

Good theory, I had been considering whether the backups had been an issue, or whether there was some issue with repeated restoring.

However, I tried to counter this by creating a template file with just URLs for content, and these are still provoking the same symptoms and message. (ie there are no quizzes, but the links are still duplicated).

Could the upload course be tripping up over itself ie finding its own backup prior to restoring as a new course ? Does the uploader run through every course i sequence, or complete one course at a time ?

best

Paul

 
Average of ratings: -
Picture of Paul Fynn
Re: Bulk course upload 2.0
 

Just tried single upload, no quizzes, unfortunately it's still doubling.

 
Average of ratings: -
Picture of Peter Stacey
Re: Bulk course upload 2.0
 

Hi

I've been testing this and have found that the script is consistently doubling all course content when I use a template course, but not when I use a backup file.

It appears to me that the restore is happening twice in the case of using a template course.

I have managed to fix this by commenting out lines 724 - 753 of index.php

It doesn't appear (so far) to have any detrimental effects.

Thanks for your work on this Piers, it's something that I've been missing since I've started using Moodle. I have developed a (non APi) script for making a course copy directly on the database, but obviously using the backup and restore functionality should be a more future-proof approach.

 

Cheers

Peter Stacey

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi Peter -

This is 'my bad' - there is a bug where one restore is handing the same file onto the next.

I've pushed a fix up to https://github.com/piersharding/moodle-tool_uploadcourse - could you test it again by any chance and let me know?

Thanks,

Piers Harding.

 
Average of ratings: -
Picture of Peter Stacey
Re: Bulk course upload 2.0
 

Hi Piers,

Thanks for that, I've tested it out and it's working fine now.

Cheers

Peter

 
Average of ratings: -
Picture of Paul Fynn
Re: Bulk course upload 2.0
 

Hi Piers,

Just to confirm the same as Peter.

Appears to be working fine on 2.2.3, on our test site, but not on 2.2.2 on our main site, so we are upgrading later today.

I'm interested to see how this handles Turnitin Assignment types in the past, as the API has a history of being troublesome in backup and restore.

Interesting that a copy of the course backup file is also created in the new course at the time of upload.

Many thanks for all your work, and particularly the very quick turnaround that you have generously provided on the bug fixes.

best

Paul

 
Average of ratings: -
Picture of Paul Fynn
Re: Bulk course upload 2.0
 

Hi Piers,

We've upgraded our main site to 2.2.3 and here's a rundown of the results.

The first record uploads and creates a new course as required, but the upload then fails and does not load the second course. With full debugging on I get the following error message:

Upload courses results

Notice: Undefined index: enrolmethod in /home/httpd/vhosts/web1/web/admin/tool/uploadcourse/index.php on line 847

Notice: Undefined index: enrolmethod in /home/httpd/vhosts/web1/web/admin/tool/uploadcourse/index.php on line 847

Notice: Undefined index: enrolmethod in /home/httpd/vhosts/web1/web/admin/tool/uploadcourse/index.php on line 854

Notice: Undefined index: in /home/httpd/vhosts/web1/web/admin/tool/uploadcourse/index.php on line 854

Fatal error: Call to a member function add_default_instance() on a non-object in /home/httpd/vhosts/web1/web/admin/tool/uploadcourse/index.php on line 855

Turnitin Handling

As I anticipated, the upload programme replicated all the course content including the old Turnitin assignments, which effectively links the new course submission back to the old template submission dates and settings. I don't know whether it would bê feasible to have an option field in the upload file either to ignore Turnitin, or to convert it to a placeholder.

""Notice
Duplicate copies of the same Turnitin assignments were found, duplicates occur when one or more Moodle Direct assignment is connected to the same one Turnitin assignment via the Turnitin API. This is known to cause issues, most notably submissions made to one of the duplicates will show up in the submission inbox of the others. To fix this you should either delete the duplicate assignments or reset the course where the duplicate assignments are.

The following duplicates were found:""


Overall your plugin is looking like a really important contribution to moodle admin.

Thanks again

Paul

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi Paul - can you send me the firs few rows of your file so I can test the particular scenario - you can send it directly to me at piers

at ompka dot net.

Thanks,

Piers Harding.

 
Average of ratings: -
Picture of John St
Re: Bulk course upload 2.0
Group Developers

I'm trying to use 1.9 backup files with the tool and seems to fail. Well actually it just says it can't find the file. Is it looking for an mbz, doesn't see one, then quits? 

When I remove the mbz check, it fails with:

error/cannot_precheck_wrong_status

Any thoughts on how this can be made to work with 1.9 backups? Thanks for a great tool!

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi -  It has not been created to work with 1.9 backups, as there is an in built assumption that the backups are 2.0+.  Having said that, it probably is't that hard to make it work with 1.9 files - I just haven't tried.

There is a convert method on the restore controller:

// check if the format conversion must happen first
if ($rc->get_status() == backup::STATUS_REQUIRE_CONV) {
$rc->convert();
}

Cheers,

Piers Harding.

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi -

 

I've pushed a change up for the backup format conversion 1 -> 2.  I would be interested to hear back from anyone who can test it.

 

Cheers,

Piers Harding.

 
Average of ratings: -
Picture of John St
Re: Bulk course upload 2.0
Group Developers

It still checks for an mbz file and errors out (if you specify a backup with csv). Once I removed that, it seems to work! The only thing I noticed is that even if I specify to use Course Files, it does not appear. Not sure why that might be yet. 

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi - thanks for testing.  I've widdened the file extension test to include mbz and zip, and pushed the fix up to https://github.com/piersharding/moodle-tool_uploadcourse.

 

With the course files option - are you sure that they are in the backup file?  Can you test that independently with the restore dialog?

Cheers,

Piers Harding.

 
Average of ratings: -
Picture of John St
Re: Bulk course upload 2.0
Group Developers

Thanks! That is working great. Is there any way to get the shortname and course name from the backup file instead of specifying it in the csv file? 

The legacy course files still aren't created, but they are if I do a normal restore as a new course. However, if I do a normal restore into an existing course, the legeacy course files are not created... So, I'm a bit stumped by that. 

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi - thanks for the feedback.

 

Therre must be away to unpack (unzip) the backup files and target the moodle.xml file inside.  If you parse this out you can get all the basic course details.

I don't have an example of doing this, but I don't think it would be that difficult to do.

Cheers,

Piers Harding.

 
Average of ratings: -
Picture of Pat OKeefe
Re: Bulk course upload 2.0
 

Hi,

I am having trouble using the course upload feature. Specifically I do not seem to be able to get the scipt to allow assigning of a valid user to the course context teacher role.  Here is my .csv upload file:

fullname,shortname,category,summary,idnumber,teacher

CSE(317-001),CSCE317-001,Computer Science & Engineering/Summer 2012,Computer Systems Engineering,SM2012CSCE317001,<somevaliduserID>

I have tried both "teacher" and "editingteacher".  No luck.  I have to manually add the role.  Any ideas?

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi -

 

The fields teacher, editingteacher, student, manager are text descriptive fields only (Role names).  If you wnat to enrol people in a course then you must use the User upload tool.

Cheers,

Piers Harding.

 
Average of ratings: -
Picture of Pat OKeefe
Re: Bulk course upload 2.0
 

HI Piers,

Thanks for the reply.  I have managed a way to bulk add teacher (editingteacher) roles to specific teacher/users using the flatfile enrollment method.  It is a real step backwards if you ask me.

I use an external database to manage enrollment records we receive from our main registrar office. As you can imagine the course-student records change quite a bit.....  Manually updating the list is impossible.  We had the whole scheme wired in version 1.9.  Including faculty/teachers assigned to each course. (used bulk course upload to manage teachers) 

For some reason when I update our enrollment database and add either "editingteacher" or "teacher" to the role field,  moodle 2.2.3 defaults the role to student.  I have worked this problem for days and finally gave up when I  found the flatfile workaround.

For the record, remote enrolment sync fileds are correct and match the entries for our working 1.9 site:

Default role: enrol_database | defaultrole -> Student

Remote role field: enrol_database | remoterolefield -> role

 

Perhaps my table element type for the role is not making moodle happy?

------------------

field, type, null

role, varchr(15), YES

-----------------

Here is the table entry:

CSCE215-851|pokeefe|editingteacher

When I sync up the databases:

# /usr/local/bin/php -c /etc/php.ini /info/docs/www/moodle2/enrol/database/cli/sync.php -v
Course synchronisation skipped.
Starting user enrolment synchronisation...
  enrolling: 3 ==> CSCE215-001 as student
...user enrolment synchronisation finished.

note the enrolled as a student output.

Just wanted to thank you for your time and see if I missed something.

Cheers,

Pat O'Keefe

 
Average of ratings: -
Picture of John St
Re: Bulk course upload 2.0
Group Developers

I still can't get it to make the legacy course files. I don't think this is an issue with your plugin specifically, but maybe a more global moodle one. When it makes the courses, regardless of what I set globally (or in your plugin) the course settings have legacy course files set to 'no' upon creation. 

 
Average of ratings: -
Picture of Richard Oelmann
Re: Bulk course upload 2.0
Group Particularly helpful MoodlersGroup Testers

Hi Piers,

I've used your tool succesfully in the past to create a number of blank courses, but I'm now trying to use it to upload a series of course backups - as new courses. These are 1.9 course backups into Moodle 2.2.3+ (Build: 20120612) and using the version of your tool from the github link as above. I'm running into a problem that when I try to upload the courses I get an error message in the status column of every course saying Template file not found - except I am not using any template files, either in the csv or in the options on the tool's page. Do I need to be? Am I doing something wrong?

Below is the first few lines of my csv file and I've added a screenshot of the error in the hope someone can help point out what is probably a silly error on my part smile

 

fullname,shortname,summary,category,backupfile
Applied Mathematics 1 ,AP1,Applied Mathematics 1 ,Engineering_and_Science_Pathway,/home/richard/Desktop/olncoursebackups/backup-ap1-20120607-0404.zip
Applied Mathematics 2 ,AP2,Applied Mathematics 2 ,Engineering_and_Science_Pathway,/home/richard/Desktop/olncoursebackups/backup-ap2-20120607-0406.zip
Political thought,Apt,Political thought,Unsorted,/home/richard/Desktop/olncoursebackups/backup-apt-20120607-0106.zip
Basic Skills,B_S,Basic Skills,Unsorted,NULL
Biology 1 (Cell Biology),B1,Biology 1 (Cell Biology),Engineering_and_Science_Pathway,/home/richard/Desktop/olncoursebackups/backup-b1-20120607-0408.zip

screenshot

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi -

I have just tested the backupfile parameter and it is working, so I'm wondering if it's one of the following problems:

* it's client server so you are referencing files on your client, but the server needs to have direct file system access to them

* file names are slightly incorrect - case sensitive

* permissions are not set on the files so that the web server user can actually read them, even though the names etc. are correct.

Cheers,

Piers Harding.

 
Average of ratings: -
Picture of Richard Oelmann
Re: Bulk course upload 2.0
Group Particularly helpful MoodlersGroup Testers

Hi Piers

Many thanks for your thoughts - I didn't consider problems with the backup files as the error message was for Template thoughtful but I will check your suggestions when I get to work on Monday.

At the moment the testing I'm doing is all on a localhost with the moodle in /var/www and the folder of backup files on my desktop - I don't think, then that it will be a client/server issue and I thought I had changed permissions on the folder, but it would make sense that being in /home/richard/Desktop the permissions are not correct. As I said, I'll try on Monday and see what happens

I think the file names are all correct as I generated them from a file listing into a text file and copied that into the csv file, so I don't think there'll be any case errors (but I will check smile )

- I know the plugin works as others have had success with it so it's got to be an issue my end. So again Piers, thanks for a great tool and thanks for your helpful suggestions smile

Richard

 
Average of ratings: -
Picture of Paul Fynn
Re: Bulk course upload 2.0
 

Hi Piers, we're approaching that time again for uploading new courses for the next semester, greatly eased by your tool (we do this three weeks before the end of the previous semester).

We'll be installing teh latest version from github for testing shortly.

Something that has been discussed previously on Turnitin handling :

the upload programme replicates all the course content including the old Turnitin assignments, which effectively links the new course submission back to the old template submission dates and settings.

Would it bê feasible to have an option field in the upload file either to ignore Turnitin, or to convert it to a placeholder.


Many thanks for any thoughts you have,

best

Paul

 

 
Average of ratings: -
Picture of Jamie Tinley
Re: Bulk course upload 2.0
 

Thank you Paul for posting your example!  I had issues with startdate which kept defaulting to todays date which it says it would at the main download site here.  Your example helped me realize it is in the form dd.mm.yyyy like 23.01.2013     If anyone else reads this and is updating other courses then make sure you choose from the dropdown to update existing courses only AND vip next dropdown choose override with file.

Thanks!  

 
Average of ratings: -
Picture of Moshe Golden
Re: Bulk course upload 2.0
 

Hello Piers,

I installed your course upload module and it works beautifully.  I was wondering if it would be possible to create a version which would work the same way, but with a "fixed name and path" input file and  be launched via a cron script.

 

Moshe

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi - Yes, this would be possible, but it is a big job, and will have to wait for later.

Cheers,

Piers Harding.

 
Average of ratings: -
Picture of Nick Gault
Re: Bulk course upload 2.0
 

Hi,

First of all, thanks to Piers for this tool - it is going to make life easier.

I am currently moving old (1.9) courses on to a new install(2.3) using this tool and course backups.

I am getting a few problems though and am wondering if anyone has any suggestions.

Issue 1: If I include any categories in the csv the course creation fails.

Issue 2: If I create/restore from an automated backup file the course files do not appear in the new course, but if I manually back the course up and then run the tool the course is restored in full. The automated backup files contain all of the documents required but something is stopping them from appearing in the new course. -   It looks like the automated backup files don't include resources despite the presence of the files. 

Thanks in advance.

Nick

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi Nick -

Have you created the course categories in advance? uploadcourse does not automatically create them.  If you have a lot to setup, then you may wish to look at https://github.com/piersharding/moodle-tool_uploadcoursecategory to help with this?

Cheers,

Piers Harding.

 
Average of ratings: -
Picture of Aleks Macheta
Re: Bulk course upload 2.0
 

Hi Piers,

Many thanks for developing this really useful tool.

I am using it to test uploading courses to 2.2.4 and have noticed that the 'summary' field seems to be a required field in the csv i have created.  If i omit this field in my csv i get a warning "summary is missing". 

I don't really need to populate this field - Is it easy to make the field optional in the code?  If so, please let me know where to look to do so.

with many thanks,
Aleks

 
Average of ratings: -
Picture of Michael Hughes
Re: Bulk course upload 2.0
 

small workaround to this is just to use a non-breaking space in summary field &nbsp; 

...,summary,...

...,&nbsp;,...

 
Average of ratings: -
Just wondering . . .
Re: Bulk course upload 2.0
Group Particularly helpful Moodlers

In some random search I found this today: https://tracker.moodle.org/browse/MDL-38501

"Create a "Bulk course action" menu, allowing several operations (backup, delete, reset, change settings...)"

11 votes in under a month.  This looks like a promising goal.  But there is also this rather old item from Anthony: MDL-17017

"Admin: Evaluate file structure for administrative functions for consistency"

Now I understand why this seemingly simple item (ie bulk course create) is so complex: the database structure is complex, and all the options in setting up a course are difficult.

Clone buttons may to someway towards this.  (it's great now to be able to duplicate activities) but at the course level there are names, shortnames etc to consider and feature creep kicks in immediately.

I still miss, every day, bulk user actions for these areas

  1. users
  2. courses
  3. activities

-Derek

 
Average of ratings: -
Picture of R.T. Brown
Re: Bulk course upload 2.0
 

Hi Piers, Thanks for the great tool. I've been testing and noticed that when a course is restored it has several blocks that the templated did not. Do you know why that might be?

Thanks --

R.T. Brown

 
Average of ratings: -
Just wondering . . .
Re: Bulk course upload 2.0
Group Particularly helpful Moodlers

R.T., Just in case you are interesed: see https://tracker.moodle.org/browse/MDL-13114

You ma like to chip in there.

-Derek

 
Average of ratings: -
Picture of Jeff Hague
Re: Bulk course upload 2.0
 

Hello all! I am struggling with this and need some help!

I have downloaded master.gz and unzipped it using both Winzip and 7zip. In both cases it extracts a single file called master with no extension.

The instructions say:

"

Download https://gitorious.org/moodle-tool_uploadcourse/moodle-tool_uploadcourse/archive-tarball/master
unpack the file (probably called master) with tar -xzvf master
This will give you a directory called moodle-tool_uploadcourse-moodle-tool_uploadcourse
Move this directory and rename it into its final position:
mv moodle-tool_uploadcourse-moodle-tool_uploadcourse <Moodle dirroot>/admin/tool/uploadcourse"

 

but I dont get a directory and cant seem to figure out how to extract it as such. What am I doing wrong?

Thanks!

 
Average of ratings: -
Picture of Jeff Hague
Re: Bulk course upload 2.0
 

Sorry - I figured it out... Apparently, unzipping a tarball on Windows using 7zip is a 2-step process.

Thanks!

 
Average of ratings: -
Rosario playing soccer
Re: Bulk course upload 2.0
Group Particularly helpful Moodlers

Please see also this one:

http://moodle.org/mod/forum/discuss.php?d=128317

I developed it for 1.9.x to import courses and users at the same time. Courses are created first and in a second pass through the cvs file users/teachers/students are enrolled in their respective roles.

Unfortunately categories have to be created first in the GUI. I still have to add this feature to my code, to create the category if it does not exist. And my code is not at click and go stage yet. But you can tailor it to fit your exact needs because I am used to make comments in every line of code to facilitate reading.

Rosario

 
Average of ratings: -
Picture of Tim Geiger
Re: Bulk course upload 2.0
 

The script works perfect for me. However I would like to some how automate the process. Is there any way to point this script towards a specific .csv file on the server without having to manually log in and select a file. I can then schedule the server to run the script daily. Any help would be appreciated.

Thanks

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi - this could be done, but would be quite a significant amount of work, as the entire index.php, and form code would need restructuring.

Command line parameters would need to e developed for all of the defaulting options in the leading forms, and then the csv, and message/error handling would need remodelling to deal with both interactive, and command line use.

 

It's a great idea, but one that will take a lot of time which I don't have at the moment.

One for later I think.

Cheers,

Piers Harding.

 
Average of ratings: -
Picture of chhavi garg
Bulk course upload 2.0
 

I need to upload coures as well as categories from same excel file .Can some one tell me how to do it??

 
Average of ratings: -
Me!
Re: Bulk course upload 2.0
Group Developers

Hi - it is more likely to work if you split out the columns for categories into a separate file, and try using https://github.com/piersharding/moodle-tool_uploadcoursecategory

Then you can upload the courses using https://github.com/piersharding/moodle-tool_uploadcourse

Cheers,

Piers Harding.

 
Average of ratings: -
Picture of chhavi garg
Re: Bulk course upload 2.0
 

hi,

I just have 1 ecxcel sheet containing details of categories and courses.Can you please tell me how to divide that excel sheet by php program.

 
Average of ratings: -
C'est moi :-)
Re: Bulk course upload 2.0
Group Documentation writersGroup Particularly helpful MoodlersGroup TestersGroup Translators

Piers : I've tested your tools to automatically create courses (or course categories), and they are useful !

In README, you show which fields can be specified, but it would be more useful to have the list of required fields opposed to optional fields (like in Upload user documentation).
Moreover, it seems several fields are required by your tools (like idnumber), but are not (required) in Moodle. I'm not sure it's really a good idea. Would better let only Moodle required fields to be required...

Séverin

 
Average of ratings: -
C'est moi :-)
Re: Bulk course upload 2.0
Group Documentation writersGroup Particularly helpful MoodlersGroup TestersGroup Translators

It seems i've seen a bug in your tool for upload courses, with Moodle 2.4.3 (not tested with other versions).

Everything is OK while creating courses via the web interface, using fields defined in the text file. It's also OK if i use a backup file, specying it's name via the web interface (and not in text file).

But if i select a course to use as template (not specifying it in the text file), activities are restored twice (one time the several activities, and then another time) in the generated courses mixed

Problem is the same for activities in section 0 or in other sections.

 
Average of ratings: -
Picture of ahmad hamwi
Re: Bulk course upload 2.0
 

The tool is amazing,

but we're having a slight problem.

All the material is posted in topic 0

Anyone knows the reason?

thank you

 
Average of ratings: -
Picture of ahmad hamwi
Re: Bulk course upload 2.0
 

In addition any luck with the legacy files showing?
And last thing, is there a way to auto generate the CSV file from moodle 1.9? So we don't populate the CSV file manually which can take a long time?

 

thank you

 
Average of ratings: -