Bulk course upload 2.0

Bulk course upload 2.0

by Mark Johnson -
Number of replies: 95
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I've just had a bash at updating the Bulk Course Creation tool (uploadcourse.php) for Moodle 2. I've got it working, and it should work with databases other than MySQL (it uses $DB now).

It's not very tidy,  and it's not packaged as a plugin, just drop it into /admin as before.  To be honest, I've done just enough to get it working for me and commented out the features I don't need and therefore haven't upgraded (teachers, templates and topics), so any contributions and improvements are welcome.  It's up on github: https://github.com/marxjohnson/moodle_admin-uploadcourse.php

Average of ratings: Useful (3)
In reply to Mark Johnson

Re: Bulk course upload 2.0

by saif nat -

thanks for you i will testing it

and i will give you my notes

 

thanks

 

saif

In reply to Mark Johnson

Re: Bulk course upload 2.0

by Mark Johnson -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

There was a bug in the default_category function, fix is now on github.

Just a quick note about performance: The old version of the script used native mysql functions to speed up processing (the comments at the top mention cutting down from ~30 minutes via the old ADODB API to ~1 minute.  All queires (that affect the way I use it) now use Moodle 2's DB API - I just tested a file creating about 650 courses and it processed in about 40 seconds.

In reply to Mark Johnson

Re: Bulk course upload 2.0

by Jason Shelvy -

Hello, I have a question that maybe some one can help with. When I run the uploadcourse script I get the following error:

An error occurred creating category with name BIO, meaning a total 1 category(ies) failed

not sure what is creating this problem. This happen if a category is already created or not.

any help would be appreciated

In reply to Jason Shelvy

Re: Bulk course upload 2.0

by Mark Johnson -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Hi Jason,

I've implemented creation of categories. If the category does already exist, you currently need to use the ID, not the name.

Hope that helps,

Mark

In reply to Mark Johnson

Re: Bulk course upload 2.0

by steve brydges -

Hi Mark,

   I'm trying this for the first time on a brand new Moodle 2.1. The csv file looks like this:

category,format,fullname,idnumber,numsections,shortname,showgrades,showreports,sortorder,visible
Medical School,topics,MBCHB,a101,10,MBChB,0,0,1,1
Maths,topics,Integers,ma231,10,Integers,0,0,1,1
English,topics,Shakespeare,en231,10,Shakespeare,0,0,1,1

and it is failing like this:

Parsed 3 course(s) from CSV
An error occured creating category with name Medical School, meaning a total 1 category(ies) failed
Course with ShortName MBChB could not be created because parent category(ies) failed
An error occured creating category with name Maths, meaning a total 1 category(ies) failed
Course with ShortName Integers could not be created because parent category(ies) failed
An error occured creating category with name English, meaning a total 1 category(ies) failed
Course with ShortName Shakespeare could not be created because parent category(ies) failed
Created 0 course(s) out of 3
3 course(s) failed due to errors
3 new category(ies) failed due to errors
 
Do you have an example of a correctly formatted csv file?
 
In reply to steve brydges

Re: Bulk course upload 2.0

by Mark Johnson -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Hi Steve,

With the current version of the script, the category must contain the ID, not the name, of the category the course should be under.  I think I minced my words above - as I recall (it was a while ago now) I haven't implemented creation of categories that don't already exist.  You need to create them first, then upload your file using the category IDs.

In reply to Mark Johnson

Re: Bulk course upload 2.0

by Steve Anatai -

Hi Mark.

Thanks for writing this script - it is a much needed feature.

I am in the process of trying this out on our testserver at school.  Will let you know how it goes.

I recently wrote a similar script that bulk-creates categories (we make heavy use of SIMS class synchronisation and the course importer allows courses to be assigned to categories - the only snag is, they have to already exist!).  I didn't see an obvious way of just creating categories with your script, but please let me know if I've missed something.  I'm happy to make my code available on request.

All the best,

Steve

In reply to Steve Anatai

Re: Bulk course upload 2.0

by Paul Fynn -

Looking forwards to hearing how this goes....... 180 courses to create sad

BTW - anyone know who chose the acid tablet as the logo for the smileys ?

In reply to Steve Anatai

Re: Bulk course upload 2.0

by Derek Chirnside -

This request is the top-equal all time issue on the tracker with 88 votes, dating back to January 2008.  The top one, ical import has been written and is looking good)  If someone votes for this, then it will be the top.

Top Voted issues on Tracker

Here is the course create issue is you want to vote: http://tracker.moodle.org/browse/MDL-13114

-Derek

In reply to Derek Chirnside

Re: Bulk course upload 2.0

by Paul Fynn -

125 courses created by hand.....another 60 to go sad

Can anybody report back on using Marks script with 2.0 or, preferably, 2.1?

Many thanks

Paul

In reply to Paul Fynn

Re: Bulk course upload 2.0

by Bruno Morais -

Just used the script on a new 2.2 install using MySQL and php 5.3.8, worked swiftly enough to save me an afternoon of manual labour. Many thanks, Mark!

 Slight issue, though, not sure if it's fixable or not:

- Being a PT school, several course names had special characters. These got "ignored" during upload. I tried to code the csv file in UTF-8 and in every alternative to ANSI. To no avail.

 As a script/module it would be much more powerful without this handicap.

Still - awesome work

BM 

In reply to Bruno Morais

Re: Bulk course upload 2.0

by Mark Johnson -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Hi Bruno,

What sort of special characters are you have trouble with? Accented letters etc?

I'm not sure why any of that would cause a problem, but I very rarely work with anything other than basic latin characters and punctuation so I'm far from an expert on the matter, I'm afraid.

In reply to Mark Johnson

Re: Bulk course upload 2.0

by Bruno Morais -

Hi

 Yes, the problem was with accented letters, tilde and comma on c... the following characters, give or take:

á, à, é, í, ó, ã, ç 

As I said, your code didn't accept files coded in any other form than ASCI. If you (or any one) could turn it into UTF-8 friendly, it would suit the international community. Between Portugal, Spain, Romania, Turkey, etc.. I can think of so many countries with special characters needs :D

 Cheers and happy new year!

Average of ratings: Useful (1)
In reply to Bruno Morais

Re: Bulk course upload 2.0

by Graham Paull -

Bruno,

How did you get this working with MySQL ?

My CSV seems to fail at teacher1_account with the following error:

string(5) "Admin"

get_record() not available anymore

I have attached my CSV if you don't mind having a look.

Thanks,

In reply to Derek Chirnside

Re: Bulk course upload 2.0

by Piers Harding -

For those that maybe interested, an admin tool plugin has been developed for 2.2+ that does bulk course upload operations in a similar vein to user upload.

This enables create/update/delete of courses, and the ability to nominate either a Moodle course backup, or existing course as a template.

https://github.com/piersharding/moodle-tool_uploadcourse

Cheers,

Piers Harding.

Average of ratings: Useful (2)
In reply to Piers Harding

Re: Bulk course upload 2.0

by Sam Stegers -

Piers,

I am very much looking forward to give your tool a go. Somehow I have not been able to find a .csv template which I could use. Could you supply a .csv file where all available columns are present, so I know what to put where.

TIA,

 

Sam

In reply to Sam Stegers

Re: Bulk course upload 2.0

by Piers Harding -

Hi Sam -

Please remember that you must be on 2.2 and above.  I have extended the documentation at https://github.com/piersharding/moodle-tool_uploadcourse with column names and a trvial example.

Cheers,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by Erik Nauman -

Piers,
Thank you for all your work on this! This is a very nice tool. Have you looked at integrating teacher enrollment along with the courseupload? The tool for pre-2.2 had that capability and now we are having to do a lot of data manipulation to get enrollments sorted by teacher for the user upload tool rather than simply enrolling the teachers with the course upload. Fortunately we are only dealing with 250 courses. I believe the enrollment module has changed significantly even between 2.2 and 2.3 so maybe it's not possible but it would really help for next time.
Thanks,
Erik

In reply to Erik Nauman

Re: Bulk course upload 2.0

by Piers Harding -

Hi Erik - I think it would be a case of replicating too much functionality that already exists in the CSV user upload tool that is part of Moodle core - I understand that it is a pain having to juggle both tools, but I would prefer not to do that.

 

Cheers,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by Richard Heath -

Hi Piers,

I'm experimenting with your tool at the moment, as we have a large number of courses to set up (>1000), some with templates, others default. I'll probably do it in batches, so as not to overwhelm the system.

Could you please explain what the "teacher" and "teachers" fields are for, and what the format of them should be? I'm imagining that the "teacher" field is the role name, and the "teachers" is the users - should this be the database ID, login id, fullname...?

Many thanks in advance, and thank you for *hopefully* making my life so much easier!

Richard

In reply to Richard Heath

Re: Bulk course upload 2.0

by Piers Harding -

Hi Richard -

This was not working correctly, and I have revamped the code around it so you will need to get an updated version from https://github.com/piersharding/moodle-tool_uploadcourse

I have relabelled the field names to:

'teacher', 'editingteacher', 'student', 'manager', 'coursecreator', 'guest', 'user'

Which are the role name substitutions that you can do when you go and edit the course details eg: for manager you might want 'purse string holder' - or some such.

Please test on a small group of courses first - the effects you could possibly get from uploading course backups over the top of an existing course, or doing a template copy can vary wildly depending on how you setup those templates, or make those backups.  Make sure you backup before you go.

Let me know how you get on.

Cheers,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by Richard Heath -

Thanks Piers, I understand now. I guess I'll have to also employ the csv enrolment plugin to get the correct teachers on the courses.

Don't worry, I'll be taking a cautious approach! I'll report back here once I have some results...

Thanks again,

Richard

In reply to Piers Harding

Re: Bulk course upload 2.0

by Flavio Martins -

Hi, Piers

I've been trying to upload a csv file that has this format:

fullname;shortname;category;summary;format;showgrades;numsections
Moodle course 1;Course 1;Courses/Discipline 4/Classroom 1;a;topics;0;6
Moodle course 2;Course 2;Courses/Discipline 4/Classroom 1;a;topics;0;6
Moodle course 3;Course 3;Courses/Discipline 4/Classroom 3;a;topics;0;6

When I upload it, moodle reads it fine at first, but when I click on the "upload courses" button, moodle shows a page with only the table headers without lsting the results. And here comes the bizarre: Later checking the course list I found that only the second line of the csv file (first course) were added in moodle.

If I try to upload the csv file again, it adds the next course cause the previous one is already recorded...

What did I do to make it fail like that? It was supposed to add all courses in one way, right?

Thanks!!

In reply to Flavio Martins

Re: Bulk course upload 2.0

by Piers Harding -

Hi Flavio  - can you try turning on Moodle debugging and see if that gives us some further information.  You are correct - it should process all your course creations sequentially, so it sounds like there is some kind of error happening after the first course is created.

 

Cheers,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by Damien Kempt -

Dear Piers,

 

I've been trying out your course upload plugin and it seems to work great.  I have one question, is it possible to assign courses to a sub-category with an upload.  I tried entering a sub-category in the category column in this format "Category Name/Sub-Category Name", but it creates the following error:


Invalid value for field category
Can not find data record in database table course_categories.


The category and sub-category definitely exist.  Do you have any suggestions?


Thank you.

In reply to Damien Kempt

Re: Bulk course upload 2.0

by Piers Harding -

Hi Damien - there was no handling for sub categories, but I have since added this support, so you will need to download an update from https://github.com/piersharding/moodle-tool_uploadcourse

Cheers,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by David O'Brien -

Hi

I have a dev version of Moodle 2.3 Moodle 2.3dev (Build: 20120511) and have been trying to use the uploadcourse tool from https://github.com/piersharding/moodle-tool_uploadcourse

I have tried all formats of a csv but get a stack trace of:

Stack trace:
  • line 467 of /lib/setuplib.php: moodle_exception thrown
  • line 105 of /admin/tool/uploadcourse/index.php: call to print_error()

A csv I have tried with the above error is:

fullname,shortname,category,idnumber,summary
Computer Science 101,CS101,Masters,CS101,The first thing you will ever know

Can anybody shed any light on the cause of this issue so that I will know where to look further.

Regards

In reply to David O'Brien

Re: Bulk course upload 2.0

by Piers Harding -

Hi -

 

I've just tried with your file as described and it works for me.  Where that error is being thrown is where the generic CSV file uploader is executed.  Usual causes for errors are a mismatch of the column counts (more or less headings than columns), or something to do with the delimiter you have specified.  It could also be something to do with the file encoding.

Cheers,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by A Barger -

Hi Piers,

I am currently using your course upload tool with much success. I have one question.  We have a sub category that actually has a "/" in the name.  When I use the tool, it is treating the "/" as a character for a sub sub category.  Is there any way to have a slash in your name without it being delimited in this way?

Category: "Science & Technology"

Sub-Category: "Computer Science/Industrial Technology"

Thanks!

In reply to A Barger

Re: Bulk course upload 2.0

by A Barger -

I just decided to change the source code and alter this line $categories = explode('/', $course->category);

 

In reply to A Barger

Re: Bulk course upload 2.0

by Piers Harding -

Hi -

The '/' as a delimiter is kind of traditional for categories, as this is used in the display of hierarchies.  If we change this then it would be counter intuitive for others.  However, if we can think of a decent way to 'escape' '/' then I'll support it.

 

Cheers,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by Richard Oelmann -
Picture of Core developers Picture of Plugin developers Picture of Testers

I faced this exact problem recently and to be honest I tool a completly different solution - I didn't want to change the code, neither did I want to change the delimiter - as Piers says the / makes sense.

My solution was to change the problem course titles for the upload and then manually change them back later - I only had about 3 or 4 to edit having uploaded about 650 trial courses with Piers' tool and felt that manually editing those 3 or 4 would be less problematic than code alterations or changing the category divider for the other 640+

I realise this may not be of help in all situations, but that was my fix - and it took under 5 minutes to achieve for my small number of courses that were an issue smile

HTH

Richard

In reply to Piers Harding

Re: Bulk course upload 2.0

by Matt Gleeson -

Hi Piers,

What about // to escape a forward slash, as in regex etc?

ie. in /campus//mode/online/Diploma of Graphic Design the // would be treated as a forward slash literal

What do you think?

-Matt.

In reply to Matt Gleeson

Re: Bulk course upload 2.0

by Piers Harding -

Hi Matt -

I would prefer taking this a step further - the defacto standard is to escape things with a backslash so this would be:

$x = 'campus\/mode/online/Diploma of Graphic Design';  // no leading slash

array(3) {

[0]=> string(11) "campus/mode"
[1]=> string(6) "online"
[2]=> string(25) "Diploma of Graphic Design"
}

I've put up a fix for this at: https://github.com/piersharding/moodle-tool_uploadcourse

Cheers,

Piers Harding.

 

In reply to Piers Harding

Re: Bulk course upload 2.0

by Matt Gleeson -

Oops, sorry, I don't know where my head was at - of course it's a backslash to escape, not a forward slash :P

Cool, thanks for the fix Piers, that's awesome! smile

-Matt.

In reply to A Barger

Re: Bulk course upload 2.0

by Frederic Nevers -

Hi, 

I've not tested this but I'm pretty sure it would work fine (it's not pretty though). How about you swap your '/' with another obscure character in your CSV file, a character that you're pretty sure will never get used in any of your category names, and then upload the CSV file using the bulk upload tool. 

Once the data is in your database, you could try and run a UPDATE statement replacing that character with '/'   This is not a real solution but should help anyone with the same problem, especially if they have a lot of subcategories.

Cheers,

Fred 

In reply to Piers Harding

Re: Bulk course upload 2.0

by David O'Brien -

Hi Piers,

Thank you for the reply. My main problem was encoding. For anybody who needs help with that I found some good instructions:

(1) Open an Excel file where you have the info (.xls, .xlsx)
(2) In Excel, choose "CSV (Comma Delimited) (*.csv) as the file type and save as that type.
(3) In NOTEPAD (found under "Programs" and then Accessories in Start menu), open the saved .csv file in Notepad
(4) Then choose -> Save As..and at the bottom of the "save as" box, there is a select box labelled as "Encoding". Select UTF-8 (do NOT use ANSI or you lose all accents etc). After selecting UTF-8, then save the file to a slightly different file name from the original.

This file is in UTF-8 and retains all characters and accents and can be imported, for example, into MySQL and other database programs.

Otherwise a fantastic tool. Thank you for the work.

Regards

In reply to David O'Brien

Re: Bulk course upload 2.0

by Susan Mangan -

Hi there,

I'm running v2.3

I also got this same error.  I followed your instructions and saved the .csv as UTF-8 but it still wouldn't go through sad 

I ended up just removing the summary column which contained the offending code.  I just wanted to post in case someone else has the same problem/error that isn't resolved.  You might have to search for the offending character(s) before your file will upload. 

In reply to Piers Harding

Re: Bulk course upload 2.0

by Noveck Gowandan -

Good day,

I'm just wondering, does this script also create subcategories if it does not exist?

For example, if the csv file contains a nested category

eg.

ParentCat/SubCat1/SubCat2

Would it create the subcategories if the names are int he CSV?

Much thanks for your help.

-noveck

 

In reply to Piers Harding

Re: Bulk course upload 2.0

by Paul Fynn -

Hi Piers,

Thanks for the tracker response.

I'm very appreciative of your work on the bulk upload, and find it bizarre that we are still awaiting a core Moodle solution.

Currently I'll be attempting to replicate just under 300 courses using your upload program. We saved a lot of time last semester with your previous version, but the copy over facility is very important to avoid the need for users to back up and restore.

Unfortunately all our users make their own choices of format and topic numbers, so ideally it would be better for us if there was an automatic pick up of format and weeks/topic number. (Alternatively would setting the maximum number of sections in th eupload file work ?)

Two difficulties I'm having at the moment

1) finding a comprehensive list of what the permissible content is for each field (format field, for example, seems to be a bit picky)

2) Currently the uploads seem to be loading as SCORM format, irrespective of the format settings .

Thanks for any tips you can offer,

best

Paul

 

 

In reply to Paul Fynn

Re: Bulk course upload 2.0

by Paul Fynn -

As a quick update,whilst I can create empty courses with the right number of topics, any attempt to import content during, or seperately after the process, seems to result in the copied content appearing twice in topic 0 in the new or updated course.

In reply to Paul Fynn

Re: Bulk course upload 2.0

by Piers Harding -

Hi -

If you have things like 'News Forum' automatically added by default on the creation of a new course, then this will give the appearance of duplication when you import a backup file (or copy from template course Id).

Perhaps look at your course format defaults and blocks?

Cheers,

Piers Harding.

In reply to Paul Fynn

Re: Bulk course upload 2.0

by Piers Harding -

Hi Paul -

I've been testing the 'format' issue, and it appears to work fine for me, if you put the format in the CSV file or setting the default via the screen options after the initial load of the file.

The options for the format value are 'scorm', 'social', weeks', and 'topics'.  As a general rule, the input values for fields are what you find on the data entry form if you inspect the HTML element.

I've also checked - and as you say, the value set in the file, or as defaults is the format that the course is created with irrespective of what it would have been in the course backup that is loaded over the top.

There does not appear to be any obvious way of passing parameters to the restore controller to get it to pull these details from the backup file.  The only way it appears to work is when you run the restore controller in backup::TARGET_NEW_COURSE.  This adds a restore step that pulls all of the main course settings - which would not be ideal either.

We need another solution...

 

Cheers,

Piers Harding.

 

In reply to Piers Harding

Re: Bulk course upload 2.0

by Paul Fynn -

Hi Piers,

Many thanks - on reflection I'm wondering whether the duplication issue (it is content as well as news) occured after I used the 'overwrite...with defaults' option.

I'll do some more checking.

I'm hoping that I can install Tim Hunts SQL plugin, and that might give a way of 'querying out' the data on format type and number nof sections.

I'll post back here on progress on the main issues,

Many thanks again

Paul

In reply to Paul Fynn

Re: Bulk course upload 2.0

by Paul Fynn -

Hi again,

having switched on the debugging, I'm now seeing this and wondering whether this implicates the upload file, template file or config.

Upload File contents:

fullname  shortname  idnumber summary format enrolmethod_1 role_1  startdate  numsections  maxbytes  lang  templatename
2012.2 HFG122  2012.2 HFG122 2012.1 HFG122 a topics manual student 23.07.2012 20 20971520 en 2012.9 HFG122
2012.2 PG610 Marketing 2012.2 PG610 2012.2 PG610 a topics manual student 23.07.2012 20 20971520 en 2012.9 610PG
2012.2 708 Independent  2012.2 708 2012.2 708 a topics manual student 23.07.2012 20 20971520 en 2012.9 708

Error report:

Upload courses results

Error: mdb->get_record() found more than one record!
  • line 1309 of /lib/dml/moodle_database.php: call to debugging()
  • line 1372 of /lib/dml/moodle_database.php: call to moodle_database->get_record_sql()
  • line 151 of /backup/moodle2/restore_qtype_plugin.class.php: call to moodle_database->get_field_sql()
  • line 131 of /backup/util/plan/restore_structure_step.class.php: call to restore_qtype_plugin->process_question_answer()
  • line 103 of /backup/util/helper/restore_structure_parser_processor.class.php: call to restore_structure_step->process()
  • line 125 of /backup/util/xml/parser/processors/grouped_parser_processor.class.php: call to restore_structure_parser_processor->dispatch_chunk()
  • line 91 of /backup/util/helper/restore_structure_parser_processor.class.php: call to grouped_parser_processor->postprocess_chunk()
  • line 148 of /backup/util/xml/parser/processors/simplified_parser_processor.class.php: call to restore_structure_parser_processor->postprocess_chunk()
  • line 92 of /backup/util/xml/parser/processors/progressive_parser_processor.class.php: call to simplified_parser_processor->process_chunk()
  • line 169 of /backup/util/xml/parser/progressive_parser.class.php: call to progressive_parser_processor->receive_chunk()
  • line 253 of /backup/util/xml/parser/progressive_parser.class.php: call to progressive_parser->publish()
  • line ? of unknownfile: call to progressive_parser->end_tag()
  • line 158 of /backup/util/xml/parser/progressive_parser.class.php: call to xml_parse()
  • line 137 of /backup/util/xml/parser/progressive_parser.class.php: call to progressive_parser->parse()
  • line 105 of /backup/util/plan/restore_structure_step.class.php: call to progressive_parser->process()
  • line 153 of /backup/util/plan/base_task.class.php: call to restore_structure_step->execute()
  • line 148 of /backup/util/plan/base_plan.class.php: call to base_task->execute()
  • line 157 of /backup/util/plan/restore_plan.class.php: call to base_plan->execute()
  • line 310 of /backup/controller/restore_controller.class.php: call to restore_plan->execute()
  • line 747 of /admin/tool/uploadcourse/index.php: call to restore_controller->execute_plan()
Error: mdb->get_record() found more than one record!
  • line 1309 of /lib/dml/moodle_database.php: call to debugging()
  • line 1372 of /lib/dml/moodle_database.php: call to moodle_database->get_record_sql()
  • line 151 of /backup/moodle2/restore_qtype_plugin.class.php: call to moodle_database->get_field_sql()
  • line 131 of /backup/util/plan/restore_structure_step.class.php: call to restore_qtype_plugin->process_question_answer()
  • line 103 of /backup/util/helper/restore_structure_parser_processor.class.php: call to restore_structure_step->process()
  • line 125 of /backup/util/xml/parser/processors/grouped_parser_processor.class.php: call to restore_structure_parser_processor->dispatch_chunk()
  • line 91 of /backup/util/helper/restore_structure_parser_processor.class.php: call to grouped_parser_processor->postprocess_chunk()
  • line 148 of /backup/util/xml/parser/processors/simplified_parser_processor.class.php: call to restore_structure_parser_processor->postprocess_chunk()
  • line 92 of /backup/util/xml/parser/processors/progressive_parser_processor.class.php: call to simplified_parser_processor->process_chunk()
  • line 169 of /backup/util/xml/parser/progressive_parser.class.php: call to progressive_parser_processor->receive_chunk()
  • line 253 of /backup/util/xml/parser/progressive_parser.class.php: call to progressive_parser->publish()
  • line ? of unknownfile: call to progressive_parser->end_tag()
  • line 158 of /backup/util/xml/parser/progressive_parser.class.php: call to xml_parse()
  • line 137 of /backup/util/xml/parser/progressive_parser.class.php: call to progressive_parser->parse()
  • line 105 of /backup/util/plan/restore_structure_step.class.php: call to progressive_parser->process()
  • line 153 of /backup/util/plan/base_task.class.php: call to restore_structure_step->execute()
  • line 148 of /backup/util/plan/base_plan.class.php: call to base_task->execute()
  • line 157 of /backup/util/plan/restore_plan.class.php: call to base_plan->execute()
  • line 310 of /backup/controller/restore_controller.class.php: call to restore_plan->execute()
  • line 747 of /admin/tool/uploadcourse/index.php: call to restore_controller->execute_plan()
  • StatusCSV lineIDFull nameShort nameCategoryID numberSummaryDelete
    New course 2 735 2012.2 HFG122 2012.2 HFG122   2012.1 HFG122 a  
    New course 3 736 2012.2 PG610 Marketing 2012.2 PG610   2012.2 PG610 a  
    New course 4 737 2012.2 708 Independent 2012.2 708   2012.2 708 a  

    Courses created: 3
    Errors: 0

 

In reply to Paul Fynn

Re: Bulk course upload 2.0

by Paul Fynn -

Hi Piers,

We uploaded to 2.2.3 and the May 16th git version of the upload which on initial inspection appears to be working fine.

Will post a progress report here.

Many thanks for all your help

Paul

In reply to Paul Fynn

Re: Bulk course upload 2.0

by Paul Fynn -

Update: The content is now posting appropriately to the required sections, but continues to duplicate content.

This looks like the pointer as to where the issue may be

Error: mdb->get_record() found more than one record!

  • line 1309 of /lib/dml/moodle_database.php: call to debugging()
In reply to Paul Fynn

Re: Bulk course upload 2.0

by Piers Harding -

Having a look at the code, this appears to be an issue with the Question Type restore function - it seems to be reporting that it is getting duplicate values for answers (if I'm looking at the same Moodle version as you).  Are you by any chance, restoring the same file twice - you can specify the backup multiple times -  templatename (course shortname), the backup file uploaded in the gui, and a specific backup file in the CSV file.  If there is overlap with these then they will clash - the uploader works through all of these options sequentially for each course, so in theory it is possible to restore 3 different (or the same) courses over a course in the same run.

Cheers,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by Paul Fynn -

Hi Piers, as the sun goes down in Switzerland, it illuminates, I presume, NZ ?? Good to catch you on the crossover.

Good theory, I had been considering whether the backups had been an issue, or whether there was some issue with repeated restoring.

However, I tried to counter this by creating a template file with just URLs for content, and these are still provoking the same symptoms and message. (ie there are no quizzes, but the links are still duplicated).

Could the upload course be tripping up over itself ie finding its own backup prior to restoring as a new course ? Does the uploader run through every course i sequence, or complete one course at a time ?

best

Paul

In reply to Paul Fynn

Re: Bulk course upload 2.0

by Paul Fynn -

Just tried single upload, no quizzes, unfortunately it's still doubling.

In reply to Paul Fynn

Re: Bulk course upload 2.0

by Peter Stacey -

Hi

I've been testing this and have found that the script is consistently doubling all course content when I use a template course, but not when I use a backup file.

It appears to me that the restore is happening twice in the case of using a template course.

I have managed to fix this by commenting out lines 724 - 753 of index.php

It doesn't appear (so far) to have any detrimental effects.

Thanks for your work on this Piers, it's something that I've been missing since I've started using Moodle. I have developed a (non APi) script for making a course copy directly on the database, but obviously using the backup and restore functionality should be a more future-proof approach.

 

Cheers

Peter Stacey

In reply to Peter Stacey

Re: Bulk course upload 2.0

by Piers Harding -

Hi Peter -

This is 'my bad' - there is a bug where one restore is handing the same file onto the next.

I've pushed a fix up to https://github.com/piersharding/moodle-tool_uploadcourse - could you test it again by any chance and let me know?

Thanks,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by Peter Stacey -

Hi Piers,

Thanks for that, I've tested it out and it's working fine now.

Cheers

Peter

In reply to Peter Stacey

Re: Bulk course upload 2.0

by Paul Fynn -

Hi Piers,

Just to confirm the same as Peter.

Appears to be working fine on 2.2.3, on our test site, but not on 2.2.2 on our main site, so we are upgrading later today.

I'm interested to see how this handles Turnitin Assignment types in the past, as the API has a history of being troublesome in backup and restore.

Interesting that a copy of the course backup file is also created in the new course at the time of upload.

Many thanks for all your work, and particularly the very quick turnaround that you have generously provided on the bug fixes.

best

Paul

In reply to Paul Fynn

Re: Bulk course upload 2.0

by Paul Fynn -

Hi Piers,

We've upgraded our main site to 2.2.3 and here's a rundown of the results.

The first record uploads and creates a new course as required, but the upload then fails and does not load the second course. With full debugging on I get the following error message:

Upload courses results

Notice: Undefined index: enrolmethod in /home/httpd/vhosts/web1/web/admin/tool/uploadcourse/index.php on line 847

Notice: Undefined index: enrolmethod in /home/httpd/vhosts/web1/web/admin/tool/uploadcourse/index.php on line 847

Notice: Undefined index: enrolmethod in /home/httpd/vhosts/web1/web/admin/tool/uploadcourse/index.php on line 854

Notice: Undefined index: in /home/httpd/vhosts/web1/web/admin/tool/uploadcourse/index.php on line 854

Fatal error: Call to a member function add_default_instance() on a non-object in /home/httpd/vhosts/web1/web/admin/tool/uploadcourse/index.php on line 855

Turnitin Handling

As I anticipated, the upload programme replicated all the course content including the old Turnitin assignments, which effectively links the new course submission back to the old template submission dates and settings. I don't know whether it would bê feasible to have an option field in the upload file either to ignore Turnitin, or to convert it to a placeholder.

""Notice
Duplicate copies of the same Turnitin assignments were found, duplicates occur when one or more Moodle Direct assignment is connected to the same one Turnitin assignment via the Turnitin API. This is known to cause issues, most notably submissions made to one of the duplicates will show up in the submission inbox of the others. To fix this you should either delete the duplicate assignments or reset the course where the duplicate assignments are.

The following duplicates were found:""


Overall your plugin is looking like a really important contribution to moodle admin.

Thanks again

Paul

In reply to Paul Fynn

Re: Bulk course upload 2.0

by Piers Harding -

Hi Paul - can you send me the firs few rows of your file so I can test the particular scenario - you can send it directly to me at piers

at ompka dot net.

Thanks,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by John St -

I'm trying to use 1.9 backup files with the tool and seems to fail. Well actually it just says it can't find the file. Is it looking for an mbz, doesn't see one, then quits? 

When I remove the mbz check, it fails with:

error/cannot_precheck_wrong_status

Any thoughts on how this can be made to work with 1.9 backups? Thanks for a great tool!

In reply to John St

Re: Bulk course upload 2.0

by Piers Harding -

Hi -  It has not been created to work with 1.9 backups, as there is an in built assumption that the backups are 2.0+.  Having said that, it probably is't that hard to make it work with 1.9 files - I just haven't tried.

There is a convert method on the restore controller:

// check if the format conversion must happen first
if ($rc->get_status() == backup::STATUS_REQUIRE_CONV) {
$rc->convert();
}

Cheers,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by Piers Harding -

Hi -

 

I've pushed a change up for the backup format conversion 1 -> 2.  I would be interested to hear back from anyone who can test it.

 

Cheers,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by John St -

It still checks for an mbz file and errors out (if you specify a backup with csv). Once I removed that, it seems to work! The only thing I noticed is that even if I specify to use Course Files, it does not appear. Not sure why that might be yet. 

In reply to John St

Re: Bulk course upload 2.0

by Piers Harding -

Hi - thanks for testing.  I've widdened the file extension test to include mbz and zip, and pushed the fix up to https://github.com/piersharding/moodle-tool_uploadcourse.

 

With the course files option - are you sure that they are in the backup file?  Can you test that independently with the restore dialog?

Cheers,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by John St -

Thanks! That is working great. Is there any way to get the shortname and course name from the backup file instead of specifying it in the csv file? 

The legacy course files still aren't created, but they are if I do a normal restore as a new course. However, if I do a normal restore into an existing course, the legeacy course files are not created... So, I'm a bit stumped by that. 

In reply to John St

Re: Bulk course upload 2.0

by Piers Harding -

Hi - thanks for the feedback.

 

Therre must be away to unpack (unzip) the backup files and target the moodle.xml file inside.  If you parse this out you can get all the basic course details.

I don't have an example of doing this, but I don't think it would be that difficult to do.

Cheers,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by Pat OKeefe -

Hi,

I am having trouble using the course upload feature. Specifically I do not seem to be able to get the scipt to allow assigning of a valid user to the course context teacher role.  Here is my .csv upload file:

fullname,shortname,category,summary,idnumber,teacher

CSE(317-001),CSCE317-001,Computer Science & Engineering/Summer 2012,Computer Systems Engineering,SM2012CSCE317001,<somevaliduserID>

I have tried both "teacher" and "editingteacher".  No luck.  I have to manually add the role.  Any ideas?

In reply to Pat OKeefe

Re: Bulk course upload 2.0

by Piers Harding -

Hi -

 

The fields teacher, editingteacher, student, manager are text descriptive fields only (Role names).  If you wnat to enrol people in a course then you must use the User upload tool.

Cheers,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by Pat OKeefe -

HI Piers,

Thanks for the reply.  I have managed a way to bulk add teacher (editingteacher) roles to specific teacher/users using the flatfile enrollment method.  It is a real step backwards if you ask me.

I use an external database to manage enrollment records we receive from our main registrar office. As you can imagine the course-student records change quite a bit.....  Manually updating the list is impossible.  We had the whole scheme wired in version 1.9.  Including faculty/teachers assigned to each course. (used bulk course upload to manage teachers) 

For some reason when I update our enrollment database and add either "editingteacher" or "teacher" to the role field,  moodle 2.2.3 defaults the role to student.  I have worked this problem for days and finally gave up when I  found the flatfile workaround.

For the record, remote enrolment sync fileds are correct and match the entries for our working 1.9 site:

Default role: enrol_database | defaultrole -> Student

Remote role field: enrol_database | remoterolefield -> role

 

Perhaps my table element type for the role is not making moodle happy?

------------------

field, type, null

role, varchr(15), YES

-----------------

Here is the table entry:

CSCE215-851|pokeefe|editingteacher

When I sync up the databases:

# /usr/local/bin/php -c /etc/php.ini /info/docs/www/moodle2/enrol/database/cli/sync.php -v
Course synchronisation skipped.
Starting user enrolment synchronisation...
  enrolling: 3 ==> CSCE215-001 as student
...user enrolment synchronisation finished.

note the enrolled as a student output.

Just wanted to thank you for your time and see if I missed something.

Cheers,

Pat O'Keefe

In reply to Piers Harding

Re: Bulk course upload 2.0

by John St -

I still can't get it to make the legacy course files. I don't think this is an issue with your plugin specifically, but maybe a more global moodle one. When it makes the courses, regardless of what I set globally (or in your plugin) the course settings have legacy course files set to 'no' upon creation. 

In reply to Piers Harding

Re: Bulk course upload 2.0

by Richard Oelmann -
Picture of Core developers Picture of Plugin developers Picture of Testers

Hi Piers,

I've used your tool succesfully in the past to create a number of blank courses, but I'm now trying to use it to upload a series of course backups - as new courses. These are 1.9 course backups into Moodle 2.2.3+ (Build: 20120612) and using the version of your tool from the github link as above. I'm running into a problem that when I try to upload the courses I get an error message in the status column of every course saying Template file not found - except I am not using any template files, either in the csv or in the options on the tool's page. Do I need to be? Am I doing something wrong?

Below is the first few lines of my csv file and I've added a screenshot of the error in the hope someone can help point out what is probably a silly error on my part smile

 

fullname,shortname,summary,category,backupfile
Applied Mathematics 1 ,AP1,Applied Mathematics 1 ,Engineering_and_Science_Pathway,/home/richard/Desktop/olncoursebackups/backup-ap1-20120607-0404.zip
Applied Mathematics 2 ,AP2,Applied Mathematics 2 ,Engineering_and_Science_Pathway,/home/richard/Desktop/olncoursebackups/backup-ap2-20120607-0406.zip
Political thought,Apt,Political thought,Unsorted,/home/richard/Desktop/olncoursebackups/backup-apt-20120607-0106.zip
Basic Skills,B_S,Basic Skills,Unsorted,NULL
Biology 1 (Cell Biology),B1,Biology 1 (Cell Biology),Engineering_and_Science_Pathway,/home/richard/Desktop/olncoursebackups/backup-b1-20120607-0408.zip

screenshot

In reply to Richard Oelmann

Re: Bulk course upload 2.0

by Piers Harding -

Hi -

I have just tested the backupfile parameter and it is working, so I'm wondering if it's one of the following problems:

* it's client server so you are referencing files on your client, but the server needs to have direct file system access to them

* file names are slightly incorrect - case sensitive

* permissions are not set on the files so that the web server user can actually read them, even though the names etc. are correct.

Cheers,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by Richard Oelmann -
Picture of Core developers Picture of Plugin developers Picture of Testers

Hi Piers

Many thanks for your thoughts - I didn't consider problems with the backup files as the error message was for Template thoughtful but I will check your suggestions when I get to work on Monday.

At the moment the testing I'm doing is all on a localhost with the moodle in /var/www and the folder of backup files on my desktop - I don't think, then that it will be a client/server issue and I thought I had changed permissions on the folder, but it would make sense that being in /home/richard/Desktop the permissions are not correct. As I said, I'll try on Monday and see what happens

I think the file names are all correct as I generated them from a file listing into a text file and copied that into the csv file, so I don't think there'll be any case errors (but I will check smile )

- I know the plugin works as others have had success with it so it's got to be an issue my end. So again Piers, thanks for a great tool and thanks for your helpful suggestions smile

Richard

In reply to Paul Fynn

Re: Bulk course upload 2.0

by Paul Fynn -

Hi Piers, we're approaching that time again for uploading new courses for the next semester, greatly eased by your tool (we do this three weeks before the end of the previous semester).

We'll be installing teh latest version from github for testing shortly.

Something that has been discussed previously on Turnitin handling :

the upload programme replicates all the course content including the old Turnitin assignments, which effectively links the new course submission back to the old template submission dates and settings.

Would it bê feasible to have an option field in the upload file either to ignore Turnitin, or to convert it to a placeholder.


Many thanks for any thoughts you have,

best

Paul

 

In reply to Paul Fynn

Re: Bulk course upload 2.0

by Jamie Tinley -

Thank you Paul for posting your example!  I had issues with startdate which kept defaulting to todays date which it says it would at the main download site here.  Your example helped me realize it is in the form dd.mm.yyyy like 23.01.2013     If anyone else reads this and is updating other courses then make sure you choose from the dropdown to update existing courses only AND vip next dropdown choose override with file.

Thanks!  

In reply to Piers Harding

Re: Bulk course upload 2.0

by Moshe Golden -

Hello Piers,

I installed your course upload module and it works beautifully.  I was wondering if it would be possible to create a version which would work the same way, but with a "fixed name and path" input file and  be launched via a cron script.

 

Moshe

In reply to Moshe Golden

Re: Bulk course upload 2.0

by Piers Harding -

Hi - Yes, this would be possible, but it is a big job, and will have to wait for later.

Cheers,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by Nick Gault -

Hi,

First of all, thanks to Piers for this tool - it is going to make life easier.

I am currently moving old (1.9) courses on to a new install(2.3) using this tool and course backups.

I am getting a few problems though and am wondering if anyone has any suggestions.

Issue 1: If I include any categories in the csv the course creation fails.

Issue 2: If I create/restore from an automated backup file the course files do not appear in the new course, but if I manually back the course up and then run the tool the course is restored in full. The automated backup files contain all of the documents required but something is stopping them from appearing in the new course. -   It looks like the automated backup files don't include resources despite the presence of the files. 

Thanks in advance.

Nick

In reply to Piers Harding

Re: Bulk course upload 2.0

by Aleks Macheta -

Hi Piers,

Many thanks for developing this really useful tool.

I am using it to test uploading courses to 2.2.4 and have noticed that the 'summary' field seems to be a required field in the csv i have created.  If i omit this field in my csv i get a warning "summary is missing". 

I don't really need to populate this field - Is it easy to make the field optional in the code?  If so, please let me know where to look to do so.

with many thanks,
Aleks

In reply to Aleks Macheta

Re: Bulk course upload 2.0

by Michael Hughes -

small workaround to this is just to use a non-breaking space in summary field &nbsp; 

...,summary,...

...,&nbsp;,...

In reply to Michael Hughes

Re: Bulk course upload 2.0

by Derek Chirnside -

In some random search I found this today: https://tracker.moodle.org/browse/MDL-38501

"Create a "Bulk course action" menu, allowing several operations (backup, delete, reset, change settings...)"

11 votes in under a month.  This looks like a promising goal.  But there is also this rather old item from Anthony: MDL-17017

"Admin: Evaluate file structure for administrative functions for consistency"

Now I understand why this seemingly simple item (ie bulk course create) is so complex: the database structure is complex, and all the options in setting up a course are difficult.

Clone buttons may to someway towards this.  (it's great now to be able to duplicate activities) but at the course level there are names, shortnames etc to consider and feature creep kicks in immediately.

I still miss, every day, bulk user actions for these areas

  1. users
  2. courses
  3. activities

-Derek

In reply to Piers Harding

Re: Bulk course upload 2.0

by R.T. Brown -

Hi Piers, Thanks for the great tool. I've been testing and noticed that when a course is restored it has several blocks that the templated did not. Do you know why that might be?

Thanks --

R.T. Brown

In reply to Piers Harding

Re: Bulk course upload 2.0

by Jeff Hague -

Hello all! I am struggling with this and need some help!

I have downloaded master.gz and unzipped it using both Winzip and 7zip. In both cases it extracts a single file called master with no extension.

The instructions say:

"

Download https://gitorious.org/moodle-tool_uploadcourse/moodle-tool_uploadcourse/archive-tarball/master
unpack the file (probably called master) with tar -xzvf master
This will give you a directory called moodle-tool_uploadcourse-moodle-tool_uploadcourse
Move this directory and rename it into its final position:
mv moodle-tool_uploadcourse-moodle-tool_uploadcourse <Moodle dirroot>/admin/tool/uploadcourse"

 

but I dont get a directory and cant seem to figure out how to extract it as such. What am I doing wrong?

Thanks!

In reply to Jeff Hague

Re: Bulk course upload 2.0

by Jeff Hague -

Sorry - I figured it out... Apparently, unzipping a tarball on Windows using 7zip is a 2-step process.

Thanks!

In reply to Steve Anatai

Re: Bulk course upload 2.0

by Rosario Carcò -

Please see also this one:

http://moodle.org/mod/forum/discuss.php?d=128317

I developed it for 1.9.x to import courses and users at the same time. Courses are created first and in a second pass through the cvs file users/teachers/students are enrolled in their respective roles.

Unfortunately categories have to be created first in the GUI. I still have to add this feature to my code, to create the category if it does not exist. And my code is not at click and go stage yet. But you can tailor it to fit your exact needs because I am used to make comments in every line of code to facilitate reading.

Rosario

In reply to Mark Johnson

Re: Bulk course upload 2.0

by Tim Geiger -

The script works perfect for me. However I would like to some how automate the process. Is there any way to point this script towards a specific .csv file on the server without having to manually log in and select a file. I can then schedule the server to run the script daily. Any help would be appreciated.

Thanks

In reply to Tim Geiger

Re: Bulk course upload 2.0

by Piers Harding -

Hi - this could be done, but would be quite a significant amount of work, as the entire index.php, and form code would need restructuring.

Command line parameters would need to e developed for all of the defaulting options in the leading forms, and then the csv, and message/error handling would need remodelling to deal with both interactive, and command line use.

 

It's a great idea, but one that will take a lot of time which I don't have at the moment.

One for later I think.

Cheers,

Piers Harding.

In reply to Piers Harding

Bulk course upload 2.0

by chhavi garg -

I need to upload coures as well as categories from same excel file .Can some one tell me how to do it??

In reply to chhavi garg

Re: Bulk course upload 2.0

by Piers Harding -

Hi - it is more likely to work if you split out the columns for categories into a separate file, and try using https://github.com/piersharding/moodle-tool_uploadcoursecategory

Then you can upload the courses using https://github.com/piersharding/moodle-tool_uploadcourse

Cheers,

Piers Harding.

In reply to Piers Harding

Re: Bulk course upload 2.0

by chhavi garg -

hi,

I just have 1 ecxcel sheet containing details of categories and courses.Can you please tell me how to divide that excel sheet by php program.

In reply to Piers Harding

Re: Bulk course upload 2.0

by Séverin Terrier -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Testers Picture of Translators

Piers : I've tested your tools to automatically create courses (or course categories), and they are useful !

In README, you show which fields can be specified, but it would be more useful to have the list of required fields opposed to optional fields (like in Upload user documentation).
Moreover, it seems several fields are required by your tools (like idnumber), but are not (required) in Moodle. I'm not sure it's really a good idea. Would better let only Moodle required fields to be required...

Séverin

In reply to Séverin Terrier

Re: Bulk course upload 2.0

by Séverin Terrier -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Testers Picture of Translators

It seems i've seen a bug in your tool for upload courses, with Moodle 2.4.3 (not tested with other versions).

Everything is OK while creating courses via the web interface, using fields defined in the text file. It's also OK if i use a backup file, specying it's name via the web interface (and not in text file).

But if i select a course to use as template (not specifying it in the text file), activities are restored twice (one time the several activities, and then another time) in the generated courses mixed

Problem is the same for activities in section 0 or in other sections.

In reply to Tim Geiger

Re: Bulk course upload 2.0

by ahmad hamwi -

The tool is amazing,

but we're having a slight problem.

All the material is posted in topic 0

Anyone knows the reason?

thank you

In reply to ahmad hamwi

Re: Bulk course upload 2.0

by ahmad hamwi -

In addition any luck with the legacy files showing?
And last thing, is there a way to auto generate the CSV file from moodle 1.9? So we don't populate the CSV file manually which can take a long time?

 

thank you

In reply to ahmad hamwi

Re: Bulk course upload 2.0

by Mohan Paliwal -

It worked for me with a little trick, as the bulk course upload module have a bug even with the 2.9 version. If you can access your database just find out the actual id for the category you want to upload (id is an internal auto number assigned to category when we add a category) so upload  your file with id not with idnumber which is user defined number. (Mysql/Mariadb sql command for knowing the actual id (select id,name from mdl_course_categories).