It sure would be nice if one could simply run uploaduser.php or uploadcourses.php from a script, with no admin interaction. From what I understand this is not possible. The flat file plug-in for enrolments does work this way, but not for creating new user accounts or courses.
Do I have this straight?
Is there a workaround?
I need to be able to process flat files (created by an external ERP system) to create and/or modify user accounts and courses, ideally from the cron job or if need be inside a php script to which the file location was specified in a POST request.
All help appreciated as this is urgent.
(running Moodle 1.9 on Windows Server 2003 with Mysql and PHP)
Alan - Sorry for a less than quick response to your inquiry. Have you been able to make any progress? I think from a security perspective, I would be fearful of having something that takes as a parameter the location of a file and uses that to create users and/or courses. In any case, if you could update me on your status as I know there has been some progress with folks doing course creation via CSV files and am not sure if you have figured out what was needed by now. If I can be of assistance just let me know. Peace - Anthony
Better late than never, Anthony!
Actually, I did figure out how to adapt the flat file plugins to use in the way I wanted, but there was so much customization required that I decided to go with LDAP for authentication and user account creation, and External DB for authorization (role assignments). I configured the latter so it does automatic course creation, so I ended up not needing a special "batch course upload" function at all.
The downside is that LDAP insists on processing ALL the records in the LDAP directory, which takes for ever, even when there is nothing that needs to be done for them (nothing has changed). I am going to add a "last modified" parameter and adapt the LDAP plugin to use it as a filter in the LDAP query, but I am afraid that the plugin still insists on checking every existing account in mdl_user to see if it needs updating in terms of possible changed in LDAP. So it is really slow.
In the future we will start using the IMS upload plugin, but at the moment it is easier to avoid writing the code on the ERP side that would be needed to prepare XML files with just the right records.
Have a happy New Year.
Alan
Actually, I did figure out how to adapt the flat file plugins to use in the way I wanted, but there was so much customization required that I decided to go with LDAP for authentication and user account creation, and External DB for authorization (role assignments). I configured the latter so it does automatic course creation, so I ended up not needing a special "batch course upload" function at all.
The downside is that LDAP insists on processing ALL the records in the LDAP directory, which takes for ever, even when there is nothing that needs to be done for them (nothing has changed). I am going to add a "last modified" parameter and adapt the LDAP plugin to use it as a filter in the LDAP query, but I am afraid that the plugin still insists on checking every existing account in mdl_user to see if it needs updating in terms of possible changed in LDAP. So it is really slow.
In the future we will start using the IMS upload plugin, but at the moment it is easier to avoid writing the code on the ERP side that would be needed to prepare XML files with just the right records.
Have a happy New Year.
Alan
Alan - Glad to hear you were able to get things working (even if slowly) and are working toward a hopefully better solution. Happy New Year! Peace - Anthony
I am looking for the same thing. I already have a daily download of the schedule from our student management system that puts students in their courses and automatically makes the groups based on the course sections. This works with a slightly modified uploadusers.php file.
I would love to make this pickup the file automatically each night. I really don't think it would be a security issue since the location of the file is secure, and I can hardcode the file name and location into the code.
If anyone has a solution to this please let me know!
Brian
I would love to make this pickup the file automatically each night. I really don't think it would be a security issue since the location of the file is secure, and I can hardcode the file name and location into the code.
If anyone has a solution to this please let me know!
Brian
Okay...I didn't find it anywhere else so I modified uploaduser.php and made a new file autouploaduser.php.
I hardcoded the file folder and import file as $CFG->dataroot/moodleimport/userimport.csv. So I am using a folder moodleimport in the moodledata to hold the import. This folder is actually mounted to a ftp directory on a 2000 box where the DA site downloads the file nightly.
I then added the autouploaduser.php in the cron to run at 4:30 AM each day so the new users and schedule changes happen the next morning that the secretaries put them in.
My version of uploaduser adds the users, as well as propogates them into the courses, and adds all the groups (course sections) that they are in as well.
We are using 1.87+ Hope this helps you...and anyone else looking for an more automated way to get the users in.
Oh...I am LDAP authenticated so there is no password file and the email is optional in the file as well. The autoupload gives no feedback...it just runs. It is set to make changes automatically...but that can easily be changed in the file.
Brian
I hardcoded the file folder and import file as $CFG->dataroot/moodleimport/userimport.csv. So I am using a folder moodleimport in the moodledata to hold the import. This folder is actually mounted to a ftp directory on a 2000 box where the DA site downloads the file nightly.
I then added the autouploaduser.php in the cron to run at 4:30 AM each day so the new users and schedule changes happen the next morning that the secretaries put them in.
My version of uploaduser adds the users, as well as propogates them into the courses, and adds all the groups (course sections) that they are in as well.
We are using 1.87+ Hope this helps you...and anyone else looking for an more automated way to get the users in.
Oh...I am LDAP authenticated so there is no password file and the email is optional in the file as well. The autoupload gives no feedback...it just runs. It is set to make changes automatically...but that can easily be changed in the file.
Brian
I don't know that I ever got back with you on the sample file.
The process is a little convoluted. On my server's FTP folder is a file usernames.csv which includes just the studentid and username. It is used by the DA site to get our usernames (I want to make them based on graduation year) and tie them into the download they give us.
They download the file with the student information to my ftp folder nightly, which is a cumulative of all currently scheduled students and courses. There is one line, per student, per course. An example is attached.
It updates the student information in moodle...adds new classes...and created the accounts if required.
The Groups are made based on the period-section and the course names are the course numbers with the instructors initials appended. This way if I have more than one instructor teaching the same course the student will go into the correct moodle course.
Brian
The process is a little convoluted. On my server's FTP folder is a file usernames.csv which includes just the studentid and username. It is used by the DA site to get our usernames (I want to make them based on graduation year) and tie them into the download they give us.
They download the file with the student information to my ftp folder nightly, which is a cumulative of all currently scheduled students and courses. There is one line, per student, per course. An example is attached.
It updates the student information in moodle...adds new classes...and created the accounts if required.
The Groups are made based on the period-section and the course names are the course numbers with the instructors initials appended. This way if I have more than one instructor teaching the same course the student will go into the correct moodle course.
Brian
I should say that it doesn't add new classes...it adds students to existing classes and does make new groups (if they didn't already exist) based on the course sections.
Sorry about the confusion on that one.
Brian
Sorry about the confusion on that one.
Brian
Brian,
Thanks. I'll take a look.
Meanwhile, do you a similar script that auto import courses too?
dav
Thanks. I'll take a look.
Meanwhile, do you a similar script that auto import courses too?
dav
Hi Brian
Thanks you for your time and effort in modifyng the uploaduser.php file.
I was trying to use your autouploaduser.php with version 1.9.2 and I think it is not compatible. I was wondering if you have the modified verison of autouploaduser.php for version 1.9.2.
many thanks in advance
Brian
This looks like a great script. I am not a developer per se, but I am learning. How difficult would it be to modify this script to only update custom profile fields for existing users? I am using 1.9.4+ currently.
Scott Karren
This looks like a great script. I am not a developer per se, but I am learning. How difficult would it be to modify this script to only update custom profile fields for existing users? I am using 1.9.4+ currently.
Scott Karren
I was hoping to use this on our site ... but we're running 1.9.1+ (from June 2008) and I get an error saying field "" does not exist.
I've tested it with both your sample file and with one of my own that works with the manual upload users.
I don't suppose you've updated to 1.9?
I've tested it with both your sample file and with one of my own that works with the manual upload users.
I don't suppose you've updated to 1.9?
In reply to Mairsile Hourihane
Re: Automated CSV file processing (users, courses)
by Alberto Permuy Leal -
Hi!
Anybody find any script or file for autoupload users with data in a CSV file? I've reading last days in moodle.org forums and a lot of sites but I don't find any script for 1.9.5 version.
Thanks in advance!
Anybody find any script or file for autoupload users with data in a CSV file? I've reading last days in moodle.org forums and a lot of sites but I don't find any script for 1.9.5 version.
Thanks in advance!
In reply to Alberto Permuy Leal
Re: Automated CSV file processing (users, courses)
by Anthony Borrow -
It has been a while since I have poked around with this question of enrollments and users; however, I wonder if the Flat_file enrollment type might be of use to you.Peace - Anthony
Hi,
Does anyone created the file "autouploaduser.php" for version 1.9 of Moodle? I try, but it's more complicated in 1.9 then in 1.8.
Thank you
do you have uploaduser script for Moodle 2.2. It looks like moodle doesn't have uploaduser.php, but has a folder with multiple files like index.php, locallib.php, user_form.php etc. Any help will be greatly appreciated.
I'm also wondering if anyone has done this before with Moodle 2.x?
Thanks for this Brian,
I'm sure it will be useful...
As a related question - can anyone help out with information about External Database authentication? There is an option (v1.9) for type of DB as "csv".
I can't find any information on Moodle Docs or the forums about what this does or how it works.
Can anyone shed any light on this?
Thanks,
Matt.
I'm sure it will be useful...
As a related question - can anyone help out with information about External Database authentication? There is an option (v1.9) for type of DB as "csv".
I can't find any information on Moodle Docs or the forums about what this does or how it works.
Can anyone shed any light on this?
Thanks,
Matt.
Hi Matthew,
If you are a programmer, take a look at lib/adodb/drivers/adodb-csv.inc.php
This is the implementation for the CSV driver. You'll see that it doesn't actually work on a flat CSV file, it requires it to be on a website and passes it parameters like
GET /myfile.csv?sql=SELECT+%2A+FROM+table+WHERE+User+%3D+%27blah%27%0A+AND+Pass+%3D+%27mypass%27+&fetch=2
So you would actually have to make a server side script to process the SQL and return the results in CSV format... so not actually very useful!
In cases where I've needed to automatically upload users from a csv, I did what Brian has done and modified the uploaduser.php code to remove all the interactive web interface parts, so that it can be ran from command line.
Where possible, it's more managable to use a proper external database, like MySQL, and run the auth_db_sync_users.php script to sync them in.
Cheers
Ashley
If you are a programmer, take a look at lib/adodb/drivers/adodb-csv.inc.php
This is the implementation for the CSV driver. You'll see that it doesn't actually work on a flat CSV file, it requires it to be on a website and passes it parameters like
GET /myfile.csv?sql=SELECT+%2A+FROM+table+WHERE+User+%3D+%27blah%27%0A+AND+Pass+%3D+%27mypass%27+&fetch=2
So you would actually have to make a server side script to process the SQL and return the results in CSV format... so not actually very useful!
In cases where I've needed to automatically upload users from a csv, I did what Brian has done and modified the uploaduser.php code to remove all the interactive web interface parts, so that it can be ran from command line.
Where possible, it's more managable to use a proper external database, like MySQL, and run the auth_db_sync_users.php script to sync them in.
Cheers
Ashley
Thanks Ashley,
I'm not a developer, so I think I'll stick with modifying uploaduser.php!
Regards,
Matthew.
I'm not a developer, so I think I'll stick with modifying uploaduser.php!
Regards,
Matthew.
Hi all,
Has anyone managed to develop a script for auto uploading and enrolling users that works on 2.7/2.8?
Does "Flat file" not meet your requirements?
Thanks for replying Just H,
The automated flat file method is only useful if the user accounts already exist. I'm after something automated that will create user accounts and enrol/group, as the original poster requested.
The autoupload.php script that Brian Pool originally posted is exactly what I need, I originally tried adapting it but I hit a brick wall as it utilises the uploaduser.php file, which no longer exists in Moodle 2.7/2.8.
Thanks
My apologies, missed that ... in that case have you had a look at Elis? The Elis Data Hub should do it; may be overkill for your requirements.
Thanks for the recommendation, I'll look into it, although on quick inspection it might be slightly overkill, as you said.
As I thought, the ELIS system is overkill for what we want. The effort and cost involved to get it up and running outweighs the benefits. Thank you anyway for the suggestion Just H.
Has anyone had any luck with scripting the functionality? Or know of any other alternatives?
Thanks,
Andy
I thought it might be (not a bad set of plugins if needing more business type functionality). Good luck with finding a solution, must be a guru out there some place that has a fix for this issue.
The external database authentication plugin (in core) can apparently handle CSV input. Never used it, but I know that the option is there. Probably worth a look.
Hi Kevin,
Thank you for the suggestion. I tried this yesterday but couldn't get it to work, mainly because I couldn't figure out where to put the path to the csv file!
I know you said you haven't tried it yourself, but I don't suppose you know a good source of information that might document this process? Googled and scoured the internet but came up with nothing.
Many thanks!
Hello, I am a Moodle newbie, have you reached a solution for that on 2.9? or a way to hardcode the csv source location, path or name so once we run the cron script the csv file will be uploaded automatically and students will be synced? thanks a lot for your help and suggestions.
I think I re-developed what you need. I had done this some time ago for version 1.9 but just redid it for 3.1.
You simply drop the attached file in the admin/tool/uploaduser directory. There are then 2 variables inside the script that you need to set.
$customfileLocation = "$CFG->dataroot/moodleimport/userimport.csv";
$defaultdomain = "@yourdomain.com";
The first is where your csv file sits. Mine is a mount inside the moodledata folder that the csv is dropped into automatically from our student management system (DASL) each night. The file conforms to the same format as the manual userupload in Moodle and the file is simply an edited version of that (from 3.1).
The second variable is for uploads without email fields. It will automatically make all new user emails username@yourdomain.com so that the upload functions without emails. This is the same as putting %u@yourdomain.com in the manual upload.
I then make this a nightly CRON job on my system.
I have been using this system for years. Automatically makes and updates existing users, adds them to correct course, and into the correct groups. Makes everything hands off once the courses are in Moodle.
Hope that helps!
I
would add this as a upload but the upload feature does not allow me to
upload something without it being a full mod, which this is obviously
not, since it is a single file.
Brian
Thanks Brian.
What columns do you need to have in the userimport.csv file?
Can you post and example .csv file?
The file is EXACTLY the same as is used to manually upload files through the manual upload. It is a .csv file. You do not have to have the email in the file and it will then use the username with the email domain that you specify in the file. The only required are username, firstname, lastname. eMail is required by Moodle but as stated I coded around that. Mine is automatically linked to our student software so the CSV file is created each night on an export. Then the cron job runs this and imports all new users, class additions, and puts them into the appropriate group as well.
The example file is just one student and is typical.
Hope that helps!
Brian