LDAP Synchronisation

LDAP Synchronisation

by Mike Cowley -
Number of replies: 12

Hello.

I am a new user to Moodle, and have been tasked by my school to create a sample system to see how it can benefit our staff and students. I have managed to get a simple Moodle set up, and I am using an LDAP connection to our Active Directory to allow users to login via their school IT accounts.

However, I would like to use the synchronisation options available to automatically add and update user accounts to the Moodle database. However, for some unknown reason, this only works for staff and not for any pupils.

Our Active Directory is organised so that the staff users are contained within one OU called staff, and our pupils are contained within a sub OU within one called pupil.

When I run the auth_ldap_sync_users.php file from the command line in Windows a count is shown that there are 866 users in one OU (pupil) and 120 users in another (staff). The system then proceeds to add / update the staff records but totally skips the pupils.

If I edit my LDAP settings in Moodle so that only the pupil OU is listed and then re-run the script, the same count is given (866) followed by an message stating "Did not get any users from LDAP - error? - exiting".

Does anyone have any idea why one OU will work whilst another will not? Also, why does both OUs work for authentication at login but not synchronisation?

Thanks in advance for any ideas.

Average of ratings: -
In reply to Mike Cowley

Re: LDAP Synchronisation

by Steve Power -

Mike

Sorry but I cannot directly solve your problem. I can however ask a few questions based on my experiences using auth_ldap_sync_users.php.

Are you running the script from the command line? If so have you specified to use extra memory. My synch fails unless I use -d memory_limit=256M.

I have also had problems with AD not returning more than 1000 results and so we had to increase the limit there.

What have you got set in the user ID number field. The synch script needs sAMAccountName to be there. When I had something else (our ID number) stored there all of my users were deleted and then recreated.

Hope some of this helps.

Steve

Average of ratings: -
In reply to Steve Power

Re: LDAP Synchronisation

by Mike Cowley -

Thanks for your reply Steve.

The script is being run from the command line. I have increased the level of memory for PHP but am seeing the same results. In regards to ADs limit on the number of objects to return I have attempted to use a sub OU of Pupil. This OU contains around 180 pupil records and receives the same error message.

I did not have anything set in the LDAP settings field for the student ID. I entered sAMAccountName as mentioned in your post but nothing changed when running the script.

I would understand things more if both the staff and pupil OUs were failing and not just one of them. Using the LDAP browser I am able to see the full LDAP tree with the bind user I have created.

Thanks for your time.

Average of ratings: -
In reply to Mike Cowley

Re: LDAP Synchronisation

by Steve Power -

Mike

Yes it is puzzling. I had problems with students not staff but my staff were in the hundreds and students in the thousands.

Having said that I am still not happy with the results of synching and am only experimanting on a backup of my live system on our development server.

Hope you solve the problem.

Steve

Average of ratings: -
In reply to Mike Cowley

Re: LDAP Synchronisation

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
> Does anyone have any idea why one OU will work whilst another will not? Also, why does both OUs work for authentication at login but not synchronisation?

Authentication and synchronisation use different code paths (as they do different things), so one thing may work and the other not. This is most probably due to configuration settings.

What are the values for ldap_user_attribute and ldap_objectclass in your LDAP settings? Also, what database server (name and version, please) are you using? And what field are you mapping into ID Number user setting?

Saludos. Iñaki.
Average of ratings: -
In reply to Iñaki Arenaza

Re: LDAP Synchronisation

by Mike Cowley -

Thanks for your reply.

I have tried leaving the ldap_user_attribute and the ID Number field as blank and with sAMAccountName. I have also left the ldap_objectclass empty.

The version of MySQL is 5.0.22 and is set on localhost.

All of the above combinations cause the same error when the script is run.

Average of ratings: -
In reply to Mike Cowley

Re: LDAP Synchronisation

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Ok, let's add some debugging info to the syncing script to see what's going on.

You don't say which Moodle version you are using, so I'll assume it's 1.6.x. Edit file .../moodle/auth/lib.php and around line 438 (for 1.6.3+) you'll see the following lines of code. Add the ones in blue:
 foreach ($contexts as $context) {
 $context = trim($context);
 if (empty($context)) {
 continue;
 }

 echo "context: |" . var_dump($context) . "|\n";
 echo "filter: |" . var_dump($filter) . "|\n";
 echo "ldap_user_attribute: |" . var_dump($CFG->ldap_user_attribute) . "|\n";

 begin_sql();
 if ($CFG->ldap_search_sub) {
Next go to line 706 or so, and add the following code:
 // bulk insert -- superfast with $bulk_insert_records
 $sql = 'INSERT INTO '.$CFG->prefix.'extuser (idnumber) VALUES ';
 // make those values safe
 array_map('addslashes', $users);
 // join and quote the whole lot
 $sql = $sql . '(\'' . join('\'),(\'', $users) . '\')';
 print "+ " . count($users) . " users\n";
 $success = execute_sql($sql, false);
 echo "auth_ldap_bulk_insert (success): |" . var_dump ($success) . "|\n";
and tell us what you get when you run the syncing script from the command line.

Saludos. Iñaki.

Average of ratings: -
In reply to Iñaki Arenaza

Re: LDAP Synchronisation

by Mike Cowley -

Thank you for your suggestion.

Sorry I forgot to mention that we are using Moodle version 1.7. I made the changes to the script as you mentioned and now receive the following output when the script is run.

Configuring temp table
connecting to ldap
string(42) "ou=intake02,ou=pupil,dc=thematrix,dc=local"
context: ||
string(24) "(&(cn=*)(objectClass=*))"
filter: ||
string(2) "cn"
ldap_user_attribute: ||
+ 177 users
bool(false)
auth_ldap_bulk_insert (success): ||
Did not get any users from LDAP -- error? -- exiting

For testing purposes I have changed the LDAP settings so that only one of the non working OUs is scanned.

Average of ratings: -
In reply to Mike Cowley

Re: LDAP Synchronisation

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

bool(false) auth_ldap_bulk_insert (success): ||

Aha! Here we have it. Insertion of the temporary user data is failing. You should have a look at your MySQL logs to see why this kind insertion is failing. Otherwise you can add the following three lines:

global $db;
$old_db_debug = $db->debug;
$db->debug = true;

to function auth_ldap_bulk_insert() just before the:

execute_sql($sql, false);

line, and this one after the same line:

$db->debug = $old_db_debug;

to get similar error output.

Saludos. Iñaki.

Average of ratings: -
In reply to Iñaki Arenaza

Re: LDAP Synchronisation

by Steve Power -

Iñaki

Thank you this is also my problem. I have six contexts and two of them are failing to insert values according to your debug test above. I have added my own code and can see that the values are read from LDAP (AD) correctly and the MySQL logs also show the values aparently inserted with no indication of a problem. Interestingly one of the failures is the first 1000 INSERTs from an OU containing 2626 users.

How do I proceed from here? Is there something I have missed in setting MySQL logging which stops any indication of an error?

I have also added your debug code above but cannot see what it is intended to do. It made no differences in my trials.

Regards
Steve

Average of ratings: -
In reply to Iñaki Arenaza

Re: LDAP Synchronisation

by Steve Power -

Iñaki (and Mike too)

I just worked it out whilst answering your post in another thread. My problem is caused by two users Mary O'Brien and David o'Donnell who have never even logged in.

Bless AD but it allows the single quote in a sAMAccountName and so of course our tech people put it in. This then breaks the sql INSERT within the bulk_insert code and so ends up wiping out a whole OU of users.

I will give the resolution of this some thought but I will need to change both ldap_auth_sync.php and the ldap auth code too as I am updating idnumber from ldap on each login.

Regards
Steve

Average of ratings: -
In reply to Steve Power

Re: LDAP Synchronisation

by Iñaki Arenaza -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Steve, see my reply in the other thread. I think I've found the culprit smile

(just for reference, the other thread is here: http://moodle.org/mod/forum/discuss.php?d=59753)

Saludos. Iñaki.
Average of ratings: -
In reply to Iñaki Arenaza

Re: LDAP Synchronisation

by Steve Power -

Iñaki

Thank you for this.

Mike does this fix your problem?

Regards
Steve

Average of ratings: -