which looks reasonably good to me so far (please review it further)
It seems the next step is to write all these functions in a way so that all web services can access them (as per http://docs.moodle.org/en/Development:Web_services_API)
(and after that we need to work on /mnet/soap and /mnet/rest to provide interfaces to them).
Does anyone feel like tackling this? It might be a while until Moodle HQ can get onto it.
we (dfwikiteam) will get to programming this right away.
The additional stuff I'm working on will provide a security context within which these functions will be called, so that they can be executed with the privileges for a given user.
It seems like the functions outlined on the docs page:
...are being executed by an imaginary god-like user. Will that be a problem? Will that cause issues for logging and audit?
Reading the API, however, I think that all of those calls should be "switched to the plural" to make things more scalable. Each XML-RPC msg has a huge cost, if I'm going to add 10K enrolnments, or 40K users, this will fall apart, badly.
For example, "add user" should be "add users", taking an multidimensional array where each element of the outer array is a "user array" with the elements described in the wikipage. If the caller only has one user to add, they outer array will have just one element...
Pluralising the proposed functions is a good approach, but it might be better accomplished by wrapping these functions in the future. For example - add_user looks like it ought to be an atomic transaction that adds a single user, and succeeds or fails on the basis of the data supplied for that user.
Adding many users as a batch (in the way that you suggest) will make this simple function less elegant. As a subscriber to this API, I won't want to have to package my user data up as an array of arrays before I can send it to the function, and I don't want to have to unpack my result set from an array either. I just want to call the damn function already.
With that in mind, it would be possible to give the function a signature like:
create_user(username, firstname, lastname, etc.....)
Where the parameters can be either scalars (where you're adding a single user) or arrays of scalars (if you're adding a batch of users). if the consensus is that the functions should be pluralised, this signature might be preferable.
The result would have to also be polymorphous; I can see this causing serious hair-tearing at some point.
If MD's interested in eventually calling this work from core, I'd be keen to use something a bit more OO and maybe chained:
$user = new User();
I'm not 100% sold on chaining, but it's pretty.
OO code currently needs to be wrapped for export, which adds some indirection, but if the XML-RPC API gets to define how Moodle core looks, then the tail is very definitely wagging the dog.
What we are talking about are calls into Moodle internals - user creation, enrolment, etc, will happen under no user context. Just like they happen for current auth and enrolment plugins.
In terms of passing arrays for scalability and performance, 1.7 was elegant, dealing with every record as a glorious object. V1.9 deals in SQL and multidimensional arrays.
It's no joke, going forward, when we design an API we must design it scalable, even if it's not the prettiest (like heavy machinery ain't always pretty). Otherwise we'll need 2 APIs, one for "single-record-operation" and another one for bulk operations. When we can have just the bulk ones, and if people want to pass a single record to them, fine.
As a subscriber to this API, I won't want to have to package my user data up as an array of arrays
Why? There's no downside, just an extra pair of brackets in most languages, and in low-level terms, an extra struct and a pointer. It's trivial, specially since most modern languages support anonymous arrays:
// from $params = array(username => 'bla' password => 'foo'); // to $params = array(array(username => 'bla' password => 'foo')); /// sometime later... $x = xmlrpc_encode_request('adduser', $params) ;
On the server side, it's a foreach wrapping around the code you need to insert the record.
The alternative function signature you propose (an array per field) would force both callers and the server side to do a lot more unpacking. Instead of being record-oriented, you are making it column oriented -- if the caller has a CSV file or a database query, they'll have thing record-oriented. And the moodle end will want to arrange things in record-oriented object-ish arrays. If you draft the code to handle that you'll see that you'll need a lot more code.
If you're not doing that, there's no additional cost to using a wrapper function for batch creates.
Re: YAGNI; Is the batch-creation of users the use-case that this function ought to be optimised for?
If that is the case, is it important enough to saddle the rest of Moodle with a fugly API?
It's probably wise to defer optimisation of APIs and interfaces until we have some actual running code.
Couple of quick comments
I'm not thinking of bulk "in-line SQL" inserts (though they could be used for even better performance if the DB supports them).
The scalability I am talking about is on the cost of each XML-RPC. Even with precompilers enabled and all the tuning, I am fairly confident that over a reasonably low-latency LAN (your best case scenario) you are going to get around 2 orders of magnitude slower performance.
In other words, what I am proposing is ~100 times faster on a fast network. On a slow/high latency network, the difference will be even larger.
From the POV of someone who's written several backend integrations, YADGTNI (You Are Definitely Going To Need It). It is important that we build public APIs that are scalable.
What do you see as the downside? Something that would outweight 2-orders-of-magnitude higher costs?
...and I am?
I'll get on with writing some code.
Do you want to create new functions for this stuff in core, that will then be called by the web services wrappers, or do you want to re-implement this functionality outside of core with a new set of functions?
So - is core going to be touched by this, or is this going to be fresh new code that sits alongside (and to some extent duplicates) core?
Overall I think for long-term viability these really need to be implemented in core libraries, with the web services wrappers being really thin. Some of those "new functions" are going to be really similar to existing functions.
Without having a real close look yet I think we might have to deprecate a few existing functions (deprecatedlib.php) and update any calls to them. Have to look at each one in turn.
The bug for all this is here: MDL-12886
I wonder if it would be more advisable to have tightly coupled webservers (they are inside and internal to the application), re-writing functions means that support will have to happen on a per function layer, not only with the Web services implementation but as well with the core implementation.
If the Web Services are treating simply as an interface to the core functions, we then win in terms of maintenence, and sustatinability, if the function is changed in the core, then we maintain only one copy. Efficiency should happen at the core level.
This would enable a thin layer of Web Services, WS essentially becomes an interface as opposed to an API, the transactions should happen using the core and the data transformation at the interface.
I haven't yet taken a look at the OK Technologies WS implementation, I will check it out when I get home and take a look. We dont need to re-invent the wheel, we just need to change the tires
This then begs the question, should WS be implemented as a enrolment & auth plugin? What about grades via WS?....
Just some thoughts...
In OK Tech WS implementation (OKTWS) the high layer (mdl_soapserver.class.php) provides a list of operations in sync with the published WSDL and relies on a low layer (server.class.php) for the Moodle API communication.
In the low layer, we use API calls (get_record most of the time) and some direct SQL calls and currently we must trap all possible HTML error messages that might be emitted (using ob_start(), ob_get_content(), ob_end() ). Something which is not really clean...
The idea under the "Standard core functions for web services" (SCFFWS) will be to take care of all this messy part and return arrays of results or false
with some error messages in case of failure.
Furthermore the problem of rights to do such and such operation should be also addressed by SCFFWS using Moodle's roles to which core developers are more familiar than us.
To my opinion the performance penalty will be minor as compared with the overall cost of a WS operation (XML packing/unpacking, network ...)
Current revision 1.5.10 do retrieve grades as per Moodle 1.7 1.8. With the advent of the new gradebook in 1.9, all this part must be rewritten. Another argument for a SCFFWS to retrieve/set grades of a student identified by its id, regardless of the gradebook format.
The list of "required operations" may increase with the proposed fusion of the two SOAP based WS see http://tracker.moodle.org/browse/MDL-12886
Would it make sense to add a "reset_courses" function to the list of standard core functions?
One of the requirements in our moodle implementation is to reset all courses by category at the end of a term. Since we have over 500 courses, resetting these manually would have been an administrative challenge.
We use the Ok Tech WS to control enrolments and retrieve grades into our SIS and have modified some of their functions to work with collections of courses under a category. We added a simple reset_courses() web method and since it it was a good solution for our requirements, I thought it might be a good candidate to add to the list of standard webservices core functions...
Hello...new to this...but from the searching I've done...I don't this has been addressed.
I am keen to link Zoho Creator and/or Zoho CRM with Moodle...for real-time user creation and course enrollment.
Zoho Creator seems to deal with the REST web services protocol quite well...so that seems like the logical best place to start.
For those of you not familiar with Zoho Creator...it is a web-based application development environment using a scripting language called Deluge. It's part of the Zoho suite of office applications which, as best I can determine, is growing in popularity.
Love to hear any thoughts/feedback on this...or if anyone is already working on this...it would be great to hear.
I have been doing some research on this topic lately. You may want to take a look at this publication
The paper is quite theoretical, but it addresses some important topics that deserve to be taken into account when integrating Moodle with some third-party tool.
Not sure if you plan to integrate Zoho with Moodle-as-third-party, or Moodle with Zoho-as-third-party. It would be great to read any thoughts about this topic, and also some feedback about Zoho.
As far as I know, Moodle provides a quite good Web Services API, so it should not be difficult to consume those Web Services from the Zoho side. I guess that integration in this direction can be fruitful: Zoho orchestrating Moodle courses and users.
One other thing: have you considered to implement some kind of single-sign-on mechanism?
University of Vigo, Spain
Thanks for your reply. Your comments and paper are very interesting - good luck with your research work - I guess the end goal of this work is to enable a workflow controlled learning experience at university - nice idea.
My idea is Zoho with Moodle-as-third-party...a Zoho client would make REST calls to a Moodle 2.0 instance and receive a reponse with an XML 'file' which can be decoded within Zoho Creator.
I am not certain this may be possible, but I am interested in exploring it...it works for interactions between Zoho Creator and Zoho CRM.
One problem that I think is a major issue is that Zoho creator will only allow a GET, not a POST...and I gather that Moodle 2.0 only allows a REST POST....so I don't know how one gets around that.
Yes, that is correct Zoho orchestrating Moodle courses and users is the end-goal. We could make a solution available as a Zoho App.
As I said, I think Zoho is rapidly gaining popularity, so hooking it into Moodle would link two great forces of the universe
Regarding your comment about single-sign on. Obviously Zoho shares this with Google Apps (and Gmail too I think)...I presume it would be entirely up to the owners of Zoho, I presume, to set this up.
Also, if I did want to test this...does anyone know of a 'live' Moodle 2.0 instance that has WS enabled that I could try to link up with?