Following on from this list here:
http://docs.moodle.org/en/Development:Web_services
which looks reasonably good to me so far (please review it further)
It seems the next step is to write all these functions in a way so that all web services can access them (as per http://docs.moodle.org/en/Development:Web_services_API)
(and after that we need to work on /mnet/soap and /mnet/rest to provide interfaces to them).
Does anyone feel like tackling this? It might be a while until Moodle HQ can get onto it.
Standard core functions for web services
Number of replies: 18Re: Standard core functions for web services
Hi Martin,
we (dfwikiteam) will get to programming this right away.
Cheers
Ludo
we (dfwikiteam) will get to programming this right away.
Cheers
Ludo
Re: Standard core functions for web services
Well, we'll be continuing our work on the ones we started a while back (that do provide similar API's to the ones posted). We'll work with the way you want them to work, as defined in your links, so that we're all on the same page.
mike
mike
Re: Standard core functions for web services
Hey - I have some similar/related work to do for a client, so I'll be implementing some related functionality and I'll need to write some of these functions and then wrap them with the other code I need.
The additional stuff I'm working on will provide a security context within which these functions will be called, so that they can be executed with the privileges for a given user.
It seems like the functions outlined on the docs page:
http://docs.moodle.org/en/Development:Web_services
...are being executed by an imaginary god-like user. Will that be a problem? Will that cause issues for logging and audit?
The additional stuff I'm working on will provide a security context within which these functions will be called, so that they can be executed with the privileges for a given user.
It seems like the functions outlined on the docs page:
http://docs.moodle.org/en/Development:Web_services
...are being executed by an imaginary god-like user. Will that be a problem? Will that cause issues for logging and audit?
Re: Standard core functions for web services
I think it's a good idea to force an explicit IP whitelist and log the actions to a file. The god-like "user" is unavoidable 
Reading the API, however, I think that all of those calls should be "switched to the plural" to make things more scalable. Each XML-RPC msg has a huge cost, if I'm going to add 10K enrolnments, or 40K users, this will fall apart, badly.
For example, "add user" should be "add users", taking an multidimensional array where each element of the outer array is a "user array" with the elements described in the wikipage. If the caller only has one user to add, they outer array will have just one element...
Reading the API, however, I think that all of those calls should be "switched to the plural" to make things more scalable. Each XML-RPC msg has a huge cost, if I'm going to add 10K enrolnments, or 40K users, this will fall apart, badly.
For example, "add user" should be "add users", taking an multidimensional array where each element of the outer array is a "user array" with the elements described in the wikipage. If the caller only has one user to add, they outer array will have just one element...
Re: Standard core functions for web services
The god-like user is not unavoidable - what makes you think that?
Pluralising the proposed functions is a good approach, but it might be better accomplished by wrapping these functions in the future. For example - add_user looks like it ought to be an atomic transaction that adds a single user, and succeeds or fails on the basis of the data supplied for that user.
Adding many users as a batch (in the way that you suggest) will make this simple function less elegant. As a subscriber to this API, I won't want to have to package my user data up as an array of arrays before I can send it to the function, and I don't want to have to unpack my result set from an array either. I just want to call the damn function already.
With that in mind, it would be possible to give the function a signature like:
create_user(username, firstname, lastname, etc.....)
Where the parameters can be either scalars (where you're adding a single user) or arrays of scalars (if you're adding a batch of users). if the consensus is that the functions should be pluralised, this signature might be preferable.
The result would have to also be polymorphous; I can see this causing serious hair-tearing at some point.
If MD's interested in eventually calling this work from core, I'd be keen to use something a bit more OO and maybe chained:
$user = new User();
$user->userName('donal')
->firstName('Donal')
->lastName('McMullan')
->commit();
I'm not 100% sold on chaining, but it's pretty.
OO code currently needs to be wrapped for export, which adds some indirection, but if the XML-RPC API gets to define how Moodle core looks, then the tail is very definitely wagging the dog.
Pluralising the proposed functions is a good approach, but it might be better accomplished by wrapping these functions in the future. For example - add_user looks like it ought to be an atomic transaction that adds a single user, and succeeds or fails on the basis of the data supplied for that user.
Adding many users as a batch (in the way that you suggest) will make this simple function less elegant. As a subscriber to this API, I won't want to have to package my user data up as an array of arrays before I can send it to the function, and I don't want to have to unpack my result set from an array either. I just want to call the damn function already.
With that in mind, it would be possible to give the function a signature like:
create_user(username, firstname, lastname, etc.....)
Where the parameters can be either scalars (where you're adding a single user) or arrays of scalars (if you're adding a batch of users). if the consensus is that the functions should be pluralised, this signature might be preferable.
The result would have to also be polymorphous; I can see this causing serious hair-tearing at some point.
If MD's interested in eventually calling this work from core, I'd be keen to use something a bit more OO and maybe chained:
$user = new User();
$user->userName('donal')
->firstName('Donal')
->lastName('McMullan')
->commit();
I'm not 100% sold on chaining, but it's pretty.
OO code currently needs to be wrapped for export, which adds some indirection, but if the XML-RPC API gets to define how Moodle core looks, then the tail is very definitely wagging the dog.
Re: Standard core functions for web services
What we are talking about are calls into Moodle internals - user creation, enrolment, etc, will happen under no user context. Just like they happen for current auth and enrolment plugins.
In terms of passing arrays for scalability and performance, 1.7 was elegant, dealing with every record as a glorious object. V1.9 deals in SQL and multidimensional arrays.
It's no joke, going forward, when we design an API we must design it scalable, even if it's not the prettiest (like heavy machinery ain't always pretty). Otherwise we'll need 2 APIs, one for "single-record-operation" and another one for bulk operations. When we can have just the bulk ones, and if people want to pass a single record to them, fine.
As a subscriber to this API, I won't want to have to package my user data up as an array of arrays
Why? There's no downside, just an extra pair of brackets in most languages, and in low-level terms, an extra struct and a pointer. It's trivial, specially since most modern languages support anonymous arrays:
// from
$params = array(username => 'bla' password => 'foo');
// to
$params = array(array(username => 'bla' password => 'foo'));
/// sometime later...
$x = xmlrpc_encode_request('adduser', $params) ;
On the server side, it's a foreach wrapping around the code you need to insert the record.
The alternative function signature you propose (an array per field) would force both callers and the server side to do a lot more unpacking. Instead of being record-oriented, you are making it column oriented -- if the caller has a CSV file or a database query, they'll have thing record-oriented. And the moodle end will want to arrange things in record-oriented object-ish arrays. If you draft the code to handle that you'll see that you'll need a lot more code.
Re: Standard core functions for web services
Re: scalability; it sounds like you're proposing that the 'add user' function would create a large query which inserts tens of thousands of new user records in one hit. I'd be happier about batch inserting records if we were using placeholders, etc. but with things as they are, I'm not sure that's a great approach.
If you're not doing that, there's no additional cost to using a wrapper function for batch creates.
Re: YAGNI; Is the batch-creation of users the use-case that this function ought to be optimised for?
If that is the case, is it important enough to saddle the rest of Moodle with a fugly API?
It's probably wise to defer optimisation of APIs and interfaces until we have some actual running code.
If you're not doing that, there's no additional cost to using a wrapper function for batch creates.
Re: YAGNI; Is the batch-creation of users the use-case that this function ought to be optimised for?
If that is the case, is it important enough to saddle the rest of Moodle with a fugly API?
It's probably wise to defer optimisation of APIs and interfaces until we have some actual running code.
Re: Standard core functions for web services
Couple of quick comments
I'm not thinking of bulk "in-line SQL" inserts (though they could be used for even better performance if the DB supports them).
The scalability I am talking about is on the cost of each XML-RPC. Even with precompilers enabled and all the tuning, I am fairly confident that over a reasonably low-latency LAN (your best case scenario) you are going to get around 2 orders of magnitude slower performance.
In other words, what I am proposing is ~100 times faster on a fast network. On a slow/high latency network, the difference will be even larger.
From the POV of someone who's written several backend integrations, YADGTNI (You Are Definitely Going To Need It). It is important that we build public APIs that are scalable.
What do you see as the downside? Something that would outweight 2-orders-of-magnitude higher costs?
Re: Standard core functions for web services
"From the POV of someone who's written several backend integrations"
...and I am?
I'll get on with writing some code.
...and I am?
I'll get on with writing some code.
Re: Standard core functions for web services
MD: A question about these features:
http://docs.moodle.org/en/Development:Web_services
Do you want to create new functions for this stuff in core, that will then be called by the web services wrappers, or do you want to re-implement this functionality outside of core with a new set of functions?
So - is core going to be touched by this, or is this going to be fresh new code that sits alongside (and to some extent duplicates) core?
http://docs.moodle.org/en/Development:Web_services
Do you want to create new functions for this stuff in core, that will then be called by the web services wrappers, or do you want to re-implement this functionality outside of core with a new set of functions?
So - is core going to be touched by this, or is this going to be fresh new code that sits alongside (and to some extent duplicates) core?
Re: Standard core functions for web services
Sorry, just saw this. Good question.
Overall I think for long-term viability these really need to be implemented in core libraries, with the web services wrappers being really thin. Some of those "new functions" are going to be really similar to existing functions.
Without having a real close look yet I think we might have to deprecate a few existing functions (deprecatedlib.php) and update any calls to them. Have to look at each one in turn.
The bug for all this is here: MDL-12886
Overall I think for long-term viability these really need to be implemented in core libraries, with the web services wrappers being really thin. Some of those "new functions" are going to be really similar to existing functions.
Without having a real close look yet I think we might have to deprecate a few existing functions (deprecatedlib.php) and update any calls to them. Have to look at each one in turn.
The bug for all this is here: MDL-12886
Re: Standard core functions for web services
" Some of those "new functions" are going to be really similar to existing functions."
I wonder if it would be more advisable to have tightly coupled webservers (they are inside and internal to the application), re-writing functions means that support will have to happen on a per function layer, not only with the Web services implementation but as well with the core implementation.
If the Web Services are treating simply as an interface to the core functions, we then win in terms of maintenence, and sustatinability, if the function is changed in the core, then we maintain only one copy. Efficiency should happen at the core level.
This would enable a thin layer of Web Services, WS essentially becomes an interface as opposed to an API, the transactions should happen using the core and the data transformation at the interface.
I haven't yet taken a look at the OK Technologies WS implementation, I will check it out when I get home and take a look. We dont need to re-invent the wheel, we just need to change the tires
This then begs the question, should WS be implemented as a enrolment & auth plugin? What about grades via WS?....
Just some thoughts...
I wonder if it would be more advisable to have tightly coupled webservers (they are inside and internal to the application), re-writing functions means that support will have to happen on a per function layer, not only with the Web services implementation but as well with the core implementation.
If the Web Services are treating simply as an interface to the core functions, we then win in terms of maintenence, and sustatinability, if the function is changed in the core, then we maintain only one copy. Efficiency should happen at the core level.
This would enable a thin layer of Web Services, WS essentially becomes an interface as opposed to an API, the transactions should happen using the core and the data transformation at the interface.
I haven't yet taken a look at the OK Technologies WS implementation, I will check it out when I get home and take a look. We dont need to re-invent the wheel, we just need to change the tires
This then begs the question, should WS be implemented as a enrolment & auth plugin? What about grades via WS?....
Just some thoughts...
Re: Standard core functions for web services
Hello,
In OK Tech WS implementation (OKTWS) the high layer (mdl_soapserver.class.php) provides a list of operations in sync with the published WSDL and relies on a low layer (server.class.php) for the Moodle API communication.
In the low layer, we use API calls (get_record most of the time) and some direct SQL calls and currently we must trap all possible HTML error messages that might be emitted (using ob_start(), ob_get_content(), ob_end() ). Something which is not really clean...
The idea under the "Standard core functions for web services" (SCFFWS) will be to take care of all this messy part and return arrays of results or false
with some error messages in case of failure.
Furthermore the problem of rights to do such and such operation should be also addressed by SCFFWS using Moodle's roles to which core developers are more familiar than us.
To my opinion the performance penalty will be minor as compared with the overall cost of a WS operation (XML packing/unpacking, network ...)
Current revision 1.5.10 do retrieve grades as per Moodle 1.7 1.8. With the advent of the new gradebook in 1.9, all this part must be rewritten. Another argument for a SCFFWS to retrieve/set grades of a student identified by its id, regardless of the gradebook format.
The list of "required operations" may increase with the proposed fusion of the two SOAP based WS see http://tracker.moodle.org/browse/MDL-12886
Cheers
In OK Tech WS implementation (OKTWS) the high layer (mdl_soapserver.class.php) provides a list of operations in sync with the published WSDL and relies on a low layer (server.class.php) for the Moodle API communication.
In the low layer, we use API calls (get_record most of the time) and some direct SQL calls and currently we must trap all possible HTML error messages that might be emitted (using ob_start(), ob_get_content(), ob_end() ). Something which is not really clean...
The idea under the "Standard core functions for web services" (SCFFWS) will be to take care of all this messy part and return arrays of results or false
with some error messages in case of failure.
Furthermore the problem of rights to do such and such operation should be also addressed by SCFFWS using Moodle's roles to which core developers are more familiar than us.
To my opinion the performance penalty will be minor as compared with the overall cost of a WS operation (XML packing/unpacking, network ...)
Current revision 1.5.10 do retrieve grades as per Moodle 1.7 1.8. With the advent of the new gradebook in 1.9, all this part must be rewritten. Another argument for a SCFFWS to retrieve/set grades of a student identified by its id, regardless of the gradebook format.
The list of "required operations" may increase with the proposed fusion of the two SOAP based WS see http://tracker.moodle.org/browse/MDL-12886
Cheers
Re: Standard core functions for web services
Would it make sense to add a "reset_courses" function to the list of standard core functions?
One of the requirements in our moodle implementation is to reset all courses by category at the end of a term. Since we have over 500 courses, resetting these manually would have been an administrative challenge.
We use the Ok Tech WS to control enrolments and retrieve grades into our SIS and have modified some of their functions to work with collections of courses under a category. We added a simple reset_courses() web method and since it it was a good solution for our requirements, I thought it might be a good candidate to add to the list of standard webservices core functions...
Re: Standard core functions for web services
looks like a good idea, please add it as acomment in the tracker.
best
best
Zoho Creator and CRM - usage of ...Re: Standard core functions for web services
Hello...new to this...but from the searching I've done...I don't this has been addressed.
I am keen to link Zoho Creator and/or Zoho CRM with Moodle...for real-time user creation and course enrollment.
Zoho Creator seems to deal with the REST web services protocol quite well...so that seems like the logical best place to start.
For those of you not familiar with Zoho Creator...it is a web-based application development environment using a scripting language called Deluge. It's part of the Zoho suite of office applications which, as best I can determine, is growing in popularity.
Love to hear any thoughts/feedback on this...or if anyone is already working on this...it would be great to hear.
Regards,
Liam O'Duibhir
SoccerFit (Australia)
Re: Zoho Creator and CRM - usage of ...Re: Standard core functions for web services
Hi Liam
I have been doing some research on this topic lately. You may want to take a look at this publication
http://fie-conference.org/fie2010/papers/1223.pdf
The paper is quite theoretical, but it addresses some important topics that deserve to be taken into account when integrating Moodle with some third-party tool.
Not sure if you plan to integrate Zoho with Moodle-as-third-party, or Moodle with Zoho-as-third-party. It would be great to read any thoughts about this topic, and also some feedback about Zoho.
As far as I know, Moodle provides a quite good Web Services API, so it should not be difficult to consume those Web Services from the Zoho side. I guess that integration in this direction can be fruitful: Zoho orchestrating Moodle courses and users.
One other thing: have you considered to implement some kind of single-sign-on mechanism?
Regards,
Roberto Perez-Rodriguez
University of Vigo, Spain
Re: Zoho Creator and CRM - usage of ...Re: Standard core functions for web services
Hello Roberto,
Thanks for your reply. Your comments and paper are very interesting - good luck with your research work - I guess the end goal of this work is to enable a workflow controlled learning experience at university - nice idea.
My idea is Zoho with Moodle-as-third-party...a Zoho client would make REST calls to a Moodle 2.0 instance and receive a reponse with an XML 'file' which can be decoded within Zoho Creator.
I am not certain this may be possible, but I am interested in exploring it...it works for interactions between Zoho Creator and Zoho CRM.
One problem that I think is a major issue is that Zoho creator will only allow a GET, not a POST...and I gather that Moodle 2.0 only allows a REST POST....so I don't know how one gets around that.
Yes, that is correct Zoho orchestrating Moodle courses and users is the end-goal. We could make a solution available as a Zoho App.
As I said, I think Zoho is rapidly gaining popularity, so hooking it into Moodle would link two great forces of the universe
Regarding your comment about single-sign on. Obviously Zoho shares this with Google Apps (and Gmail too I think)...I presume it would be entirely up to the owners of Zoho, I presume, to set this up.
Also, if I did want to test this...does anyone know of a 'live' Moodle 2.0 instance that has WS enabled that I could try to link up with?
Thanks again,
Liam.