Roles infrastructure questions - questions, performance, etc...

Roles infrastructure questions - questions, performance, etc...

by Martín Langhoff -
Number of replies: 23
Starting to really get into v1.8 now (lived in 1.5/1.6 land for quite a while, barely dipping my toes in 1.7) and I have some questions about roles infrastructure that I am needing help with.

They boil down to

- What is the public API and how do we use it efficiently/scalably?

- How do we denote things that are not "capabilities" but aspects of a role?

- How do we migrate to more granular capabilities in a sustainable way?

It may very well be that I need to RTFM or RTF Source -- I've been working a bit with the code already, and reading (and hacking) the source with a few people at Catalyst. Still, I make have missed some key concepts here and there... wink

There's more detail to my questions.. I'll open subthreads...
Average of ratings: -
In reply to Martín Langhoff

Re: Roles questions - Public API

by Martín Langhoff -
"What is the public API and how do we use it efficiently/scalably?"

Right now, this page is extremely light in API details (hopefully we can beef it up!)
http://docs.moodle.org/en/Development:Roles#Programming_Interface
it only talks about get_context_instance(), as_capability() and require_capability().

But that's only a subset of what we need... naturally there's more. We don't have good details of internal vs external calls in lib/accesslib.php -- . I ended up writing a findcallers.pl utility to figure this out. Accesslib is huge - it has 81 functions.

Looking at the output of findcallers.pl, I found these functions that I think form the rest of the API:

load_all_capabilities()
get_roles_with_capability()
role_assign() role_unassign()
enrol_into_course()
get_all_roles()
get_assignable_roles()
get_default_course_role()
get_users_by_capability()
get_role_users()
get_roles_on_exact_context()
get_user_capability_course()
role_switch()
user_has_role_assignment()
role_get_name()

So I guess my questions are...
- Is this list right? smile
- Which of those are the key calls, and could we have some "design" notes about them explaining how they are expected to be used?
- Which of those are costly (RAM, CPU, DB load, DB traffic), and how do we minimise their cost?

Pre 1.7, we had the user_students and user_teachers tables, and it was all simple and always cheap to check these things. With the flexibility and abstraction that roles brings, some of these operations are _really_ expensive.

In that sense, I want to work towards making roles much more efficient -- using less db queries, running the queries _outside_ of the loop and caching more smartly. For example, in the scenarios below:

- User looking at courses... this is probably the worst - we spend a huge effort asking if the user can see the course listing, and the teachers' names, and it's horribly costly to list the teachers' names.

- Participants page. Very costly.

- User in course page, or course resources/activities. I think this is reasonably cached already but the cache seems to be large in memory.

Phew. Long post. More to come.

Edit: attached output of findcallers.pl - the script itself is in cvs:/contrib/tools/devtools/findcallers.pl
Average of ratings: -
In reply to Martín Langhoff

Re: Roles questions - Public API

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
As far as I'm concerned, everything in accesslib.php is the API, even if a lot of it is not useful for most modules. Mostly they really will just need those three in the docs. I totally agree we need a lot more documentation about the other functions and how to use them.

There is an awful lot of caching already at many levels, much of it added over the past couple of months. The accesslib functions are complex and inter-related so if you do want to make any changes to them be verrrry careful.

General use of courses and blocks is pretty fast these days.

I'm sure there are still other particular situations that can be further optimised, we just need to know about them. About the course listing (teachers) loop I thought you were already fixing that?
Average of ratings: Useful (1)
In reply to Martin Dougiamas

Re: Roles questions - Public API

by Martín Langhoff -

As far as I'm concerned, everything in accesslib.php is the API, even if a lot of it is not useful for most modules.

I'd say that right now only 20% of the calls are ever used outsides of accesslib and admin/roles. I guess it's good to know it... those are the ones that are important to keep stable (in behaviour) and the ones to document... at least a bit! wink

There is an awful lot of caching already

Agreed - though I suspect it's not very efficient because it can't be shared across http processes -- at the most, you can stash it in the session. (And session growth has its own perf issues).

So I am considering putting some of that info in a shmem cache if one is available.

The accesslib functions are complex and inter-related so if you do want to make any changes to them be verrrry careful.

200%! my intention is to have an experimental branch in git to play with this stuff and show some patches and ideas. If they pan out, we'll put them in HEAD.

About the course listing (teachers) loop I thought you were already fixing that?

I got lost along the way, trying to understand all of this. sad (and of course a lot of other work!)

Average of ratings: -
In reply to Martín Langhoff

Re: Roles questions - Public API

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Can't you make findcallers.pl count the number of times each one in called. That would be a first attempt at making a priority list. Then instead of saying 'please document these 14 functinos' you may be able to say 'please document these few'.
Average of ratings: -
In reply to Tim Hunt

Re: Roles questions - Public API

by Martín Langhoff -
It only counts them in the source, for anything else I'd need a profiler, which takes just a tad more time to get going wink

But... click on the report attached to my earlier post and at the end of it you'll find the list with the counts to prioritise at the bottom of the list... instead of asking for doco for 81, I ask for doco for 14 but if someone can do at least a few. smile
Average of ratings: -
In reply to Martín Langhoff

Re: Roles questions - aspects vs capabilities

by Martín Langhoff -
"How do we denote things that are not "capabilities" but aspects of a role?"

One scenarios that we are facing right now: Create a course and a forum with forced subscription. Posts to that forum will go to the Moodle admin - the Moodle admin is auto-subscribed because we check for the course:view capability. But having course:view does not mean that admin is a participant in the course.

If we deal with it by creating new capabilities "forum: is_forcesubscribed" and "forum: is_autosubscribed", then I'm not sure we are in the clear... admin has "can_do_all" which will still say yes to that capability if I undestand correctly.

On the other hand, we _could_ limit the capability check to exclude CONTEXT_SYSTEM but I am not sure if assigning a sitewide "student" role will still work (maybe this is helped by the CONTEXT_SYSTEM vs CONTEXT_SITEWIDE split?). In any case, a sitewide "course creator" will see the same problem -- course creators want to have all sorts of rights within the course, but don't want to be spammed wink

Similarly, how do I ask who are the "gradeable" participants in this course? For example when showing the quiz report and wanting to show students (grade-able enrolled users) who have not taken the quiz.

As you can see, there's plenty I don't quite understand here. I think the CONTEXT_SYSTEM split that seems to have happened between 1.7 and 1.8 may help with this but I don't quite understand all its aspects and implications...
Average of ratings: -
In reply to Martín Langhoff

Re: Roles questions - aspects vs capabilities

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
"the Moodle admin is auto-subscribed because we check for the course:view capability."

Actually, the admin should not have course:view explicitly by default. I think admins DID have that at some stage but it was a bug that was fixed - admins are "above" needing to be participants (participants are simply defined as those who have explicit course:view on a course).

Whenever you use has_capability for something like that you should use the $doanything=false argument to exclude people who have the "doanything" capability.


Average of ratings: Useful (1)
In reply to Martin Dougiamas

Re: Roles questions - aspects vs capabilities

by Martín Langhoff -
Ah -- that clarifies things for admin. Does that save "course creators" from the same problem?

Are there any notes on the splitting up of CONTEXT_SYSTEM from the sitecourse? I remember Petr and you talking about it...
Average of ratings: -
In reply to Martín Langhoff

Re: Roles questions - aspects vs capabilities

by Petr Skoda -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
The problem with original sitecourse implementation was that it required many hardcoded hacks and was not flexible enough. The separation was mainly about removing of this special handling. Site course should be now nearly the same as the rest of courses with minor exceptions such as breadcrums.
Average of ratings: Useful (1)
In reply to Petr Skoda

Re: Roles questions - aspects vs capabilities

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Just to add to that, the interface to set roles/overrides in the "site course" context can be found under:

Admin > Frontpage > Frontpage roles
Average of ratings: Useful (1)
In reply to Martin Dougiamas

Re: Roles questions - aspects vs capabilities

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Instead of relying on $doanything=false, it is better to try to make capabilites that are a positive thing, that is, where $doanything for admins is the logical thing.

So for your example, I would make a capability mod/forum:ignoreforcesubscribe, that exempts peple from being forcibly subscribed.
Average of ratings: Useful (1)
In reply to Tim Hunt

Re: Roles questions - aspects vs capabilities

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Absolutely. $doanything is a big fat exception. Setting it to false is only when you want to test absolute membership in some 'group' of people as defined by a capability.
Average of ratings: Useful (1)
In reply to Martín Langhoff

Re: Roles questions - upgrade

by Martín Langhoff -
"How do we migrate to more granular capabilities in a sustainable way?"

The scenario I am wondering about goes a bit like this

- v1.8: admin overrides role foo to deny cap bar.
- v1.9: upgrade adds bar_z, which is more granular, does the upgrade process do the right thing and autocreate an override?

In the current code to define new caps I don't think we have a way to say "during upgrade, inherit what cap bar had".

Overall - this is a minor one wink
Average of ratings: -
In reply to Martín Langhoff

Re: Roles questions - upgrade

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Yes, it's the correct thing to do so that behaviour on a site doesn't suddenly change.

No, there's no function for this but we've done this several times in upgrades already (it's a few lines only).

A new function would be nice! (that'd make 82 big grin)
Average of ratings: Useful (1)
In reply to Martin Dougiamas

Re: Roles questions - upgrade

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
How do you do that? I thought it was impossible. Is there a good example to look at?
Average of ratings: -
In reply to Tim Hunt

Re: Roles questions - upgrade

by Jamie Pratt -
It would be very helpful if someone could point me to some code for upgrades where a capability has been split into more granular capabilities and the code gives everyone who have capability x capability x_y in all contexts that they had x. I need something like this for the new question code.
Average of ratings: -
In reply to Jamie Pratt

Re: Roles questions - upgrade

by Jamie Pratt -

Below is what I came up with. To migrate permissions in all contexts for :

'moodle/question:manage'
to :

'moodle/question:add', 'moodle/question:editmine', 'moodle/question:editall', 
 'moodle/question:viewmine', 'moodle/question:viewall', 'moodle/question:usemine', 
 'moodle/question:useall'

Any comments??

Jamie
 if ($result && $oldversion < 2007051901) {
 $role_capabilities = get_records('role_capabilities', 'capability', 'moodle/question:manage');
 foreach ($role_capabilities as $role_capability){
 unset($role_capability->id);
 $role_capability->timemodified = time();
 foreach (array('moodle/question:add', 'moodle/question:editmine', 'moodle/question:editall',
 'moodle/question:viewmine', 'moodle/question:viewall', 'moodle/question:usemine',
 'moodle/question:useall') as $capname){
 $role_capability->capability = $capname;
 //assign_capability will update rather than insert if capability exists
 $result = $result && assign_capability($capname, $role_capability->permission,
 $role_capability->roleid, $role_capability->contextid, true);
 }
 $result = $result && delete_records('role_capabilities', 'capability', 'moodle/question:manage');
 }
 }
Average of ratings: -
In reply to Jamie Pratt

Re: Roles questions - upgrade

by Jamie Pratt -
Here is a more succinct version of the code with some redundant bits removed and in an attachment to preserve indentation.
Average of ratings: -
In reply to Jamie Pratt

Re: Roles questions - upgrade

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
The only problem I can see is won't this cause problems if the definition of the new capabilites have not been loaded yet?
Average of ratings: -
In reply to Tim Hunt

Re: Roles questions - upgrade

by Jamie Pratt -
The definition of the new capabilities have not been loaded yet when the upgrade.php is executed but this doesn't seem to cause a problem. The new capabilities are loaded after upgrade.php
Average of ratings: -
In reply to Jamie Pratt

Re: Roles questions - upgrade

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Oh, right, I must have been thinking about the sanity checks in has_capability, which do give warnings in developer debug mode if you refer to a capability that does not exist.
Average of ratings: -