Potential problem with the logging API

Potential problem with the logging API

by Tim Hunt -
Number of replies: 6
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Last night I had a long and helpful discussion with with Mark Nelson about MDL-40063 (implementing the log API in the quiz) during which I learned a lot more about the log API than I previously knew.

In trying to work out the best way to do certain things, we came across a problem that has not been addressed yet, but I fear it is going to cause problems in Moodle 2.8.

The problem has not arisen yet because in Moodle 2.7, the scope was restricted to only cover converting existing add_to_log calls to the new API. There has not yet been an attempt to add in missing logging.

Another contributory factor to the problem is that, in the new log/events API, the recommendation is to raise the event relevant library function where the change is implemented, whereas in the past the rue was to log things in the outer-most UI layer. To give a specific example, in the past mod_quiz\delete_attempt used to be logged in the report class report.php, but now it should be logged in quiz_delete_attempt in locallib.php. There are many ways in which this is a good change. For example, it means that a quiz attempt being deleted is always logged in the same way. There are two ways in which it is bad:

  1. It can be bad for performance. It leasts easily to n+1 DB query performance problems.
  2. The point of logging is learning analytics. It is not necessarily enough to know that an attempt was deleted. You need to know who deleted it, and possibly you want some other context that might let you deduce why they deleted it.

This problem of context becomes more acute when you start thinking of core subsystems. The specific example we were grappling with was question_manually_graded. This processing happens in the question system, and following the design principles of the new log API, the event should be raised at the lowest level, which question_behaviour::process_comment. However, questions work the same wherever they are used, and the question API is well designed so extraneous information is not passed around. Hence, the necessary information to raise a semantically rich event is not available there.

When we spotted this problem, Mark and I went looking for other places where this problem might have arisen and been solved. The ideas that immediately came to mind were:

  • Ratings (e.g. you can rate this forum post)
  • Comment API
  • Question (we ought to be logging more interesting stuff that just teachers manually grading).

When we went to look, we found no add_to_log calls in either ratings or comments, and so no attempt has been made to solve these issues in 2.7.

I think it is clear that these are all educationally interesting events that you might want to log for learning analytics, so we will have to solve this problem of educationally interesting events that are both generic (provided by a core API) but also where context matters, and so we need to include it in the event.

One option is to change all the APIs just to pass in the extra information required to trigger the events. However, that seems like a bad design. The tail wagging the dog.

So, suggestions welcome.

Average of ratings: -
In reply to Tim Hunt

Re: Potential problem with the logging API

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Thinking about how to solve this problem, and also while writing the previous post, I have one suggestion that might lead to a solution.

The suggestion is that while doing the processing, the low-level API (questions/comments/ratings/etc.) only generates the necessary events. It does not actually trigger them. It puts them in some sort of event buffer. Then, once all processing is complete, the calling code can add the necessary context to the events, and trigger them all.

This ties in with another principle of the new events system: events should only be raised after processing is complete.

 

I suppose there is a similar solution where we just turn things around. Before calling the API that will raise events, the calling code make some sort of API call like ...->set_events_context(), and then that context is used to raise the events. (Argh! context is a bad word here. Moodle uses context to mean something else.)

 

However, in the case of questions, the first proposed solution works better. It would work very naturally for created events to be added to the question_engine_unit_of_work object, and then they are only triggered when the calling code calls question_engine::save_question_usage_by_activity(...) which calls question_engine_unit_of_work::save().

Both those methods could have an $eventscontext argument added, with default null, and then the change would be largely backwards compatible.

For getting events from the question behaviour to the question_engine_unit_of_work, we can add an add_event($event) method to question_attempt_pending_step so that the behaviour can raise events, and then the question_attempt class can move any added events to the unit of work.

Yay! I think that lets us trigger events without any breaking API change for questions.

 

Now we just need to know what events people doing learning analytics want, and what context they need in them.

Mind you, people have been successfully doing learning analytics on quizzes for years, without using logging. There is lots of high quality data stored in the question engine database tables, and there is hight quality analysis available in things like the quiz statistics report.

Average of ratings: -
In reply to Tim Hunt

Re: Potential problem with the logging API

by dawn alderson -

Hi Tim,

Understand most of the techy information here on a level that is probably not the same as there smile

In sum, it seems to me that decisions need to be made about solving the problem of where and when the infrastructure relating to the API is enacted.... or fired/triggered!  Not sure if my lingo fits....but it appears a generic formula is required.  I see that the soulution you offer, very briefly here-is to trigger events without breaking API change for questions.  I see.

In addition, I would like to pick up on context....I seem to share a meaning....with what was in the first post...in that, a generic approach appears necessary for core, but what about the different contexts in practice?  For example, I understand logging for learner analytics can include collecting data about outcomes...and latterly (practice-wise) on learning processes.  So, quantifiable, shareable and trackable API required for that-bear with me smile 

So, context-and the need to know what events people doing learning analytics want....is probably going to be that of a broad spectrum. 

In simple terms then, for those who need it that way (moi)  the adjustments in question, will enable the following:

To track quiz attempts, specific (generic) activity and outcomes.  This info will be shareable and quantifiable.  Is that it?  Because, this does not sound very different from the lines in your last para in the last post.  This leads me to think that this problem and solution outlined is about back-end issues only-am I wrong?

Many thanks

Dawn

 

 

 

 

 

 

 

 

 

     

Average of ratings: -
In reply to Tim Hunt

Re: Potential problem with the logging API

by Dan Poltawski -

Hi Tim,

I feel like a rubber duck :P

I agree this seems to be a problem which needs to be solved and actually, I already seen this problem come up even with what we have now.

However, in the case of questions, the first proposed solution works better.

I'm not actually clear on what your proposed solution was? smile

Average of ratings: -
In reply to Dan Poltawski

Re: Potential problem with the logging API

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

"I already seen this problem come up even with what we have now." - Do you have a URL (tracker issue id) with more details? Would be interesting to learn more, about the problem and any solution that was found.

I had two proposed solutions:

  1. Library code creats partial event objects, and either returns them somewhere, or stores them somewhere that the calling code can get at. Then the calling code addes the extra contextual information to the events and triggers them.
  2. The calling code provides the contextual information to the events system, to be used in all the events from the following API call(s), then makes the API call which triggers the events including the contextual information.

Then, for the question engine events case, I gave a more detailed sketch of how the first of those might be implemented with only backwards compatible API changes.

Average of ratings: -
In reply to Tim Hunt

Re: Potential problem with the logging API

by Anne Krijger -

Hi Tim,

A bit late to the party, but...

1. and 2. sound fairly similar to me.

If I understand correctly the outer layer sets up the transaction or event container,
the container can be accessed and enriched by the inner layers,
at any point the information in the container can be used to either provide some type of logging or trigger an event.

Is that what you are proposing?

A problem I see there is the sequentiality (is that a word? smile) of it.

If a container (context, event, whatever you want to call it) is created, it must be destroyed at a certain point.
But I guess this will be the case at the latest at the end of the http request in most cases, so may not be much of a problem.

Another issue would then be how the code in the middle (the lower layers) would know which context/container to retrieve to add their information to. But maybe I'm thinking too much about multthreading environments here and would this also not be an issue in PHP.

BTW I'm working under the assumption here that you would not want to pass an id or object along all the functions/methods that are executed.

Anne.

Average of ratings: -