Future major features

 
 
Picture of Petr Škoda
Request for comments - new logging API
Group DevelopersGroup Documentation writersGroup Moodle HQGroup Particularly helpful Moodlers

Hello,

in the last few months the backend team was working on a specification and a small demonstration of a new logging API - see http://docs.moodle.org/dev/Logging_2. It is a continuation of the new events API introduced in Moodle 2.6.

If you are interested in this new API please comment here or in the MDL-41266, thanks.

 
Average of ratings: -
Picture of Tomasz Muras
Re: Request for comments - new logging API
Group DevelopersGroup Particularly helpful MoodlersGroup Translators

This has been already mentioned on the wiki document, to get some functionality Moodle will need to issue SQL query against the log data. This means that Moodle will have to know the exact structure of the logging implementation. Only the interface methods for getting logs may not be enough.

One way to deal with it, would be to keep a minimal core SQL DB log table forever. So an even would be created but for some logs they would go to mdl_log as well. This could be a minimal set of log events with a minimal amount of data - just to suffice for common functionality that uses log table at the moment.

Another way to deal with it, would be to create a set of interfaces for each functionality that we need from logs, eg:

interface login_information with log_login_errors method to cover http://docs.moodle.org/dev/Logging_usage#lib.2Fcronlib.php.

This way, when some functionality would be required (e.g. finding failed logins since), log mechanisms would be queried. The first one that implements appropriate interface would be used. If none, then the feature would not be available.

 

Another point, invoking get_events like this:


$reader->get_events($select, $params, 'timecreated DESC, id DESC', $page*$perpage, $perpage);

looks to me like every log plugin will *have to* expose/store data in SQL table, is that intentional?

Tomek

 
Average of ratings:Useful (1)
Picture of Petr Škoda
Re: Request for comments - new logging API
Group DevelopersGroup Documentation writersGroup Moodle HQGroup Particularly helpful Moodlers
The log-in related information should be IMHO stored in a separate table.

SQL queries are the only fast way to access data in our PHP code, so yes if storage plugin wants to make data available in standard reports it needs to expose it as a standardised table view.

The new design allows you to write add-ons at different levels:
* reports - using SQL interface to fetch or aggregate data
* storage plugins - read/write entry point for event data
* custom storage manager with custom readers and writers
* any custom logging solution based on simple event observer

The goal is to capture more structured data and to allow more customisations in all parts of the logging and reporting process. Do not expect any major changes in the standard distribution now because the first step is to cover the same functionality compared to 2.5. Do not expect performance improvements in standard installation either, however admins should have more ways to tune their servers to handle the increasing amounts of logged data and developers may create some novel solutions for specific use cases.
 
Average of ratings: -
Picture of Tomasz Muras
Re: Request for comments - new logging API
Group DevelopersGroup Particularly helpful MoodlersGroup Translators

Quote from wiki page

There are performance and scalability problems, such as:

  • log tables are joined in many popular queries and are slow;
  • all logs go to one standard moodle database table, which is not scalable;
  • it is not possible to configure level of logging at present; and
  • it is not possible to disable logging completely.

How is this proposed solution solving problem #1? By changing all current joins to event listeners, as suggested by Martin?

Database table can be scalable - it can be done on the DB level, eg by partitioning or using MySQL Proxy (for MySQL).

Problems #3 and #4 do not require complete rewrite - they can be easily implemented in the existing framework.

 

cheers,
Tomek

 
Average of ratings: -
Picture of Tomasz Muras
Re: Request for comments - new logging API
Group DevelopersGroup Particularly helpful MoodlersGroup Translators

Two more problems from wiki, "Existing problems" section:

  • Logging is added ad-hoc by developers and has spotty coverage.
  • Some logged actions do not contain important information needed later.

New framework won't help for #1 - logging will still need to be added ad-hoc by developers. To fix this, we would need to think about front-end controller that does the logging without having a need for developer to implement that explicitly.

The same goes for #2 above - we need more data passed not the new framework.

As for:

  • log tables are joined in many popular queries and are slow

This can easily be fixed without complete rewrite as well - it'd be enough to log as we do now and also trigger an event.

So - what are existing issues with logging at the moment? I don't think we need a complete rewrite and new architecture to solve issues listed in http://docs.moodle.org/dev/Logging_2#Existing_problems . By creating whole new logging layer we are:

  • reducing reliability (introducing new bugs, new layer, more components)
  • introducing more complexity (logging is dead simple now, does it have to be complex?)
  • decreasing performance (direct write to table  vs event -> observer/listener -> write to table)
  • creating more work just to catch up with the existing features
  • at the end of that phase we will have no added value (as there will be no other logging plugins)

I would think about what really is important and implement just that, for example:

  • events seem like a great idea, maybe just trigger an event after we do a normal write to DB? If anyone needs to consume logging event, then even creating local plugin may be OK, do we really need more interfaces? Look at messaging plugins for example - nobody is really using them and that has added more complexity into Moodle
  • structuring log data is again a great idea - we can do it already without complete rewrite (e.g. add more columns to mdl_log)
  • we need to tackle SQL joins on mdl_log no matter what - so let's do it now and then think about the rewrite. Maybe we should put some data (like logins) into a separate table, as suggested by Petr?

In general, I would suggest to first see what we need and then create a framework/layer for our needs, instead of doing the opposite - creating architecture that we may need in the future.

Tomek

 
Average of ratings: -
Picture of Petr Škoda
Re: Request for comments - new logging API
Group DevelopersGroup Documentation writersGroup Moodle HQGroup Particularly helpful Moodlers
1/ mdl_log is a huge table in existing installs - it would be technically very hard to tweak it, it would not be possible to migrate old data to new structure - I believe that the only way forward is to rely on new database table(s) in plugins

2/ The "new framework" fixes original events and logging at the same time - both of them now receive more and better data.

3/ The new events and logging includes full backwards compatibility, this would not be possible if we just altered events and logging.


Anyway the new events are already part of Moodle 2.6, I am afraid there is no way back. I expected developers would discuss the actual PHP interfaces now.
 
Average of ratings: -