Future major features

Pluggable Search Engines

 
This discussion has been locked because a year has elapsed since the last post. Please start a new discussion topic.
Picture of Phil Lello
Pluggable Search Engines
 
Hi all,

I'm considering patching Moodle to support pluggable search engines for searching content. This is largely inspired by customisations I've previously worked on using the GSOC Global Search plugin. I envisage two stock search plugins, a purely SQL based one that will run in a standard Moodle environment, and a SOLR plugin that will require a connection to a SOLR instance/master-slave setup.

To keep my life simple, and release code to the community in a reasonable timeframe, I'd like to incrementally patch different search functions in moodle core, starting with course descriptions, then moving on to other content at a later date. Would people find it useful/desirable to merge a partial implementation as a base for future development, or is the preference for a fully complete feature?

A key part of this implementation would be providing a standard mechanism to add custom metadata to indexed resources, to allow drill-down by these additional criteria in large catalogues, in a similar way to many e-commerce sites (e.g. e-Bay, Amazon, etc.)

Note that I'm likely to largely ignore the current SOLR global search plugin (possibly using it as a reference in some instances), as there are quite a few limitations I'd like to get away from - not least is the requirement for a PHP plugin when talking XML to a SOLR instance would work just as well.


Feedback desired,


Phil

 
Average of ratings: -
Picture of Daniel Neis Araujo
Re: Pluggable Search Engines
Group Core developersGroup Particularly helpful MoodlersGroup Plugin developersGroup Translators

Hello, Phil


nice to know that there is more people interested on this =)


I recently worked with MDL-31989 , also known as Global Search 2, using Solr and found it very useful.


Altough the code was a little bit old-style, i've managed to update it to work on 2.7 with not much work (github link on tracker issue) and also implemented a "more like this" feature for foruns, also known as MDL-46429 or "Implement "related forum discussions" list on top of Global Search 2" =)


I know that having to install a php library may be problematic for many people, but moodle already depends on other php libraries so the pretty-decent API of this library avoids a lot of work dealing with the XML and other APIs of Solr.


Please note that I am not here defending Solr as the best tool for the job, just addressing some points =)


I would also like to see http://sphinxsearch.com/ integrated on Global Search for Moodle but i had no time to work on this yet and i think that sphinx does not have the "more like this" feature out-of-the-box as solr have and I don't know about the term similarity algorithms sphinx uses.


If you don't mind,  I would like you to say a little about the "limitations you'd like to get away from".


Kind regards,

Daniel

 
Average of ratings: Useful (1)
Picture of Phil Lello
Re: Pluggable Search Engines
 

Hi Daniel,

Other than the need to build the SOLR plugin from source (for Ubuntu 12.04/14.04), the main limitation I found was not being able to set the endpoint for the SOLR instance - e.g. it would only work against /solr/search; since I was using faceted queries, I really wanted to be able to use a moreLikeThis endpoint where I could specify facets; unfortunately, there doesn't seem to be any way of doing this with the existing PHP module.

Other areas I'd like to improve on (which can be addressed in PHP) are the ability to send updates to a different host to queries - this would allow a load-balanced hostname providing node-failure protection for queries, with updates going the the master (I'm assuming a SOLR master-slave set-up).

The way I envisage a pluggable search engine API to work is to provide a base class/interface that different engines implement, and allow hooks for pre-query and post-query updates. This would allow, for example, facets to be layered on top of a search for engines the support faceted queries, and results to be filtered for engines that don't.

Phil

 
Average of ratings: -
Picture of Tomasz Muras
Re: Pluggable Search Engines
Group Core developersGroup Particularly helpful MoodlersGroup Plugin developersGroup Plugins guardiansGroup Translators
Hi Phil,

Unless I missed something you mention 2 limitations of current GS2 implementation that force you to do a rewrite:

1. "requirement for a PHP plugin"
2. "not being able to set the endpoint for the SOLR instance"

I think both can be resolved in the current code - you can replace functions used in solr request (point 1 above) with your own and surely PHP code can be updated to address point 2.

There was quite a lot of work already put into the GS2 implementation and the code from Daniel at https://github.com/danielneis/moodle/tree/MDL-31989_master seems to be the best starting place. I would definitely start there if I were to pick it up again.

Your plan is very ambitious (that's always good) - and original plans (including mine) for this plugin were pretty ambitious as well. Now I think that better way to approach that would be to start with something small (less features) but fully work-able - and then extend it.

cheers,
Tomek
 
Average of ratings: -
Picture of Николай Сурмин
Re: Pluggable Search Engines
 

Hi Guys! 

I am about to finish my study and working on my Master thesis, the topic of which is "To find and test the plugins for Global Search in Moodle. As far as understood this is a quite actual problem. I have already looked through a lot of information but have not found anyone actual plugin which I could use for my work.

Could you please be so kind and help me find such a plugins. All useful information will be highly appreciated. 

Sincerely, Nick.

 
Average of ratings: -
Picture of Tomasz Muras
Re: Pluggable Search Engines
Group Core developersGroup Particularly helpful MoodlersGroup Plugin developersGroup Plugins guardiansGroup Translators

Hi Nick,

Global Search itself is not yet implemented in Moodle, so it's going to be hard to find any plugins for it! smile


Tomek

 
Average of ratings: -