Writing Moodle's Global Search

Writing Moodle's Global Search

by Prateek Sachan -
Number of replies: 65

Hi, I'm Prateek and planning to write Moodle's Global Search.

Earlier, it was written by Tomasz Muras. 
The previous code is here : https://github.com/prateeksachan/moodle/tree/gs (forked from Tomasz's code). The code shouldn't be re-used and used only as a reference. 

The official docs on Global Search are here. 

The code of Search APIs for modules here could be used in the new implementation.

These are the current search features needed to be implemented. 

I wanted to invite suggstions and ideas regarding any other features that need to be included in Global Search's version 1.0. I would also like to have ideas about the UI of search results shown to the user (as this will be an important part).

For this version, we plan to integrate Solr. 

My motive is to build Global Search and the core modules supporting it.

Suggestions are welcome.

Thanks.

Average of ratings: -
In reply to Prateek Sachan

Re: Writing Moodle's Global Search

by Joseph Thibault -
Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Having the ability to perform advanced searches would be great in my opinion (https://moodle.org/mod/forum/search.php?id=5) just an example.

Average of ratings: -
In reply to Joseph Thibault

Re: Writing Moodle's Global Search

by Prateek Sachan -

Yes!
Currently, you can search within forums. Imagine, if you have to find important documents within some course's modules (like lessons, pages, etc.).

Global Search will prove very handy in such situations.

Average of ratings: -
In reply to Prateek Sachan

Re: Writing Moodle's Global Search

by gaurav singh -
Dear Prateek,

Pls solve my my problem.
I have 2 courses category named Jan-June 2015 and Aug-Dec 2015.
On clicking one category,it shows the result from both categories.
Please help me out what i have to do. My moodle version is Moodle 2.8.3
I am new in moodle,kindly help me.
Thanks
Gaurav
Project Associate
IIT Mandi

Average of ratings: -
In reply to Prateek Sachan

Re: Writing Moodle's Global Search

by Stuart Lamour -
Picture of Plugin developers

Hi Prateek,

the most important thing is to define your requirements specification based on interviews/research with actual students.

otherwise its all just speculation on what might useful or nice to have.

get a clear scope from real users - not like the people on moodle.org - and create a very clear mvp spec on that.

this way you can avoid scope creep or nice to have, but no one actually uses.

jessse james garret has a useful diagram on this https://wiki.bath.ac.uk/download/attachments/34409648/elementsofuserexperience.jpg?version=1&modificationDate=1285603076000

and i cannot recommend http://www.amazon.co.uk/Designing-Digital-Age-Human-Centered-Products/dp/0470229101 enough for anyone about to embark on such a task!

good luck Prateek- looking forward to seeing your user research and resulting requirements spec for this smile 

 

Average of ratings: -
In reply to Stuart Lamour

Re: Writing Moodle's Global Search

by Prateek Sachan -

Currently, I prepared two simple templates regarding the Search's UI.

The pop-up will instruct users about "how to use search features" (similar to editing one on issue page. ).

The results could be displayed in the following manner (its just a simple-basic template):

  

For the first version, I just want to create the minimal things required to get the stuff started.

Thanks.

Average of ratings: -
In reply to Prateek Sachan

Re: Writing Moodle's Global Search

by Ralf Hilgenstock -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Translators

Some aspects while thinking about in for five minutes:

- where is the search located: in a course, on mymoodle page?
- where does it search?
- does it reflect on group memberschip?- does it reflect conditional activities?
- what happens with file attachment content

Search is mostly asked for file content and folder content.

What does the result show? the search text in its content 10 words before an after? What happens if a search term is used very often in a lesson or book on different pages, what shows the results page

What should the search show from a quiz before the student made an attempt, after the attempt?  (description text only,  question text, answering options,  feedback text, essay question student answer, teachers feedback for essay question). To keep it simple I would suggest to limit it here on description text.

 

Average of ratings: -
In reply to Ralf Hilgenstock

Re: Writing Moodle's Global Search

by Prateek Sachan -

Hi Ralf,

To answer your queries:

where is the search located: in a course, on mymoodle page?
Search will be more like a plugin. It will be in your moodle site - integrated inside modules.

where does it search?
It will search for keywords based on your options. You may choose whether to include data from quizzes or forums or any other module in your search.
 
does it reflect on group memberschip?- does it reflect conditional activities?
Yes, it will reflect group memberships. As an example, a student from group A will not be able to view search results that are a part of assignment/quiz submission from group B. (However, the teacher will be able to see them). It will handle proper security. 

what happens with file attachment content
Keywords will also be searched inside HTML, PPT or PDF files. If a user has access (or proper "capability"), then only the results will be shown along with the link of the file attached. 

What happens if a search term is used very often in a lesson or book on different pages, what shows the results page?
The results will be based on "relevance weightage". For example:
Case I:
Lesson A contains 1 occurances of "batman superman" (as a phrase..both words are together)
Lesson B contains 10 occurances of batman and 6 occurances of superman.

If a user searches for "batman superman" Lesson B wil have higher priority (or relevance) as comparerd to Lesson A

Case II:
Lesson A contains 10 occurances of "batman superman" (as a phrase..both words are together)
Lesson B contains 10 occurances of batman and 6 occurances of superman.

If a user searches for "batman superman" Lesson A wil have higher priority (or relevance) as comparerd to Lesson B

What should the search show from a quiz before the student made an attempt, after the attempt? 
Search will look for "keywords". For example,
One the quiz questions is "Define thermodynamics".
If the student searches for "thermodynamics" before makin the attempt, search will return the content from the quiz question.

If he makes an attempt (and answers the question: "Thermodynamics is a branch of natural science concerned with heat and its relation to energy and work".), and then searches, the search willl then include both the attempt and quiz questions. (The answer may be nested inside the question itself!).


Thanks for the feedback. Would love to hear if you have any ideas. In the meantime, go through this page listing the features of Global Search: 
http://docs.moodle.org/dev/Global_Search#Requirements_and_assumptions

Average of ratings: -
In reply to Prateek Sachan

Re: Writing Moodle's Global Search

by Amir Elion -

Hi Prateek,

this is very welcome. Hope it becomes fruitful soon.

I would suggest considering more capabilities to control which roles see which results. E.g. You could add a view search module A results capability to role 1 and not to role 2. This would allow several levels of search to different types of users.

Would also highly stress the usability option for refining search or search within results.

Obviously very effective UI ideas and features can be learned from Google search - e.g. Tabs to allow switching between different types of results (text, images, news etc in google, perhaps All, Content, Users, Courses, etc. in Moodle). Also ability to filter results quick according to latest update or creation (within 30 days, 3 months, year). 

Average of ratings: -
In reply to Amir Elion

Re: Writing Moodle's Global Search

by Prateek Sachan -

Hi Amir,

Thanks for the feedback. For the first version I will be trying to concentrate on including only important search features, just to get it started. Once it has been done, more and more features will be added with time.

UI will be a very important critical feature : the very reason why I posted about it here as I wanted ideas from the community. Yes, Google Search is really very good in terms of offering the user elegant search UI. Will definitely takes ideas from it.

Thanks.

Average of ratings: -
In reply to Prateek Sachan

Re: Writing Moodle's Global Search

by Ralf Hilgenstock -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Translators

Hello Prateek

thanks for your detailled answer. Just one feedback:

If search is possible for quiz question content should be defined by teacher and not by student. The teacher should be able to define that student could not serch in quiz question content.

 

ralf

Average of ratings: -
In reply to Ralf Hilgenstock

Re: Writing Moodle's Global Search

by Prateek Sachan -

Hi Ralf,

Thanks for the concern and feedback. I will try my best to handle security with the Global Search. Proper security will be handled while showing the results. (Like students couldn't view other students' grades/answers etc. in addition to the situation you mentioned).

We don't want people to get access to documents that aren't meant for them. That is one of the most important aspects of Global Search. 

Thanks.

Average of ratings: -
In reply to Prateek Sachan

Re: Writing Moodle's Global Search

by Luis de Vasconcelos -

> what happens with file attachment content
> Keywords will also be searched inside HTML, PPT or PDF files.

Will other "Office" file formats like Word documents (.DOC/.DOCX) and Excel spreadsheets (.XLS/.XLSX) be included?

Average of ratings: -
In reply to Luis de Vasconcelos

Re: Writing Moodle's Global Search

by Tomasz Muras -
Picture of Core developers Picture of Plugin developers Picture of Plugins guardians Picture of Translators

The plan is to use solr for full text extraction, which in turn uses Apache Tika. See list of supported formats.

cheers,
Tomek

Average of ratings: Useful (2)
In reply to Tomasz Muras

Re: Writing Moodle's Global Search

by Luis de Vasconcelos -

So there is a big dependency on several Apache "extensions" (Tika, Solr...) How does this affect Moodle sites that are hosted on non-Apache web servers like IIS?

Average of ratings: -
In reply to Luis de Vasconcelos

Re: Writing Moodle's Global Search

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

It's not any Apache web server dependency that concerns me it is the server side Java dependency. It takes Moodle from generic LAMP software that can run on affordable web space to something rather specialised.

I have worked extensively with Java and server side java and I know of what I speak.

 

Average of ratings: -
In reply to Marcus Green

Re: Writing Moodle's Global Search

by Tomasz Muras -
Picture of Core developers Picture of Plugin developers Picture of Plugins guardians Picture of Translators

That's correct Marcus and we should be very clear here. Global Search will be implemented in a way to allow for plugging other search engines than solr but at this moment only solr support will be implemented.

To make long story short (I've spent quite a bit of time on this): there is no search engine written in PHP, so it's either Java or nothing. I doubt anyone will ever implement support for anything else than solr.

We will depend on solr features as well. For example text extraction from documents is provided by solr, so we will not be implementing our own text extraction.

Because of the above, Global Search will not work with Moodle out of the box. To get in setup you will need to get Java hosting, setup solr, add PHP extension that talks to solr, etc. We will of course produce installation instructions.

Hope that helps,
Tomek

Average of ratings: -
In reply to Tomasz Muras

Re: Writing Moodle's Global Search

by Aparup Banerjee -

Having one search engine like Solr to reference while building GS is really useful but as Tomek has mentioned, the plan is to have it pluggable.

It would be useful to have some other search engine backends/docs to match up with the developing API to cover off any holes, but failing that, i believe this won't be the case with Solr as the backend basis.

 

 

Average of ratings: -
In reply to Tomasz Muras

Re: Writing Moodle's Global Search

by Luis de Vasconcelos -

And Global Search will only work on Apache web servers, so other web servers like IIS are excluded and won't be supported?

Average of ratings: -
In reply to Luis de Vasconcelos

Re: Writing Moodle's Global Search

by Hubert Chathi -

As I wrote in my reply to your previous comment, Tika and Solr are standalone products that are unrelated to the Apache web server, other than the fact that both are managed by the Apache Foundation.  There should be no problems running it on other servers, as long as it supports Java.

Average of ratings: -
In reply to Hubert Chathi

Re: Writing Moodle's Global Search

by Luis de Vasconcelos -
According to Prateek, Global Search will initially only work on Apache: http://docs.moodle.org/dev/Talk:Global_search_(GSoC2013)
Average of ratings: -
In reply to Luis de Vasconcelos

Re: Writing Moodle's Global Search

by Prateek Sachan -

There was a slight confusion regarding the apache solr readme. I confirmed with those people. So, the module could be compiled w/o any dependencies on the used webserver.

Sorry, for the slight mess up here.

I will be listing down the dependencies over here (which is not yet final and will be updated): http://docs.moodle.org/dev/Global_search_%28GSoC2013%29#Installation

Average of ratings: -
In reply to Luis de Vasconcelos

Re: Writing Moodle's Global Search

by Hubert Chathi -

Tika and Solr are not Apache extensions.  They are standalone products that are maintained by the Apache foundation, but otherwise are unrelated to the Apache web server.

Average of ratings: Useful (1)
In reply to Tomasz Muras

Re: Writing Moodle's Global Search

by Alex Walker -

I'm glad to hear Moodle will use Apache Solr.

I've just implemented a Solr server at my institution. Although it has a steep learning curve (and it's Java!), it is fantastic.

The code I use to submit my Moodle courses to Solr (and Mahara user profiles, and a few custom in-house applications) is really simple, but if anyone wants it, they're welcome to it.

Average of ratings: -
In reply to Prateek Sachan

Re: Writing Moodle's Global Search

by Ankit Agarwal -

Hey Prateek,
First, welcome to the Moodle community and good luck with the selection process for Gsoc.

Some suggestions/things to consider:-

# Ranking.  Imho, ranking is one of the very important factor for this project. Ranking just based on occurence , might not give the most desired result. Google uses something called as Page rank concept (along with dozens of other factor), not sure if we can build something similar for us or not, but it surely is something worth looking into. For example the content from a book is more important than the content of a log entry, irrespective of the occurences.

# Security. Security will be a big concern while filtering the results, am pretty sure you already have some proposal on how to handle it. Just make sure it doesn't get too expensive.

# Events. You might want to look into the new event system proposed(work in progress) for Moodle 2.6. Might help you with indexing and other stuff. (http://docs.moodle.org/dev/Event_2)

# Not searchable content. - just an idea, that there would be a way to mark some of the content as not searchable (mimicking the robots.txt for normal search engines).

Good work so far,

Thanks

Average of ratings: -
In reply to Ankit Agarwal

Re: Writing Moodle's Global Search

by Prateek Sachan -

Hi Ankit,

Thanks a lot for the constructive feedback. 

#Ranking: Thanks for this thought, I will definitely consider it and discuss it with my mentor if/when my GSoC project gets approved. I would try my best to implement the Page Rank concept and other factors (as you outlined) as far as possible.
However, I guess that on a general note, all modules would have the same priority. Log entries would however, have a lower priority than the modules. 

#Security: It is a big concern. I will try my best not to compromise even a little in terms of security, because then the Global Search would still be undone. Currently, the manner in which I plan to handle it wouldn't have any expenses regarding security as I will be treating search modularily. But then, we will be 100% sure only after the testing.

#Events : As I've planned to to build the Global Search on Moodle 2.5 (closest to master), and the plan is to make it available in Moodle 2.6, hence I will keep this into account.

#Not Searchable Content: Currently, only the modules (as a whole) would be searchable/not-searchable. However, I will remember to discuss it with my mentor regarding your idea. But, for the first version of Global Search, I personally feel it is not that important. I want to implement just the necessary things required for Global Search first. Once, the basics (the basic first version) have been laid out in working mode, we can always improve it on a whole new level making it richer and better. 

Thanks again for the kind feedback. It was really helpful. smile

Average of ratings: -
In reply to Prateek Sachan

Re: Writing Moodle's Global Search

by Aparup Banerjee -

" if/when my GSoC project gets approved"

Its full steam ahead for this project now smile

Average of ratings: -
In reply to Prateek Sachan

Re: Writing Moodle's Global Search

by Luis de Vasconcelos -

A related discussion and tracker issue:

Global Search rewrite

MDL-31989 - Global Search 2

Average of ratings: -
In reply to Luis de Vasconcelos

Re: Writing Moodle's Global Search

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Calling Mr Dougiamas, Calling Mr Dougiamas...

"Solr is written in Java and runs as a standalone full-text search server within a servlet container such as Jetty"

Average of ratings: -
In reply to Prateek Sachan

Re: Writing Moodle's Global Search

by Niall Julian -

Good luck with this Prateek. A working global site search is badly needed!

Average of ratings: -
In reply to Niall Julian

Re: Writing Moodle's Global Search

by Prateek Sachan -

Thanks Niall.

The specs for Global Search are being updated here.

If you have any suggestions, feel free to post it here in the forum.

Average of ratings: -
In reply to Prateek Sachan

Re: Writing Moodle's Global Search

by Jürg Hoerner -

In the search results I like to see the course name  and the course path (and  a course link). Maybe it will nice to see if the cours is activ or not, filter.

Thank you very much vor the great work.

 

 

Average of ratings: -
In reply to Jürg Hoerner

Re: Writing Moodle's Global Search

by Prateek Sachan -

Thanks for the feedback.

Currently, the results do show the Course Name as a link.

 

Thanks. 

 

Average of ratings: -
In reply to Prateek Sachan

Re: Writing Moodle's Global Search

by toby saunders -

Hi Prateek

I was hoping to use global search in my moodle 2.5 project - will this search be backwards compatible with moodle 2.5? I was also wondering what the time scale was until a release as this will be an enterprise site and they won't allow any software which is in alpha/beta state.

Many thanks

Toby

 

Average of ratings: -
In reply to toby saunders

Re: Writing Moodle's Global Search

by Arjun Baliga -

Dear Prateek,

I was just referreing your page http://docs.moodle.org/dev/Global_search_(GSoC2013)#Introduction .

I was able to install solr-php extension .While installing Apache Solr ,I guess I am missing something which I am not able to understand.

In Setting up Global Search for Moodle section I cannot find files that you mentioned for  Apache Solr release 4.6 in exact location.also I could not find global search folder.

Thanks in advance,

Arjun Baliga

 

 

 

 

Average of ratings: -
In reply to Arjun Baliga

Re: Writing Moodle's Global Search

by Prateek Sachan -

Hi Arjun, 

You may refer Global Search docs. You need to download Apache Solr and keep it in an external directory (not inside Moodle directory). The xml files that need to be replaced are included in the Global Search repo. See here.

In case of any doubts on installing it, you may email me directly.

Thanks.

 

 

Average of ratings: -
In reply to toby saunders

Re: Writing Moodle's Global Search

by Prateek Sachan -

Hi Toby,

I've written Global Search both for 2.5 and 2.6. However, it hasn't been integrated into the official repo yet. Moodle HQ developers will review it for 2.7 release next year.

In case of any doubts on installing it, you may email me directly.

Thanks.

Average of ratings: -
In reply to Prateek Sachan

Re: Writing Moodle's Global Search

by Andreas Weber -

Hi Prateek,

just installed global search and it throws an error in cron,
please look:

http://globalsearch.haspaakademie.de/admin/cron.php

Thanks in advance!
Andreas

Average of ratings: -
In reply to Andreas Weber

Re: Writing Moodle's Global Search

by Daniel Neis Araujo -
Picture of Core developers Picture of Plugin developers Picture of Translators

Hello,

 

i've managed to make it work on top of current master (2.7 beta):

https://github.com/danielneis/moodle/tree/MDL-31989

 

I've tested it creating a medium test course (with 500 forum posts + 200 pages) and it worked well.

Also, the current php-pecl-solr extension version 2.0.0b works well with the Solr 4.8.0

 

Kind regards,

Daniel

Average of ratings: -
In reply to Prateek Sachan

Re: Writing Moodle's Global Search

by Patricia Moura -

Hi Prateek,

Congrats on your amazing work! Any news about the 2.7 release?

Thanks,

Patricia

Average of ratings: -
In reply to Patricia Moura

Re: Writing Moodle's Global Search

by toby saunders -

Hi - is there any news or a timeline for this getting merged into the main  moodle codebase?

Average of ratings: -
In reply to toby saunders

Re: Writing Moodle's Global Search

by Николай Сурмин -
Dear community!

I am struggling with Global Search installment. 
I have installed Linux on my computer, downloaded and installed Moodle 2.5 and after a week of affords even solr, but in the end of the day I still have an error: 

I see solr running at http://localhost:8983/solr/
But I see "Solr Server is not running!" at http://www.moodle.local/search/admin.php

SolrClient->ping() returns false.
At "if (!$search_engine_check_server($client)) {"

Could somebody please be so kind and give me a peace of advise what can be the problem and how I can solve it.

Thanks in advance.

B.R. 
Nick.
Average of ratings: -
In reply to Николай Сурмин

Re: Writing Moodle's Global Search

by Daniel Neis Araujo -
Picture of Core developers Picture of Plugin developers Picture of Translators

Hello,


you may take a look at https://tracker.moodle.org/browse/MDL-31989

and try the newest code on top of 2.9dev that is mentioned on the last comment.


Kind regards,

Daniel

Average of ratings: -
In reply to Daniel Neis Araujo

Re: Writing Moodle's Global Search

by Mae Church -

I am very interested in the status of the Global Search, I think it is MDL-31989.  I need to provide a capability to our users and would prefer to use a moodle capability instead of an external source.  I have read the documentation but am not clear if this function is available.  Any update would be greatly appreciated.  

Average of ratings: -
In reply to Mae Church

Re: Writing Moodle's Global Search

by Daniel Neis Araujo -
Picture of Core developers Picture of Plugin developers Picture of Translators

Hello,


this feature is not fully available on Moodle,

but if you manage to apply the patch following links on the tracker,

the result is pretty usable, altough it lacks various capabilities checks

to verify if user can really access the content searched.

Also, Solr is a little complicated to get running and is the only

search engine supported right now, so I've started to write some code

to get elastic search integrated too:

https://github.com/danielneis/moodle/commit/d24d6e165e440c69240a3f2e465c65ba3e60a16a


Finally, using only Moodle and a relational database to implement

this feature should not be the best option even if you use something like

sphinx, the features to index content of different file types and decouple

from the main moodle database and webservice are the mais advantages of

this approach with solr/elasticsearch.


If you are interested in funding further development of these features, let us know =)


Kind regards,

Daniel

Average of ratings: -
In reply to Daniel Neis Araujo

Re: Writing Moodle's Global Search

by Mae Church -


Thank you I appreciate your quick and thorough response.  I am not sure if I can fund further development but I will look into that.

 

Is there another recommended approach while we wait for this approach to be completed?

We are on 2.9

Average of ratings: -
In reply to Daniel Neis Araujo

Re: Writing Moodle's Global Search

by Nathalie Georgiou -

Hello !

I would like to know if anything has been finished on this issue? I am very interested in a Global Search on Moodle. 

I'm currently developing a training portal for my company, and being able to look for documents which contain certain key words would make the entire training process a lot easier.

Can you please let me know if there is any such search available on Moodle?

Thanks!
Nathalie

Average of ratings: -
In reply to Nathalie Georgiou

Re: Writing Moodle's Global Search

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Global search is not available in core Moodle. All discussions I have read involve additional software including a server side setup called Solr 

https://lucene.apache.org/solr/

which is java based. 

Average of ratings: -
In reply to Marcus Green

Re: Writing Moodle's Global Search

by Nathalie Georgiou -

Thank you so much for your answer Marcus!

I will check this option more. 

If anyone has used it and has feedback, feel free to share with me. 

Thanks!

Nathalie

Average of ratings: -
In reply to Nathalie Georgiou

Re: Writing Moodle's Global Search

by Daniel Neis Araujo -
Picture of Core developers Picture of Plugin developers Picture of Translators

Hello,


tha "Global Search" feature was integrated in Moodle last week (https://moodle.org/mod/forum/discuss.php?d=328852) and yes, it needs an external "indexing and searching" service.


Kind regards,

Daniel

Average of ratings: -
In reply to Daniel Neis Araujo

Re: Writing Moodle's Global Search

by Nathalie Georgiou -

Great! It looks like I have asked just at the perfect time! Thanks for the answer Daniel

Best,
Nathalie

Average of ratings: -
In reply to Daniel Neis Araujo

Re: Writing Moodle's Global Search

by Nathalie Georgiou -

Sorry to get back to you again, but I'm hoping you can help.
My system team has now installed the tool, but how do I make it visible on the entire website?



Thanks!
Best,
Nathalie

Average of ratings: -
In reply to Nathalie Georgiou

Re: Writing Moodle's Global Search

by David Monllaó -

Hi Nathalie,

You can follow https://docs.moodle.org/dev/Global_search, this might slightly change during the next weeks though.

Average of ratings: -
In reply to David Monllaó

Re: Writing Moodle's Global Search

by Nathalie Georgiou -

Thank you David for your answer, I will follow that page for further developments.

Best,

Nathalie

Average of ratings: -
In reply to Nathalie Georgiou

Re: Writing Moodle's Global Search

by Daniel Neis Araujo -
Picture of Core developers Picture of Plugin developers Picture of Translators

Hello,


if anyone is interested, I've developed a search engine plugin to use Elasticsearch:

https://github.com/danielneis/moodle-search_elasticsearch


With elasticsearch you do not need any php extensions, it uses curl to access the rest api of elasticsearch.

Just point your moodle to the url and port of elasticsearch and it works.


Kind regards,

Daniel


Average of ratings: -
In reply to Daniel Neis Araujo

Re: Writing Moodle's Global Search

by David Monllaó -

I'm sorry Daniel but admins with interest in keeping their site contents secured should not use your plugin. You are making all indexed contents available to everyone and you are not filtering the index contents by the context where the user has access, also all filters are ignored and this solution will not scale properly. Please do not recommend this for any production site.

We included Elastic search as a proposal for a GSOC project (https://docs.moodle.org/dev/Projects_for_new_developers#Elastic_search) where we aim to have a secure and scalable elasticsearch plugin for Moodle 3.2. Moodle 3.1 will come with support for Solr and, if we have time to complete it, support for a "local db table" search engine (only for small moodle sites - MDL-53226)

Average of ratings: Useful (1)
In reply to David Monllaó

Re: Writing Moodle's Global Search

by Daniel Neis Araujo -
Picture of Core developers Picture of Plugin developers Picture of Translators

Hello, David


thanks for your review.

The plugin is not really ready for production and that is why it is "only" on the Github and not yet sent to Plugins directory.

I've followed the instructions on docs and thought that calling the "check access" function was enough to check by cotext as you can see at:
https://github.com/danielneis/moodle-search_elasticsearch/blob/master/classes/engine.php#L91-L99


Also, I've found that in the docs it says to use \core_search\document::set_data_from_engine , but instead the Solr plugin uses $this->to_document and it worked for me too, because when I try to use the method on the documentation, Moodle says it should not be called statically.


Finally, I would like to know why you think this plugin is not scalable.


Kind regards,

Daniel


Average of ratings: -
In reply to Daniel Neis Araujo

Re: Writing Moodle's Global Search

by Mike Churchward -
Picture of Core developers Picture of Plugin developers Picture of Testers

Hi Daniel -

Since your new plugin is still under development, can I suggest you make the following changes to your "version.php" file?

  • Add a line for "maturity" and set it to beta or alpha. So:
    • $plugin->maturity = MATURITY_ALPHA;
  • Add a line for "release" for a more readable version description. So something like:
    • $plugin->release = '0.1 - Alpha';
  • Change the "requires" to be the Moodle 3.1 version. This will prevent it from being installed on any version prior to 3.1. So:
    • $plugin->requires = 2016031000;

See the details on version.php for more info.

These changes will help ensure  that a Moodle admin will be aware of its current development status.

mike

Average of ratings: Useful (2)
In reply to Mike Churchward

Re: Writing Moodle's Global Search

by Daniel Neis Araujo -
Picture of Core developers Picture of Plugin developers Picture of Translators
Hello, Mike

thanks for the suggestions!
I've made those changes and they are now on the repo.

Kind regards,
Daniel
Average of ratings: -
In reply to Daniel Neis Araujo

Re: Writing Moodle's Global Search

by Eric Merrill -
Picture of Core developers Picture of Moodle HQ Picture of Peer reviewers Picture of Plugin developers Picture of Testers

The docs don't yet fully represent the current state of the API. When I get done with file indexing, I'm going to work on updating them. I would look at engine and the mock stuff (tests/fixtures) to see what the bare minimum is.

Average of ratings: -
In reply to Daniel Neis Araujo

Re: Writing Moodle's Global Search

by David Monllaó -

Hi Daniel,

Search engines do not need to worry much about the indexing process, search areas are responsible of deciding which info should be indexed, but search engine plugins are fully responsible of all security checkings when retrieving data, if you send a query against all indexed data only checking the access when you already got the results back from the engine you will probably get 0 results for the current user in any site with more than 10 courses (to say a number). More info in https://docs.moodle.org/dev/Search_engines#Retrieve_contents.

We are already receiving proposals for this GSOC project, if you just wanted to share an elasticsearch plugin proof of concept many thanks, but if you want to develop a serious plugin that people can use in their sites I would wait for GSOC outcome, we would not like to waste anyone's time.

As Eric said, the search API and search engine plugintype interfaces will not be final until 3.1 is released, we are still refining it.

Average of ratings: -
In reply to Daniel Neis Araujo

Re: Writing Moodle's Global Search

by Cesar Resendiz -
Hello Daniel, 

I installed elasticserach in my moodle 3.2, I followed the instructions that you have in Github but I did not manage to activate the search.


PasoEstado
1. Seleccionar motor de búsquedaElasticsearch
2. Habilitar áreas de búsquedaSí
3. Configurar el motor de búsqueda
4. Indexar datosNo
5. Habilitar búsqueda globalSí

When I try to search, I get the following message: The search engine is not available. Please contact your administrator.

Where do you think the problem is?

Average of ratings: -
In reply to Cesar Resendiz

Re: Writing Moodle's Global Search

by Mike Churchward -
Picture of Core developers Picture of Plugin developers Picture of Testers

The message indicates that it cannot find the elasticsearch server you specified in the plugin setup. Do you have an Elasticsearch server running somewhere that you can use (See https://www.elastic.co/)?

Average of ratings: -