Hi, I'm Prateek and planning to write Moodle's Global Search.
Earlier, it was written by Tomasz Muras.
The previous code is here : https://github.com/prateeksachan/moodle/tree/gs (forked from Tomasz's code). The code shouldn't be re-used and used only as a reference.
The official docs on Global Search are here.
The code of Search APIs for modules here could be used in the new implementation.
I wanted to invite suggstions and ideas regarding any other features that need to be included in Global Search's version 1.0. I would also like to have ideas about the UI of search results shown to the user (as this will be an important part).
For this version, we plan to integrate Solr.
My motive is to build Global Search and the core modules supporting it.
Suggestions are welcome.
Currently, you can search within forums. Imagine, if you have to find important documents within some course's modules (like lessons, pages, etc.).
Global Search will prove very handy in such situations.
Pls solve my my problem.
I have 2 courses category named Jan-June 2015 and Aug-Dec 2015.
On clicking one category,it shows the result from both categories.
Please help me out what i have to do. My moodle version is Moodle 2.8.3
I am new in moodle,kindly help me.
the most important thing is to define your requirements specification based on interviews/research with actual students.
otherwise its all just speculation on what might useful or nice to have.
get a clear scope from real users - not like the people on moodle.org - and create a very clear mvp spec on that.
this way you can avoid scope creep or nice to have, but no one actually uses.
jessse james garret has a useful diagram on this https://wiki.bath.ac.uk/download/attachments/34409648/elementsofuserexperience.jpg?version=1&modificationDate=1285603076000
and i cannot recommend http://www.amazon.co.uk/Designing-Digital-Age-Human-Centered-Products/dp/0470229101 enough for anyone about to embark on such a task!
good luck Prateek- looking forward to seeing your user research and resulting requirements spec for this
Currently, I prepared two simple templates regarding the Search's UI.
The pop-up will instruct users about "how to use search features" (similar to editing one on issue page. ).
The results could be displayed in the following manner (its just a simple-basic template):
For the first version, I just want to create the minimal things required to get the stuff started.
Some aspects while thinking about in for five minutes:
- where is the search located: in a course, on mymoodle page?
- where does it search?
- does it reflect on group memberschip?- does it reflect conditional activities?
- what happens with file attachment content
Search is mostly asked for file content and folder content.
What does the result show? the search text in its content 10 words before an after? What happens if a search term is used very often in a lesson or book on different pages, what shows the results page
What should the search show from a quiz before the student made an attempt, after the attempt? (description text only, question text, answering options, feedback text, essay question student answer, teachers feedback for essay question). To keep it simple I would suggest to limit it here on description text.
To answer your queries:
where is the search located: in a course, on mymoodle page?
Search will be more like a plugin. It will be in your moodle site - integrated inside modules.
where does it search?
It will search for keywords based on your options. You may choose whether to include data from quizzes or forums or any other module in your search.
does it reflect on group memberschip?- does it reflect conditional activities?
Yes, it will reflect group memberships. As an example, a student from group A will not be able to view search results that are a part of assignment/quiz submission from group B. (However, the teacher will be able to see them). It will handle proper security.
what happens with file attachment content
Keywords will also be searched inside HTML, PPT or PDF files. If a user has access (or proper "capability"), then only the results will be shown along with the link of the file attached.
What happens if a search term is used very often in a lesson or book on different pages, what shows the results page?
The results will be based on "relevance weightage". For example:
Lesson A contains 1 occurances of "batman superman" (as a phrase..both words are together)
Lesson B contains 10 occurances of batman and 6 occurances of superman.
If a user searches for "batman superman" Lesson B wil have higher priority (or relevance) as comparerd to Lesson A
Lesson A contains 10 occurances of "batman superman" (as a phrase..both words are together)
Lesson B contains 10 occurances of batman and 6 occurances of superman.
If a user searches for "batman superman" Lesson A wil have higher priority (or relevance) as comparerd to Lesson B
What should the search show from a quiz before the student made an attempt, after the attempt?
Search will look for "keywords". For example,
One the quiz questions is "Define thermodynamics".
If the student searches for "thermodynamics" before makin the attempt, search will return the content from the quiz question.
If he makes an attempt (and answers the question: "Thermodynamics is a branch of natural science concerned with heat and its relation to energy and work".), and then searches, the search willl then include both the attempt and quiz questions. (The answer may be nested inside the question itself!).
Thanks for the feedback. Would love to hear if you have any ideas. In the meantime, go through this page listing the features of Global Search:
this is very welcome. Hope it becomes fruitful soon.
I would suggest considering more capabilities to control which roles see which results. E.g. You could add a view search module A results capability to role 1 and not to role 2. This would allow several levels of search to different types of users.
Would also highly stress the usability option for refining search or search within results.
Obviously very effective UI ideas and features can be learned from Google search - e.g. Tabs to allow switching between different types of results (text, images, news etc in google, perhaps All, Content, Users, Courses, etc. in Moodle). Also ability to filter results quick according to latest update or creation (within 30 days, 3 months, year).
Thanks for the feedback. For the first version I will be trying to concentrate on including only important search features, just to get it started. Once it has been done, more and more features will be added with time.
UI will be a very important critical feature : the very reason why I posted about it here as I wanted ideas from the community. Yes, Google Search is really very good in terms of offering the user elegant search UI. Will definitely takes ideas from it.
Thanks for the concern and feedback. I will try my best to handle security with the Global Search. Proper security will be handled while showing the results. (Like students couldn't view other students' grades/answers etc. in addition to the situation you mentioned).
We don't want people to get access to documents that aren't meant for them. That is one of the most important aspects of Global Search.
That's correct Marcus and we should be very clear here. Global Search will be implemented in a way to allow for plugging other search engines than solr but at this moment only solr support will be implemented.
To make long story short (I've spent quite a bit of time on this): there is no search engine written in PHP, so it's either Java or nothing. I doubt anyone will ever implement support for anything else than solr.
We will depend on solr features as well. For example text extraction from documents is provided by solr, so we will not be implementing our own text extraction.
Because of the above, Global Search will not work with Moodle out of the box. To get in setup you will need to get Java hosting, setup solr, add PHP extension that talks to solr, etc. We will of course produce installation instructions.
Hope that helps,
Having one search engine like Solr to reference while building GS is really useful but as Tomek has mentioned, the plan is to have it pluggable.
It would be useful to have some other search engine backends/docs to match up with the developing API to cover off any holes, but failing that, i believe this won't be the case with Solr as the backend basis.
There was a slight confusion regarding the apache solr readme. I confirmed with those people. So, the module could be compiled w/o any dependencies on the used webserver.
Sorry, for the slight mess up here.
I will be listing down the dependencies over here (which is not yet final and will be updated): http://docs.moodle.org/dev/Global_search_%28GSoC2013%29#Installation
I'm glad to hear Moodle will use Apache Solr.
I've just implemented a Solr server at my institution. Although it has a steep learning curve (and it's Java!), it is fantastic.
The code I use to submit my Moodle courses to Solr (and Mahara user profiles, and a few custom in-house applications) is really simple, but if anyone wants it, they're welcome to it.
First, welcome to the Moodle community and good luck with the selection process for Gsoc.
Some suggestions/things to consider:-
# Ranking. Imho, ranking is one of the very important factor for this project. Ranking just based on occurence , might not give the most desired result. Google uses something called as Page rank concept (along with dozens of other factor), not sure if we can build something similar for us or not, but it surely is something worth looking into. For example the content from a book is more important than the content of a log entry, irrespective of the occurences.
# Security. Security will be a big concern while filtering the results, am pretty sure you already have some proposal on how to handle it. Just make sure it doesn't get too expensive.
# Events. You might want to look into the new event system proposed(work in progress) for Moodle 2.6. Might help you with indexing and other stuff. (http://docs.moodle.org/dev/Event_2)
# Not searchable content. - just an idea, that there would be a way to mark some of the content as not searchable (mimicking the robots.txt for normal search engines).
Good work so far,
Thanks a lot for the constructive feedback.
#Ranking: Thanks for this thought, I will definitely consider it and discuss it with my mentor if/when my GSoC project gets approved. I would try my best to implement the Page Rank concept and other factors (as you outlined) as far as possible.
However, I guess that on a general note, all modules would have the same priority. Log entries would however, have a lower priority than the modules.
#Security: It is a big concern. I will try my best not to compromise even a little in terms of security, because then the Global Search would still be undone. Currently, the manner in which I plan to handle it wouldn't have any expenses regarding security as I will be treating search modularily. But then, we will be 100% sure only after the testing.
#Events : As I've planned to to build the Global Search on Moodle 2.5 (closest to master), and the plan is to make it available in Moodle 2.6, hence I will keep this into account.
#Not Searchable Content: Currently, only the modules (as a whole) would be searchable/not-searchable. However, I will remember to discuss it with my mentor regarding your idea. But, for the first version of Global Search, I personally feel it is not that important. I want to implement just the necessary things required for Global Search first. Once, the basics (the basic first version) have been laid out in working mode, we can always improve it on a whole new level making it richer and better.
Thanks again for the kind feedback. It was really helpful.
Good luck with this Prateek. A working global site search is badly needed!
In the search results I like to see the course name and the course path (and a course link). Maybe it will nice to see if the cours is activ or not, filter.
Thank you very much vor the great work.
Thanks for the feedback.
Currently, the results do show the Course Name as a link.
I was hoping to use global search in my moodle 2.5 project - will this search be backwards compatible with moodle 2.5? I was also wondering what the time scale was until a release as this will be an enterprise site and they won't allow any software which is in alpha/beta state.
I was just referreing your page http://docs.moodle.org/dev/Global_search_(GSoC2013)#Introduction .
I was able to install solr-php extension .While installing Apache Solr ,I guess I am missing something which I am not able to understand.
In Setting up Global Search for Moodle section I cannot find files that you mentioned for Apache Solr release 4.6 in exact location.also I could not find global search folder.
Thanks in advance,
You may refer Global Search docs. You need to download Apache Solr and keep it in an external directory (not inside Moodle directory). The xml files that need to be replaced are included in the Global Search repo. See here.
In case of any doubts on installing it, you may email me directly.
i've managed to make it work on top of current master (2.7 beta):
I've tested it creating a medium test course (with 500 forum posts + 200 pages) and it worked well.
Also, the current php-pecl-solr extension version 2.0.0b works well with the Solr 4.8.0
Congrats on your amazing work! Any news about the 2.7 release?
Hi - is there any news or a timeline for this getting merged into the main moodle codebase?
I am very interested in the status of the Global Search, I think it is MDL-31989. I need to provide a capability to our users and would prefer to use a moodle capability instead of an external source. I have read the documentation but am not clear if this function is available. Any update would be greatly appreciated.
this feature is not fully available on Moodle,
but if you manage to apply the patch following links on the tracker,
the result is pretty usable, altough it lacks various capabilities checks
to verify if user can really access the content searched.
Also, Solr is a little complicated to get running and is the only
search engine supported right now, so I've started to write some code
to get elastic search integrated too:
Finally, using only Moodle and a relational database to implement
this feature should not be the best option even if you use something like
sphinx, the features to index content of different file types and decouple
from the main moodle database and webservice are the mais advantages of
this approach with solr/elasticsearch.
If you are interested in funding further development of these features, let us know =)
Thank you I appreciate your quick and thorough response. I am not sure if I can fund further development but I will look into that.
Is there another recommended approach while we wait for this approach to be completed?
We are on 2.9
Hi, Please guide me to integrate it with moodle 2.7
I would like to know if anything has been finished on this issue? I am very interested in a Global Search on Moodle.
I'm currently developing a training portal for my company, and being able to look for documents which contain certain key words would make the entire training process a lot easier.
Can you please let me know if there is any such search available on Moodle?
Thank you so much for your answer Marcus!
I will check this option more.
If anyone has used it and has feedback, feel free to share with me.
tha "Global Search" feature was integrated in Moodle last week (https://moodle.org/mod/forum/discuss.php?d=328852) and yes, it needs an external "indexing and searching" service.
Great! It looks like I have asked just at the perfect time! Thanks for the answer Daniel
Sorry to get back to you again, but I'm hoping you can help.
My system team has now installed the tool, but how do I make it visible on the entire website?
Thank you David for your answer, I will follow that page for further developments.
if anyone is interested, I've developed a search engine plugin to use Elasticsearch:
With elasticsearch you do not need any php extensions, it uses curl to access the rest api of elasticsearch.
Just point your moodle to the url and port of elasticsearch and it works.
I'm sorry Daniel but admins with interest in keeping their site contents secured should not use your plugin. You are making all indexed contents available to everyone and you are not filtering the index contents by the context where the user has access, also all filters are ignored and this solution will not scale properly. Please do not recommend this for any production site.
We included Elastic search as a proposal for a GSOC project (https://docs.moodle.org/dev/Projects_for_new_developers#Elastic_search) where we aim to have a secure and scalable elasticsearch plugin for Moodle 3.2. Moodle 3.1 will come with support for Solr and, if we have time to complete it, support for a "local db table" search engine (only for small moodle sites - MDL-53226)
thanks for your review.
The plugin is not really ready for production and that is why it is "only" on the Github and not yet sent to Plugins directory.
I've followed the instructions on docs and thought that calling the "check access" function was enough to check by cotext as you can see at:
Also, I've found that in the docs it says to use \core_search\document::set_data_from_engine , but instead the Solr plugin uses $this->to_document and it worked for me too, because when I try to use the method on the documentation, Moodle says it should not be called statically.
Finally, I would like to know why you think this plugin is not scalable.
Hi Daniel -
Since your new plugin is still under development, can I suggest you make the following changes to your "version.php" file?
- Add a line for "maturity" and set it to beta or alpha. So:
- $plugin->maturity = MATURITY_ALPHA;
- Add a line for "release" for a more readable version description. So something like:
- $plugin->release = '0.1 - Alpha';
- Change the "requires" to be the Moodle 3.1 version. This will prevent it from being installed on any version prior to 3.1. So:
- $plugin->requires = 2016031000;
- $plugin->requires = 2016031000;
See the details on version.php for more info.
These changes will help ensure that a Moodle admin will be aware of its current development status.
The docs don't yet fully represent the current state of the API. When I get done with file indexing, I'm going to work on updating them. I would look at engine and the mock stuff (tests/fixtures) to see what the bare minimum is.
Search engines do not need to worry much about the indexing process, search areas are responsible of deciding which info should be indexed, but search engine plugins are fully responsible of all security checkings when retrieving data, if you send a query against all indexed data only checking the access when you already got the results back from the engine you will probably get 0 results for the current user in any site with more than 10 courses (to say a number). More info in https://docs.moodle.org/dev/Search_engines#Retrieve_contents.
We are already receiving proposals for this GSOC project, if you just wanted to share an elasticsearch plugin proof of concept many thanks, but if you want to develop a serious plugin that people can use in their sites I would wait for GSOC outcome, we would not like to waste anyone's time.
As Eric said, the search API and search engine plugintype interfaces will not be final until 3.1 is released, we are still refining it.
I installed elasticserach in my moodle 3.2, I followed the instructions that you have in Github but I did not manage to activate the search.
|1. Seleccionar motor de búsqueda||Elasticsearch|
|2. Habilitar áreas de búsqueda||Sí|
|3. Configurar el motor de búsqueda|
|4. Indexar datos||No|
|5. Habilitar búsqueda global||Sí|
When I try to search, I get the following message: The search engine is not available. Please contact your administrator.
Where do you think the problem is?
The message indicates that it cannot find the elasticsearch server you specified in the plugin setup. Do you have an Elasticsearch server running somewhere that you can use (See https://www.elastic.co/)?