General developer forum

 
 
Picture of Prateek Sachan
Writing Moodle's Global Search
Group DevelopersGroup Testers

Hi, I'm Prateek and planning to write Moodle's Global Search.

Earlier, it was written by Tomasz Muras. 
The previous code is here : https://github.com/prateeksachan/moodle/tree/gs (forked from Tomasz's code). The code shouldn't be re-used and used only as a reference. 

The official docs on Global Search are here

The code of Search APIs for modules here could be used in the new implementation.

These are the current search features needed to be implemented

I wanted to invite suggstions and ideas regarding any other features that need to be included in Global Search's version 1.0. I would also like to have ideas about the UI of search results shown to the user (as this will be an important part).

For this version, we plan to integrate Solr. 

My motive is to build Global Search and the core modules supporting it.

Suggestions are welcome.

Thanks.

 
Average of ratings: -
Picture of Joseph Thibault
Re: Writing Moodle's Global Search
Group Particularly helpful MoodlersGroup Testers

Having the ability to perform advanced searches would be great in my opinion (https://moodle.org/mod/forum/search.php?id=5) just an example.

 
Average of ratings: -
Picture of Prateek Sachan
Re: Writing Moodle's Global Search
Group DevelopersGroup Testers

Yes!
Currently, you can search within forums. Imagine, if you have to find important documents within some course's modules (like lessons, pages, etc.).

Global Search will prove very handy in such situations.

 
Average of ratings: -
Picture of Stuart Lamour
Re: Writing Moodle's Global Search
Group Particularly helpful Moodlers

Hi Prateek,

the most important thing is to define your requirements specification based on interviews/research with actual students.

otherwise its all just speculation on what might useful or nice to have.

get a clear scope from real users - not like the people on moodle.org - and create a very clear mvp spec on that.

this way you can avoid scope creep or nice to have, but no one actually uses.

jessse james garret has a useful diagram on this https://wiki.bath.ac.uk/download/attachments/34409648/elementsofuserexperience.jpg?version=1&modificationDate=1285603076000

and i cannot recommend http://www.amazon.co.uk/Designing-Digital-Age-Human-Centered-Products/dp/0470229101 enough for anyone about to embark on such a task!

good luck Prateek- looking forward to seeing your user research and resulting requirements spec for this smile 

 

 
Average of ratings: -
Picture of Prateek Sachan
Re: Writing Moodle's Global Search
Group DevelopersGroup Testers

Currently, I prepared two simple templates regarding the Search's UI.

The pop-up will instruct users about "how to use search features" (similar to editing one on issue page. ).

The results could be displayed in the following manner (its just a simple-basic template):

  

For the first version, I just want to create the minimal things required to get the stuff started.

Thanks.

 
Average of ratings: -
Picture of Ralf Hilgenstock
Re: Writing Moodle's Global Search
Group DevelopersGroup Translators

Some aspects while thinking about in for five minutes:

- where is the search located: in a course, on mymoodle page?
- where does it search?
- does it reflect on group memberschip?- does it reflect conditional activities?
- what happens with file attachment content

Search is mostly asked for file content and folder content.

What does the result show? the search text in its content 10 words before an after? What happens if a search term is used very often in a lesson or book on different pages, what shows the results page

What should the search show from a quiz before the student made an attempt, after the attempt?  (description text only,  question text, answering options,  feedback text, essay question student answer, teachers feedback for essay question). To keep it simple I would suggest to limit it here on description text.

 

 
Average of ratings: -
Picture of Prateek Sachan
Re: Writing Moodle's Global Search
Group DevelopersGroup Testers

Hi Ralf,

To answer your queries:

where is the search located: in a course, on mymoodle page?
Search will be more like a plugin. It will be in your moodle site - integrated inside modules.

where does it search?
It will search for keywords based on your options. You may choose whether to include data from quizzes or forums or any other module in your search.
 
does it reflect on group memberschip?- does it reflect conditional activities?
Yes, it will reflect group memberships. As an example, a student from group A will not be able to view search results that are a part of assignment/quiz submission from group B. (However, the teacher will be able to see them). It will handle proper security. 

what happens with file attachment content
Keywords will also be searched inside HTML, PPT or PDF files. If a user has access (or proper "capability"), then only the results will be shown along with the link of the file attached. 

What happens if a search term is used very often in a lesson or book on different pages, what shows the results page?
The results will be based on "relevance weightage". For example:
Case I:
Lesson A contains 1 occurances of "batman superman" (as a phrase..both words are together)
Lesson B contains 10 occurances of batman and 6 occurances of superman.

If a user searches for "batman superman" Lesson B wil have higher priority (or relevance) as comparerd to Lesson A

Case II:
Lesson A contains 10 occurances of "batman superman" (as a phrase..both words are together)
Lesson B contains 10 occurances of batman and 6 occurances of superman.

If a user searches for "batman superman" Lesson A wil have higher priority (or relevance) as comparerd to Lesson B

What should the search show from a quiz before the student made an attempt, after the attempt? 
Search will look for "keywords". For example,
One the quiz questions is "Define thermodynamics".
If the student searches for "thermodynamics" before makin the attempt, search will return the content from the quiz question.

If he makes an attempt (and answers the question: "Thermodynamics is a branch of natural science concerned with heat and its relation to energy and work".), and then searches, the search willl then include both the attempt and quiz questions. (The answer may be nested inside the question itself!).


Thanks for the feedback. Would love to hear if you have any ideas. In the meantime, go through this page listing the features of Global Search: 
http://docs.moodle.org/dev/Global_Search#Requirements_and_assumptions

 
Average of ratings: -
Picture of Amir Elion
Re: Writing Moodle's Global Search
Group Translators

Hi Prateek,

this is very welcome. Hope it becomes fruitful soon.

I would suggest considering more capabilities to control which roles see which results. E.g. You could add a view search module A results capability to role 1 and not to role 2. This would allow several levels of search to different types of users.

Would also highly stress the usability option for refining search or search within results.

Obviously very effective UI ideas and features can be learned from Google search - e.g. Tabs to allow switching between different types of results (text, images, news etc in google, perhaps All, Content, Users, Courses, etc. in Moodle). Also ability to filter results quick according to latest update or creation (within 30 days, 3 months, year). 

 
Average of ratings: -
Picture of Prateek Sachan
Re: Writing Moodle's Global Search
Group DevelopersGroup Testers

Hi Amir,

Thanks for the feedback. For the first version I will be trying to concentrate on including only important search features, just to get it started. Once it has been done, more and more features will be added with time.

UI will be a very important critical feature : the very reason why I posted about it here as I wanted ideas from the community. Yes, Google Search is really very good in terms of offering the user elegant search UI. Will definitely takes ideas from it.

Thanks.

 
Average of ratings: -
Picture of Ralf Hilgenstock
Re: Writing Moodle's Global Search
Group DevelopersGroup Translators

Hello Prateek

thanks for your detailled answer. Just one feedback:

If search is possible for quiz question content should be defined by teacher and not by student. The teacher should be able to define that student could not serch in quiz question content.

 

ralf

 
Average of ratings: -
Picture of Prateek Sachan
Re: Writing Moodle's Global Search
Group DevelopersGroup Testers

Hi Ralf,

Thanks for the concern and feedback. I will try my best to handle security with the Global Search. Proper security will be handled while showing the results. (Like students couldn't view other students' grades/answers etc. in addition to the situation you mentioned).

We don't want people to get access to documents that aren't meant for them. That is one of the most important aspects of Global Search. 

Thanks.

 
Average of ratings: -
Sketch...
Re: Writing Moodle's Global Search
Group Particularly helpful Moodlers

> what happens with file attachment content
> Keywords will also be searched inside HTML, PPT or PDF files.

Will other "Office" file formats like Word documents (.DOC/.DOCX) and Excel spreadsheets (.XLS/.XLSX) be included?

 
Average of ratings: -
Picture of Tomasz Muras
Re: Writing Moodle's Global Search
Group DevelopersGroup Particularly helpful MoodlersGroup Translators

The plan is to use solr for full text extraction, which in turn uses Apache Tika. See list of supported formats.

cheers,
Tomek

 
Average of ratings:Useful (2)
Sketch...
Re: Writing Moodle's Global Search
Group Particularly helpful Moodlers

So there is a big dependency on several Apache "extensions" (Tika, Solr...) How does this affect Moodle sites that are hosted on non-Apache web servers like IIS?

 
Average of ratings: -
Picture of Marcus Green
Re: Writing Moodle's Global Search
Group Particularly helpful Moodlers

It's not any Apache web server dependency that concerns me it is the server side Java dependency. It takes Moodle from generic LAMP software that can run on affordable web space to something rather specialised.

I have worked extensively with Java and server side java and I know of what I speak.

 

 
Average of ratings: -
Picture of Tomasz Muras
Re: Writing Moodle's Global Search
Group DevelopersGroup Particularly helpful MoodlersGroup Translators

That's correct Marcus and we should be very clear here. Global Search will be implemented in a way to allow for plugging other search engines than solr but at this moment only solr support will be implemented.

To make long story short (I've spent quite a bit of time on this): there is no search engine written in PHP, so it's either Java or nothing. I doubt anyone will ever implement support for anything else than solr.

We will depend on solr features as well. For example text extraction from documents is provided by solr, so we will not be implementing our own text extraction.

Because of the above, Global Search will not work with Moodle out of the box. To get in setup you will need to get Java hosting, setup solr, add PHP extension that talks to solr, etc. We will of course produce installation instructions.

Hope that helps,
Tomek

 
Average of ratings: -
MD shot of me from his iphone4
Re: Writing Moodle's Global Search
Group DevelopersGroup Moodle Course Creator Certificate holdersGroup Moodle HQGroup Particularly helpful MoodlersGroup Testers

Having one search engine like Solr to reference while building GS is really useful but as Tomek has mentioned, the plan is to have it pluggable.

It would be useful to have some other search engine backends/docs to match up with the developing API to cover off any holes, but failing that, i believe this won't be the case with Solr as the backend basis.

 

 

 
Average of ratings: -
Sketch...
Re: Writing Moodle's Global Search
Group Particularly helpful Moodlers

And Global Search will only work on Apache web servers, so other web servers like IIS are excluded and won't be supported?

 
Average of ratings: -
Picture of Hubert Chathi
Re: Writing Moodle's Global Search
 

As I wrote in my reply to your previous comment, Tika and Solr are standalone products that are unrelated to the Apache web server, other than the fact that both are managed by the Apache Foundation.  There should be no problems running it on other servers, as long as it supports Java.

 
Average of ratings: -
Sketch...
Re: Writing Moodle's Global Search
Group Particularly helpful Moodlers
According to Prateek, Global Search will initially only work on Apache: http://docs.moodle.org/dev/Talk:Global_search_(GSoC2013)
 
Average of ratings: -
Picture of Prateek Sachan
Re: Writing Moodle's Global Search
Group DevelopersGroup Testers

There was a slight confusion regarding the apache solr readme. I confirmed with those people. So, the module could be compiled w/o any dependencies on the used webserver.

Sorry, for the slight mess up here.

I will be listing down the dependencies over here (which is not yet final and will be updated): http://docs.moodle.org/dev/Global_search_%28GSoC2013%29#Installation

 
Average of ratings: -
Picture of Tomasz Muras
Re: Writing Moodle's Global Search
Group DevelopersGroup Particularly helpful MoodlersGroup Translators

As long as you can install solr PHP extension it should work on IIS just fine.

Tomek

 
Average of ratings: -
Picture of Prateek Sachan
Re: Writing Moodle's Global Search
Group DevelopersGroup Testers

Hi Marcus, installation procedures are being updated here.

Thanks.

 
Average of ratings: -
Picture of Hubert Chathi
Re: Writing Moodle's Global Search
 

Tika and Solr are not Apache extensions.  They are standalone products that are maintained by the Apache foundation, but otherwise are unrelated to the Apache web server.

 
Average of ratings:Useful (1)
Picture of Alex Walker
Re: Writing Moodle's Global Search
Group Particularly helpful Moodlers

I'm glad to hear Moodle will use Apache Solr.

I've just implemented a Solr server at my institution. Although it has a steep learning curve (and it's Java!), it is fantastic.

The code I use to submit my Moodle courses to Solr (and Mahara user profiles, and a few custom in-house applications) is really simple, but if anyone wants it, they're welcome to it.

 
Average of ratings: -
Picture of Ankit Agarwal
Re: Writing Moodle's Global Search
Group DevelopersGroup Moodle Course Creator Certificate holdersGroup Moodle HQGroup Particularly helpful MoodlersGroup Testers

Hey Prateek,
First, welcome to the Moodle community and good luck with the selection process for Gsoc.

Some suggestions/things to consider:-

# Ranking.  Imho, ranking is one of the very important factor for this project. Ranking just based on occurence , might not give the most desired result. Google uses something called as Page rank concept (along with dozens of other factor), not sure if we can build something similar for us or not, but it surely is something worth looking into. For example the content from a book is more important than the content of a log entry, irrespective of the occurences.

# Security. Security will be a big concern while filtering the results, am pretty sure you already have some proposal on how to handle it. Just make sure it doesn't get too expensive.

# Events. You might want to look into the new event system proposed(work in progress) for Moodle 2.6. Might help you with indexing and other stuff. (http://docs.moodle.org/dev/Event_2)

# Not searchable content. - just an idea, that there would be a way to mark some of the content as not searchable (mimicking the robots.txt for normal search engines).

Good work so far,

Thanks

 
Average of ratings: -
Picture of Prateek Sachan
Re: Writing Moodle's Global Search
Group DevelopersGroup Testers

Hi Ankit,

Thanks a lot for the constructive feedback. 

#Ranking: Thanks for this thought, I will definitely consider it and discuss it with my mentor if/when my GSoC project gets approved. I would try my best to implement the Page Rank concept and other factors (as you outlined) as far as possible.
However, I guess that on a general note, all modules would have the same priority. Log entries would however, have a lower priority than the modules. 

#Security: It is a big concern. I will try my best not to compromise even a little in terms of security, because then the Global Search would still be undone. Currently, the manner in which I plan to handle it wouldn't have any expenses regarding security as I will be treating search modularily. But then, we will be 100% sure only after the testing.

#Events : As I've planned to to build the Global Search on Moodle 2.5 (closest to master), and the plan is to make it available in Moodle 2.6, hence I will keep this into account.

#Not Searchable Content: Currently, only the modules (as a whole) would be searchable/not-searchable. However, I will remember to discuss it with my mentor regarding your idea. But, for the first version of Global Search, I personally feel it is not that important. I want to implement just the necessary things required for Global Search first. Once, the basics (the basic first version) have been laid out in working mode, we can always improve it on a whole new level making it richer and better. 

Thanks again for the kind feedback. It was really helpful. smile

 
Average of ratings: -
MD shot of me from his iphone4
Re: Writing Moodle's Global Search
Group DevelopersGroup Moodle Course Creator Certificate holdersGroup Moodle HQGroup Particularly helpful MoodlersGroup Testers

" if/when my GSoC project gets approved"

Its full steam ahead for this project now smile

 
Average of ratings: -
Sketch...
Re: Writing Moodle's Global Search
Group Particularly helpful Moodlers

A related discussion and tracker issue:

Global Search rewrite

MDL-31989 - Global Search 2

 
Average of ratings: -
Picture of Marcus Green
Re: Writing Moodle's Global Search
Group Particularly helpful Moodlers

Calling Mr Dougiamas, Calling Mr Dougiamas...

"Solr is written in Java and runs as a standalone full-text search server within a servlet container such as Jetty"

 
Average of ratings: -
Picture of Niall Julian
Re: Writing Moodle's Global Search
 

Good luck with this Prateek. A working global site search is badly needed!

 
Average of ratings: -
Picture of Prateek Sachan
Re: Writing Moodle's Global Search
Group DevelopersGroup Testers

Thanks Niall.

The specs for Global Search are being updated here.

If you have any suggestions, feel free to post it here in the forum.

 
Average of ratings: -
Picture of Jürg Hoerner
Re: Writing Moodle's Global Search
 

In the search results I like to see the course name  and the course path (and  a course link). Maybe it will nice to see if the cours is activ or not, filter.

Thank you very much vor the great work.

 

 

 
Average of ratings: -
Picture of Prateek Sachan
Re: Writing Moodle's Global Search
Group DevelopersGroup Testers

Thanks for the feedback.

Currently, the results do show the Course Name as a link.

 

Thanks. 

 

 
Average of ratings: -
Picture of toby saunders
Re: Writing Moodle's Global Search
 

Hi Prateek

I was hoping to use global search in my moodle 2.5 project - will this search be backwards compatible with moodle 2.5? I was also wondering what the time scale was until a release as this will be an enterprise site and they won't allow any software which is in alpha/beta state.

Many thanks

Toby

 

 
Average of ratings: -
Picture of Arjun Baliga
Re: Writing Moodle's Global Search
 

Dear Prateek,

I was just referreing your page http://docs.moodle.org/dev/Global_search_(GSoC2013)#Introduction .

I was able to install solr-php extension .While installing Apache Solr ,I guess I am missing something which I am not able to understand.

In Setting up Global Search for Moodle section I cannot find files that you mentioned for  Apache Solr release 4.6 in exact location.also I could not find global search folder.

Thanks in advance,

Arjun Baliga

 

 

 

 

 
Average of ratings: -
Picture of Prateek Sachan
Re: Writing Moodle's Global Search
Group DevelopersGroup Testers

Hi Arjun, 

You may refer Global Search docs. You need to download Apache Solr and keep it in an external directory (not inside Moodle directory). The xml files that need to be replaced are included in the Global Search repo. See here.

In case of any doubts on installing it, you may email me directly.

Thanks.

 

 

 
Average of ratings: -
Picture of Prateek Sachan
Re: Writing Moodle's Global Search
Group DevelopersGroup Testers

Hi Toby,

I've written Global Search both for 2.5 and 2.6. However, it hasn't been integrated into the official repo yet. Moodle HQ developers will review it for 2.7 release next year.

In case of any doubts on installing it, you may email me directly.

Thanks.

 
Average of ratings: -
Picture of Andreas Weber
Re: Writing Moodle's Global Search
 

Hi Prateek,

just installed global search and it throws an error in cron,
please look:

http://globalsearch.haspaakademie.de/admin/cron.php

Thanks in advance!
Andreas

 
Average of ratings: -