General developer forum

More proposed global search improvements

 
Picture of sam marshall
More proposed global search improvements
Core developersParticularly helpful MoodlersPlugin developersTesters

Thanks to various HQ developers for reviewing and integrating the code, the two global search improvements I developed so far have made it into Moodle 3.4. (Allowing block search MDL-58957; supporting partial indexing MDL-59039.)

As we move towards implementing global search in our Moodle-based VLE, there are more improvements I'd like to put in, and before I start I thought I'd post about them here to see if anyone has opinions.

(By the way I wonder if not too many people are actually using global search, or not on a large scale? It's actually a really nice feature and seems to work quite well, albeit there are some problems with it. And it's pretty easy to set up an Apache Solr server, or you can pay somebody to host it for you.)

Here are the things I'd like to implement - I will create tracker issues and so on when I'm ready to work on these (except there's one I already set up in a sort of half-arsed effort to see how easy it was, turned out not very).

1. A Behat step to mock search results for testing the UI (I already mentioned this last thread)

2. The ability to search courses you are not enrolled in

This is a complicated one. Currently the system only lets you search courses you are enrolled in, but in our VLE, there are some courses students can access even if not enrolled (using moodle/course:view). And we will have custom search UI to search only the current course. It would be nice if it works on those courses.

The obvious solution would be to make it possible to search all courses you can access even if not enrolled, but this could cause a large expansion in the number of contexts users can access (problem because of MDL-54992) and might be difficult for other reasons too.

What I'm thinking instead is simply to make it possible from the back end so that if you supply a list of courses to search, then it will include all those courses if you have access to them, even if you aren't enrolled. There are probably some details that need working out. Anyhow, this way it won't increase the number of contexts in a 'full' search...

3. Group-aware searching (already filed as MDL-58885)

Supposing you have 50 tutor groups and a tutor group forum with lots of posts, set to 'separate groups' mode. Currently, if a user searches for something that is mentioned in the forum, it will return all the results regardless of group, and then filter out results the user can't see - on average, 98% of results will be filtered out. This is really inefficient.

Also it would be nice if you can search for only results in a specific group.

Neither thing is possible because at the moment, group IDs are not passed to Solr or other search engines along with the other document metadata. I'd like to change this. I don't see a fundamental reason why it shouldn't work... It might be quite hard though, I didn't finish figuring it out.

4. Ability to search a single context

Our current in-house search system lets you search, for example, just in the current forum. At the moment there isn't a way to do this in the search API, you can only search a whole course. There's no fundamental reason why this shouldn't be possible, it should be quite easy to add. (I'm proposing adding the back-end for it, not necessarily a user interface.)

5. Ability to search by user

The search API already stores the user id of people who created a search document, but there's no way to use that for searching - it would be nice if you could search for 'posts by my mate Anthea that include the word asbestos', for instance. Again this should be reasonably feasible to add.

6. Use one server to search, another to index

If you have a large system and you want to switch between search servers (e.g. changing provider), you need to reindex on the new server, which might take many hours. It would be nice if there was a way to keep using the existing server (albeit with an out of date index) at the same time as indexing on the new one.


That's it. Anyone else interested in any of these? Some of them seem to me like pretty basic things that are missing from the API.

--sam
 
Average of ratings: Useful (3)
Picture of Paul Holden
Re: More proposed global search improvements
Core developersParticularly helpful MoodlersPlugin developers

By the way I wonder if not too many people are actually using global search

We're using it here (Moodle 3.1); it works great and the API is nice for module developers to easily add support for their own plugins too Yes

My wishlist would be to allow custom sorting of returned results - seems a bit haphazard at the moment, would be helpful to order results a-z, by date, etc

 
Average of ratings: -
Picture of sam marshall
Re: More proposed global search improvements
Core developersParticularly helpful MoodlersPlugin developersTesters

Interesting, and yes I second the point about the API being nice. (This might be a good point to mention that if anyone's using the OU collaborative tool plugins, the next versions of those will include global search support.)

By the way, in terms of wishlists, the changes I listed are basically ones we need in order to replicate functionality we have in our existing search. I think once we get further on with the project and actually deploy it we might come up with other things we would like to change. The product owners have been muttering* about prioritising results, although I'm not certain that's really necessary.

--sam

* This is the point where I find out if they read this forum and object to my characterisation!

 
Average of ratings: -
Picture of David Monllaó
Re: More proposed global search improvements
Core developersMoodle Course Creator Certificate holdersMoodle HQParticularly helpful MoodlersPlugin developersTesters

Hi Sam,

Cool, some comments:

1. +1

2. Linking a couple of related issues https://tracker.moodle.org/browse/MDL-53169 (we could have a site setting to allow search in open courses) and https://tracker.moodle.org/browse/MDL-55303 to search potential courses to enrol.

3. 50 groups courses is not a case we prioritised when designing the API. I agree that it is not efficient but not the most common use case either (sorry, defensive mode on :P) The only problem I see is that groups only exist below course level and search API is open to any core component so depending on what people searches for groups filter would not make much sense. Adding this new field would require a full site re-index.

4. It sounds good, and it shouldn't require many changes. If the backend is updated to accept a contextid filter ideally we should add a filter for it in the UI (not a strong opinion about it)

5. This was part of the proposed global search implementation and I remove it because..... I can't remember, probably less priority than other parts of the API and not many resources but yes, it should be easy to add an author autocomplete field that applies a userid filter.

6. Not much to add here sorry, wouldn't us require to duplicate all solr settings?

 
Average of ratings: -
Picture of sam marshall
Re: More proposed global search improvements
Core developersParticularly helpful MoodlersPlugin developersTesters

Thanks for considering this!

2. Interesting - your comments in MDL-53169 go further than I was thinking of (I was only thinking about courses where the user is allowed to access it without enrolling, i.e. they are never going to enrol, not ones where they allow guest access). I'll have to investigate this further before submitting anything. Maybe it could be a site setting like you say, or possibly even a site setting (to allow it) in addition to a search filter setting.

3. I agree it's not the most common use case but probably we are not the only place with large courses (~1,000 students split into 50 groups of 20, for instance) and a lack of group suppor might make deploying search infeasible in those cases.

UI for this feature could be a problem (I'm basically proposing the backend) - it only really makes sense when searching within a specific course but the current search UI doesn't even let you do that, you can select multiple courses... See below for UI thoughts.

Regarding the need to reindex, oh dear, I've not fully investigated this yet but was wondering if we can change the schema without a full reindex if the group field is optional - most search areas aren't going to need to support groups... Maybe Solr/other engines don't let you do that...

4. UI option for single/specific contexts would be good but again this is limited because there is currently no UI for searching within a course (we can feasibly do a UI for selecting e.g. activities within the course, but not really for activities within all the courses you have access to systemwide, there would be issues with duplicate names and all sorts). See below for UI possibility.

5. Yes this one can have UI! smile

6. This is obviously my lowest priority change... Maybe better done as a plugin rather than core if we really need it, I guess we could make a search engine that extends Solr to include two copies of (most of) the settings! I'll leave this one out of consideration.

Regarding the search UI... A lot of these I'm sort of wanting to improve the back-end search facilities in order that a different UI could be implemented by customisers (e.g. us) rather than really proposing a change to the current Moodle search UI. Because I don't think our UI would necessarily be right for Moodle. But I think it's still worth having these changes because it gives the flexibility to change/improve the Moodle UI later, too.

About our UI (for info): the main focus of our system (our current one that we want to replicate) is that the search box is context-sensitive. So it's always in the same place but if you are looking at a course page (say for course shortname B747) the placeholder hint/label say 'Search B747' and it takes you to course search; if you are inside a forum it says 'Search this forum' and takes you to activity search. Basic principle is that search results are going to be more relevant if we can search a restricted area, and often you do know you want to search for something within a specific course or forum. For example if you're studying 3 courses and one of them is about statistics, and you want to search for 'confidence', that could return results across other courses on other topics as well, but if you're currently thinking about statistics, you want that type of confidence.

Obviously Moodle 'global' search works the opposite way where it assumes you want to search everything. Maybe one way of implementing something like this in Moodle search UI without changing it much could be to just note the context that the user came from where they typed the search (optional param &fromcontext=12345), and automatically show this as an option in the filters section. Just a simple checkbox like 'Search only within B747 Airline management' (if you came from that course) or 'Search only within Tutor group forum' (if you came from that forum). Rather than users having to select where they're searching, just a single checkbox to search where you came from.

That would provide a UI for point 4, specific contexts (and possibly also point 3, groups - if you came from an activity with separate/visible groups enabled) and also maybe point 2 (searching courses you aren't enrolled in) as well as a quicker alternative UI for the existing ability to search selected courses. Quite an easy change too.

How does that sound? I might be able to implement that (possibly)...

--sam

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: More proposed global search improvements
Core developersDocumentation writersParticularly helpful MoodlersPlugin developers

Lots of good stuff here that I am not going to join in with.

Just one thought re the UI bit: Might be good to offer a choice of the context all all parents, so Tutor group forum, B747, Business school, Whole site.

 
Average of ratings: -
Picture of David Monllaó
Re: More proposed global search improvements
Core developersMoodle Course Creator Certificate holdersMoodle HQParticularly helpful MoodlersPlugin developersTesters

Sorry for the delay responding to your comment Sam.

2. Yes, site setting + filter option also sounds good, I thought that site setting would be ok because it seems something that mainly depends on how the site admins manage user accesses.

Your context-aware search proposal also sounds good; would be nice to hear from people using it to know if they would prefer global search to work this way by default; my +1 for it. The required changes to allow it do not require any major refactoring either and they should be quite straight forward although there are a couple of points that is important commenting about:

  1. In a strict sense filtering by a course context means getting stuff that is stored in the search engine using the course contextid, this would exclude blocks and activities as they are indexed using their own contexts. I would propose to internally interpret a "Search in course ABC" as a courseid=X filter, not contextid=X
  2. A set search context autocomplete filter in the UI could bring scalability issues as I understand that it would contains all Moodle contexts' get_context_name, I guess no one better that you OU guys to check it ;)
  3. The search box would pass the page context to search/index.php (at least this is how I imagine it would work) and results will be limited to that context until the user unticks whatever filter we show as active in the UI. If we do this automatically for users I think that it is important that is clear in the UI that a contextid filter is applied. Sorry I don't have any suggestion. 
  4. Groups filter or other course-level filters can be displayed if we pass a course contextid but this is not fully solving the UI groups problem, I still don't have any idea to display this filter when no context is passed. Up to you if you want to always display it, only display it if there is a context filter or any other solution you can think of
  5. We should decide what we do in context system pages, all the stuff that does not belong to any lower context belongs to system context. I am more inclined to propose to not filter by context system in context system pages. People should still be able to apply the context system filter in search/index.php or filter by search area.

 
Average of ratings: -
Picture of Matt Porritt
Re: More proposed global search improvements
Core developersPlugin developers

Hi Sam & All,

Just wanted to link some relevant tracker issues here:

MDL-59434 Global Search: implement content aware searching / alternate results sort orders

MDL-59074 Allow Global Search plugins to override search form

MDL-59459 Global Search: Increase file indexing coverage



 
Average of ratings: -
Picture of Ralf Hilgenstock
Re: More proposed global search improvements
Core developersParticularly helpful MoodlersTranslators

Hi Sam,

from my opinion it would be fine to make the search options more configurable. Here some first ideas.

Search result levels:

- Search results are visible for public (people who are not logged in). May be intersting for people who want to use thsi for marketing purposes, also for not public courses.

- Search results are visible for logged in users even if they are not enrolled

- Search results only for users who are enroled in a course and the content is available for them

- Search results only for editing teachers and admins, managers

- Blocked for global search.

Context settings:

Site

Course category

Course  (sometimes courses are used by the Board or a Project group and content should not be visible to anybody else, even not by global search.)

Activity


Ralf


 
Average of ratings: -