Global Search 2 - Zend Lucene indexing performance

Global Search 2 - Zend Lucene indexing performance

by Tomasz Muras -
Number of replies: 1
Picture of Core developers Picture of Plugin developers Picture of Plugins guardians Picture of Translators

Hello,

As promised in the previous discussion and in MDL-31989, I've tested the performance of Zend_Search_Lucene (so far only indexing performance). Full details are in my blog post but here is the summary:

  • Zend Lucene indexes about 50k of text per second when indexing 1GB of data. The speed of adding new documents will be slowly degrading as index grows
  • There are no memory leaks in the current code - the memory usage will be constant no matter how many documents will be indexed at once

I think that the speed of indexing is acceptable and I'm moving forward with Zend_Search_Lucene as search solution.

cheers,
Tomek

Average of ratings: -
In reply to Tomasz Muras

Re: Global Search 2 - Zend Lucene indexing performance

by Tomasz Muras -
Picture of Core developers Picture of Plugin developers Picture of Plugins guardians Picture of Translators

Hello,

another update on Global Search work.
Following the indexing speed tests, I've started testing the "optimize index" operation. Optimizing index is pretty important, as the performance will keep degrading when documents are removed or added. Unfortunately as it turns out, the optimization of big indexes is a problem for Zend Lucene. I tried hard tweaking Lucene settings but I couldn't get it to successfully optimize big (1GB) index. I've also spent couple of days trying different versions of luke and lucli - the tools that could be used to optimize Lucene index "outside" of Moodle but unfortunately they read index but corrupt it during optimization.

The work-around is to delete index and re-build it from scratch and this will be required for bigger installations.I don't think this is a show-stopper but please comment if you feel different.

I wanted to have Global Search ready for Moodle 2.3 but this will not be the case - there is not enough time for me to finish it, so the goal has been moved to Moodle 2.4.

cheers,
Tomek

Average of ratings: Useful (1)