[3.8.4+] Users get blank page when using search function

[3.8.4+] Users get blank page when using search function

by Jeff Noel -
Number of replies: 12

After updating 3.5.9+ to 3.8.4+ on our production server, the global search hangs and loads for almost a minute before showing a blank page (See screenshot below - we have the header, but no content).

I tried forcing the indexation of all courses (12000) and made a search after that... still have the same issue.

We are not using Solr, but Simple Global Search. Everything is setup just fine (was working for years).


Any idea to fix this quickly ?




Average of ratings: -
In reply to Jeff Noel

Re: [3.8.4+] Users get blank page when using search function

by Jeff Noel -
On the same note, course categories cannot be opened up in the navigation menu anymore:


The AJAX call seems to work just fine, but we can't see the courses within the categories. I feel like it's related to the search issue.

Here's the content of getnavbranch.php when we try to open up a course category:
{"id":"expandable_branch_10_88","name":"2019-1","type":10,"key":"88","class":"type_category expandable_branch_10_88","requiresajaxloading":true,"icon":{"component":"moodle","pix":"i\/navigationitem","classes":["navicon"],"alt":"","aria-hidden":"true"},"title":"","expandable":"88","link":"https:\/\/moodle.mydomain.ca\/course\/index.php?categoryid=88","hidden":false,"haschildren":true}

Instead of loading the content, it shows the empty arrow icon and does nothing.
Average of ratings: -
In reply to Jeff Noel

Re: [3.8.4+] Users get blank page when using search function

by Ken Task -
Picture of Particularly helpful Moodlers

Quick fix????!!!

Upgrading moodle 3.5.x to 3.8.4+ ... did you have to upgrade your DB also?

Both of your problems kinda sounds like DB server returning data issue.

Soooo ... debugging shows?

How about server error logs - web as well as DB logs?

The simple search does have 2 scheduled task now.

\core\task\search_index_task

\core\task\search_optimize_task

What do you see if you run those manually from admin/tool/task/cli/ and the schedule_task.php scripts option to run just those?

php schedule_task.php --execute="\core\task\search_index_task"

php schedule_task.php --execute="\core\task\search_optimize_task"

'SoS', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: [3.8.4+] Users get blank page when using search function

by Jeff Noel -
Hi Ken,

Tasks are running fine. I have deleted the whole index and restarted the indexation process. Once it was all done I ran the "search_optimize_task" task and it just went blazingly fast:

Execute scheduled task: Optimisation de l'index de recherche globale (core\task\search_optimize_task)
... used 3 dbqueries
... used 0.073920011520386 seconds
Scheduled task complete: Optimisation de l'index de recherche globale (core\task\search_optimize_task)

I have about 230K files, 150K forum posts, 12000 courses...
I tested a few things and realized that I don't have a blank page when logging in as my admin user. Whenever I'm a standard user or a "limited manager" (can view anything, anywhere but cannot edit courses), I land on a blank search page for the same search term.

You might be right about the DB issue...

I checked the web server logs (and moodle logs) and everything seems fine there.
We had a recent OS switch and changed to another VM for the DB (everything running fine for the past 6 months) and your post made me realize that the log settings were wrong on it.

Kind-of bummed that I can't simply reboot the MySQL instance right now since it's a production server. It'll have to wait.

Thanks for your cues! I'll report back once I have DB logs.
Average of ratings: -
In reply to Jeff Noel

Re: [3.8.4+] Users get blank page when using search function

by Ken Task -
Picture of Particularly helpful Moodlers

In 6 months time, table for 'who done it' (considering the number of courses/users) could grow very very large ... as other tables as well.   Unless they have setting to cull them once reaching thresholds.

Simple Search in moodle could also grow and those tables won't be culled.

Now when you say logging .. are we talking about Moodle logging or DB server logging?

Have made that mistake after migrating a site and forgetting I turned on true DB server logging to monitor for a bit ... got bushswhacked into another project and forgot to monitor then turn OFF true DB server logging.   'client' complained one day about 'slow' ... well, yeah, duh!   Dummy me! :|

'SoS', Ken


Average of ratings: -
In reply to Ken Task

Re: [3.8.4+] Users get blank page when using search function

by Jeff Noel -
Hi Ken,

Yes, I was talking about the DB Server logging. We keep putting them on and off depending on our needs and looks like they were off while I was debugging this whole issue.
Now that they're on though, I got an update to my issue:

I have been able to reproduce the problem on my test server. I then upgraded PHP from 7.1.14 to 7.4.8... the page still takes anywhere between 15-30 seconds to load, but at least it shows results now.

The issue seems to be happening to people that:
- Are not admins
- Are not students
- Have a role where they can see and access hundreds or thousands of courses (limited manager, etc.).

Per the DB query logs, it seems like Moodle checks the permission level for each course context, thus making thousands of queries to display the course list in the search results.



If I look at table sizes, search_simpledb_index is only 306.51Mb in size (about 1 million rows). I ordered all my tables by size and it,s the 11th biggest (files, question and question_attemps, logstore_standard_log being way bigger, etc.).

I don't think it's a disproportionate size for an index considering the amount of content we have in this instance.

I'll test the newer PHP version on my production server to see if things get better.

As always, I appreciate your help Ken. Feel free to chime in if you have any other ideas !
Average of ratings: -
In reply to Jeff Noel

Re: [3.8.4+] Users get blank page when using search function

by Ken Task -
Picture of Particularly helpful Moodlers

Version 3.8.x of Moodle's PHP Max version is PHP 7.3, I thought.  You've PHP 7.4.x.

See: http://www.syndrega.ch/blog/#php-and-dbms-compatibility-of-major-moodle-releases

There are some settings for global search in:

admin/settings.php?section=manageglobalsearch

Searchable courses

If set to search within enrolled courses only, course information (name and summary) and course content will only be searched in courses which the user is enrolled in. Otherwise, course information and course content will be searched in all courses which the user can access, such as courses with guest access enabled.

Include all visible courses

If enabled, search results will include course information (name and summary) of courses which are visible to the user, even if they don't have access to the course content.

MIght search web server error logs - mysql has gone away or similar + PHP time for a script to run or memory a script can consume.   Blank screens returned to user usually means some resource/setting exhausted or too low.

Also, turning on MySQL logging full time and not ever turning it off could lead to some issues.

One can turn that on, watch in realtime (multitail), then turn it off:

slowlogon
# to turn on:
mysql -e ‘set global log = 1′ -u root -p’pass’
mysql -e ‘set global log_slow_queries = 1′ -u root -p’pass’
mysql -e ‘set global slow_query_log = 1′ -u root -p’pass’
multitail -i /var/log/mysql/mysql-general.log -i /var/log/mysql/mysql-slow.log;

The multitail line will show in realtime.

slowlogoff
# to turn off
set global slow_query_log = 0
set global log_slow_queries = 0
set global log = 0;

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: [3.8.4+] Users get blank page when using search function

by Jeff Noel -
Hi Ken,

We're only using MySQL logs as a temporary measure since we're also aware of the issues that can cause (especially performance wise).

PHP 7.4 is supported by Moodle 3.8 branch since 3.8.3 (see notes in server requirements here: https://docs.moodle.org/dev/Moodle_3.8_release_notes)

We simply set the general MySQL log to output to a log file on our logs drive.
I'll start tracking issues related to lack of resource (whether it is RAM, CPU or something else) through the logs as you mentioned. Thanks for the cue.

We have searchable courses set to search for all courses, since our "limited managers" are not enrolled automatically in the various departments they manage, they have access to all courses but are not enrolled.

I'll post whenever I find something. Thanks again for your time !

Jeff
Average of ratings: -
In reply to Ken Task

Re: [3.8.4+] Users get blank page when using search function

by Jeff Noel -

BIIIIIIIIG progress today !

After my last post (below or above) where I found that there was millions of calls made to check user permissions... one of our team member though about /course/search.php.

We tried manually querying /course/search.php?q=search_term_whatever with the users experiencing timeouts and the results were instantaneous. 1.5 second load time.

Meanwhile, the page /search/index.php?q=search_term_whatever still does timeouts and takes forever to load... for users with the specific roles and permissions mentioned earlier.

I could bypass the global search terrible performance issues by running a javascript script that changes the form action in /course/search.php to /course/search.php (itself) instead of targetting /search/index.php. Results are lightning fast now.

Should I open a ticket about it ? Does that trigger any idea of what could be the issue ?
I'd still want to fix the global search performance for these people if feasible.

Jeff

Average of ratings: -
In reply to Jeff Noel

Re: [3.8.4+] Users get blank page when using search function

by Justin Hunt -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Just a thought, but what is your PHP max_execution_time set to?
Average of ratings: -
In reply to Justin Hunt

Re: [3.8.4+] Users get blank page when using search function

by Jeff Noel -
Hi Justin,
max_execution_time is set at 3000 seconds right now.
We set this value after having issues with excessively long upgrade times.
Average of ratings: -
In reply to Justin Hunt

Re: [3.8.4+] Users get blank page when using search function

by Jeff Noel -

Hi,

I took a snap from XDebug and it seems like a LOT of calls are made to check user capabilities, user admin status (when they're not an admin), etc.
The page times out after ~4-6 minutes.


Average of ratings: -
In reply to Jeff Noel

Re: [3.8.4+] Users get blank page when using search function

by Jeff Noel -
Before checking all of this I thought my theme could be causing the problem, but it doesn't seem to be the issue at all !

This issue only happens with users that have:

* Access to all courses (read only)
* The right to view all courses (even invisible ones)

And make a broad search (terms like "2020" or "2020-2" when our course names are all named similar to this: 2020-2-ABC1234-00
Average of ratings: -