Slow loading Admin pages

Slow loading Admin pages

by Emma Richardson -
Number of replies: 16
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

Ok, after just building two new servers and splitting out the database from the app server, I am still experiencing extremely slow load times, just on the admin pages (about 10 seconds).  

Running Moodle 3.9.2.  Servers both running Debian Buster - PHP 7.3, Mariadb 10.6.

Originally this was running on a single server with Debian Stretch, PHP 7.2, MySql.  

The rest of the site is suitably zippy.  If someone could give me some advice on how to see what is actually going on when I go to the Admin pages, I would greatly appreciate it.

Average of ratings: -
In reply to Emma Richardson

Re: Slow loading Admin pages

by Ken Task -
Picture of Particularly helpful Moodlers

Suggest installing MySQLTuner on both app servers and DB servers.

Run tuner from app server first and note recommended changes ... one will probably related to networking - 'skip networking'.  DB server will do a DNS lookup upon every query ... admin user has more profile therefore more queries.

DB server needs to have host restrictions ... IP address of the app server.

There could be other things ... but look into networking first since that was not involved before the split. 

'SoS', Ken


Average of ratings: -
In reply to Emma Richardson

Re: Slow loading Admin pages

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Emma

The Admin requests are known to cause more load than the non-admin ones. The non-tech answer is that the roles and permissions model of Moodle has more work to do in the roles and permissions model of Moodle. For a techy answer you profile such processes as documented in https://docs.moodle.org/dev/Profiling_PHP. For and example of the kind of results see https://moodle.org/mod/forum/discuss.php?d=411581#p1663606.
Average of ratings: Useful (2)
In reply to Visvanath Ratnaweera

Re: Slow loading Admin pages

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers
Ok, this morning I went back through this thread and finally visited the page you linked to!!! Thank you Visvanath - I think I have found that the problem is a plugin I am using. That profiler is pretty slick!!
Average of ratings: -
In reply to Emma Richardson

Re: Slow loading Admin pages

by Mahmut TEKER -
Hi,
We have exact same server software spec and after moving to Buster we have discovered that Xdebug is enabled by default on PHP (dont know why). Especially admin pages were slow. Maybe you can check it with php -v and see whether it is enabled or not.

Also you can install https://moodle.org/plugins/report_benchmark (if not allready) when we ran the benchmark it gave warning as "Processor Processing Speed" has a problem and disaapeared afer disabling Xdebug.

Best.
Average of ratings: Useful (2)
In reply to Mahmut TEKER

Re: Slow loading Admin pages

by Ken Task -
Picture of Particularly helpful Moodlers
+1 same thing on CentOS 7 ... xdebug not really needed for moodle. Not loading xdebug resolved some memory issues as well.
'SoS', Ken
Average of ratings: -
In reply to Ken Task

Re: Slow loading Admin pages

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers
Thanks for all the advice. Will start with sqltuner. Don't think this is network as problem was there before the split. Sadly, don't appear to have Xdebug - was hoping for an easy fix!
Average of ratings: -
In reply to Emma Richardson

Re: Slow loading Admin pages

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Emma

So the problem is not the slow response to Admin pages per se, but your new server duo (split web server and db server) perform below your expectations?

You said the problem was there before the split. You mean the previous single Debian Stretch performed performed equally bad as the current Buster duo? Based on what, the loading times of the Admin pages? You take measurements when the servers are idle otherwise? What is the hardware?
Average of ratings: -
In reply to Emma Richardson

Re: Slow loading Admin pages

by Alain Raap -
Picture of Particularly helpful Moodlers
Hi Emma,

What caching (server) are you using? Or do you use a PHP-FPM server? We have a double server configuration too, and with Moodle 3.9.2 we have a much better performance than with our current version (Moodle 3.5). Automated backup was much faster too. Mysqltuner could be of help too, do you cache the database in memory? That's really a performance improvement.
Average of ratings: -
In reply to Alain Raap

Re: Slow loading Admin pages

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers
Ok, more info - the problem has always been there so this is nothing new with the new setup - I was just hoping that migrating to the dual server setup would resolve the issue.
It takes a good 10 seconds to load the admin settings page and about 10-20 seconds to make changes to any admin settings. The rest of the site is fast. Set up on HyperV with 8GB Mem and 8 cores CPU. I have run Mysqltuner on both sites. Asides from it think that the moodle database columns are all of the wrong type(!), I am seeing recommendations to turn off query-cache...I do also see a reference on the db server to the fact that is dns name resolution and I should add that I am running this through a nginx reverse proxy which I am wondering if that is contributing to the problem. I have opcache enabled for php. Just reinstalled apache2 with default config - haven't played with those settings yet.
Questions - should I turn off query-cache? Could nginx reverse proxy be causing this?
Average of ratings: -
In reply to Emma Richardson

Re: Slow loading Admin pages

by Ken Task -
Picture of Particularly helpful Moodlers

The advice from tuner should have included 'turn off DNS queries' ... each request of DB server does a lookup on request ... if you have DNS issues, slow.   Run tuner from app server for a true network involved test.

Use IP address for DB host in config of moodle.

On the DB server allow ONLY the IP of the code server.

The other DB setting is normally buffer pool ... the BP has to be equal to if not slightly more than the data.   IF you can fit all of the DB into the memory of the dedicated DB server there is no disk IO ... thus faster for all users.

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: Slow loading Admin pages

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers
Well, still struggling. I added various settings to etc/mysql/my.cnf file that shut off query caching and raised the size of tmp table stuff and turned off dns queries. Looking at global variables, they were updated. No change at all except it was maybe a little bit slower. Then I discovered I could no longer log into mysql from the server as it didn't like the entries I put in the my.cnf file and I ended up having to take them all out. The documentation seems to indicate that was the correct file to update. I went back and just did a SET GLOBAL from mysql but no improvement. I am having trouble getting the skip-name-resolve entry to take. MYSQL says there is an error in the syntax - tried SET and SET GLOBAL skip-name-resolve = 1; I am beginning to think that this is not a database issue. Developer tools are telling me the initial server response time is 12 seconds...does anyone have experience with using nginx as a reverse proxy ? I am wondering if that is the bottleneck...
Average of ratings: -
In reply to Emma Richardson

Re: Slow loading Admin pages

by Ken Task -
Picture of Particularly helpful Moodlers

Set Globals are temporary .. next time db server restarted/rebooted ... those are lost.

Ok to test with on a temp basis I guess.

How are these boxen networked?   Reason asked ... Rackspace assigns 2 ip addresses to leased dedicated servers.   A public - which is mapped in DNS to access apache.   And a 10. IP ... not in DNS and used by support folks from RS.  No rules that say you can't use the 10 dots.   I had 5 RS servers Webmins linked via 10 dots.

Could do and have done same with DB server ... one client had courses offered to clients in China ... that box always had pokes and probes.   For a test, set the DB server to a 10. on another RS server ... DB server there listened on 10. IP.   No proxy ... no any special added in front of and possibly messing with security for DB.

What's purpose of Nginx Proxy ... basically to hide DB server.

You could do that in other ways!

As far as secure ... hmmmm .... newer versions of MySQL/MariaDB on could run under TLS ... all traffic encrypted.

No other host need access the DB server ... so in host table of the DB server allow only the public IP of the code server.   Also on DB server restrict ssh access to a single IP (other than localhost) .... to that of the code server.

One way to find out of Nginx proxy is the problem .... turn it off! smile

'SoS', Ken

Average of ratings: -
In reply to Emma Richardson

Re: Slow loading Admin pages

by Alain Raap -
Picture of Particularly helpful Moodlers
Just a few more questions from my side, where is your moodledata and how is it connected? NFS (we had in the past) is a very slow network device, so just asking if you use it.
And what is the response time of a page load (I usually check this with the debugger in the browser (F12))? Have you had any issues with caching? Performance boosts are
usually made after tweaking caching of the webserver and / or database.
Average of ratings: -
In reply to Alain Raap

Re: Slow loading Admin pages

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers
Moodledata is on the same drive and server as the apps server. Both servers are on a 10. network and communicating with each other that way. (I do not think this is a server to server connection issue as the same problem existing when db and app were on the same server.
The nginx reverse proxy is used for sharing a single external ip with multiple sites. I will be testing the site without that piece today...looking at developer tools when I load the admin pages, it is the initial server response time that is 12 or more seconds - once that piece clears everything is in expected ranges.
Average of ratings: -
In reply to Emma Richardson

Re: Slow loading Admin pages

by Alain Raap -
Picture of Particularly helpful Moodlers
Does it makes a difference to load the admin pages from the outside (internet) or on the inside network? Sometimes loadbalancers (like F5) can be hard to configure with your Moodle configuration (in our case). We had the same kind of problems with long load times in the beginning, it's often difficult to find the bottleneck with these problems. Logging like Splunk or your monitoring tools could be of help. We got timeouts somewhere in the network and found this in the network logfiles (searched on IP address or FQDN).
Average of ratings: -
In reply to Alain Raap

Re: Slow loading Admin pages

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers
Finally got the profiler working and pretty sure the problem is a plugin (one I actually paid to have written - coursetocal) - just emailed the developer to see if it makes sense to him!
Average of ratings: -