New Moodle 3.3 service super slow

New Moodle 3.3 service super slow

by Jason Menard -
Number of replies: 13

Hi all,

We're spinning up our new Moodle server for the academic year; running Moodle 3.3 from GitHub on a dual-core CentOS 7 VMware VM with 16GB of RAM. This is a better-resourced VM than what we've used in the past, so it should be speedy. This time the PostgreSQL database is running on the same VM as Moodle; normally we have it running on another server that supports multiple databases.  

We're running PHP 7.0.19, and PostgresSQL 9.3. So here's the problem-- pageload times are ridiculously long, especially on the front page. Pageload times can be up to a minute to 1.5 mins. the main culprits are things loaded by /ajax/service.php, waitimes for message_popup_get_unread_popup_notification_count, and core_fetch_notifications take the most time (the former usually taking a minute to 1.5 minutes, the latter ~30S). We tried downgrading to PHP 5.6.5, thinking PHP 7 might confuse Moodle somehow and got nothing.

We have a pretty stable Moodle installation and support process; we stick to Core and standard themes, and don't add much custom code (the only thing we do install is Javascript and doesn't really impact Core). The only thing we've done different this year is installing from GitHub and using Ansible to configure the installation, and we built the Ansible script by working from our normal installation docs to get our test server going.

Based on past experience, this server should just work nicely, but clearly something's not right and we've gone down a few rabbit holes that just go deeper and don't reveal a clear place where something is misconfigured. We'd really like to resolve this quickly, since I have faculty that are ready to start moving into the new server, and I can barely get it fully configured, much less invite them to use it. As it is the server is not functional for endusers.

Any advice folks can provide will be much appreciated.

Thanks,

Jason



Average of ratings: -
In reply to Jason Menard

Re: New Moodle 3.3 service super slow

by Jason Menard -

We actually identified the issue. We had to create a new iptables rule to better talk to our NetApp volumes that use qtrees. So not a Moodle Core issue, or PHP, more a talking-to-NetApp issue.

Average of ratings: Useful (1)
In reply to Jason Menard

Re: New Moodle 3.3 service super slow

by Jeff White -
Hi Jason,


Was all file access to the Netapp impaired or was it just this ajax/service.php? I am rather curious as it would confuse the heck out of me for just this to act up if your file storage was not acting up to par. 

Average of ratings: -
In reply to Jeff White

Re: New Moodle 3.3 service super slow

by Alain Raap -
Picture of Particularly helpful Moodlers

We had the same issue with long wait time/timeouts on our server (3.1.5+) in a lab environment without routing to the internet.
We found out that the problem was that we hadn't used the proxyserver in Moodle (in Server -> HTTP). After configuring this the problem was solved! The Ajax call via XML-RPC in the service.php was the bottleneck in this case. 

Average of ratings: Useful (3)
In reply to Alain Raap

Re: New Moodle 3.3 service super slow

by Benoît Lathière -

We have the same problem: often, the 'service.php' called in Ajax is very slow: delay = 30 seconds..

We did add a proxy server but the problem still appears.

Is a cache engine (ie: memcached) really required ? Is the 'service' result cached in Moodle?

in the admin (Server -> HTTP), what the most powerful method, "HTTP" or "SOCKS5"?

Thanks for your help.

Average of ratings: -
In reply to Benoît Lathière

Re: New Moodle 3.3 service super slow

by Alain Raap -
Picture of Particularly helpful Moodlers
Benoît we used HTTP and that works fine for us. We don't use memcached (yet), but it could be a performance improvement. We found out that service.php was the reason of the performance problem with the help of Firebug (in Firefox).
Average of ratings: -
In reply to Alain Raap

Re: New Moodle 3.3 service super slow

by Benoît Lathière -

Thanks for your infirmations.

It seems we have a speed problem on the NFS mount for moodledata sad I will try to store sessions in memcached.

Have a good day.

Benoit.

Average of ratings: -
In reply to Benoît Lathière

Re: New Moodle 3.3 service super slow

by Alain Raap -
Picture of Particularly helpful Moodlers
Benoît Lathière we had the same problem with performance of the NFS share and changed some of the settings of the share. Just try to increase some of these values, rsize=65536,wsize=65536. We use nfs4 btw
Average of ratings: -
In reply to Benoît Lathière

Re: New Moodle 3.3 service super slow

by Alain Raap -
Picture of Particularly helpful Moodlers

Benoit we use Memcached and local caching on the webtier now and that's a huge performance improvement.

Don't cache on your (slow) NFS share!

Average of ratings: Useful (1)
In reply to Alain Raap

Re: New Moodle 3.3 service super slow

by Benoît Lathière -

Hello Alain,

We do that too: Memcached for sessions, Redis for application cache, in local on each node (so, no SPOF).

"Don't cache on your (slow) NFS share!" I agree with You!

And check the scheduled tasks: some plugins have bad scheduled huge tasks (ie: use_stats) and must run only the night.

Best regards.

Average of ratings: Useful (2)
In reply to Alain Raap

Re: New Moodle 3.3 service super slow

by Jerry Lau -

We are experiencing the same situation albeit an upgrade to 3.2.4+ build from our 3.0.10+.

Running on RHEL 6.6 64-BIT on php 5.6.25 (Zend OPcache) + MySQL 5.5.41

No clustering, failover, no proxy services, etc... nothing fancy and noticed that the MySQL spiked to 67.5% cpu usage.

On VMWare with 16 GB of ram...

what else could be causing the bottleneck?

Thanks




Average of ratings: -
In reply to Jerry Lau

Re: New Moodle 3.3 service super slow

by Chris Fryer -

What values does your MySQL configuration file have for innodb_buffer_pool_size and friends?

https://dev.mysql.com/doc/refman/5.5/en/innodb-buffer-pool.html



Average of ratings: -
In reply to Chris Fryer

Re: New Moodle 3.3 service super slow

by Jerry Lau -

Also noticed the very high amount of http traffic so much so that no one can log in.

My moodledata is pretty big .. about 925 GB and ram is 16 GB and db size is about 275 GB


In mysql query...

mysql> SHOW VARIABLES LIKE '%innodb_buffer_pool_size%';

+-------------------------+-----------+

| Variable_name           | Value     |

+-------------------------+-----------+

| innodb_buffer_pool_size | 134217728 |


In my.cnf (I don't have that value)


[mysqld]

#datadir=/opt/rh/mysql55/root/var/lib/mysql

datadir=/srv/mysql

socket=/var/lib/mysql/mysql.sock

# Disabling symbolic-links is recommended to prevent assorted security risks

symbolic-links=0

max_allowed_packet=1G

wait_timeout = 31536000

connect_timeout=100

net_read_timeout=3600

net_write_timeout=3600


[mysqldump]

max_allowed_packet=1G

[mysqld_safe]

log-error=/var/log/mysql55-mysqld.log

pid-file=/opt/rh/mysql55/root/var/run/mysqld/mysqld.pid


Should I put in the "innodb_buffer_pool" in my my.cnf to say, 12 GB?

innodb_buffer_pool = 12G;




Average of ratings: -
In reply to Jerry Lau

Re: New Moodle 3.3 service super slow

by Alain Raap -
Picture of Particularly helpful Moodlers

Jerry how is your caching configured and your temp directory?

Average of ratings: -