Moodle extremely slow

Moodle extremely slow

by Cosmas Kasumba -
Number of replies: 61
Picture of Testers

Please our Moodle is extremely slow, maximum number of users in a day is about 200, when traffic is extremely high. Users registered on the system is 5000. The server has the following

Php 7.2

Moodle version 3.8.2 (upgraded yesterday)

CPU 8 core /16 thread

RAM 30gb 

SSD drive

Average of ratings: -
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Can you provide more information.
In reply to Marcus Green

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
What information exactly must I add? Opening a page takes about 3 - 7 minutes
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
When it was on shared hosting it was much faster as compared to how it is now on the dedicated server with the specifications above
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators

Was this on the shared server: Moodle Error 500 https://moodle.org/mod/forum/discuss.php?d=400066 ? Did you find out why?

And this escalation happened just after the upgrade yesterday?

In reply to Visvanath Ratnaweera

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
@Visvanath Ratnaweera yes, the Error 500 happened on the shared server, the Error 500 was caused by the high resource usage of the CPU, everytime the error came our CPU usage was closer or above to the 25% threshold we were allocated by hostgator. We had also passed the number of files were allowed to keep on the server, our account ended up being suspended and as result we had to pay for a dedicated server. Before the migration of yesterday(probably 2 weeks ago) we had to move the files to a server which had 4 Core/ 8 thread. Its when I realised that moodle was too slow and the teachers could not mark online, students were complain they basically couldnt do their work, and the first thought was the server size and we decided to update again which now led me to this issue. Quite a story..
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Is it a VM / something else? You say the disk is SSD... is the 'moodledata' stored there as well or somewhere else?

Run this report - Performance_overview ??

Consider installing and running - https://moodle.org/plugins/report_benchmark - what's the output?
In reply to Howard Miller

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
This is the output from the benchmark report
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
I see my main problem is on the login, according to the benchmark
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
That's strange... given everything else is ok, my first thought is "what's different about your system?". Do you have a non core theme (try again with boost)? Do you have any optional plugins?
In reply to Howard Miller

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
Yes, Am using eguru as the theme, I need to check if I still have a few more optional plugins I had removed most if not all of them, let me quickly try with boost then we see.
In reply to Howard Miller

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
This is what Am getting with boost https://snipboard.io/fam3l0.jpg
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Hmmm.... I'm quickly running out of ideas. I'm hoping somebody else jumps in. I'll keep thinking...
In reply to Howard Miller

Re: Moodle extremely slow

by Ken Task -
Picture of Particularly helpful Moodlers

Having 'too much fun', Howard? smile

2 cents ... both areas of Moodle bench relate to users and authentications ... so how are users authenticating?

Moodle is backended by DB server ... suggest, IF MySQL or MariaDB ... installation of MySQLTuner ...

https://github.com/major/MySQLTuner-perl

Does mdl_user table have any issues?   Is the table and all the columns in that table match up with character set/collation (variable in config.php).  How about mdl_log_store_standard_log ... the 'who done it' table?

Scripts in moodlecode/admin/cli/ will help check DB.

'SoS', Ken

In reply to Ken Task

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
which script in moodlecode/admin/cli/ will help check DB. do I check for collation?
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Ken Task -
Picture of Particularly helpful Moodlers

Still strongly suggest MySQLTuner. smile

'SoS', Ken

In reply to Ken Task

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
Ken is it fine to PM you?
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Ken Task -
Picture of Particularly helpful Moodlers

I take it PM has to do with MySQLTuner? which is NOT a Moodle plugin but an operating system thang.   You probably have an icon for something like MySQL Bench in cPanel ... but that's not near as good as Tuner.

Do you have a Terminal icon in your cPanel?

Hostgater?  Hmmm ... guess they don't think you will ever need to tweak your DB's.... too bad! sad

'SoS', Ken

In reply to Ken Task

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
Yes, its in connection with MySQL Tuner. On the cPanel I dont have MySQL bench but the terminal is there
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Ken Task -
Picture of Particularly helpful Moodlers

More than likely, but dunno for sure, when you click the terminal icon a new browser window will appear and in it a 'termanal' shell.

Has a prompt like 'accountlogin>'

And at the prompt one can type commands.

Try some for info purposes:

whoami [ENTER]

pwd [ENTER]

Biggy:

sudo -s [ENTER]

You will be prompted to provide the password of the account.  Successfully entering that, prompt should change to:

root:>

which means you should have no trouble installing MySQL Tuner.

To exit sudo and terminal, type:

exit [ENTER]

with sudo have to do that twice:

exit [ENTER]

'SoS', Ken


In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators

This is a powerful dedicated server - 8 Core / 16 Thread Intel Xeon-D CPU, 30 GB RAM, 1 TB SSD, Unmetered Bandwidth (https://www.hostgator.com/dedicated-server -> Enterprise) - still extremely slow even with the 200 users per day! So something is extremely wrong.

Sorry to say this, but from your approach, and the fact that you want Cpanel instructions, you are unfamiliar with the Unix Shell and avoid it. Remember, this is a dedicated server, the hoster can't restore the system. What if you do more damage and lose the whole system? For me the risk it too high and too much of a responsibility. I'm glad that I haven't given any specific instructions here, if there are, please ignore them!

In reply to Visvanath Ratnaweera

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
Well thank you for the response, indeed Am not good with the Unix Shell and on the server, I have mostly worked using the Cpanel that's why I had talked about the Cpanel, as for the risks involved, Am well aware of the risks involved and I take extra caution before trying out what has been suggested and like I mentioned earlier the speed has improved a lot all thanks to the people like yourself who have taken their time to help. I am more than willing to learn from people like you.
In reply to Ken Task

Re: Moodle extremely slow

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
The character set etc. won't account for this sort of slowness. In fact you'd have to put some effort in to "tune" MySQL down to this speed.

However, that doesn't mean that there's nothing wrong with the database.
In reply to Howard Miller

Re: Moodle extremely slow

by Ken Task -
Picture of Particularly helpful Moodlers

Just tryin' to cover all the bases!  Haven't seen a moodle server yet that couldn't benefit from DB tweaks ... minor as they might be! smile

'SoS', Ken

In reply to Ken Task

Re: Moodle extremely slow

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Agreed - but that's unlikely to be the problem here.

However, in the absence of any other leads it's worth running anything and everything just to see what shows up.
In reply to Ken Task

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers

Not too sure what to look for under MySQL tuner but i see the pic below suggesting that the maximum memory usage is dangerously high

Attachment db.jpg
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Ken Task -
Picture of Particularly helpful Moodlers

Did you run tuner with superuser for DB credentials? The DB user in your config.php of Moodle may not have same privs.

That's unusual ... normally, Tuner finds MySQL/MariaDB could/should use more!

So get into the terminal again and type: top [ENTER]

At the 'top' of 'top', it shows memory.

What does that look like?

Also, further up in Tuner output there is a 'Performance Metrics' section.

Maximum reached memory usage
Maximum possible memory usage

There is a section on InnoDB Metrics

InnoDB buffer pool / data size

InnoDB buffer pool instances

'SoS', Ken


In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Ken Task -
Picture of Particularly helpful Moodlers

Now that you know you have terminal access to server, there are some command line only php scripts in moodlecode/admin/cli/ that are specifically for MySQL/MariaDB.

Issuing them with no switches brings up help:

php mysql_compressed_rows.php

For info:

php mysql_compressed_rows.php -i

Database version:      5.7.30
Database name:         moodle
Database engine:       InnoDB
innodb_file_per_table: ON
innodb_file_format:    Barracuda

BTW, mysql did have an update recently bringing version to 5.7.30

php mysql_collation.php -l

will list all tables and columns in those tables collations and end with a summation - all tables should have collation that matches what's in config.php for site.

Of interest/attention would be mdl_users and mdl_logstore_standard_log.

Either of those are related to what is shown as a performance issue in bench.

'SoS', Ken

In reply to Ken Task

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers

php mysql_collation.php -l

I don't see any info on collation in the config.php. Is it ok with you @Ken I send you the log details so you can check it . 

Attachment bara.jpg
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Ken Task -
Picture of Particularly helpful Moodlers

config-diet.php file for a 3.8.highest version of moodle has this in
$CFG->dboptions = array(
.... other settings here do not use this line ....
'dbcollation' => 'utf8mb4_unicode_ci', // MySQL has partial and full UTF-8
                                // support. If you wish to use partial UTF-8
                                // (three bytes) then set this option to
                                // 'utf8_unicode_ci', otherwise this option
                                // can be removed for MySQL (by default it will
                                // use 'utf8mb4_unicode_ci'. This option should
                                // be removed for all other databases.

Using:
php mysql_collation.php -a
You should see  utf8mb4_general_ci listed as well as
one that is 'default':
 utf8mb4_unicode_ci (default)

Tuner
InnoDB buffer pool is 128M ... data is 3.5G
Highest usage of available connections ... default is 151
IF you ever have many students in taking a quiz at one time
150 could come into play ...

DB server on same box as Apache/php-fpm is somewhat a balancing act.
DB performs better if it can use more memory ... but not overly so
that fewer php-fpm's can launch to handle user request of site.

In my.cnf (config file for MySQL) for your server and tuner output,
the following:

skip-name-resolve=1
innodb_buffer_pool_size=3.5G
innodb_buffer_pool_instances=3

restart mysql and check for errors

'SoS', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
Please help on changing the innodb_buffer_pool_size=3.5G I can't seem to get it right.
In reply to Ken Task

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
I did almost all the instructions you gave me and I must add that the site is now very fast(unfortunately benchmark giving an error: cannot write to database), but i still need help on these two:
  1. innodb_buffer_pool_size=3.5G (Not too sure how to save it, I tried opening with vim but getting the error that username not a sudo doer.
  2. innodb_buffer_pool_instances=3( I can't see this one in my.cnf) 
  • NOTE: I have also put the site on https
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Ken Task -
Picture of Particularly helpful Moodlers

Both of those settings were not present in my.cnf file.

In that file you might have sections like [mysqld] and all setting under that apply to server.  Below might be [mysqlclient] and all settings below that apply to clients.

So you can't add those lines just anywhere in my.cnf

The issue with sudoers ... that probably correct ... user you login with in cPanel is also the user of the terminal shell.  Providers often do not provide to customers superuser creds to DB server.  But they are there ... have to be.

Don't know how GoDaddy does that ... so contact GoDaddy Tech support.

Think you *might* have to tweak your web server/php-fmp settings to launch fewer copies to make sure we have a balance of memory usage.

While tech support if fixing access to my.cnf for ya, may as well ask about those settings as well.

'SoS', Ken

In reply to Ken Task

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
Thank you, let me go through and check. On the server I have root access, would that be the same as superuser?
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Ken Task -
Picture of Particularly helpful Moodlers

As a matter of security, the shell root user and password is not the same as superuser in DB.   DB superuser might say 'root' but that's not really the shell root user and not the same password either.

Again ... *** contact GoDaddy Helpdesk ***.

'SoS', Ken


In reply to Ken Task

Re: Moodle extremely slow

by Ken Task -
Picture of Particularly helpful Moodlers

Were you able to get any help from GoDaddy Technical on superuser credentials for DB server on your dedicated to you only VPS?

You've reported server is faster now, but without the last steps server is still not as optimal as it could be and if usage/users increase, you might slide back into 'slow'.

'SoS', Ken

In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Ken Task -
Picture of Particularly helpful Moodlers

You should be able to copy text clips of log to clipboard, paste into Notepad or Textedit on your local machine.   Then come back here to paste those clips.   I don't want to be sent what could be a Gig or more log file.  Am sure you can understand why! :_

'SoS', Ken

In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
  1. Login time for the guest 35.9 sec, acceptable limit 0.3 sec (320 times longer)
  2. Login time for a fake user 29.6 sec, acceptable limit 0.3 sec (100 times longer)

This can't be a dedicated server! Moodle BenchMark doesn't say anything about scalability, but when it finds an error, you have to take it seriously.

So the story is, you had a shared server hitting the limits, migrated to a dedicated server 1 two weeks ago, still "insufficient", migrated to dedicated server 2 two days ago. This is the result? Talk to the hosting provider.

N.B. What are these "snapboard" screen-shots? Why can't you upload them here? Are you sure the visitors will find those images tomorrow? (A forum is a knowledge base, which others access - not only for today!)

In reply to Visvanath Ratnaweera

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers

My apology on the use of snapboard, I didnt think of the visitors, will definitely load them here next time. Let try to tuner MySQL and if no improvement then I can talk to the host.

In reply to Howard Miller

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
Yes the moodledata is stored on the same server
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

Where is the database?  Could this be a database issue?

In reply to Emma Richardson

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
Hi Emma, Am not sure I fully understand the question on where the database is. Its on the same server as the moodle files
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers
That would answer that question then. You mentioned you are on HostGator- do they have logs? I have seen some companies limit bandwidth or server calls - you would normally see some server log notices regarding that? Oh, in Moode, by any chance is Theme Designer mode on? If so, turn that off...
In reply to Emma Richardson

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers

Theme designer is off, as for the log, their advert says no limit on the bandwidth but let me look up and see if I can see the logs

In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Ken Task -
Picture of Particularly helpful Moodlers

Do you have a WAF (web application firewall) now that you did not have before?  A WAF would watch all request to server but, would think, especially with login.  Are your students using smartphones more than before? (webservices turned on, I assume)

'SoS', Ken

In reply to Ken Task

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
Yes students are using smartphones more than before as all classes are now being done online. For security the only one we have that I know of is sitelock which we had even on the previous hosting.
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Ken Task -
Picture of Particularly helpful Moodlers

Does 'sitelock' keep logs?

Check them out.

Mox nix if you had it before or not ... that was then ... this is now! ;)

'SoS', Ken

In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
FEEDBACK: Now we are now constantly getting the following error
Attachment mem.jpg
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Ken Task -
Picture of Particularly helpful Moodlers

This one is a local browser error.   Not Moodle.

'SoS', Ken

In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Alain Raap -
Picture of Particularly helpful Moodlers
Hi, have you any caching installed on your server?
Redis is a good caching server that can speedup your
performance.

What’s the size of the database? If loaded in memory (InnoDB) you’ll need to have enough Memory on your server.
In reply to Alain Raap

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
We have Memcache installed,
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Cosmas Kasumba -
Picture of Testers
Update on the moodle. I did all the turning, the site running fast, the problem comes when writing a test, right now I had given students a test to write and 24 students started to write and now the site is timing out. When students are to writing the test we are able to have a lot of students on without any problem, the only problem is on the test. Any idea how I can resolve it. As I write this post the site is off cause of less than 50 students who are trying to write.

EDIT: 127 students online, system keeps coming on and off (ERR_CONNECTION_TIMED_OUT)
Attachment test.png
In reply to Cosmas Kasumba

Re: Moodle extremely slow

by Ken Task -
Picture of Particularly helpful Moodlers

So we've come full circle.   Error screen isn't coming from Moodle thus not reaching Moodle.   Sent you a PM.   Please respond to that.

'SoS', Ken