Laggy site

Laggy site

by Keithz Pecayo -
Number of replies: 10
Our moodle site becomes slow when too many users are using it simultaneously and it will get this error eventually 

Error: database connection failed 

I'm talking about 100 to 300 users on a daily basis. Below are the specs and the configuration of the server.




Should we get an upgrade or there's something wrong with the configuration? Any help will be appreciated. thanks

Average of ratings: -
In reply to Keithz Pecayo

Re: Laggy site

by Ken Task -
Picture of Particularly helpful Moodlers

Would help to know Moodle version, MySQL version, PHP version.

Run the following query with superuser credentials to DB server:

mysql> show variables like 'max_connections';

and

mysql> show variables like 'innodb_buffer%';

Am willing to bet Bitnami may not have a .sh script that checks DB settings and offers suggestions for tweaking DB server settings so suggest installing MySQLTuner (a perl script) and run it with superuser credentials.   It will make recommendations as to needed tweaks.

https://github.com/major/MySQLTuner-perl

See an example of output of tuner ...

https://www.linode.com/docs/guides/how-to-optimize-mysql-performance-using-mysqltuner/

'SoS', Ken


In reply to Ken Task

Re: Laggy site

by Keithz Pecayo -
Hi ken,

TY for the reply below are the screenshot for the queries above:

mysql> show variables like 'max_connections';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 151 |
+-----------------+-------+
1 row in set (0.01 sec)

mysql> show variables like 'innodb_buffer%';
+-------------------------------------+----------------+
| Variable_name | Value |
+-------------------------------------+----------------+
| innodb_buffer_pool_chunk_size | 134217728 |
| innodb_buffer_pool_dump_at_shutdown | ON |
| innodb_buffer_pool_dump_now | OFF |
| innodb_buffer_pool_dump_pct | 25 |
| innodb_buffer_pool_filename | ib_buffer_pool |
| innodb_buffer_pool_in_core_file | ON |
| innodb_buffer_pool_instances | 1 |
| innodb_buffer_pool_load_abort | OFF |
| innodb_buffer_pool_load_at_startup | ON |
| innodb_buffer_pool_load_now | OFF |
| innodb_buffer_pool_size | 268435456 |
+-------------------------------------+----------------+
11 rows in set (0.00 sec)

For the tuner will get into it. 

PHP 7.3.7 
MySql Ver 8.0.16
Moodle 3.7.1 (Build: 20190708)
In reply to Keithz Pecayo

Re: Laggy site

by Ken Task -
Picture of Particularly helpful Moodlers

Since you've reported database connections issue ...

The queries are very appropriate:

max_connections is default 151 ... think tuner might tell you it needs to be higher.

Also ... innodb buffer pool size and instances ... am certain Tuner will have something to say about them! smile

Forgot to ask is this server is an all-in-one .. apache + mysql on same server.   If so, tweaks to tuning MySQL's memory needs to fit with what apache is using/configured to use.

So one more command:

free -h

You might also wanna look at 'top' ... the top of top and how mysqld is behaving in list of processes - espeically when server begins to be hit by many students.

'SoS', Ken

In reply to Ken Task

Re: Laggy site

by Keithz Pecayo -
Yes it's an all-in-one server.






In reply to Keithz Pecayo

Re: Laggy site

by Ken Task -
Picture of Particularly helpful Moodlers

1.9  Gig total memory to work with?   Wow!

See mysqld 101% of CPU?

@Visvanath - what do you think?   That enough to handle very many users?

'SoS', Ken

In reply to Ken Task

Re: Laggy site

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Ken

Yep, 138 online users (last 60 min) are on the high side for a 2 vCPU, 2 GB RAM VPS. Still, there is a huge margin of error, since we don't know what they are doing.

Still, the output of 'top' is puzzling, the server has RAM under control (no swapping). As you pointed out 101% mysqld is prohibitively high. As a rough estimate 33% is OK, dividing CPU power amoung the database, web server and PHP. Necessary to investigate why mysqld consumes it all. (@OP, did you go through the forum docs?)

Another odd thing: The name of the process is not mysqld but mysqld.bin?

The OP has blinded the top of 'top': Where are the load averages? number of tasks? zombies? CPU utilization breakdown?

I fear, one needs to dive in to the Bitnami story. I won't be available for that. See my reply to the OP https://moodle.org/mod/forum/discuss.php?d=425544#p1713729.
In reply to Visvanath Ratnaweera

Re: Laggy site

by Keithz Pecayo -

Hi Ken, Hi Visvanath

TY for your patience, i've restarted the server


Also run up the tuner, below are the recommendations:

-------- Recommendations ---------------------------------------------------------------------------

General recommendations:

    Control warning line(s) into /opt/bitnami/mysql/data/mysqld.log file

    Control error line(s) into /opt/bitnami/mysql/data/mysqld.log file

    MySQL was started within the last 24 hours - recommendations may be inaccurate

    Reduce your overall MySQL memory footprint for system stability

    Dedicate this server to your database for highest performance.

    We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.

             See https://dev.mysql.com/doc/internals/en/join-buffer-size.html

             (specially the conclusions at the bottom of the page).

    Consider installing Sys schema from https://github.com/mysql/mysql-sys for MySQL

    Consider installing Sys schema from https://github.com/FromDual/mariadb-sys for MariaDB

    Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: https://bit.ly/2TcGgtU

Variables to adjust:

  *** MySQL's maximum memory usage is dangerously high ***

  *** Add RAM before increasing MySQL buffer variables ***

    join_buffer_size (> 256.0K, or always use indexes with JOINs)

    innodb_buffer_pool_size (>= 46.1G) if possible.

    innodb_log_file_size should be (=32M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.


In reply to Keithz Pecayo

Re: Laggy site

by Ken Task -
Picture of Particularly helpful Moodlers

Bottom line ... server is underpowered to run your moodle.

Total memory shows 2Gig ... and tuner recommends increasing things that would put your server into a completely unstable state.

this item:

innodb_buffer_pool_size (>= 46.1G) if possible

is not possible ... you only have 2G memory for everything.

So it's time for you to investigate what options you have with service provider ... where you host.

No ... I do not know any work-around and won't even try to suggest any! smile

'SoS', Ken

In reply to Keithz Pecayo

Re: Laggy site

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Freshly started, load average jumps to 1.45 (out of 2)? Not good? Does the column CPU% add up to 145%? (I don't know, why you keep on pruning the output. Now the head is there, but the chest is gone!)

For comparison here's is what 'top' shows in a MoodleBox, Moodle on a Raspberry Pi 3B+ (4 much weaker cores and 1 GB of RAM) in its idle state:


Assuming the CPU% add up to 145% and the heavy CPU load is still coming from mysqld and you restarted when the server it is not busy, the heavy idle consumption of mysqld must be the source. Somehow it is at full throttle with the clutch disengaged. If that is the case, the cause is not Moodle, but your system.

Talking of the system, you conveniently avoid my questions on Bitnami. Not that I can answer them, I am just not familiar, but there is a slight chance that there is somebody in the community who knows. But as already said, if MySQL is busy with itself, it is not a Moodle problem. Well, Moodle will have a problem, because it depends on the database.
In reply to Keithz Pecayo

Re: Laggy site

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

Daily numbers say little about the load. Concurrency is the critical factor. When you say "too many users are using it simultaneously" how many were there? What were they doing? Writing an essay into a  forum causes very little load, opening an MC exam is a heavy operation. Your initial pointers to the subject are linked to the header of this forum. Check them first.

Don't expect too specific answers though. Every case is different. And there are no formulas to calculate the maximum load. People here may judge whether a situation is too much for your server if you tell us what your server is. For the kind of required information pl. read Before you post... read this....

Coming to the information you've provided, I see "Bitnami" mentioned a couple of times. I believe you run one of these https://bitnami.com/stack/moodle. Which one, there are six, Single-Tier, Mutli-Tier, etc mentioned? I hope there is somebody here, who knows the product you use. I don't.