Conducting quiz for 2000 users

Conducting quiz for 2000 users

by manoj joseph -
Number of replies: 23

I have installed moodle 3.8 in amazon ec2 cloud and want to conduct quiz for 2000 concurrent users.I have installed xampp and configured moodle.my total users may become above 10000+.

Is it possible for me to conduct test and what all settings i needs to do with Apache and MySQL?

My both Apache and Mysql running on the same system.

Now I am using EC2 instance with 16 core cpu and 64gb ram.


Kindly help me to configure.

Thanking you


Average of ratings: -
In reply to manoj joseph

Re: Conducting quiz for 2000 users

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Interesting. Almost the same question has been asked yesterday: Login 1,200 students at a time on quizzes. Possibly you could follow the answers there.
Average of ratings: Useful (1)
In reply to manoj joseph

Re: Conducting quiz for 2000 users

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
You should not use xampp for a live system. It's only designed for development and testing. I doubt that it will end well.
In reply to Howard Miller

Re: Conducting quiz for 2000 users

by manoj joseph -

Thank you for your reply  can i go for lamp stack is that better

https://medium.com/better-programming/how-to-install-lamp-stack-on-ubuntu-db77ac018116

can you suggest which one is better for live system as i am new in production side.

In reply to manoj joseph

Re: Conducting quiz for 2000 users

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

Not claiming to have conducted a quiz for 2000 simultaneous candidates I believe that it won't be as simple as an "amazon ec2 cloud", whatever that is, not to mention "xampp". (See Howard's post.)

You can see the kind of infrastructure needed in many discussions in the Hardware and performance forum. In fact in the very same discussion you've reposted https://moodle.org/mod/forum/discuss.php?d=395623 they are talking of clusters of web servers.

In any case, this topic is more suitable for the Hardware and performance forum.
In reply to Visvanath Ratnaweera

Re: Conducting quiz for 2000 users

by manoj joseph -
I Changed Xampp

Now I have installed apache 2.4
PHP PFM 7.4
and Mysql server 8 in another instance.
I am not able to find why system get's slow which goes above 1000.server cpu load was only 20% peak.
one finding was in ubuntu userlimit default was only 1024
#ulimit -n
may be that was the issue.
Now i changed the userlimit to 65535 on both db system and webserver.
Needs to check the quiz and will give a reply
In reply to manoj joseph

Re: Conducting quiz for 2000 users

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
" am not able to find why system get's slow which goes above 1000.server cpu load was only 20% peak."
To get useful answers you need to be specific.
"get's slow"
is fairly meaningless. Be specific
In reply to Marcus Green

Re: Conducting quiz for 2000 users

by manoj joseph -
Thank you Marcus.
The site 'gets slow' means Quiz taking more time to load questions

My questions contains images.Once the number of concurrent login exceeds 1000,students get error messages and they are automatically logged out of the quiz. I guess the problem related to database tuning.
In reply to manoj joseph

Re: Conducting quiz for 2000 users

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
'The site 'gets slow' means Quiz taking more time to load questions'
10 seconds, 50 seconds, 2 minutes, 3 minutes?
'I guess the problem related to database tuning.'
It might be but it could also be other things, are these actual students or is this a performance testing script?
You need to give a complete and detailed explanation of your setup to receive any useful help. I am not an expert on quiz performance, but I am quite knowledgable about getting answers to technical questions.
In reply to Marcus Green

Re: Conducting quiz for 2000 users

by manoj joseph -

Now I changed the settings like below.Needs to check with these settings.Database running separately

I changed apache mpm_prefork to mpm_event  .

and the changed  all the configurations like below

mpm_event


<IfModule mpm_event_module>

        ServerLimit             4800

        StartServers             16

        MinSpareThreads          25

        MaxSpareThreads          75

        ThreadLimit              64

        ThreadsPerChild          25

        MaxRequestWorkers         4800

        MaxConnectionsPerChild   1500

</IfModule>


php.ini


max_input_time = 60

max_input_vars = 10000

memory_limit = 52G


mysqld.conf


key_buffer_size         = 512M


innodb_undo_log_truncate=off

# general

thread_cache_size = 100

table_definition_cache=2000

table_open_cache = 20000

open_files_limit = 40000

table_open_cache_instances=64

back_log=3500

max_connections=15000


# buffers

innodb_buffer_pool_size=40G

innodb_buffer_pool_chunk_size=2G

innodb_buffer_pool_instances=16

innodb_log_buffer_size=64M


# files

innodb_file_per_table

innodb_log_file_size=8G

innodb_log_files_in_group=2

innodb_open_files=4000



# tune

innodb_doublewrite= 1

innodb_thread_concurrency=0

innodb_flush_log_at_trx_commit= 0

innodb_flush_method=O_DIRECT_NO_FSYNC

innodb_max_dirty_pages_pct=90

innodb_max_dirty_pages_pct_lwm=10

innodb_lru_scan_depth=2048

innodb_page_cleaners=4

join_buffer_size=1G

sort_buffer_size=512M

innodb_use_native_aio=1

innodb_stats_persistent = 1


#innodb_spin_wait_delay=96

innodb_adaptive_flushing = 1

innodb_flush_neighbors = 0

innodb_read_io_threads = 16

innodb_write_io_threads = 16

innodb_io_capacity=1500

innodb_io_capacity_max=2500

innodb_purge_threads=4

innodb_adaptive_hash_index=0

max_prepared_stmt_count=10000

innodb_monitor_enable = '%'

performance_schema = ON


references:

https://medium.com/@sbuckpesch/apache2-and-php-fpm-performance-optimization-step-by-step-guide-1bfecf161534

https://www.percona.com/blog/2019/02/25/mysql-challenge-100k-connections/


In reply to manoj joseph

Re: Conducting quiz for 2000 users

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

I am also keen to know which parameter or parameters double the performance of a server. i.e. takes you from 1000 simultaneous quiz users to 2000. Do you simulate the user scenario or have to wait for (or organize) 2000 real users?
In reply to Visvanath Ratnaweera

Re: Conducting quiz for 2000 users

by manoj joseph -

How can we simulate user scenario?

In reply to manoj joseph

Re: Conducting quiz for 2000 users

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
There are many. Some are listed in https://en.wikipedia.org/wiki/Web_server_benchmarking#Tools_for_benchmarking. Being perhaps one of the earliest and the most known tool, JMeter gets some assistance in Moodle, see https://docs.moodle.org/dev/JMeter. You should understand that these tools are outside of Moodle. For support you need to go to their respective communities.
In reply to manoj joseph

Re: Conducting quiz for 2000 users

by Ricardo Caiado -
Picture of Particularly helpful Moodlers
Hi, Manoj.

Take a look at this as your dealing with images: https://moodle.org/mod/forum/discuss.php?d=217355

Did some tweak yesterday on my prod server and notice some improvement right away.

All the best,

Ricardo
Average of ratings: Useful (1)
In reply to Ricardo Caiado

Re: Conducting quiz for 2000 users

by manoj joseph -

Thank you Mr Ricardo Caiado .I followed the post  of xsendfile images

In the beging i put the below settings in my config.php

$CFG->xsendfilealiases = array(
//         '/dataroot/' => $CFG->dataroot,
//         '/cachedir/' => '/var/www/moodle/cache',    // for custom $CFG->cachedir locations
//         '/localcachedir/' => '/var/local/cache',    // for custom $CFG->localcachedir locations
//         '/tempdir/'  => '/var/www/moodle/temp',     // for custom $CFG->tempdir locations
//         '/filedir'   => '/var/www/moodle/filedir',  // for custom $CFG->filedir locations
//     );

Issue was images were not loading .Then I followed
https://moodle.org/mod/forum/discuss.php?d=367379

Similar way i put both path to my virtualhost settings and

<VirtualHost *:80>

XSendFile on
        XSendFilePath /path/to/moodle-code
        XSendFilePath /path/to/moodle-data
</VirtualHost>


Now picture is loading.How do we test my Xsendfile is working or not?


In reply to manoj joseph

Re: Conducting quiz for 2000 users

by Ricardo Caiado -
Picture of Particularly helpful Moodlers
Manoj,

I did not test it, but I noticed some improvement right away. Faster image loading time.

Ricardo
In reply to Ricardo Caiado

Re: Conducting quiz for 2000 users

by manoj joseph -
What is this opcache hit rate?I am getting  following things Should I need to change any thing?

In reply to Ricardo Caiado

Re: Conducting quiz for 2000 users

by manoj joseph -

Ricardo 

Thank you.It seems my moodle goes fine.Now db load reduced only 18 maximum connections up to 1200 concurrent users.Total 2500 users quiz done but peak was only 1200.In between due to some issues i opened another forum link follows.

https://moodle.org/mod/forum/discuss.php?d=406236&parent=1640931

Now i am getting another error while regarding the result it goes to  500 Internal error.What may be the issue?

In reply to manoj joseph

Re: Conducting quiz for 2000 users

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
You need to find that error recorded in your web server's error log to get a better idea. The 500 error tells you nothing useful.
In reply to Howard Miller

Re: Conducting quiz for 2000 users

by manoj joseph -
Thank you Howard

I made it ready after setting request_terminate_timeout=300 in /etc/php/7.4/pool.d/www.conf.

Now its working fine
In reply to Howard Miller

Re: Conducting quiz for 2000 users

by manoj joseph -
How do we check Peak user login all time in my moodle server? In moove theme dashboard shows last 5minutes log.Is there any other method to get peak concurrent user report?