Moodle in English: Shared temp folders in config.php

Shared temp folders in config.php

by Juan Luis Maestre - Sunday, May 17, 2020, 12:45 AM

Hello, I am tunning my moodle site. It's hosted on AWS with the following configuration:

Moodle v3.7.3
Frontend web servers fleet with nginx-1.14 (with proxy_cache to ramdisk) + php-fpm-7.2.24 (with Zend Opcache engine)
Database: Aurora Mysql 5.6.10 with read replica failover
Redis: for MUC caching with igbinary serializer enabled
Memcache: for session data and lock factory
EFS: for moodledata (EFS is NFS-based)

The bottleneck of this configuration is in the EFS. It's well known the poor performance of NFS-based filesystems on multiple files of small size. This matter has already been discussed on many occasions in this and other forums.

My specific problem is due to the shared temp folders between the web servers in the fleet: $CFG->tempdir, $CFG->cachedir and $CFG->backuptempdir. These folders are stored in the moodledata folder, slowing down certain operations, and obtaining disastrous results in the benchmark test: Reading file performance=6,57 seconds and Writing fileperformance = 54,69 seconds.

I'm testing other configurations that include a dedicated node to store these folders with:

Option 1: NFS server(this solution avoid the EFS metadata overhead for the small files).
Option 2: GlusterFS volume with gluster native mount on web frontends
Any other suggestion?

In addition, this dedicated node could be used for cron and backup tasks.

With option 1 I have managed to write file performance of 9.8 secs (compared to 58.67 secs) and read file performance of 0.048 secs (compared to 6.57 secs). I have verified it with a course backup that previosly took ~94 secs, now I get ~45 secs.

On the other hand, with option 2 I have managed to write file performance of 3.262 secs (compared to 58.67 secs) and read file performance of 0.673 secs (compared to 6.57 secs). In the same way, the course backup that previously took ~94 secs, now I get ~37 secs.

Are these three shared temp folders a critical point of failure? I think they store very volatile files that, in case of failure of this dedicated node, could be replaced by another fresh node without any other problem than the web servers would have to regenerate the shared temp content. I'm right?

Initially I thought to create a GlusterFS replicated volume between several dedicated nodes but it seems to me a waste of resources if the data is not critical

I had in mind a third option which was to create a replicated Glusterfs volume between the different frontend web servers to achieve this, but the results are obviously worse due to the overhead of having to replicate between all the nodes.

What is your opinion? Any other suggestions?

Thanks

Re: Shared temp folders in config.php

by Howard Miller - Sunday, May 17, 2020, 1:25 AM

I'm not convinced. How do you know (for certain) that the NFS performance is a problem. All the stuff that really matters has been dealt with. There's nothing much left writing to NFS that should cause significant performance issues.

As I like to say - prove it

Average of ratings: -

Re: Shared temp folders in config.php

by Juan Luis Maestre - Sunday, May 17, 2020, 2:06 AM

Hello Howard, my purpose was not to indicate that NFS is a problem in itself (I use it, a lot of people too) in my production environments with good results.

However, with EFS, my actual implementation in production:

(benchmark with EFS)

Without EFS, in local SSD storage for example:

(benchmark without EFS, SSD local based)

This seems to indicate that there is some kind of problem or bottleneck in my EFS-hosted moodle temporary shared folders, right? Some of my users complain about slowness when creating course backups...

I cannot use local-SSD based temporary folders because I have more than one web server, and they must share these, right? So I'm looking for an alternative solution for it: NFS (not EFS based), GlusterFS, ...

Thanks!

Average of ratings: -

Re: Shared temp folders in config.php

by Howard Miller - Sunday, May 17, 2020, 2:37 AM

I would expect NFS to be slow. It's the nature of the beast. It doesn't indicate a problem in itself. It would certainly indicate a problem if you didn't have Redis.

What are your page load times? What does the load average look like (you've got monitoring I hope)? If it's all happy then I wouldn't be concerned.

Average of ratings:Useful (1)

Re: Shared temp folders in config.php

by Juan Luis Maestre - Sunday, May 17, 2020, 4:05 AM

Hi,

I am happy with the results. Response times remain at good levels. Page load times at 800-900msecs. I agree with you that redis is the key. Without redis the response times are greatly increased. At the moment I am doing load tests and identifying possible problems, preparing the platform for a future scenario of approximately 3K-4K concurrent users making on-line exams. Maybe it's like you say, and I shouldn't be so worried. Still, I prepare for the worst case scenario. Is it possible that quizzes are not very dependent on moodledata?

Thanks!

Average of ratings: -

Re: Shared temp folders in config.php

by Howard Miller - Sunday, May 17, 2020, 5:25 AM

Quizzes are not dependent on moodledata at all if you don't have any files involved. Which you almost certainly do not.

Average of ratings: -

Re: Shared temp folders in config.php

by Visvanath Ratnaweera - Sunday, May 17, 2020, 2:47 AM

From what I can see the I/O performance is disastrous. Reading takes 13.5 times the acceptable limit, writing 55 times! Could you post both benchmark results in full?

What is the size of the overall operation? How many registered users? How many users are logged in (last 5 minutes) at peak times? Do you have group chats with https://docs.moodle.org/en/Chat_activity ? What is the largest number of students entering a synchronous on-line exam https://docs.moodle.org/en/Quiz_activity in the same minute?

Average of ratings: -

Re: Shared temp folders in config.php

by Juan Luis Maestre - Sunday, May 17, 2020, 4:32 AM

Hi Visvanathan, these benchmarks have been done without users in the system and without load, in our test environment. I agree with you that the results are disastrous. No group chats.

I just repeated the tests right now:

(test with EFS)

Test with SSD:

I'm doing load tests with 1K-2K virtual users doing a quiz at the same time and the results are good. For 2K users, 5' ramp up (login, view course, view quiz, start quiz, ....):

In general the results are good, with max response times in the clients ~2-3 seconds

Maybe soon we will have about 3K-4K users doing quizs at the same time. So my concern about being able to scale-up my resources and handle all the load, including the concern that I previously mentioned (shared temp folders)

I am still amazed that it is capable of handling such a load with such poor performance in temp shared folders. It is possible?

Thanks!!

Average of ratings: -

Re: Shared temp folders in config.php

by Howard Miller - Sunday, May 17, 2020, 5:27 AM

I'm not that familiar with the inner workings of the quiz module, but I can't imagine what it would need temporary folders for.

Hopefully, Tim will be along and can confirm or deny that.

I would go to some trouble to make sure your database is tuned, though.

Average of ratings: -

Re: Shared temp folders in config.php

by Tim Hunt - Monday, May 18, 2020, 12:24 AM

Right. The only possible shared disc use during quizzes is caching, or images in the question text/feedback. So if you set up Redis of another good cache, then disc should not matter.

Generally, the HTTP requests for a quiz attempt looks like:

view.php -> startattempt.php -> (attempt.php -> processattempt.php ->) x N -> summary.php -> processattempt.php -> (review.php)? -> view.php

(https://docs.moodle.org/dev/Quiz_user_interface_overview). And for most of those pages, the work done is:

A bunch of standard DB queries to check the login and load the quiz settings (would be similar in any other Moodle activity like a forum).
One big DB query (very similar to https://docs.moodle.org/dev/Overview_of_the_Moodle_question_engine#Detailed_data_about_an_attempt) to load the attempt.
A number of gets from the question definition cache, one for each question. (Obviously cache misses will need to do a bunch of DB queries, but the cache will soon be warm.)
Then execute a bunch of PHP code.

Then, for processattempt.php, there will additionally be 5) some writes to the question_attempt_steps, question_attempt_step_data and quiz_attempts tables if anything has changed.

The two most expensive requests in this process are startattempt.php, which has to create a new quiz attempt (if the student does not already have on) and the finally call to processattemp.php when the student clicks Submit all and Finish. The reasons these are expensive is that they have to updated every question in the attempt. The other requests only have to touch the questions on the current (or previous) page. Especially once MDL-67183 lands in Moodle 3.9.

I hope that helps understand what is going on. We have done our best to make it efficient (although I have always said: number one priority is to allow educationally exciting question types like https://moodle.org/plugins/qtype_stack, https://moodle.org/plugins/qtype_coderunner and https://moodle.org/plugins/qtype_pmatch, efficiency of the code comes second.)

I have never fully understood why the quiz is so good at crashing some people's servers. My hypothesis is that there are two main factors:

Working through a quiz, each student is typically doing more HTTP requests per minute than is typical for other activities (e.g. reading a forum or book, or writing an assignment).
Quiz attempts do more DB writes than other activities, and all on the few tables mentioned above. Therefore, you need a database that can cope well with multiple concurrent writes to the same table. (That is, everyone should always use Postgres 😉)

P.S. I occurs to me that for step 1), there might be another possible saving. I think some modules use a newer way to get the basic settings, which makes more use of caching. We should probably think about updating quiz to do it that way too.

Average of ratings:Useful (4)

Re: Shared temp folders in config.php

by Juan Luis Maestre - Monday, May 18, 2020, 1:02 PM

Hi Tim, I am very grateful for your detailed explanation.

I will tune the database and redis cache. The impact of moodledata/shared temp folders is less than I thought.

I am surprised by your recommendation to use postgres vs mysql, due to better multiple concurrent writes performance. I will keep it in mind.

Thank you very much!

Average of ratings: -

Re: Shared temp folders in config.php

by Tim Hunt - Friday, May 22, 2020, 4:57 PM

Note, I have now created MDL-68806, to document another possible performance improvement in the quiz.

Average of ratings:Useful (1)

Re: Shared temp folders in config.php

by Juan Luis Maestre - Saturday, May 23, 2020, 2:53 AM

Very grateful, surely it will be a very useful improvement

Average of ratings: -

Hardware and performance

Shared temp folders in config.php