System Benchmark Writing file performance +/- 40

System Benchmark Writing file performance +/- 40

by Dominique Palumbo -
Number of replies: 16
Picture of Particularly helpful Moodlers Picture of Plugin developers

Hi,

I would like to share experience about filesystem on Moodle 3.7.x in cluster.

We've a score near 40 for Writing file performance.

Your Moodle data is on a share we access it using NFS and the filesystem itself is CEPH.

We've six computers behind a load balancer. (16 cores x64 intel, 24gb ram) it's on Debian 9, Apache2, we use Memcached for sessions and for the muc.

I've some questions :

1. Could you please share your performance on your Moodle if they using share for Moodle data and Moodle is on clustering ?

2. Is someone have already change from NFS to GlusterNFS (or else) and had a good performance gain ?

Your Moodle have an overall good performance more 35000 unique visit every days and +/- 500 concurent users.

The bad performance happen when we've to restore courses and backup it.

Any advice are welcome, I hope I was clear enough. I've read the Moodle advice on performance it's the reason I ask for the GlusterNFS but we also wish to have also feedback from others about their infrastructure and performances.


Dominique.

system benchmark

Average of ratings: -
In reply to Dominique Palumbo

Re: System Benchmark Writing file performance +/- 40

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Six computers, 16 cores x64 intel, 24gb RAM each, showing a score of 4274 points? Are they buried in an avalance?
sad

A single one of them will score below 50 points (lower the better). Ref. [Moodle BenchMark] Need Beta-tester. Even a $ 35 Raspberry Pi (in the MoodleBox) scores around 160.
Average of ratings: Useful (1)
In reply to Visvanath Ratnaweera

Re: System Benchmark Writing file performance +/- 40

by Dominique Palumbo -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Hi Visvanath,

sometime truth hurt : ).

Without the test of read and write files. The score is 154. it's still not extravagant. In fact,
I've seen someone posting a score of +/-55.

It's really questionning about use one very fast server instead of six.

I'll read carrefully all the reply. And I'll comeback.

Thanks for your fast reply.
(I'll propose the Raspberry Pi : ) )
Average of ratings: -
In reply to Dominique Palumbo

Re: System Benchmark Writing file performance +/- 40

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Dominique

Yep, lot of stuff for you to go through, and possibly some hurting.
wink

Don't be concerned whether the score is 50 or 150. See my comment on over-interpreting and the corr. thread in another post. As a first approximation, a decent server should not bring any result in red or pink, kind of go-no-go criteria.

As it stands, if you take two machines from the cluster and make a conventional single server web server and a single server database server, that pair will give a throughput many times of your current cluster where a NFS mount pulls the hand-brake.
Average of ratings: -
In reply to Dominique Palumbo

Re: System Benchmark Writing file performance +/- 40

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
How is your shared cache and sessions configured?

In particular what's your cache server (Redis, memcached etc.)?

Have you looked at / tried database locks rather than file locks?

Shared file storage is always slow (often terminally slow) without tuning these performance-related services that write to the file store by default. 
Average of ratings: Useful (1)
In reply to Howard Miller

Re: System Benchmark Writing file performance +/- 40

by Dominique Palumbo -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Hi Howard,

We use memcached for the session (on the 6 servers) and we've two more memcached servers for the MUC.
(we want to go to redis for the next academic year)

The lock is already configure in the database.
in the config.ini ($CFG->lock_factory = "\\core\\lock\\db_record_lock_factory";)

Yes, this slow performance also explain the performance problem that we've with the backup...
We would like to know if your slowness is more slow than other systems with shared storage.

Thanks for your reply.

Dominique.
Average of ratings: -
In reply to Dominique Palumbo

Re: System Benchmark Writing file performance +/- 40

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Dominique

This time with more useful information, hopefully.
;)

About the benchmark:
- All the numbers, except the final score, are in seconds. So the 40 is total duration in seconds. The score is 4274, which is inverse - lower the better. Typical servers score below 100. (See the original thread.)

- A decent server should not show any result in orange, let alone in pink. In reality these are not go-no-go tests. But how close a result is to the two limits mentioned gives you an indication of the strengths and weaknesses of the server.

- One should not over-interpret the result. For example, I don't know what exactly it measures. Ref. [Moodle Benchmark] Is it applicable to non-x86 architectures?. Also, I don't know whether the test is suitable for clusters either.

All that said, there is something severely wrong with the NFS mount for moodledata. The solution could be something as simple as some filesystem or mounting parameter. In the worst case you have chosen the wrong technology. As Howard mentioned, shifting certain functions to the database of RAM may improve the performance, but not at a price.
Average of ratings: Useful (1)
In reply to Visvanath Ratnaweera

Re: System Benchmark Writing file performance +/- 40

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Yeh - that's a broken NFS server there without a doubt. Probably some NFS diagnostics are required. It could be a number of things....

Even if you resolve that my original question still applies.... even performant NFS is typically far too slow to be used for the (MUC) cache.
Average of ratings: Useful (1)
In reply to Visvanath Ratnaweera

Re: System Benchmark Writing file performance +/- 40

by Dominique Palumbo -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Hi Visvanath,

the first post was usually useful. We need it : ) (like a good kick in the ass (french expression))

Maybe another important information is that 6 servers are virtualized.

I'll ask to the infra team if they can make a diagnostic of the NFS.


Thanks.

Dominique.
Average of ratings: -
In reply to Dominique Palumbo

Re: System Benchmark Writing file performance +/- 40

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Dominique

Yes, need to talk a word with your IT staff!
wink

In your OP you mentioned "six computers behind a load balancer. (16 cores x64 intel, 24gb ram)". You say they are virtual machines. Must be 16 vCPU and 24 GB each(?) What is the forumula behind that calculation. May be I had too much physics at school. Dividing hardware in to six pieces and putting them together supposed to yield more that the original hardware?
Average of ratings: -
In reply to Dominique Palumbo

Re: System Benchmark Writing file performance +/- 40

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Dominique,
give https://moodle.org/mod/forum/discuss.php?d=310501#p1242382 and following posts a read: you could find some suggestions even if it is an old thread.

HTH,
Matteo

Average of ratings: Useful (1)
In reply to Matteo Scaramuccia

Re: System Benchmark Writing file performance +/- 40

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I would only add that... some time having elapsed... that I would personally recommend Redis over Memcached these days.
Average of ratings: Useful (2)
In reply to Howard Miller

Re: System Benchmark Writing file performance +/- 40

by Matteo Scaramuccia -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Howard,

I would personally recommend Redis over Memcached these days

+1 approve Yes
See also MDL-60037.

BTW a slow dataroot could severely impact on file serving, that was the main meaning of my ref.

HTH,
Matteo

Average of ratings: Useful (2)
In reply to Matteo Scaramuccia

Re: System Benchmark Writing file performance +/- 40

by Dominique Palumbo -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Hi,

your Memcached version is memcached 1.4.33.
So, it's seems good.
Thanks.

Dominique.
Average of ratings: -
In reply to Howard Miller

Re: System Benchmark Writing file performance +/- 40

by Dominique Palumbo -
Picture of Particularly helpful Moodlers Picture of Plugin developers
It also a question I've and you already answer it : )

I've another one about Redis.
For the session Redis is still more performant than have it in DB with the session table in memory ?
In the event of a server failure (Moodle in clustering), is Redis able to manage sessions that were on this server on other machines ?
(with memcache Moodle just go in error)

Thanks.

Dominique.
Average of ratings: -
In reply to Dominique Palumbo

Re: System Benchmark Writing file performance +/- 40

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
(The last time I looked) Moodle still does not support Redis clusters. Which is unfortunate. However, this means you have a single Redis instance for all your front ends. A single point of failure if you will. We run a hot spare in case of problems. I've never had Redis go wrong to find out what happens. I'm rather of the view that cache is by its very definition disposable and sessions, while highly inconvenient if lost, is still not an end-of-the-world thing.

I have no idea if this answers your question.
Average of ratings: Useful (1)
In reply to Matteo Scaramuccia

Re: System Benchmark Writing file performance +/- 40

by Dominique Palumbo -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Hi Matteo,

thank you very much.
I'll communicate the configuration you suggest in the post you link. It seem it work well.
(noatime which could seriously change your performances whilst you could try rsize=8192,wsize=8192,tcp,timeo=14)
Dominique.
Average of ratings: -