NFS write performance issues

NFS write performance issues

由Karsten keck發表於
Number of replies: 7

Hello everyone,

I'm new to Moodle, so this is my first post. Please be patient if I get something wrong or if my question is confusing.

I am experiencing extremely high latency when writing to an NFS share in my Moodle installation, which is severely affecting performance. I'd like to summarize the tests and benchmarks we've conducted and hope for some advice on how to resolve these issues.

Here is my problem:

grafik.png

Setup:

    Web Servers: 3x (web01, web02, web03)
    Database: Postgres Patroni Cluster
    Redis Server: redis01 used for sessoin and application cache I'm not aware what the application cache stores and if it's a good idea to have the request cache left on the filesystem.
    File Server: file01
    Hardware: 8x 2.5 GHz CPU, 16 GB RAM per server
    NFS Mounts:
        moodledata is mounted over NFS to all web servers.
        tempdir and cachedir are also mounted over NFS.
        localcachedir and localrequestdir are stored locally on the web servers.

The NFS server is a cloud system hosted by our provider. The disk on the file server is mounted with the following options:

/dev/disk/by-id/scsi-0HC_Volume_XXX /data ext4 discard,nofail,defaults 0 0

On the web servers, the NFS mounts are configured with the following options:

file01:/moodledata /moodledata nfs4 _netdev,auto,noacl,nocto,rsize=32768,wsize=32768,noatime,nodiratime,ac 0 0

We also tested using async, but this didn’t improve performance.

Tests we did:

    File writing:
        Extremely high latency of 49.773 seconds (acceptable: < 1 second, critical: > 1.25 seconds).
    Other tests:
        Database operations, PHP functions, and file reading are within an acceptable range.

Moodle directory configuration:

    $CFG->tempdir = '/moodledata/temp'; → NFS
    $CFG->cachedir = '/moodledata/cache'; → NFS
    $CFG->localcachedir = '/moodledata-local/localcache'; → local on web servers
    $CFG->localrequestdir = '/moodledata-local/temp'; → local on web servers

Server-side tests:

    Writing with dd on the NFS mount (from web01):
        5.4 GB in 23.66 seconds (~227 MB/s)

    Fio test on the NFS mount (from web01):
        IOPS: 6176, Bandwidth: 24.1 MiB/s
        Average latency: 662 ms, Maximum: 3488 ms.

    Fio test directly on the NFS server:
        IOPS: 4196, Bandwidth: 16.4 MiB/s
        Average latency: 975 ms, Maximum: 13 seconds.

    Fio test on the local moodledata directory (web01):
        IOPS: 273k, Bandwidth: 1066 MiB/s
        Very low latencies (Average 15 µs).

Network tests:

    iperf (web01 → file01):
        Stable bandwidth of 7.33 Gbit/s, no bottlenecks detected.

Did a nfsstat on the fileserver:

Server RPC statistics:

calls      badcalls   badfmt     badauth    badclnt
2750390491   957        231        726        0

nfsstat shows a high usage of getattr (18%), putfh (27%), and sequence (26%) operations. even while device in fstab is mounted with ac option.

Question:

Where should I start to troubleshoot the write speed issues? The high write latency to the NFS share is slowing down the entire system.
Has anyone suggestions how to fix / optimize this?

Thank you in advance for your help!

評比平均分數：Useful (1)
In reply to Karsten keck

Re: NFS write performance issues

由Brett Dalton發表於
Moodle HQ的相片 Particularly helpful Moodlers的相片
Thanks for the detailed information. One question I have is when you say its " cloud system hosted by our provider" is this in the same infrastructure as the rest of your Moodle servers? This is really an infra problem which your hosting provide should be able to help with.
評比平均分數： -
In reply to Brett Dalton

Ri: Re: NFS write performance issues

由Sergio Rabellino發表於
Particularly helpful Moodlers的相片 Plugin developers的相片

Yes, very detailed information: taking care of numbers I can think that you did FIO tests with cache enabled, that's why probably FIO on web01 perform better than directly on the NFS server.

For NFS server, the number of threads is significant when you have performance issues (how to verify and change them changes a lot between OSes, but should be easy to find 'howto' with google...). NFS Server RAM it's another important thing to verify: much is better. And verify the quality of the underlying volume, as it performs not so good.

Lastly, check naming resolution on NFS Server (e.g. put in /etc/hosts the IP of your web01 server), it seems that the server is doing "something" under the hood before answering, and typically these are dns/naming resolution that wait until a timeout.

HTH.

評比平均分數： -
In reply to Sergio Rabellino

Re: Ri: Re: NFS write performance issues

由Karsten keck發表於
Thanks for the hints.

Ok, I'll try to keep it short this time.

tl;dr:
As you said, the bottleneck seems to be local on the fileserver.
Meanwhile, I got a backup fileserver and did fio tests without the latency peaks. Maybe because it's still not in use.
I'll try to switch to that one first. If that doesn't work, I'll talk to my hoster and may come back with more questions about NFS configuration.

###

About infrastructure: Yes, they're all in a shared private subnet with a 10Gb backend.
But I noticed something special with the disk: the hoster provides additional disks (where I have moodledata) as cloud volumes,
advertised as fast SSD-based network hard drives with triple replication. Might be interesting.

The huge gap between the minimum and maximum latency (up to almost a minute) on the fileserver's 'local' disk is really frustrating.
The performance seems inconsistent, and I can't figure out why. I'll see if the switch fixes this.

DNS is fine. All servers are in the hosts file.

The configuration of the NFS server can surely be improved. I think the threading was a good tip, but I still need to find some guidelines for the recommended settings based on the RAM/CPU setup.

Nevertheless, as mentioned, the bottleneck seems to be the local disk on the NFS server, not the network mount.
評比平均分數： -
In reply to Karsten keck

Re: NFS write performance issues

由Visvanath Ratnaweera發表於
Particularly helpful Moodlers的相片 Translators的相片
Whether your question is confusing? I'd say, if all enquiries on moodle.org are that much confusing the site will become "boringly unconfusing".

About the Moodle Benchmark: As many benchmarks you don't know how those numbers ultimately translate in to numbers you are interested in, like "How many concurrent users will my Moodle support". It does best is for comparison. See the original discussion [Moodle BenchMark] Need Beta-tester. There were many more. Use the forum advanced search.
 
There is obviously something very wrong in your results. Writing performance of your data volume is prohibitively slow. The name Moodle Benchmark gives false hopes. It is not Moodle doing something, it uses the access to the server (LAMP) given by the data in Moodle's config.php. Be convinced by testing the performance of your data volume directly from the shell, with Bonnie++ or other tools you know. Then ask the provider. They will not react to "Moodle numbers" saying that it is a Moodle problem, but have to respond to the widely known system benchmarks in Linux. There are simply too many to count.
 
Oh, BTW, don't forget the initial pointers linked to this forum, Performance and Performance FAQ.
評比平均分數： -
In reply to Visvanath Ratnaweera

Re: NFS write performance issues

由Karsten keck發表於
Hi,
Thanks for the Links
I'm aware, that the benchmark isn't representative for official system performance. Also did not look in the code but guess that it writes thinks via php fput or frwrite?
So i chose fio, iostat and for nfs nfsstat. This gives a more detailed output than bonnie++
You can find a rough breakdown of the numbers in my first post.
As mentioned in my last post above, I'll take a look at the local disk on the fileserver.
評比平均分數： -
In reply to Visvanath Ratnaweera

Ri: Re: NFS write performance issues

由Sergio Rabellino發表於
Particularly helpful Moodlers的相片 Plugin developers的相片
Only for information, The mentioned tool FIO it's a disk benchmark tool like Bonnie++, but newer and much better, very much better...
評比平均分數：Useful (1)
In reply to Karsten keck

Re: NFS write performance issues

由Jeff White發表於
If you are only using one web node then try to stick to local disk. You will have a much easier time for disk performance.

If you MUST have multiple web nodes AND have the resources, consider using GFS/GPFS over NFS as it provides near local disk speeds for a shared storage. At least that is what I got when I did Jmeter load tests on my Moodle environment. It is a massive resource hog, memory, but it will improve your environment's shared storage response times.
評比平均分數：Useful (1)