NFS for dataroot

NFS for dataroot

by Keith Weinheimer -
Number of replies: 13

I saw in other discussions where it has been noted that an NFS mounted directory is used for the dataroot.

I am taking over a installation in AWS that was spun up using the bitnami marketplace offering.

From the config I have this.

$CFG->dataroot  = '/opt/bitnami/apps/moodle/moodledata';

I created a new directory in the same root called moodledata-efs, mounted that to an EFS in AWS and then copied the contents of the original moodledata to it using rsync

sudo rsync -avzh /opt/bitnami/apps/moodle/moodledata/ /opt/bitnami/apps/moodle/moodledata-efs/

So permissions and ownership of the directory and all contents are identical

I renamed moodledata to moodledata-orig 

Renamed moodledata-efs to moodledata and then started up the services again and i get the below error

Fatal error: $CFG->dataroot is not configured properly, directory does not exist or is not accessible! Exiting.

I have confirmed that the bitnami user which apache is running under can create and delete a file in the new NFS mounted path for dataroot.

I am not looking to cluster I just want to store dataroot on a more durable data store.







Average of ratings: -
In reply to Keith Weinheimer

Re: NFS for dataroot

by Keith Weinheimer -

I should also note this is moodle version 2.8.2, getting prepared to upgrade

Average of ratings: -
In reply to Keith Weinheimer

Re: NFS for dataroot

by Keith Weinheimer -

I have followed these exact same steps on the fresh install of 2.8.2 on amazon linux 2 and I did not run into this dataroot issue.

Average of ratings: -
In reply to Keith Weinheimer

Re: NFS for dataroot

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

You wrote:
> So permissions and ownership of the directory and all contents are identical

Did you check? I suggest 
sudo -u webowner touch /path/to/directory/foobar
which should create a file foobar inside the complaining moodledata.

'webowner' is of course the owner of the web server process, sometimes www-data or just www or apache or something else. Use 'ps' to find out.

Average of ratings: -
In reply to Keith Weinheimer

Re: NFS for dataroot

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

That message is almost always true. 

First, I'm just noting that it's hard to support the Bitnami install - it's not an official installer and we don't know if it did anything strange.

It's worth enabling Debugging, reproducing the problem and then checking your web server's error log to see if anything interesting got logged.

Average of ratings: -
In reply to Howard Miller

Re: NFS for dataroot

by Keith Weinheimer -
I came across the dbmigrate script and may have inadvertently found the file system issue, will know soon.


I was getting a permission issue in the file system when trying to run migrate.php from command line as the web user.

Did a chown -R to make sure the web user owned all the files in the file system (this is using the local filesytem, not NFS) and was able to get past the error. 

With that I was able to migrate the DB to an RDS instance which is half the battle.

I am trying to get this instance off bitnami and yes bitnami does all kinds of stuff with the install and environment to make it hard to support.

Average of ratings: Useful (1)
In reply to Keith Weinheimer

Re: NFS for dataroot

by Gillian Webb -

A bit of an aside - but don't use EFS for Moodle. It has some major performance issues that make Moodle randomly time out - this took me a while to track down myself on our AWS deployment. Turns out that for Moodle's access pattern, EFS will just randomly have extreme latency spikes. (We were not running out of burst credits, this is a different issue with small file request latency.)

NFS mounts themselves are fine, just don't use Amazon's EFS product to provide them for a Moodle installation. If you run your own NFS server it'll be fine, or you can do what we did and get a third party storage hosting provider to provide a NFS mount for you.

Average of ratings: Useful (1)
In reply to Gillian Webb

Re: NFS for dataroot

by Keith Weinheimer -

Ian, how long ago did you try and move away form EFS?

What OS platform were you on?

Were you using amazon utilities to mount NFS?

https://docs.aws.amazon.com/efs/latest/ug/mounting-fs.html#mounting-fs-mount-helper

I have used NFS with several off the shelf applications like Jira, Confluence, Vtiger and EFS has performed well. We were previously using our own NFS server.


Average of ratings: -
In reply to Keith Weinheimer

Re: NFS for dataroot

by Gillian Webb -

This was mid-2018. Yes, we were using Amazon Linux and the Amazon utilities to mount the EFS share. Here's what AWS support eventually said:

While the distributed architecture of Amazon EFS enables high levels of availability, durability, and scalability, it results in a small latency overhead for each file operation. Due to this per-operation latency, overall throughput generally increases as the average I/O size increases, since the overhead is amortized over a larger amount of data.
 
If the workload involves reading/writing lots of small files(or copying these files), the per-operation latencies can result in EFS performance being different from a local file system performance. This is why performance is degraded when dealing with small r/w IOPs - it is a limitation of the EFS architecture.

Basically, EFS works just fine for many use cases, but some specific architectural decisions Amazon make for EFS cause Moodle to have inconsistent and slow performance when the Moodle dataroot is on an EFS volume.

Using Redis caching for sessions and a local RAM disk localcache folder didn't even fix this. No matter what, Moodle will do quite a few small file operations on the moodledata folder, and that increased per-operation latency kills your site performance.

Average of ratings: Useful (1)
In reply to Gillian Webb

Re: NFS for dataroot

by Jeff White -
This is likely not a cheap solution but you might want to look into using  GPFS or GFS2. I have had great success with our Moodle environment that makes the NSD have near local disk performance. We have not scaled out to where we have replication in the US, EU, and Asia but it should in theory still maintain near the same performance as our replication between two data centers in the same state though I am not 100% sure how. 
Average of ratings: -
In reply to Jeff White

Re: NFS for dataroot

by Keith Weinheimer -

Thanks Jeff, our implementation is relatively small, I will try EFS to start to take advantage of the easy of backup and growth management. we'll see how it goes.

Average of ratings: -
In reply to Gillian Webb

Re: NFS for dataroot

by chaitanya varanasi -

@Ian Webb- Could you share the nfs configurations that work for you?

I am especially looking at /etc/fstab of the client. The mount command can have additional options like rsize and wsize. 

Also, let me know if you made any changes to the NFS server in terms of ulimit, or the settings within /etc/exports (can't think of other parameters).

Average of ratings: -