Problems mounting Moodledata via NFS on CentOS

Problems mounting Moodledata via NFS on CentOS

by António Godinho -
Number of replies: 20

Hi,


Maybe this is the wrong place for this, but I'm killing myself over this...

I want to cluster our Moodle, to run tests for the students.


Right now The DB runs on 3 DB nodes with Galera MariaDB, with 2 HaProxys for balance and HA.


I want to split to original server into several nodes with Apache and PHP-FPM.

The Moodle data (and Redis) will run on a separated server, and will be mounted on each node running Apache.

NFS on the server has this options on exports:

/mnt/nfs_shares/moodledata 10.47.1.192(rw,sync,all_squash)

Clients will have on fstab:

10.47.1.184:/mnt/nfs_shares/moodledata  /var/www/moodledata     nfs4 _netdev,rw,noatime,sync,rsize=8192,wsize=8192,tcp,timeo=14,intr 0 0

If I setup SElinux off with "setenforce 0" I can see the moodle page. If I thrun it on:

Fatal error: $CFG->dataroot is not writable, admin has to fix directory permissions! Exiting.

I'e tryed using the "setsebool -P httpd_use_nfs 1" but no sucess...

Any advice on this?


Looking at our scenario, any other advice on something I should change?


Thank you.

Average of ratings: -
In reply to António Godinho

Re: Problems mounting Moodledata via NFS on CentOS

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
From your web server machine, what do you get when...

ls -al /var/www/moodledata

?? Do they user and permissions mean it is writeable by the Apache user?
Average of ratings: -
In reply to Howard Miller

Re: Problems mounting Moodledata via NFS on CentOS

by António Godinho -
Yes it's writeable by Apache user:
On the Web server node:
[root@Srv-MoodleWeb1 ~]# ls -al /var/www/
total 12
drwxr-xr-x.  6 apache apache   65 Apr 17 14:39 .
drwxr-xr-x. 21 root   root   4096 Apr 16 11:18 ..
drwxr-xr-x.  2 apache apache    6 Dec 23 20:47 cgi-bin
drwxr-xr-x.  2 apache apache   22 Apr 16 11:23 html
drwxr-xr-x. 53 apache apache 4096 Mar 23 07:03 moodle
drwxrwxrwx. 13 apache apache 4096 Apr 16 06:55 moodledata
[root@Srv-MoodleWeb1 ~]# ls -al /var/www/moodledata
total 52
drwxrwxrwx.  13 apache apache 4096 Apr 16 06:55 .
drwxr-xr-x.   6 apache apache   65 Apr 17 14:39 ..
drwxrwxrwx.   6 apache apache 4096 Apr  7 16:07 cache
drwxrwxrwx. 258 apache apache 4096 Aug 29  2019 filedir
-rw-rw-rw-.   1 apache apache  128 Aug  1  2019 .htaccess
drwxrwxrwx.  14 apache apache 4096 Apr 17 04:12 lang
drwxrwxrwx.  12 apache apache 4096 Apr 17 12:08 localcache
drwxrwxrwx. 258 apache apache 4096 Aug  6  2019 lock
drwxrwxrwx.   4 apache apache 4096 Aug  1  2019 models
drwxrwxrwx.   2 apache apache 4096 Aug  1  2019 muc
drwxrwxrwx.   2 apache apache 4096 Aug  1  2019 sessions
drwxrwxrwx.   5 apache apache 4096 Apr  7 11:54 stack
drwxrwxrwx.  12 apache apache 4096 Apr 17 09:05 temp
drwxrwxrwx.  71 apache apache 4096 Apr 17 12:00 trashdir
[root@Srv-MoodleWeb1 ~]#

On the NFS server:

[root@MoodleNFS1 ~]# cd /mnt/nfs_shares/
[root@MoodleNFS1 nfs_shares]# ls -la
total 12
drwxr-xr-x.  3 apache apache 4096 Apr 14 09:15 .
drwxr-xr-x.  3 root   root   4096 Apr 14 09:15 ..
drwxrwxrwx. 13 apache apache 4096 Apr 16 06:55 moodledata
[root@MoodleNFS1 nfs_shares]#

Like I've said, When I disable the SELinux everything works. I think it's a SELinux/CentOS issue...

Average of ratings: -
In reply to António Godinho

Re: Problems mounting Moodledata via NFS on CentOS

by António Godinho -

Also, using a shell, I'm able to create a file with apache user. Well the owner and group isn't exactly what I've expected...

[root@Srv-MoodleWeb1 ~]# sudo -u apache touch /var/www/moodledata/testing.txt
[root@Srv-MoodleWeb1 ~]# ls -la /var/www/moodledata/testing.txt
-rw-r--r--. 1 nobody nobody 0 Apr 18 10:08 /var/www/moodledata/testing.txt
[root@Srv-MoodleWeb1 ~]#


Average of ratings: -
In reply to Howard Miller

Re: Problems mounting Moodledata via NFS on CentOS

by António Godinho -
Disabling SELinux makes it work. I just think disabling it is NOT a valid long term solution.
Average of ratings: -
In reply to António Godinho

Re: Problems mounting Moodledata via NFS on CentOS

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
No, but it gives you a clue where to look for a long-term solution. There must be one specific thing which SE-linux blocks by default, but which is needed here. Try looking in the applicable log file, then work out how to selectively enable that.
Average of ratings: -
In reply to António Godinho

Re: Problems mounting Moodledata via NFS on CentOS

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
It depends on your attitude towards SELinux. Other Linux distros seem to get on fine without it. It's a truly horrible thing to configure.

Anyway, don't switch it off. At least set it to 'Permissive' and watch the logs. 
Average of ratings: -
In reply to António Godinho

Re: Problems mounting Moodledata via NFS on CentOS

by Ken Task -
Picture of Particularly helpful Moodlers

Set to permissive, selinux will log it's objections ... and give you hints/clues/suggestions as to what to do about them - actual selinux commands.

Might leave selinux in permissive mode for a while and check logs often as site is used.  For example, updating a plugin, selinux might throw a different objection from say uploading a file via Moodle UX to filedir.

Once selinux stops complaining, then set to enforcing.

'SoS', Ken


Average of ratings: Useful (1)
In reply to António Godinho

Re: Problems mounting Moodledata via NFS on CentOS

by Camilo Ravelo Durán -
Hello one question. In NFS do you only store all the moodledata subfolders? And where is the session and cache folder? I have 5 mirror moodle servers that share 1 NFS, this NFS has the moodledata folder (filedir, session, cache, localcache cache) and it is really slow. I also use centos for NFS the server and for the client.
Average of ratings: -
In reply to Camilo Ravelo Durán

Re: Problems mounting Moodledata via NFS on CentOS

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
NFS is almost always too slow for the cache. The first/simplest spec is to set up a dedicated cache server. If in doubt, try Redis. You can then point the MUC/Cache at this. You should see a difference!
Average of ratings: Useful (1)
In reply to Howard Miller

Re: Problems mounting Moodledata via NFS on CentOS

by Camilo Ravelo Durán -
thank you very much. Can it also be memcached ?. With the sessions folder of the moodledata you can also do that?
Average of ratings: -
In reply to Camilo Ravelo Durán

Re: Problems mounting Moodledata via NFS on CentOS

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Yes and yes. However, my personal (not terribly scientific) experience is that Redis provides better performance. Seeing as we're there, also consider moving the locking from NFS to the database (DON't move sessions to the database). The configuration for sessions and locks is in config.php (See config-dist.php).

I haven't used memcached in a while but it certainly used to be the case that you needed a separate instance for sessions - not the case with Redis.
Average of ratings: Useful (1)
In reply to Howard Miller

Re: Problems mounting Moodledata via NFS on CentOS

by António Godinho -
I've followed the instrutions on https://docs.moodle.org/38/en/Redis_cache_store, and now I have REDIS on Application and session:

This plugin is still valid for version 3.8?
https://github.com/blackboard-open-source/moodle-local_redislock
Average of ratings: Useful (1)
In reply to António Godinho

Re: Problems mounting Moodledata via NFS on CentOS

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
No idea. I would just use the database locking.
Average of ratings: -
In reply to Howard Miller

Re: Problems mounting Moodledata via NFS on CentOS

by António Godinho -
Ok I'll do that!
Anyway, that plugin gives a error Lock Factory set does not exist...
Average of ratings: -
In reply to António Godinho

Re: Problems mounting Moodledata via NFS on CentOS

by Camilo Ravelo Durán -
Antonio, how did you configure redis for the different moodle nodes? Should it be a separate machine, where the moodle nodes point to it ?. I am lost with the redis configuration since I have a load balancer with 5 machines with the same moodle. Thanks....
Average of ratings: -
In reply to Camilo Ravelo Durán

Re: Problems mounting Moodledata via NFS on CentOS

by António Godinho -
Hi Camilo,

I'm not there yet. I'm working on a webnode, that I'll replicate into 3 or 5 once it's donne.
The server that runs the NFS server, will also run the redis. So all will point at the same server, using the $CFG->MR_SHORT_NAME.
In theory it should work, but hopefully someone else can confirm that.
Average of ratings: -
In reply to Camilo Ravelo Durán

Re: Problems mounting Moodledata via NFS on CentOS

by António Godinho -
I'm will also use a NFS cluster of 2 servers, just to be safe.
Average of ratings: -
In reply to António Godinho

Re: Problems mounting Moodledata via NFS on CentOS

by Camilo Ravelo Durán -
I'm not sure if using redis on the same NFS server is the best....
Average of ratings: -
In reply to Camilo Ravelo Durán

Re: Problems mounting Moodledata via NFS on CentOS

by António Godinho -
Hi again,

Maybe Howard or someone else can help and clarify if I'm thinking this the wrong way.

I'll have a NFS failover with 2 servers, both will server redis. One will provide the services, the other running as backup. 
This means that at any time, only one will server NFS and redis services.

NOTE: On the diagram below, there are 3 Nginx+HAproxy. I have those running for several years, serving multiple websites including Moodle. 

My ideia is this:
Moodle
Average of ratings: Useful (2)
In reply to António Godinho

Re: Problems mounting Moodledata via NFS on CentOS

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Failover is incredibly tricky and expensive. Often, all you do is increase complexity and move your single point of failure somewhere else.  It's very rare that Moodle is *really* so critical that you need it. Sure, have hot backups but you can probably live with the minutes it takes to swap wires about or change DNS settings on the very rare occasions that you actually have to do it.
Average of ratings: Useful (1)