Randomly get "Dataroot not writable" error

Randomly get "Dataroot not writable" error

by Adam H -
Number of replies: 21

Hi All

We are running Moodle 2.2.5 on a CentOS Linux server and are constantly randomly getting the "Fatal error: $CFG->dataroot is not writable, admin has to fix directory permissions! Exiting." error just randomly,, the site will function fine for a while then suddenly you'll click on something (anything that's clickable in Moodle), and then will be presented with this error on a white screen..

We've tried all sorts of things to remidy the issue, first off we altered the SELinux permissions on the moodle data directory using "chcon -Rv --type=httpd_sys_content_t /var/moodle_data" ,, and this would work for a couple of days, then suddenly we would start to see the dataroot error appearing again randomly (possibly every 20 clicks or so).. We tried the "semanage fcontext -a -t httpd_sys_content_t "/var/moodle_data(/.*)?" command and this seemed to apply a more permenant fix,, albeit about 5 days later the error would randomly return again..

Growing disheartened by the problem we simply turned SELinux off per advice on many other forum posts regarding the issue,, needless to say a day or two later the error simply returns (completely randomly)..

This is driving everyone completely bonkers now.. Has anyone else encountered a similar problem????

Thank you for any help in advance!!

..Adam

Average of ratings: -
In reply to Adam H

Re: Randomly get "Dataroot not writable" error

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Switch on Debugging. Watch your logs. Hope for some commonality in the errors that gives you a clue.

Are you *sure* that your permissions are correct - in *all* the dataroot subdirectories?

In reply to Howard Miller

Re: Randomly get "Dataroot not writable" error

by Adam H -

Debugging doesn't report any other messages..

Moodle data and ALL subdirs are set to 777 and owned by apache:root

There is no commanality with the error,, sometimes it happens when you click 'login',, sometimes it happens when you click on a course category,, some times when you click a users profile,, etc etc

I'm leaning towards there being a problem with the physical hard disk at the moment mixed because it doesn't make sense to be so random..

In reply to Adam H

Re: Randomly get "Dataroot not writable" error

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

If debugging is on properly - I am very doubtful that there are no additional messages. Have you checked your web server error log? The underlying PHP error should be logged somewhere.

You could be correct about the disk but, in my experience anyway, the Linux filesystem would normally have dropped into 'read only' mode by now.

In reply to Howard Miller

Re: Randomly get "Dataroot not writable" error

by Adam H -

Ok I tell a lie there are thousands of the following error:-

PHP Notice: Trying to get property of non-object in /var/www/html/lib/setup.php on line 76

and

PHP Notice: Trying to get property of non-object in /var/www/html/lib/setup.php on line 69

which is a call to the function if(!is_writable($CFG->dataroot))

But the directory is definitely writable because Moodle functions fine.. What would cause it to suddenly not be writable every few clicks and flag up this message?

I think you might be right regarding the harddrives,, i've seen harddisks failing before and they do indeed get forced into readonly causing all sorts of other problems.

Thank you for your help,, it's much appreciated.

In reply to Adam H

Re: Randomly get "Dataroot not writable" error

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

So it's got nothing to do with writing to the disk then wink

The issue is that $CFG isn't being setup properly. That's even weirder. Are there no other notices at all? Is there any possibility you have been hacked (most likely you will have left config.php or the whole of your moodle program files writable by the web server user for that to be likely)

EDIT:
Hmmm.... let's see your config.php (please obfusticate passwords)!!

In reply to Howard Miller

Re: Randomly get "Dataroot not writable" error

by Adam H -

Here is the config file:--

<?php  // Moodle configuration file
 
unset($CFG);
global $CFG;
$CFG = new stdClass();
 
$CFG->dbtype    = 'mysqli';
$CFG->dblibrary = 'native';
$CFG->dbhost    = 'localhost';
$CFG->dbname    = 'moodle';
$CFG->dbuser    = 'moodle_user';
$CFG->dbpass    = 'xxxxxx';
$CFG->prefix    = 'mdl_';
$CFG->dboptions = array (
  'dbpersist' => 0,
  'dbsocket' => 0,
);
 
$CFG->wwwroot   = 'http://moodle.someserver.com';
$CFG->dataroot  = '/var/moodle_data/data';
//$CFG->tempdir = '/backup/temp';
$CFG->admin     = 'admin';
 
$CFG->directorypermissions = 0777;
 
$CFG->passwordsaltmain = '';
 
require_once(dirname(__FILE__) . '/lib/setup.php');
 
// There is no php closing tag in this file,
// it is intentional because it prevents trailing whitespace problems!
//$CFG->debug = 6143;
//$CFG->debugdisplay = 1;

I've dealt with many moodle's and I can't see anything wrong with the setup of this particular config file sad.. What a nightmare!

Thank you Howard

(Edited by Howard Miller - original submission Wednesday, 28 November 2012, 2:42 PM)

In reply to Adam H

Re: Randomly get "Dataroot not writable" error

by Olja Petrovic -

I'd obfuscate the password salt too (any moderators?).

That's strange, it's not even the permissions of config.php. I'll think about it.

In reply to Olja Petrovic

Re: Randomly get "Dataroot not writable" error

by Olja Petrovic -

Still no ideas.

Have you addes any custom code, any contributed modules? Anyone including setup directly? Do a search to find out.

Do you have a local install to test on, maybe put some error_debug and backtracing around in local or test install?

Before replacing the source code make a copy of it to study later. Database also, of course.

In reply to Olja Petrovic

Re: Randomly get "Dataroot not writable" error

by Olja Petrovic -

Even though setup.php checks that $CFG is set right away. As if it was set, but not an object!? Absurd.

In reply to Adam H

Re: Randomly get "Dataroot not writable" error

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I know they're commented out, but the require for setup.php should be the last thing in this not the debugging switches.

Other than that, I'm desperately hoping you've found some other notices in your logs. I'm struggling really. Just to be completely safe, I would replace your Moodle code with a clean copy from moodle.org just in case something has got screwed up. Also, run a repair of your database tables.

In reply to Adam H

Re: Randomly get "Dataroot not writable" error

by Ken Task -
Picture of Particularly helpful Moodlers

Due to randomness, wonder if it's a cacheing issue:
http://www.php.net/manual/en/function.is-writable.php
in the Notes: The results of this function are cached
 
Apache also has some control over that … mod_expires
https://httpd.apache.org/docs/2.2/mod/mod_expires.html

'spirit of sharing', Ken

In reply to Ken Task

Re: Randomly get "Dataroot not writable" error

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I think we've established that it's nothing to do with writing the files. It's actually the $CFG object getting stuffed up. I'd have been a lot happier if it was writing to the files.

I wonder if the silence means he's fixed it?

In reply to Howard Miller

Re: Randomly get "Dataroot not writable" error

by Adam H -

Hi All

Well I upgraded to 2.2.6+ today out of fears there might have been something dodgy in the 2.2.5+ code that the site was running previously...

Since the upgrade there have been no reports of the $CFG->dataroot error, but i'm not really holding my breath that it is fully resolved yet.. Previously any changes i'd make to recifiy the problem (ie, SELinux permission changes etc) would result in no occurance of the error for even bouts of 1 or 2 days or more, then we would suddenly start to see it again.

So far though, no reports of it from the last two hours.. Here's to hoping we have a fix!!

Thank you for your help!!

In reply to Adam H

Re: Randomly get "Dataroot not writable" error

by David M -

Im having the exact same issue here with Moodle 2.4, i know directory permissons are correct, im at a total loss of what to do next. Running on CentOS 6.3 with stock PHP/Apache/MySQL packages. 

In reply to David M

Re: Randomly get "Dataroot not writable" error

by Adam H -

Hi David

It may well be worth updating your code to the latest release of the version you're using.. When you say 2.4 do you mean 2.2.4? or the latest beta 2.4 release?

We had this issue for a good number of weeks on 2.2.5+ and I'm happy to say that since updating to 2.2.6+ last wednesday we have not seen a single occurence of the dataroot error. (touch wood!!!)

Regarding your CentOS stack, is SELinux running on your machine?

Regards

In reply to David M

Re: Randomly get "Dataroot not writable" error

by Susana L. -

Same problem!
Randomly "$CFG->dataroot is not writable, admin has to fix directory permissions! Exiting."

Same PHP Notices on moodle-error_log:
Trying to get property of non-object in /var/www/html/moodle/lib/setup.php on line 69,...
line 70 and line 76

Moodle 2.2.4.

In reply to Susana L.

Re: Randomly get "Dataroot not writable" error

by Susana L. -

Reloading Apache seems to have (temporarily??) solved the problem.

In reply to Susana L.

Re: Randomly get "Dataroot not writable" error

by Ken Task -
Picture of Particularly helpful Moodlers

First ... this is a guess - won't hurt to try IF one can edit php.ini.  The fact that apache restart solves the problem leads me to think that maybe the following would fix.

Apologize for the technical in this, but maybe someone more knowledgeable than I can understand/correct if this is in-correct:

Line 69 shows:

$CFG->dataroot = realpath($CFG->dataroot);

Which looks to me like it's setting php's realpath from the config.php file's dataroot variable.

From what I an gather, php's realpath is cached in php which is controlled.  Because of the randomness behavior, wondering if there's an issue with the realpath cache.

Create a flie: realpath.php
with the following contents:


<?php
var_dump(realpath_cache_get());
?>

make sure ownership/permissions belong to the apache user/group, then it the file via browser.

The information will not be formatted, but in browsing one can see expires settings.  On that page that's returned do a find for your moodle directory.  In the example below, the file was placed at the moodle document root in a moodle23 directory.
Find 'moodle23'
One might see something similar to:

 ["/var/www/html/moodle23"]=> array(4) { ["key"]=> int(-1678145344390156167) ["is_dir"]=> bool(true) ["realpath"]=> string(22) "/var/www/html/moodle23"  ["expires"]=> int(1357236007)

Info:
http://php.net/manual/en/function.realpath-cache-size.php
Comment/notes quote:

"realpath_cache_size" is used by PHP to cache the real file system paths of filenames referenced instead of looking them up each time.  Every time you perform any of the various file functions or include/require a file and use a relative path, PHP has to look up where that file really exists. PHP caches those values so it doesn’t have to search the current working directory and include_path for the file you are working on.

Consider editing php.ini
See: http://us2.php.net/manual/en/ini.core.php#ini.realpath-cache-size
Increasing

realpath_cache_size -> 16K which is default

Decreasing:

realpath_cache_ttl <- 120 which is default

In a CentOS Linux server php.ini the following:

; Determines the size of the realpath cache to be used by PHP. This value should
; be increased on systems where PHP opens many files to reflect the quantity of
; the file operations performed.
; http://www.php.net/manual/en/ini.core.php#ini.realpath-cache-size   
; realpath_cache_size = 16k

; Duration of time, in seconds for which to cache realpath information for a gi$
; file or directory. For systems with rarely changing files, consider increasin$
; value.
; http://www.php.net/manual/en/ini.core.php#ini.realpath-cache-ttl
; realpath_cache_ttl = 120

any change to php.ini requires restart of apache service.

NOTE: cannot be used in .htaccess it is PHP_INI_SYSTEM only.

'sprit of sharing', Ken

Average of ratings: Useful (1)
In reply to Susana L.

Re: Randomly get "Dataroot not writable" error

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Which PHP accelerator are you using? I wonder whether it could be a mis-behaving accelerator having issues with the $CFG variable?

It's unusual, but the fact that restarting apache seems to fix it makes me wonder.