moodledata owner and test->prod migration

moodledata owner and test->prod migration

by Josiah Carberry -
Number of replies: 7

This question is about both a specific detail and a general approach.

In the course of trying to delete some backup files, I got this message after saving the change (the deletion of the files):

Warning: chmod(): Operation not permitted in /home/www/blahbalhbalh/web/moodle/lib/filestorage/file_storage.php on line 1900

I am assuming that this error has to do with the ownership of files under moodledata/filedir. The files themselves have 777 permissions, so that, in and of itself, is not the issue.

So, the detailed question is whether someone can confirm that the files concerned are in the filedir area and can someone tell me what the program was trying to do when it failed.

The more general question concerns the strategy for moving from a test to a prod environment, and its impact on the above issue. My initial strategy was to develop a site in a test environment and when it was ready, to copy the file system from test to prod (both the moodle and the moodledata areas), and to backup the test database and restore it to prod. The configurations that depended on things like the host name and installation location were also updated in prod.

Until, this approach has worked fine. But, because my prod environment is a shared platform, I cannot manage the files that belong to the Apache account. So, when files are added programmatically to anywhere under moodledata, their owner is the Apache account. But the original files in prod all belong to an account that I manage. While I can take ownership of Apache account files, I cannot do the opposite.

I am assuming that this issue is at the source of the problem I am now seeing above, although I am not really sure why. But more importantly, is there a different way of migrating wholesale the files in moodledata/filedir to another site such that the owner remains the Apache account on my shared platform? The only things I can think of are:

a) upload all those files again using Moodle, which is both extremely tedious, error-prone, and would require my abandoning the whole approach of the initial setup of the prod env. based on the test env.

b) asking my hosting provider to change the ownership of the files, which is something they have been loathe to do in the past.

c) moving to a new hosting provider or to a platform dedicated to me, that I more fully control, which is a major step I would like to avoid, if at all possible.

Any other suggestions?

Average of ratings: -
In reply to Josiah Carberry

Re: moodledata owner and test->prod migration

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

> This question is about both a specific detail and a general approach.

The general approach is this: https://docs.moodle.org/en/Moodle_migration.

To the specific problem:
> In the course of trying to delete some backup files, I got this message after saving the change (the deletion of the files):
>
> Warning: chmod(): Operation not permitted in /home/www/blahbalhbalh/web/moodle/lib/filestorage/file_storage.php on line 1900
>
> I am assuming that this error has to do with the ownership of files under moodledata/filedir. The files themselves have 777 permissions, so that, in and of itself, is not the issue.

Obviously a permission is missing in moodledata/filedir for your webserver process. You stressed that the files have 777 permission, how about the directory moodledata/filedir ($ ls -ld /path/to/moodledata/filedir)?

> So, the detailed question is whether someone can confirm that the files concerned are in the filedir area and can someone tell me what the program was trying to do when it failed.

Hardly, unless that someone has shell or equivalent access to your hosting account.

> The more general question concerns the strategy for moving from a test to a prod environment, and its impact on the above issue. My initial strategy was to develop a site in a test environment and when it was ready, to copy the file system from test to prod (both the moodle and the moodledata areas), and to backup the test database and restore it to prod. The configurations that depended on things like the host name and installation location were also updated in prod.

In that case, it must have been compatible with the Moodle_migration doc mentioned above.

> Until, this approach has worked fine. But, because my prod environment is a shared platform, I cannot manage the files that belong to the Apache account. So, when files are added programmatically to anywhere under moodledata, their owner is the Apache account. But the original files in prod all belong to an account that I manage. While I can take ownership of Apache account files, I cannot do the opposite.
>
> I am assuming that this issue is at the source of the problem I am now seeing above, although I am not really sure why. But more importantly, is there a different way of migrating wholesale the files in moodledata/filedir to another site such that the owner remains the Apache account on my shared platform? The only things I can think of are:

You have to come to terms with the fact that the ownership of moodledata is different between your test environment and the prod environment and find a suitable copying procedure.

> a) upload all those files again using Moodle, which is both extremely tedious, error-prone, and would require my abandoning the whole approach of the initial setup of the prod env. based on the test env.
>
> b) asking my hosting provider to change the ownership of the files, which is something they have been loathe to do in the past.
>
> c) moving to a new hosting provider or to a platform dedicated to me, that I more fully control, which is a major step I would like to avoid, if at all possible.

It is not yet proven that the radical solutions a) and c) are necessary. I would try to find a suitable copying method. You haven't told us how exactly you copy them.
Average of ratings: -
In reply to Josiah Carberry

Re: moodledata owner and test->prod migration

by Ken Task -
Picture of Particularly helpful Moodlers

A lot to digest/respond to in a single posting!   But ... I'll pick a couple ... MUCH depends upon some questions asked here though ... saves time in looking at your previous postings to see what version of Moodle you are running on which system ... development/production.  And it doesn't hurt to be specific about WHERE your production site is hosted either.  That's not disclosing any deep/dark secrets ... heck, we're talking about a Web based application here! :|

Your development site is local but what flavor of OS is it on?  Linux/Windows/Mac?

We know that you are on shared hosting ... which greatly limits what one can do ... might need to do ... but that's where the service provider is supposed to take care of customers needs to run whatever app.  From hints you've provided I'd say Linux.   Some remotely hosted systems have 'extra protections' for PHP apps customers run on shared systems ... suPHP is one ... it has sometimes messed with stuff.   Got anything like that on the remotely hosted site?

When one logs onto the remotely hosted shared system, whatever you upload is tagged (or should be tagged) as belonging to you (owner and group).   Your group, however, should be included with apache web services group.   There is one exception to that ... if your development box is Linux the backups might be tagged as owner/group apache ... depending upon the OS.

If you suspect an ownership/group issue not resolved when uploading backups, etc., from development system to production system, one could 'cheat' and setup the development system to run under the same users/group names.

But, for your last comment ... it appears that you are not ready to take on more.

So here's a question ... If developing on local gives you so many issues, why develop on local?  Why not just hide on production server what you are working on ... ie, not available to any student/teacher and only you as Admin level user can see it.  Besides that, what you are developing locally can sometimes fool you ... a large video *will* play fast locally ... but take that large video (oh say 20+ gigs) and put it on a shared Web server will NOT play fast ... IF at all.  So developing on production server is true to that setup - what you see/can do is what you get - and what your students will see/get!

Bottom line ... are you setting up a situation where you are having to work to work?   Suggest making your Moodle life simple ... ONE SERVER ... develop on the production server ... and all of your current issues will be resolved! ;)

'spirit of sharing', Ken

Average of ratings: -
In reply to Ken Task

Re: moodledata owner and test->prod migration

by Josiah Carberry -

Well, Ken, testing in production new plugins, making sure that I have the permissions right, testing emails sent to students, and ensuring that all the criteria for making a resource visible are set correctly, and so forth, really, really rubs against the grain. I abhor doing this directly in prod. But once I get everything set up in test and demonstrate that it works, redoing the exact same thing in prod is an error-prone activity. We all make little mistakes. This might be OK for some little things that are easy to test and easier to correct. But it is much harder to do if there is a complex exam with 40 questions, or a course certificate that should only be visible once you have completed all the requirements for the course. In my particular environment, any functionality available in prod is susceptible to being used as soon as it is visible to students. My courses are all self-paced, with no start or end dates for the individual lessons.

I appreciate the issue of performance differences, but I have sorted that out in advance of any course being available. But a proper test environment should really be lodged on the same sort of platform at the same hosting provider, if performance issues are really at risk.

The bottom line is that I really have only a very few issues, once you eliminate the issues that are part of my learning curve. 99.9% works OK. I just want to get all my system administration procedures working smoothly and reliably.

Average of ratings: -
In reply to Josiah Carberry

Re: moodledata owner and test->prod migration

by Ken Task -
Picture of Particularly helpful Moodlers

Ok, now there is a good reason for complicating one's Moodle 'life' ... testing/learning new add-ons/plugins - info not disclosed before and cannot be aquired via 'Vulcan mind meld'! ;)

Somewhere in these you mention failure of automated backups? and Moodle not being able to finish nor clean up.  Known issue if the course backups get 4Gig +.   Relates to operating system limitations which on remotely hosted shared system might have other factors as well.  After Moodle creates the backup zip/tar.gz by mbz name, the code uses 'copy' to move the backup file to destination ... could be filedir, could be filedir  + other, or just other.   In any event, the OS  has an issue with 'copy' ... if one had command line access one could find the completed .mbz file and 'MOVE' it to a location with no problem but even as super user a 'cp' command fails.   Not sure there is a solution to that although the default of using tar.gz in backup as opposed to zip for the .mbz file has solved it somewhat.

As far as Moodle not cleaning up ... any process that ends before completion would/could not know if it needs to clean up or not.   Nature of the beast, am afraid.   Sometimes manual clean up is needed/required.   Those things that involved files in the new file system as you know require both manually removing records in the DB but also in filedir (VERY CAREFULLY).

The desire for 'all system administration procedures working smoothly and reliably' always a goal ... forever changing as one marches Moodle upwards.

BTW, may have a 'badge', but that doesn't say how many times the owner of that badge tried something and failed ... this to say, have more experience with what NOT to do than most! ;)  A long the way, however, one learns some 'backdoors'/other ways of arriving at destination.

'spirit of sharing', Ken

Average of ratings: -
In reply to Josiah Carberry

Re: moodledata owner and test->prod migration

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

If the files and their containing directories are "chmod -R 0777" then you wouldn't have got that error. If everybody is allowed to write then the ownership is irrelevant. 

If you turn on Debugging then you might get more info. But really, it's a simple permission mistake. 

Make sure you understand how ownership and permissions work on your system. It's a very good investment wink

Average of ratings: -
In reply to Howard Miller

Re: moodledata owner and test->prod migration

by Josiah Carberry -

OK - so I checked once again the permissions of the directories under filedir and  they are all 777. The files themselves are 666. Then, I checked in the table mdl_files to see where the files are physically located. I am assuming that this is indicated by the first 4 characters of the value in the hash value. Out of 5 files that were supposedly created by the automatic backup, 4 are simply not there. This is obviously the cause of the error I say, not a problem with permissions.

The question of why they are not there is probably related to an earlier problem I had where the backup was crashing after the backup was created in the temp directory, but failed to be moved over to the ultimate place under filedir. So moodle is apparently inserting a row in mdl_files before the file itself is confirmed as created.

So now I need to get rid of the spurious rows in mdl_files. I cannot do this programmatically, as the program crashes with the error messaged mentioned in the original message. Can I simply delete them in the database? Or is there more to it than that?

Average of ratings: -
In reply to Josiah Carberry

Re: moodledata owner and test->prod migration

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I believe that should be safe. But, as ever, take a backup first wink

Average of ratings: -