Installing and upgrading help

(1.9 - 2.2) Moodledata missing/deleted files :(

 
Picture of Angel MM
(1.9 - 2.2) Moodledata missing/deleted files :(
 

Good morning mates, I'm having problems while updating moodle from 1.9 to 2.2:

The thing is that in some step of the auto upgrade proccess the course files uploaded moodledata directory (700 permissions for directory and 600 permissions for files) get deleted without getting any trace/debug error. I tried to manually restore moodledata before installation but it seems that the access to those files in moodle 2.2 is done differently than the 1.9 method, so I don't really know why does that happened :/

There is some tracker issue concerning something similar thing in the 2.3 version upgrade, but it is not helpful.

My moodle is stored in an external hosting and all files/directories have the same owner:group. Could that be the problem? It is so strange...

Anyway, thanks in advance for the help, best regards smile

 
Average of ratings: -
Picture of Howard Miller
Re: (1.9 - 2.2) Moodledata missing/deleted files :(
Group Core developersGroup Documentation writersGroup Particularly helpful MoodlersGroup Plugin developers

Yes - the file storage is completely changed. 

The most likely thing is that something went wrong with your upgrade and you started again without fully restoring. When going from 1.9 to 2.x it is *vital* that you fully restore the site if something goes wrong before trying again. That means both files and database as both could be in an undefined state. 

 
Average of ratings: -
Picture of Angel MM
Re: (1.9 - 2.2) Moodledata missing/deleted files :(
 

If I understand it right, you mean that i have to start over again, huh? Well, I already did that with a full database/moodledata backup I had and it happened again. Also, css and icons are missing too.

Thanks for your fast reply smile

 
Average of ratings: -
Picture of Emma Richardson
Re: (1.9 - 2.2) Moodledata missing/deleted files :(
Group Documentation writersGroup Particularly helpful Moodlers

Can you walk us through how you upgraded?  Did you copy over all your plugins?  Has the website moved?  Different url?

Just noticed that you said auto upgrade?  Are you using a builtin installer?  Don't do that - I suspect that your moodledata is wrongly located within your moodledata folder.  So, you need to download the new moodle code from moodle, name it moodlenew, copy over your moodledata folder, your config.php file and any plugins to the new code folder.  Then, rename your old moodle folder to moodleold and your new one to moodle (presuming that is what your folder is called now) and you should be good to go.

 
Average of ratings: -
Picture of Angel MM
Re: (1.9 - 2.2) Moodledata missing/deleted files :(
 

That's exactly what I did (called it auto upgrade, my bad hehe). I also tried it locally on a machine I have and the upgrading process from 1.9 to 3.2 worked perfectly (This moodle of mine was not that big but the file I uploaded in a test course remained untouched).

The more I investigate, the more issues I found: JavaScript is not working properly, so I cannot expand configuration menus...

Thanks for the reply smile

 
Average of ratings: -
Picture of Angel MM
Re: (1.9 - 2.2) Moodledata missing/deleted files :(
 

I also have another question: Do files on new versions are stored in the DB as BLOB datatypes or they should be in the moodledata directory?

Thanks! smile

 
Average of ratings: -
Picture of Emma Richardson
Re: (1.9 - 2.2) Moodledata missing/deleted files :(
Group Documentation writersGroup Particularly helpful Moodlers

The files are in the moodledata folder.  If the upgrade worked on a different machine then obviously something went wrong.  Have you checked permissions?  Made sure that they are recursive?  

 
Average of ratings: -
Picture of Angel MM
Re: (1.9 - 2.2) Moodledata missing/deleted files :(
 
I did it with a find linux command, plus i checked the second time that all were set correctly. The only thing that cannot be changed is the owner:group, so the moodle installation/upgrading recommendations cannot be satisfied, but it sounds so strange to me that an owner:group settings make moodle remove all of the files, huh? I'll have to call the hosting and see if someone had this problem too or try a fresh installation with those settings and see if the same happens to the moodledata folder.

Thanks smile

 
Average of ratings: -
Picture of Emma Richardson
Re: (1.9 - 2.2) Moodledata missing/deleted files :(
Group Documentation writersGroup Particularly helpful Moodlers

The moodle installer will not remove the moodledata file folder.  I suspect it did not copy over correctly.

 
Average of ratings: -
Picture of Angel MM
Re: (1.9 - 2.2) Moodledata missing/deleted files :(
 

That's what i though at first but, if moodle is not able to copy files, it shouldn't be able to remove them either, huh?

 
Average of ratings: -
Picture of Emma Richardson
Re: (1.9 - 2.2) Moodledata missing/deleted files :(
Group Documentation writersGroup Particularly helpful Moodlers

Moodle can create and delete files through the interface when needed.  But the installer does not know what has been added to an old install and does not know automatically where the moodledata folder is.  Normally moodledata is kept outside of the moodle code folder and so it is not an issue during upgrade.  Did you just copy within your controlpanel or did you download and upload again with ftp?  I wonder if you could not simply move it to the new moodle folder and avoid the whole copy process - that should keep it intact.  Hopefully your webhost does have backups...

 
Average of ratings: -
Picture of Angel MM
Re: (1.9 - 2.2) Moodledata missing/deleted files :(
 
Good evening, I kept investigating what could be happened and I found out that Moodle 2.2 stores files in a different way than Moodle 1.9 (as I suspected). It does it using something called Moodle Repositories and Moodle 1.9 files are referred as Legacy Course Files. Could this be the problem? If so, where are those repository files to be accessed by all migrated courses?
Sorry if I'm being annoying and thanks for helping me smile
 
Average of ratings: -
Picture of Emma Richardson
Re: (1.9 - 2.2) Moodledata missing/deleted files :(
Group Documentation writersGroup Particularly helpful Moodlers

That is not going to affect your css etc.  And you should still have access to your legacy files as before.  Are your courses all still there?  Could you show us some screenshots of what you are seeing.

You mentioned migrated?  Has this changed to a different server/different url?

 
Average of ratings: -
Picture of Angel MM
Re: (1.9 - 2.2) Moodledata missing/deleted files :(
 

It can be seen above, there are so much less pdfs in the moodle22 moodledata directory. And, at first, moodle22 css didn't load properly (I had to reload the page); plus moodle22 is slower than moodle19 (same internet speed connection). I should notice too that moodle22 upgrading has to be manually reloaded and I thought that the process were automatic (progress bars and all of that stuff, and yes, JavaScript is enabled and working). I refered as migrated courses to the "new version" of the moodle19 courses that should work correctly.

Thanks for your time smile

 
Average of ratings: -
Picture of Emma Richardson
Re: (1.9 - 2.2) Moodledata missing/deleted files :(
Group Documentation writersGroup Particularly helpful Moodlers

So when you do the same count on the dev server that worked, do you see the full number of pdfs?

If so, this goes back to my point that the copy probably did not work successfully and you should try moving the folder instead of copying it.

 
Average of ratings: Useful (1)
Picture of Ken Task
Re: (1.9 - 2.2) Moodledata missing/deleted files :(
Group Particularly helpful Moodlers

Pardon the intrusion ... have done (many moons ago) several of these 'mirgrations' from 1.9.x to 2.2.x in the past and had to investigate such issues.

The following will not specifically address the specific issues you are seeing as no two such migrations had the same results.

Emma could be correct.

Hopefully, the following explanation will help trouble shoot.

So ... having said that ...

First screen shoot shows the data directory has already been touched by
a migration attempt - evidence of 'filedir' in moodledata.

The numbered directories are the old 1.9.x course ID numbers (CID's).
Your find command might have found PDF's in those course ID directories ...
certainly NOT in filedir.

Adjust the command and limit it to single digit CID's and drop the word count.

find ./? -name "*.pdf"


If you try the same command in the filedir directory, it will return no hits.
That's because of the massive change to filesystem in Moodle.
There are no files in filedir that have a file name extension.

Instead, to find a file in filedir, it's a two step process one of which is
a DB query of mdl_files to find the contenthash then using that information
to find the directories/subdiretories/contenthash fileanme of a file then
using something like file -b to double check it is a PDF or other file type.

DB query from command line looking for any file that ends with .pdf:

mysql -u root -p'YOURPASSWORD' -e 'use YOURMOODLEDB;select filename,contenthash from mdl_files where filename like "%.pdf";'

Will/should produce a long listing like this:

| Harry Potter_1_0.pdf | 4c4fa72addf6cfc16c6149bde7cedd01b70af1cf |
| Harry Potter_2_0.pdf | f18a1fb0516afcd77dbdf6380d03a7cbba72e1d2 |

Now one can use the file -b command on the contenthash from filedir directory:
(example looking for the 2nd file listed above):

file -b ./f1/8a/f18a1fb0516afcd77dbdf6380d03a7cbba72e1d2

Would have returned:

[root@sos filedir]# file -b ./f1/8a/f18a1fb0516afcd77dbdf6380d03a7cbba72e1d2
PDF document, version 1.4

Can't view it so you could *copy* that file out of filedir sea of files to web root and change it's name to view with
browser.  (not recommending you do this for all files ... it's just an example);

Still in filedir

cp ./f1/8a/f18a1fb0516afcd77dbdf6380d03a7cbba72e1d2 /var/www/html/HarryPotter_2_0.pdf

Then use your browser:

http://site/HarryPotter_2_0.pdf

* one exception has to do with Certificates issued in the 1.9.
So to see if that's true in your situation, change into the moodledata directory and issue the following command
on just one of the course ID directories you see:

ls -lR ./CIDNUMBER/

IF you see a modinfo/##/##/ and then a file name like 'Certificate blah blah.pdf' ... those are certificates that could not be moved into new file system.

Attached is a screen snap of a moodledata directory and it should look once cleaned up.

*** what happened? *** Dunno ... hard to say.

You've given some hints that you might be hosted with a provider that moved a shared account to a VPS system ... fact you could not use the true web server user (assuming CentOS - that would be apache:apache - on Ubuntu that would be/should be www-data:www-data).

Even VPS systems might have CAPS ... when processes reach those CAPS the script is killed ... no warnings, no errors recorded anywhere one might look. :\   I would hope not, but that depends upon the 'VPS package' purchased.

The other hint is where you are working ... /home/username/public_html/ indicates even if on a VPS the provider has set you up in a 'user jail' ... and not at true Apache web root .... which on CentOS 7 and Ubuntu 16.04 which are both Linux Standards Based distro's now ... is /var/www/

Now how to fix ... gets in to a lot more detail ... which could be very specific to your situation.  I've never been able to 'fix' anything via 'Vulcan Mind Meld'! smile   And, NO, am not asking for access to fix either.

just re-called .. I did have to tweak a MySQL setting ... number of opened files ... for some of the migrations.   Got to the point where I did that in advance before the first attempt.  NOTE: am not a DB admin by trade ... just a lowly Intructional Tech person.


'spirit of sharing', Ken



 
Average of ratings: Useful (1)
Picture of Angel MM
FOUND THE ANSWER
 
Both of you have completed the solution at 90%.

It's true (as Emma said) that files in the fresh installation modified too, but not deleted. Here is where Ken add his part: as I suspected, files are stored (differently in >2.x versions, I think) in the filedir directory, hashed or something like that (that file command is so useful, isn't it? hehehe).

Well then, investigating why new Moodle files were not reachable (clicking in the resource linked to a pdf, for example), I search that error and discovered something called Slash Argument. It seems that some hosting php settings cause an incompatibility with this feature.

I dissabled it and all back to normal.

Administration > Site administration > Server > HTTP

https://docs.moodle.org/32/en/Using_slash_arguments

Thanks for all the support, 10/10 community smile

 
Average of ratings: -
Picture of Ken Task
Re: FOUND THE ANSWER
Group Particularly helpful Moodlers

Yeah!  That was one of those little details that happened somewhere in the 'Moodle March'.   So congrats!   Job well done.  There might have been a clue for that in web server error logs.

But, let this also be a lesson for you cause no one knows just how smoothly a jump from Moodle 3.x to a 4.x will go - unless one researches every version in between.

Advice ... don't wait too long to 'march' a moodle forward.   Don't get into a situation where one is looking at what I'll call a 'hyperjump'.   'Not in Kansas anymore, Toto!' ;)

Set up a sandbox instance and plan your 'march' *IF* you wait to 'hyperjump'.   That sandbox instance can but spun up rather easily *IF* using Google Compute Engine or Amazon me thinks.

Last piece of advice ... learn how to use git (you do have Linux) and the commands found in /moodlecode/admin/cli/.   Using git and the scripts in cli one can rather easily update but also upgrade.  Performing a "Moodle March" is possible then and one doesn't have to attempt a 'hyperjump'.   Yes, git meant for programmers ... but why shouldn't you take advantage of what programmers use?

2 cents worth ... I know! ;)

'spirit of sharing', Ken




 
Average of ratings: -