Broken Moodle - please help!

Broken Moodle - please help!

by Elaine Topham -
Number of replies: 26

Hello,

Please excuse my lack of knowledge but it is my first time posting here and I am hoping for some help with our Moodle problems ... I would appreciate any help/suggestions at all at this point!

So, to give a little background on the issue, a couple of weeks ago a request came in to reactivate the Calendar block that we had disabled... however, we didn't want to bring it back on all 2000 pages, so someone in the team decided it would be good to delete the block manually and reinstall... this didn't go as planned!!  So we had to manipulate things a little, and eventually "fixed" things.   A week later we attempted to update a plugin, however this appeared to corrupt our Moodle installation and it was stuck in a loop trying to update the Moodle database.  When we got Moodle back up and running, it still wasn't right - please see the print screen attached for an easier explanation of what it looked like.  Our IT guys (they deal with the server side stuff, my team deal with the web administration) attempted to restore Moodle to the backup that was taken at 4am that day, however they experienced problems with this also (I haven't got details about the issue here, sorry)

We were running Moodle 2.9.4 on IIS/SQL Server on 3 virtual servers (clustered?) - forgive my knowledge in this area, I don't deal with this normally!  Something happened to 2 of the clusters, so we have been running Moodle on one since about 15th January.  Last Thursday, something happened and our Moodle went offline and we haven't been able to revive it since!

Yesterday, we updated Moodle to 3.0.2, in the hopes that this may fix the problem... it did for a while, though many of the plugins were giving errors - e.g. Files not on disk.  This morning, we tried to remove the outdated/missing plugins, starting with the tutorial bookings plugin, but this reverted us back to the issue shown in the print screen.

Also, we have located unusual spaces in the filenames for REALLY key files, like index.php and view.php... so they were "index. php" and "view. php".  Removing the spaces fixes the mycourses issue... but we are now stuck with a Moodle that we can't remove outdated plugins, can't install new ones and basically daren't touch anything!

As we only received access to the server yesterday, my team are now desparately trying to fix this, so my questions are:

1) Any suggestions what might be happening?

2) Are there any logs I can check on the web server, SQL server or in Moodle which might give me ideas as to what is happening?

3) Is it possible to do a fresh install of Moodle and restore the old data into it?

This afternoon I may try and install a fresh Moodle installation on our dev server and either try and import the data from the current database into a new DB, or to point it at the current DB where it is... I don't know if it will work, but it's worth a try at this point!


Any suggestions please - any help at all would be HUGELY appreciated!

Thanks in advance,

Elaine

Attachment Moodle issue.png
Average of ratings: Useful (1)
In reply to Elaine Topham

Re: Broken Moodle - please help!

by Gareth J Barnard -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers

Quick glance.  The double squared brackets are a missing language strings issue.  Check that your '/lang/en' folder is completely populated with all of the files from the standard Moodle download then perform a 'Purge all caches' (Purge_all_cache).

Average of ratings: Useful (2)
In reply to Gareth J Barnard

Re: Broken Moodle - please help!

by Elaine Topham -

Hi Gareth,

Thanks for your reply!

Do you mean the lang folder in the Moodle files or the Moodledata folder?  I'm assuming you meant the Moodle files, but I've checked and these are the same as the files in the ZIP file for the Moodle 3.0.2 download.

Haven't tried "purge_all_caches" yet... will try this in the morning and report back smile

Average of ratings: -
In reply to Elaine Topham

Re: Broken Moodle - please help!

by Gareth J Barnard -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers

The lang folder in the Moodle files area not the data folder.

Also look at your web server logs and hunt around for the php.log file too.  I run Apache (on Windows) so cannot comment on IIS.

Average of ratings: Useful (1)
In reply to Gareth J Barnard

Re: Broken Moodle - please help!

by Elaine Topham -

Thanks, that's the folder I thought you meant, but I just wanted to check.

Found the PHP log this afternoon, need to have a better look through it tomorrow because there seems to be a lot in there, but I've not yet looked in IIS, so will have a dig for those tomorrow smile

Average of ratings: -
In reply to Gareth J Barnard

Re: Broken Moodle - please help!

by Elaine Topham -

Thank you for your suggestions and support Gareth - it was indeed a problem with the moodle.php file in the lang folder ( see my other post somewhere on this thread!)


Thanks for your time, I really appreciate it!

Average of ratings: -
In reply to Elaine Topham

Re: Broken Moodle - please help!

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

What a well asked question! To echo Gareth it is almost certainly something to do with your language string files. Let us know how you progress.

Average of ratings: Useful (1)
In reply to Marcus Green

Re: Broken Moodle - please help!

by Elaine Topham -

Thanks Marcus... though I'm new to the Moodle "backend" and forum, I have some experience with computers and know how important details are smile


I'll let you know tomorrow on my progress with purging the caches, but having checked the Lang folder all files appear to be the same... though I haven't looked into the code in these files to see if anything's been changed in there yet....

Average of ratings: -
In reply to Elaine Topham

Re: Broken Moodle - please help!

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

You might check permissions on the new moodle installation.  That could be half your battle.  

Average of ratings: -
In reply to Emma Richardson

Re: Broken Moodle - please help!

by Elaine Topham -

Hi Emma, we haven't actually done a fresh install yet, we have restored from a backup and tried an update to Moodle 3.0.2.  Thanks for the advice though, it may come in handy as I think we are trying a fresh install option tomorrow, so I'll bear that in mind when installing it smile

Average of ratings: -
In reply to Elaine Topham

Re: Broken Moodle - please help!

by Usman Asar -
Picture of Plugin developers Picture of Testers

Elaine,

You have to ask your IT guys for few more things, how is your cluster formed? where you mentioned 3 VM's, I am assuming it would be 2 web servers and 1 database server. Now comes additional information of how web servers are setup, as your web server doesn't seem to have any issues, nor your PHP application server.

What I am assuming considering your case is, your database got corrupted,  assuming your server was setup using Application Request Routing (ARR), then setting up ARR requires few folders to be shared across all load balancing nodes, when comes the time of upgrades/installations, then ideally one node is to be worked upon while shutting down web servers on others, and once replication occurs then to start other web servers on other nodes with newly replicated data. This may have not been done at your end, so what could have happened is, once you have touched plug-ins on one instance where other nodes were still running, database must have updated the changes, whilst replication may not have occurred and database was again updated with 2nd node where you mentioned updating plug-in sent your server in a loop, now this loop happened where one node is showing new plug-in folder present, while other node telling database it's not.

Now you can try, importing database again and doing a fresh install, but assumption is, database got corrupted to the point that it MAY need to be recreated, also if you are doing so, make sure you're working on one web server node only, also can you ask your IT guys, how is cache's shared across? as moodle is bit different than other scripts so cache folders in case of multiple nodes are to be shared across all.


Average of ratings: Useful (1)
In reply to Usman Asar

Re: Broken Moodle - please help!

by Elaine Topham -

Hi Usman,

Thank you for your input - I will attempt to find the answers to these questions tomorrow, though this may be difficult for reasons I won't go into here.

As far as I know, we have 3 web server clusters with a separate database server - the database is definitely on a different server.  As for whether it is setup using ARR, I don't know but will try to find out, though how you describe the situation sounds terribly familiar.  We are working on one web server at present.

I came to the same resolution this afternoon, I think our only option is to do a fresh install and try and restore the data back to it... and hope the database isn't corrupted.  If so, my next job will be to work out how to recreate the database as you suggest smile

Average of ratings: -
In reply to Elaine Topham

Re: Broken Moodle - please help!

by Usman Asar -
Picture of Plugin developers Picture of Testers
Fingers crossed your DB comes back good, but you now know the drill, it will be just IIS server that you've to shut down on other nodes while upgrading, whilst windows server remains alive for replication. (less than a minute it takes).
Once you're done with it, do ask your IT Guys, if it was combination of NLB (Network Load Balance) & ARR, or just ARR implemented, or may be Level 5/7 Hardware Load balancer <--- this for my books wink
Average of ratings: -
In reply to Usman Asar

Re: Broken Moodle - please help!

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Hi Elaine

What is the status of your system this morning? Is it possible to bring up a single web server and the database as I was wondering if there is an issue with having different files on the different servers?

Average of ratings: -
In reply to Marcus Green

Re: Broken Moodle - please help!

by Elaine Topham -

Hi Marcus,

Status of our Moodle this morning... we are only running on one web server connecting to the same database (this has been the case since Monday).  I don't have access to the other clusters, I believe they are broken.

I've tried Purge Caches and my moodle is now broken again, which is not good... currently updating our dev server Moodle to 3.0.2 in readiness for attempting a fresh install and then attempting to get the data imported into the new database...

Though... I've just checked the files in the lang folder again.. now there is a missing file called moodle.php... have copied this over and it's fixed the broken links... now checking plugins again smile

Average of ratings: -
In reply to Elaine Topham

Re: Broken Moodle - please help!

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Is your Moodle currently usable by students? What plugins are you using.

Update: I just realised you are running 2.9, plugins mostly work on a move from 2.9 to 3.0 and where updates are required they are fairly trivial.

Average of ratings: Useful (1)
In reply to Marcus Green

Re: Broken Moodle - please help!

by Elaine Topham -

Hi Marcus,


No students can access it at present (since last week), we have the site down in maintenance mode.  We have a lot of plugins installed, which isn't helping matters I know...


UPDATE: I don't want to jinx things, but things are improving we think... have followed Gareth's suggestion of purging the caches (which really broke things again) and I have now replaced the missing file in the lang folder (which seems to have fixed things!).  Moodle links are all back and I am now able to remove/install/update plugins... I think we are progressing along the right path now - I'll keep you updated and let you all know how things turn out and SOOO much thanks for all the suggestions and help so far!

Average of ratings: -
In reply to Marcus Green

Re: Broken Moodle - please help!

by Elaine Topham -

We have cracked it I think Marcus!!  Everything seems to be working ok at last!  I have now removed all outdated plugins, plugins that said the files were missing from disk and I have also now managed to update and to install new plugins!!  The only plugins we haven't put back yet are the Planet eStream plugins, as the Moodle Plugin directory shows that they aren't available for Moodle 3.0 yet, so going to contact eStream regarding this and look into this problem another day!  For now, we have our VLE back smile currently working on a full backup..


It was the Purge All Caches smile  obviously, when I checked the Lang folder yesterday the files all matched, but clearly the moodle.php file had been deleted somehow and was "resident" in the cache (I think anyway!).  When I cleared the caches this morning, it seriously broke Moodle (and the cached file disappeared), but we've replaced the missing file into the lang folder and everything seems to be working ok now!

Thanks for your help and suggestions!

Average of ratings: -
In reply to Elaine Topham

Re: Broken Moodle - please help!

by Luis de Vasconcelos -

In your initial post you said that, "we updated Moodle to 3.0.2, in the hopes that this may fix the problem... it did for a while, though many of the plugins were giving errors - e.g. Files not on disk."

And then you said: "we have located unusual spaces in the filenames for REALLY key files, like index.php and view.php... so they were 'index. php' and 'view. php'."

And then two of your clusters went down!

Then on Thursday "Moodle went offline"!

What a week!!!

Now that your site is back up again I would be asking some serious questions: "How did this happen? What went wrong...?"

Have you worked out what caused the above "unusual spaces in the filenames" problem? The answer to that would be really interesting!

Average of ratings: -
In reply to Luis de Vasconcelos

Re: Broken Moodle - please help!

by Elaine Topham -

Hi Luis,


Yes, it's been quite a mad week!

I would also like to know how the spaces in the filenames occured, and how Moodle.php disappeared!!  We literally don't know... If I find answers/solutions/ideas about how it happened I will post them here.

Average of ratings: -
In reply to Elaine Topham

Re: Broken Moodle - please help!

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Spaces appearing in file names is wierd. Is there some kind of synchronisation going on between a master and other servers and if so what technology does it use.

Average of ratings: -
In reply to Marcus Green

Re: Broken Moodle - please help!

by Elaine Topham -

Yes, as far as I know there is some replication between virtual servers, though I'm not privvy to how this is setup... the server administration is dealt with by our CSU team.

Average of ratings: -
In reply to Usman Asar

Re: Broken Moodle - please help!

by Elaine Topham -

Hi Usman,

We've fixed it, see my other post on this thread somewhere!

We didn't actually need to reinstall, but I will try and find out the answers to your questions (for your books) when our IT guys come back... I do know there is a Kemp load balancer, if that means anything to you?

Average of ratings: -
In reply to Elaine Topham

Re: Broken Moodle - please help!

by Usman Asar -
Picture of Plugin developers Picture of Testers

Hello Elaine,

Glad to know things worked out for you pretty well, and thank you for taking time replying everyone who assisted, and yes KEMP does sound familiar to me as providers of Hardware, Virtual, Cloud Based load balancers, you don't have to ask your IT guys any more question for this is  all I needed to know if you're making use of built in technologies that Windows server platform provides or using third party.

For future, do shut down other nodes while updating from one node, once your shared data is replicated over, then safely turn on other nodes, to avoid any more incidents wink

Average of ratings: Useful (1)
In reply to Usman Asar

Re: Broken Moodle - please help!

by Elaine Topham -

Haha I will Definitely bear this in mind in future! FYI ... something I found out today was that the Cluster that we were working on the other week, wasn't actually the Primary cluster... methinks that might have been where all the problems started smile ...

Average of ratings: -
In reply to Elaine Topham

Re: Broken Moodle - please help!

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
I would immediately restore the back-up and experiment the upgrade in a copy. See https://docs.moodle.org/en/Moodle_migration.
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Broken Moodle - please help!

by Elaine Topham -

Hi Visvanath,

Thanks for your suggestion, I think we have cracked it wink

Average of ratings: -