Moodle site corrupted

Moodle site corrupted

by Doug Moody -
Number of replies: 16

I am using Moodle 3.1. Everything has been running perfectly for over a year. I didn't even log in during the weekend. I have classes right now that can't use Moodle because when I tried to login this morning (beginning of the week, Monday), I was presented with a simple black and white moodle login page, and after putting in my credentials, I got something similar to the following:

"enamqePu59cU8cnJAP9K3YhwOR2zSEAdOGPelPHpm kB7AUhod25puOCKUH160ZzxQADIHrQCOcUADHI6Ucdv1pAO4HakPNId2eMY70nI5yaB3Hd6X3ApO 2TxR1PtQAo5GaOmcnrR70dKBgPajoOKTr2ozz0oEGKUdKP5UfjQAUUc9qKQwHtQPrmgDAxmgDHSm IPaiiikMAAOKKT8MUtMVxaTPpRzn2oPPFAwBzRQQOuMmkOGPrQK4uaO9IABzigUBcXjPvRmkAA6U Yz1GKAFoo+lGPWkFxaQj3ox/k0YoGGQKWkIGc0CgBaSiigBaT6/lRjNGKAFopKKAFoopKAFopMZ6 0EZoAWik+lGPzoAWkoP1ooAWjp1pDRigAzg9evQUA5ox/wDqo4PfpTAWk4oIoCgDFIAz70ZFN8tf SlK+9MWovejnNJtGQQOR3o6n3oC4tAz7mk79KPegBegzmlpMUEUhiiikI9aMCgLsBz3zS0mOeKMU AFLSAck5o4z60AFFFIVz9aYDqTvRj3o70gFopKMGgBaKSigBaO9JjnNGKAFopCARg0AAcCgBaTNG BRQAd6DRj0oxQAtFJj60AfWgLi0nvRgEg0YP1pgGaMZ+lJgE+9AHNAhaOxpMfWj8OaAF+tGaQrx3 oAPds0BqLQPrRtpNvzZyfpQGotGaTA6etGB+VAXHZopuzI64pegpDQvFH6UUUAFFJz2FLQAUUUlA C0UlFAC0UUUDCiiigAooooAKKKKADNFJ9aWgQUUUGgYUUUUAFFJS0CEoFFLQAlLSUtACYo74zR1o "

There were pages and pages of this. I WAS able to finally get the main pages (in black and white plain text), but when trying to navigate beyond the menus, I again got the garbage text seen above.

I also tried running cron from the command line, and Got the message "Moodle upgrade pending. CRON execution suspended"

Any ideas what is going on? Ideas how to restore things to normal? My students are languishing....

Moodle upgrade pending, cron execution suspended.
Moodle upgrade pending, cron execution suspended.
Moodle upgrade pending, cron execution suspended.
Moodle upgrade pending, cron execution suspended.

(Edited by Howard Miller to make subject more sensible - original submission Monday, 8 January 2018, 2:15 PM)

Average of ratings: -
In reply to Doug Moody

Re: Moodle site corrupted

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I suspect you may have been hacked. 

Check the first line of some likely php files (config.php, index.php and so on). Do they start with <?php followed by masses of characters like above?

Average of ratings: -
In reply to Doug Moody

Re: Moodle site corrupted

by Ken Task -
Picture of Particularly helpful Moodlers

+1 to what Howard said ... with addition that the injected code may not be on the first line of a php file.

Got Linux?   Install ClamAV.   ClamAV won't 'clean' php files (advise you not try that), but it can ID some (not all) PHP 'viruses' ... like Trojans.

cd /path/to/moodlecode/

Then issue:

clamscan -ri

See: https://linux.die.net/man/1/clamscan

IF that ID's any files inspect them via command line.

Hope you have a recent (prior to the holidays) backup of code.   BTW, as a former K12 person, made it a habit to always do full site backups just prior to holidays.

IF those backups are tar balls, one can extract .php files from the backups and replace the 'infected' files.

That, would be a quick fix to get and and running, but may not/won't solve the issue.   Still need to investigate what/how that happened an address it.

Cleaning up a hacked site is no fun and does take some time.

Also hope you have a recent backup of DB as one might need to restore the entire site to a point in time when site wasn't experiencing the current issue - which means some work might be lost.

'spirit of sharing', Ken



Average of ratings: -
In reply to Ken Task

Re: Moodle site corrupted

by Doug Moody -

I do automatic backups, and I do have an automated one. I inspected the php files that are common, and there are no files anywhere I can see that have a telltale date stamp that is new.

Ken, I don't have Linux, so can't use that. Any other ideas?

Average of ratings: -
In reply to Doug Moody

Re: Moodle site corrupted

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers

Doug, remind me.  How do you host your moodle?  If you are using a hosting company, have you spoken to them?  Who you your system admin?  If you don't have Linux, does this mean you have Windows?  How do you check the health of your server?   Have you rebooted your server?

Average of ratings: -
In reply to Rick Jerz

Re: Moodle site corrupted

by Doug Moody -

I'm now getting a message from the server that apparently there is a disk space issue.


Average of ratings: -
In reply to Doug Moody

Re: Moodle site corrupted

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers

Okay.  You can clean (delete) old files trying to make more space, or you can add more disk space.  For example, if you have old moodle backups, you might consider deleting a few of these to free up some space, especially if your moodle database and moodledata folders are large.  Temporarily, you might try clearing Moodle's cache.

Average of ratings: -
In reply to Doug Moody

Re: Moodle site corrupted

by Ken Task -
Picture of Particularly helpful Moodlers

Time stamp may not change so that's not really the ONLY thing to look for.   A really good hack would take that into account and 'cover it's tracks' so to speak.

Whatever antivirus you have or can install on the server use that.  IF I re-call correctly from previous talks with you the chosen platform was decided in hopes of getting internal IT departments assistance.   What do they recommend?

While your server may not have the specific issue mentioned in the URL, server might have something similar:

https://www.microsoft.com/en-us/wdsi/threats/malware-encyclopedia-description?Name=Backdoor:PHP/WebShell.A

Again ... ask your internal IT department for their recommendations as to AV software for Windows.

The basics of hacked site recovery are not really that much different in Windows - although the end result of investigation might lead to the only way one can be absolutely certain server has nothing malicious on it ... that of complete re-install of OS + apps + code + DB restore, etc..

Howard is correct that more than likely code has been injected into key files for Moodle ... like index.php or config.php or other.

One should be able to open those files with NotePad.   Many of these things try to hide their presense by adding their code wwwwwwaaaaaayyyyy out to the right of what looks to be a 'normal line' ....

So using your arrow keys, in NotePad one should be able to check if the end of a line has a carriage return where one would expect to find it .... at the end of the text you see.

The first line would have <?php

IF you put the cursor at the end of the last 'p' in that line then hit right arrow does the cursor continue out to the right, or does is wrap and go to the beginning of the second line? 

IF it doesn't wrap ... go to the end of that line to see the tail end of that line .... might see a closing tag for a script or data.   That file is suspect and probably contains injected code.

There are other file types in Moodle code and a bunch of files to scan/inspect.

A Moodle 3.4 has approximately 728130 total files.

Are you running a version of Moodle that is capalble of search?   If so, it means you have apache solr and you also have JavaRuntime SDK server running.   Has that been updated to highest/most secure version?

Like I said ... nothing easy about this situation and if one doesn't have the time to research/investigate - nor the technical assistance ... might have no choice but to wipe it all out, install fresh OS and apps, then restore site backups after they have been scanned , etc..

Wishing you good luck! smile

'spirit of sharing', Ken


Average of ratings: -
In reply to Ken Task

Re: Moodle site corrupted

by Doug Moody -

And NOW, when I attempt to login to Moodle, it is going directly to the plugins upgrade page. I can't get out of that loop. When I try to update the database, it gives an error message about some dependencies missing. I can't resolve those dependencies without getting to the site.

Ideas on how I can get to the home page manually?

Average of ratings: -
In reply to Doug Moody

Re: Moodle site corrupted

by Floyd Saner -

Doug, 

Try

<your moodle url>/my

or

<your moodle url>/course


Floyd

Average of ratings: -
In reply to Doug Moody

Re: Moodle site corrupted

by Ken Task -
Picture of Particularly helpful Moodlers

Wow!  This is getting to be stranger and stranger ... from corrupted files to out of space to infinite loop on a pending update/upgrade.  So you must be doing something to fix/address ... moving forward?

So did you resolve the out of space issue?

Automated backups can create some interesting issues ... do you re-call the settings for that?  Keep all? and are you saving those automated backups to moodledata/filedir (the default) or to a designated backup area ... like C:\backups?

That, relies upon cron running correctly so may as well throw that into the investigation as well.

Infinite loop ... how many extra plugins do you have and what are they?

IF the drive where the DB server/files reside ran out of space, you could have DB corruption as well.

????????

'spirit of sharing', Ken

Average of ratings: -
In reply to Doug Moody

Re: Moodle site corrupted

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers

From what I know, the only time that my moodle wants to update the database is when I update moodle, update a plugin, or add a plugin.  So someone might have done something to your moodle without you knowing (and maybe didn't do it correctly.)  I am just trying to provide some ideas for you.  I am not the best server admin.

Average of ratings: -
In reply to Ken Task

Re: Moodle site corrupted

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I doubt this is a virus as such. If it's what I think it might be it's very common if you're not careful with Moodle permissions (or any other PHP files). This hack injects encrypted code at the top of PHP files which then look for other PHP files throughout the server that are writeable by the web server and infects those. 

You have to be quite diligent about removing it. One important step is to replace ALL the Moodle files with new ones from a known reliable source. Your code backups are NOT a reliable source!! You should get new code (and plugins if needed) from moodle.org. The actual Moodle data and database is unlikely to be affected so don't worry about that. But do clear ALL the directories under 'moodledata' except /filedir (as some contain code files)!

Recreate config.php from (the new) config-dist.php copying over details manually. Do not reuse the old config.php. It only needs one infected file. 

THEN... set your permissions such that the web server cannot write to the Moodle code files. 

Essentially, the service that allows you to install and upgrade from the Moodle interface is, IMO, a big security hole. 

Average of ratings: -
In reply to Howard Miller

Re: Moodle site corrupted

by Doug Moody -

Thank you all for your willingness to help....

Long story short, I upgraded to 3.4 and things are running again.

I will try to safeguard against future problems like this by following your advice. When this is all said and done, it will have wasted 3 days of my life, but the good news is that I now have 3.4 installed. I like what I see so far!

I shall never find out for sure what really happened, but again, thank you all!

Average of ratings: -
In reply to Doug Moody

Re: Moodle site corrupted

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers

Doug, you didn't waste three days, you just got 30 hours of education (3x10).

I just upgrade to 3.4 a couple days ago.  Actually, I got a complete new VPS running, with php7, MariaDB, and 3.4. I too like 3.4.  My server is also much faster.

When I put this perspective.php file into my moodle folder, and run it from my browser, I get the following results, which are 2-10 times better than my old server and php5.x:

Speed test

In one second you can do...

  • 22708000 function calls
  • 186000 16KB files read from disk (cache)
  • 38100 regular expression replaces over 1KB of text
  • 9000 16KB files written to disk (cache)
  • 19040 get_record calls on the course table
  • 7790 insert_record calls on the course table
  • 7360 update_record calls on the course table
  • 1.01 seconds to delete 7790 entries

Script took 29.26 seconds to execute.



Average of ratings: -
In reply to Doug Moody

Re: Moodle site corrupted

by Ken Task -
Picture of Particularly helpful Moodlers

All is well that ends well ... or so I've heard. 

Not knowing, however, would make me nervous! :\  I do hope that, come Spring Break, there is not a similar event upon returning from Spring Break.

As a former K12 wide area private network admin (multiple ISD's), a suggestion ... the Friday/day the students are dismissed for a 'holiday' where students will be away for an extended time (not just a weekend or three day), do a full site backup for sure.

I've never regretted taking the time it took to do that.

'spirit of sharing', Ken


Average of ratings: Useful (1)
In reply to Howard Miller

Re: Moodle site corrupted

by Doug Moody -

Hi Howard,

A quick question about your post above...

When you said that I should set permissions so that the server can't write to the moodle code files, which directories in particular are you referring to? 

Right now, my permissions are set to 0755. Which permissions do you recommend, and on what directories? I was thinking 644???

Average of ratings: -