404 Error on /admin/index.php (and an ErrorDocument issue)

404 Error on /admin/index.php (and an ErrorDocument issue)

由John Doe發表於
Number of replies: 20

Hey everyone,

I ran into a very strange issue (the first time I run into it, even though I've installed and maintained a lot of moodle installations before).

Error:

The requested URL /admin/index.php was not found on this server.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

What I did:

  • I confirmed the file exists
  • I triple checked the permissions and owners of the file and the entire moodle directory. All good.
  • I replaced the file with an original one from the moodle repo
  • Tried upgrading by pulling the latest weekly release (4.4.3 to 4.4.5). Didn't work so I reverted back
  • Cleared the cache (CLI)
  • Tried to perform the upgrade via the CLI
  • Eventually I undid everything above by just restoring a week old backup... turns out the issue still exists. (note that I did no admin tasks the past 3 weeks at least on the platform)


To reproduce the issue:

  • I log in as an admin
  • I get redirected to this page: https://ibb.co/mgjg1M1
    • The URL in this page is: /admin/index.php , (which is weird because in the next step it says the file is not found)
  • I click continue, then get the error mentioned above: https://ibb.co/5X6V6rz9
    • The strange thing about this page is the password field. I've never encountered something like this before. I'm not sure what it is and how it got there.

Any help troubleshooting this issue is highly appreciated. And please let me know if I missed something or if more info is required.

Using moodle 4.3.3. 

評比平均分數： -
In reply to John Doe

Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由Ken Task發表於
Particularly helpful Moodlers的相片

The environment check screen shot you shared clipped off the bottom.   Are there any yellows at the bottom?

In config.php of site check to see if there is a line:

$CFG->admin     = 'admin';

It's there to assist Moodle in-case one has a wordpress on top of the moodle code.

And, turn on debugging ... all the way to developer .... to see if Moodle can report more details.

Password fields have 'extra protections' ... javascript related, I think.  Can you look at server error logs?   Check those out as well.

'SoS', Ken

評比平均分數： -
In reply to Ken Task

Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由John Doe發表於
Hey Ken,

1) The environment screenshot is clipped on purpose for ""security"" reasons. In any case there's only 1 warning regarding async backups. I don't think it's relevant
2) The line $CFG->admin = 'admin'; exists in the config. Note that I'm not using wordpress.
3) Debug mode was already on when I captured that screenshot and I did not see any errors being reported.
4) I just checked the logs. The error log is not reflecting anything weirdly enough (it doesnt update when I reproduce the issue while Im tailing it). The access log is showing no errors, only 200 OK requests which is even weirder.

Access log:

[01/Feb/2025:15:10:24 +0000] "GET / HTTP/1.1" 303 1997 "REDACTED_URL/login/index.php?loginredirect=1" "Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0"
REDACTED_IP - - [01/Feb/2025:15:10:25 +0000] "GET /admin/index.php HTTP/1.1" 200 6048 "REDACTED_URL/login/index.php?loginredirect=1" "Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0"
REDACTED_IP - - [01/Feb/2025:15:10:34 +0000] "GET /admin/index.php?cache=0&confirmrelease=1&confirmplugincheck=0&confirmupgrade=1 HTTP/1.1" 200 1324 "REDACTED_URL/admin/index.php" "Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0"

5) I ran a curl command to try to open the file in question and it worked fine:

curl -I https://URL/admin/index.php
HTTP/1.1 200 OK
Date: Sat, 01 Feb 2025 15:14:54 GMT
Server: Apache/2.4.52 (Ubuntu)
X-Accel-Buffering: no
Set-Cookie: MoodleSession=p1hrjavupaae8dngnpdq2trbgf; path=/; secure
Expires: Mon, 20 Aug 1969 09:23:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Content-Language: en
Content-Script-Type: text/javascript
Content-Style-Type: text/css
X-UA-Compatible: IE=edge
Cache-Control: post-check=0, pre-check=0, no-transform
Last-Modified: Sat, 01 Feb 2025 15:14:55 GMT
Accept-Ranges: none
X-Frame-Options: sameorigin
Content-Type: text/html; charset=utf-8

6) I added $CFG->theme = 'boost'; but it also made no difference.

Any clue how to proceed?
評比平均分數： -
In reply to John Doe

Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由John Doe發表於
Update:

I was able to get it to work again. Long story short, my moodle site got compromised in some way. Very similar to this thread: https://moodle.org/mod/forum/discuss.php?d=459873 (if not the exact same).
I deleted some files and folders that should not exist in the core moodle code and it just simply worked.

Some folders and files that I deleted:
/moodle/more
/mod/lanciau
/theme/lanciau

I also deleted the webshell and the moderated rss plugins. Not sure about the latter but the former is definitely what the attacker used to inject some code into my index.php.

Not sure which one of them was the direct culprit. I kep a copy of the branch aside for further investigation because unfortunately this isn't the first time something like this happens.
評比平均分數： -
In reply to John Doe

Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由Ken Task發表於
Particularly helpful Moodlers的相片

So we suspect the site has been compromised somehow and .php code might have injections in them or there are .php files that look like they belong but don't?

Get fresh code via git into a directory at same level as document root.   Example: document root is /var/www/html/ and moodle code is in /html/

Git command from /var/www/ location:

git clone git://git.moodle.org/moodle.git githtml

cd githtml

Then issue the following git commands changing the numbers to match the version you had ... a 4.3.   Is 403 in git lingo.

git branch --track MOODLE_403_STABLE origin/MOODLE_403_STABLE
git checkout MOODLE_403_STABLE

Check that you have the version needed:

fgrep '$release' version.php

Once you've acquire core code and set the version, copy ... that's copy ... plugins from html/x into same location in githtml/

Check ownerships/permissions.

Then move /var/www/html to /var/www/htmlhacked

mv githtml to html

cd into /var/www/html/admin/cli/ - the newly acquired via git and issue:

php upgrade.php

That might result in a minor point release of that version.

If that has no errors, then visit site via GUI!

Fingers crossed!

'SoS', Ken

評比平均分數：Useful (1)
In reply to John Doe

Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由Ken Task發表於
Particularly helpful Moodlers的相片

Glad to hear.

Not the first!!!  That begs deeper inspection and questions as to hosting.  If hosted with a provider, wonder if there aren't 'malicious neighbors' ... which are possible! 傷心

Since you kept an archive of old code directory ... get a listing of all files/folders > a txt file.   Do same with fixed new code > a txt file by another name in same location as the other txt file.   Then diff them.

Also inspect the header of that index.php file ... actually all .php files.   Think clamscan would whoopie that those.

'SoS', Ken

評比平均分數：Useful (1)
In reply to Ken Task

Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由Ken Task發表於
Particularly helpful Moodlers的相片

And one more thought and reason to use git for core code ....

if any file in core code has been altered or any file that doesn't belong is present, a git pull would 'whoopie' and complain that it can't update/upgrade because of files that it would list.

Don't stash them via git but cancel the update/upgrade and do some sluething on what git reported.

'SoS', Ken

評比平均分數：Useful (1)
In reply to Ken Task

Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由John Doe發表於
Hey Ken, you're absolutely right about using git for managing moodle installations.

I did something similar to your other comment - basically I staged the changes that I want to keep (i.e plugins and other files), I stashed them, then I ran:
git reset --hard origin/MOODLE_403_STABLE and then finally popped the stash. This way I get to keep the files I want while making sure everything else is the same as the remote branch.

I'm still getting the hang of git (took a while to understand that I should leave the plugins and custom files as UNTRACKED and just move them with me to other branches if needed). Let me know if there's something i missed in my process.
評比平均分數：Useful (1)
In reply to John Doe

Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由John Doe發表於
EDIT

git reset --hard origin/MOODLE_403_STABLE only resets TRACKED files.
It does not remove untracked files or directories (in other words, the infected files remain)

So I had to run git clean with --exclude to remove the unwanted files and exclude the files i want.
評比平均分數：Useful (1)
In reply to John Doe

Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由Ken Task發表於
Particularly helpful Moodlers的相片

At code root of git acquired code there is a .gitignore file.

Comments in header:

# This file specifies intentionally untracked files that all Moodle git
# repositories should ignore. It is recommended not to modify this file in your
# local clone. Instead, use .git/info/exclude and add new records there as
# needed.
#
# Example: if you deploy a contributed plugin mod/foobar into your site, put
# the following line into .git/info/exclude file in your Moodle clone:
# /mod/foobar/
# 
# See gitignore(5) man page for more details

Thus no need to stash ... keeping core nothing but core.

Also glad to hear the sos page useful. 微笑

'SoS', Ken

評比平均分數：Useful (1)
In reply to John Doe

Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由Ken Task發表於
Particularly helpful Moodlers的相片

Now that you've been successful in cleaning up code, 2 more suggestions as the bad actors will be back and more than likely trying the same things that were successful.

Install logwatch - perl script - set to 10 (max reporting) and have the summary go to root user email .. never leaves server.

logwatch will report bad actors - at least the what ... but not the IP.  Also create a 'slogs' script to take one parameter ... that of the attempted exploit.   Using slogs one can see IP address.

Suggest blocking those IP addresses or range of IP addresses at the network layer using server firewall.

Do be careful not to block yourself! 微笑

'SoS', Ken

評比平均分數：Useful (1)
In reply to Ken Task

Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由Ken Task發表於
Particularly helpful Moodlers的相片

A 3rd suggestion ...

Install multitail
https://www.vanheusden.com/multitail/
allows watching server logs in realtime.
As a bad actor comes back, note the IP address and block at network layer.

Might monitor with multitail for some time ... now that you having to do this, it's a constant/regular activity by you as bad actors will get more 'stealthy' etc.

'SoS', Ken

評比平均分數：Useful (1)
In reply to John Doe

Er: Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由Visvanath Ratnaweera發表於
Particularly helpful Moodlers的相片 Translators的相片
I am still not sure why you needed those Git somersaults: You had the core code changed and want to keep those changes? The additional plug-ins don't count to that category, they (usually) not tracked by the main Git repo (the .git/ under Moodle core).

And one more alarm bell: the lanciau thing. See Unexpected and strange update page after login.
評比平均分數： -
In reply to Visvanath Ratnaweera

Re: Er: Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由John Doe發表於
Basically these files were not under .gitignore or the exclude file (which is understandably my fault). So I had to stash them while I dealt with the rest of the changes.
I'm always up for a simpler approach if there is one though.
評比平均分數： -
In reply to John Doe

Re: Er: Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由Visvanath Ratnaweera發表於
Particularly helpful Moodlers的相片 Translators的相片
So you have not changed the core code. You added (new) files instead (not counting the files the additional plug-ins bring).

About a simpler approach: Well, this is about how Git (versioning) works. Not specific to Moodle.
評比平均分數： -
In reply to John Doe

Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由Visvanath Ratnaweera發表於
Particularly helpful Moodlers的相片 Translators的相片

Very similar to this thread: https://moodle.org/mod/forum/discuss.php?d=459873 (if not the exact same).

Means, 

  • either the 'admin' user's password or the username and the password of a different user with Site administrator privileges has been found by the intruder - probably on Have I Been Pawned or in one of the username/password lists making rounds in the web - and logged in to Moodle
  • the plug-in installation was enabled by the system administrator by giving the web server write privileges in places inside the Moodle code directory.

Those two are enough for a full-scale attack. The intruder installs https://moodle.org/plugins/local_webshell - which allows running Shell comands in the server through the Moodle GUI! See it in action at https://www.youtube.com/ watch?v=D09TIj5ZHyg&t=650s (delete the blank).

That is a total meltdown. (I know why I don't allow GUI installation of plug-ins on production servers.) I don't trust deleting certain files in the code tree will heal the site.

評比平均分數：Useful (1)
In reply to Visvanath Ratnaweera

Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由John Doe發表於
Hey Visvanath, good to hear from you as usual!

Yeah it's definitely either of those if not both. I will start forcing 2FA for all admins and definitely disabling GUI installations of plugins on production. Thanks for that.
評比平均分數： -
In reply to John Doe

Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由Visvanath Ratnaweera發表於
Particularly helpful Moodlers的相片 Translators的相片
Well, you have destroyed the evidence. Otherwise from the date stamps of those unwanted plug-ins and the people with site admin privilege who were logged in at that time, you can find out whose credentials were compromised. Moodle even keeps a log of site configuration changes. I assume, installing new plug-ins are logged there too.
評比平均分數： -
In reply to Visvanath Ratnaweera

Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由John Doe發表於
I took a backup of the entire VPS before I started. I was able to restore it and check the logs and I found out the user and date in question.
評比平均分數：Useful (1)
In reply to John Doe

Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由Ken Task發表於
Particularly helpful Moodlers的相片

When you restored site, did you do a complete site restore ... new DB, then import the .sql backup into new DB.   Change DB name in config.php?

Could be the mdl_config_plugins table has been touched (update/upgraded) and you might have references to in-compatible plugins in there.

Do you have any WAF? (web application firewall) or is site behind CloudFlare?
Looks like you are getting redirected to something related to config of site.

Since you can access your server via Terminal/ssh,
there are other scripts in code/admin/cli/ to run.
checks.php and check_database_schema.php
I'd also run php uninstall_plugins.php --show-contrib
That will show addon plugins.
Do any of those not have compat versions for destination?

Sent you a PM with a link to a zip file

Unzip in code root.   That will create an sos.html static web page which you can edit.  Has a reminder and direct links to the following admin areas:

Don't forget to put this page into the same location as the 
code for your Moodle!   Login to site to authenticate, then in a new tab go directly to the sos.html page via browser url line.

Search
Environment
Theme Settings
Theme Selector
Notifications
Plugins Overview
Additional Plugins
Manage Blocks
Manage Activities
Maintenance Mode
Debugging
Purge All Caches
Users
PHP Info
Server
Manage Auth
Manage Activities

'SoS', Ken

評比平均分數：Useful (2)
In reply to Ken Task

Re: 404 Error on /admin/index.php (and an ErrorDocument issue)

由John Doe發表於
Super useful when/if I get locked out again! I received the file and saved it. Thanks Ken 微笑
評比平均分數： -