Error upgrading from 3.8 to 3.10.3 latest release

Error upgrading from 3.8 to 3.10.3 latest release

by Perry Way -
Number of replies: 29

I'm attempting to prove the viability of upgrading a copy of our current Moodle site which is running on 3.8 to the latest release of 3.10.3. We are running Moodle on Windows IIS.

After going through the process of logging in, the server responds with an error redirecting "/login/index.php" to "/login/<a> </a>" with the verbiage "A potentially dangerous Request.Path value was detected from the client (<)". I've never seen this behavior in any previous versions of Moodle, whether installed freshly or through an upgrade process. This is rather puzzling to me. Does anyone know anything about this sort of behavior? I've attempted to do a search for this on this site but have gotten no examples before me.

What appears in the browser is a standard IIS form of reporting errors. But the data is not all in that page. So I looked for the corresponding error in the Windows Event Log and this is the entire error text:

Event code: 3005 Event message: An unhandled exception has occurred. Event time: 4/12/2021 1:06:40 PM Event time (UTC): 4/12/2021 8:06:40 PM Event ID: 9dc4a7b396db46998614c6dae155f001 Event sequence: 4 Event occurrence: 1 Event detail code: 0 Application information: Application domain: /LM/W3SVC/16/ROOT-3-132627315898376896 Trust level: Full Application Virtual Path: / Application Path: C:\Websites\moodle_testing\website\ Machine name: ADMIN3 Process information: Process ID: 8552 Process name: w3wp.exe Account name: IIS APPPOOL\moodle_testing Exception information: Exception type: HttpException Exception message: A potentially dangerous Request.Path value was detected from the client (<). at System.Web.HttpRequest.ValidateInputIfRequiredByConfig() at System.Web.HttpApplication.PipelineStepManager.ValidateHelper(HttpContext context) Request information: Request URL: https://{our domain}:443/login/<a></a> Request path: /login/<a></a> User host address: 97.93.27.15 User: Is authenticated: False Authentication Type: Thread account name: ADMIN3\Administrator Thread information: Thread ID: 7 Thread account name: ADMIN3\Administrator Is impersonating: False Stack trace: at System.Web.HttpRequest.ValidateInputIfRequiredByConfig() at System.Web.HttpApplication.PipelineStepManager.ValidateHelper(HttpContext context) Custom event details:  
 

Update, it appears that after logging in any path winds up being mangled with "<a> </a>" anchor tags being added to the end. So I am unable to use the site at all until I resolve this issue. Even if I close the browser and then open a new one and manually type in the URL, the path is appended with "<a> </a>". Very odd. 

Average of ratings: -
In reply to Perry Way

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Perry Way -
Poking around further, it appears I can get to some pages without this error. If I know the exact path, those pages where it's not index.php that would be used to respond to a path with a trailing / appear to be working at first. But there are some exceptions to this. I created a folder /zzzzz and put an index.php file in that path that simply had a call to phpinfo(); and that worked so it appears that if it's moodle code in that index.php file it responds with this reported error.

Additionally I have attempted to turn on debugging, by typing in the path /admin/settings.php?section=debugging directly for the page where you would set the debugging flag. And, I get a page with two lines of text reporting:

"Error reading from database
Error reading from database"

The same error message returns when I type in the path /theme/index.php to set the theme to the one I created for our site.

So some kind of database issue exists for those two pages. But what exactly? I do not know since debugging is not turned on. I know for a fact that the database is accessible and working fine from other paths that I've attempted, such as looking at my user account profile and navigating to some courses.

Very puzzling. If nobody responds I'll be taking a dive through the code tomorrow with the knowledge of the variation of effects based on location of paths. But I'm hoping that someone has experienced this sort of thing before and knows what the issue is to save me some time.

Thanks in advance!
In reply to Perry Way

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Perry Way -
Woops spoke too soon. I set the debugging in the config.php manually since I can't do it interactively with this site problem and interestingly enough it no longer reported the "Error reading from database" but responded with the redirection to the path minus the page and query parameters and the problem happened again. Clearly something is redirecting to an invalid form of path. Also of interest is I turned the debugging off again manually in config.php and the behavior did not go back to reporting the "Error reading from database" but remained in this whacky redirection error. Hmm...
In reply to Perry Way

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Perry Way -

I found where this behavior originates. The redirect function in /lib/weblib.php midway into the code there's a place where the URL being passed to the function is adjusted to transform encoded paths. Starting at line 2933, this bit of code here causes this behavior

    // Sanitise url - we can not rely on moodle_url or our URL cleaning
    // because they do not support all valid external URLs.
    $url = preg_replace('/[\x00-\x1F\x7F]/', '', $url);
    $url = str_replace('"', '%22', $url);
    $encodedurl = preg_replace("/\&(?![a-zA-Z0-9#]{1,8};)/", "&amp;", $url);
    $encodedurl = preg_replace('/^.*href="([^"]*)".*$/', "\\1", clean_text('<a href="'.$encodedurl.'" />', FORMAT_HTML));
    $url = str_replace('&amp;', '&', $encodedurl);

If I comment out the last line in this chuck of code (where $url gets replaced by the decoded url), then redirection happens without any problems and I am prompted to enter in my Upgrade Key so that I can perform the site upgrade.

This seems messy to me to comment out that last line. But I'm tempted to do just that so I can get to second base with the upgrading test.

Anyone know anything more? I can't believe I'm the only one experiencing this problem...


Average of ratings: Useful (1)
In reply to Perry Way

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Matt T -
You're not the only one, I've seen posts on here with people encountering the same URL anchor tag appending issue, albeit on Linux boxes, not Windows boxes.

Really good job nailing down the offending line of code. According to a quick check of git blame it doesn't look like that code has been touched in years. Although it's possibly causing issues somewhere else. 

Do you have access to error logs? Can you add error_log($url); above and below that block of code (uncommented), load upgrade as before, and see what is being logged? I'm curious to see the string value before/after it passes through the filtering functions.
Average of ratings: Useful (1)
In reply to Matt T

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Perry Way -

Hi Matt, Oh, thank goodness I'm not the only one. I had to do some checking but I found that error_log() gets ported to the Windows Event Log so after doing what you suggested the before value of the path was:

https://{our temporary testing domain}/admin/index.php

and the after was

<a></a>

This is interesting because my own way of checking variable contents (using echo $url; die;) provided an empty string for $url variable at the end. So some different behavior happens when using error_log() function.

+edit... ahh I just figured out why using echo($url); on the after value would show blank in the browser window because it's an anchor tag and has no path and no label, so of course it would show blank in the browser..


In reply to Perry Way

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Matt T -
What output do you get from this script (place in root dir)? Replace $url at top with the first string (the full path). 


I have plugged in many variations of URLs and am unable to get it to output empty anchor tags. Nor is it immediately apparent to me why the regex would do do.
In reply to Matt T

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Perry Way -

Hi Matt,

Well, I tried your script and it returns a 500 error response to the browser. In the Windows Event Log it reports:

PHP Parse error:  syntax error, unexpected ',' in C:\Websites\moodle_testing\website\specialtest.php on line 17

so urlencode() apparently throws the exception translating to a 500 error. 

If I modify your script and comment out that line of code the previous echo command returns: 

string(7) "

+edit: I am wondering if there's some kind of problem with the url itself. Let me explain. I have created a private domain, one that is not made public in our DNS. On Windows in /windows/system32/drivers/etc/host file you can enter in an IP address followed by the domain name and that will override any DNS settings. I did that both on my client machine and on the server. Why did I do this? Because I want to prohibit external traffic from poking around that site because it's not been proven and I might have issues pertaining to upgrading. On a Unix/Linux box I don't know how that is done but I know there's a way to do the same sort of thing. I'm wondering if somewhere in the chain of events in the Moodle core code if there's any external calls to DNS to resolve a domain, which would then return nothing perhaps? It's a far fetched idea but I thought maybe this extra information might be pertinent in getting to the bottom of this issue. I've never had this issue before when say working on my own development machine with a local copy. I've always just made up some domain and used the hosts file to point that domain to my local IP (127.0.0.1).   

In reply to Perry Way

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Ken Task -
Picture of Particularly helpful Moodlers

Not a Windows person and really can't suggest technical fixes on that platform, but have been a DNS server admin before.

Think all OS's (Mac's/Windows/Linux [any flavor]) all use the priority of hostfile mappings first, then DNS.

So in a Linux box, since apache does like to know who it is, I usually add a line to /etc/hosts which maps the IPv4 address to the fully qualified domain name and alias of the server.

Have also used same for a 'stealth' moodle by FQDN where that FQDN doesn't exist in DNS.

Example entry in /etc/hosts

10.209.1.241    int.somenet.org    intserver

or

sameIPv4publicIP   int.somenet.org intserver

If I use the 2nd example in my Mac's /etc/hosts file, I can use a browser to go to int.somenet.org - even though that doesn't exist in public DNS.

'SoS', Ken


In reply to Ken Task

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Perry Way -
Thanks Ken, you confirmed to me that other OS's have the same ability. I like how you referred to this as "stealth" Moodles smile Yeah, that's exactly my reason for going this way.
In reply to Perry Way

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Leon Stringer -
Picture of Core developers Picture of Particularly helpful Moodlers

The:

PHP Parse error:  syntax error, unexpected ',' in C:\Websites\moodle_testing\website\specialtest.php on line 17

is because there's an extra bit on the last line of the script that shouldn't be there:

              11,33         All

(I think that's the last line from Matt's text editor). Try removing this so the last line is:

echo urlencode($url);
Average of ratings: Useful (1)
In reply to Leon Stringer

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Perry Way -

Thanks Leon,

I saw that on the code view on that link Matt shared but I downloaded the file and didn't see it in the file at first. But it was hiding in the same line (17) but off the screen to the right so far that it didn't show up in my editor. So I thought the problem was with urlencode() but you were right, it was the offending line of code.

Having removed that bit (and adding a <br /> after each echo command) now I'm getting the result:

string(7) ""
%3Ca%3E%3C%2Fa%3E

which decodes as <a></a> 

In reply to Perry Way

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Leon Stringer -
Picture of Core developers Picture of Particularly helpful Moodlers

Without tracing too deep presumably it's clean_text() that's returning that. Of course this shouldn't happen.

I've tried Matt's script on my IIS system, I get:

string(9) "REPLACEME" REPLACEME

which I think is what is expected. My system is IIS 10 on Windows Server 2016, PHP 7.2.32. It's Moodle 3.7.

I suggest replacing the Moodle source code folder with a newly downloaded one: rename the current folder, download a new one and extract it to the root location for the site, then copy config.php from the backup folder (along with any plugins or themes the site has). This is to ensure the source code is definitely OK because my first guess would be that something isn't correct in the code.

Also if you can: purge caches (web, command line).

Average of ratings: Useful (1)
In reply to Leon Stringer

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Leon Stringer -
Picture of Core developers Picture of Particularly helpful Moodlers

Before changing the source code you could try moving the echo lines to different points and refresh the page to see which line it is where $url gets changed from "REPLACEME" to "%3Ca%3E%3C%2Fa%3E". For example move them before the first preg_replace() on line 10 and see what the output is, then move them after that line, etc.

In reply to Leon Stringer

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Perry Way -

Hi Leon,

Yeah this is a good exercise. I see that this is where the transformation occurs:

$encodedurl = preg_replace('/^.*href="([^"]*)".*$/', "\\1", clean_text('<a href="'.$encodedurl.'" />', FORMAT_HTML));
echo var_dump($encodedurl) . "<br />";
echo urlencode($encodedurl) . "<br />";

when I echo out $encodeurl, that's where I get the output 

string(7) ""
%3Ca%3E%3C%2Fa%3E

I'm bushed for the day and working on overtime at this point. Tomorrow I will attempt another fresh copy of code. The thing is when I ran into this problem first I though as you that maybe I got some mangled code. So I went through the process a second time of unzipping the download. But next time (tomorrow) I will re-download and try again to see if there's any new behavior. If it has the same behavior my next plan would be to download 3.10.0 instead of the latest version and see if that is the same.

In reply to Perry Way

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Ken Task -
Picture of Particularly helpful Moodlers

Been following this (painfully) but really can't contrubute (Leon is your best bet), but ... gotta state/ask ...

There is another way to perform the upgrade that doesn't involve weblib nor web service.    Yes, even on Windows, one could use code/admin/cli/ php scripts to upgrade.   Takes web services out of the loop.

Am wondering if a CLI upgrade would result in same issues?

'SoS', Ken

In reply to Ken Task

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Perry Way -

Hi Ken,

Yes I plan to do the upgrade process with CLI scripts. This problem happens to non-administrators logging in as well. It's not intrinsically related to the upgrade process. So I thought I should solve this problem before doing the actual upgrade. Additionally, it really benefits you to do the upgrade through CLI scripts on IIS because Microsoft defaulted IIS to timeout at a certain time (I think 90 seconds) and some of the upgrade processes could take longer than that before responding to the client. I've had that happen to me years ago and since then have stuck with CLI upgrades.

In reply to Leon Stringer

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Matt T -
+1 for cache clearing, it's possible this relates to bad cache including HTML purifier cache which may explain why you get the problem and we don't.
Average of ratings: Useful (1)
In reply to Matt T

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Perry Way -
Oh that's a good idea Matt! I didn't think about that. I will try that first, this morning.

+edit: Ahh, I just did it and tried to log in and the problem is still happening, so it's not a caching issue. I guess I'll do what I planned to last night and reinstall again from a new .zip download.
In reply to Perry Way

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Perry Way -
I'm not really sure where I should post this next response/feedback since there are some branches in previous interactions and responses, so I'm responding to my original post.

I just realized a few new things in this situation. Apparently I presumed we were already using MySql 5.7 but we aren't, we are using My Sql 5.6.2, which satisfies the requirements for our present Moodle version 3.8. But in order to migrate to version 3.10 we will need to be using MySql 5.7 or higher and preferably version 8. Since the server we are running our site on affects more than our Moodle, this will entail having to take down our main site, our student store site and all of our historical sites we keep active for accreditors to perform audits and whatnot while we are upgrading MySql to version 8. So I will be postponing our upgrade to Moodle 3.10 until our Spring/Summer break in late June early July and will be attempting to upgrade our current Moodle to 3.9 latest release.

So I attempted to do that today and would you believe, a fresh unzip of Moodle 3.9 latest is providing the same behavioral problems? It's hard to believe I know, but it is! I find this very odd. I'll need to do more digging around which is a bit cumbersome since I don't have the luxury of having so many versions of code installed on my development machine and will need to work remotely on our server. But I plan to put serious time into that.

For a hoot, I again set the debugging to be on, on the Moodle 3.9 latest upgrade site and found a ton of errors being reported on the /theme/index.php path. I have to type in the index.php actually otherwise the problem of prevents getting to that page. I wanted to set the theme to our new theme before doing the actual upgrade you see. Perhaps this is putting the horse before the cart but anyway, it led to a deeper discovery, and that being about our database issue. If you read some of my other responses I mentioned that I was getting an "error reading from database" twice on some pages I hand typed the URL's in the browser. Well with the debugging turned on now, I get more details about that, and here's something that's been puzzling me for hours. Apparently there's been some kind of change to the mdl_license table and that change has been referenced by a change in the code interacting with that table in the latest Moodle 3.9 release:

Here's the debug information:

Debug info: Unknown column 'sortorder' in 'order clause'
SELECT * FROM mdl_license ORDER BY sortorder ASC
[array (
)]
Error code: dmlreadexception
Stack trace:
line 486 of \lib\dml\moodle_database.php: dml_read_exception thrown
line 1273 of \lib\dml\mysqli_native_moodle_database.php: call to moodle_database->query_end()
line 1453 of \lib\dml\moodle_database.php: call to mysqli_native_moodle_database->get_records_sql()
line 232 of \lib\licenselib.php: call to moodle_database->get_records_select()
line 362 of \lib\licenselib.php: call to license_manager::get_licenses()
line 379 of \lib\licenselib.php: call to license_manager::get_active_licenses()
line 33 of \admin\settings\license.php: call to license_manager::get_active_licenses_as_array()
line 8670 of \lib\adminlib.php: call to require()
line 8559 of \lib\adminlib.php: call to admin_get_root()
line 33 of \theme\index.php: call to admin_externalpage_setup()

You can see that the column sortorder is missing in my database which is version 3.8 ready to upgrade to 3.9 latest. Now, being the kind of guy I am I am tempted to alter table and add that column as an int or a large int datatype to fix the problem but then I know we're not supposed to modify these tables ourselves but instead should use the XMLDB editor. So I went to that editor and found that mdl_license is not in the XMLDB editor (at least by name or category). Subsequent searching on the internet revealed no further details about that column. I found documentation at the path to generate all documentation: /admin/tool/xmldb/index.php?action=generate_all_documentation that showed that table minus that column. I am rather perplexed by this. Not being privy to all the history of design changes to the database structure leaves me in a state of not knowing what to do. 

Does anyone know more about this and can shed light on possibly what I should do to get past this impasse?  
In reply to Perry Way

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Ken Task -
Picture of Particularly helpful Moodlers

3.8.highest

mdl_license
Field name    Type    Allow nulls?    Key    Default value    Extras
id     bigint(10)     No     Primary     NULL     auto_increment
shortname     varchar(255)     Yes     None     NULL    
fullname     longtext     Yes     None     NULL    
source     varchar(255)     Yes     None     NULL    
enabled     tinyint(1)     No     None     1    
version     bigint(10)     No     None     0

sortorder first appeared in 3.9 and another field called custom

Field name    Type    Allow nulls?    Key    Default value    Extras
id     bigint(10)     No     Primary     NULL     auto_increment
shortname     varchar(255)     Yes     None     NULL    
fullname     longtext     Yes     None     NULL    
source     varchar(255)     Yes     None     NULL    
enabled     tinyint(1)     No     None     1    
version     bigint(10)     No     None     0    
custom     tinyint(1)     No     None     0    
sortorder     mediumint(5)     No     None     0

310
Field name    Type    Allow nulls?    Key    Default value    Extras
id     bigint(10)     No     Primary     NULL     auto_increment
shortname     varchar(255)     Yes     None     NULL    
fullname     longtext     Yes     None     NULL    
source     varchar(255)     Yes     None     NULL    
enabled     tinyint(1)     No     None     1    
version     bigint(10)     No     None     0    
custom     tinyint(1)     No     None     0    
sortorder     mediumint(5)     No     None     0

Can you take upgrading one hop at time?

3.8 -> 3.9.highest ... check ... then 3.10.highest

'SoS', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Perry Way -

Hi Ken,

Thanks for the information. Yes, I can take upgrading one step at a time and that's what I'm attempting to do. I have been under the impression that there is the option available to do the upgrade using the web interface rather than solely through the CLI interface. So, I decided to log on and see if the <a></a> situation had cleared up with version 3.9. That was my sole purpose actually for logging on. Interestingly this database error prevented me from doing anything past logging on, and I also discovered that the <a></a> redirection path was unavoidable. Due to this behavior I can't navigate (or go to) to the admin path for performing the upgrade via the web interface. 

Well, I will attempt to do it solely through the CLI interface and see if that is successful.

Thanks for your inputs.

In reply to Perry Way

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Perry Way -

Well, I have successfully upgraded to Moodle 3.9.6+ (latest release) and the URL rewriting to <a></a> is still happening. The database errors I referred to yesterday are now no longer occurring. There are some other residual issues but not debilitating at this point. The most important thing is the <a></a> rewriting that occurs. I suppose since this issue is not able to be reproduced by those who have commented on this thread that I'll have to dig deeply and find out where exactly this is happening and what is causing it.

The one thing good I have to work with is that others have had this issue before. So I'm not the only one. That eliminates spooky mystical reasons smile


In reply to Perry Way

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Perry Way -

Hallelujah! I just got to second base now! Here is a brief synopsis of my findings:

in weblib.php function redirect() was passing "https://moodle_testing.lauruscollege.edu/admin/registration/index.php?returnurl=%2Findex.php" to function clean_text(). Function clean_text() was returning "<a></a>"

I decided to type that url in and registered the site, and after doing that the problem went away.

I don't quite understand the chain of events that led to this issue, but I frankly find this a real impediment for anyone in the future who is planning to do any form of testing and upgrade on an alternate domain. I think it should be corrected because I am a professional developer and it took me days of testing and hacking around to figure out how to get to second base. I only got to second base because I intuited that since the URL (pointing to site registration) was being formed to any URL ending with /index.php or a trailing / that somewhere in the chain of events it would generate the URL to the site registration. But then the function clean_text() was returning <a></a> rather than redirecting to the site registration page.

How could one take a copy of a live site's database (with all its settings including site registration) and a copy of a live site's moodledata folder, put it on another domain for testing the viability of an upgrade without running into this problem? I see that as a pretty basic problem.

That said, now that I'm on second base there are other site errors that I have to work out. But now i know how to correct this situation in the future and I hope this eventually winds up helping others at some point. (And I hope someone in Moodle development looks into this issue as well)


In reply to Perry Way

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Matt T -
I plugged your $url in my test script and I think I have figured out the cause now.

In short my working theory is that this is caused by an underscore in the URL, which is fairly specific but not very uncommon which explains why this issue comes up occasionally.

An underscore in $url causes is_purify_html_necessary($url) to return true. Removing the underscore causes it to return false. If that function returns true, clean_text() calls purify_html() which I believe is causing the empty anchor tags to be returned. Otherwise, clean_text() simply returns $url untouched.

If I confirm this, I may have a look at submitting a patch to resolve it.
Average of ratings: Useful (2)
In reply to Matt T

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Leon Stringer -
Picture of Core developers Picture of Particularly helpful Moodlers

Well done Perry tracking this down.

I think name components for DNS host records may only contain letters, numbers and hyphens. If this had been a record on a DNS server it would have failed to add it. So I don't think it's a bug.

Underscores in the resource part of the URL don't cause this problem because they're allowed.

Average of ratings: Useful (2)
In reply to Leon Stringer

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Matt T -
You're right. An underscore is not a legal character for use in hostnames, as defined in RFC 882 (I suppose there's an exception for SPF records etc, or perhaps they're just not hostnames proper).

The code doesn't trigger purify_html() if the underscore is in the file path. 
Average of ratings: Useful (1)
In reply to Matt T

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Perry Way -
Thanks guys for your inputs! I was not aware of the underscore issue. I'll change that right quick and report back if the other residual errors I've been getting are resolved. It could explain for some other issues perhaps. Your help so far has been well received! Thanks!
In reply to Perry Way

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Matt T -
I've taken a fairly blinkered approach to your post, in just focusing on getting the anchor tag issue fixed. I've now read about the other issues.

Just to be clear: are you encountering those errors *before* running the database upgrade script for 3.10, or after? If you haven't yet run the upgrade script, it might explain why those errors are being encountered.

As has already been foreshadowed, and you have appreciated, you should use the CLI upgrade script and forget the web one (although it's nice that the web based upgrade issues alerted you to a potential issue with the underscore).
Average of ratings: Useful (1)
In reply to Matt T

Re: Error upgrading from 3.8 to 3.10.3 latest release

by Perry Way -
Hi Matt,

In my defense, my desire to log in first was I wanted to set the theme before doing the upgrade. We are also using a new theme and the previous one we were using on our live site I don't think would be supported by the boost theme in 3.9 or higher and I didn't want to get locked out so to speak. I don't know of a way to set the theme with a prepared CLI script so thought I should do that straight away before doing the upgrade.

I got those additional errors both before and after, and it appears they were all related to the underscore in the subdomain used. Once I cleared that up by removing the underscore all other residual problems went away, except for one. The last error I believe is not related to this issue I had, and I just created a new post for it. It is something I don't have time to chase at the moment since this Friday is a very busy day in preparation for new term start on Monday. Today is the day all new term courses are created by my software that backs up templates and restores to new term courses. And we had plenty of problems. I can return to trying to identify the issue next week sometime, but in the mean time if you're interested, here's the link to that. I think a simple change to the /index.php file would prevent this from ever happening to anyone else in the future:

https://moodle.org/mod/forum/discuss.php?d=421075