Seeking whatever assistance I can get on this serious issue: About a week ago (date not certain) we started to experience very slow Moodle login / logout times (2-3 minutes) for browser users, and mobile-app users are unable to log in at all (they receive an error code "logintokenerror: timeout has occurred"). Once logged in, Moodle actions all appear to work at "normal" speed, or close to it. The server appears to be working normally and other apps on this server are working normally. But the Firefox display of page-element load times shows that moodle/login/index.php is taking 1.7 min to load.
What is baffling is that there has been no substantive change to Moodle in the past several months except for adding courses and users - and not a lot of that either. We're not a large or busy site (~800 courses, ~3000 users total but only 5-10 concurrent users). We do have some system connections running over web services (Edwiser Bridge connecting to an external Wordpress website, ampEducator SIS course-syncing and SSO) but those were all implemented months or years ago with no recent changes. Further, we have a Test Moodle site which is an exact copy of our production site (literally a copy/restore from a few months ago) with the same configuration, connections, courses and users and it doesn't have these issues.
In the Moodle logs the failed Moodle app logins appear as "Web service authentication failed with code: "invalid_token" so I thought for a moment that perhaps our web services token had expired. So I recreated the token (this time without an expiry date) but that did not solve the issue. And our Test Moodle had the exact same token, and Moodle app connections are working for it.
Any clues where to look for this issue? What diagnostics should I be running to figure out what is happening?
We're running Moodle 3.9.11 on a MS Azure server LAMP (Linux, Apache, MySQL, PHP) stack.
I appreciate any ideas you can offer...
Stan