I probably should have posted here first, I posted in the Quiz forum first.
Anyways, I work in a K-12 public school district (10K students) in the United States. Originally, we were going to hire a consultant or third party company to help us with our bigger vision of "making Moodle better". This includes infrastructure work, UI work, integration with other tools (like Drupal or our SIS), repository/Hub work and work with Quizzes.
We are still doing that but we have an immediate need to optimize our Moodle server. We are having too many performance issues, especially around Quizzes and robust courses. I believe our cron job set up needs help as well. Basically it takes forever to do things like adding an assignment to a category or duplicating a quiz. It's becoming unreliable and teachers are forced to do work on off peak hours. We need someone who knows Moodle and Server optimization. We need help with our Moodle server set up.
Would a Moodle Partner be the best way to go? Any advice?
I believe the Quiz part of the problem was answered in the Quiz forum: https://moodle.org/mod/forum/discuss.php?d=280607.
I can't say much about the major rewrite of Moodle. You are going the need a huge amount of resources. (I read somewhere about the monetary value of major open source projects incl. Moodle. They weren't small numbers. ;) )
About the performance issues: If you have the local staff you should first try with the (free) help in the Hardware and performance forum. Failing that, you need to out source. Either you can advertise, for example in Moodle jobs https://moodle.org/mod/data/view.php?id=7232, or contact Moodle Partners directly.
I said _answered_, not _solved_. As I understood the discussion https://moodle.org/mod/forum/discuss.php?d=280607 the problem is a combination of the site-wide quiz bank related issues (tracker) and your particularly slow server. People here can only pinpoint to the cause of the problem. To solve this kind of problems you need good local IT staff. (I don't claim to understand the Quiz system from inside, but you got replies from its developer ;) )
If your Moodle server and install is badly set up, then it will be a lot slower than it needs to be.
Have your server admin staff worked through https://docs.moodle.org/28/en/Performance_recommendations, particularly the sections PHP performance, Apache performance, and MySQL performance?
Those are the basics, and if you got in the right outside expert, they could go through those sorts of checks on your servers on the basis of a lot of valuable experience. (However, I have no idea who the right person would be near you, looking at the list of Moodle partners would be a start.) They might also be able to advise on whether more powerful hardware is likely to make a significant difference, and also set up monitoring (e.g. nagios) so you can see what is happening on your servers at any time.
One way or another you need to know what's going on. It may be as simple as buying some more memory (it rarely hurts), it may be a slow disk subsystem (quite common with many VMs) or your expectations of what your hardware can handle may just be too optimistic.
In any case, you (or someone) needs to be running monitoring software (e.g. Munin, Nagios) to work out what is going on. Certainly many/most Moodle partners will be able to help you. If you like - and you are not running Windows - my own MP can help if you are stuck (http://www.e-learndesign.co.uk/). However, in the context of this forum it is proper that I recommend that you talk to more than one Partner.
The hard part is that I'm not a "Server" person and our network people here aren't Linux people. We do contract with a local partner for help but they are not "Moodle" people. I'll give one last ditch effort to work with them and go over those docs but these quiz issues are just so specific to us and specific to courses that we've spend literally hours and hours in forums, researching, debugging, bug tracker, etc. and our teachers are still locking up their course when duplicating an assignment, kids are still randomly getting errors. It really shouldn't be this hard to not have your Moodle course lock up.
While I shouldn't generalise - a decent Linux admin should not be completely stuck even if they are not familiar with Moodle. If Moodle has, apparently, ground to a halt then the server is doing *something*. Is the load low or high? Is memory getting swapped out to disk? What is the database doing? How busy is the disk subsystem?
This is all basic stuff...
My point is that if a page takes nearly 30 minutes to generate (which is pretty crazy) then *something* is happening on the server. The bottle-neck is someplace. As Moodle works fine for the majority of users we can (mostly) discount dreadful programming in Moodle. So, what is the server doing or what is the server waiting for? There must be something...
"So, what is the server doing or what is the server waiting for? There must be something..."
Which is exactly what I can't figure out. But now that we have an environment where we can replicate it, hopefully we'll figure it out and I might have to reach out to a Moodle Partner it sounds like. Thanks.