make quiz support 60 or more questions

make quiz support 60 or more questions

by Eran Levin -
Number of replies: 16

Hello there,

I use moodle 2.9 with my students, 

and I believe that the major flaw of it, that it supports only handful of questions.

If you try with large sites running quizzes with say, 100 questions or more, you'll get episodic CPU peaks. It usually occurs when a user is being examined with quiz and get feedback only when it finish his attempt.

another consequence of that is that ,presenting the questions is very slow, and some hosts just don't support executing scripts so long.

A good solution will be a  web "comet", or pooling a service, and getting the questions gradually.

this will lead that the load will be distributed more evenly.


also, user should not wait long for "attempt quiz now". I believe that initiating attempt should first init only the first questions ,printing the quiz, and then initiating the rest.


Thanks!

Average of ratings: Useful (1)
In reply to Eran Levin

Re: make quiz support 60 or more questions

by Emma Richardson -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

It is true that the quiz can be hard on the CPU.  But there are options that help:

1.  Upgrade, later versions help with server load.

2.  Put questions on separate pages - do not try to put all 60 questions on one page.

Average of ratings: Useful (2)
In reply to Emma Richardson

Re: make quiz support 60 or more questions

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

I think that having 1 question per page is better for the students as well as performance. It means they can focus on one question at a time.

Average of ratings: Useful (1)
In reply to Emma Richardson

Re: make quiz support 60 or more questions

by Daniel Thies -
Picture of Core developers Picture of Plugin developers Picture of Testers

Hi Emma and Marcus,

I think that Eran may already have one question per page. He says that the performance problem is on start quiz attempt and on final review. The quiz initializes all questions at the start of the attempt which would could be slow if many students are doing this at once. The final review places all questions on a single page even if they are one per page in the quiz. When time runs out all questions will be graded for all open attempts. It might help to have a maximum page size of review or an option not to display the review. It is possible that using immediate feedback or another behaviour instead of deferred feedback may also help.

In reply to Daniel Thies

Re: make quiz support 60 or more questions

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Sounds like good advice Daniel. The other thing is that the Quiz is  the most resource intensive of Moodle modules and I would recommend anyone doing high stakes quizzing to run them on a separate server. I believe it can help if the start of the quiz is staggered, even by a few tens of seconds. I used to run quizzes for about 15 students on some very cheap shared hosting space. 

That used to work fine so long as nobody else with hosting on the same machine was doing anything very resource intensive. 


In reply to Daniel Thies

Re: make quiz support 60 or more questions

by Daniel Thies -
Picture of Core developers Picture of Plugin developers Picture of Testers

I think the review problem could be eased by unchecking the attempt under the Immediately after the attempt in the review options. This returns the students to the quiz summary instead of showing the review. There would still be a delay for grading, but the server would be spared rendering the questions and feedback.

The quiz starts all questions at the beginning of the attempt with question_usage::start_all_questions even if the question is not displayed immediately. I am not sure why this is necessary. Is there a reason why the question start could not be done just before it is rendered the first time?

Average of ratings: Useful (1)
In reply to Daniel Thies

Re: make quiz support 60 or more questions

by Eran Levin -


I believe that the current quiz engine is implemented incorrectly. It should start the questions JIT. Otherwise, no quiz with 50 questions or more will be fast. This flaw I believe is easy to fix, and I would do so, if I knew what the moodle practices are. 


Do you know whether moodle team plans refactor some of this script's behavior?

In reply to Eran Levin

Re: make quiz support 60 or more questions

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I'm glad you think it is easy. We look forwards to seeing your patch.

https://docs.moodle.org/dev/Process

https://docs.moodle.org/dev/Coding_style

By the way, http://wiki.c2.com/?PrematureOptimization. Have you actually done any profiling to find where the bottleneck is?

In reply to Tim Hunt

Re: make quiz support 60 or more questions

by Eran Levin -

nobody pejoratives Moodle's developers, I just wondered, why such talented developers, let such incredibly popular feature to be broken.

I will read the guidelines, but till I'll write code that fits moodle's decision, I believe that its best interest is to solve what I believe, in my humble opinion , an easy problem.


P.S. have you scheduled a date to solve this issue?

In reply to Eran Levin

Re: make quiz support 60 or more questions

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Why have we not fixed it? Because it is not actually broken for most people. At the Open University, we have been running Moodle quizzes for more than 10 years, and in that time our students have probably made around five million quiz attempts, and this has just never been an issue. 

In that time, the quiz code has been massively reworked, to make it much more flexible, reliable and performant, so actually, if this did now become important, it would be much, much easier to implement this now, than it was in Moodle 1.6 when we started.

In fact, there are two possible approaches to improving performance here. As well as your suggestion, of only starting questions when a user moves to a given page of the quiz, the other approach would be to create one quiz attempt for each student as a batch job, before they click the 'start attempt' button.

Similarly, you could make it so that when students click 'Submit all and finish', process_finish is not called immediately as a synchronous call, but instead is processed asynchronously. Those changes would smooth out the two biggest spikes in server load in relation to 'all students at the same time' quiz attempts, and they would be safer changes to make, because you don't change what any of the quiz processing code does, just what time it gets called.

So, anyway, I've thought about it, but it has never been important enough to get done.

Are you sure your server is tuned as well as it could be? It would be good to do some profiling (https://tjhunt.blogspot.co.uk/2013/05/performance-testing-moodle.html) to find what is taking all the time. If I had to guess, I would expect it to be all the database writes that need to be done, but you can never be sure. If that guess is correct. DB writes is something you may be able to tune for. Which DB engine are you using?

Average of ratings: Useful (4)
In reply to Tim Hunt

Re: make quiz support 60 or more questions

by Daniel Thies -
Picture of Core developers Picture of Plugin developers Picture of Testers

The quiz cron already runs process_finish when the option is set to submit automatically when time expires. A quiz access rules plugin could be written to prevent the immediate processing of the submission without changing the core code.

In reply to Tim Hunt

Re: make quiz support 60 or more questions

by Eran Levin -

I believe that this feature is the most absent feature. If your system can't handle more than 60 questions, then block the user using more than 60 questions. But if you do let them add 60+ questions, handle it properly or at least logically. I hope that you do recognize that the open university's requirements are by no means  any guidance of what the other moodle admins need or need not.

I've seen that the quiz code changes constantly, I really undecided whether to fork the quiz activity and just change it there, or try to make changes and try to merge with the current branch.


I see that there is an auto save in the new version, but is there auto load (via ajax?)

In reply to Eran Levin

Re: make quiz support 60 or more questions

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I still do not believe there is a problem.

As a test, I just downloaded this set of 152 questions: https://moodle.net/mod/data/view.php?d=8&rid=425

In at test course on demo.moodle.net (which is just some random test site where people can try out Moodle, not especially powerful hardware) I made a quiz, and added all those questions.

Then I clicked the start attempt button, and the attempt started in a couple of seconds. (I did not try to time with any accuracy.)

Actually, I did try to time accurately, by turning on Site admin -> Development -> Debugging -> Show performance info, but unfortunately, startattempt.php where all the work is done does not display anything. It ends with a redirect to attempt.php, so the timings and DB query counts for starting a huge attempt do not get shown in any simple way.

For what it is worth, I think the time will be dominated by the DB writes, which I think should be

  • 1 INSERT into quiz_attempts
  • 1 INSERT into question_usages
  • 152 INSERTS into question_attempts (one for each question)
  • 152 INSERTS into question_attempt_steps
  • A lot of INSERTS into question_attempt_step_data, but done as a single large batch using insert_records. (Efficient insert_records implementation seems to have been done for MySQL and Postgres.) 
  • A few other things, like inserts into the log tables.


I really think, before you persist in telling us to rewrite all the quiz code, you should do some basic performance tuning on your server. https://docs.moodle.org/29/en/Performance_recommendations

Average of ratings: Useful (2)
In reply to Tim Hunt

Re: make quiz support 60 or more questions

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers

Tim, I was surprised to read about some of these comments about the moodle quiz engine.  Granted, I don't have a big moodle, so I might not ever see the volume of quiz-taking that might cause a problem, but I do run my moodle on a small VPS, so performance is always a concern.  In 8 years of using moodle, I haven't seen any quiz performance issues.  Having said that, I do run my quizzes asynchronously, don't typically have 60 questions (usually 20-40), and don't put one question on a separate page.

In reply to Emma Richardson

Re: make quiz support 60 or more questions

by Eran Levin -
have any changes been made during versions 2.9-3.2 to the quiz activity performance ?



In reply to Eran Levin

Re: make quiz support 60 or more questions

by Germán Valero -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators

Hi Eran,

Have you looked at the Exam additional plugin ?

"It allows the teacher to build quizzes with multiple choice question based on Quiz module. Main purpose of Exam module is to improve performance of quiz and to reduce database overhead, so that a large number of users can attempt quiz simultaneously. Built for people who want to conduct large scale MCQ Quizzes."

It comes with a warning: "Warning  - Technically it is possible for students to manipulate their exam grades if they know right parameters or they could see correct answers if they could read javascript objets being transmitted in their browser. Use with Caution."

In reply to Germán Valero

Re: make quiz support 60 or more questions

by Eran Levin -

 believe it is a solution for the problem for some users, but not for all including me.  I do want to have 1 question per page, and correcting it would involve, I believe :
1.  Implementing AJAX UI solution, whenever user switches questions . or auto load of 10 questions in-a-row.
2. JIT questions initialization.


we all agree that quizzes with 50 question or more, is a core feature of the system and not a plug-in associated, extreme case feature.