Performance with large quizzes and STACK

Performance with large quizzes and STACK

by George Schaathun -
Number of replies: 5
I have trouble getting satisfactory response time in large quizzes using STACK.  Moodle 3.9 with the fix of MDL-67183, but I still have similar problems, just with larger numbers of students and attempts.   Is there a chance that this will improve further in Moodle 4.0, e.g. by further lazy loading?  Or is there any other advice around on how to deal with it?

The problem is illustrated by this callgraph from the profiler, from the view when a single question is presented to a student.
-

1. I do note that the run time is doubled by a separate block module reinstantiating the activity.  I know how to deal with that, but I am hoping for a better improvement than 50% ...
2. The second problem is that question_attempt::load_from_records is called once for each attempt (yes, 19966 is exactly twice the number of attempts) by question_usage_by_activity::load_from_records . This seems superfluous, as these attempt records are not used when a question is presented.
3. Most of the actual work is done inside STACK, so the superfluosity is not necessarily a problem for other question types.  I must admit that STACK is used without pre-instantiated variants, so we do not benefit from the cache.  However, monitoring with top(1) shows that the CPU is used by apache and not by maxima, so I am not convinced that this is a problem.
4.  The reason why pre-instantiation is not used is that I need such a large number of variants (as the students get many variants each) that it is very tedious to set up, exported XML and backup files become huge.  I simply have not found a practical and efficient way to do it.
Average of ratings: -
In reply to George Schaathun

Re: Performance with large quizzes and STACK

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I see this relates to a custom activity called mod_capquiz - and I see that you are one of the maintainers of that module. (But that is all I know about this activity so far. I have just looked at the screen-grabs in the plugins DB.)

1. I looked at the diagram before reading your post, so I was going to point this out.

2. I am glad you pointed this out. OK, so you have a single question_usage_by_activity containing approximately ten thousand question-attempts. That is never going to work. Surely one student has not attempted ten thousand questions?

Normally, one question_usage_by_activity should correspond to something like a quiz attempt. The typical quiz attempt has about 6 questions, and it is rare to go above 30. However, from reading this forum over the years, I know that some teachers are sadists, so if I had to put a number of it, I would say that the quiz/question system should cope fine up to about 200 question-attempts in one usage (although I would not go that far with STACK questions). Anyway, you are about two orders of magnitude beyond that. I am afraid you are going to have to re-organise how your activity manages its data, if you want this to work at this scale.

It seems that with capquiz, iit seems that from time to time, students come in and attempt a batch of questions. I think you should create a separate question_usage_by_activity for each time a student does a session attempting some questions. Then, to display overall statistics, you will need to aggregate the data, but there are ways to pull out aggregate data efficiently, you don't need to fully load all the question_usage_by_activitys. You may be able to use the same helper functions that the quiz module uses to display its repots, or you may need to keep your own statistics and update them every time a student completes a qestion, or batch of questions.

I hope that is useful. Sorry to be the bearer of bad news.

Average of ratings:Useful (2)
In reply to Tim Hunt

Re: Performance with large quizzes and STACK

by George Schaathun -
That's very helpful, Tim. You are right that I am one of the maintainers. Unfortunately the original developer has left the team, and so has the only other developer who has ever been this deeply into the question bank API.

To explain the numbers. There are a little more than 150 students enrolled on the activity, having made a little more than 60 question attempts each on average. Thus, one question_usage_by_activity seems to aggregate all students, all questions, and all attempts by each student on each question.

Are you saying that the normal and/or recommended approach is to make one question_usage_by_activity per student? That would probably suffice to make the system scale. Are there any API functions to get parts of the data structure, such as the questions without the attempts?

I am not worried about the statistics. The main statistic is the student's rating which is maintained and updated for each attempt. For analysis I need all the attempts, but when that's needed, I can wait.

BTW. I have found that STACK is not the issue here. A similar test with an activity using Calculated give similar results.
Average of ratings:Useful (1)
In reply to George Schaathun

Re: Performance with large quizzes and STACK

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Changing to one question_usage_by_activity per student would be much better. The basic docs for using the question engine is https://docs.moodle.org/dev/Using_the_question_engine_from_module.

https://docs.moodle.org/dev/Overview_of_the_Moodle_question_engine#Detailed_data_about_an_attempt gives an overview of the database structure. Though, it is better to use the API if you can (because Moodle tries to keep the API backwards-compatible, but the DB schema can change). The helper functions that exist are https://github.com/moodle/moodle/blob/master/question/engine/datalib.php#L837 - but to be honest, the ones that exist are the ones that the quiz needed.

I acutally would expect STACK to be slower than other question types, but I can well believe that it is the 20,000-attempt usage that kills it, irrespective of question type.

I hope those pointers help. Please ask more specific questions when necessary. (Also, myself, and other people who know the question system like Marcus and Luca can often be found in the Developer's chat room https://docs.moodle.org/dev/Chat.)

Average of ratings:Useful (1)
In reply to Tim Hunt

Re: Performance with large quizzes and STACK

by George Schaathun -
Thanks. I am sure I will be back when I get stuck again, but this certainly sets ut moving in the right direction.

Maybe I exaggerated about STACK. The 15-20 multiple choice questions in the quiz seem to make no contribution to the CPU usage at all. My point was that STACK is not the only question type which can't take 10-20000 attempts in a quiz. It is also a problem with Calculated.

For the future, beyond 4.0, I am dreaming of question engine API designed for flexibility and reuse, beyond the needs of the core Quiz. In my opinion, the modularisation of the question bank, question types and question using activities is a major selling point of Moodle, because questions can be used didactically in so many different ways.
Average of ratings:Useful (1)