Pattern Match questions that jam the quiz (no review or student access)

Pattern Match questions that jam the quiz (no review or student access)

by scott hallman -
Number of replies: 2

Hi All

we ahve been using pattern match for a while and think it's great. We are getting better and the auto marking and using it in bigger exams. We often even use it for learning object using quiz as the tool, as pattern match (with a dummy answer) allows the student to compare their answer and the model answer on the same page (unlike essay).

However, recently when unticking the 20 word warning, and having the quiz set to immediate feedback we have had loads of issues. Which isvery concerning, since we would like to use this question type for a big exam in a month.

For the students who wrote 2-300 words, answered all the questions but did not finalise, as they wanted to go back, it locked them out and gave a 500 error timed out. If we clicked on review of the attempt it does the same. We changed the setting on the quiz to deferred feedback and that at least let the students back into the quiz to complete it. Nevertheless we still can't click on review to see the in progress attempt.

Students who wrote less and finalised their attempt, causes no issues and response times to vew and review are fine. But for the students in progress and with longer answers they are stuck in limbo not being able to get in or take another attempt.

The bug appears to be a combination of things:

  • long answers from students
  • quiz settings "question behaviour" set to immediate feedback/interactive with multiple tries
  • students completing the test but not finalising

We are reluctant to use this question type until we are sure it can handle 2-300 word answers and the auto marking, otherwise we are forced to use the "essay question" which has no possibility of auto marking and therefore we might as well do the exam on paper.

Has anyone had the same issues with students writing longer answers. Is there likely to be a fix. This quiz only had 5 questions in it. The exam we want to run has about 100. So it's probably good we found the bug now

Average of ratings: -
In reply to scott hallman

Re: Pattern Match questions that jam the quiz (no review or student access)

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I would not be surprised if there was some sort of compbinatorial explosion in the pmatch matching code when you give it a very long answer - it was not especially designed with that case in mind - as you can tell by the default 20-word limit.

Now that this problem is known, it would be possible to debug the performance of the code, and there are probably some pretty obvious optimisations that could be made. However, as you might be able to tell from my aggressive use of the passive voice there, I don't see myself having time to do that.

Average of ratings: -
In reply to Tim Hunt

Re: Pattern Match questions that jam the quiz (no review or student access)

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Just in case anyone else wants to have a go at this, to investigate this, I would:

  1. Make sure XHProf (or the PHP7 equivalent) was set up and working in my Moodle site. https://docs.moodle.org/dev/Profiling_PHP
  2. Make a very simple test.php script, which does the minimal code to match a long string against a pmatch expression - something like https://github.com/moodleou/moodle-qtype_pmatch/blob/master/tests/pmatch_test.php#L38 in a stand-alone script.
  3. Profile that script, to see if you can work out any performance optimisations ni the code (my guess would be that there is quite a lot of low-hanging fruit).
  4. Use the full PHPunit test suite, to verify that your proposed code changes do not break anything.
Average of ratings: -