Quiz

 
 
Picture of Duncan O'Hara
Definitive answers for quiz timing and saving
 

We have been running quizzes using Moodle (v.1.9) for over three years with vast numbers of students and staff.  Over the years we have encountered a wide variety of student behaviour during quiz attempts that have caused a range of different problems.

One of the persistent issues we experience is students' ability to go over time i.e. the quiz attempt is recorded as taking 53 minutes when the quiz time limit is 50. While we are not confident we have identified every reason for this we have a number of scenarios that seem to explain most cases. Using the browser back button, opening multiple browser tabs and/or windows being a few prime candidates.

We have recently upgraded to v.2.3 and we were excited about the quiz having had an overhaul.  We just ran a quiz for over 300 students and once again some students have managed to go over the alloted time limit - 50mins.  I was hopeful that the new settings that allows an open attempt to be submitted would help us get around some of the 'quiz attempt has gone over time' issues. It was not uncommon for students (in v.1.9) to have saved (but not submitted) their questions, exited the quiz only to return a day later open the quiz which they thought they had submitted and for their attempt to be sent and graded zero as they were well over the allocated time limit.  Forcing submission of open attempts in v.2.3 I thought would pretty much eliminate this from happening.  The latest quiz has shown that the dreaded overtime scenario still exists.  Although the details of what the student did during his attempt is sketchy it does seem he had multiple browser tabs open and was doing things like navigating the Moodle site in which the quiz was launched from while in another tab his quiz attempt was open. The quiz was set to submit open attempt with no grace period.

I realise my question is unlikely to get a definitve answer, but I wonder if there is a single reason that would explain the overtime scenario?  Alternatively, maybe there are some tactics that other people have employed to assist in reducing/eliminating the number of cases whereby quiz attempts have gone over? I have a number of lecturers who are very frustrated about having to make judgement calls on whether a student should receive marks for quizzes that have gone overtime.  It is not uncommon that in every quiz taken (and we run hundreds each week) that there is up to 5% of attempts for each quiz that have gone overtime.  For class of a 1000 students, 5% results in a lot of work and loss of confidence in the integrity of running assessments in this way.

Somewhat related to the above is the question about questions being saved.  One of things that we found in v.1.9 was that when a quiz was broken over multiple pages some students decided to open up each page in a different browser window or tab and answer questions in each tab. This behaviour resulted in some very odd adding up of scores and lost answers. Some of our staff then reverted to having all questions on one page as it was just too hard relying on students to read instructions not to open multiple tabs.  While in v.1.9 there was the save button at the bottom of the page which we encouraged students to use regularly, in v.2.3 this save button has gone.  Apart from splitting the quiz questions over multiple pages, which we encourage staff to do, but which some steadfastly refuse to do due to previous bad experiences, how do we encourage students to save their question answers as they go given there is no save button?  It does seem rather odd to ask them to click on the next button, which takes them to an overview page,  as a means of saving.  Also, it is my understanding that unless a student does something like hit the next button then answers are not sent to the Moodle server. Simply there is no automated send of answers that happens in the background? I have seen some talk about using ajax (??) to send information back to the server on a regular basis but it seems this solution creates additional issues.

Sorry for the length of this post and if the answers to these questions have already been posted. I did trawl the forum posts for some time to see if I could find answers but with no luck.

Duncan

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Definitive answers for quiz timing and saving
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

Why do we have to allow quizzes to go overtime at all? The answer is because if the server is under heavy load, then that is not the student's fault, and we should not penalise them for that. Therefore, we have to build some slack into the system. The amount of slack is now configurable at Plugins -> Activity modules -> Quiz -> Minimum grace period. By default it is set to 60 seconds.

Also, even when Moodle is set to automatically submit quiz attempts, the code that does this is only triggered at appropriate moments, which is either

  • When the student does something relating to that quiz attempt (e.g. tried to go to the next page); or
  • Cron runs.

So, the time the attempt gets automatically submitted may not be exactly the time the time limit ran out. Despite that, in Moodle 2.3, no student responses will be processed that Moodle sees later than quiz end time (= later of quiz close time, and attempt start time + time-limit) + minimum grace period, even if the recorded submit date is later than that. (http://docs.moodle.org/23/en/Quiz_settings#Examples_of_how_timing_is_handled give some more examples, if you really want a definitive answer, you will have to read the code and understand what it does.)

In terms of either ajax auto-save, or a manual save button, I am being asked to think about this at the OU. I think autosave is better if it can be made to work, but there are two potential isses that would need to be dealt with:

  1. The amount of data that might end up in the database, and
  2. a technicality to do with adaptive behaviour quizzes.
 
Average of ratings: -
Picture of Susan Mangan
Re: Definitive answers for quiz timing and saving
Group Particularly helpful Moodlers

Hi Tim,

Sorry, can you clarify something for me? 

If a quiz is set to close attempts automatically, does Moodle save the student responses at that time regardless of whether or not it was over the time limit (eg, due to server load, etc.)

I understand a student can be given a score of zero if the time is over the allotted period but I thought the answers were saved regardless.

We're running 2.4.3+ (Build: 20130405)

Thanks!

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Definitive answers for quiz timing and saving
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

Moodle does not save student responses that are submitted (too) late.

However, in normal circumstances (student actively working on the quiz in a web browser that has JavaScript enabled) the quiz will be submitted as time expires, and those latest responses will be saved and graded.

(Because networks and Moodle can be a bit slow, these final responses will be processed even if they only get to Moodle a little bit after time has expired. However, if Moodle is very overloaded, responses may arrive too late even for that, in which case they are discarded. This amount of slack can be set under admin -> Plugins -> Activity modules -> Quiz -> graceperiodmin)

If the student is not actively working on the quiz as time expires, well the latest responses will have been saved already, and they are what will be graded.

 
Average of ratings:Useful (1)
Picture of Susan Mangan
Re: Definitive answers for quiz timing and saving
Group Particularly helpful Moodlers

Thanks Tim that helps me to better understand what I need to be aware of to help ensure quiz answers are being saved properly.

However, re:

"If the student is not actively working on the quiz as time expires, well the latest responses will have been saved already, and they are what will be graded."

Unless you re running 2.5 the latest responses will not have been saved if the following conditions are met:

- all the questions are one one page and no attempt to click the Next button was made during the time the student was typing in answers

- the quiz is set to automatically close when time expires

- for whatever reason the server response is slower than the time-out setting in gradeperiodadmin rendering the attempt late which loses all of the student responses

I have been encouraging our Teachers to NOT put all questions on one page for some time now (and we have left the default to be one question per page, even though this is not ideal) because this is the only way to reassure responses are actually saved at the time the quiz is being performed. 

However, sometimes it is difficult to reach all our Teachers with this crucial information and sometimes a teacher will think they have paginated correctly because it says so in the quiz settings however, this is very misleading because it does not accurately reflect what the actual quiz looks like since further pagination can happen outside of the quiz settings.

All in all most of our quizzes have been successful with not a lot of problems however from time-to-time these little intricacies trip us up - would be nice if I had the time to be more proactive with keeping up to date on the newer documentation, testing and training ...

I have to say documentation at moodle.org has improved immensely over the last couple of years - very impressed with this smile  Because we've used Moodle since 2004 I sometimes forget to check the docs - it's always been a matter of testing, testing testing and workout out the glitches.  Now the docs are usually on spot!  Nice wink

 

 
Average of ratings: -
Tim at Lone Pine Koala Sanctuary
Re: Definitive answers for quiz timing and saving
Group DevelopersGroup Documentation writersGroup Particularly helpful Moodlers

If it takes more than 60 seconds between the quiz timer expiring and triggering and auto-submit, and Moodle processing the student's responses, then something is probably wrong somewhere.

However, we do try to constantly improve things. For example there is MDL-39246 in Moodle 2.4.4, and of course autosave in 2.5. Really, there are strong incentives to have a plan to update when you can.

You say you are running 2.3. I hope you mean 2.3.7, or at least something close to that.

 
Average of ratings: -
Picture of Susan Mangan
Re: Definitive answers for quiz timing and saving
Group Particularly helpful Moodlers

Well something must be wrong then because it just happened to us mixed

We are running 2.4.3 not 2.3 (I think the guy that did the original post was running 2.3 ...)

 

 
Average of ratings: -