On quiz 1, there were about 10 questions.
Item analysis runs perfectly.
On quiz 2, there are 50 questions.
When we click on item analysis, nothing ever happens. The browser progress just keeps spinning.
Too many students?
ໃນການຕອບກັບຫາ Dallas Ray Smetter
Re: Item Analysis hangs on some quizzes, but not others.
ໂດຍ Dallas Ray Smetter -
ໃນການຕອບກັບຫາ Dallas Ray Smetter
Re: Item Analysis hangs on some quizzes, but not others.
ໂດຍ Jean-Michel Védrine -
Hello,
Can I ask what Moodle version you are using ? The version of the mod/quiz/report/analysis/report.php file would help also if available.
item analysis do a lot of calculations and database queries but given your numbers of students and questions no time out should occur on a normal dedicated server.
Also can you give a rought estimate or the 50 questions types ? How many MCQ, how many short answers, ... do you have any random questions ?
Of course the problem to do such tests is that I don't have a big number of students and attempts on my development Moodle install (and not even on my production server only used by me and my 300 students !)
Can I ask what Moodle version you are using ? The version of the mod/quiz/report/analysis/report.php file would help also if available.
item analysis do a lot of calculations and database queries but given your numbers of students and questions no time out should occur on a normal dedicated server.
Also can you give a rought estimate or the 50 questions types ? How many MCQ, how many short answers, ... do you have any random questions ?
Of course the problem to do such tests is that I don't have a big number of students and attempts on my development Moodle install (and not even on my production server only used by me and my 300 students !)
ໃນການຕອບກັບຫາ Jean-Michel Védrine
Re: Item Analysis hangs on some quizzes, but not others.
ໂດຍ Tim Hunt -
The OU is probalby about to oustsource some developments of the Item Analysis report. When I have written a spec (hopefully next week) I will post it publicly, so people can comment. I think one of the things I will include in the requirements is that this report should do the calculations once and then store the results in the database, rather than recomputing everything every time.
Effectively one of the bottleneck in the development of the extended calculated questiontype is that the actual Item Analysis does not handle multianswer question types (Cloze is not handled also).
I think I will continue my work on calculated (and cloze) on all the other aspects and wait to see your specs.
Pierre
P.S.Once your specs are defined, do you plan to do this development yourself?
I think I will continue my work on calculated (and cloze) on all the other aspects and wait to see your specs.
Pierre
P.S.Once your specs are defined, do you plan to do this development yourself?
One of the main problem is related to the call in Item Analysis report.php to the
function get_all_responses(&$question, &$state) .
the default question/type/questiontype.php function
function get_all_responses(&$question, &$state) {
if (isset($question->options->answers) && is_array($question->options->answers)) {
However multianswer has no answers, they are in the subquestions, so the multianswer (cloze) is not recognized.
The multinaswer question type could perhaps defined a specific get_all_responses(), however the actual report.php could not handle multiresponses from subquestions well.
Pierre
function get_all_responses(&$question, &$state) .
the default question/type/questiontype.php function
function get_all_responses(&$question, &$state) {
if (isset($question->options->answers) && is_array($question->options->answers)) {
However multianswer has no answers, they are in the subquestions, so the multianswer (cloze) is not recognized.
The multinaswer question type could perhaps defined a specific get_all_responses(), however the actual report.php could not handle multiresponses from subquestions well.
Pierre
ໃນການຕອບກັບຫາ Pierre Pichet
Re: Item Analysis hangs on some quizzes, but not others.
ໂດຍ Pierre Pichet -
We could use the function actual_number_of_questions() to give the number of questions included is one cloze question. However this will disturb the quiz display as actually all the questiontypes use the default actual_number_of_questions() return 1 with the exception of the description questiontype.
If quiz rendering does not split a question with actual_number_of_questions > 1 , this parameter could be used in the reports to correctly display the subquestions responses.
Pierre
If quiz rendering does not split a question with actual_number_of_questions > 1 , this parameter could be used in the reports to correctly display the subquestions responses.
Pierre
ໃນການຕອບກັບຫາ Pierre Pichet
Re: Item Analysis hangs on some quizzes, but not others.
ໂດຍ Jean-Michel Védrine -
This thread speaks of 2 related but very distinct things :
1) Improving item analysis efficiency
Believe it or not, the actual code is a lot more efficient that it was at the begining of this report !
But I agree it could certainly be improved a lot more. Enrique would surely not be mad at me if I say he wasn't a SQL guru when he wrote this report and neither was I when I took the task to maintain his code.
One of the problem is the lot of options you need to support in the code :
- you can analyze just one quiz attempt for each user. This particular attempt may be the one with the highest overall score, the first attempt or the last attempt of those performed. Alternatively all attempts data may be combined for a cumulative analysis.
- Some attempts can be excluded from analysis by setting a low limit for the score of the attempts to analyze. This limit is specified as a percentage (0-100) of the maximum grade achievable in the quiz.
But all people really using the analysis report will agree with me that these options are really necessary and their removal would greatly reduce analysis report usability.
One other problem is that at the time this report was written, we has to support very old MySQL versions. Now that MySQL 4.1 is the minimum requirement, I think some of the queries could be rewritten resulting in a a great gain in efficiency.
2) support more question types in item analysis report
As Piere said analysis report need to call the get_question_responses function wich in turn call the get_all_responses method of the questiontype.
It is very interesting to read again MDL-5379 "Analysis report should not have a list of accepted types" because it was at that time we had to decide how analysis will decide if a qtype is supported or not.
I am strongly in favor of specific get_all_responses methods for some qtypes but Pierre is right, the actual code is written with the assumption that the question can return a list of all possibles students responses. More exactly this assumption is not made by the analysis report but by Classical Test Theory : how could we calculate parameters such as facility index or discrimination index if we can't have a list of all responses ?
This list of all possible responses can either be in ->options->answers and fetched by the default get_all_responses or constructed by the get_all_responses method specific to the question type.
Can some Test experts answer to the following question :
For a question where a student must answer several subquestions is analysis of each subquestion separately the only sensible way to go ?
Tim I am very interested to read your specs because even if I live in a country (France) where most of the teachers (including those using MCQs) never heard about facility or discrimination index, I use item analysis quite a lot.
1) Improving item analysis efficiency
Believe it or not, the actual code is a lot more efficient that it was at the begining of this report !
But I agree it could certainly be improved a lot more. Enrique would surely not be mad at me if I say he wasn't a SQL guru when he wrote this report and neither was I when I took the task to maintain his code.
One of the problem is the lot of options you need to support in the code :
- you can analyze just one quiz attempt for each user. This particular attempt may be the one with the highest overall score, the first attempt or the last attempt of those performed. Alternatively all attempts data may be combined for a cumulative analysis.
- Some attempts can be excluded from analysis by setting a low limit for the score of the attempts to analyze. This limit is specified as a percentage (0-100) of the maximum grade achievable in the quiz.
But all people really using the analysis report will agree with me that these options are really necessary and their removal would greatly reduce analysis report usability.
One other problem is that at the time this report was written, we has to support very old MySQL versions. Now that MySQL 4.1 is the minimum requirement, I think some of the queries could be rewritten resulting in a a great gain in efficiency.
2) support more question types in item analysis report
As Piere said analysis report need to call the get_question_responses function wich in turn call the get_all_responses method of the questiontype.
It is very interesting to read again MDL-5379 "Analysis report should not have a list of accepted types" because it was at that time we had to decide how analysis will decide if a qtype is supported or not.
I am strongly in favor of specific get_all_responses methods for some qtypes but Pierre is right, the actual code is written with the assumption that the question can return a list of all possibles students responses. More exactly this assumption is not made by the analysis report but by Classical Test Theory : how could we calculate parameters such as facility index or discrimination index if we can't have a list of all responses ?
This list of all possible responses can either be in ->options->answers and fetched by the default get_all_responses or constructed by the get_all_responses method specific to the question type.
Can some Test experts answer to the following question :
For a question where a student must answer several subquestions is analysis of each subquestion separately the only sensible way to go ?
Tim I am very interested to read your specs because even if I live in a country (France) where most of the teachers (including those using MCQs) never heard about facility or discrimination index, I use item analysis quite a lot.
Here is the actual code flow:
there is no reference to the $QTYPES[$quizquestions[$i]->qtype] for calculations
Is Improving this code flow possible so that even with 700 students and 90 questions, we could under the 30 seconds or are we with the same problem that occurs when you backup courses or even import more than 50 calculated questions?
Pierre
P.S. Somebody told me about the possibility to set (in Unix) different execution time to different php files, is this rigth?
/// Here we rewiew all attempts and record data to construct the table
$questions = array();
$statstable = array();
$questionarray = array();
foreach ($attempts as $attempt) {
$questionarray[] = quiz_questions_in_quiz($attempt->layout);
}
$questionlist = quiz_questions_in_quiz(implode(",", $questionarray));
$questionarray = array_unique(explode(",",$questionlist));
$questionlist = implode(",", $questionarray);
unset($questionarray);
foreach ($attempts as $attempt) {
switch ($attemptselection) {
case QUIZ_ALLATTEMPTS :
$userscore = 0; // can be anything, not used
break;
case QUIZ_HIGHESTATTEMPT :
$userscore = $attempt->sumgrades;
break;
case QUIZ_FIRSTATTEMPT :
$userscore = $attempt->timemodified;
break;
case QUIZ_LASTATTEMPT :
$userscore = $attempt->timemodified;
break;
}
if ($attemptselection == QUIZ_ALLATTEMPTS || $userscore == $usermax[$attempt->userid]) {
$sql = "SELECT q.*, i.grade AS maxgrade, i.id AS instance".
" FROM {$CFG->prefix}question q,".
" {$CFG->prefix}quiz_question_instances i".
" WHERE i.quiz = '$quiz->id' AND q.id = i.question".
" AND q.id IN ($questionlist)";
if (!$quizquestions = get_records_sql($sql)) {
error('No questions found');
}
// Load the question type specific information
if (!get_question_options($quizquestions)) {
error('Could not load question options');
}
// Restore the question sessions to their most recent states
// creating new sessions where required
if (!$states = get_question_states($quizquestions, $quiz, $attempt)) {
error('Could not restore question sessions');
}
$numbers = explode(',', $questionlist);
$statsrow = array();
foreach ($numbers as $i) {
if (!isset($quizquestions[$i]) or !isset($states[$i])) {
continue;
}
$qtype = ($quizquestions[$i]->qtype=='random') ? $states[$i]->options->question->qtype : $quizquestions[$i]->qtype;
$q = get_question_responses($quizquestions[$i], $states[$i]);
if (empty($q)){
continue;
}
$qid = $q->id;
if (!isset($questions[$qid])) {
$questions[$qid]['id'] = $qid;
$questions[$qid]['qname'] = $quizquestions[$i]->name;
foreach ($q->responses as $answer => $r) {
$r->count = 0;
$questions[$qid]['responses'][$answer] = $r->answer;
$questions[$qid]['rcounts'][$answer] = 0;
$questions[$qid]['credits'][$answer] = $r->credit;
$statsrow[$qid] = 0;
}
}
$responses = get_question_actual_response($quizquestions[$i], $states[$i]);
foreach ($responses as $resp){
if ($resp) {
if ($key = array_search($resp, $questions[$qid]['responses'])) {
$questions[$qid]['rcounts'][$key]++;
} else {
$test = new stdClass;
$test->responses = $QTYPES[$quizquestions[$i]->qtype]->get_correct_responses($quizquestions[$i], $states[$i]);
if ($key = $QTYPES[$quizquestions[$i]->qtype]->check_response($quizquestions[$i], $states[$i], $test)) {
$questions[$qid]['rcounts'][$key]++;
} else {
$questions[$qid]['responses'][] = $resp;
$questions[$qid]['rcounts'][] = 1;
$questions[$qid]['credits'][] = 0;
}
}
}
}
$statsrow[$qid] = get_question_fraction_grade($quizquestions[$i], $states[$i]);
}
$attemptscores[$attempt->id] = $attempt->sumgrades;
$statstable[$attempt->id] = $statsrow;
}
} // Statistics Data table built
Most of the data is already stored in the database and if the response is already stored in the $questions[$qid]['responses'] array $questions = array();
$statstable = array();
$questionarray = array();
foreach ($attempts as $attempt) {
$questionarray[] = quiz_questions_in_quiz($attempt->layout);
}
$questionlist = quiz_questions_in_quiz(implode(",", $questionarray));
$questionarray = array_unique(explode(",",$questionlist));
$questionlist = implode(",", $questionarray);
unset($questionarray);
foreach ($attempts as $attempt) {
switch ($attemptselection) {
case QUIZ_ALLATTEMPTS :
$userscore = 0; // can be anything, not used
break;
case QUIZ_HIGHESTATTEMPT :
$userscore = $attempt->sumgrades;
break;
case QUIZ_FIRSTATTEMPT :
$userscore = $attempt->timemodified;
break;
case QUIZ_LASTATTEMPT :
$userscore = $attempt->timemodified;
break;
}
if ($attemptselection == QUIZ_ALLATTEMPTS || $userscore == $usermax[$attempt->userid]) {
$sql = "SELECT q.*, i.grade AS maxgrade, i.id AS instance".
" FROM {$CFG->prefix}question q,".
" {$CFG->prefix}quiz_question_instances i".
" WHERE i.quiz = '$quiz->id' AND q.id = i.question".
" AND q.id IN ($questionlist)";
if (!$quizquestions = get_records_sql($sql)) {
error('No questions found');
}
// Load the question type specific information
if (!get_question_options($quizquestions)) {
error('Could not load question options');
}
// Restore the question sessions to their most recent states
// creating new sessions where required
if (!$states = get_question_states($quizquestions, $quiz, $attempt)) {
error('Could not restore question sessions');
}
$numbers = explode(',', $questionlist);
$statsrow = array();
foreach ($numbers as $i) {
if (!isset($quizquestions[$i]) or !isset($states[$i])) {
continue;
}
$qtype = ($quizquestions[$i]->qtype=='random') ? $states[$i]->options->question->qtype : $quizquestions[$i]->qtype;
$q = get_question_responses($quizquestions[$i], $states[$i]);
if (empty($q)){
continue;
}
$qid = $q->id;
if (!isset($questions[$qid])) {
$questions[$qid]['id'] = $qid;
$questions[$qid]['qname'] = $quizquestions[$i]->name;
foreach ($q->responses as $answer => $r) {
$r->count = 0;
$questions[$qid]['responses'][$answer] = $r->answer;
$questions[$qid]['rcounts'][$answer] = 0;
$questions[$qid]['credits'][$answer] = $r->credit;
$statsrow[$qid] = 0;
}
}
$responses = get_question_actual_response($quizquestions[$i], $states[$i]);
foreach ($responses as $resp){
if ($resp) {
if ($key = array_search($resp, $questions[$qid]['responses'])) {
$questions[$qid]['rcounts'][$key]++;
} else {
$test = new stdClass;
$test->responses = $QTYPES[$quizquestions[$i]->qtype]->get_correct_responses($quizquestions[$i], $states[$i]);
if ($key = $QTYPES[$quizquestions[$i]->qtype]->check_response($quizquestions[$i], $states[$i], $test)) {
$questions[$qid]['rcounts'][$key]++;
} else {
$questions[$qid]['responses'][] = $resp;
$questions[$qid]['rcounts'][] = 1;
$questions[$qid]['credits'][] = 0;
}
}
}
}
$statsrow[$qid] = get_question_fraction_grade($quizquestions[$i], $states[$i]);
}
$attemptscores[$attempt->id] = $attempt->sumgrades;
$statstable[$attempt->id] = $statsrow;
}
} // Statistics Data table built
foreach ($responses as $resp){
if ($resp) {
if ($key = array_search($resp, $questions[$qid]['responses'])) {
$questions[$qid]['rcounts'][$key]++;
} else {
if ($resp) {
if ($key = array_search($resp, $questions[$qid]['responses'])) {
$questions[$qid]['rcounts'][$key]++;
} else {
there is no reference to the $QTYPES[$quizquestions[$i]->qtype] for calculations
$test = new stdClass;
$test->responses = $QTYPES[$quizquestions[$i]->qtype]->get_correct_responses($quizquestions[$i], $states[$i]);
if ($key = $QTYPES[$quizquestions[$i]->qtype]->check_response($quizquestions[$i], $states[$i], $test)) {
$questions[$qid]['rcounts'][$key]++;
} else {
$questions[$qid]['responses'][] = $resp;
$questions[$qid]['rcounts'][] = 1;
$questions[$qid]['credits'][] = 0;
}
}
}
$test->responses = $QTYPES[$quizquestions[$i]->qtype]->get_correct_responses($quizquestions[$i], $states[$i]);
if ($key = $QTYPES[$quizquestions[$i]->qtype]->check_response($quizquestions[$i], $states[$i], $test)) {
$questions[$qid]['rcounts'][$key]++;
} else {
$questions[$qid]['responses'][] = $resp;
$questions[$qid]['rcounts'][] = 1;
$questions[$qid]['credits'][] = 0;
}
}
}
Is Improving this code flow possible so that even with 700 students and 90 questions, we could under the 30 seconds or are we with the same problem that occurs when you backup courses or even import more than 50 calculated questions?
Pierre
P.S. Somebody told me about the possibility to set (in Unix) different execution time to different php files, is this rigth?
ໃນການຕອບກັບຫາ Pierre Pichet
Re: Item Analysis hangs on some quizzes, but not others.
ໂດຍ Pierre Pichet -
ໃນການຕອບກັບຫາ Pierre Pichet
Re: Item Analysis hangs on some quizzes, but not others.
ໂດຍ Jean-Michel Védrine -
Of course but if I remember well before quiz was rewritten and the actual overview and analysis reports introduced, the old "detailed statistics" didn't support random questions and it was a frequent request.
From a conceptual point of view a quiz with random questions is just a quiz with a bigger set of actual questions where each student don't attempt all questions. So from a conceptual point of view item analysis of a quiz with random question is just item analysis of a bigger quiz
My conviction is that the key to optimize this report is :
From a conceptual point of view a quiz with random questions is just a quiz with a bigger set of actual questions where each student don't attempt all questions. So from a conceptual point of view item analysis of a quiz with random question is just item analysis of a bigger quiz
My conviction is that the key to optimize this report is :
- better sql queries
- as Tim said, store in memory some data to "ease" and speed item analysis work.
ໃນການຕອບກັບຫາ Jean-Michel Védrine
Re: Item Analysis hangs on some quizzes, but not others.
ໂດຍ Pierre Pichet -
One way to improve the code is to have better interface between the quiz and the report so that random questions are identified and this could let the report load the questions only once.
The report should know something about the questions and the valid reponses for example in the case where there is more than one valid response (like multichoice in the non single version).
Calculated are a special case as there can be 100 versions of one question as the datasetitems are currently limited to 100.
This why I think that Tim is setting standards (for actual and plug-in).
Most of the actual code is build around the one question and one valid response which is put in a [''] array before testing or grading.
For multiple answers, the questionstypes usually code them separated by ',' ';' ':' convention which is known to the question type.
The report should know something about the questions and the valid reponses for example in the case where there is more than one valid response (like multichoice in the non single version).
Calculated are a special case as there can be 100 versions of one question as the datasetitems are currently limited to 100.
This why I think that Tim is setting standards (for actual and plug-in).
Most of the actual code is build around the one question and one valid response which is put in a [''] array before testing or grading.
For multiple answers, the questionstypes usually code them separated by ',' ';' ':' convention which is known to the question type.
As an example the new extended calculated in his multichoice multiple responses version will have to store in the attempt text response field the dataset item number, the responses order and the responses using ',' '-' ':' as separators.
Pierre
ໃນການຕອບກັບຫາ Jean-Michel Védrine
Re: Item Analysis hangs on some quizzes, but not others.
ໂດຍ Dallas Ray Smetter -
ໃນການຕອບກັບຫາ Dallas Ray Smetter
Re: Item Analysis hangs on some quizzes, but not others.
ໂດຍ Pierre Pichet -
Have you any control on the server that you use, so that you can set longer execution time or larger memory?
There was no real important code changes recently and your version is very similar to the actual newest one.(2.0 ).
Pierre
P.S. Is this an exceptional case or not ?
Improving the code for a specific quiz type with all of type MC, with four options, none of which are random can be done but setting your server differently and doing the report when the user number is low is an easier solution.
There was no real important code changes recently and your version is very similar to the actual newest one.(2.0 ).
Pierre
P.S. Is this an exceptional case or not ?
Improving the code for a specific quiz type with all of type MC, with four options, none of which are random can be done but setting your server differently and doing the report when the user number is low is an easier solution.
ໃນການຕອບກັບຫາ Pierre Pichet
Re: Item Analysis hangs on some quizzes, but not others.
ໂດຍ Dallas Ray Smetter -