At the OU we are proposing some changes to the quiz statistics report which (all being well) would probably end up in Moodle 2.6. There is a link to the full spec at the end of this post, but here are the highlights of what we are proposing:
1. In the screen-grab below, you can see what happens at the moment for random questions. Question 1 in the quiz is a random question picking from a category containing 5 questions. You can see how many students got each specific question, and what the statistics are. (So, if one variant of the question is harder or easier than the rest, you can see it.)
We are proposing to do the same analysis for question types like calculated, that have internal variants (that is, datasets).
2. This analysis is fine when you have just a few questions in a category (which is how we commonly do things at the OU). However, it is useless when you have a question bank with hundreds of questions. Above a certain limit, we will not show all variants. We will just show three rows for the min, average and max value of each statistic. If you want to expand that to see all the details, then you will be able to click a link to a separate page.
3. If you have a random question that picks different calculated questions with internal variants, then that is fully analyised, but only a summary is shown on the front page of the report, and you have to click through for details. (I don't think this is a common case, but it needs to work.)
Moving on from the statistics to the detailed analysis of responses (http://docs.moodle.org/24/en/Quiz_statistics_report#Analysis_of_individual_question_responses )
4. At the moment it only analyses the student's final response. When you are using Adaptive or Interactive behaviour, that is not necssarily what you want. We will add a new option to analyse either the first, last or all responses.
5. At the moment, all the statistics can only be calculated for each students first quiz attempt, or all quiz attempts. However, the quiz has two other scoring modes: Last attempt and Best attempt. We will extend the report so that it can analyse those attempts to, so you are analysing the attempts that determined the final grade.
6. A trivial point, but computing the statistics can be slow, so we will display a progress bar while the calculations are being done.
Here is the spec with the full details of our proposal. I'm afraid it is a PDF, and contains details of lots of OU-specific question types: https://docs.google.com/file/d/0B_W3H1TnOxK-UnJIcDZ0RWMxaWM/edit?usp=sharing
