New quiz statistics pages

New quiz statistics pages

by Gustav W Delius -
Number of replies: 22
The existing quiz statistics pages need to be rewritten to make them more robust and scalable. Jon has more or less agreed to do this provided we can let him know what exactly it is we want out of these pages.

Here is my brief summary of what is there currently (I should hasten to add that I have not actually ever used these pages myself so this summary may not be a very accurate one):
  • Simple statistics page: This is simply a table listing all the students in the course and their grades at all the questions in the quiz.
  • Detailed statistics page: This page actually contains three separate tables:
    • Responses of Individuals to Each Item: A table whose rows are the students and the columns are the questions and in each field there is a summary of the student's answer at that question.
    • Item Response Analysis: A table whose columns are the questions and whose rows are the different answers given by students. The fields contain the number of times the answer was given for that question. This table as it is now does not work very well for short answer questions because of course there will be a large number of different questions given by the students which gives this table a large number of rows.
    • Listing of Items in Quiz with Summary Statistics: This gives for each multiple choice or true/false question the possible answers and the number and percentage of students who have chosen each answer. No information is given for other question types (I believe).
I would like to hear from anyone who actually uses these tables to find out which features are useful and how they could be improved. What for example is the difference between the "Item Response Analysis" and the "Summary Statistics"? Do we need to keep both?

There are some obvious things that should be improved:
  • each table that we decide to keep should be on its own page.
  • any table that has one row per student should be paged (for scalability) and sortable by names, grades, ...
  • The overview page with Jon's new paged and sortable table should be given download capabilities because then the simple statistics page will no longer be needed because Jon's table on the overview page can optionally display the grades at the individual questions.
  • The summary statistics for questions should be provided by the question types so that each question type can decide what is a suitable way to summarize the information. Clearly the results of numerical questions will need to be summarized differently from multiple choice questions for example.
Average of ratings: -
In reply to Gustav W Delius

Re: New quiz statistics pages

by Bill Click -
Actually, I like the current choices for statistics.  My only problem is with the sequencing of the data, sorting, which I hear is due out in release 1.5 .  I would be interested to hear others options though.
Average of ratings: -
In reply to Bill Click

Re: New quiz statistics pages

by Gustav W Delius -
Bill, thanks for your feedback. For what purposes do you use the second table on the detailed statistics page rather than the third?
Average of ratings: -
In reply to Gustav W Delius

Re: New quiz statistics pages

by Bill Click -
I like statistics.  I like the ability of seeing exactly what my students are learning and what I need to re-cover, or change my teaching approach.  The third one I would like to see some type of identifier to show the correct answer.  Currectly there is nothing showing on this report.
Average of ratings: -
In reply to Bill Click

Re: New quiz statistics pages

by Dennis Daniels -
Yes. I totally agree to an extant. The new 1.5 offers some more statistical analysis but it is not sufficient IMHO.

Please give 1.5 a run and evaluate its output.

We manage what we measure.

Dennis
Average of ratings: -
In reply to Gustav W Delius

Re: New quiz statistics pages

by Don Hinkelman -
Picture of Particularly helpful Moodlers Picture of Plugin developers
I use the Detailed Statistics data each year several times to analyze placement test data in language learning courses.  It is wonderful!

I use all parts, but the Item Response Analysis summary table (in the middle?) is the most important one (so maybe it should be moved to the top).  There I find a problem (poor IF or poor ID) then go to see the data in other tables.   To present the data of my 50 question test to my colleagues, I take screenshots of the rather long table (three screens wide) and manually tape 3 shots together.  Something more elegant would be great.

I cannot say exactly what else should be calculated or in what format, so we really need to hear from Tom Robb, who created and coded this functionality.
Average of ratings: -
In reply to Don Hinkelman

Re: New quiz statistics pages

by Don Hinkelman -
Picture of Particularly helpful Moodlers Picture of Plugin developers
To give more details on how I used and reported the Detailed Statistics, here is a conference paper I presented to the Pacific Computer Assisted Language Learning association in December.  On the last page, in the Appendix, you can see the how I pasted screen shots of Item Response Analysis onto a document.
Average of ratings: -
In reply to Don Hinkelman

Re: New quiz statistics pages

by Gustav W Delius -
You are making very good use of the item response analysis. Currently the information about each item is spread over two tables on the detailed statistics page, the "Item Response Analysis" table and the "Listing of Items in Quiz with Summary Statistics" table. Would you object to incorporating the information from the "Item Response Analysis" into the later?
Average of ratings: -
In reply to Gustav W Delius

Re: New quiz statistics pages

by Don Hinkelman -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Thank you for working on this.  I am not sure yet whether combining the tables is best.  My first impression is that I need the Item Response Analysis separate.  But I will take a careful look at all the data and see the best way to report it.  Also we should hear from Tom.  I don't think he knows we are working on this.
Average of ratings: -
In reply to Don Hinkelman

Re: New quiz statistics pages

by Thomas Robb -
Sorry to be chiming in a bit late on this.
I'm really glad to see that people find the module useful and are keen on extending it. I basically made it (selfishly) for the types of quizzes that I conceived language teachers to use. Also, since I have no formal training, my programming 'talents' are limited. I'm elated that capable hands are taking it over!

When I created the Detailed Stats module, my main purpose was 1) to show the individual answers for each student, and 2) to show summary statistics for each item.

Now with the summary stats, it soon became obvious that not all info could be neatly summarized in a single chart. In particular, the actual choices for each item were too long and messy. If someone can find a neat way of putting all of the info into one chart, that would be great.
Average of ratings: -
In reply to Thomas Robb

Re: New quiz statistics pages

by Don Hinkelman -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Dear Gustav and Tom,

I took a careful look at Detailed Statistics just now, and realized that I do not use the first table at all.   I am not concerned with responses by individuals, just composite totals.   So the second and third tables are critical.  I start with the second table for problems with items then look at the third table to see more specifically if a particular rejoinder (answer) had a problem. 

I like the two tables separate.  But if you have a compelling reason to combine  tables 2 & 3, I would not object.  But most of all I would like a way to transfer/print a page nicely with the data of 2 & 3  either combined or separate. 

I also would not object if table 1 was removed, but I am sure Tom had a good reason for putting that in.
Average of ratings: -
In reply to Don Hinkelman

Re: New quiz statistics pages

by Timothy Takemoto -

While there are no end of things that could be done to the statistics, the only thing I really need is the addition of random questions to detailed statistics. Pretty much all my questions are random so I get deliscious looking tables filled with rows and rows of "--"

Timothy

Average of ratings: -
In reply to Timothy Takemoto

Re: New quiz statistics pages

by John Bristor -
Mr. Takemoto..  I hear you were looking for me concerning Moodle stats on random questions.
Average of ratings: -
In reply to John Bristor

Re: New quiz statistics pages

by Timothy Takemoto -

Dear Mr. Bristor

Yes, and thank you very much for coming back.

When you let me have a test of your stats on random questions code, I did not know how much I would need it.

These days, to prevent cheating, all, or nearly all of my tests are random. I will be adding more and more questions in the future making all the quizes random. So the absense of random questions is rather a stumbling block for me. No random question stats means no stats at all.

As you can see however, moodle's stats are changing so perhaps the work you have done (on Moodle 1.1?) would now no longer be applicable. But If you would be so kind as to post the patch somewhere then perhaps (I hope), the quiz module developers will be interested.

Thanks again for coming back, and thanks again for the secure screen.

Timothy

Timothy Takemoto

Average of ratings: -
In reply to Timothy Takemoto

Re: New quiz statistics pages

by John Bristor -

(I had a little time to type up some thoughts)

It should work with some minor required changes unless there have been wholesale structural changes or data storage requirements to the quiz tables that I don't know about (is there a technical list of quiz table changes since v1.1.1?)  Everything else the module needs is built as needed.

One requirement is that for historical statistical analysis, I don't allow the deletion of questions that have ever been used.  A field called 'disabled' was added to the questions table.  The idea was that if students seem to have difficulties with a certain question, it could be evaluated, then the original question disabled and a new re-worded or replacement question added.  The resulting stats for the new question could then be compared over time to stats for the original question and answers, to measure improvement (or not).

My exams are all timed, and the 'general' stats page has overall averages on time used for the exam.  This might use features I added with the security changes and may have to be disabled in the stats module to be used by others.  

The third possible problem is that the module was designed for use by a single administrator.  Tables are dropped and built on the fly as needed.  For multiple admins or teachers, a session ID, or username field would need to be added to my stats tables to avoid any conflicts between multiple concurrent users.  Based on the code structure, it isn't as difficult as it may sound.

My exams are all 100 question exams pulled from a pool of hundreds or thousands of questions and they are all random.  The stats module does work with random or non-random multiple choice answers and though I believe it will work with true-false, I never tested it.  For multi-answer questions, a minor change or two would have to be incorporated.

Bascially, there are 4 main programs to my stats module:

1) buildstats - designed for use as a cron job or called periodically to go thru all quizes/exams and build base question and answer stats needed by the module.

2) Top Tier general stats - gives a high level look for all exams combined and for each individual exam broken out by category.  Each category for all exams or by individual exams has links to "drill-down" to the next level/tier"

3) Second Level/Tier - this level shows all questions for the category selected, sorted in descending order by percentage of incorrect answers with accompanying stats. Each question has a hot link to "Drill-down" to the next tier/level. (Disabled questions are noted in italics)

4) Third Tier/Level - this level shows the specific question to be evaluated.  All potential answers are listed with how many times each answer has been selected and any feedback that is in the table.  Each answer has associated stats.  

If you'll give me a couple of days, I'll create some links that give examples of the 'tiers'..  That may give you and possibly others a better idea of any questions to ask about its general intended operation or at least will be talking the same terminology when anyone starts looking thru or modifying the code.

Average of ratings: -
In reply to Timothy Takemoto

Re: New quiz statistics pages

by Dennis Daniels -
I share your model of testing Mr. Takemoto. Almost all of my tests are now random... it's taken a while to get a large enough database to make the random test truly random but I'm nearly there. The stats though, as you mentioned, are lacking. I've no idea what questions students are having universal trouble with as only those who speak up, asking for help, are pointing out their difficulties.

Stats for the random answer quizzes would be a HUGE improvement in most every testing environment, I think.

Question about 1.5, is it true that teachers can set the number of questions that appear on a test page? Three questions per page with five pages for a total of fifteen questions?


best
dennis
Average of ratings: -
In reply to Don Hinkelman

Re: New quiz statistics pages

by Thomas Robb -
Perhaps Don doesn't use the first table, but I do, which is why I made it! One use of the 'quiz' module for me, in the absence of a decent 'survey' module has been, has been to use if for surveys. I've just told my students to ignore the scores. (There was talk a while back of adapting the quiz module by simply having a survey/quiz toggle on the set-up page, but nothing seems to have come of that....).

At any rate, for this kind of application, one would want to see the individual responses of the students.

Another use for Chart 1 would be to 'eyeball' the discrimination of the items. If the items can be sorted into order by total score, then the top students should have all items right, with a progressively increasing number of wrong answers as you scroll through the report. Items that don't discriminate would show up because you could observe that there was no variation in the proportion of students getting a specific item wrong no matter where you scrolled on the listing.
Average of ratings: -
In reply to Thomas Robb

Re: New quiz statistics pages

by Don Hinkelman -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Tom wrote:
One use of the 'quiz' module for me, in the absence of a decent 'survey' module has been, has been to use if for surveys. I've just told my students to ignore the scores. (There was talk a while back of adapting the quiz module by simply having a survey/quiz toggle on the set-up page, but nothing seems to have come of that....).

Cool!  I never thought of that application.  Yes, it would be an excellent way to get survey results--until we get a good survey module.  Paging these results would help, wouldn't it?


 
Average of ratings: -
In reply to Don Hinkelman

Re: New quiz statistics pages

by Michael Penney -
-until we get a good survey module

Hi Don, I think we have a great survey module already: questionaire (phpESP).

IMO, would be nice to have this as part of the core, and then have the standard surveys from the current survey mod installed into questionaire by default (and call questionaire survey to avoid all the confusion).
Average of ratings: -
In reply to Michael Penney

Re: New quiz statistics pages

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
The current Survey module implements standardised, validated research questionnaires that work in all languages, which would be difficult to do in phpESP, I think, though i haven't looked at it for a while.  I believe there's a place for Survey still and in fact I'd love to add more academic surveys to it ... I still have plans about eventually doing things with the collected data (cross-site comparisons, aggregations etc).

Perhaps these two modules should swap names. big grin mixed
Average of ratings: -
In reply to Michael Penney

Re: New quiz statistics pages

by Don Hinkelman -
Picture of Particularly helpful Moodlers Picture of Plugin developers
Yes, Michael, I am eager to try Churchward's new improved questionnaire version with reporting and public/private viewing.  It has never made it into standard release, but I will try it soon.

We critically need a module that teachers can design quick internal daily, weekly, and summative evaluations.  And one that **students** can also use for creating research questionnaires/polling cross-course, course-site, and public.
Average of ratings: -
In reply to Don Hinkelman

Re: New quiz statistics pages

by John Rodgers -
Just to add to the discussion, I do use the student responses on individual items to identify student misconceptions and find out what parts of the questions need to be tweaked to make sure I'm not picking up weak readers instead of weak math..er (ers)
Average of ratings: -
In reply to Gustav W Delius

Re: New quiz statistics pages

by Darren Smith -

I haven't got any concrete suggestions (shy) but it would be nice if students could have access to some basic anonymous stats (course ranking, group ranking and the like). Possibly for individual quizzes and also for all attempted quizzes?

I know there is going to be a quiz block which I haven't looked at in detail so sorry if there is overlap.

Just a thought

Darren

Average of ratings: -