item analysis course wide

item analysis course wide

by Ro Eng -
Number of replies: 9

Hi, I'm kind of new to the forum, so please be lomd smile

I have been reading alot of about quiz/questions/item analysis.  We have moodle 1.9.X and I just want to confirm a some of my thoughts about the item analysis portion.  I think from what I have read that the information is based on QA for that particular Quiz.  It is not overall cumulative throughout the courses or courses.  So, if I read the DC, DI of item analysis report of Question ID #1199 ..that information is only for Question Id#1199 in that particular Quiz..regardless of whether that same question has been used in a few quizzes.  Am I correct in that thought?  If yes, are there any solutions to tabulate the information to go course wide instead of per quiz?  Does that make sense?  Thank you for any responses, and of course..forgive me if you feel the question is not appropriate for this site. 

Average of ratings: -
In reply to Ro Eng

Re: item analysis course wide

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
It is a very good question.

And note that the quiz reports are changing a bit in Moodle 2.0. You may be interested in Development:Quiz_report_enhancements, although the changes do not really affect your question.

On one level, if you think of Item Analysis as a statistical technique for highlighting problematic questions. Since generally statistical techniques work better with more data, your suggestion to aggregate all the places the question is used seems sensible.

However, one must look more closely at the assumptions that underlie the item analysis calculations. They are:

1. The quiz is attempting to measure some ability of the students. The total score on the quiz is meant to be an estimate of that ability.

2. We want each question to contribute to the quality of that estimate.

3. Therefore, the correlation co-efficient between the score for a particular question, and the score for the whole quiz, should not be too low.

In other words, item analysis does note tell you about a particular question in isolation. It only tells you about a particular question in the context of a particular quiz.

So aggregating the results does not make sense.

However, I think it would be useful if there was a way to find out for a particular question, all the quizzes that use that question, perhaps with a link to the corresponding quiz reports.
Average of ratings: -
In reply to Tim Hunt

Re: item analysis course wide

by Ro Eng -
Thank you for the information. I did read the info on the upcoming 2.0 and am anxious to see it arrive smile

I appreciate your help and input and insight into the correlation co efficient. What you said definitely make sense. It would be nice to see where questions are used as a whole and some kind of statistical info.

Thank you for your help...great info
Average of ratings: -
In reply to Tim Hunt

Re: item analysis course wide

by Daniel McCloy -
The issue of how to interpret correlation coefficients is separate from the issue of how to interpret the raw item analysis data, i.e., what fraction of students answered correctly and what answers were chosen by those students who got it wrong. The latter is a really useful metric for determining whether questions that I think are clear and easy are in fact confusing to a significant number of my students. Thus I think you overstate your point when you say that "aggregating the results does not make sense." It is, in fact, quite a sensible wish. I would even go further than Rhonda and say that it would be nice to be able to aggregate that data across multiple courses that are drawing from the same shared bank of questions.
Average of ratings: Useful (1)
In reply to Daniel McCloy

Re: item analysis course wide

by Itamar Tzadok -
However, unless all these courses are homogeneous the multiple-courses-aggregation metric may not be very useful even if the questions are drawn from the same bank. Are all these courses homogeneous? smile
Average of ratings: Useful (1)
In reply to Itamar Tzadok

Re: item analysis course wide

by Daniel McCloy -
What I'm thinking of is the same class taught over multiple terms, so yes, fairly homogeneous.
Average of ratings: -
In reply to Daniel McCloy

Re: item analysis course wide

by Ro Eng -

From reading your responses and pondering etc..this is how I think I can get the data our instructors hope for:

Create a course, say Course 101 (year created 2009)

Group students who are in the course, say by graduation year, example RT 2010. 

When the course is over for this first group, backup the course and archive it as Course 101 2009.  Reopen the original course 101 with the data from the previous students (but unenroll them) and create the new group of students RT 2011 (year they graduate again).  That way the item analysis stays and you can actually separate it by group.I have completed some small tests, and have not seen a reason it wont work.  Again, I am new to this school, but they are currently archiving differently then what I said above.  They archive the complete course and then start a new course (same thing, but different).

Any thoughts or comments why this may not work?  And, does anyone know if the data will stay if the student is deleted from the system.  I did one test and it appears no, but I have read the data does not stay..??

Average of ratings: -
In reply to Ro Eng

Re: item analysis course wide

by Chris Collman -
Data is not deleted when students are deleted. I have deleted students then brought them back into a course and their scores and test are there. But I can not see their tests when the student are un-enrolled from the course.

We use groups in some courses and in at least 1.5.3 (huge grin), you have mapped out the way item analysis works.

Soapbox: I do not like groups and prefer archiving the old as you described. IMO tests should be vetted every cycle. Once a test is given you can not toss questions, and regrading for one group because the instructor said something different gets to be a nightmare. Even editing a question to make it clearer for the next group means it is not the same question. So in any case it is garbage in and garbage out, despite how analytical things look in an item analysis. Either archive the old or the poor students/faculty are stuck with identical tests that we can only hope are valid over time/cycles.

Moodle 2.0 will offer some alternative that will help me look at my "good questions" over time and I am thrilled.

Hope this helps

Average of ratings: -
In reply to Chris Collman

Re: item analysis course wide

by Ro Eng -

Thank you Chris.  I am not a fan of carrying the questions through a course ..I think it is valid for that test, with those students, but I am not the expert.  smile

I actually had the same conversation about editing questions and thus changing not only the question but the potential outcome of the test.  I think my 'guys' are the only ones who want to do it...for some strange reason. 

I have not met many fans of 'groups' or groupings...but I think it may have potential at least as an example.

I can't wait for 2.0 and hope that it helps too.  smile

Average of ratings: -
In reply to Ro Eng

Re: item analysis course wide report

by Russ H -

Hi,

I'm wondering if anyone knows of a way to export all of the item analysis reports for a course at the same time. We have a few instructors who would like to download and archive the reports but would rather not have to go into each quiz, then it's item analysis page, and download the individual files. If anyone has any ideas on how to do this, it would be greatly appreciated!

I also agree with the other posts that course wide reports for what answers were commonly chosen for questions regardless of answer would be helpful. It may not be the same stats as the quiz-isolated report, but having the option would be nice.

Best,

-Russ

Average of ratings: -