Interpretation of Skewness in Quiz report statistics

Interpretation of Skewness in Quiz report statistics

by Vilmantas Pupkis -
Number of replies: 9
Hello!

While looking in https://docs.moodle.org/dev/Quiz_report_statistics about the interpretations of Quiz report statistics, I found such information about skewness.

Skewness: A measure of the asymmetry of the distribution of scores. Zero implies a perfectly symmetrical distribution, positive values a ‘tail’ to the right and negative values a ‘tail’ to the left. Aim for a value of -1.0. If it is too negative, it may indicate lack of discrimination between students who do better than average. Similarly, a large positive value (greater than 1.0) may indicate a lack of discrimination near the pass fail border.


Is it correct that one should really aim for skewness = -1? As far as I understand, a good test should give a symmetric normal distribution of grades, thus skewness should be equal to 0, and not -1.

Thanks in advance.
Average of ratings: -
In reply to Vilmantas Pupkis

Re: Interpretation of Skewness in Quiz report statistics

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I don't understand that recommendation. I don't see a value in aiming at a particular value.
In reply to Vilmantas Pupkis

Re: Interpretation of Skewness in Quiz report statistics

by Dominique Bauer -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

Student scores is not a random event, like rolling dice. It is the result of the effort the students put into the activity.

To say that there are as many students who make less effort than the average as students who make more effort than the average does not reflect reality, or at least should not reflect it. If an activity is well prepared and the students are interested, the skew should be negative.

If the skew is zero or positive, either your grading scheme is not adequate or your students are not interested. Either way, you need to remedy the situation. A zero skew for a good class is unfair to students.

If you are grading your students any which way for the final score, your department head should ask you to review your grades and possibly improve your course as well.

In short, a negative bias is desirable and should be aimed at.

Average of ratings: Useful (3)
In reply to Dominique Bauer

Ats.: Re: Interpretation of Skewness in Quiz report statistics

by Vilmantas Pupkis -
Thank you very much, Dominique, now the logic is much clearer to me.
If I might bother you again, do you happen to know any scientific literature that would justify why is it that "To say that there are as many students who make less effort than the average as students who make more effort than the average does not reflect reality"? Why (and how) does skewness correlate with student effort and engagement? Does this relation depend on the sample size?
In reply to Vilmantas Pupkis

Re: Ats.: Re: Interpretation of Skewness in Quiz report statistics

by Dominique Bauer -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

The ideal situation would be for all students to obtain 100% in their courses. The reality is that it won't happen, but it's still an ideal.

Some exams, not necessarily academic, are pass or fail. For those, it is clear that the result of the exam actually depends on its purpose or its nature. For example, if the exam is a job interview and only one candidate gets the job, all the other candidates have failed and the results curve will be extremely positively skewed. On the other hand, if the exam consists of being able to walk on two legs, the exam will be extremely negatively skewed because most humans can walk on two legs (unless they are completely drunk).

Ask yourself what the grades you give students are used for, to punish them or to encourage them. Results are very subjective. Some teachers will tend to give low marks, others high marks. But what comes into play? The teacher's preferences in the attribution of marks, the quality of his/her teaching, the difficulty of the exams, the level of the students, etc.

To me, low grades mostly indicate a failure on the part of the teacher to ensure that the students do well in their course. With flipped classrooms, it becomes clear that teachers should have a coaching role, that they should motivate students by helping them to do well. It is the success of the students that counts and the means to achieve it must be taken. I don't think giving students low marks helps them much, on the contrary.

You will find in the literature many articles written by statisticians who comment that a positive skew is the result of a hard test and a negative skew the result of an easy test. This is partly true, but far from taking into account all the elements.

Anyhow, here is an original article written by Curtis Miller, a statistician who has taught and thought about the issue. Mind you, you may not find too many articles with the same practical considerations and the same degree of thought.

Curtis Miller clearly demonstrates how applying a zero skew to a good class is downright unfair to students. He also destroys the myth of the normal distribution. I agree with everything he says in the article, and disagree with some comments, at the bottom of the article, which I find quite disappointing.

Miller, Curtis. 2019. Grades Aren't Normal. Curtis Miller's personal website. Posted on July 30, 2019.
https://ntguardian.wordpress.com/2019/07/30/grades-arent-normal/➚

Average of ratings: Useful (3)
In reply to Dominique Bauer

Ats.: Re: Ats.: Re: Interpretation of Skewness in Quiz report statistics

by Vilmantas Pupkis -
Thank you very much again.

I definitely agree that artificially imposing a distribution on the grades is an unfair and tragic practice, whatever that distribution might be. Still, a lot of grades do make a distribution, and without much thought, I observed that there are more students with average abilities than either excellent or abysmal, so it did look a bit like a normal distribution. The skewness is usually negative even in my own grading, but I used to attribute that to small samples and the "ceiling" effect.

It's a really fascinating question - if a grading system is more or less fair (at least let's pretend it is) - what sort of distribution do the grades naturally form? What are the fundamental (and, hopefully, measurable) reasons that make it that way? How do they interact?

It's probably as always - once one starts digging deeper, a very simple answer to a question turns out to be rather complicated if there even is an answer after all.
In reply to Vilmantas Pupkis

Re: Ats.: Re: Ats.: Re: Interpretation of Skewness in Quiz report statistics

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I think there are other renevant questions (which could be asked about any assessed activity). For example:

  • What outcomes to we care about most: the numeric grade that goes into the gradebook, or the learning that happens for the student when they struggle to complete the assessed task using what we have been teaching them? (Probably a mixture, based on the context.)
  • These numbers we get - how accurate are they really? We often quote then as percentages, which might imply accurate to +/-1%, but really they are not. For example, look at the research literature about what happens when different poeople grade the same essay (with the same grading criteria). Or, if the same student had sat the assessment yesterday, or tomorrow, instead of today, would they have go the same grade?
  • ... and so on.
The statistics are useful, and try to estimate the answer to some of the questions like this, but only up to a point.
Average of ratings: Useful (3)
In reply to Dominique Bauer

Re: Ats.: Re: Interpretation of Skewness in Quiz report statistics

by Dominique Bauer -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

Here are my own thoughts. Sorry if anyone else has posted similar ones without me referring to them. You can also disagree with me if you like.

In the Undergraduate studies regulations of my school, one can read:

The evaluation of the student in a course is the responsibility of the teacher.

Well, it's good all the same that they mention it.

The evaluation is the assessment of the level of learning achieved by the student in relation to the objectives of the courses.

It is obvious that the grades serve this purpose. Assess students, give them a grade, and rank them from weakest to strongest. Sure, but there are other purposes of evaluations.

I would say there are three types of quizzes. The first aims to help students become familiar with and learn a subject, the second aims to help them master it and the third aims to encourage them to strengthen their understanding of the subject and to let them check for themselves that they have understood it correctly. This third type is often described as a "summative" assessment.

But suppose an important summative exam is canceled at the very last minute, say just as the students are taking their places in the exam room. The students will complain and some will even say that they studied "for nothing". Obviously, the assessment of the level of learning achieved by the students is not made. However, while this may sound awkward, much of the goal of the exam has been achieved. Indeed, the students prepared for the exam and thus strengthened their understanding of the subject. This is the most important. Mind you, I'm not suggesting that canceling exams at the last minute would be a good thing.

In my view, school is not a place where a competition is held to find a winner, which would be a degenerate view of it. School is a place where students learn and that should be its main goal.

One can also read in the regulations:

Not all students should receive the same grade.

The average for a class of the results attributed should not be too far from the probable or normal average for such a course.

These two points are interesting. 1) The grades must be reasonably well distributed and 2) their average must be as expected. The regulations further says that if this is not the case, the teacher must revise the grades at the request of the Dean of Studies, but in reality most probably of the Head of the Department. Note that nothing is said about the skewness of the distribution.

Cont'd in my next post.

Average of ratings: Useful (2)
In reply to Dominique Bauer

Re: Ats.: Re: Interpretation of Skewness in Quiz report statistics

by Dominique Bauer -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers

A nice, slightly negatively skewed curve may be an ideal, but several factors make it not easy to achieve.

Let's take a given course as an example. Factors that influence the grades of the class include among others:

  • The level of the students, which can vary from one class to another. It may happen that some groups are measurably stronger or weaker than others, but in general the composition of the student population is fairly constant session after session.
  • The difficulty of the course. First-year courses in a program are arguably more challenging for students than final-year courses. Indeed, many things are new in the first year, and the rate of failure or dropping out is surely higher than in the last year, hence the averages lower.
  • How the teacher assesses and grades students. This is probably by far the most important factor influencing the grades for the class. Some teachers are harsh, others are lenient in awarding grades. So much so that our School provides teachers with an in-house program, prepared by our statistics teachers, to work on final grades. Professors can do whatever they want with this tool, for example automatically adjust the average, its standard deviation or its kurtosis (leptokurtic or platykurtic) and, manually, its skewness. But the marks are always submitted to the Head of the department for approval .

As a teacher, you are free, to a certain extent, to assign grades as you see fit. One thing is certain, there will be a certain distribution of grades. Is there one that is natural? Maybe yes under some conditions, but the conditions are so many and varied that the answer is maybe that there are several natural curves or maybe none at all.

As far as I am concerned, I give marks whose average and distribution correspond to what my School's administration expects. The curves also have a negative skew which comes from the way I assess students in my course.

By the way, I worked in industry for several years before becoming a teacher. For various reasons, I had to change jobs often and therefore had to go through many job interviews. The only company where I was asked for my academic transcript was one where, as I quickly realized, I felt lucky not to get the job.

Some students don't give a hoot about the grades they get, as long as they pass the course. In a way, I envy these students. Maybe they will have a better life.

Average of ratings: Useful (1)
In reply to Dominique Bauer

Re: Ats.: Re: Interpretation of Skewness in Quiz report statistics

by Dominique Bauer -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Plugin developers
...than those who care too much about their grades.