WebSequitur - Scoring anomaly

WebSequitur - Scoring anomaly

by Glenys Hanson -
Number of replies: 29

Hi Gordon,

A student has just pointed out a scoring anomaly in WebSequitur (unmodified source code): After you click on the first correct segment (word,sentence, image...) you see "Your score is 100%". So she stopped - she thought she had finished.  In fact, she had doubts and wrote to me about it - it's a distance learning course and her English is very weak.

I realise that this is a WebSequitur problem and not really a Hot Potatoes Module or QuizPort problem and I've written also on the Hot Potatoes List, but the scoring produced in QuizPort is weird. In the example below I've just clicked on the first correct segment. As you can see, I've uploaded it to your site.

Cheers,

Glenys

For some reason the video doesn't appear in the QuizPort version on your site - it does on mine. In the Hot Pot version it does on your site too. thoughtful

Attachment 13-10-2010 02-22-59.png
Average of ratings: -
In reply to Glenys Hanson

Re: WebSequitur - Scoring anomaly

by Gordon Bateson -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Glenys,
thanks for reporting this.

> but the scoring produced in QuizPort is weird

My brain battery is rather dim today. Could you expand on this. I can see the score says 50%, and I understand that you think this is weird. I need help understanding exactly why you think it is weird.

Is this the same score that WebSequitur reported?  Do you find it "weird" because WebSequitur showed a different score to the one  shown by QuizPort?

Or are you saying the WebSequitur score and the QuizPort score are the same, i.e. 50%, but you think this 50% is "weird", because having only clicked one segment, you expected you score to be just a few percent, not 50%, which implies that the exercise is half finished?

thanks
Gordon

Average of ratings: -
In reply to Gordon Bateson

Re: WebSequitur - Scoring anomaly

by Gordon Bateson -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

A message such as

  • Your score so far is 50%, but you haven't finished yet - keep going!

would be better then?

Average of ratings: -
In reply to Gordon Bateson

Re: WebSequitur - Scoring anomaly

by Glenys Hanson -

Hi Gordon,

>Is this the same score that WebSequitur reported?

When you're actually doing the exercise in Hot Potatoes, Hot Potatoes Module or QuizPort it's the same when you make a first correct choice, you see "Your score is 100%" It stays like that until you make a mistake. I understand the logic as being "You're 100% correct in what you've done so far."

>Do you find it "weird" because WebSequitur showed a different score to the one  shown by QuizPort?

Yes, because what has been shown as 100% inside the exercise is shown as 50% by QuizPort. 50% of what? mixed

>Or are you saying the WebSequitur score and the QuizPort score are the same, i.e. 50%, but you think this 50% is "weird", because having only clicked one segment, you expected you score to be just a few percent, not 50%, which implies that the exercise is half finished?

As I've just said the WebSequitur scores and the QuizPort scores are not the same, but yes for the second part.

>"Your score so far is 50%, but you haven't finished yet - keep going!" is much better but shouldn't it be 100%? See above.

It's amazing that I've used WebSequitur for years with dozens of exercises and hundreds of students and nobody's noticed this before.

Cheers,

Glenys

Average of ratings: -
In reply to Glenys Hanson

Re: WebSequitur - Scoring anomaly

by Gordon Bateson -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers
Ah yes, I see. Thank you. If the scores are different, I think i can fix that. Not sure what to do about the message. Maybe it is possible to change from within WebSequitur when making the quiz? Anyway, I will look into it. I'm a little busy at the moment, but I have put these two things on my "to do" list. regards Gordon
Average of ratings: -
In reply to Gordon Bateson

Re: WebSequitur - Scoring anomaly

by Gordon Bateson -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Glenys, probably you have found this already, but you can change the scoring message via the Sequitur Options -> Configure Output settings (see attached screenshot). Gordon

Attachment sequitur.options.gif
Average of ratings: -
In reply to Gordon Bateson

Re: WebSequitur - Scoring anomaly

by Gordon Bateson -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Glenys,

as you reported there was a discrepency between the score displayed by WebSequitur as it progressed through a exercise, and the score displayed by QuizPort for an abandoned attempt at a WebSequitur exercise.

I have found and fixed the cause of the problem, so the scores should be the same from now on. Unfortunately I can't fix the scores that are already stored in the Moodle database sad Please note that this issue only affects scores of attempts whose status is "abadoned", "timed out" or "in progress". There should be no discrepency for "completed" attempts at WebSequitur exercises.

It was an educating experience for me to investigate this issue. I learned that WebSequitur scores don't mean "Final score" they mean "Score so far". The only way to know the difference, is to check the status of the quiz attempt in the QuizPort pages that report score and status.

I could well understand that this may not be what students or teachers expect.

Please download and install the new QuizPort scripts when you have a chance.

Many thanks again for reporting this issue approve
Gordon

Average of ratings: -
In reply to Gordon Bateson

Re: WebSequitur - Scoring anomaly

by Glenys Hanson -

Hi Gordon,

Yes, I have thought of changing this to "Your score so far is:" but this would be shown also when the exercise is finished.

With WebRhubarb I've already done that but in that app there's also a "Finished" message (see below).

I've discussed all this with Martin Holmes Re: Scoring anomaly in WebSequitur and he's not willing to do major work when only one person has reported a problem. I see his point.

So for me now the problem is interpreting the QuizPort scores. But it's not urgent. The lady in question now knows what she has to do.

Cheers,

Glenys

Attachment 14-10-2010 11-15-33.png
Average of ratings: -
In reply to Glenys Hanson

Re: WebSequitur - Scoring anomaly

by Gordon Bateson -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

> So for me now the problem is interpreting the QuizPort scores.

If the status of a quiz attempt says "completed" then this is a final score.

If the status of a quiz attempt is "abandoned", "timed out" or "in progress", then the score is a "score so far".

HTH smile
Gordon

Average of ratings: -
In reply to Gordon Bateson

Re: WebSequitur - Scoring anomaly

by Glenys Hanson -

Hi Gordon,

Thanks a lot for fixing this so quickly.

By the way, when I posted my last message at 11:25, I hadn't read yours of 11:19 (hope one day we'll have ForumNG here where it's so much easier to read a whole discussion when you're replying to one message).

Now I understand about WebSequitur I can explain it to the students and make sure my instructions are clear. "Put the text in order" is obviously not enough for some people. I need to add something like: "Continue clicking on A, B or C until the text is complete."

I'll download the new version and test it out.

Thanks again. smile

Glenys

Average of ratings: -
In reply to Glenys Hanson

Re: WebSequitur - Scoring anomaly

by Glenys Hanson -

Hi Gordon,

Unless there's something I'm doing wrong, I'm afraid we're not out of the forest yet.

I've made a very simple WebSequitur and uploaded it to your site. Even when the attempt is abandoned after the first two words (there are 6 in all) it appears in the Grade Book as 100%.

It's the same in the Hot Pot Module and in QuizPort.

Bon courage !

Glenys

Attachment 14-10-2010 18-47-39.png
Average of ratings: -
In reply to Glenys Hanson

Re: WebSequitur - Scoring anomaly

by Gordon Bateson -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers
But 100% is the score reported by WebSequitur, so it seems to me that this system is working as it was designed to, even if that is not how you expect it to work.

Notice that the quiz attempt status is "abandoned" so you know it is not completed yet.

To get the scores and grades that you want, it would be necessary to change the scoring system for WebSequitur, in a similar way to how I imagine you changed the WebRhubarb scoring.

Gordon
Average of ratings: -
In reply to Gordon Bateson

Re: WebSequitur - Scoring anomaly

by Glenys Hanson -

Hi Gordon,

I never did anything to WebRhubarb scoring system - it's always worked that way as far as I know.

Would it be possible for "abandoned" "in progress" or "timed out" scores not to be reported in the Grade Book? From the student's point of view, when they see "100%" in the Grade Book it's natural for them to think they've completed the exercise, isn't it? And from the teacher's point of view it means a lot more checking.  At the moment, I download the gradebook once a week in spreadsheet form to see where the students are at. It would be a lot of extra work to open the individual QuizPort results for each student to check if they'd really completed the WebSequiturs or not. (I have 42 students doing 2 WebSequiturs a week for 9 weeks.)

I'm wondering now if there are JQuizzes which are being reported as 100% in the Grade Book when in fact the student has only answered the first question correctly and stopped. This is less likely because there is a message that says "Questions completed so far: 1/4". But I'll check and report back.

Cheers,

Glenys

Average of ratings: -
In reply to Glenys Hanson

Re: WebSequitur - Scoring anomaly

by Gordon Bateson -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Yes, it would be possible, but I'm sure some of your students would complain about, or be puzzled by, or feel it was anomalous to have apparently scored 100% in the activity but only receive 0% in the gradebook.

It seems to me that the cause of this issue is that WebSequitur reports a "score so far", which you and your student(s) interpret as a final score". I therefore think that the solution is to modify WebSequitur, so that it distinguishes between "score so far" and the "final score" for a quiz attempt. Then we ensure that it only reports the "final score" back to Moodle.

What do you think would be a reasonable "final score" to appear in the gradebook for a student who completed some but not all of a WebSequitur exercise?

Could you also report back on the proportion of your students you think this issue affects. What percentage of your students have "abandoned" WebSequitur quiz attemtpts which generate inflated grades in the Moodle gradebook?

Gordon

Average of ratings: -
In reply to Gordon Bateson

Re: WebSequitur - Scoring anomaly

by Gordon Bateson -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

I have just checked on the Moodle site I use for my univiersity classes. Having checked the results for 100 students doing 4 WebSequiturs, I could not find a single instance of WebSequitur producing inflated Moodle grades.

The only "abadoned" quiz attempts I found were either 0% (probably caused by students "just looking" at the exercise), or were superceded by a later completed attempt (e.g. the first student in attached screenshot).

In my students' case then, they seem to understand, or feel, a need to complete the WebSequitur activity.

Gordon

Attachment quizport.websequitur.results.jpg
Average of ratings: -
In reply to Gordon Bateson

Re: WebSequitur - Scoring anomaly

by Glenys Hanson -

Hi Gordon,

I'll do that checking and report back.

It seems to me that once students have understood how WebSequitur fonctions they should always complete it - not necessarily with a perfect score, of course. It's not a very difficult exercise. For me, it's usually the first exercise in set; the pedagogical objective being that students read (and listen to) the text to the end before attempting more difficult exercises.

I did check JQuiz and though it also shows "Your score is 100%" (if this option is activated) when only the first question out of 4 has been answered, in the grade book the score is correctly shown as 25%. Is there some reason that this is not possible for WebSequitur?

Martin Holmes said it would be a lot of work to change the scoring system for WebSequitur and I understand his reluctance to do it if it almost never causes a problem.

I may not have time to do the checking today - but I will do it.

Cheers,

Glenys

Average of ratings: -
In reply to Glenys Hanson

Re: WebSequitur - Scoring anomaly

by Gordon Bateson -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

> Is there some reason that this is not possible for WebSequitur?

This question is answered by the sentence which follows it, namely ...

> Martin Holmes said it would be a lot of work
> to change the scoring system for WebSequitur

In other words, it is possible but it requires some extra work because WebSequitur doesn't currently maintain a "final score". It only has an "ongoing score" which  becomes the "final score" when all the segments have been put in the correct order.

JQuiz on the other hand maintains a score for each question, so the final score can be found at any time by totalling up the individual question scores for the questions that are used the cuurent JQuiz attempt.

So modifying the scoring in WebSequitur is possible, but it requires some work, by somebody.

Maybe the person you found to modify the behavior of WebRhubarb could do the modifications for WebSequitur? I would be happy to include the modifications in a new QuizPort output format.

If you would like me to create the new QuizPort output format for WebSequitur, that is possible too. You may have to wait a little while though, as I have hit a busy patch.

regards
Gordon

Average of ratings: -
In reply to Gordon Bateson

Re: WebSequitur - Scoring anomaly

by Glenys Hanson -

Hi Gordon,

I haven't worked with the colleague who did the modifications to WebSequitur for some time so I feel a little shy of asking him - but he can always say no, can't he?

If I'm not shy of asking you and Martin it's because I 'talk" to you practically every day and I know you'll say "No" when you have other priorities. As you know, it's difficult for non-coders to know if what we ask for will take 2 minutes or 2 weeks.

I still haven't had time to check if there are any unjustified 100% in the gradebook. I've been busy sprinkling the course with admonitions to continue until they see the OK button and click on it. Far more students this year than last are "completing" their exercises so my instructions must be getting better. I think, thanks to you, I'm doing the settings for QuizPort better too.

I haven't said it for a while, though the Hot Potatoes module was great, QuizPort is Wonderful! cool

Thanks so much,

Glenys

Average of ratings: -
In reply to Glenys Hanson

Re: WebSequitur - Scoring anomaly

by Gordon Bateson -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Glenys,
it's good to know QuizPort is working well for you. Thanks for that!

I had a moment of inspiration early this morning. It relates to the score calculation in WebSequitur.

At any given time, WebSequitur calculates the score like this:

  • ScoreSoFar = PointsScoredSoFar / PointsAvailableSoFar

The value of PointsScoredSoFar increases as the student clicks on correct segments. However, the issue that has caused the confusion resulting in this thread, is that PointsAvaiableSoFar also changes as new segments are presented to the student along with new distractors.

We can produce a more expected score if we caculate a new value, TotalPointsAvailable, at the beginning of the quiz, and then use that whenever we calculate the score, thus:

  • ScoreSoFar = PointsScoredSoFar / TotalPointsAvailable

So the new QuizPort output format for WebSequitur will dispay a score that grows from 0% at the start of the quiz, to possibly 100% if all the segments are chosen in the correct order. At any time the score can be sent back to Moodle and used meaningfully in the "grade" caculation for the activity.

The downside is that the display of the score will be different from the current behavior of this type of quiz, which may surprize some students and teachers who have gotten used to the way WebSequitur currently works. However, the current output format will still be available. The new output format will only be used when teachers choose it from the QuizPort quiz settings page.

I will probably have time make the new output  in November.

regards
Gordon

Average of ratings: -
In reply to Gordon Bateson

Re: WebSequitur - Scoring anomaly

by Gordon Bateson -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Programmer's note: the following Javascript should do what is required if used as replacement for WebSequitur's standard "CalculateScore()" function:

function CalculateScore(){
if (typeof(window.TotalPointsAvailable)=='undefined') {
　window.TotalPointsAvailable = 0;

　// points for questions with complete number of distractors
　TotalPointsAvailable += (TotalSegments - NumberOfOptions) * (NumberOfOptions - 1);

　// points for questions with less than the total number of distractors
　TotalPointsAvailable += (NumberOfOptions * (NumberOfOptions + 1) / 2) - 1;
}
if (TotalPointsAvailable==0) {
　return 0;
} else {
　return Math.floor(100*ScoredPoints/TotalPointsAvailable);
}
// standard score calculation
// return Math.floor(100*ScoredPoints/TotalPoints);
}

Average of ratings: -
In reply to Gordon Bateson

Re: WebSequitur - Scoring anomaly

by Glenys Hanson -

Hi Gordon,

It's so kind of you to take all this trouble.

I tried copying and pasting your code into sequitur6.js_ but it stops the exercise functioning correctly: see below.

There are no other modifictions to the source code in the exmple I've uploaded to your site as WebSequitur Newscore

There's absolutely no rush for this now as far as I'm concerned - I think my students now understand they have to continue to the end.

Cheers,

Glenys

Attachment 18-10-2010 13-34-42.png
Average of ratings: -
In reply to Glenys Hanson

Re: WebSequitur - Scoring anomaly

by Gordon Bateson -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers
Oh, thanks for trying it out, but I intended for that code to be the just the basis of a solution. As you have found , it probably needs a little more polish before it is ready for action. I will get it working in Nov and let you know when it is ready to be put through it's paces. All the best, Gordon
Average of ratings: -
In reply to Gordon Bateson

Re: WebSequitur - Scoring anomaly

by Gordon Bateson -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Thanks for uploading the quiz.

The following function will probably work better.

function CalculateScore(){
if (typeof(window.TotalPointsAvailable)=='undefined') {
window.TotalPointsAvailable = 0;
TotalPointsAvailable += (TotalSegments - NumberOfOptions) * (NumberOfOptions - 1);
TotalPointsAvailable += (NumberOfOptions - 1) * NumberOfOptions / 2;
}
if (TotalPointsAvailable==0) {
return 0;
} else {
return Math.floor(100*ScoredPoints/TotalPointsAvailable);
}
}

I have tested it to a certain extent. Please could you add it to your sequitur.js_ and give it a trial run. While you are testing, please vary the total number of segments, and also the number of segments viewable at one time. Be extreme, if you like. Try to break it smile

regards
Gordon

Average of ratings: -
In reply to Gordon Bateson

Re: WebSequitur - Scoring anomaly

by Glenys Hanson -

Hi Gordon,

You're amazing! It works just perfectly now. I've tested it every way I can think of and have modified 14 different exercises with various other customisations plus video, images, audio... I've uploaded them to my production site so the students will be testing most of them too.

The only thing I noticed was nothing to do with scoring.  In my simple test exercise "one, two, three..." when I switched from "left to right" to "right to left" it continued to present the words from left to right. But of course, WebSequitur is intended for ordering big chunks, not words.

Can I tell Martin on the Hot Potatoes list that a solution has been found, or would you prefer to do it yourself?

Cheers,

Glenys

Average of ratings: -
In reply to Glenys Hanson

Re: WebSequitur - Scoring anomaly

by Gordon Bateson -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Oh great - thanks for giving the function a good testing Glenys.

> Can I tell Martin on the Hot Potatoes list
> that a solution has been found,
> or would you prefer to do it yourself?

That's entirely up to you.

From my point of view, I will add the function to a new QuizPort output format for WebSequitur. Teacher's will be able to choose between the standard WebSequitur scoring system, that starts at 100% and comes down if you make mistakes, or the new incremental scoring system that starts at 0% and goes up as you choose correct segments.

The new output format will be available to html and xml versions of WebSequitur exercises.

As long as the WebSequitur exercises are run through QuizPort, Martin H. doesn't need to change the WebSequitur application at all, and teachers can choose which scoring system they use. I'm guessing that Martin H. might prefer that solution wink

all the best
Gordon

Average of ratings: -
In reply to Gordon Bateson

Re: WebSequitur - Scoring anomaly

by Gordon Bateson -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Glenys,

just a note to say that the "incremental" scoring Sequitur output format for QuizPort is now available. It works for Sequitur exercises run from both xml and html files.

I really like it and have already started using with gusto.

thanks very much for bringing about this useful functionality approve
Gordon

Average of ratings: -
In reply to Gordon Bateson

Re: WebSequitur - Scoring anomaly

by Glenys Hanson -

Hi Gordon,

How nice of you to say that. smile  I sometimes feel I'm forever giving you more work as a developer. I forget you're a teacher too and actually create exercises for your students like the rest of us.

I use WebSequitur a lot as a way of getting students to read a text until the end, and not stop and look up a dictionary at the first problem. When they get to the end, they usually realise they've understood a lot of the text (I'm not talking about complete or near beginners, of course).

I'd love to hear how you and other people use it.

Cheer,

Glenys

Average of ratings: -
In reply to Glenys Hanson

Re: WebSequitur - Scoring anomaly

by Gordon Bateson -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers

Hi Glenys,

I agree that once students get involved in a Sequitur exercise they get read a lot of material before they can stop themselves wink

In the English conversation classes that I teach, we use the Sequitur quizzes to introduce sample dialogs. Students listen to an audio version of the dialog and click on the text segments of the dialog so that the completed dialog builds line by line in the bottom area of the exercise.

Next, they have a JCloze with one blank per line.

Finally, there are three JMatches in which the students have to rebuild the jumbled lines of dialog on the right and put them in a meaningful position on the left:

  1. English audio on left, English text on right
  2. 6 items (2 are fixed), English text on right
  3. 6 items (none fixed), English audio on right

At that point students leave QuizPort and do a Moodle offline Assignment which is to pair up and write their own dialog (based on the sample) and then perform it to me.

That usually takes them between 60-90 minutes, during which time I hover, encourage, console, awaken from slumber, and generally help them complete the tasks and instill in them nurture some useful learning habits (e.g. attend regularly, use a notebook, work independantly when you are able to, confer and cooperate when you need to, set and achieve goals).

regards
Gordon

Average of ratings: Useful (1)
In reply to Gordon Bateson

Re: WebSequitur - Scoring anomaly

by Deborah Delin -

Hi Gordon,

Thanks very much for sharing this.

I really like your idea of using fixed items on the right to gradually make the exercises more challenging.  I had never thought of doing that.

Deborah

Average of ratings: -
In reply to Deborah Delin

Language learning sequences, was Re: WebSequitur - Scoring anomaly

by Glenys Hanson -

Hi Gordon and everyone,

I find it very useful to read descriptions of other teachers' lessons. I hadn't thought of using JMatch with fixed items in that way either. I love the image of Gordon's role to "hover, encourage, console, awaken from slumber...nurture some useful learning habits."

Here's a sequece I've used on a distance learning course for French adults most of whom have an A2/B1 level on the Common European Framework of Reference for Languages. The objective is mainly reading and listening comprehension. It's based on a 350 word text and its audio recording. It's the 7th in a set of 12.

  1. Listen and put the text in order. WebSequitur
  2. Vocabulary help. JGloss
  3. Vocabulary practice: matching words to pictures. JMatch
  4. Choose the correct word: time expressions. JCloze
  5. Choose the correct word: this/that/these. JCloze
  6. Choose the correct word: actions. JCloze
  7. Choose the correct word: adjectives. JCloze
  8. Choose the correct word: numbers. JCloze
  9. Choose the correct word: few/many/more/most. JCloze
  10. Choose the correct word: some prepositions. JCloze
  11. Put the events of her life in chronological order: JMatch
  12. Find the mistakes. Find-It! a
  13. Correct the mistakes. Find-It! b
  14. 10 sentences: put the words in order. JMix
  15. Dictation/Text reconstruction. WebRhubarb modified so that only the consonants appear.
  16. Finally, they have to write their opinion of the text in a forum.

It takes them, on average, 2 hours to go through the whole sequence. I tell them not to do it all at one sitting but they don't heed my advice.

You can see the actual set of exercises here:

Sweet Potato Queens: Put the text in orderSweet Potato Queens: Exercises 1 - 12Sweet Potato Queens: Text reconstruction

(For various reasons, the numbering system is different from that above.)

If you are logged in as a guest, you can see the exercises by clicking on "Preview". You can also create your own account on the site.

Cheers,

Glenys

Average of ratings: -