pattern match syntax testing - out of moodle?

pattern match syntax testing - out of moodle?

by Adriano Ceccarelli -
Number of replies: 8

Hi All,

I have been trying to grasp the what's and how's of this type of question (which I like very much) and at the end of the various attempts to generate rules with a sensible syntax the only way of testing is to construct a question with the rules and then run a new attempt of the quiz for each possible statement one can think of.... A very tedious procedure as 5-6 clicks each time are involved. 

Has anyone found a way of testing rapidly a number of different sentences in a single run with the same rule? it would be great if one could type a number of different possible answers and have them all scanned at once with the same set of rules incorporated in the question.

Thanks

Regards

Adriano

Average of ratings: -
In reply to Adriano Ceccarelli

Re: pattern match syntax testing - out of moodle?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

If you have a recent version if the pattern-match question type, you should find it has a very useful tool built in.

Preview any pattern-match question, and there will be a "Test this question" link in the top right. When you click that, you get to a screen where you have import/create a list of possible responses, and whether you think they should be graded right or wrong, and it will use the grading rules you have set up, and hilight any where the automated grading does not match what you said it should be.

If you keep that screen open in one browser tab, and edit the question in a second brower tab, that is quite an effective way to develop questions.





Average of ratings: Useful (4)
In reply to Tim Hunt

Re: pattern match syntax testing - out of moodle?

by Adriano Ceccarelli -

Hello Tim,

many thanks for your reply. Very helpful tool indeed, however I must be grossly missing something....

I made a set of rules but apparently never got hem to work according to the test question tool. Just before giving up I went to the quiz interface and tried the same answers I uploaded with the csv file one at a time with the same rules I tested and everything worked fine, resulting in the scores I was aiming at.

There are a few things not clear to me in that screen: why don't the human marks correspond to the computed marks if running the answers in quiz one at a time I got the correct (i.e. wanted) scores? Why are some answers marked in red some in blue and some stay plain text? Why does this highlighting change by clicking on the human mark value? there is more doubts actually, so I might better ask: please, where is the help for that screen, or at least a legend or some documentation, so that I can first learn how to use it and then try?

However, at least now I know I can use pmatch syntax .... smile)

Many thanks

Adriano

Average of ratings: -
In reply to Adriano Ceccarelli

Re: pattern match syntax testing - out of moodle?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

There are two types of mistake that the automated grading can make:

  • It can take a response that you think should be graded correct, and grade it wrong.
  • It can take a response that you think should be graded wrong, and grade it correct.

Those are called false-positive and false-negative errors, and they get highlighted in red and blue. So, that is why they are highlighted - they are the ones you need to worry about. And, if you change the 'Human mark' which is how you say whether you think that response should be correct or not, then that affects whether the result of automated grading is considered a false-positive or not.

It would be helpful if you could give a minimal example that lets us replicate the problem. Ideally just one or two grading rules that are set up in the question, and one example response that it highlighted incorrectly for that question.

There is one worry I have about this. The question testing tool was initially designed for the simplest case where all the rules lead to a mark of 100%, even though in reality it is sometimes helpful to have have some rules to detect clearly wrong answer and give them 0%. I though that had all been fixed, but it might explain what you are seeing?

Average of ratings: -
In reply to Tim Hunt

Re: pattern match syntax testing - out of moodle?

by Adriano Ceccarelli -

Hello, 

what I was actually surprised of was that despite having them graded in the wrong why while running the 'test this question' tool I got exactly what I expected when entering each assertion separately in 5 different attempts of the same quiz. I tried again deleting all answers, clearing the browser cache  quitting the browser and starting it over from new but I had exactly the same behaviour. I would be grateful if you could instruct me as to where to send the examples. I would send the csv file, a text file with the starting question and the rules and a screenshot documenting the type of feedback I get from it. 

The intermediate grading of this particular question are only partial statements where only 2 or 1 out of the three elements necessary to make it complete, and the rules are set in decreasing order. I followed (or I intended to..) the logic of what it states in the help whereby an answer is matched to each rule until it finds the match.

Many thanks for your help

Adriano

Average of ratings: -
In reply to Adriano Ceccarelli

Re: pattern match syntax testing - out of moodle?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

The grading should be the same when answering the question as in the testing too. If not, that is a bug we definitely want to fix.

If you don't want to attach the examples here (or make an issue at https://github.com/moodleou/moodle-qtype_pmatchjme/issues and attach them there), the you can email them to me. My email address is in my profile.


Average of ratings: -
In reply to Tim Hunt

Re: pattern match syntax testing - out of moodle?

by Adriano Ceccarelli -

Hello Tim,

I  have attached a zip file with the three documents I mentioned in my last post. I will be looking forward to hear the verdict .... smile Please note that I have the rules also commented in my own language (//...), while the FB=... is  the text included in the feedback 

Many thanks for your help

Adriano

Average of ratings: -
In reply to Adriano Ceccarelli

Re: pattern match syntax testing - out of moodle?

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

It would have been much easier if you had exported the question in Moodle XML format, rather than making me re-type it. Also, just copying and pasting from your text file did not work. I had to correct several typos in the syntax before the question would save (e.g. missing closing ) and match-all where it should be match_all.)

So, I am not sure I set it up right, but for me, none of your sample answers match any of the rules.

Also, at the moment, the question testing tool can only handle responses that are completely right or completely wrong (0% or 100%), so the 50% and 20% answers definitely won't work.

Average of ratings: -
In reply to Tim Hunt

Re: pattern match syntax testing - out of moodle?

by Adriano Ceccarelli -

I apologize for having caused this hassle. I wish I knew there was a moodle XML format, and that it was possibile to export  the samples in that format. When I described what I would have attached I did it exactly on purpose to check if the file formats I was providing were suited for the needs. Obviously some misunderstanding occurred in between posts.

Similarly, I wasn't aware of the 0-100% threshold of the question testing tool, is it mentioned somewhere in the documentation of the tool?  That  obviously explains why I am not getting the intermediate scores proportionally represented in the computed scores column. 

As for the overall negative result in matching rules and answers I provided, I cannot really explain why, when tried individually they worked.

I attach a .zip file with the screenshots of the results from each attempt, in case you wanted to compare. Files are .gif or it would have been too large.

However, many thanks anyway for the time spent trying to fix my problem.

Regards

Adriano

Average of ratings: -