Yet another variant of cloze tool illustrated below. It's part of recompiling and uploading tools I've developed and used in courses. Open for public use at http://substantialmethods.com/subject/view.php?id=6&topic=1.
We should definitely start a thread on understanding pedagogies behind question types.
Here's an illustration of a simple rating application with the Dataform. The entries were added by the instructor so it's not quite peer rating, but it would work in a similar way if the entries were added by the students. Currently it's only one rating criterion per entry. It has just been uploaded, so it remains to see how engaging it can be.
The activity view displays all the entries for rating (and comments).
The 3 top rated philosophers (entries) in descending order are displayed in a block on the course main page:
Not quite Gettier problem b/c such a problem requires two levels of justification and transitivity, and in my example there is no justification for 6x12=74 (confidence doesn't count as justification). It's rather basic logic. True statements can follow validly from false ones.
At the end you seem to agree with what I've been proposing. If understanding is any thing that enables me to answer your questions correctly, then we can safely dismiss the word 'understanding' as unhelpful for measurement and admit that ultimately we measure correct answers. Your responsibility as an instructor is to ensure that the correct answers are available to me, one way or another (or preferably multiple ways), when I prepare for the exam. If the exam requires me to make inferences to answers that were not available, you need to make sure I explicitly practice making such inferences in the same problem domain.
There is no need to underestimate the extent of parrot learning and assume that any parrot could not possibly answer 6x4=24, 3x4=12, 2x3x4=24, 2x12=24, Yes. But then again, by your definitions you may be inclined to call such a display 'understanding' and look for the human ghost in the parrot machine so as to preserve the elusive distinction between parrot learning and "real" learning ("understanding").
Here is the problem Tony. You say
Understanding is a matter of inter-relating correctly all the different pieces of knowledge that you have.
If what counts as correct is not just your personal whim, please give me a list of all the inter-relations I should master or a way to systematically extract them from the knowledge you do make available and let me learn them the way that works for me, even if that way doesn't work for you and even if its mere parroting. Otherwise you only measure my ability to guess what you think is right. And that's wrong.
At any rate, I don't think your multiplication tables example works. If I hesitate over whether I am sure that 7x12 is 84, I may be able to immediately relate this to other things - e.g. that this is 10 more than 74 which I am confident is 6x12, so it has to be right. It's right but I'm wrong and that's because confidence has nothing to do with it.
Have just seen the fallacies site (thanks for sharing!). Indeed pretty funny and nicely designed. Not sure, though, that I agree with the opening statement.
A logical fallacy is usually what has happened when someone is wrong about something.
Surely, we can be usually wrong without making any logical fallacies. We just need to start with being wrong about some things, and then conclude from those things many other wrong things (among occasional right things) by perfectly valid reasoning. Logical fallacies do not entail wrongness just as valid reasoning does not entail correctness.
The course format has not been released yet. It currently works on 2.2.2 and will be adjusted to 2.3 in a month or so. I've been asked to release the current version and plan to do that next week.
What is so sacred about essay writing? (Here's a retrospectively amusing anecdote) Put another way, what do we need it for? If it is just a written expression of critical reasoning, then maybe it's good for not much. As hinted above, valid reasoning is not necessarily more useful than logical fallacies (or more broadly rhetoric) for the good life (a long standing debate from the dawn of rationality). If it serves some practical purpose then often using templates and filling the blanks does the trick. But for the latter not much of assessment is needed.
It seems to me that for self-assessment of essay writing the problem is not so much the self part as the assessment part, which in turn is a problem of the self which manifests itself in our inclination to preserve the right to make subjective human errors just to remind ourselves that we are better than other things.
That said, peer-view (where peer could be one's other self) is important for engagement and engagement is important for learning. I will be adjusting and posting more of my experimental work for essay writing and we could take it from there.