Adapting preg question to the new question engine

Adapting preg question to the new question engine

על ידי Oleg Sychev בתאריך
מספר תגובות: 17

Tim & others - please help.

I have very little time, but people using my question really want to get 2.1 version before September 1,  so I have to upgrade it quickly.  So I hope people don't mind stupidness of some questions I'm asking - I really don't have time.  Just point me to the specific source file (and function) or dev wiki page if answer is already written somewhere. There are way too many TODO on the wiki page about upgrading questions to new engine עצוב

Summary: preg question used to be descendant of shortanswer with regular expression matching. It uses "best fit" answer grading strategy (not "first fit" as shortanswer) and include a "hint" button.

Questions:

  1. is there grading strategy intended for "best fit" type of grading or I should write a my own one?
  2. how should I integrate a hint button functionality in a question type? Is there some special behavour pattern programmed which I could use, or hint button and it's behavour should be programmed into a question class? 
  3. how should I treat different penalties for submitting wrong answer and using a hint?
  4. my question used to save best fit answer in $responses during grading to avoid expensive recalculating it when displaying hint in print_question_formulation_and_controls. What is the correct way to do it now?
ממוצע דרוגים: -
בתגובה ל: Oleg Sychev

Re: Adapting preg question to the new question engine

על ידי Tim Hunt בתאריך
תמונה של Core developers תמונה של Documentation writers תמונה של Particularly helpful Moodlers תמונה של Peer reviewers תמונה של Plugin developers

Have you seen http://docs.moodle.org/dev/Developing_a_Question_Type#Converting_a_Moodle_2.0_question_type

 

1. You may wish to avoid using a grading strategy altogether. Might be worth looking at the numerical qtype.

2. Hints really need a new behaviour (rather than implementing then in specific question types). My inclination would be to remove hints from your question type in 2.1, and then make a new 'Deferred feedback with hints' behaviour later.

3. Don't try to combine hints with adaptive mode, unless you already have a way of doing that which works well, in which case git back to us.

4. Jean-Michel and I discussed this recently, and concluded it was not possible, even though when I started with the question engine, it was intended to be possible. You can get most of the performance gain you need to storing the result of grading in a field of the questoin_definition class. (Write a get_matching_answer method that returns $this->matchinganswer if set, otherwise computes it and sets it.)

ממוצע דרוגים: -
בתגובה ל: Tim Hunt

Re: Adapting preg question to the new question engine

על ידי Oleg Sychev בתאריך

Thanks, Tim.  Hope wem could manage to upgrade question without downgrading it functionality. For now new engine look like it adds more problems for qtype than solves, but that's for now.

  1. I noted that creating grading strategy means defining quite a lot of classes and interfaces. However, I wonder if "best fit" strategy actually may be of interest to you. Current "first fit" could be easily done as subset of it (returning fitness 0 or 1 only), but "best fit" would allow to add next character hinting to shortanswer without much trouble.
  2. I don't think next character hinting (as in Joseph's regex qtype, it's originally Joseph's idea) may be implemented somewhere instead of qtype - it contains too much qtype-specific code. The only thing behavour could do is provide a button and ask a qtype for hint.  People like hinting and rely on it in their teaching strategy. I couldn't just downgrade that, leaving them helpless with their current lessons (decision to upgrade might be an instituional one, and teachers may not influence that). I need to implement it for 2.1 somehow, hack or whatever, and lighting fast (doing other job in day - so I'm afraid I'm too tired for getting new bright ideas when I'm going to work on this). Any hints? Could I just output button in a renderer::formulation_and_controls and where should it lead?
  3. Could you be more specific about a problems that await me? Again people want this functionality for 2.1 and quick, and they have their reasons. I couldn't just say them "sorry, new Moodle isn't suitable for that, so forget it for a while". It is used to be two penalties: one for submitting wrong answer, another (usually greater) for using hint. It should be not worse now. Am I correct that for this to work correctly I have to subclass every behavour reminding adaptive mode and somehow set in the question type that is intend to use those subclasses instead of their parent classes, if they are selected by module?
  4. caching is OK, but are we still looking to get this possible?

On the overall, it is strange that I could do less with new engine, not more. The question should be able to do any reasonable thing it is done in previous versions, isn't it?

ממוצע דרוגים: -
בתגובה ל: Oleg Sychev

Re: Adapting preg question to the new question engine

על ידי Tim Hunt בתאריך
תמונה של Core developers תמונה של Documentation writers תמונה של Particularly helpful Moodlers תמונה של Peer reviewers תמונה של Plugin developers

I just installed the question type to try to understand what the hint feature was doing.

Sadly, I could not work it out. (This may be because I just got back from a bar קריצה). Anyway, could you give me a suggested question to try, and some suggested ways to interact with it, to understand what the hint feature does (and the other features of the question type).

ממוצע דרוגים: -
בתגובה ל: Oleg Sychev

Re: Adapting preg question to the new question engine

על ידי Tim Hunt בתאריך
תמונה של Core developers תמונה של Documentation writers תמונה של Particularly helpful Moodlers תמונה של Peer reviewers תמונה של Plugin developers

Acually, I will say a bit more about some of your questions:

4. Yes, this needs to be solved, but I cannot currently see a simple solution that covers all the cases, so I don't know when this will be solved. Sorry. The extreme example I have in mind when thinking about this is the sojunit question type, which needs to comile and execute Java code to grade the question, or even to validate that the response is valid.

2 & 3. About your comment that the new code is less flexible. One point of the new system is the separation of concerns between the behaviour and the question type. That gives the system as a whole more flexibility because, within reason, you can use any behaviour with any question type. So you get O(m*n) possibilities from O(m+n) code.

So, what you need to try to do is descibe your hint behaviour in a generic way, without reference to the preg question type.

I assume it would include something like "When the hint button is pressed, a hint is computed (by the question type) depending on the partial response the student has given. This hint is then displayed up until the student next submits an answer. As a result of this, the hint-penalty is subracted from the final mark"

Obviously, to implement this, the question definition class will need some extra methods over-and-above the standard ones, something like compute_hint($partialresponse), but then any question type that can implement that method can be used with your hint behaviour.

Once you have worked out worked out the flow chart/state diagram for your desired behaviour, and then created some unit tests, it is quite easy to implment the behaviour class. Even the most complex like interactive and adaptive are quite a small amount of code.

ממוצע דרוגים: -
בתגובה ל: Tim Hunt

Re: Adapting preg question to the new question engine

על ידי Oleg Sychev בתאריך

Hi, Tim

It's a pity you can't get hinting to work. There are quite detailed docs at http://docs.moodle.org/20/en/Preg_question_type and they are linked from Modules and Plugins database, you could get an impression about hinting from there (so I don't repost it there, but you idea was correct). Creating question you should select matching engine that would allow you to activate hinting (DFA engine only for now, thought there are 2 more on the way), then in adaptive mode (in 2.0 terminology) you would get hinting.

The problem introduced by hinting is that you need to find a "nearest" answer to compute a hint even if there are no full matching answer. For regular expression it is done during matching (for shortanwer it could use LCS, which is better), so it's naturally done in grading process to avoid parsing and executing each regex twice (actually I don't think parse regex, build Finite Automata and execute it for every answer is much better than compile Java code once), and it returns two data: matching answer and whether matching is full or partial - that's why existing strategy don't help. 

 

Well, I spent some time looking at the code in questiontype/question classes (and so you should get one more easy issue on the tracker).

My current plan is: add hint button in the renderer::formulation_and_controls() (what link should it point?), and add it to the question::get_expected_data()  (what type of PARAM_xxx is best for the button?). Then I expect it to appear in reponses along with the input field (when pressed of course).  Question class will inherit question_graded_automatically_with_countback - I hope it works? -  and then analyzing response array it could know how many submissions and hints were done, and so calculate appropriate penalty.  Hinting information would be printed in renderer::specific_feedback(). A bit hacky, but did you know something that prevent it from working?

Later, when I would have some more time to get familiar with behavours (I still can't find in wiki a definition what they exactly intended for), I'll may move hinting to the behavour, and maybe create an advanced grading strategy, so that other questions could use it. Actually I consider writing behavour a good solution only if it gets into core, since otherwise PREG become two dependent plugins instead of one, which is bad. But shortanswer could compute hint very easy, so it may be of interest.

ממוצע דרוגים: -
בתגובה ל: Oleg Sychev

Re: Adapting preg question to the new question engine

על ידי Tim Hunt בתאריך
תמונה של Core developers תמונה של Documentation writers תמונה של Particularly helpful Moodlers תמונה של Peer reviewers תמונה של Plugin developers

Sorry, I should have remembered that the question type was well documented. I will try again some time, but not tonight.

Buttons are best as PARAM_BOOL, though in fact if you do that, you will either get true, or the variable won't be set at all.

Something 'a bit hacky' but which actually works is always a good starting point. It is easier to take working code, and refactor it to improve the design than to start something from scratch.

I don't see a big problem with two dependent plugins. Going from 1.9 -> 2.1 I had to split qtype_opaque into qtype_opaque and qbehaviour_opaque (both in my github, not in core). It's no big deal.

I can't think of a very good way to explain behaviours, although the idea is clear in my head. The phrase I used yesterday: "separation of concerns" is an important part of it, but does not really explain.

One way to think about it: In 1.9/2.0 there was the option Adaptive mode, yes/no. That difference was handled by a lot of if statements in the core of the question system, that is, in the question_process_responses function. That was a horrible mess, and a big maintenance headache, so effectively I refactored to use the strategy pattern. Now there are separate classes to represent Adaptive mode yes (qbehaviour_adaptive and adaptivenopenalty) and Adaptive mode no (qbehaviour_deferredfeedback). Also, lurking in the old code was more special cases for handling manually graded questions. That has now been cleanly separated into qbehaviour_manualgraded. A reason for doing this refactoring, in addition to making the code easier to maintain, was because we wanted to introduce new options like qbehaviour_interactive, qbehaviour_deferredcbm, and so on. Trying to add those to the existing mess would not have been possible.

So, that is the summary: Stragegy pattern to abstract one part of what happens when a student attempts a question. Of course, the key thing to understand what the boundaries of that 'one part' is. There is, of course, another part, that is the responsibility of the question type. And everything that is left is the responsibility of the core question engine. The design tried to make the core question engine as simple and dumb as possible. On the other hand, I was able to make the core question engine responsible for loading and saving the attempt data to the database, so the question type and question behaviour don't have to worry about that.

ממוצע דרוגים: -
בתגובה ל: Oleg Sychev

Re: Adapting preg question to the new question engine

על ידי Tim Hunt בתאריך
תמונה של Core developers תמונה של Documentation writers תמונה של Particularly helpful Moodlers תמונה של Peer reviewers תמונה של Plugin developers

OK, I have now read the docs, and sort-of understand what I was doing wrong. To use the example there, I was effectively typing "are blue, " as my answer, and was getting confused becuase it just seemed to be echoing back my input in blue.

ממוצע דרוגים: -
בתגובה ל: Tim Hunt

Re: Adapting preg question to the new question engine

על ידי Oleg Sychev בתאריך

Unfortunately, looking for errors in first characters for regular expressions is quite complicated. I think about issue, but it seems that it's need a whole bunch of research - that'll take time.  So preg looking for error when it could find a valid start.

Well, questions now are:

1. does I need to change onclick action on "hint" button - it is used to be form.action+#questionid ?

2. Would countback question work with all behavours as intended when implemented, or it is another good idea that was dropped while implementing?

Having two plugins for a project isn't am issue for developer directly, but it is an issue with users, they tend to be confused over such things. Give me a way to deliver them one package with could be unzipped in one place, and I'll make as many plugins as you want.

ממוצע דרוגים: -
בתגובה ל: Oleg Sychev

Re: Adapting preg question to the new question engine

על ידי Tim Hunt בתאריך
תמונה של Core developers תמונה של Documentation writers תמונה של Particularly helpful Moodlers תמונה של Peer reviewers תמונה של Plugin developers

1. Look at qbehaviour_renderer::submit_button, in particular you need the js_init_call line. That adds necessary event handlers, which deal with changing form.action.

2. The count-back thing is subtle, and probably best explained by way of example (but feel free to skip this rather long explanation): Suppose you have a matching question with four subquestions, and let us suppose it is marked out of a maximum of 4 marks (we can think of that as one mark for each sub-question). Suppose you are running it with a penalty of 1/3, so effectively the student has three changes to get it right. (I am thinking of interactive behaviour, but adaptive would be similar.) Now suppose a student interacts with the question as follows:

  1. On their first try, the student gets the first two subQs right, and the other two wrong.
  2. On their second try, they are absolutely sure about the first two, so they leave them as is, but try different guesses for the other two parts ... still wrong.
  3. Finally, on their third try, as well as getting the frist two parts right, they also get the third part right, but the 4th part is still wrong.
Now, what is a fair grade to allocate to this attempt?
Our decision was that subQs 1 & 2 should get full marks, because the student got them right first time. Then part 3 should get 1/3 becuase they were right on the third try, and the 4 part, which was wrong, should get 0 marks. So, 2 1/3 should be the total mark.

The question is, how to implement that. Obviously, it only applies to questions with sub-parts. In core Moodle we implemented that for matching, but we have also implemented it in several ou-specific question types. Anyway, questions that want to do this sort of complex scoring need to implement the question_automatically_gradable_with_countback interface, and they work with the interactivecountback behaviour instead of interactive. But, as I say, these things are basically implementation details, which is why qbehaviour_interactivecountback::IS_ARCHETYPAL is false.

 

To answer your last point, they have nearly finished the new modules and plugins database: http://moodle.org/plugins. That does not yet understand dependancies between different plugs, but that could be added.

The idea is, in future, that they would link the Moodle admin pages to that, so Moodle can automatically check that all your plugins are up-to-date. So, handling plugins will only get easier with time.

ממוצע דרוגים:Useful (1)
בתגובה ל: Tim Hunt

Re: Adapting preg question to the new question engine

על ידי Oleg Sychev בתאריך

Thanks, Tim.  

1. The one unclarified bit is a penalties in adaptive mode, there are no adaptivecountback. Still, adaptive mode used to have penalties. What are the problems with them you are mentioned and how the question could intervene in the penalty calculation for adaptive mode? (you may well assume that preg, as shortanswer, doesn't have sub-questions, but is in need of having different penalties for submitting and hinting; having responses of all steps is enought for me to calculate penalty). The question still neeed to behave the same when upgraded, and I expect adaptive quizzes are upgraded to adaptive behavours...

2. In interactive mode, now hint would be equal to the try, and that will be potential problem too, that need to be adressed later. And I lost looking where to set the number of tries for interactive mode in UI. Does it dependent on question penalty or something?

P.S. As a teacher I was quite puzzled by internal help about quiz behavours (that vaguely mentions two of them only), and using link to Moodle Docs on the page woudn't help too. I managed to find good descriptions of behavours in http://docs.moodle.org/dev/Question_Engine_2:Overview but dev docs is hardly an area where typical teacher would look for behavour description for their quizzes. Something must be done with that IMHO.

The descriptions on dev page seems to be out of date too, since I don't find "each attempt build on last" behavour in moodle.

ממוצע דרוגים: -
בתגובה ל: Oleg Sychev

Re: Adapting preg question to the new question engine

על ידי Tim Hunt בתאריך
תמונה של Core developers תמונה של Documentation writers תמונה של Particularly helpful Moodlers תמונה של Peer reviewers תמונה של Plugin developers

1. Interactive is the behaviour the OU uses most, and wanting to get it into standard Moodle (or into Moodle at all!) was one of the drivers for making the new question engine. So, that is were we implemented our more complex scoring rules.

The only reason for making the adaptive behaviour was that it was vital that quizzes made in Moodle <= 2.0 would still work the same way Moodle 2.1. So the goal with the new adaptive behaviour was to work the same way as the old adaptive mode = Yes. That is why there is no adaptivecountback.

I still think you would be better off creating a new behaviour to capture the Hint button behaviour. It can't be that hard. Indeed, try the attached. That is completely untested, but seems conceptually right to me. Given a suitable question type, it would be possible to write some unit tests, and make it work, I am sure.

 

2. In interactive mode the number of tries allowed is the number of hints in the question definition + 1. So, a question with 2 hints gives the students 3 tries.

 

P.S. You are right, the help could be better. That is the help my colleague Phil wrote, with the OU audience in mind. Better suggestions welcome. We definitely need a link to Moodle docs at the end, and a page there that describes all the behaviours in user-friendly langauge.

I also need to update the question engine developer docs. Too much to do, and not enough time.

OK, the current content of the help pop-up is:

How questions behave

Students can interact with the questions in the quiz in various different ways. For example, you may wish the students to enter an answer to each question and then submit the entire quiz, before anything is graded or they get any feedback. That would be 'Deferred feedback' mode. Alternatively, you may wish for students to submit each question as they go along to get immediate feedback, and if they do not get it right immediately, have another try for fewer marks. That would be 'Interactive with multiple tries' mode.

My proposal is ... well insipration is failing me, but I think it is much better if we jsut break it into shorter paragraphs. Also add the two bits at the end:

How questions behave

Students can interact with the questions in the quiz in various ways.

For example, you may wish the students to enter an answer to each question and then submit the entire quiz, before anything is graded or they get any feedback. That would be 'Deferred feedback' mode.

Alternatively, you may wish for students to submit each question as they go along to get immediate feedback, and if they do not get it right immediately, have another try for fewer marks. That would be 'Interactive with multiple tries' mode.

The best way to understand all the different behaviours is to preview a question, you can change the behaviour and see what happens.

More information ... (this last line is a link to MoodleDocs)

If anything, this is a bit too long for a help pop-up, but not excessively so.

ממוצע דרוגים: -
בתגובה ל: Tim Hunt

Re: Adapting preg question to the new question engine

על ידי Oleg Sychev בתאריך

Hmm, you experience make me think. Do you think it would be useful to add something like colored "..." to the answer to indicate that it is correct beginning, but incomplete?

ממוצע דרוגים: -
בתגובה ל: Oleg Sychev

Re: Adapting preg question to the new question engine

על ידי Tim Hunt בתאריך
תמונה של Core developers תמונה של Documentation writers תמונה של Particularly helpful Moodlers תמונה של Peer reviewers תמונה של Plugin developers

I expect that the question type makes perfect sense once you are familiar with it, but I was certainly very confusing coming to it for the first time without reading the instructions. I was impossible to guess what the blue meant, particularly becuase what I had typed was all blue. There was no red strike-through.

So, some change would be good, but I don't have any ideas what change would make it easier to understand.

ממוצע דרוגים: -
בתגובה ל: Oleg Sychev

Re: Adapting preg question to the new question engine

על ידי Jean-Michel Védrine בתאריך

Hello Oleg,

There is something I don't understand. As I remember your preg question the hint button was only displayed in adaptive mode ? So I don't understand why you say :

Am I correct that for this to work correctly I have to subclass every behavour reminding adaptive mode and somehow set in the question type that is intend to use those subclasses instead of their parent classes, if they are selected by module?

For me the only behaviour you have to subclass will be adaptive to define a new " adaptivehint" behaviour because I don't see how hints and penalties could be used with deferred or deferredcbm, or immediate behaviours, even adding hints to interactive seems strange to me given there is no penalty in interactive behaviour !

So, as I see it, your adaptivehint will define a hint button, will receive data when this button is used by the student, and will adjust grade as needed calling your question definition to know the penalty for using hint in the adjusted_fraction method

But maybe I don't understand well your problem and how the preg question is used.

ממוצע דרוגים: -
בתגובה ל: Tim Hunt

Re: Adapting preg question to the new question engine

על ידי Oleg Sychev בתאריך

Tim, you wrote that saving additional information in response is impossible for qtype.

What if I create a hidden field on the question form and add it to the get_expected_data? Could then grade_response() - or other function - safely change this field in last response?

ממוצע דרוגים: -
בתגובה ל: Oleg Sychev

Re: Adapting preg question to the new question engine

על ידי Oleg Sychev בתאריך

Another question: when writing grade_response() should I return state based on fraction only, regardless of penalties for using hints or multiple tries?

ממוצע דרוגים: -
בתגובה ל: Oleg Sychev

Re: Adapting preg question to the new question engine

על ידי Tim Hunt בתאריך
תמונה של Core developers תמונה של Documentation writers תמונה של Particularly helpful Moodlers תמונה של Peer reviewers תמונה של Plugin developers

The state should be based on the student's final response, rather than the fraction. It is easier to explain with an example.

Suppose you have a questions with penalty 33.3%, and the student gets it right on the third try. Then, the overall fraction will be 33.3%, but the state will be question_state::$right.

Regarding your suggested work-around for the inability to save state information in grade_response. Using a hidden form field does not work very well. First, grade_response is called from processattempt.php, while the renderer is only called from attempt.php, so you will have to re-calculate at least twice.

What is required is a way for grade_response (and similar methods) to call $pendingstep->set_qt_var(), and the problem is that those methods don't have access to $pendingstep. I need to think of the safest way to make that possible.

The OU pmatch question type is gets quite slow in certain situations (a few seconds to process one response in one particularly complex case) so this issue is causing me pain as well as you. I do need to find a solution, but I don't want to be rushed into implementing a bad solution that causes problems in the long term.

ממוצע דרוגים: -