## Quiz

### Comments wanted: Self Assesment Activity Using the Question Bank

Comments wanted: Self Assesment Activity Using the Question Bank

Hi I'm Jayesh and I am going to work on Project "Self Assesment Activity Using the Question Bank".

I have been selected to work on this project in this year's Google Summer of Code.

To start with the idea of the project is to implement a practice module that will use existing Question Bank API and help to develop a module that will help student's to analyze themselves.

I have created some mock-ups and all other technical details are listed on the wiki page.

http://docs.moodle.org/dev/Self-assessment_activity_using_the_question_bank

Thanks

Average of ratings:Useful (7)
Re: Comments wanted: Self Assesment Activity Using the Question Bank

I like the idea, this is something from which my students could benefit.

My only hang up is that in my question database some questions are linked to each other, they contain synchronised variables. If the question picker randomly chooses one part of them it will actually be impossible for the student to answer it correctly.

This is probably a problem with the way I organise my question database though.

Average of ratings:Useful (1)
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Hi Chris,

Thanks for the feedback!

Yeah dependency is one issue that I will need to look at.

There are lot other things that have to be looked at. Basically since this is first time module is being implemented I wanted to know from teacher's and student's point of view that is there any functionality that we can add to this in order to make it more useful for students.

There might be certain things that you will want to get added to teacher's UI too. For now we have mock-ups for Student UI ready which are on wiki page. Teacher UI mock-ups will be ready soon and updated on same wiki page.

Suggestions are welcome so that we can add them to basic module.

Thanks,

Cheers.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Piggybacking a bit on Chris' comments:

How will questions be added to a Self Assessment?  I don't see any specifics on this and I can imagine a few options.

2) All questions in a category can be added.
3) The category itself can be added.

What is most appropriate is going to depend on how people have organized their question banks.

I think it is less important to see #2 as opposed to #1 and #3.  In particular, having similar functionality as in the quiz by using #3 would be very helpful.  When I train the teachers at my school on quizzes and questions, I spend a good bit of time talking about building a hierarchy in the question bank to facilitate random questions.  Applying some of the same logic here would be useful.

For that reason I'd almost prefer not to have #2 since there is no analogous structure in quiz.  I would imagine that it could confuse people who add questions later to a category and are used to quiz.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Hi Joshua,

First of all the thanks for the feedback!

Questions will be added in same way we used to add in quiz but they are going to be used in slightly different manner in practice module.

We can take a small example to illustrate this.

1. For an example you are having a question bank for a course let say PHP.
2. What teachers do is they organize their question bank according to categories.
3. Teachers will later on create a practice module (as we do in quiz) and add questions to practice that teacher thinks are useful for students.Questions will be added in same way we used to enter questions in quiz.
4. Now you might be having different categories inside practice. Let's say Topic-1,Topic-2 are name of different categories inside it.
5. After this any student who wants to Practice questions from PHP course he will log in to PHP course, will see a practice that was created by teacher in third step and a screen having 2 options will be displayed to him/her.

What we have planned to do is we will display all the categories from a particular course to a student.As you can see in mock-up there is a option for a student to select one or more categories.So if a student wants to practice questions for more than categories he just needs to select those categories and questions will be displayed randomly from both the caetegories.

Now the question arises how many questions will be selected from each of them. Since this is first time Practice module is being implemented we are working on formula on how randomly should a question should be displayed to student and if student selects more than one category then how many questions from each category should be shown to students.This formula is being developed. I will let you know as soon as we are done with it.

Cheers.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Hello Chris,

For that kind of questions maybe you could look at some question type that permit multi part questions like Stack or formulas, that way each questions could be independent and considered as a whole which would simplify a lot random choice of questions in an activity like self assessment ?

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Welcome to our GSOC 2013 student Jayesh Anandani,

I think it is very importing for him to get input from the Moodle Community on this self assessment activity project.

Very often people say "I can't contribute back to Moodle because I know nothing about programming"

But you don't need to be a developer to help Jayesh: just read the specifications he wrote, look at the dialogs mockups and ask yourself: what should the teacher/student see during a self assessment session ? what settings, feature is missing ?

Maybe some suggestions will have to be postponed for the first version, to keep the project doable during the summer but they will be recorded, and this is important for the future.

Doing so you will help this new activity to be really interesting for students and teachers.

IMHO this is how open source software works.

Average of ratings:Useful (3)
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Hello all,

All the inputs received till now have been very helpful for the project. It has brought out many prospects that may be revised and added to module. Some other inputs like that can help us making a module better.

Any comments that you have regarding module from a teacher or a student point view are welcome.

Cheers.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Jayesh,

Two thoughts:

1. the test banks available should be designated by the teacher role as available for self-assessment or not

2. make a quick start button that defaults perhaps to a default setting configured by the teacher role (so that students dont' have to configure their own self-assessment every time).

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Joseph "the test banks available should be designated by the teacher role as available for self-assessment or not"

+1 (Joseph too)

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Hello Joseph and Joseph,

I must admit I don't understand (in fact i'm lost and I think Jayesh is too ).

In Jayesh's proposal when the teacher create (or edit) the practice activity he choose the question bank categories he want the questions to be drawn from in this activity.

Can you explain what you want ?

But maybe it will be best to wait that Jayesh has posted the mockups for the teacher UI to understand his proposal and comment after that.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

I also didn't got those 2 points. It will be very helpful of you if you can elaborate those 2 points.

I will be posting mock-ups for Teacher UI by tonight.

Cheers.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Yes that is what I want.  Sorry for being unclear.

As for the 2nd point, I would not want it a burden for students to create their own self-assessment.  Too often Moodle over engineers a feature, I think a quick start self-assessment option would be great (so the student can jump right in rather than consider the three options I see in the activity that students are forced to configure/think about before getting started: number of questions, what topics, timer, etc.).  I was thinking there should be a default set by the teacher so that the student can simply get started with the self-assessment.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Hello Joseph,

I think we are clear with first part.

Moving towards second part as far as this module is concerned it is a self assesment module and it is aimed at student't assesment. This module will aim at student's assesment in which student will analyze themselves. I think by  imposing restricition from teacher's side is not a good option.This will be something we don't want to happen.

Student as per their confidence can practice the way in which they want to practice which is the main aim of our module. This practice sessions can be monitored by teachers. But it is the student that should really select the option the way he wants it. Some want it to be normal , some may want to frame it within a time limit and some by goals.

This is my opinion. I think this is the best option that student can have.

We can  wait and take look at what other's have to say about this.There will be always be difference in opinions so no offence. ;)

Cheers.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

I think what Joseph Thibault meant and I agreed to is that, amongst the various question categories within a course's question bank, a teacher should have a way to designate which categories will be usable by the students for self assessment and which ones should not be used (e.g. because their questions will be used for exams). That's all.

Joseph

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Hello Joseph Rézeau,

Thanks for the kind feedback!

I disagreed with the second point.

The first point that you elaborated will be working in same way as you are suggesting. Student will be able to see only those categories which teacher wants to show them.

Cheers.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

I've been using versions of the quiz for self-assessment for a couple of years so this is a nice development. I've been reading the comments and agree with the ideas that are circulating.

In addition, I would like to add that from my perspective, I would like to have the option to be able to block access to such a feature while students are writing a graded quiz.  Here's some background... we give students practice self-assessments and then require them to write a quiz for course marks from a secure proctored location but we have a few students who are accessing the self assessments while writing the quiz despite the guidelines we give them (i.e. closed book, exam conditions etc.).  While we use other software to block access to other websites, we can't currently block access to other items within our Moodle site.

For example could access be blocked/denied to specific IP addresses at certain times ? i.e. the opposite of allowing access only to specific IP addresses.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Hello Ian,

Thanks for the kind feedback!

Yeah I agree with you! As far as I have seen moodle there is an option for hiding certain activities. Since the main practice is created by teacher,and later on each student uses the practice in his/her own way to create sessions as mock-up suggests. Teacher can easily hide parctice so that students won't be able to see that practice during quiz.

Cheers.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

I'm not sure I agree with "teacher can easily hide...."

In my situation I have 700 students writing a graded quiz across several days from the controlled, proctored environment. I don't want to have to turn an activity off to ALL users during the day while some students are writing the quiz because this prevents some students from using the self-assessments. I also don't want to have to be remembering to hide / reveal the self assessments at the start / end of each day (I've been there and done that with a previous LMS...)

A better solution in my case / opinion would be to be able to block / deny access based on parameters such IP address and time / date etc.  This way students could still practice from computers outside of the controlled environment while other students are writing the graded quiz.

Hope that makes sense.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Ah yes. It will create a problem in that case. I didn't think of such a case.

Can you provide me some example that deal with IP related thing? I am not too sure on how to go on with this. Definetly it is a good way. There must be some other way to solve this.

Can we provide an interface to teacher listing them all  the students enrolled in a particular course out of which teacher can select student's whom he/she wants practice to be shown. For rest of them practice will be deactivated?

I will have to consider it with my mentors. It is a good thing to be looked upon. Any other inputs to resolve this will also be great.

Cheers.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

In terms of the IP address issues, the existing Moodle quiz module allows an instructor to have the ability to allow access based on IP addresses (i.e. to restrict access to a particular set of addresses). It's a standard feature via the quiz settings page in the same manner as time can be controlled. I was assuming (hoping) that there was a way to do the exact opposite i.e. deny access.  If this could be tied to time as well it could be quite powerful I think. If it could be flexible to allow specific times on specific days (rather than from a specific time / date to another specific time / date) so much the better (may be based on a calendar type interface so that time range on a day to day basis can be easily set (?))

As for the roster idea, I think that with a large class that would be impractical and too prone to errors that require a manual fix. The settings for a particular student would also be time specific anyway.

Average of ratings:Useful (1)
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Yes. I saw that. I think idea  given by you will be easier to implement and will be of great help.

+1 for the idea and suggestion.

Cheers.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Can I start by thanking you all for giving Jayesh such excellent feedback while I was away on holiday.

(In case you were interested, I was staying here, and I walked up both the hills/mountains that you can see in the background during my week there. I recommend visiting the Scottish highlands if you get the chance, despite the midges.)

OK, so there is enthusiasm for the general idea of this module, which is nice to see, but there are some details about how it should work that we still need to sort out.

I am going to list what I think are the outstanding questions here (and in the other thread). In most cases, I will give my opinion on the answers in separate posts that I will make later.

(Note, I read this thread some days ago, and I have been thinking about these questions ever since, but it has taken me a while to find time to write this post, and I have run out of time again, so I will have to continue with the promised follow-up posts tomorrow, sorry again Jayesh. In the mean time, everyone else is welcome to continue the discussion.)

1. Simple point, but Jean-Michel asked (I think only in an email) what should we call this activity? Well, Jayesh has started doing the work with the code called mod/qpractice, which I think is an OK name. At the moment, that is appearing as 'Question practice' in the user interface. Is that OK, or can someone think of a better name.

2. Who should be in charge, student or teacher? I think the focus of this activity should be putting the student in charge of their own learning as much as possible. On the other hand, I completely take the point (made by Joshua and others) that it is the teachers job to guide the student as they take charge of their own learning, and we should give teachers the tools they need to do that.

The next few points are inter-related

3. How will teachers indicate which questions should be used in this activity?

4. After the student has finished each question, how does the next question get selected?

5. Should starting a practice session be a one-click operation for the student, or should they have to think about and select the options they want?

6. Which question behaviour should be available for use with this activity? (which links back to question 2.)

Now some other unrelated issues

7. What reports should we provide for teachers? of should we guarantee students privacy from teachers snooping, as Tony advocates?

I final point, which Jean-Michel has already made above. There are more good ideas in this thread than we could possible implement in one Google Summer of Code project. The goal of the project is to make a basic version of the activity that is a useful learning activity. As we know from Moodle and the quiz module. Once you have some working code, then you can add more features later.

Given that, I will dismiss some issues that have been raised in this thread (for now at least).

• Dependencies between questions / linked questions / shared variables. Sorry. At least for now, this activity will only work with questions that can be used independently. (See also Jean-Michel's response about using formulas / STACK / Cloze as a work-around.)
• Students creating their own questions for use in this activity. I think this is out-of-scope, but I will note that you can override the roles to let students edit questions if you like, so perhaps it will be possible with no extra code.
• IP address restrictions. I can see why Ian wants this, but I am afraid I think it is out-of-scope for the first version. It is the sort of thing that is quite easy to add later.

Some things that are probably out of scope for version 1, but have a better chance of getting done sooner rather than later (but no promises):

• Integration with Mozilla open badges, or something, to recognise achievements by a student.

Average of ratings:Useful (2)
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Tim, glad you got some vacation.  It is well deserved.

My thoughts:

1.  Every core activity module seems to have a single word name.  I'd suggest just "Practice."

2+5+6.  At the end of the day, teachers are always going to be in charge based on what categories they allow to be used.  If I want to make a Practice where only one category is allowed, then that's what I'll do.  With that being said, having some flexibility for both extremes is helpful.  I would imagine a system where teachers could select defaults for how a session behaves.  Those settings could be locked or unlocked for students to change.  So this could be a one click operation for students who wanted to start with the teacher suggested defaults.  Or, if its allowed, then students could have more flexibility.  For simplicity's sake, I'd suggest that students be allowed to select whatever categories they want from the allowed list.

6.  I think every question behavior makes sense here except for deferred.  So I'd vote for including them all, including CBM.  However, teachers should be able to eliminate options from the list.  For example, I don't use Interactive yet and I haven't written hints for any of my questions yet.  If a student selected that then they would be very confused when no hints showed up.

3.  I think adding questions make sense only at a category level.

7.  I am in favor of using a similar system found in Questionnaire.  When you create a Questionnaire, you can chose whether it is anonymous or not.  If you select anonymous, once someone has completed a questionnaire, you cannot go back and change that setting.  This setting could/should be advertised to students when they complete a Practice session.  If you implement a system like this, then its just a matter of what reports get displayed in the different contexts.

With that said, I was hoping for some reports similar to the quiz reports.   I'd like to be able to see everything about each student's practice sessions, with questions they saw and response histories.  I'd like to see an overview for a particular student that includes how many practice sessions, what categories were used for each, and a "score" for how well they did.  It would also be useful to have item analysis pages similar to the quiz reports as well.

Badges/Recognition:  Rather than reinventing the wheel, how about just using activity completion?  Just make lots of options for what to require for completion (see the Forum completion options) to give us some flexibility.   Since Badges can be tied to completion then they'll just work.

Thanks to all who are participating in this discussion and particularly to Jayesh for working on this.  I'm quite excited about this new module and I'm really looking forward to using it.

Average of ratings:Useful (1)
Re: Comments wanted: Self Assesment Activity Using the Question Bank

1. At least for the moment, this activity module will be an add-on. I am literal minded. I like Question practice.

Badges: Good point. Linking into activity completion is a reasonable approach. Except that, an activity can only be completed once. We might want several badges from one practice activity (e.g. accuracy and perseverance).

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

OK, so to start making the promised follow-up posts.

2. & 6. Who should be in control, and a good example of that is, what question behaviours should we allow, and who chooses - that is a good specific example of the general question 2.

So, when a student starts a new practice session, Moodle needs to know a number of options for how that session will work. For example we might have three option:

Question behaviour: Immediate feedback, Immediate feedback with CBM, etc.
Stopping criterion: Open ended, After 10 questions, etc.
Question selection method: Chosen at random from topic 1, etc.

For each of these options, there are a range of possibilities, and let us use behaviour as an example.

There are a range of behaviours installed in Moodle, by default:

1. deferred feedback
2. immediate feedback
3. deferred feedback with CBM
4. immediate feedback with CBM
5. interactive with multiple tries

Now straight away we can eliminate 1, 3 and 8. For practice, the students need to be able to submit their response and get a grade while interacting with the question. So, that leave 5 possible behaviours that we can use with question practice.

Ultimately, when a practice session is started, a single possible option must be chosen. Starting a practice session is done by the student, so they get to make the final decision.

How much control should we give teachers? Well, my suggestion is that we let them decide which options are available to students. So, on the form where a teacher sets the settings for the question practice, we will have

Available question behaviours: [ ] adaptive
[x] immediate feedback
[ ] immediate feedback with CBM
[x] interactive with multiple tries

Then, only the options that the teacher things are relevent are presented to the student.

Now, a teacher may only allow one possible option. That is both bad (the goal of this activity is to put students in charge) but also good (the student does not have to make a decision about this themselves, they can just concentrate on answering questions.

I think that this approach that works for behaviours can be generalised to all the options: for each option there are a range of possible settings, and the teacher gets to select which of those possibilities are available to students.

Now, as we have said, it is a big advantage if the student can just click once to start a practice session, without having to set any options. (It also has disadvantages.) One way you can achieve that result is for the teacher to fix all options, so the student has no choice. A better approach, as already suggested by Joseph Thibault it to have a system of presets. The settings to start a practice settings are just a chunk of date: Option1 => value1, Option2 => values2, etc. We can, of course store data like that in the Moodle database. What this means is that we can let the teacher (or student) create preset or default practice sessions, and we can let students start one of those practice sessions with a single click of the mouse (without removing their freedom to create completely custom sessions).

I think that is something that should be added in version 2.0 of the practice module. It is not necessary for the first version, and is clearly something that can be added later.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

I thought you were a bit quiet Tim, then my notifications binged for you all of a sudden.

I also hope you are rested and refreshed.

-Derek

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

So, the next topic to talk about is 3. How will teachers indicate which questions should be used in this activity?

Jayesh, Jean-Michel and I had a Google hang-out where we talked about this. I had a suggestion that I thought would be controversial, but they both thought it seemed sensible. Here is a mock-up of the idea:

The idea is that the question practice activity will use all the questions from the "Question categories for 'Question practice: Example question practice'" area, and no others.

This is a good idea because

• it is the simplest possible solution, so it is easy for users to understand;
• it makes it very clear which questions are available for practice, and which are not, so you won't accidentally end up with students able to practice questions from the final exam;
• it is also the easiest thing to code, which means fewer bugs;
• and, for anyone who knows about all the bugs relating to backing up and restoring quizzes or courses that use shared questions, you know we really don't want to go there again.

This is idea problematic because

• if you want to re-use existing questions from elsewhere in Moodle, you have to copy them into the question practice area;
• currently the question bank user-interface does not provide a very good way to let you do that.

However, we all know that the question bank user-interface needs to be improved, so I think that leads to more benefits:

• If the Question practice activity requires the questoin bank to work well, then that is more pressure on us to finally fix the question bank.
• And, perhaps more importantly, the Question practice activity will be another good example (beside the quiz) of what we require from the question bank, so it will help us to work out how to fix the question bank.

(Given the current state of the question bank UI, copying categories of questions from elsewhere into a Question practice is most easily done using Moodle XML import and export.)

Does this idea make sense? Can you point out any insurmountable problems, before it is too late? Thanks.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

That makes sense from a time perspective with this being a Summer of Code project.

With that said, I hope over the long term that this can be expanded out.  I don't like the idea of duplicating questions.  My course backups with all the questions (1500ish) already take forever.  My main concern revolves around making both a targeted practice and a general cumulative practice.  There is likely to be heavy duplication between these two categories.  A rough estimate is that my total number of questions would grow to 2100ish.

This does run counter to my general advice to new teachers to be careful to setup their categories in the question bank and avoid the use of the Activity level categories.   I give this advice for maximum flexibility.

Is the desire to simplify this to just limit it to one category for easy coding?  Would it be nearly as simple from a code perspective to mandate the same "everything in the category" and "only one category" option but in the main question bank?  Or not?  I've already professed my general ignorance on coding...

I'd prefer the second option to reduce the duplication of questions.

However, I do realize the time constraints.  I'm not opposed to your proposal Tim, its just not my favorite option.  It does make a good bit of sense to simplify things.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

The slow question backups is caused by MDL-29439 (and MDL-39725MDL-39726). Someone braver than me is trying to solve that, and making good progress.

One of the reasons question back up ended up so slow is because quizzes can use questions from anywhere, and that makes backup/restore hard to get right. In Moodle 2.0 they were happy enough just to get it working, performance was a secondary concern.

We don't want to limit it to just one category. We want to limit it to all the categories in one context.

My preference (long term, for the quiz as well as for question practice) is to require all the questions used be quiz (or question practice, or lesson, ...) to be copied into that specific area of the question bank. Then to provide really good tools within the question bank so that it is trivial to keep the copy of the questions used in the quiz in synch with your master copy in the main question bank. Sadly I don't know exactly how that should work. As I said, I want to build qpractice this way, and learn from that more about how the question bank should work.

I can confirm that your aspiration for how it should work: questions created and beautifully organised in a shared area of the question bank, and then used in quizzes and other activities, is exactly how I think it should appear to teachers. However, we need to change how things work on the inside to really make that work reliably.

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

Let me just make sure that I understand what you're saying here.  We're talking about an activity level context, right?  It may contain categories within it.  All questions in the activity level context can be used in the practice session.  Will students be able to select activities in the activity level context to practice on in a more targeted way?  Or will it just mix them all up together?

Average of ratings: -
Re: Comments wanted: Self Assesment Activity Using the Question Bank

It really boils down to: At any given moment, how to we select the next question to present to the student? How, and how much, do we let teachers and students control that?

And we don't have an answer to that yet. We have lots of ideas, but one task for this week is to discuss the answer (while Jayesh goes on writing the bits of code we are clear on). Fortunately, this bit of code is very self-contained. Basically we have a function

$questionid = get_next_questionid($story_so_far, \$options);

and it is quite easy to experiment with different implementations of that function without changing anything else.

Average of ratings: -