Sequential quiz - WCAG - legal

Sequential quiz - WCAG - legal

de Nilanjan Basu -
Número de respuestas: 21

Hello everyone,

I have recently received a guide from my university that indicates that using the sequential option in Moodle quizzes will run afoul of the Web accessibility guidelines (WCAG available at https://www.w3.org/TR/WCAG20/) and are therefore not allowed in the future. I understand that this is not just my university but has been happening in many other universities as well. Has anyone else experience this issue and have any insights / experiences to share on how this may play out? Thank you in advance for your input.

Promedio de valoraciones:Useful (1)
En respuesta a Nilanjan Basu

Re: Sequential quiz - WCAG - legal

de Rick Jerz -
Imagen de Particularly helpful Moodlers Imagen de Testers

I am not an expert on this, but I do note that the "G" in "WCAG" stands for "guidelines."  If it were a "legal" issue, the acronym should become WCAL.

But I know what you are going through.

En respuesta a Nilanjan Basu

Re: Sequential quiz - WCAG - legal

de Tim Hunt -
Imagen de Core developers Imagen de Documentation writers Imagen de Particularly helpful Moodlers Imagen de Peer reviewers Imagen de Plugin developers
My main concern about the sequential option is that it is horrible pedagogy which reduces the opportunities for students to demonstrate good metacognitive skills. You don't have to use it.
En respuesta a Tim Hunt

Re: Sequential quiz - WCAG - legal

de Germán Valero -
Imagen de Documentation writers Imagen de Particularly helpful Moodlers Imagen de Plugin developers Imagen de Testers Imagen de Translators
During this COVID epidemic, my (veterinary pathology) undergraduate students seem to have developed a great ability to communicate during online quizzes and share information.

This has caused an unprecedented increase in grades in quizzes, which clearly do nor correlate with their knowledge of the tested subjects.

I have also noticed a large increase in similarly-worded incorrect answers to essay questions.

This has forced me to try sequential navigation of quizzes with random questions.

The widespread complaints, from mostly students who had low grades in proctored quizzes before the COVID and now have much higher grades in unproctored quizzes, supports the notion that they were indeed cheating.

My University can not afford proctored online testing.

I have added many questions to enlarge the question bank, but writing good questions for medical subjects is a slow task, and after writing them, I thought that most of these might be best used in formative, rather than summative assessment. So, I made them many practice formative quizzes with multiple tries, but I still need to make summative assessments for final term exams.
En respuesta a Germán Valero

Re: Sequential quiz - WCAG - legal

de Rick Jerz -
Imagen de Particularly helpful Moodlers Imagen de Testers
Germán, have you considered making randomized exams? Basically, each student might have different questions.
En respuesta a Rick Jerz

Re: Sequential quiz - WCAG - legal

de Germán Valero -
Imagen de Documentation writers Imagen de Particularly helpful Moodlers Imagen de Plugin developers Imagen de Testers Imagen de Translators
@Rick,
Thanks for replying.
That was my first approach.

The exams do use randomly selected questions, but the total number of questions in the bank is barely 3x the number of students in the class. There are only 2-3 available questions for each category being assessed.

One issue is that all students must be asked a similar number of questions from the same syllabus subject. Otherwise,some students might complain that their exam was unfair, and somebody else got an easier exam than their's.

I did not want to use the questions I already used for formative assessments, as the students have the habit of collecting them and I must assume they are available for a quick look and cheat.
En respuesta a Germán Valero

Re: Sequential quiz - WCAG - legal

de Rick Jerz -
Imagen de Particularly helpful Moodlers Imagen de Testers
Yes, I understand. One approach to adding questions is to duplicate a question, then modify it so that it is similar in concept, yet different wording. With this approach, questions will be similar. Let the students complain. If an administrator complains, the show them all the questions in a category to prove that they are similar. This is 2020, not 1958!

In my own case, the textbook author provides a lot of questions.

Also, if you can "shuffle" the questions and "shuffle" MC answers, exams will be different even if they happen to have the same questions. You are using "timed" exams, right?
En respuesta a Rick Jerz

Re: Sequential quiz - WCAG - legal

de Germán Valero -
Imagen de Documentation writers Imagen de Particularly helpful Moodlers Imagen de Plugin developers Imagen de Testers Imagen de Translators
Thanks for your answer Rick.

I am using timed exams with sequential navigation, and I have decreased the usual (1 minute default average) time per question to 45 seconds per question. I do not think it would be fair to decrease the allowed time any further, as already 20% of all test are automaticlly submitted when time runs out.

Actually, I am one of the editors for the mexican textbook for this subject (http://www.libros.unam.mx/patologia-general-veterinaria-9786070292804-ebook.html ). As we have provided free access PDF files of all the textbook chapters to all students during this COVID crisis, the students can search the textbook pretty fast.

I am trying to make questions that require that the student know the basic concepts, and then apply this knowledge in a real-life case story with gross and microscopic images. But I am pretty sure the students are communicating amongst themselves in real time while they are answering the exam.

One strategy I have used is to include an easy to remember gross image in the question (e.g. a smiling monkey) , and then duplicate the question and change the description of the clinical signs and lesions, and ask a question that (obviously) should have a different answer from the original question. This was a very good technique initially (many student failures in the first exams so devised) , but the students have realized about it and have seemingly improved the way they communicate about the questions.

The problem, as I see it, is that too many students currently think it is more useful for them to devise ways to cheat in exams than it is to dedicate time and effort to study hard subjects.

Because of national privacy laws, we are not allowed to make oral exams by ZOOM. That would be a very straightforward way to deal with some students that seem to cheat on exams.
En respuesta a Germán Valero

Re: Sequential quiz - WCAG - legal

de Joseph Rézeau -
Imagen de Core developers Imagen de Particularly helpful Moodlers Imagen de Plugin developers Imagen de Testers Imagen de Translators

Hi Germán!

"The problem, as I see it, is that too many students currently think it is more useful for them to devise ways to cheat in exams than it is to dedicate time and effort to study hard subjects."

An eternal problem. Your analysis could be applied to many other situations of human life. Thieves, drug-dealers and (some) politicians think it's more useful for them to steal, cheat, etc. than to dedicate time and effort to live an honest life.

In all these domains the answer is the same: education, education, education. As a wise educator yourself, Germán, I'm sure you will know this. As a former educator I know how education can often feel like a Sisyphean task.Guiño

Joseph

En respuesta a Germán Valero

Re: Sequential quiz - WCAG - legal

de Ewout ter Haar -

"and have seemingly improved the way they communicate about the questions."

So they are learning, improving a vital skill in 21th century society: collective problem solving. Maybe you think, well, they are not learning the skills I want them to demonstrate. But the obvious way forward is to allow communication and give harder problems to solve.

My colleagues are asking how to do group work in Quiz.  The expectation is things would work like Assignment, but that's not possible now off course. I'm looking at the Group quiz report plugin. Does anyone have experience with it?

En respuesta a Ewout ter Haar

Re: Sequential quiz - WCAG - legal

de Nilanjan Basu -
This is a two-edged sword. I have in the past taught upper level research classes and the students used these tools in the best possible ways to seamlessly transmit any insights from office hours to the entire group.
And of course now they are using Discord (a multiplayer gaming oriented version of Slack/Teams that can of course be repurposed as any kind of group work task) as a way to collaborate on exams. So yes, many of them are very well prepared for the future workplaces that require digital coordination of teamwork. We just need to be able to make sure it is used in the right places and not the wrong ones.
En respuesta a Germán Valero

Re: Sequential quiz - WCAG - legal

de Rick Jerz -
Imagen de Particularly helpful Moodlers Imagen de Testers
Relative to "oral exams," why not have student record their responses (audio) and then upload these? This would avoid using Zoom. Of course, you have to have space on your server or some other way of handling big sizes.

In one of my courses, every student needs to give a presentation where they show that they have applied what they have learned. I make them do a voice-over-PowerPoint presentation, using the Pecha Kucha technique, which says the presentation must be 6:40 minutes long. These PowerPoints end up being between 10MB to 30MB in size. So maybe you could use a technique like this. (My technique gets slightly more involved because I like to post videos, so there is another step that I use to convert the PowerPoints into videos.) Here are my project guidelines.
En respuesta a Germán Valero

Re: Sequential quiz - WCAG - legal

de Rick Jerz -
Imagen de Particularly helpful Moodlers Imagen de Testers
(As a separate post...)

Yes, if all that you want students to do is to repeat something (i.e. some facts or definitions) from the textbook, they can easily search the textbook for an answer. I teach quantitative courses, so it is a bit easier for me to configure problems that are not directly in the textbook, but rather cause students to have to think. (Then I do random questions, too.)

Some of the problems that you are facing are why I claim that teaching online is harder than teaching face-to-face.  It takes a lot more effort to design an effective online course.
En respuesta a Rick Jerz

Re: Sequential quiz - WCAG - legal

de Dominique Bauer -
Imagen de Documentation writers Imagen de Particularly helpful Moodlers Imagen de Plugin developers
The subject is difficult, complex and controversial.

I have students take exams in my course on Structural steel design, but ... and I could talk about it for some time.

While I may not necessarily agree with all of the content of the following articles, I think there is some food for thought here.







Note that I didn't write these articles, don't even know if they are well-founded, and I don't really want to discuss their seriousness or content. triste

En respuesta a Dominique Bauer

Re: Sequential quiz - WCAG - legal

de Rick Jerz -
Imagen de Particularly helpful Moodlers Imagen de Testers
Yep, complex, and controversial.

If one, just as an example, did not evaluate students' knowledge in a course on Structural Steel Design, which of these students would a company want to hire to build a new bridge?
En respuesta a Rick Jerz

Re: Sequential quiz - WCAG - legal

de Nilanjan Basu -
Thanks - I am trying that approach. Sitting with a spreadsheet and creating similar clones of every question with changed numbers. Then I can randomly ask one from each such subgroup in an exam.
A colleagues suggested a similar technique for non-numerical reasoning question: create clones based on True/False logic. For example version one is built around "when does X happen" and version 2 is "when does X not happen". By combining multiple such conditions and a little language rephrasing this can generate multiple clones of non numerical reasoning question.
En respuesta a Germán Valero

Re: Sequential quiz - WCAG - legal

de Steven Ouellette -
You might look into a student-administered honor code. We had GREAT success with this at the University of Colorado, Boulder. In our department, we had a "two strikes you are out" policy. If you accrued two honor code violations, you were kicked out of the program.

Such a thing must be multi-pronged. It can't just be teachers trying to catch the students - you only train them to cheat better.

Here is what we did:

First, a student-administered honor code. Students are harder on students that faculty. They understand that the value of their degree or certificate is directly related to how well other students do after graduation. They don't want cheaters to sully what they have worked hard to achieve. Procedurally, the students would create the honor code and then make decisions on the violations that were contested. The professor has sole control over what the consequences are in each case, the student board merely determines if the violation occurred when contested. If a student contests the professor's finding and it is found by the board to be without merit, there are no consequences. If the student contests the violation and it is found to have merit, they are subject to both the professor AND student board sanctions. If the student does not contest it, then they are subject only to the teacher's sanctions. Obviously, there needs to be some training for the student board.

Second, at the beginning of each semester, each professor presents briefly on why the honor code is important to them along with some standard slides. This helps acculturate the students into accepted norms. In our case (engineering management) a professor might mention the consequences in the past of poor management ethics and their consequences not only for the individual, but for those around them that bear the real cost. This personalizes it beyond a list of rules.

Third, we had an online honor code quiz. Each student had to pass with 100%, though they were given three tries to do so. If they did not, they were not allowed to register. The idea was to help teach the students (especially ones from other cultures) what was and was not acceptable in the US higher ed system. This was more about concepts than process. For example, "sources need to be cited" rather than how exactly to do that, or what levels of students working together was allowed in different scenarios. It was interesting to analyze these results. Often students were too strict in their interpretation (perhaps wrongly thinking that all the questions were violations) so they learned something by missing those. But we did get a number of answers that seemed to say that research fraud was perfectly OK, so something was learned when they missed that one too.

There is a LOT of research showing that adherence to an honor code is very strong if it is seen as the norm. The opposite as well - if students see that cheating is the norm, they will adhere to that.
En respuesta a Steven Ouellette

Re: Sequential quiz - WCAG - legal

de Steven Ouellette -
...and I just learned today it was expanded to the whole of the College of Engineering and Applied Sciences! From the Interim Dean:

Dear colleagues,

This year the college is undertaking a new initiative to support academic integrity among our student body. All degree-seeking undergraduate and graduate students enrolled in the college will be required to pass an Academic Integrity Quiz. This initiative was strongly endorsed by both the Undergraduate Education Council and Graduate Education Council in the college and builds upon the quiz successfully administered by the Engineering Management Program for many years. (Thank you, EMP, for your leadership!) 

The Academic Integrity Quiz is housed in Canvas and will be administered annually each fall to our students to emphasize the importance of ethics in academics, research, and their professional careers. Students will be required to complete the quiz with a score of 100% to ensure they understand expectations as a CU student and as an engineering professional. The quiz pulls from a bank of questions, so if a student has to repeat the quiz, they will receive a fresh set of questions. In our current environment of mixed modality instruction, supporting the education and training of our students in this realm is more important than ever.

Students will be invited to take the quiz in mid-September and need to complete it by mid-October or an ethics hold will be placed on their student account that prevents registering for classes. 


En respuesta a Steven Ouellette

Re: Sequential quiz - WCAG - legal

de Rick Jerz -
Imagen de Particularly helpful Moodlers Imagen de Testers
I notice a few things about this information that you shared.

1) Your school is using Canvas, and not Moodle. Boo!
2) This Academic Integrity quiz is using a quiz bank with enough questions whereby they are using randomness to protect cheating, even on this anti-cheating assessment. This method supports some of what we have said.
3) The administration does not seem to be concerned that every student must have the same exam. I believe that Nilanjan  was worried about this issue. In your case, no need to worry.
4) From what I can tell, Canvas does provide a "sequential" delivery of questions. I am not sure how similar this is to Moodle. Your thoughts? Which settings in Moodle are you using for sequential question delivery?
En respuesta a Rick Jerz

Re: Sequential quiz - WCAG - legal

de Steven Ouellette -
Yeah, I tried hard to get them to use Moodle. The first Student Honor Code quiz was indeed on Moodle for a number of years. (I made it when we had, *gack* ecollege at the time which was so horrible.)

I think the quiz bank is not about cheating, it is about giving the student different, challenging scenarios so they think/learn about the principles of academic honesty. At least it was when I was in charge of it. But really the point is not the quiz itself, it is part of a system that is intended to change the on-campus culture with respect to cheating/plagiarism. Professors cannot by themselves change this (IMHO and supported by research) with any test technology up to and including TurnItIn. It takes a multi-pronged approach as I describe above to change the culture, and a quiz about the honor code by itself is not going to work. As part of an overall process, a quiz requirement can reinforce its importance and some of what seem to students to be the more subtle points. (The one I ran into was submitting work you did for one class to another. Specifically forbidden by the SHC, but we ran into it frequently until we started doing the SHC quiz.)

When I taught at the business school, I had one student per class per semester that plagiarized. (Was even featured on the local news about this.) And that was lower than a lot of profs. When I moved over to Engineering, it wasn't that frequent due to the subject matter (Engineering Management), but maybe one student per year. Until we instituted the full suite. Then we caught a pair and sanctioned them. Caught them again and kicked them out of the program (heartbreaking as they were foreign students). I can think of maybe two more events after that, in a department of maybe 200-400 students, almost all remote. The good thing about students being ephemeral is you can change the culture very quickly with a faculty that apply the rules consistently. We were required to report all HC violations to the student honor code board.

Accessibility is a HUUUGE issue at CU Boulder. The DOJ sued them first right out of the gate for accessibility issues. Anyone running into accessibility issues would be wise to contact CU Boulder's Office of Information Technology.

I never taught with Canvas. I love Moodle though, and still use it for my business.
En respuesta a Nilanjan Basu

Re: Sequential quiz - WCAG - legal

de Nilanjan Basu -
Rick - thanks for the pointer. I will dig into that angle but given the layers of government guidelines and institutional responses by the university I suspect it will be a long time before I get a clear answer to whether it is a guideline or a law.
Tim - I agree with you in general. However, this is a very specific situation where a mid/lower level undergraduate course that is tools /calculations focused and has a moderately large class size is perforce taught and evaluated online thanks to Covid. The experience of the last 5 months tells us that unless there is proper monitoring there can be extensive cheating. Which brings us to a few unpalatable options: a) Online proctoring which is invasive on privacy grounds and makes students genuinely uncomfortable b) The sequential randomized quiz / exam or c) exams /quizzes that ask an impossibly large number of fairly easy questions such that the payoff to doing your own work is significantly higher than trying to coordinate. I would argue that given these three terrible choices the sequential quiz is not necessarily / clearly the worst.
En respuesta a Nilanjan Basu

Re: Sequential quiz - WCAG - legal

de Nilanjan Basu -
Thanks everyone - it was great to see the perspectives.
Summary from my side: I got a second opinion through formal faculty group channels that came through really quickly. As Rick had indicated earlier, it is a guideline and interpreting it as a legal requirement was the misguided (I will assume that is all it was) notion of someone in university administration. I am free to use it.