How can I improve the workflow when grading exams?

How can I improve the workflow when grading exams?

Titus von der Malsburg -
Erantzun kopurua: 18

Hi, I'm currently grading exams that I implemented in Moodle using the assignment feature (Moodle version 3.1).  The process is fairly slow and this is in part due to issues with the user interface which I'm going to list below. I'm looking for ways to improve the workflow.

1. When I open someone's solution (just text, no attached files), the text is folded and I have to click a little [+] icon to see the full solution.  Just one extra click but since there are many assignments and many students this does cause some appreciable friction.  It's also unnecessary.  If I grade solutions, of course I have to see everything.  So I think the solution shouldn't be folded in the first place.  Question: Is  there a way (for users, I'm not admin) to show the complete solution by default?

2. When I give feedback for an solution, I want to do it in a certain format (say markdown).  However, Moodle doesn't remember which text format I chose last time and for every assignment I have to use the dropdown menu to select markdown. Question: Is there a way, to choose a certain text format (plain, moodle, markdown, plain text) by default?  I couldn't find this in the user settings.

3. When I'm done with a solution and move to the next student, I have to confirm that the grade and feedback should be saved.  The answer is always yes here, so this is another step that needlessly slows me down. (Why would I enter a grade and feedback and then not save it?)  Question: Is there a way to disable these confirmations?

4. When I choose to save the grade and feedback, I get another dialogue box that tells me that the data was in fact saved. Again I have to click a button. I don't understand the purpose of this confirmation. Usually, I assume that the data is saved and that I get a message only when that failed. This is standard pretty much everywhere. For example, when I save a file in Word, there is no dialogue box informing me that it worked.  Question: How can I disable these informational dialogue boxed?


Thank you very much in advance for your input.


  Titus

Puntuazioen batez bestekoa:Useful (1)
Titus von der Malsburg(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

Colin Fraser -
Documentation writers-ren irudia Testers-ren irudia

Wow, you have certainly done some seriously critical thinking here Titus, and good on you. Every one of those comments are thoughtful and would be really helpful I suggest. The problem is that this would mean a large rewrite of the marking module. For such advances in the quality of the product, I would suggest you place this item into the Moodle Tracker, as is, for an improvement to the core product. I understand that there is some work being undertaken, or about to be done, on this module for v3.5 so please get this into the Moodle tracker as soon as you can. 

An alternative is to join the Moodle User Association, at about €67 Euro, per person, and make some impression there to get support for these changes. 

Excellent ideas that should be well supported. 


Titus von der Malsburg(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

AL Rachels -
Core developers-ren irudia Particularly helpful Moodlers-ren irudia Plugin developers-ren irudia Testers-ren irudia

Hi Titus,

From first reading yesterday, and a re-read today, am I right in thinking your assignments are set up for online text? If so, have you considered using the Journal plugin, instead?

In one of my classes I had students writing short, five paragraph essays on a regular basis in online text assignments. Like you I learned that there was a WHOLE lot of extra clicking needed to grade these the first time, and then, regrade them after the students re-wrote them to fix problems.

I switched to using the Journal. It can show all the entries for a whole class, or for a group, so it is easy for the teacher to scroll down the page, grading as needed. Then, one click to save all the grades and feedback/comments. I used Justin Hunt's Snippet tool for Atto to create "boiler plate" comments to make commenting quicker and easier. Eventually I wound up with one snippet that was a template for a grading rubric where I would just need to type X's to mark what was good or bad.

I even modified the Grade Me plugin so that it would show when students needed a Journal submission graded, and had Grade Me installed on the front page of Moodle which really helped cut down on "clicks" on a daily basis.

Titus von der Malsburg(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

Lynn Wilhelm -
This has become an issue for my graders too. I love the confirmation that the grade and feedback was saved, but clicking to dismiss that pop-up and then clicking "change user" to move to the next student. Because we like to use grading (marking) workflow which already requires extra clicks, our graders are complaining about the extra work. For 100 students this can really add time to grading. In order to help them, we've turned off grading workflow.


I, too, would like to see the inline text show up immediately when grading. It would be wonderful if it would show up in the viewing pane (where file submissions appear). The viewing pane also needs to be adjustable. When using advanced grading, the grading pane is to hard to view if the viewing pane is open.


Lynn Wilhelm(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

Joost Elshoff -
Particularly helpful Moodlers-ren irudia Testers-ren irudia

There have been quite some improvements on later versions of Moodle, also to this part of the environment. If this is about the Assignment module, you can use the grading interface which can convert submitted text to pdf so a teacher can annotate it. Also, in this interface, you shouldn't have to save every change for every student every time, but you should be able to move through the list of students and save your work at the end. 

Before upgrading to Moodle 3.3, you may want to have a look at the official Moodle demo sites, to see what has improved.

Joost Elshoff(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

Lynn Wilhelm -

Thanks Joost,

I don't know when my institution will upgrade. We are using 3.2 now.

I'll look at the demo site to see. I would worry about using the grading page (interface) and switching users without saving each time. Before updating to 3.2 we had graders complain that they entered grades but "Moodle lost them". I believe it was user error, but now the forced save is good. A Save and Next, button like before, would be best but I'd still like to see the confirmation notice.


Joost Elshoff(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

Colin Fraser -
Documentation writers-ren irudia Testers-ren irudia

Joost, the PDF converter should work every time, but it doesn't... occasionally. I have often asked students to save their work as PDF, upload it so I can mark it with annotations, but often they don't save it or for some docs, they just don't appear. I have to download the doc, then upload it after marking.  

What I have considered doing is to get hard copies of things from students and using a document camera, video my marking it, and upload the video for that to the the students... but a proper document camera is seriously expensive and the ideas I have come up with so far to get around that have not proven as successful as I would like. 

The point being that I really would like to get away from PDFs altogether, and that is happening, but slowly. 


Colin Fraser(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

Rick Jerz -
Particularly helpful Moodlers-ren irudia Testers-ren irudia

Colin, I have taken note of this conversation and some of the interesting challenges.  Quite honestly, I have not experienced these problems but this is probably because I have a different workflow.  Also, I am at a university that has a different makeup of students than those who teach at different educational levels.

In my case, I give students assignments, they work on the assignment and submit their files using Moodle's Assignment activity.  These assignments can involve a wide range of software products, including MS Project, MS Visio, CAD, etc.  To grade these, I download all of the files to my computer, download the (csv) grading report, open this grading report in Excel, individually open each of the student's file, look at it, then enter a grade and comments in the grading worksheet, and finally upload the grading worksheet to moodle.  I am done.  And this is not bad, it's pretty efficient.

However, I never find the need to create "marks" on their submitted files and return these to the student.  I find that my comments to them are sufficient.  Also, I don't use any multiple submission attempts, where the student's work might go back and forth several times.  But I do appreciate that this could happen.

So although I understand how PDF is one way to mark up a students file, it seems like it may not be the best, which I think is what you are saying.

So I don't know the best approach.  If the student submits a Word file, I would use Word's "track and make comment" feature.  If the student submits an Excel file, I would use Excel's text tools to make comments.  In other words, I might try to use the application's tools to do the marking up.

If I had a need for this submit and resubmit system, I wonder if a "single user" forum might be the better way to handle this.  A student would submit and provide comment.  The instructor downloads the file, looks at it, maybe marks it up, and then replies to the student's post with mark ups and/or comments.

I recall a product called "VoiceThread" that seemed to be focused on allowing instructors to give "audio" feedback.  I never used it, but it seemed interesting.

Another thing that I have done when I want to type a lot of feedback is to use my Mac's voice to text tool.

Well, I am just sharing some thoughts about this issue.


Puntuazioen batez bestekoa:Useful (1)
Rick Jerz(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

Colin Fraser -
Documentation writers-ren irudia Testers-ren irudia

Interesting thoughts, as always, Rick. I too do not know what is the best approach when marking student work, but I have had a number of ideas thrown at me over the last year that seem to be garnishing a little more positive response. 

I'm dealing with 13-17yr olds and the demands are different. The Gr8 are still into stamps on work, so badges are good, I paste them into their work when possible. For the rest, I know that just writing stuff on their papers is a waste of time, they don't read the comments anyway, and if I ask for a resubmission, many will just give me the same thing back with a few more full stops and capitals..irribarrea So I'm really looking for alternatives.

One thing I want to start, properly, is videoing my marking of a paper. This will appear spontaneous and directly speaks to the viewer. I think I can do this using a screen casting product like Captivate or Screencast-o-matic, or if their work is on paper, a web-cam, mounted on a retort stand with a couple of strong lamps. It looks down on the paper and they will see me crossing stuff out, re-ordering and listen to me making comments. All I really need is an affordable, reliable video editor for Windows. Mac has Final Cut which, apparently, is fairly easy to use. The beauty here will be that I can edit the video and replace the sound track if I need to. I know, it will take a bit more time, initially, but once I get used to the techniques involved, I should be able to do the videos with a minimal amount of editing. Windows has a built in voice recognition app that I have never used, so an audio feedback becomes possible

Without Moodle, this becomes a huge task, and your idea of a single user forum...mmmm very interesting for some things I think.  


   

Colin Fraser(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

Rick Jerz -
Particularly helpful Moodlers-ren irudia Testers-ren irudia

Colin, if students don't read what you write on paper I wonder if they will watch your videos?  It might be a lot of work without your desired benefit.  I can begin sharing some ideas with you about what you would like to do with video.

1) Where are you going to put all of these videos?  On YouTube?  Managing all of the videos could take some effort.  Do you have a video server available?  I do, and it works well.  My university provides it to me, so I have no idea how it operates and what it costs.  As a minimum, you will spend twice the amount of time producing and delivering your video for every minute of content.  So if you give a student 5 minutes of feedback, it is 10 minutes of your time.  If you have 20 students in a course, that's over 3 hours per assignment.

2) I assume that your students are remote?  If not, why not just arrange a physical review time with them?

3) When I make my own "live" videos, I just mount my iPhone on a tripod.  Smartphone quality has become great, and there is no need to invest in any fancy cameras or equipment.  Consider mocking up this setup and experimenting.

4) It is likely that you will want to edit each video, many cutting out some of the wasted beginning or end.  Any you will probably want to compress and encode it.  This also takes time.  The output file format is easy... mp4.  But there are many settings that affect the size and quality of your final video.

5) Remember, the Mac has always been the better multimedia computer.  In my 30 years of working with PCs and now Macs, I believe the Mac is still best.  Sure, one can do this with a PC.  It will take more time, it will crash more often, and it doesn't manage resources as effectively.  But go for the PC if you wish.

6) I purchased a "green screen" system, with lamps, for around $125 on ebay.  I believe a "lamp only" system should cost less.

7) I use two video editing products: Camtasia, and Adobe Premiere.  A teacher license for Camtasia costs about $100/year.  Right now, through my university, The complete Adobe Suite costs me $10/year.  But this is to change soon, and it may end up costing me $20/month (regular teach rate.)

Camtasia shines at computer-screen video captures and annotations.  Premiere shines at video composition and output.  Depending upon what I am doing, I will take output from Camtasia and further edit it in Premiere.  However, one can do a lot with Camtasia, and Camtasia does output good mp4s, it lack some of the powerful editing and management tools of Premiere.

8) Consider initially doing this not for every paper, but as a review for the entire group.  Your video could take on "I want to review with everyone some of the things that I saw were good with everyone's papers, and some things that need improvement."  This might be a 20 minute video once each assignment, or once each week.  (Or it could be "I want to show you 10 good things and 10 wrong things with your submittals.")

9) Many interesting pedagogical things that you can do.  Students who do well do not need to resubmit, unless they want to improve their grade.  Students who don't meet a certain grade threshold have to resubmit.  Many others.

10) Learning the world of video will consume a lot of your time.  It does for me, but I feel it is the way of the future of education and worth my effort.

If you go to the bottom of my homepage, you will see some links to my Moodlemoot presentations, and videos.  If you go to my rjMoodle links, at the top of my Moodle, click Help, and you will see more of my videos.

Last year, at MountainMoot, I tried to talk about video production.  I only had a half hour of so.  It didn't work well.  One attendee mentioned that it was "trying to squeeze an elephant into a tuna can."

Well, in summary, I think you could do what you want to do.  You probably already have a smart phone.  For other equipment, probably less than $150.  Software, probably $100 to $300 per year.  Time?  Probably more than you want to spend, but you decide. 


Colin Fraser(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

AL Rachels -
Core developers-ren irudia Particularly helpful Moodlers-ren irudia Plugin developers-ren irudia Testers-ren irudia

Hi Colin,

Rick's points about video is "spot on" but don' worry about what he says in point #5...PC's work just fine for video editing. barrea The main thing is, that both Mac's and PC's will do a fine job, so use whichever you have.

A good Windows video editor that is reasonably price is Corel VideoStudio Ultimate. On sale right now for $89.99, plus it does have education licensing if you need it for a whole lab, like I used to do.

Another good video editor (free one) believe it or not, is Blender, which is actually an Open Source 3D creation program, that has built in video editing capability. The only down side is the program has a fairly steep learning curve. Like Rick I use Camtasia, but my version is an old one that still works fine.

One thing I have started recommending for creating videos, is OBS Studio, Open Broadcaster Software, which is free. Versions for Windows, macOS 10.10+, and Linux are available. It allows you to live switch from multiple video cameras, computer monitors, pre-recorded videos, pre-recored audio, still pictures, documents, presentations, etc. while live streaming or recording for later editing or use. It works really nice on my current desktop with dual monitors and up to four USB cameras. Since I rarely show myself on screen, I usually use a wireless headset for narration which allows me to move around without affecting the quality and volume of the sound recording.

Since you mention you might need a document camera, setup, save your money and don't buy one of the overpriced ones with a document platform and lights. See if you can get a Siggi HD Plus USB camera from IPEVO. Depending on the model, they're only about $100 and are made with a crane arm that allows you to position and rotate the camera as needed. Also has a built in microphone, if needed.

If you ever want to add sub-titled text to your videos, another free program, Aegisub does that easily. Versions for Windows, OS X and Unix are available.

Puntuazioen batez bestekoa:Useful (1)
AL Rachels(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

AL Rachels -
Core developers-ren irudia Particularly helpful Moodlers-ren irudia Plugin developers-ren irudia Testers-ren irudia

Almost forgot, since you are using Windows, if you need to quickly crank out a video, don't forget about Windows Movie Maker 2017. Available for Windows 7, 8, 10, XP and Vista.

AL Rachels(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

Rick Jerz -
Particularly helpful Moodlers-ren irudia Testers-ren irudia

Yep Al, I knew #5 would catch a few.

Now, would you recommend as much memory in your computer as possible?

You didn't say it, but I think I can detect that you too have invested a lot of energy into learning the ins and outs of video making.

(I too don't like to put myself into my videos.  I use a Samson USB  mic when I need better-recorded audio.)

Rick Jerz(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

AL Rachels -
Core developers-ren irudia Particularly helpful Moodlers-ren irudia Plugin developers-ren irudia Testers-ren irudia

Hi Rick, YES, lots of time and energy! As to memory, my main desktop has 16GB and when I occasionally look at memory usage via Task Manager, it rarely seems to go above 31%, even when video recording. I find that as long as my current equipment is able to render faster than real time, I am happy with it. I can remember about 20 years ago for a school project, I helped my son create a 3 minute 48 second video, that took 32 hours to render on our 160 MHz home computer. We then discovered some errors that needed fixing so I rendered it on a brand new 266 MHz school computer where with extra memory and faster hard drive, it rendered in only 7 hours. barrea The year before I retired, I demonstrated to my students the same video would render in about 3 and half minutes on the computer I had set up as a video workstation.

Before retiring, most of the video work my students and I did was to create videos of school events, Very seldom after my first two years of teaching did I create any "instructional" video content due to what I taught. I taught computer applications, word processing, databases, spreadsheets, presentations, etc. and they just changed too much, too often. For instance, one June I spent over 20 hours creating a lesson on how to use Corel Draw, a vector graphics drawing program. Literally, two weeks after finishing the video, I got a grant to replace all my lab computers and software. The new version of Corel Draw that was part of my purchase made the video obsolete due to changes in the program.

AL Rachels(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

Colin Fraser -
Documentation writers-ren irudia Testers-ren irudia

mm lots of comments..irribarrea 

For starters, I don't believe that technology is making teaching any easier, its a lot of work to use properly, but the issue is what gives students the best opportunity to learn something, even if it is only how to read speed signs at 110k/h, it's something.

Thanks for the hardware/software ideas gentlemen, fabulous. Never owned a Crapple, Rick, never liked the idea of having a tool I can't build upon, where hardware is fixed and to improve, I have to buy a new one. We don't get things for the same prices here in Oz that you do, seems shipping costs can add extravagantly to the purchase price and "currency fluctuations" cause even further issues.  MacBook Air is about 35-50% more than the cost of a good PC laptop here. 

We have an ever bigger issue with software, there is no way we would get any Adobe products at that price, they have raised them by about 250% in the last two years on top of what was already a huge price disparity from the year before. We really have to go Open Source, I do anyway.  

<comment>I am not going to mention the moral or ethical perspectives that lead to these kind of things, for two reasons. One, nothing and nobody shall stand in the way of profit. Two, it is ours and we can charge what we want. I wholeheartedly agree with these views, Adam Smith would be proud of his legacy. So would Gene Roddenberry, after all, it was in the fertile imaginings of Star Trek that we learned of the Ferengi...and the 250 Rules of Acquisition - or was he describing modern corporations?... </comment>


 

Titus von der Malsburg(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

Jon Fila -

Here's what I've done to improve the grading workflow. 

https://moodle.org/mod/forum/discuss.php?d=357787

  • Getting rid of +/-
  • Configurable reports
  • Requires Grading Filter
Puntuazioen batez bestekoa:Useful (1)
Jon Fila(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

Colin Fraser -
Documentation writers-ren irudia Testers-ren irudia

You're right Jon, but unfortunately, we are in a hosted situation and cannot access the code. I am still waiting for installation of the Progress Bar or Completion Progress Bar from a request some..erm.. 3 or 4 years ago so requesting a code change is just not going to happen. 


Titus von der Malsburg(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

Javier Escalera -

This is how I like to grade text submissions, if it helps. I am on Windows and I find this more than adequate for my needs. There is some back and forth, but no waste of clicks:

Download all submissions to local computer, unzip, and open folder on file explorer, with preview.

Check quick grading on submissions page.

Read first submission on file explorer preview window, OR open file if you have to, read, go back to quick grading in Moodle and put in grade and quick feedback for student. Repeat for each student, saving every now and then.

Two monitors make it even easier.

Hope it helps someone,

Javier


Javier Escalera(e)ri erantzunda

Re: How can I improve the workflow when grading exams?

Rick Jerz -
Particularly helpful Moodlers-ren irudia Testers-ren irudia

Good post, Javier.  This is esentially how I do it too.

Depending on what the student has submitted, sometimes just looking at it in "preview" mode is adequate.  But sometimes I need to open the submittal in the actual application, which is a little slower.

Where I depart is that I "download the grading worksheet", open this in Excel, make my comments, then upload the Excel file.  I find that Excel's "quick fill" capability makes it fast for me to provide the same comments to different students, when adequate. In the grading worksheet, I am able to quickly fill in initial grades (e.g., 20 max for those who submitted, 0 for those who did not) and then modify this grade as necessary after viewing the submittal.

This is really just a subtle difference than entering grades and comments in Moodle.

It appears that neither you or I find a need to "mark-up" right on the submittal.