It doesn't appear that workshop switches into the different phases automatically. Is it something I am doing or is it a missing feature. In 1.9 once you set up the workshop it switched into every phase automatically except grade release. It is cumbersone, akward and time-consuming to have to manual switch workshops into the appropriate phase. Can we fix that ASAP?
Can we fix that ASAP?
No. There is nothing to fix as this is not a bug but an intentional feature. Many problems in Workshop 1.x were there just because Workshop did a lot of things silently in the background. So it was decided and described in the Workshop 2.0 specification document that the new version would be interactive and would require more user input - thus allowing total control over what is going on.
Bad things might happen if - for example - some student were not given a submission to assess. The workshop activity would continue with this student left behind and no-one would spot anything wrong. So there must always be a human who confirms that the workshop can continue to the next phase.
I am sorry you find the fact that you must pay attention to the workshop "cumbersone, akward and time-consuming". But Workshop 2.0 was not designed as a stand-alone tool looking after everything.
I didn't think that being more interactive and requiring more input shouldn't mean that a teacher should have go in click a button to move to the next phase.There of course must be supervision -- I am not sure why it can't be a choice to move directly into the next phase.
So instead of it happening automatically for everyone, that when a teacher sets up the workshop :
Do you want to move directly into the assessment phase?
With a question like that it would still be interactive but for those of us who know that we are not going to make any changes we can set up the workshop to proceed. I use many workshops with very large classes -- to the extent possible -- I shouldn't have to mind the software.
Here is the choice I would like to see:
(1) Step 1: set up all the phases of the work (this could be done by going into the different phases) it does not necessarily have to be a global page -- but the choices made would dictate how the workshop unless teacher change the choices. For instance:
(a) choose random allocation
(b) choose 3 reviewers per submission
(c) choose to have a faculty evalution
(4) choose the weight of the faculty evaluation
(2) Step 2. at the close of the submission stage have the submissions allocated based on the choice. Obviously, if faculty chooses to manually allocate, the assessment stage can't open until the faculty makes the allocation. However, if the faculty chooses to random allocate - their no reason why the program couldn't make those randon allocations and open the workshop for assessment based on the time and date identified.
(3) Step 3. at the close of the assessment stage, initial grades can be calculated automatically. Faculty would need to review and make any changes and the grades would not be released to the student until the faculty manually closed the workshop.
Putting myself in your shoes I can somewhat understand your point of view on the whole, Vernellia. You got used to a system which worked well for you and you want the same back. However for most people the old Workshop module was enigmatically difficult to understand, things often went wrong (or at least produced unexpected results, etc.). I find the new version easier to understand and set up while providing the best tool imaginable for peer assessment.
The development of the new Workshop module has been a very open process with lot of opportunities for feedback, comments, feature requests, etc. (see here and here for example). I believe it's a bit late to do another rewrite of the module now.
I, for one, really really like it!
The old workshop did not work realy realy well for us. It had a lot of problems. It seems to me that it should not be an either or-- Either we have no ability to have phases switch automatically or (as in the old workshop) it switch automatically without any input from the teacher.
Think of this as an additional feature of the new workshop. I assume that the workshop is not static that it will be reworked over the years bugs will be removed, features will be added.
I am just saying that it should be an option to have the workshop's switch phases automatically - I can see where it won't be the default -- But an option.
I wasn't aware that this option could not be added without doing the workshop but prephaps I am wrong.
I participated in the feature request and provided feedback -- however, somehow I missed the fact that there wasn't any way to switch automaticallty. At any rate, lack of participation in the prior process should not foreclose the opportunity for input now.
of course it doesn't foreclose the opportunity for input now, as http://tracker.moodle.org/browse/MDL-26099 proves (isn't it great? Imagine that with non-open source software...). Reading your posts I just felt that the lack of auto-switching to next phases was seen as a bug... I probably just read you wrong. Sorry if I sounded miffed...
I don't see it as a Bug -- but it is a flaw. Having the teacher tied to the software merely to click a button to switch phases will control how and when you can set up workshops because it will be tied to the teachers availability.
It also increases the likelyhood of problems -- ie the teacher missing the dates/times of switching phases.
If it set up this way because it is the only way to make other parts of workshop work than so be it.
On the other hand, the ability to switch phases ought to be a part of the program.
I'll second your request Vernellia and have voted for the tracker issue mentioned above.
I used the workshop module in 1.9 often and trained others on how to use it and am in the process of using the workshop in 2.0 with students for the first time now. In the new workshop, the phase switching and lack of automation tripped me up. Even when I thought I understood it, I didn't get it right--I had manually allocated all of the submissions to be assessed and thought I was good. It wasn't until a student pointed out that they couldn't see anything to assess that I realized I also had to click the light bulb for the assessment phase too.
It would be great to have options for automated or manual processes. In the automated version, I would see fields designed to set dates for different phases to begin and end with the option for the transition to be manual or automatic at each stage. For instance, I might want the allocation process and the move from allocation to assessment to be automatic, but may want to review the scores before making them available in the final phase.
A bonus would be to have options to open and close phases based on specific events. It would be very cool to be able to tell the workshop to move automatically into assessment phase when all student submissions had been received and/or allocated.
The workshop is a great tool but it is vastly underutlized due to its complexity. We've made progress in 2.0, but by removing the option for automation, we've put up a roadblock to adoption by making the workshop more complex than it needs to be. Teachers have so many more important things to do and contributions to make--they don't need to be "tied to the software merely to click a button."
Thanks for providing such a thoughtful comment. The lack of automatically switches has cause me to miss opening and closing several workshops this semester. In fact, only when students email me did I realize that I had missed . It seems to me if you delay allocation till the workshop has closed; and you select random allocation. There should be a feature to allow you to select to have that done automatically.
Vernillia and Michelle,
As a dedicated user of the Workshop module, and an advocate of active learning which peer evaluation provides, I too have been miffed as to what possibly the rationale was to create a totally manual system for what used to work fine without my intervention. Now I have to intervene 4 times for each course for each Workshop.
In my view, this is a design issue that needs to be addressed, if Workshop activity is to be useful. I echo your request to make the automated change of phases possible as an option for the many of us who do not wish to click so many times for what used to work just fine. Thanks all.
I can't agree more with what you said.
What wo do need is:
whether the phase switch would work automatically or maunual depends on what we set while the setup time.
See the algorithm in this module. It allows peer assessment to begin before all submissions are made.
This is much better for both students and teachers.
David, When you are doing dozens of workshops a week with hundreds of students, it is a useability issue. I understand your choice. and if it is a programming issue than so be it. But it seems to me the choice is not do everything authomatic or do nothing automatic. It seems that the way you design workshop makes pedadogical choices rather than useability choices. That given the wide range of uses and pedadogies, workshop should designed to provide the most flexibility, so that the teacher on the ground can make the choices. That to me is not an unrealistic expectation.
You can now switch phases automatically in Moodle 2.3 http://docs.moodle.org/23/en/Workshop_settings
Are you suggesting that the workshop switches all phases automatically as it did in version 1.9?
As far as I understand, it only switches from submission to assessment phases automatically and the rest need to be changed manually.
Thank you for the little clues throughout your posts that finally led me to understand how to switch manually from one phase to another. We have not used Workshop in the past since it looked to be more trouble than it was worth but with Moodle 2.0 changes, we decided to review it again.
I have to say that I 'expected' it to automatically switch from the Set up phase to the Submission phase as soon as I had completed (all the boxes were checked) the Setup phase. I didn't expect to be able to go back and edit the Setup phase, but found myself going back again and again to see if I missed a setting or forgot to set permissions because it just wouldn't switch to the next phase. I have literally spent hours searching for how this works and actually thought it was broken.
No where in any of the documentation for Workshop does is say:
"to switch from one phase to the next, click on the lightbulb icon"
This is not intuitive at all as even after I found out that I needed to manually switch phases (through reading the forums), I couldn't see where to do that. I was looking for something on the Setup screen or under the Settings menu. Perhaps the icon for that needs to be more representative of what it does. I realize that the lightbulb allows you to go back and edit the phase again or switch to the next phase but perhaps some text would be more helpful.
I do agree with the previous posts that it should automatically switch between at least the first few phases once milestones have been achieved (ie. Setup is complete; submissions have been received and reviewers have been allocated) and leave the grading section to be done manually.
It is always a challenge to please all of the people all of the time and I believe that if you can automate what people expect to have automated and allow control over the phases they need to control you will be providing a useful tool.
Thank-you so much for writing such a detailed explanation. I was in the same boat as you, but I didn't pick up on the nuance that you did. I too have spent hours back and forth.
Isn't that always the case, it's easy now, but until you know where to look, you can spend hours and hours on something.
thanks all for the valuable feedback on this! I just updated Workshop wiki docs page at http://docs.moodle.org/en/Workshop_module#Workshop_phases and added the information about switching the phase (by the way - it is wiki so you are welcome to improve it whenever you realize there's an information missing).
Regarding the support for automatic or scheduled phase switching. I can see a demand for it. Even though I see many potential issues, too, I believe it will be possible to come with a solution of this. I can imagine a side block called Workshop events/scheduler manager or so. That would be capable of executing various jobs based on time and/or conditions. For example, this block would be able to notify the teacher about new submissions and assessments (similarly to what forum does when you are subscribed). One of the supported actions would be "Switch the Workshop into ... phase". The block would inform the teacher whether the scheduled switching was successful or not.
Yes, such a tool would be interesting... Let us see!
Thanks. David. I am glad to see that you are considering. There are numerous situation where if there is a firm deadline for submission (that is if you miss the deadline you get a zero), and where you are doing random allocation after the close of submission -- where the closing and the random allocation can't be automated.
David, where is the option to automatic switch phases in your agenda? I am setting up my courses for next fall and it would be nice to be able to set up the submission phase, end of submission phase, start of the assessment phase and end of assessment phase without regard to my actual availability to change phases
Understood. I look forward when it is added back into workshop since the lack of it causes significant issues. I have missed opening several workshops and I have had to cut the amount of assessment time for students. The automation may not be appropriate in every situation but in the situation where you are going to do random assignments and not allow late admissions, there doesn't seem to be any reason not to automatically open the submission phase, close the submission phase, open the assessment phase and close the assessment phase.
Has anyone made even dirty fix for this? Say that you want go from submission phase to assessment phase at 0:00 -- and want to sleep when that happens. Only quick fix I can see is copy switchphase.php to switchphase-secret123.php, remove authorization and then wget it by at command. Not dirty, it is ugly.
I agree with Vernellia and Michelle, I'm just starting to use Workshop module and found it very dissapointing to notice that the workshop process can not be scheduled to happen automatically.
I've used earlier a system, where I set the timetable of submissions and peer reviews for the whole course when the course starts. It was very important feature in that system. I have 10 exercises in the course, and the students are informed in the beginning of the course when the deadline for each of the submissions and their assement will be. If I have to open and close each submissions and then open and close the assessment for each submissions there are bound to be delays and even mistakes, which then confuse the students and cause problems.
There is the possibility to set schedule for access control, I don't quite get why that can't be used automatically to control switching from a phase to another...
It would by no means be doing unexpected, silent things in the background. On the contrary, it would ensure a very exact, easy and timely control for the whole process.
I hope this will be added to the system, so that we don't have to try and deal with ackward and cumbersum dirty fixes to get this done.
I have 6 courses, 26 workshops in each, 4 clicks per workshop, that is 624 clicks per semester for something that required no clicks in the earlier version.
624 clicks on just the right days at just the right time. Workshop is the reason I swithched to Moodle, but this change makes things very difficult for me. I need this to just work automatically.
Is there any automated solution for this yet? Please say yes!
No, to our knowledge there is no solution, sorry
We tried to use the workshop in our couse last semester, and this was one of the reasons, we had to give up using Moodle in my course, and switch back to what we used before.
So glad to find this post! I was having the same issue. Even after I read the writer's text, I had difficulty finding the "lightbulb." I'm guessing this is because of the color scheme our school is using. In our case, the lightbulb was grayed out in a navy background, making it almost invisible. I'm good now, but I wonder how many others face the same predicament.
David, is automatic switching still online for 2.3?
To all others: if you have not voted for this feature please do so. http://tracker.moodle.org/browse/MDL-26099
Well, this is the reason I hesitated to transit to 2.x from 1.9. I did it foolish on this semester. I started to get sms and calls from my students about workshop. I have to do it for four workshops per week. I feel the great gap between the developers and teachers. Poor, Me.
I agree with you. I struggle with 2.0. I understand that this will be fixed in 2.3 but who and hell knows when 2.3 will come out.
Thanks Tim. I wasn't aware of this document. It doesn't list workshop but David has said that the update will be with 2.3.
Manly little changes never get listed on the roadmap. Really, the only way to find out about the is to search the tracker. Hopefully, when the time comes, someone will do that tracker search and write the release notes.
Thank you David
This isn't working for me in 2.3.1+ Build: 20120802 is anyone else having issues?
It doesn't work for me either, I tried it in two courses over a two month period. All the settings appeared to be correct, each course had 25+ workshops, it never automatically switched phase.
It was a real hassel because I kept forgetting to open the next phase and the windows were short.
Please bring back automatic switching of all phases (though I'll take any phase for now)
I'm sorry to hear that. I just tested it and it works as expected at my machine. Are you sure you followed the setup procedure correctly:
- "Submissions deadline" is defined
- "Switch to the next phase after the submissions deadline" is checked
- Workshop is in the Submissions phase
- In Settings > Workshop administration > Allocate submissions > Scheduled allocation, the scheduled allocation is enabled and configured. It's status is displayed as "To be executed on ..."
- The cron job is configured and running for your Moodle site (it's your admin's duty to do it but I believe you have it running - basically if you are receiving forum posts by email at your site, the cron is running).
Thank you! The problem was my cron job. it wasn't running, i am on a host and was trying to use the web interface. When I switched it to the command line insturctions it worked.
I would like to take a moment to plug full automation again. I have 4 classes, 9 workshops a week. Hundreds of students. It is nice to know that the asssesment phase will happen automatically, but I shudder to think that I still need to go in and click the button to compute the score and then click the button to close it. In my use case, I see no value in doing this manually. Perhaps there are other use cases where this makes sense. I would like it to return to the functionality of 1.9 where everything was automated.
Will there be a way to do this?
Will there be a way to do this?
I really don't want to promise anything here. As I already expressed several times in this forum, I can see big potential issues at the conceptual level with the fact that a machine (Workshop) writes final grades for students into the Gradebook and there is no human/teacher supervision over this process (even a formal one).
I can sense that this is a strong concern for you. Was this a problem for anyone back in 1.9 when the process was fully automated? It wasn't for me. I ran hundreds of assignments through the process and it was never an issue.
As I implemented it, there was a human touch: the student. The students are deeply concerned about their grades and if there was ever a problem with the grade, I was sure to be contacted by them. They look at the gradebook.
Out of hundreds of assignments, I did have one real problem come up. Through the luck of the draw, one student got 3 reviewers who did not do the reviews. So, the system gave the student a 0. It was a purely random occurance. The student contacted me and I graded the assignment. This happened in version 2.2 and I did not catch it when I clicked the button to grade the assignments.
May I suggest that the "human touch" of advancing the phases manually is a false security. I have 2 dozen workshops to advance each week, so I simply open them all at once in tabs and click through them. There is nothing inherent about me manually clicking them that requires me to review each student's grade to ensure that it is accurate. And there is nothing in the interface that would help me to know a bad grade just by looking at it.
May I suggest that complete automation be added as a plugin behavior to the workshop. It need not be there by default, but those that would benefit by it greatly could install it.
I appreciate you keeping an open mind about this and keeping the dialoge going. I feel like we can be creative and come up with a solution that meets everyone's interests.
Thanks for your input Jared.
You raised interesting points. I agree that forcing the user to proceed just a formal approval without any actual review is useless. Just to clarify - I am not concerned about grades for submission in workshop. Their calculation is simple (weighted mean of all received assessments) and there is really not much to do and check.
What keeps me reserved from the whole idea of automatic phase switching is the grade for assessment. Its value is actually a sign for the teacher to check the reliability of assessments. Teachers were expected to evaluate those cases where the calculated grade for assessment is pretty low and make sure that such assessment does not bias the calculated grade for submission much. Do you use grade for assessments and if so, what typical value do you set? (compared to the grade for submission, is the portion like 50/50 or 80/20 or what?)
Thinking about this issue more, I actually came to another out-of-this-topic idea. There might be a column in the Grading evaluation report that would display something like "reliability index" for each submission. Such an index might be defined as the lowest grade for assessment that some reviewer got. So if there is a submission assessed by three reviewers and the one of the reviewers got grade for assessment, say, 40%, the reliability index for that submission would be 0.4. It would help teacher to easily identify "malicious" submissions they should pay attention to.
The other case that happened to you (a submission with no assessment) could be better handled by Workshop, too - I agree. Eg. to highlight submissions with no assessment in a special section at the grading evaluation screen.
I actually, evaluate the assessment grades. I think a reliability index would be incredibly helpful. One thing I have wanted is an alternative way to have the assessment grade calculated. For instance, could there be some kind of formula that is used that is based on how well the students did the example assessments. But I agree, there is a need to monitor the assessments. I just think that the software should provide the choice to the teacher not force a certain choice.
Regarding the computed grade for assessment, I personally don't find it too useful. I set it to Very Lax and it still causes frustration for my students. They get very irritatated that if they don't agree with the other two reviewer they loose points. So they then get in the position of trying to guess what the other reviewers will do so they won't loose points. To me, I wish I could just give them a standard 5pts for giving the review scores, no matter what. My rubric is simple, 0 points for 0 work, 4 pts for poor work, 8 pts for acceptable work, and 10 pts for amazing work. And my assignments are more like journal entries than research papers. Students don't have trouble giving an accurate score.
I do hundreds of workshops a week (I will be doubling it next semester, one of my new classes has 260 students), so I would prefer a system that pushed more of the load to the students, rather than requiring me to look at each response. I rely on the students to give me a reliability check of assessments. It is their grade and they are motivated to look at it. They are looking at just one score, where I would need to look at hundreds. Over the course of a semester each student is evaluated 105 times so I get a nice statistically accurate assessment of their abilities, even though sometimes they are scored too high and sometimes too low. The reviews and submissions are all anynomous, I don't have any concern with malicious assessments. Over the course of the semester, the class average of all assessments is always about 8.2/10 or so, indicating the the system works. If anything, scores are a bit inflated.
I do give a grade for the submission and the assessment: 10 for submission/5 for assessment, or 15 points total. In the gradebook I use the calculated row function to merge the two scores back into one score again so the students can see how they did.
More important than the computed grade for assessment, what I wish the system did was require comments. Handing out points is easy, but I want students to be required to form sentences around their praise and objections for the paper they are reviewing. This improves the quality of the review and gives the writer feedback she can use to improve. I often wish there was a simple check on the comments box in the review section that prevented the assessment submission if there is no comment. Right now I can't easily measure it so I just encourage my students to do it.
Another feature I sometimes (but only sometimes) think would be helpful is a method for the writer to report a bad review. Perhaps a simple button that sends an email. Right now, I just have students send me an email if they feel that the average of 3 scores is unfair. They have to feel robbed enough to do the work to send me the email. In this case, I toss out the scores and grade the paper myself, for better or worse. Of course, if this was too easy then it would create unnecessary work for me.
WARNING ** TANGENT** While I am thinking about it, I wish I could set how big the text box is. Is there a way to set it's size for all workshops? Or perhaps a character or word count. I want it to be easy for my students to know when they have written the appropriate amount of material, not to much and not too little. **END TANGENT**
Wow, that was a long winded response. But I care a lot about the Workshop and use it a ton. If it were not for workshop, I would probably not be using Moodle. Most of the rest of what I use Moodle for can be done in my school's LMS. I have my own server so that I can employ workshop in my classes. It allows me to have higher learning assignments where work load would have made it impractical otherwise.
Thanks for your hard work! This is by far the best peer review tool I have found.
I actually would love a feature that would allow students to notify when the comments provide are not helpful. A big part of the learning process is for students not just rate but explain how the paper could be improved. I would like to be able to adjust grades of students who provide superficial comments. Like you I do a lot of workshops for large classes and it is impossible to monitor the feedback that students provide.
I wanted to check in on this topic. David, have you any new thoughts? Any chance full automation will make it into 2.4?
Thank you David!
I noticed that in the 2.4 version of workshop the plug in structure has improved. Will that make auto phase switching easier?
Did auto phase switching all the way to closed phase make it into 2.5?
No, it did not.
Is it still a goal? If so, any idea when it might happen? I constantly forget to go "click the button" and my students get ancy when they can't see their grades.
Vote for automation!
This week I tried the Workshop first time. I assumed that it would automatically go forward and was surprised when got messages from the students. I found this thread by searching in the Internet.
Just been badly burned by this one as set the final stage to change I thought automatically - now looking for an alternative tool after a series of serious issues with workshop requiring changes to grading and the activity.
The idea was simple: submit a draft assignment before the deadline, random allocation to a peer, write comments as a review, receive comments after deadline.
No automation meant it didn't switch over as advertised - setting dates for phase changes didn't work. However more problematic was that no comments were shared even after manual phase change intervention.
Specifically set it to grading strategy: comments. However comments were judged by the system as incomplete/insufficient and not shared even after manual intervention to change phase as they weren't complete as there was no criteria. That was the point - no grade, just comments!
Retrofitting a pass/fail and asking people to pass led to some comments being lost, others making it etc. and a lot have gone missing and we've had to fall back on email.
So can you not have text only reviewing/comments but MUST rank or grade your peers? Or was there some other setting that caused the issue here? What was it? Grading would have completely defeated the purpose of the exercise. Retrofitting it was problematic and I don't think anyone's going to support using this tool again - pity, good idea just pretty incomprehensible and unexpected and unclear in its implementation and documentation.
Problematic - anyone know of another tool to use that does do automatic allocation and switch of stages + written reviews only without having to give a grade? PeerMark?
comments were judged by the system as incomplete/insufficient and not shared
Sorry, I don't understand this. The grading strategy "Comment" let's peer-reviewers to put just a comment on the given aspects (as defined in the assessment form). These comments are made available to the author of the submission once the workshop is closed. What do you mean by "judged as incomplete"?
Good question - I somehow doubt it but live in hope... Been requested for so long but all those requesting it were apparently wrong...
This is a very long thread so I am sorry if I have misunderstood the problems you are having, but just to point out for anyone who gets here vis Google that switching phases automatically in the workshop has been available since 2.3 - see Workshop FAQ
I believe it only switches some phases automatically. it is still necessary to switch to the evaluate and close phase manually. If I am wrong, I would love to know, but I don't think I am.