I'm delighted to announce today that the Engagement Analytics suite is now available at:
Engagement Analytics provides a quick 'traffic light' snapshot view of the ten students in your course you should be concerned about based on a configurable risk calculation. It also links through to a report on all students in your course, as well as a detailed explanation of how risk is calculated for a particular student.
Indicators of engagement/risk
The risk percentages shown in the block are the result of three indicators:
- Login activity: how often, how recently, and how long are students logging in?
- Forum activity: are students reading, posting, replying?
- Assessment activity: are students submitting their assessed work, and are they submitting on time?
These indicators were selected as the most evidence-based following a review of the literature. How important are logins vs forums vs assessments in predicting success in your course? It depends on the course, so Engagement Analytics allows you to configure the weighting of each - eg 60% logins; 30% assessment; 10% forums. Inside each indicator you can modify how it calculates risk: the number of logins before someone is not at risk; how many days before an assessment is too late and therefore risky; the number of forum posts you expect students to write each week; etc.
Write your own indicators
You can write your own indicators using our documented indicator architecture, and they'll have equal status with our login/forum/assessment indicators. So if you think that completion tracking, diagnostic quiz scores, or downloading files are the real indicators of student engagement, then write an indicator! If you've got an idea feel free to propose here.
Pedagogy/Philosophy of analytics
Engagement Analytics takes a 'Level 3' approach to learning and teaching (Biggs, 1999), and views learning as "what the student does" (as opposed to "what the student is" or "what the teacher does"). By understanding what students are doing we can better guide them to do the things we know will help them learn. It does not focus on "what the student is" (age;IQ;GPA;etc). Of course, that doesn't stop you from writing an indicator that plugs in to your student management system though. As an education researcher I encourage you to have a read of the Biggs article if you haven't yet, as it has some insights that are useful for analytics.
To make the most of Engagment Analytics, a teacher needs to specify what they think learning is in their course - ideally they are already communicating this to their students anyway. Engagement Analytics is not an AI/SciFi analytics tool; its usefulness is only in knowing if students are doing what you have specified you want! It also won't follow up students automagically; you know your students best, and a carefully crafted email/phone call/message should work better than a robotic email.
Help us make it better
Please let us know if you have any issues, ideas or comments - ideally over the next few months while we still have some funds left ;)
Many thanks to:
- The NetSpot Innovation Fund for generous funding
- NetSpot developers Ashley Holman & Adam Olley for writing the code, and Kim Edgar for project management / documentation support
- The expanded project team of Jacques van der Meer, Angela Carbone, Emily Spencer, Robert Nelson, Hariz Halilovich, Keryn Pratt, Tom Apperley & Nathan Bailey
Dr Phillip Dawson, Monash University, Melbourne Australia
We had a look at this on our test site . . . and now have it on our production site - a GREAT addition to "our" world where teachers may be supporting numerous courses with many students. Being able to glance at a quick visual to see who is "in danger" has already proven to be a time saver. Would love to see the ability to quickly add/subtract course items to the list of items that are used . . . as it now stands, again, this is a great addition!!
I hope this is the right place.
I have a bare bones test Moodle that this works perfectly on.
However, on the production system, no stats are being generated. I'm unsure where to look.
Both systems live on the same Windows server so the assumption is that the server configuration is correct.
Are there any known conflicts or setup requirements?
First I'll assume it's the same version of Moodle, just a different environment (bare-bones test and prod).
What version of moodle is it btw?
I assume you've set up the indicators similar to what you did on the bare-bones one?
We are running 2.4.0.
As far as set up, we've used the same installation settings cache 5 minutes, monitoring student role.
However, on an individual course:
the test site displays the settings correctly for editing via reports->engagement analytics->update settings
The production site displays only the top 'Weighting' category i.e. The assessment, forum and login activity settings are not visible.
I've switched both sites back to the standard theme and it made no difference.
Any thoughts on where else to look would be greatly appreciated.
We were having the same problem: Moodle 2.4.1+ (Build: 20130131); LAMP server; report works fine, including display and configuration; block shows up blank when course editing is turned on, and disappears when I turn off course editing. This is on a test server (testing 2.4 before upgrading), and I saw that cron hadn't been run in more than 24 hours. So I ran it (clicked on the link on the Admin Notifications page); checked the course and the Engagement Analytics block was still empty; waited 5 minutes for the cache to clear; reloaded the course page; still nothing in the block; ran cron again; still noting in the block; waited 5 minutes; still nothing in the block; ran cron again; still nothing in the block.
I'll see if I find anything interesting in the logs for Apache, Moodle, or anywhere else. Much apprecitate any tips on where to look, what to look for, or what might need changing in the block or mod code to make this work.
Thanks for this little gem, Phil. I hope we get this into our LMS sooner rather than later. I enjoyed your reference to Biggs's book and want to suggest there could be a Theory X vs. Theory Y basis for making a variant of this block visible to students. It ties in with the idea of Social Norming - people like to belong, and behave accordingly, and this can be used to advantage (see HBR Readers Love this Article).
If the class averages were displayed alongside the individual scores, this could be used as a guide by each and every student for how they compare with the rest of the class. They may or may not wish to respond to what they see, of course.
This looks very helpful especially as more classroom content delivery moves online. I notice that two of the three components 'require Moodle 2.2' and one 'requires Moodle 2.3'. What are the plans to update the suite for 2.3 and future Moodle releases?
first thanks for a great and timely plugin.
I've been using it in a course and have found it useful for keeping tabs on students that are not completing activities that may lead them not becoming disengaged.
I work at a major VET provider that uses Moodle a lot. I'm in learning technology and we are currently trying to evaluate the effectiveness of number of learning anlaytics tools and approaches.
i would like the organisation to use the block a lot more, but first would like to test how accurate it is in predicting enagement/disengagement and what are the best settings to improve accuracy.
I would like to analyse the effectiveness of the engagement block in providing early warning of students that are likely to become disengaged by comparing the indicators from the block (r/o/g) with results of whether students have completed the course. We aim to do this by using the block with historical data from moodle logs and comparing it to the class results.
At them moment I'm trying to find a way of manipulating the scores from the block (would be at different times during the course say start, middle and 2nd last week of course) in Excel to conduct analysis.
What I'm missing are the algorithims that the block uses. I can't find this information anywhere online.
Are you able to tell me:
what the ranges are for the Red, Orange and Green lights.
how the scores are calucalted for the three variables (forum posts, time online, assignments completed).
This will help me to analyse the results from the block with the actual completion results.
I have just tried installing this on a local Moodle 2.4.3+ site for testing. I downloaded all three components of this module:
Unzipped each and put them into their respective folders on my Moodle installation (mod, report, blocks). I started up XAMPP on my computer and logged into my Moodle site.
The notifications screen all looked good and the modules installed well.
I then went to a course and tried to add the block. I got a blank screen. I went to my admin and turned on debugging, then returned to the course, still just a blank screen.
Currently, it seems that there is no way for me to get access back to the course without uninstalling the module.
Assuming you still get this with the latest versions I just uploaded, you may need to check the error log on the system itself. Depending what you're hosting your moodle on and with what provider, I can't really tell you where to find that log though.
Have you also got debugdisplay turned on?
I downloaded and installed the version from github (thank you!), and that at least hasn't broken the course when I installed the block.
However, I'm trying to get my head around how to use this. The block itself is empty, vanishes when editing is turned off, and has nothing useful in the block configuration.
When I look at the MoodleDocs for this, the very first section on using this module says to go to Reports > Engagement Analytics
This is what I see when I get there:
Just a blank page. Not quite as blank as when the block broke the course, but nothing like the screenshots on the GoogleDocs page...
PS - I have debugging turned on and the debug messages set to DEVELOPER mode. Not getting anything from that.
I assume by 'debugging turned on' you mean 'debugdisplay' is turned on.
In moodle, blocks with no content don't appear to users when editing is turned off. The block is only empty when it thinks there's no indicators to show for anyone. Are there users enrolled in this course?
If so, do they have a role that matches one of the ticked roles under:
Home / ► Site administration / ► Plugins / ► Activity modules / ► Engagement analytics
On a standard moodle, the 'student' role should be checked by default.
Lastly, I assume your site has moodle cron running? If caching is setup in the settings for the activity, the cron needs to run to generate the indicator stats used.
I've just updated the block and report plugins.
From github you can get the latest version for M2.4 from the master branch of each, or grap the latest version from the moodle.org/plugins repository.
I've fixed two issues, one with the block not having the new addinstance capability all blocks now need in 2.4+ and the block being unable to fetch the indicator statuses from the report module due to API changes with the plugin_manager.
Thanks Adam for all your work recently and advice,
Its nice to see the block and reports displaying data in Moodle 2.4.4 once the latest plugin versions were installed. I was hoping the 2.4 version of these plugins would pickup the Moodle 2.3 assignment type in the assessment criteria but only the old 2.2 assignment types are showing for me. Is that expected with this latest release?
As I had the plugin set installed already, only the Block and Report plugins required updating and the Activity plugin did not. Is there an easy way to add other graded activities to the block without dipping into code?
I'm trying to work out the feasibility of using analytics to track the progress of students across courses. For example, I might want to identify at risk students as they progress through a programme (not just a course).
Would I be right in thinking we are some way off this possibility? I can see the Engagement Analytics block will be very useful on a course by course basis but not across courses?
From elsewhere in the forums I can see that there is a logging project in progress, which might be the first stage in a more comprehensive analytics suite? How far off do you think we are from getting to the point where users can be tracked across courses?
Any advice or rough indications of timescales would really be appreciated.
Hi Martin, I was about to post an almost identical request! Did you ever get a response to this query about using the block across a set of courses (i.e. a programme)? This would also be very useful to us as we run fairly complex programmes on Moodle.
I have just tried this with Moodle 2.5.1 with the latest version from GIT hub. Everything seems OK but the block doesn't display anything.
The report seems to work. I was able to set weighting and such.
Any help is appreciated
with debug turn on, here is the warning error
I am in version 2.5.2 and I cannot get to the configuration of the engagement analytic reports or managing the indicators. Under the settings column, there is nothing displayed.
Thank you for any help that you can give me.
Thanks for this useful plugin. I'm using the traffic light system on the engagement analytics page- this is great. But when I click into the View Course Report page- I get very confused.
There are 3 headings Asssessment Activity, Forum Activity and Login Activity. I've uploaded a screengrab of one of the pages for your reference.
I'll take the first student's activity as an example -Assessment Activity 0% (0%), Forum activity 33 % (100%), Login 0% (0%). What does the percentage in brackets mean? If it is risk- is seems that it is the opposite to the first page- 33% activity on the forum gives you 100% risk? Or is that what it means?
Also in this particular course, the lecturer has not started using the forum yet, so how does that come about?
Secondly, I am also using the Activity Completion page, which is also useful. However, for the most part we've set the tickboxes so that students can tick them themselves. This page is only a measure of students who have actually ticked the boxes, right? Is there any other handy report that measures who has completed what at this type of level?
We recently upgraded our Moodle system to 2.9.1 and this module will break the installation and it was working fine in version 2.8.2+ for us.
What seems to happen is that if you update the module in 2.9.1, once you login, you will get a blank screen and that is it. No error message. Nothing.
Fortunately, we had taken a snapshot of our virtualized install and were able to go back and rule out that it was this module.
Any plans from the developers of the module to update it?