Engagement Analytics block/report available on plugins directory

Engagement Analytics block/report available on plugins directory

by Phillip Dawson -
Number of replies: 28

I'm delighted to announce today that the Engagement Analytics suite is now available at:

http://moodle.org/plugins/browse.php?list=set&id=20

Engagement Analytics provides a quick 'traffic light' snapshot view of the ten students in your course you should be concerned about based on a configurable risk calculation. It also links through to a report on all students in your course, as well as a detailed explanation of how risk is calculated for a particular student.

Indicators of engagement/risk

The risk percentages shown in the block are the result of three indicators:

  • Login activity: how often, how recently, and how long are students logging in?
  • Forum activity: are students reading, posting, replying?
  • Assessment activity: are students submitting their assessed work, and are they submitting on time?

These indicators were selected as the most evidence-based following a review of the literature. How important are logins vs forums vs assessments in predicting success in your course? It depends on the course, so Engagement Analytics allows you to configure the weighting of each - eg 60% logins; 30% assessment; 10% forums. Inside each indicator you can modify how it calculates risk: the number of logins before someone is not at risk; how many days before an assessment is too late and therefore risky; the number of forum posts you expect students to write each week; etc.

Write your own indicators

You can write your own indicators using our documented indicator architecture, and they'll have equal status with our login/forum/assessment indicators. So if you think that completion tracking, diagnostic quiz scores, or downloading files are the real indicators of student engagement, then write an indicator! If you've got an idea feel free to propose here.

Pedagogy/Philosophy of analytics

Engagement Analytics takes a 'Level 3' approach to learning and teaching (Biggs, 1999), and views learning as "what the student does" (as opposed to "what the student is" or "what the teacher does"). By understanding what students are doing we can better guide them to do the things we know will help them learn. It does not focus on "what the student is" (age;IQ;GPA;etc). Of course, that doesn't stop you from writing an indicator that plugs in to your student management system though. As an education researcher I encourage you to have a read of the Biggs article if you haven't yet, as it has some insights that are useful for analytics.

To make the most of Engagment Analytics, a teacher needs to specify what they think learning is in their course - ideally they are already communicating this to their students anyway. Engagement Analytics is not an AI/SciFi analytics tool; its usefulness is only in knowing if students are doing what you have specified you want! It also won't follow up students automagically; you know your students best, and a carefully crafted email/phone call/message should work better than a robotic email.

Help us make it better

Please let us know if you have any issues, ideas or comments - ideally over the next few months while we still have some funds left ;)

Many thanks to:

  • The NetSpot Innovation Fund for generous funding
  • NetSpot developers Ashley Holman & Adam Olley for writing the code, and Kim Edgar for project management / documentation support
  • The expanded project team of Jacques van der Meer, Angela Carbone, Emily Spencer, Robert Nelson, Hariz Halilovich, Keryn Pratt, Tom Apperley & Nathan Bailey

Cheers

Dr Phillip Dawson, Monash University, Melbourne Australia

Average of ratings: Useful (6)
In reply to Phillip Dawson

Re: Engagement Analytics block/report available on plugins directory

by Ed Main -

We had a look at this on our test site . . . and now have it on our production site - a GREAT addition to "our" world where teachers may be supporting numerous courses with many students. Being able to glance at a quick visual to see who is "in danger" has already proven to be a time saver. Would love to see the ability to quickly add/subtract course items to the list of items that are used . . . as it now stands, again, this is a great addition!!   big grin

Average of ratings: -
In reply to Ed Main

Re: Engagement Analytics block/report available on plugins directory

by Edd Talbot -

I hope this is the right place.

I have a bare bones test Moodle that this works perfectly on.

 

However, on the production system, no stats are being generated. I'm unsure where to look.

Both systems live on the same Windows server so the assumption is that the server configuration is correct.

Are there any known conflicts or setup requirements?

Thanks,

Edd

Average of ratings: -
In reply to Edd Talbot

Re: Engagement Analytics block/report available on plugins directory

by Adam Olley -
Picture of Core developers Picture of Plugin developers

First I'll assume it's the same version of Moodle, just a different environment (bare-bones test and prod).

What version of moodle is it btw?

I assume you've set up the indicators similar to what you did on the bare-bones one?

Average of ratings: -
In reply to Adam Olley

Re: Engagement Analytics block/report available on plugins directory

by Edd Talbot -

Hi Adam,

We are running 2.4.0.

As far as set up, we've used the same installation settings cache 5 minutes, monitoring student role.

However, on an individual course:

the test site displays the settings correctly for editing via reports->engagement analytics->update settings

The production site displays only the top 'Weighting' category i.e. The assessment, forum and login activity settings are not visible.

I've switched both sites back to the standard theme and it made no difference.

Any thoughts on where else to look would be greatly appreciated.

Thanks,

Edd

Average of ratings: -
In reply to Edd Talbot

Re: Engagement Analytics block/report available on plugins directory

by Eric Malone -
We're on 2.4 as well (lamp) and it appears there are some compatibility kinks. The percentages work fine under reports but the block doesn't display anything.
Average of ratings: -
In reply to Eric Malone

Re: Engagement Analytics block/report available on plugins directory

by Patrick G -

We were having the same problem: Moodle 2.4.1+ (Build: 20130131); LAMP server; report works fine, including display and configuration; block shows up blank when course editing is turned on, and disappears when I turn off course editing. This is on a test server (testing 2.4 before upgrading), and I saw that cron hadn't been run in more than 24 hours. So I ran it (clicked on the link on the Admin Notifications page); checked the course and the Engagement Analytics block was still empty; waited 5 minutes for the cache to clear; reloaded the course page; still nothing in the block; ran cron again; still noting in the block; waited 5 minutes; still nothing in the block; ran cron again; still nothing in the block.

I'll see if I find anything interesting in the logs for Apache, Moodle, or anywhere else. Much apprecitate any tips on where to look, what to look for, or what might need changing in the block or mod code to make this work.

Average of ratings: -
In reply to Phillip Dawson

Re: Engagement Analytics block/report available on plugins directory

by Enrico Canale -

Thanks for this little gem, Phil. I hope we get this into our LMS sooner rather than later. I enjoyed your reference to Biggs's book and want to suggest there could be a Theory X vs. Theory Y basis for making a variant of this block visible to students. It ties in with the idea of Social Norming - people like to belong, and behave accordingly, and this can be used to advantage (see HBR Readers Love this Article).

If the class averages were displayed alongside the individual scores, this could be used as a guide by each and every student for how they compare with the rest of the class. They may or may not wish to respond to what they see, of course.

Cheers
Ric

Average of ratings: -
In reply to Phillip Dawson

Re: Engagement Analytics block/report available on plugins directory

by Michael Lewis -

This looks very helpful especially as more classroom content delivery moves online. I notice that two of the three components 'require Moodle 2.2' and one 'requires Moodle 2.3'. What are the plans to update the suite for 2.3 and future Moodle releases?

Thanks!

Average of ratings: -
In reply to Michael Lewis

Re: Engagement Analytics block/report available on plugins directory

by Adam Olley -
Picture of Core developers Picture of Plugin developers
Hi Michael, For the mod and block components, the reason there isn't a "M2.3" version is because the M2.2 versions work on 2.3 without any changes required. The report component however did have changes necessary to be compatible with M2.3. You can safely deploy the 2.2 mod, block and 2.3 report and it will all work on M2.3. Enjoy smile
Average of ratings: -
In reply to Phillip Dawson

Re: Engagement Analytics block/report available on plugins directory

by chris x -

Hi Phillip,

first thanks for a great and timely plugin.

I've been using it in a course and have found it useful for keeping tabs on students that are not completing activities that may lead them not becoming disengaged.

I work at a major VET provider that uses Moodle a lot. I'm in learning technology and we are currently trying to evaluate the effectiveness of number of learning anlaytics tools and approaches.

i would like the organisation to use the block a lot more, but first would like to test how accurate it is in predicting enagement/disengagement and what are the best settings to improve accuracy.

I would like to analyse the effectiveness of the engagement block in providing early warning of students that are likely to become disengaged by comparing the indicators from the block (r/o/g) with results of whether students have completed the course. We aim to do this by using the block with historical data from moodle logs and comparing it to the class results.

At them moment I'm trying to find a way of manipulating the scores from the block (would be at different times during the course say start, middle and 2nd last week of course) in Excel to conduct analysis.

What I'm missing are the algorithims that the block uses. I can't find this information anywhere online.

Are you able to tell me:

what the ranges are for the Red, Orange and Green lights.

how the scores are calucalted for the three variables (forum posts, time online, assignments completed).

This will help me to analyse the results from the block with the actual completion results.

many thanks

Chris

 

Average of ratings: -
In reply to Phillip Dawson

Re: Engagement Analytics block/report available on plugins directory

by Miriam Laidlaw -
Picture of Plugin developers

Hello there,

I have just tried installing this on a local Moodle 2.4.3+ site for testing. I downloaded all three components of this module:

  • mod_engagement
  • report_engagement
  • block_engagement

Unzipped each and put them into their respective folders on my Moodle installation (mod, report, blocks). I started up XAMPP on my computer and logged into my Moodle site.

The notifications screen all looked good and the modules installed well.

I then went to a course and tried to add the block. I got a blank screen. I went to my admin and turned on debugging, then returned to the course, still just a blank screen.

Currently, it seems that there is no way for me to get access back to the course without uninstalling the module.

Average of ratings: -
In reply to Miriam Laidlaw

Re: Engagement Analytics block/report available on plugins directory

by Rex Lorenzo -

Try turning on debugging on your Moodle install to get a better idea of what might be the problem.

Debugging messages can be enabled by an administrator in Settings > Site administration > Development > Debugging.

Average of ratings: -
In reply to Rex Lorenzo

Re: Engagement Analytics block/report available on plugins directory

by Tabitha Parker (was Roder) -

Debugging is on, but there was no error messages displaying when trying to access the course with the engagement block added to it (using Firefox).

Average of ratings: -
In reply to Tabitha Parker (was Roder)

Re: Engagement Analytics block/report available on plugins directory

by Adam Olley -
Picture of Core developers Picture of Plugin developers

Hi Tabitha,

Assuming you still get this with the latest versions I just uploaded, you may need to check the error log on the system itself. Depending what you're hosting your moodle on and with what provider, I can't really tell you where to find that log though.

Have you also got debugdisplay turned on?

Average of ratings: Useful (1)
In reply to Adam Olley

Re: Engagement Analytics block/report available on plugins directory

by Miriam Laidlaw -
Picture of Plugin developers

Hi Adam,

Tabitha was responding on my behalf, actually. We both work in the same office and were trying to get this module working on my local installation. Sorry if that didn't come across.

I will try again with the updates you have implemented. Will let you know how I get on.

Miriam

Average of ratings: -
In reply to Miriam Laidlaw

Re: Engagement Analytics block/report available on plugins directory

by Miriam Laidlaw -
Picture of Plugin developers

Hi there,

I downloaded and installed the version from github (thank you!), and that at least hasn't broken the course when I installed the block.

However, I'm trying to get my head around how to use this. The block itself is empty, vanishes when editing is turned off, and has nothing useful in the block configuration.

When I look at the MoodleDocs for this, the very first section on using this module says to go to Reports > Engagement Analytics

This is what I see when I get there:

Engagement analytics

Just a blank page. Not quite as blank as when the block broke the course, but nothing like the screenshots on the GoogleDocs page...

Ideas?

PS - I have debugging turned on and the debug messages set to DEVELOPER mode. Not getting anything from that.

Average of ratings: -
In reply to Miriam Laidlaw

Re: Engagement Analytics block/report available on plugins directory

by Adam Olley -
Picture of Core developers Picture of Plugin developers

I assume by 'debugging turned on' you mean 'debugdisplay' is turned on.

In moodle, blocks with no content don't appear to users when editing is turned off. The block is only empty when it thinks there's no indicators to show for anyone. Are there users enrolled in this course?

If so, do they have a role that matches one of the ticked roles under:

Home / ► Site administration / ► Plugins / ► Activity modules / ► Engagement analytics

On a standard moodle, the 'student' role should be checked by default.

Lastly, I assume your site has moodle cron running? If caching is setup in the settings for the activity, the cron needs to run to generate the indicator stats used.

Average of ratings: -
In reply to Miriam Laidlaw

Re: Engagement Analytics block/report available on plugins directory

by Adam Olley -
Picture of Core developers Picture of Plugin developers

Hi Miriam,

I've just updated the block and report plugins.

From github you can get the latest version for M2.4 from the master branch of each, or grap the latest version from the moodle.org/plugins repository.

I've fixed two issues, one with the block not having the new addinstance capability all blocks now need in 2.4+ and the block being unable to fetch the indicator statuses from the report module due to API changes with the plugin_manager.

Average of ratings: -
In reply to Adam Olley

Re: Engagement Analytics block/report available on plugins directory

by Yvonne Hamilton -

Thanks Adam for all your work recently and advice,

Its nice to see the block and reports displaying data in Moodle 2.4.4 once the latest plugin versions were installed. I was hoping the 2.4 version of these plugins would pickup the Moodle 2.3 assignment type in the assessment criteria but only the old 2.2 assignment types are showing for me. Is that expected with this latest release?

As I had the plugin set installed already, only the Block and Report plugins required updating and the Activity plugin did not. Is there an easy way to add other graded activities to the block without dipping into code?

Regards,

Yvonne

Average of ratings: -
In reply to Phillip Dawson

Re: Engagement Analytics block/report available on plugins directory

by Martin Agombar -

I'm trying to work out the feasibility of using analytics to track the progress of students across courses. For example, I might want to identify at risk students as they progress through a programme (not just a course).

Would I be right in thinking we are some way off this possibility? I can see the Engagement Analytics block will be very useful on a course by course basis but not across courses?

From elsewhere in the forums I can see that there is a logging project in progress, which might be the first stage in a more comprehensive analytics suite? How far off do you think we are from getting to the point where users can be tracked across courses?

Any advice or rough indications of timescales would really be appreciated.

Average of ratings: -
In reply to Martin Agombar

Re: Engagement Analytics block/report available on plugins directory

by Isabel Ashburner -

Hi Martin,  I was about to post an almost identical request!  Did you ever get a response to this query about using the block across a set of courses (i.e. a programme)?  This would also be very useful to us as we run fairly complex programmes on Moodle.

thanks

Isabel

Average of ratings: -
In reply to Phillip Dawson

Re: Engagement Analytics block/report available on plugins directory

by David Ha -

I have just tried this with Moodle 2.5.1 with the latest version from GIT hub.  Everything seems OK but the block doesn't display anything.

The report seems to work.  I was able to set weighting and such.

Any help is appreciated

Average of ratings: -
In reply to David Ha

Re: Engagement Analytics block/report available on plugins directory

by David Ha -

with debug turn on, here is the warning error

Did you remember to call setType() for 'id'? Defaulting to PARAM_RAW cleaning.
  • line 1303 of \lib\formslib.php: call to debugging()
  • line 281 of \lib\formslib.php: call to moodleform->detectMissingSetType()
  • line 202 of \lib\formslib.php: call to moodleform->_process_submission()
  • line 53 of \report\engagement\edit.php: call to moodleform->moodleform()
Did you remember to call setType() for 'weighting_assessment'? Defaulting to PARAM_RAW cleaning.
  • line 1303 of \lib\formslib.php: call to debugging()
  • line 281 of \lib\formslib.php: call to moodleform->detectMissingSetType()
  • line 202 of \lib\formslib.php: call to moodleform->_process_submission()
  • line 53 of \report\engagement\edit.php: call to moodleform->moodleform()
Did you remember to call setType() for 'weighting_forum'? Defaulting to PARAM_RAW cleaning.
  • line 1303 of \lib\formslib.php: call to debugging()
  • line 281 of \lib\formslib.php: call to moodleform->detectMissingSetType()
  • line 202 of \lib\formslib.php: call to moodleform->_process_submission()
  • line 53 of \report\engagement\edit.php: call to moodleform->moodleform()
Did you remember to call setType() for 'weighting_login'? Defaulting to PARAM_RAW cleaning.
  • line 1303 of \lib\formslib.php: call to debugging()
  • line 281 of \lib\formslib.php: call to moodleform->detectMissingSetType()
  • line 202 of \lib\formslib.php: call to moodleform->_process_submission()
  • line 53 of \report\engagement\edit.php: call to moodleform->moodleform()
Did you remember to call setType() for 'login_session_length'? Defaulting to PARAM_RAW cleaning.
  • line 1303 of \lib\formslib.php: call to debugging()
  • line 281 of \lib\formslib.php: call to moodleform->detectMissingSetType()
  • line 202 of \lib\formslib.php: call to moodleform->_process_submission()
  • line 53 of \report\engagement\edit.php: call to moodleform->moodleform()
Average of ratings: -
In reply to Phillip Dawson

Re: Engagement Analytics block/report available on plugins directory

by angela russo -

I am in version 2.5.2 and I cannot get to the configuration of the engagement analytic reports or managing the indicators. Under the settings column, there is nothing displayed.

Thank you for any help that you can give me.

 

Average of ratings: -
In reply to angela russo

Re: Engagement Analytics block/report available on plugins directory

by Bob Puffer -

Seems the latest version of EA block removed being able adopt site-wide settings if that is your problem.  If you're going at it from the course level then you should turn debugging on so you can see where the error is.

Average of ratings: -
In reply to Phillip Dawson

Re: Engagement Analytics block/report available on plugins directory

by Mary O'Toole -

Hi Philip,

Thanks for this useful plugin.  I'm using the traffic light system on the engagement analytics page- this is great.  But when I click into the View Course Report page- I get very confused.

There are 3 headings Asssessment Activity, Forum Activity and Login Activity.  I've uploaded a screengrab of one of the pages for your reference.

I'll take the first student's activity as an example -Assessment Activity 0% (0%), Forum activity 33 % (100%), Login 0% (0%).  What does the percentage in brackets mean?  If it is risk- is seems that it is the opposite to the first page- 33% activity on the forum gives you 100% risk?  Or is that what it means?

Also in this particular course, the lecturer has not started using the forum yet, so how does that come about?

Secondly, I am also using the Activity Completion page, which is also useful.  However, for the most part we've set the tickboxes so that students can tick them themselves.  This page is only a measure of students who have  actually ticked the boxes, right?  Is there any other handy report that measures who has completed what at this type of level?  

Many thanks,

Mary

Attachment Engagement Analytics explanation.PNG
Average of ratings: -
In reply to Mary O'Toole

Re: Engagement Analytics block/report available on plugins directory

by Susan Mangan -

Is anyone using the EA plug-in for 2.8?  I don't see a release for anything over 2.7 and it doesn't look as though there has been any development on this plug in over the past year?

TIA

Average of ratings: -
In reply to Susan Mangan

Re: Engagement Analytics block/report available on plugins directory

by Sali Kaceli -

We recently upgraded our Moodle system to 2.9.1 and this module will break  the installation and it was working fine in version 2.8.2+ for us.

What seems to happen is that if you update the module in 2.9.1, once you login, you will get a blank screen and that is it. No error message. Nothing. 

Fortunately, we had taken a snapshot of our virtualized install and were able to go back and rule out that it was this module. 


Any plans from the developers of the module to update it? 

Average of ratings: -