General plugins

Heatmap plugin

 
My mug
Heatmap plugin
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

Hi, all.

I've created a block that, when added to a course, allows the overlay of a heatmap on activities. The heatmap is visible to teachers/admins only and can be toggled on and off.

The plugin is inspired by the Moodle Activity Viewer and uses similar queries to the Activity report.

I'd be grateful if people were willing to test the plugin and provide feedback before I submit to the Plugins directory. You can clone my Github repo or download the zip.


 
Average of ratings: Useful (6)
Picture of Ralf Brauckhoff
Re: Heatmap plugin
 

Hallo, that sounds good. Can you tell me, for what version of Moodle this is?

Ralf

 
Average of ratings: -
Picture of Marcus Green
Re: Heatmap plugin
Core developersParticularly helpful MoodlersPlugin developersTesters

Just installed on 3.0 and it looks v.nice indeed. Mmmmmm

 
Average of ratings: -
Picture of Usman Asar
Re: Heatmap plugin
Particularly helpful MoodlersPlugin developers

Michael, that's is awesome idea! Congrats on that.

Will guests be able to define color customization for heat strengths? or something like legend defined in the block?

Also, guest clicks included? or is it just enrolled users?

 
Average of ratings: -
John Provasnik at Moodle Moot US
Re: Heatmap plugin
Particularly helpful Moodlers

Color customization would be nice for people who cannot see shades of red : )

 
Average of ratings: -
Picture of Mike Churchward
Re: Heatmap plugin
Core developersParticularly helpful MoodlersPlugin developersPlugins guardiansTesters

Hi Michael -

This is indeed an interesting plugin. I am also looking at it to help me get acquainted with some of the newer Moodle development features it uses.

In the code, it determines the earliest log date to use based on the oldest date it finds in the logs of any of the readers it finds. If it has both the legacy system and the internal system, it sets the time to the earliest of the minimum time in either the legacy or internal system, and stores that in the variable "$minlog". But I'm not sure that the variable is ever used after that.
I believe, if the situation comes up that the lower internal time is the internal system, it should only use the internal system. But I think the code currently ignores that it found "$minlog" to be the internal system time, and gets the legacy logs anyway. I believe you do not have to execute the block of code in block_heatmap:get_content, around lines 172 to 195 if you determined that the internal reader has an earlier start date (in lines 164 to 169).

I believe the block of code:

            // If we are using the internal reader check the minimum time in that table.
if ($useinternalreader) {
// If new log table has older data then don't use the minimum time obtained from the legacy table.
if (empty($minlog) || ($minloginternalreader <= $minlog)) {
$minlog = $minloginternalreader;
}
}

could be rewritten as:

            // If we are using the internal reader check the minimum time in that table.
if ($useinternalreader) {
// If new log table has older data then don't use the minimum time obtained from the legacy table.
if (empty($minlog) || ($minloginternalreader <= $minlog)) {
$uselegacyreader = false;
}
}

and avoid an unneeded query.

But, I also may be misunderstanding what is happening in there.

Also, I believe that the block fetches new data when the cached data is older than five minutes from the execution time. The SQL statements that retrieve the log data does not seem to filter on time data though. I believe, every time new log data is fetched from the database, it is grabbing everything that exists in the course. If the course is large and has been running for a long time, this could be a lot of data. I think there needs to be a "WHERE" clause based on the minimum time? Also, should the "get_records_sql" statements use "recordsets" to deal with potentially large data sets being returned?

I look forward to playing with this a bit more.

mike

 
Average of ratings: -
Picture of Marcus Green
Re: Heatmap plugin
Core developersParticularly helpful MoodlersPlugin developersTesters

I have looked at this plugin a little more and am really impressed by the idea. Like Mike, I did wonder what the performance implications might be on large courses.

 
Average of ratings: -
Picture of Mike Churchward
Re: Heatmap plugin
Core developersParticularly helpful MoodlersPlugin developersPlugins guardiansTesters

Hi Michael -

I made my suggestions as a pull request if its helpful.

mike

 
Average of ratings: -
My mug
Re: Heatmap plugin
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

Thanks for having a look, everyone.

The minimum version needed to run this is Moodle 2.7 for the caching and logging support. I'll set that as the minimum version in the version.php.

The queries for gathering the hit counts and unique users are copied from the Activity report, updated mostly by Petr. They are pretty efficient in that they achieve the gathering of results in a single query. Constraining the results to a limited period might reduce the impact slightly. I'll look at starting at the beginning of the course, if that is set. I don't think building up results in the cache would be reliable and it won't improve efficiency much. The caching was added because in the Activity report, when you hit the page, it's a single query, but people will tend to view the course page multiple times in a row, so caching the query result saves hitting the log table too often. I've arbitrarily set a 5min lifespan for the cache. I might add a global setting so that can be played with.

The way the queries check both the standard log and the legacy log was done to account for the possibility that someone could start using the legacy log, then part-way through the teaching period, upgrade and transition to the new standard log, after which the results of both queries would need to be combined (which is probably less efficient than either of the queries themselves). To be honest, I don't think this would happen often. I might simplify the checking so that it only checks a single log, starting with the standard log, falling back to the legacy log and if that fails, giving up.

Gathering the results into a recordset will not help in this case. Although the logs can be big, the results delivered by the query are not (one result per activity in the course), so a recordset is not needed. Copying from a recordset into an array nullifies the benefit of using a recordset anyway.

 
Average of ratings: -
Picture of Mike Churchward
Re: Heatmap plugin
Core developersParticularly helpful MoodlersPlugin developersPlugins guardiansTesters

Hi Michael -

Regarding your comments on recordsets and single queries, you're right. I forgot about the "GROUP BY" part that would minimize the number of records to the number of activities in the course. So, you are right. In this case, recordsets would make it worse.

Regarding the legacy log checking, I agree that combining the two makes sense for the situation you described. The thing i was pointing out is that the code was performing a query on the legacy log if it was used on the site, even if it was determined that the internal log had earlier dates than the legacy log. In that situation, the data (if any) retrieved from the legacy log query would not actually be used. But the new code excludes that possibility by only using one or the other, so I guess that's moot now.

I'm still concerned about the query performance on large sites though, as my experience with very large sites show that queries on the log tables can have significant impact. I like the idea of caching, since in general, the vast majority of the data that is being queried won't change from view to view. But I think I was misunderstanding how the cache worked. I was assuming that the data could be added to. I think you are saying that it is completely replaced when the queries are rerun?

This block wouldn't be used by every user of the course, so that would minimize any impact as well.

Can I confirm with you that the code this is based on is the code in "blocks/recent_activity"? I am going to see if I can test this on a very large data set to see if there are any "real" impacts.

mike

 
Average of ratings: -
My mug
Re: Heatmap plugin
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

The caching is to avoid hitting the log too often. I don't think the queries could be made any more efficient than they are now and it would only be privileged users seeing the heatmap (and sharing the cache per course). Let us know how your testing goes; mine showed negligible impact.

The queries come from the Activity report (report_outline). Check the index.php file. The minimum check was to determine when the first entry was, but I've removed that from the heatmap code.

 
Average of ratings: -
Picture of Mike Churchward
Re: Heatmap plugin
Core developersParticularly helpful MoodlersPlugin developersPlugins guardiansTesters

Hi Michael -

I tested the Activity Report on a course with more than 700 users, approximately 30 activities and more than six months of logs. It was slow - took about 30-40 seconds to run (before caching) - but I don't think it hurt the site performance. So, I think that performance impact of the Heatmap will not be significant.

mike

 
Average of ratings: -
My mug
Re: Heatmap plugin
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

Hi, Mike (Churchwood).

The performance impact of these queries would probably have more to do with the total log size, rather than the size of the course.

I also wonder what part of that 30-40 seconds were taken up by the queries.

The Activity Report doesn't use caching.

 
Average of ratings: -
My mug
Re: Heatmap plugin
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

In relation to the colours, at the moment these are fixed in the code. I'm not sure these really need to be customisable. I hope a legend is not needed as the heatmap also injects the counts into the course page.

The counts are for all users. When looking at the legacy logs it checks for views. With the newer logging it looks for "participation" events.

 
Average of ratings: -
My mug
Re: Heatmap plugin
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

I've added a global setting to control how far back in the logs the block will check (since the start of the course or forever). I've also added a global setting to control the lifespan of the cache.

I've moved the colour settings to the styles.css file so they can be overridden or themed.

I've tested this with the Topics, Weekly and Collapsed topics formats.

I'd appreciate feedback on whether this works for you.

 
Average of ratings: -
Picture of Marcus Green
Re: Heatmap plugin
Core developersParticularly helpful MoodlersPlugin developersTesters

Excellent work on the colours, just becauase Geeks think that colour is a minor detail ....... (fill in geek stereotyping here)

 
Average of ratings: -
Picture of Marcus Green
Re: Heatmap plugin
Core developersParticularly helpful MoodlersPlugin developersTesters

Actually, excellent idea to put the colours in CSS, but unfortunately when I just upgraded there were no longer any colours sad

 
Average of ratings: -
Picture of Marcus Green
Re: Heatmap plugin
Core developersParticularly helpful MoodlersPlugin developersTesters

Ahh, probably a problem between the chair and keyboard, here is some lovely linear-gradient lovelyness put in the css file.


 
Average of ratings: -
My mug
Re: Heatmap plugin
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

Pretty. I wonder if you can make the look like they are on fire. 

Please share what you come up with. 

For those trying the latest, I should have mentioned that you have to run the upgrade from the Notifications page after grabbing the new code. 

 
Average of ratings: -
Picture of AL Rachels
Re: Heatmap plugin
Core developersParticularly helpful MoodlersPlugin developersTesters

Not perfect, but getting there.


.block_heatmap_heat_4 {
    background: red; /* For browsers that do not support gradients */
    background: -webkit-linear-gradient(yellow, red); /* For Safari 5.1 to 6.0 */
    background: -o-linear-gradient(yellow, red); /* For Opera 11.1 to 12.0 */
    background: -moz-linear-gradient(yellow, red); /* For Firefox 3.6 to 15 */
    background: linear-gradient(yellow, red); /* Standard syntax */
}

 
Average of ratings: Useful (1)
Picture of AL Rachels
Re: Heatmap plugin
Core developersParticularly helpful MoodlersPlugin developersTesters

Nice plugin. I really like the idea and I would imagine that for any site this provides "valuable" information.

I do wish I could install once on the front page and make it appear on any site course page. Even a small Moodle site, like mine, to check all my courses would make me have to install it dozens and dozens of times. If a front page install would make too big of an impact on performance, maybe the default could be set to toggled off?

 
Average of ratings: -
Picture of Marcus Green
Re: Heatmap plugin
Core developersParticularly helpful MoodlersPlugin developersTesters

One way to make make it easy to appear everywhere would be to put the functionality into a filter rather than a block.

 
Average of ratings: -
My mug
Re: Heatmap plugin
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

Thanks for playing with the colours, AL.

I can certainly make it available on the site home. That's a good idea and I didn't consider adding it across the site.

A filter could possibly achieve the same thing, but it would remove the potential for toggling and extra bits in the block.

 
Average of ratings: -
Picture of Marcus Green
Re: Heatmap plugin
Core developersParticularly helpful MoodlersPlugin developersTesters

Ahh yes, I forgot about the toggle thing.

 
Average of ratings: -
My mug
Re: Heatmap plugin
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

Hi, all.

Thanks to everyone for the feedback.

I've held back from adding scorch marks on the course page and stuck to some simple, flat colours. You can theme that still if you wish.

I've changed counts to colour-coded icons so the heatmap looks like this...



There are now three global settings: one for controlling the cache lifespan, one for controlling how far back the log search will go and one to control what is injected into the page (background colour, icons or both).



I've submitted this plugin to the Plugins Directory and it will take a bit of time to get approved. In the meantime, please keep testing this out and providing feedback. You can still pull from Github or download the zip.

I also made the video below to demonstrate how the Heatmap works.


 
Average of ratings: -
Picture of Mike Nodding
Re: Heatmap plugin
 

in our dev so may be an issue with that environment, but can't get it to display.

2.7.12 (Build: 20160111)

added the block to course using clean theme but nothing is showing in the block.

capabilities seem ok although seemed to have trouble saving the view report one - eventually did for one with manager archetype but couldn't save it for our local admin. 

debugging:

Notice: Undefined variable: totalusers in /var/www/murdoch-dev/blocks/heatmap/block_heatmap.php on line 254
Notice: Undefined variable: totalusers in /var/www/murdoch-dev/blocks/heatmap/block_heatmap.php on line 255
Notice: Undefined variable: minviews in /var/www/murdoch-dev/blocks/heatmap/block_heatmap.php on line 258
Notice: Undefined variable: maxviews in /var/www/murdoch-dev/blocks/heatmap/block_heatmap.php on line 259


 
Average of ratings: -
My mug
Re: Heatmap plugin
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

Hi, Mike (Nodding).

Thanks for reporting that error. I suspect it was caused because the course you were viewing hadn't started yet and the queries were limited to start from the beginning of the course.

I've put in code to check for that and provide a meaningful response.

 
Average of ratings: -
Picture of Mike Nodding
Re: Heatmap plugin
 

I'll check updated code when possible but i did have course start date set ok and queries set to all. 

 
Average of ratings: -
My mug
Re: Heatmap plugin
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

Thanks, Mike.

Let me know how that goes.

 
Average of ratings: -
Randy Thornton
Re: Heatmap plugin
Documentation writersParticularly helpful Moodlers

I'd like to bring up some accessibility compliance concerns with a plugin so dependent on color for its semantics. Granted that there will not (typically) be as many teachers or assistant teachers in this situation as students, there's a lot more attention being paid to this in US higher ed now than formerly.

And since this is such an improvement in interface over the standard report, it would be nice to take that into account. While each line it has numbers for total and unique views, they are color coded too and not so prominent as the color.

Can this list be sorted by the number of total or unique views in addition to the course order of activities? How does this look in greyscale?

 
Average of ratings: -
My mug
Re: Heatmap plugin
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

Hi, Randy.

Yes, accessibility was one of my concerns when I started creating this and the main form of information being communicated was colour. That's why I added the total and unique views to each activity.

I don't think it would be good to re-order the course content. I think that would remove the context of the sections and people using non-visual browsers would possibly find that more confusing.

I could add more textual information, perhaps only for non-visual browsers, that represents the same thing as the colour. Currently the coloured items are divided into five usage bands and given one of five colours that correspond to that. We could note that band alongside the total and unique views; something like "Usage band: 20-40%". Alternately, we could add a rank to each item, such as "Rank: 3rd".

What do you think?

 
Average of ratings: -
My mug
Re: Heatmap plugin
Core developersMoodle Course Creator Certificate holdersPlugin developersTesters

For those interested, the Heatmap has now been approved and is available on the Plugins Directory. Hurray!

https://moodle.org/plugins/block_heatmap

 
Average of ratings: -