Analytics and reporting

Hi Joseph,

Actually I was trying to show both, but to put a significant spin on it.

I recall a lecturer once telling me that if a student had not logged into Moodle to access the materials after one week of the course beginning, that they were already behind. This is not necessarily the case for every module, every course everywhere, but for that lecturer, that college, that course, it was.

I have seen and heard similar events from different places, however the context is king.

Only the teacher could probably know if the person who has not accessed it, is going to be at a disadvantage, or if the person who is accessing it a lot may be struggling.

The only thing that a report or analytics can say, is that the behaviour of those two people are not the norm, and that one is statistically quite far from the norm - ie, related to their peers.

So if that document is a crucial practical guidance information, or assessment guidelines which need to be read by a certain time, this is another bit of information which needs to be analysed.

Now yes, perhaps this should be a completion element, which can then be tracked as complete or not to identify the non-activity, but hyper activity cannot be tracked by that approach.

So how does one approach such things?

I love statistics, I love the buzz from analysis and the identification of habit and  or abnormalities.

I hope to be starting a test project where some analysis of relative analytis can be tested for specific criteria analysis

• relative to expectations
• relative to peers
• relative to overall individual performance

So if an expectation (e.g. study logs into Moodle within 1 day of course beginning) is not met, this is identified

So if a students (e.g. grade on an assignment) is significantly different to that of his peers on that activity it can be tracked

So if a students performance in one module is significantly different to his performance in another module, that this can be identified.

What someone does or interprets these as is a different matter, however they are statistically significant albeit, once context is added they may not be important, or they may be very important.

Great thread.  

I'm a self-confessed "data junkie", but I believe it's important to start with a realistic scope of what we hope to learn (accurately) from Analytics.  I've taken to paraphrasing George Siemens "Analytics doesn't provide answers, it just helps us to ask better questions."

If all teachers would use the gradebook, frequently and often (heck, even consistently while we're aspiring) we would get great predictive indicators of student success.   Until that time, clickstream / usage data seems like the best common data measures that we have.  Maybe we’re trying to solve a teaching/learning issue with technology – but that said, there is a lot that we can ask and advance our learning.

Here’s a chart pulled from usage data on a single course (373 students) that shows some of that usefulness, and some of the complexity in the data.  There’s an overall positive correlation between activity (hits and time) and grade (e.g. more time in Moodle, but students who are Pell-Eligible  (e.g. low income) spend much more time than students who are not Pell-eligible. 

So in this example, the amount of time v. grade is positive in some cases, but possibly problematic for others. 

What does this mean?  Is this good or bad?  That’s not a question we can answer with this data – just the what, not the “why” or the “so what”. 

That said, we’re beginning our Moodle analytics effort with more robust usage measures around adoption.  Looks like configurable reports might be a good step in that direction, maybe we could decide on a few to get “adopted” by core?  

Martin, I agree that there are many aspects which are not about learning outcomes specifically, but more about behaviour and performance outcomes.

Generally speaking,knowing that a student

- accessed material does not indicate that they know it,

- not accessing it does not indicate that they do not know it

- accessing it a lot does not mean that they dont get it, maybe it fails to download for them, or you dont know how to take a local copy,  or cant

However it does help lead to other questions and maybe other answers

Same as if someone has not logged into their online course area within a week of starting, and if it is central to the course, it could be problematic, maybe they dont know the url, cant find it, don;t know what password to log in with, cant follow the guidelines...

I think analytics goes beyond just grade or pure learning outcomes into performance and behaviour.

How often or long someone has looked at materials doesn't tell you if they've learned... but it may warn you that they're having trouble learning. We use Moodle for asynchronous distance learning, and one of the most difficult problems our instructors face is knowing how the students are doing before they achieve or fail to achieve an outcome. Outcomes are inherently summative; the learning process also needs to include attention to formative indicators.

I'd like to see a way for instructors to rate resources and activities in importance beyond simply marking them to collect completion or not, so students can see which resources are most important and instructors can see if students are paying attention appropriately. I'd also like to see built-in tools to allow students to rate the helpfulness of resources, and possibly ways to automatically sort resources by student ratings. (For that matter, I'd like to see more formalized ways for students to contribute and rate resources, but I'm planning to use Database for that.)