Yes, over-engagement can also be a problem. Level of engagement is an indicator, and however you measure it, there is probably a non-linear relationship between engagement and success in the course. (There is also a non-linear relationship between engagement and teacher visibility and number of activities.)

Currently we only have an indicator that measures how much of the learning cycle a student goes through, which is fairly coarse-grained— there are 5 levels, between “viewing” at level 1 and “revise and resubmit” at level 5. If we were to add an indicator for “time spent” or “number of actions” we would probably want our model to look for a non-linear relationship. This is somewhat complicated by the need to scale and center indicators to values between -1 and 1. I have suggested a logarithmic scale for open-ended data types like logins or activity views, but we might lose too much range on the upper end if we are trying to detect over-engagement.

Good topic! Does anyone else have any suggestions for how to handle this?