Seymour Papert offered a good critique of this type of thinking/research in "Computer criticism vs. Technocentric thinking
However, such turns of phrase often betray a tendency to think of "computers" and of "LOGO" as agents that act directly on thinking and learning; they betray a tendency to reduce what are really the most important components of educational situations -- people and cultures -- to a secondary, facilitating role (1).
The context for human development is always a culture, never an isolated technology. In the presence of computers, cultures might change and with them people's ways of learning and thinking. But if you want to understand (or influence) the change, you have to center your attention on the culture -- not on the computer.
In this particular example, the reliance on standardised tests as a measure of outcomes gives some hints to the culture at play.
He makes this argument in response to studies that showed limited impact of the Logo programming language. He uses this analogy
"Does LOGO work?" "Is LOGO good for learning this or that?" All these turns of speech are signs of the technocentric stage of computer discourse.
Consider for a moment some questions that are "obviously" absurd. Does wood produce good houses? If I built a house out of wood and it fell down, would this show that wood does not produce good houses? Do hammers and saws produce good furniture? These betray themselves as technocentric questions by ignoring people and the elements only people can introduce: skill, design, aesthetics.
My initial reaction to reading Michael's description of the study (I haven't look at the study) was to think of a problem with most learning analytics research. i.e. it's driven more by the availability of the data and algorithms, rather than the usefulness of any insight that can be gained.
In the same piece Papert critiques the "treatment model" of research. Perhaps an indication to look for alternatives to such research? Even though it's much easier to do.
In terms of Moodle and improving teaching practice, perhaps this offers one suggestion
Stated abstractly, the two studies have the same explicit intention: the children are to be given "programming"-- and the purpose of the experiments is to see what happens. But there is no such thing as "programming-in-general." These children are not given "programming." They are given LOGO. But there is no such thing as "LOGO-in-general" either. The children encounter LOGO in a particular way, in a particular relationship to other people, teachers, peer mentors, and friends. (4) They don't encounter a thing, they encounter a culture.
Measuring the impact of "Moodle" or "computers" on education is pointless. There is no such thing as X-in-general. Instead it's the culture into which X is introduced that is important. For me, raising questions such as