Use of Power Law and Standards-Based Assessment

Use of Power Law and Standards-Based Assessment

by Tim Zietz -
Number of replies: 10

I've been very impressed by Robert Marzano's approach to grading. Has this caught anyone's interest? He's currently getting tremendous attention in the US because of his meta-analysis of the past twenty five years of educational research.

Only ONE other vendor has picked up on this trend in education!

Here's the readings if you're interested. A really fresh approach to assessment (which is not new, but few have noticed how important it is to standards-based education.)

"Excel statistical functions: TREND." Microsoft Support. N.p., n.d. Web. 14 Sept. 2010. <http://support.microsoft.com/kb/828801>;.

Marzano, Robert J. Transforming Classroom Grading. Alexandria, VA: Association For Supervision & Curriculum Development, 2000. Print.

Marzano, Robert J. Formative Assessment and Standards-Based Grading: Classroom Strategies That Work. Bloomington: Solution Tree, 2009. Print.

Average of ratings: -
In reply to Tim Zietz

Re: Use of Power Law and Standards-Based Assessment

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Sounds potentially interesting.

Can one read a summary online? Are any of his papers published in proper journals (so I can get them through the OU library)?
In reply to Tim Hunt

Re: Use of Power Law and Standards-Based Assessment

by Tim Zietz -
I cannot speak to the OU library question directly. He is extensively published and has his own website. Google Robert Marzano.
In reply to Tim Hunt

Re: Use of Power Law and Standards-Based Assessment

by Joseph Rézeau -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers Picture of Translators
In reply to Joseph Rézeau

Re: Use of Power Law and Standards-Based Assessment

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
That list does not contain anything very recent, nor about Assessment.

The OU library is a pretty good University library. We have subscriptions to a lot of academic journals.

I tried both Google, and Google Scholar, but did not find anything that seemed worth reading.

His website looks like a horrible bit of self-promotion. Perhaps that is the kind of website you have to have if you are an American Entrepreneur, but it puts me right off.

So, if you want to interest us in this, give us a URL I can just click and read that explains what this is all about, or write a summary yourself here.
In reply to Tim Zietz

Re: Use of Power Law and Standards-Based Assessment

by Tim Zietz -

Thank you, T. H. for your legitimate concerns regarding the Marzano site. I'm a busy science teacher, and programmer on the side. I'm also new to the forums, so please forgive my inappropriate 'just google' response earlier.

What I've found most helpful is in the Marzano's 2000 book in my original post. Please know that my remarks should in no way be interpretted to be a marketing appeal. What Dr. Marzano has done in most of his work is not new research but a meta-analysis of the past twenty five years of educational research on teaching strategies, and assessment (i.e. bringing many studies together and summarizing them in ways that are helpful to the practitioner in the classroom.)

Let me briefly summarize for you: A student's grade should, if anything, provide feedback on 1) their mastery of the core concepts of a course, and 2) progress of their learning. All assessment should be assessed in light of those learning goals.

Perhaps an analogy will help on this point. One key skill in basketball is the ability to make foul shots. Experience tells us that a novice player might get only two or three shots out of ten, while a proficient player might sink seven or eight shots in the hoop. In other words, persistence and practice are rewarded by increasing levels of achievement. So it is in the classroom. Progress and achievement are the key features in learning. A grade should therefore provide students feedback on both.

This is how it works out in my classroom:

First, ALL assessments are judged using a scoring rubric, with adaptations where appropriate. The teacher rates student performance against the target learning goals for each unit. This is NOT the same as determining what the percent a student responded correctly on an assessment. Assessment questions and problems vary in their challenge. Some course challenges can only be met by highly skilled students, while others can be achieved by most students. Consider the rubric below:

Score

% Eq.

Description

Advanced

4

100

Made inferences, and demonstrates depth of understanding beyond what is directly taught. Applied target concepts and skills.

3.67

95

Mastery of Proficient content; high partial success at Advanced content

3.33

90

Mastery of Proficient content; some partial success at Advanced content

Proficient

3

85

Target learning goal ; more challenging content

2.67

80

Mastery of Basic content; high partial success at Proficient content

2.33

75

Mastery of Basic content; some partial success at Proficient content

Basic

2

70

Basic content mastered

1.67

65

High partial success of Basic content; major errors/omissions at Proficient content

1.33

60

Some partial success of Basic content; major errors/omissions at Proficient content

Below Basic

1

55

Basic content mastered with some assistance

0.5

48

With help, partial success at Basic content; no success at Proficient content

0

0

Even with help, no success

The target learning goals are communicated at the beginning of each unit, and drive all learning activities. Numerous assessments afford students opportunities to improve and demonstrate their success. Additionally, students should search for opportunities related to learning goals to demonstrate advanced competencies on their own initiative. Remember that a student’s success in learning increases with practice.

The second key difference in grading students is determining a marking quarter grade.

In a typical marking quarter, there are three units or broad topics planned. Each unit has a set of goals and learning line, as in the illustration above. By the end of a unit, it is anticipated that ALL students will achieve high proficiency of the learning goals. Therefore, the grade for the unit will reflect their degree of mastery, which is closer to the top of the learning line. It will NOT be determined by the average of the unit assessments, since that is a rather poor indicator of their actual success.

The MS TREND function works well to show the actual progress a student has made over a series of similar assessments on a concept or skill.

Perhaps this is not the best forum for this discussion. Perhaps the grading module forum would be more appropriate. BUT, any quiz could be assessed using the 4 point rubric above.

Thank you for your courteous consideration. The literature is available through many outlets for those who seek to go to the research.

In reply to Tim Zietz

Re: Use of Power Law and Standards-Based Assessment

by Colin Fraser -
Picture of Documentation writers Picture of Testers
Hi TimZ, we have not had a lot of luck in this country, despite the noises made by so many people, importing a lot of concepts from the US. Seems they get waterlogged coming across the Pacific and by the time they get implemented here, they appear to be severely warped so don't ever work properly.

To me, Marzano seems to have used the same ideas as found in the International Baccalaureate, you identify a level of proficiency and then assess it in terms of expression, presentation, creation and so on. Like the IB though, it makes an assumption that students have sufficient maturity, intellect and personality that they are going to take the risk of making fools of themselves and make public the connections they have understood between things that would demonstrate an Advanced level of skills. (It is all well and good for us to say, "Well if we truly developed a welcoming learning environment, this would not be an issue", to which I would agree, but reality is that the pack is going to attack anyone on either end, the really weak and the really smart are always targeted. Our language is full of such pejoratives, nerd, dweeb and so on.)

I use the word "personality" here because I have found it is usually the one aspect of our youth that is sadly lacking in them. Some have real brains, some have brains and maturity, but I have found only a very few who have that combination of brains, maturity and the personality that will allow them to take risks in their learning. This would mean that very, very few would even come close to a 4. Perhaps this is the way it should be, make learning difficult, anything that comes too easy is never learned properly, but the Darwinian approach to education has never really appealed to me.


In reply to Colin Fraser

Re: Use of Power Law and Standards-Based Assessment

by Marc Grober -
IB great idea, rather poor implementation (been there, done that.....)
Colin put his finger on it though-ability to to take a shot and fail. We really teach students just the opposite; to be conservative, constrained to the ambit of the instructor and to avoid any possibility of any grade other than superb as the grade is more important than anything one might learn.

The kind of education I would wish for my children is not to be found institutionalized as routinzation makes it virtually impossible for anything like that to survive. It is anathema to the success of the institution in the same way that heterodoxy is intolerable to the one true church (whichever that might be-lol.)

As far as Marzano, this looks like just another subjective model with "rubric" blazoned across it. If one could get 200 teachers to grade the work of 200 students on this rubric with scores that are within .05% then you would have a model. Good luck.....
In reply to Marc Grober

Re: Use of Power Law and Standards-Based Assessment

by Colin Fraser -
Picture of Documentation writers Picture of Testers
I know we are getting off topic here, but ... hahahaaaa!!! We learn more from our failures than we do from successes, so why are we trying to get every student to succeed?

One of the fundamental issues I have with modern education, well OK, modern society, is that we have managed to extend childhood for another seven years. In this country majority is reached at 18, in others it is 21, (and I am sure it is up and down in other countries) but in any case, many countries subscribe to the concept of the "teenager". Given that the term was invented by an American advertising company to delineate an emerging and economically strengthening demographic, why do we use it so freely?

To me, these are emerging adults, and we should be treating them as such and not be calling them by some technically inane label that entirely misses the point of who they are. At 13, our children are no longer children but adults in training, so why do we insist on keeping them as children? We try to provide them with the tools we believe they will need for their adult lives, but somehow, many of our students will not agree with that. Hopefully Moodle will provide a framework in which students will find a path into learning, but I doubt it. I suspect it will only be as adults will they see the advantages of what we are offering. So why do we keep them as children for those additional years?

If they were to grow and mature perhaps, then things like Marzano and IB would be better appreciated, better understood by students, and even better implemented by teachers.

In reply to Tim Zietz

Re: Use of Power Law and Standards-Based Assessment

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Thank you for explaining. And don't worry about telling me to Google it, I've been known to do that to other people wink (By the way, if you ever want to do that in a really rude way, then this is a fantastic tool: http://lmgtfy.com/?q=netiquette - I thought you might appreciate that.)

Anyway, as you say, nothing really new here, but things like rubrics are a useful tool that could stand to be used more (in the UK, or at least the OU, we just talk about 'Learning Outcomes'. See the example below); it is always good to remember that assessment is always an imperfect attempt quantify the thing that is really important, what the student has actually learned; and good to encourage students to take more control of their own learning (although whenever OU students are consulted, what they ask for is to be spoon fed what they need to know! Taking control of your own learning is much harder work.)

And, as you say, Moodle does not fully support all this, but have you found the bits it does offer?

The are disabled be default, so you need to go into the admin screens to enable them, but Moodle does support 'Outcomes' which is basically just another word for Rubric. Once it is turned on, you can grade each assignment against one or more Outcome, and the outcome is expressed as a scale like the one you show. A textual description with an underlying numerical value to correspond to each point. And then that all shows up in the gradebook.

Moodle does not support trend lines. However, when the gradebook was rewritten for Moodle 1.9, it was designed so that there was a central data store, and then you could create different reports to display that data in different ways. So, it would be possible for someone to create a gradebook report plugin that displays each student's results in a more graphical way with trendlines. There is a tutorial at Development:Gradebook_Report_Tutorial.




Learning outcomes example: There are from one part (about a week's part time study) of a course I did recently:

Having completed your reading of the study materials, you should be able to:
  • describe the mapping between the object data model and relational data model;
  • describe the areas where there is an impedance mismatch between object and relational database technologies – data types, identity, inheritance and relationships (associations) – and explain how each may be resolved;
  • produce a relational database design that represents a given design class model for an object-oriented application.

You should also be conversant with the following concepts:

  • collection data type, domains, join operation, object identity, primary/foreign key mechanism, user-defined data type, value identity.

These are the learning outcomes for the whole course:

Learning outcomes applicable to M888 can be grouped into the following overlapping categories:

Knowledge and understanding

After completing the course, you should be able to:

  • describe how relational databases are employed to maintain persistent data in software systems including web applications;
  • demonstrate a comprehensive understanding of transparent persistence and the use of persistence frameworks such as Java EE and .NET;
  • describe how DBMS services are made available through persistence frameworks via APIs such as ODBC and JDBC;
  • explain the need for object-relational mapping, describe how it is achieved, and select the appropriate approach to use in a particular situation.

Cognitive skills

After completing the course, you should be able to:

  • analyse the requirements for the management of shared persistent data in software systems;
  • apply design patterns and persistence frameworks during the development of a software system.

Key skills

After completing the course, you should be able to:

  • learn through the use of online resources – articles, journal papers and books;
  • critically evaluate information from a variety of sources on the management of shared persistent data in software systems;
  • communicate effectively in writing about approaches to managing shared persistent data in software systems.

Practical and/or professional skills

After completing the course, you should be able to:

  • apply the principles, concepts and techniques of the course in the use of database APIs and persistence frameworks to manage shared persistent data in software systems in the workplace.

In reply to Tim Zietz

Re: Use of Power Law and Standards-Based Assessment

by Tim Zietz -

After going back and reading through Marzano, I found the original work he cites regarding the use of the power law in tracking progress of student learning. Apparently, there is significant evidence that the power law is an appropriate means of modeling learning, and Newell states that its application is ubiquitous. See citations below. Quite frankly, I'm not a researcher, but a teacher. I'm tossing this into the mix for your consideration, or not. I'm one for stealing the good ideas of others when I run across one. I think this is a very good idea. Who says averaging grades (a measure of central tendency) should,  carte blanc, be the basis of a student's grade anyway?

I was able to found many writings in Google books that quote these two books, but I believe they are both out of print. Perhaps someone has a copy of these on their shelf?

For what it's worth, I've set up a spreadsheet in Google docs to demonstrate how this calculation compares with averaging grades. Feel free to play around with it. I'm also attaching an Excel sheet that is similar if you prefer to mess with it that way.

Anderson, J.R. Learning and memory: An integrated approach. New York: John Wiley & Sons. 1987.

Newell, A., & Rosenbloom, P. S. Cognitive skills and their acquisition. Hillsdale, NJ: Erlbaum. 1981.