What's all this about index assessment?

A basic definition of an index is "an indicator, sign, or measure of something." A more thorough analysis of the term might reveal a definition such as, "a number derived from a series of observations and used as an indicator or measure."

In either case, these definitions really do serve to describe what I mean by a new approach to assessment that I've been experimenting with in recent years. Thus, I have come to call it index marking or index assessment. As the years have gone by, I've been incorporating more and more index assessments into my assessment mix, and I've done this primarily because both technology and connectivity have made this form of assessment possible. 

Essentially, "index assessment" describes an assessment that is based on some sort of running total. That running total is based on numerous and ongoing collections of data. However, the index value is formative during a given unit of study, but becomes summative at the end of the unit. This allows both the teacher and the student to derive all of the value to be gained from formative assessment during the unit (such as low-stress check-ins, immediate feedback to students, and data to inform instructional next steps). However, students also enjoy one additional but highly critical aspect of index assessment: motivation. Knowing that an index mark will eventually become summative, those students who may be more motivated by marks will still be motivated to not only complete an index assessment, but to provide it their best effort as well. In my experience, motivation has been a perennial problem with formative assessment, and no amount of conversations, speeches, lectures, reminders or even infographics would solve this problem. 

Over the years, I have made great efforts to communicate the value of formative assessment to my students. While these efforts would make a modest impact on completion rates, I would still never obtain anything close to a 50% completion rate on formative assessments. With the introduction of the index approach, my completion rates are now well over 90% for the exact same assessments. Moreover, the overall level of achievement on associated summative assessments (ex. test at the end of the unit) has also increased. 

Putting Index Assessment Into Practice

At the moment, I have two index assessments that account, in total, for 15% of a student's overall grade in my courses. Specifically, these index assessments are the Ongoing Triangulation Index (OTI) and the Mastery Learning Lab (MLL). I have expanded an each of these forms of assessment in their own respective posts. Both of these index assessments can essentially be thought of as marks that are recorded during a given unit of study, remain observable by both the teacher and the student during the unit, and are always available to be improved upon through subsequent efforts made by the student. In other words, the student can respond to his mark in ways that can actually improve his mark during the unit.

The critical point is that an index assessment is formative during the unit, but becomes summative at the conclusion of the unit.  Naturally, it is critical that students understand this at the outset of the course. Given that index assessment is both new and somewhat unorthodox, this information needs to be communicated both verbally and in writing, repeatedly, to both students and parents. (More on communication to students and parents is explored below.)


The Strength of Index Assessment: Distributed Practice

Distributed practice refers to the long-noted beneficial effect of spacing out practice across numerous yet smaller periods of time. In other words, it is better to practice something for 15 minutes a day across four days, than to practice for one hour on one day. This effect was first studied by Hermann Ebbinghaus in 1885. Ebbinghaus discovered that he could successfully remember more material in less time if he spaced out the time that he spent on studying as opposed to concentrating for the same amount of time during fewer occasions. This effect is also known as the "spacing" effect, and it has held up very, very well over more than a century of study. In fact, I would dare say that most musicians and athletes naturally discover and take advantage of the spacing effect in their practice and training, as they come to clearly confront the fact that a learner cannot acquire skill within a limited period of time nearly as well as they can when spreading out their practice over extended periods of time. 


The Logistics of Index Assessment

Index assessments ask students to repeatedly take the same (or similar) assessments over an extended period of time. Moreover, the student is encouraged to repeat attempts with the knowledge that individual attempts will not count towards a final grade in the short-run, but that overall achievement will indeed count towards a final grade in the long-run. For such assessment to be practical or realistic, it must reside within some form of powerful educational technology that tracks the student's progress. Thus, we must use the appropriate technology, and we must set the scoring preferences in the most appropriate way. I use CourseSites for my Mastery Learning Lab. Specifically, I use the "Tests, Surveys, and Pools" feature made available under "Course Tools." I also make sure to organize my index assessments under unit titles, to manage the columns under the Grade Center so that the quizzes progress in order, and I set up an "Average Column" at the end of each unit. I set the preferences for each individual assessment so that the "highest" grade on each assessment is counted towards the overall mark within the unit - not the "average" or "most recent" grade. (The high score option locks high scores in place so that students can repeat attempts on quizzes or exercises without fear of losing a previously attained high score.)

​I also prefer to set up a "Smart View" for each section of a given course, as opposed to setting up an entirely new CourseSite for different sections of the same course. These Smart Views allow me to see each section in alphabetical order, which greatly assists when transposing the marks from CourseSites into my school's grade management system. At the end of the unit, I will then record the mark within the Unit Average column as a summative mark. Literally, this means that I will wait until I am entering the mark for the unit's culminating assessment (ex. unit test), and I will then set up a separate column entitled something like, "Unit #2, Mastery Learning Lab." 

​Other logistical advice that I would highlight include the need to collect lots of data and to clear the slate at the end of the unit. Given that indexes are based on collections of data, an index mark should be based on numerous assessments that each contribute to the overall index value over the course of a unit. An index mark should then be reset at the end of each unit, allowing a new value to be generated for each successive unit. This is achieved in different ways depending on the digital utility that one might be using to administer an index assessment. 
​

Why not just use formative assessments?

As I've mentioned elsewhere, formative assessments are great, but they're not perfect. Let's just acknowledge two elephants in the room when it comes to formative assessment: i) students often don't do them, and, ii) when they do, they don't tend to provide them their best effort.

1. Students tend not to do activities that are not associated with a summative mark. (Oh... did you think that was unique to your students? Nope! That's basically a universal law... and it doesn't matter how "interesting," "relevant," "fun," or "engaging" the work is. It's basic economics: time is scarce, so, all other things being equal, students will direct less effort toward school activities that are not associated with a summative mark.)
2. Related to the point above, even if we do collect formative data, we must remember that we are observing the results of efforts towards an assessment that the student knew would not count. Again, all other things being equal, results on formative assessments will generally reflect a lower degree of effort and diligence than we would see on equivalent summative assessments. Thus, steering  your classroom by observing formative assessment data is kind of like steering your car by looking through a foggy windshield. 

Thus, an index mark carries on as a fluctuating, formative value throughout each unit, but carries the promise of being recorded as a summative mark at the end of each unit. This provides the student with an extended period of time in a low-stress environment to master their knowledge and skills regarding a given topic, but then rewards the student's diligence and achievement with a summative mark that will actually make an impact on his overall grade.

​
How do you explain it to your students?

This is exactly what I tell my students regarding the quizzes in my MLL:

The quizzes in the Mastery Learning Lab are technically considered "formative" during the unit because they are not counted toward your mark during the unit. Moreover, these are mastery quizzes that you can take over and over again during the unit to help you develop your understanding of the topic. Finally, they will help both you and me identify areas of strength and weakness in your understanding of the topics as we move through the unit. However, at the conclusion of the unit, these quizzes will become "summative" because the overall average for a given unit will indeed be counted toward the calculation of your grade. Bear in mind that unattempted quizzes will receive marks of zero as of the conclusion of the unit. 


But... who can do all that marking?

I completely understand the skepticism that one might naturally have regarding index marking. It sounds like some airy-fairy, pie-in-the-sky initiative that only a partial load teacher could possibly pursue. I will point out, however, that I am a full-load teacher, and have been for more than 25 years. Index assessment is quite possible, but it is only made possible with appropriate technology and connectivity. 

In other articles, I have examined my index assessments in greater detail, and I would invite anyone who is curious about them to read more about the Mastery Learning Lab (MLL) and the Ongoing Triangulation Index (OTI). 


In Summary:

As might be evident from the above discussion, index assessment is inextricably tied to the idea of mastery learning. As I've mentioned before, genuine mastery learning requires unlimited opportunities to revisit material and then subject one's understanding of its content to an objective assessment until that assessment indicates that the material has been mastered. (It is all too easy for students to revisit material and then believe that they understand it, but one's sense of understanding can at times be found wanting when it is subjected to an empirical, objective test.)

In the final analysis, It's probably easiest to think of an index assessment as a summative assessment that both the student and the teacher can observe and improve upon as it develops. This provides a significant contrast to typical summative assessments because, with most summative assessments, by the time the teacher or the student sees the mark, it's too late for the teacher or the student to do anything about it. 

To be sure, it takes a while for students, teachers, and parents to get the gist of index marking. It's not quite formative, and it's not quite summative... it's a bit of both. I would like to think that it's the best of both, as I believe that index assessment allows students to enjoy the low-pressure feedback and remediation associated with formative assessments,  while also enjoying the motivation, acknowledgement, and reward associated with summative assessments.

References:

Bahrick, Harry P; Phelphs, Elizabeth. Retention of Spanish vocabulary over 8 years. Journal of Experimental Psychology: Learning, Memory, & Cognition. Vol 13(2) Apr 1987, 344-349

Bloom, Kristine C; Shuell, Thomas J. Effects of massed and distributed practice on the learning and retention of second-language vocabulary. Journal of Educational Research. Vol 74(4) Mar-Apr 1981, 245-248.

Donovan, John J; Radosevich, David J. A meta-analytic review of the distribution of practice effect: Now you see it, now you don't. Journal of Applied Psychology. Vol 84(5) Oct 1999, 795-805

Ebbinghaus, H. Memory: A contribution to experimental psychology. New York: Dover, 1964 (Originally published, 1885).

Ebbinghaus, Hermann (1885). Memory: A Contribution to Experimental Psychology.

Rea, Cornelius P; Modigliani, Vito. The effect of expanded versus massed practice on the retention of multiplication facts and spelling lists. Human Learning: Journal of Practical Research & Applications. Vol 4(1) Jan-Mar 1985, 11-18.
- See more at: http://www.aft.org/periodical/american-educator/summer-2002/ask-cognitive-scientist#sthash.g0xfsxpB.dpuf

Willingham, Daniel T. Allocating Student Study Time: "Massed" versus "Distributed" Practice.  http://www.aft.org/periodical/american-educator/summer-2002/ask-cognitive-scientist#sthash.g0xfsxpB.dpuf
​

Mastery learning is one of the most discussed but little pursued concepts in education. We hear a lot of talk about mastery learning, but how many teachers actually do it? I mean actual mastery learning, where students have the opportunity to establish "a level of performance that all students must master before moving on to the next unit" (Slavin, 1987). Sure, we might see the odd educator referring to test rewrites as "mastery learning," but that is hardly establishing "mastery" of the material. At best, it is a chance to revisit the material once. At worst, it is simply marking a test twice. 

Unlimited mastery learning across the entire curriculum of a given course is, without a doubt, a daunting prospect. However, it can be done. As long as a teacher leverages the tools currently available, and maintains focus on the true intent of mastery learning, then mastery learning can indeed be achieved in many modern learning environments. What follows are a few of the keys to mastery learning that I have discovered in my practice, as well as some practical advice on implementing these keys.


Key 1: Mastery learning is mastery learning... not mastery marking.

Mastery learning occurs at the formative level. Multiple opportunities provided to student to improve their summative marks is a separate (albeit related) issue from mastery learning. Make no mistake: true mastery learning is about learning – not marks. (More on that later.) Assessment designed to provide feedback and direction for improvement is, by definition, formative assessment. Ergo, genuine mastery learning occurs at the formative level. The relationship between mastery learning and the student’s summative marks should, if the endeavour is pursued correctly, simply follow as a natural positive correlation. Now, as for the whole "pursued correctly" thing, well... that's the whole trick of the endeavour. (Again, more on that below.)

Key 2: Mastery learning can occur anywhere, but is most important to pursue for foundational concepts. 

I will never be a master of memorization. My brain just doesn’t work that way. However, many people have incredible memories. We don’t want to design formative assessments to promote mastery memorization, though. The memorization of facts is, without a doubt, critically important to learning: even skill-based learning. As is often said amongst brain researchers, “You need to know something before you can think about something.” I will certainly concede that there is great value in memorization of knowledge, and I will also concede the fact that mastery learning can in fact reinforce that knowledge. (Just ask any music student who has repeatedly used flashcards to memorize the notes of the treble or bass clef.) 

Indeed, knowledge is the soil from which thinking grows, and thinking can be thought of as the branches from which the leaves of creativity spring. Yes, people, organizations, businesses, society, and civilization all need people who can think and exercise creativity, but before any of that can happen, knowledge (actual knowledge... not opinion) must be in place. 

Key 3. Formative assessment can only generate mastery learning if students have some way to examine and understand their own formative results. 

I cannot emphasize enough the need for the student to understand… not be told by someone else who understands. The student must have the opportunity to engage in self-assessment (i.e. assessment as learning) following each formative assessment. Otherwise, the student is essentially spinning a dial each time he takes the assessment, and hoping that the dial will eventually land on “Mastered!”

The formative assessment must not only provide a mark to the student, and record that mark for both the student and the teacher, but that mark must be broken down into its constituent parts as well. In other words, a quiz comprised of ten questions must present the student with a result for each question.

As counter-intuitive as this might seem, providing feedback to the student on each and every aspect of a given formative assessment, is, almost by definition, not particularly required for assessment “as” learning. The more salient issue is whether the student has the opportunity to look at an incorrect result and process, for himself, why the result is wrong. Only when that critical step is facilitated, and then the student has the opportunity to test and confirm his new understanding in subsequent assessments, will the student experience what is perhaps the most powerful form of learning available: assessment “as” learning.


But... who can do all that marking?!

A very small percentage of teachers have ever pursued unlimited mastery learning across their entire curriculum. Ask most teachers about the least favourite part of their job, and they will undoubtedly say it's the marking. Marking is relentless, and most teachers find the prospect of marking all of their assessments just once to be incredibly challenging at best. (At worst, it's impossible.) Now, add into this equation the following realities associated with formative assessment in a mastery learning environment:

1. many more formative assessments will be added into the assessment mix,
2. these formative assessments will be done several times,
3. these formative assessments will flow in at different times, and
4. the marks for these assessments must be recorded and tracked. (This step may seem optional, but it’s not. All formative results must be tracked in a grade manager. Don’t even tell me about how great your memory is. A teacher’s brain will simply not be able to retain progress results for several students, across several courses, across several assessments, across a span of several months. That’s impossible.)

Remind most teachers of these elements, and they will tell you that true, unlimited, mastery learning is simply not a reality.

Enter the new reality

For the most part, teachers who balk at the idea of genuine mastery learning would be correct were it not for one thing: the incredible opportunities now provided to educators by way of technology. The technological tools and applications available today provide educators with the opportunity to provide their students with assessments that are not only completed online, but marked immediately by the online application and then recorded in the application's central grade manager. Admittedly, that’s a lot of data crunching, but remember: that’s what computers do. 

Make no mistake though, in order for genuine mastery learning to be pursued, teachers must leverage technology that can manage the recording and tracking of marks. (Remember what I said about the "impossible" thing above. A teacher's memory and vague impressions will not suffice in the effort to administer formative assessment.)

Developing a Mastery Learning Lab

So now, what can you do in your classroom to start implementing true, unlimited, mastery learning, today? You can go out on the web and register yourself with any number of free online learning management systems (such as CourseSites or Schoology to name just a couple) or set up an account with one of the pre-developed learning systems (such as Khan Academy or Mathletics), and then establish an online Mastery Learning Lab!

To create a Mastery Learning Lab (MLL), you will need to develop a collection of quizzes and activities that the Learning Management System (LMS) can mark and record, or you will need to register for a pre-developed system that marks and records its own pre-existing quizzes and activities. Either way, the quizzes in your Lab are "mastery learning" quizzes because you can invite your students to take these quizzes repeatedly until they master them. (Note: Online Learning Management Systems generally provide teachers with the option of emphasize the most recent mark, the highest mark, or the average of all attempts on a given quiz or activity.)  

An important caveat about formative assessments 

Formative assessment is great, but it's not perfect. Let's just acknowledge two elephants in the room when it comes to formative assessment: students often don't do them, and, when they do, they don't tend to provide them their best effort.

1. Students tend not to do activities that are not associated with a summative mark. Oh... did you think that was unique to your students? Nope! That's basically a universal law... and it doesn't matter how "interesting," "relevant," "fun," or "engaging" the work is. It's basic economics: time is scarce, so, all other things being equal, students will direct less time and effort toward school activities that are not associated with a summative mark.
2. Related to the point above, even if we do collect formative data, we must remember that we are observing the results of efforts towards an assessment that the student knew would not count. Again, all other things being equal, results on formative assessments will generally reflect a lower degree of effort and diligence than we would see on equivalent summative assessments. Thus, steering  your classroom by observing formative assessment data is kind of like steering your car by looking through a foggy windshield. 

A simple solution: Make your Mastery Learning Lab the best of both worlds!

Did you know there's no law against making an assessment both formative AND summative? Yup. It's true. Go ahead... look it up. No law!

So, here's what you do. You make sure that your students understand that the quizzes and activities in your Mastery Learning Lab are "formative" during a unit because they are mastery quizzes that students can take over and over again during the unit to help them develop their understanding of the topic. Moreover, the results on these assessments will help both the student and the teacher identify areas of strength and weakness as you move through a unit, thereby helping to guide next steps. However, at the conclusion of the unit these quizzes and activities become "summative" because their final marks will indeed be counted toward the final grade as of the conclusion of a given unit. (See more on this approach by examining the post on Index Marking.)

Boom! Best of both worlds! The student gets to attempt assessments in a non-threatening, low-stress, mastery environment that is formative during the unit, but the students are also motivated to not only do these assessments but to provide them their best effort because they know that the assessments will eventually count toward their grade. 

In Summary

Setting up a Mastery Learning Lab is definitely a big time investment. This investment can be reduced markedly if one utilizes a pre-developed learning system such as Mathletics or Khan Academy, but, either way, it requires an investment. The question then becomes: Is it worth the investment? In my opinion, it is well worth the investment. Especially if you are confident that you will be teaching the same course for a number of years. I might even suggest that the payback period on a teacher's investment in a Mastery Learning Lab could be as little as two years. In other words, the time you put into your Lab will be returned, with dividends, after two years. 

Consider setting up a Mastery Learning Lab for one of your courses today, and let me know how it goes.


References

Slavin, R. E. (1987). Mastery learning reconsidered. Review of Educational Research, 57, 175-213.

Teachers in Ontario have been spending a lot of time over the last couple of years discussing the use of observations and conversations in assessment. The discussion stems from approximately one paragraph of material found on page 39 of the Ministry of Education's Growing Success policy document. The paragraph begins:

Evidence of student achievement for evaluation is collected over time from three different sources – observations, conversations, and student products. Using multiple sources of evidence increases the reliability and validity of the evaluation of student learning.

Ontario teachers have not found much guidance from the Ministry on the implementation of this directive, nor have we found any real consensus amongst teachers about how one would actually record conversations and observations, how one would use this data were it to be collected, or even how one might differentiate a "conversation," from an "observation," from a "product."

Nonetheless, I like the spirit of triangulation, and so I have set out to create a practical, reliable, and useful tool that allows a teacher to both collect data regarding conversations and observations, and then actually use this data in a predictable and transparent way to inform a student's final grade. I have called this tool the Ongoing Triangulation Index.

The OTI - A Summary

The OTI is a tool, administered via ClassDojo, that records and analyzes daily in-class observations of learning, and then translates these observations into an index value that is associated with a mark. (In my courses, this mark is worth 5% of the final grade.) The Index carries on as a fluctuating, formative value throughout each unit, but is then recorded as a summative mark at the end of each unit. The Index is reset at then end of each unit, allowing a new value to be generated for each successive unit. Thus, the OTI acknowledges the learning that is demonstrated by students in class on a daily basis.

The Ongoing Triangulation Index

The Ongoing Triangulation Index is a value based on recorded observations and conversations in class. Each observation or conversation is time-stamped, dated, and accompanied by a brief explanation. Within this index, students can accumulate both positive and negative observations, the sum of which will determine their overall Index value.
​
The Index provides each student with a percentage mark, which essentially indicates the percentage of positive to negative observations. Moreover, some analysis of these observations is also provided, which can be used by students and academic advisers to identify areas of strength and weakness, and to set goals in response to these observations.

Enabled by Technology

The OTI would not be possible without the assistance of technology. While I have experimented with a number of class management apps, I have decided to use ClassDojo to track the OTI. I made this decision because ClassDojo allows me to record a given observation for several students at once. (For example, all of the students who participated in a class discussion, or the top five scorers in a Kahoot review exercise.) In addition, ClassDojo allows me to communicate my in-class observations, as well as my general class story, to my students, parents, and even academic advisers.