What's all this about index assessment?

A basic definition of an index is "an indicator, sign, or measure of something." A more thorough analysis of the term might reveal a definition such as, "a number derived from a series of observations and used as an indicator or measure."

In either case, these definitions really do serve to describe what I mean by a new approach to assessment that I've been experimenting with in recent years. Thus, I have come to call it index marking or index assessment. As the years have gone by, I've been incorporating more and more index assessments into my assessment mix, and I've done this primarily because both technology and connectivity have made this form of assessment possible. 

Essentially, "index assessment" describes an assessment that is based on some sort of running total. That running total is based on numerous and ongoing collections of data. However, the index value is formative during a given unit of study, but becomes summative at the end of the unit. This allows both the teacher and the student to derive all of the value to be gained from formative assessment during the unit (such as low-stress check-ins, immediate feedback to students, and data to inform instructional next steps). However, students also enjoy one additional but highly critical aspect of index assessment: motivation. Knowing that an index mark will eventually become summative, those students who may be more motivated by marks will still be motivated to not only complete an index assessment, but to provide it their best effort as well. In my experience, motivation has been a perennial problem with formative assessment, and no amount of conversations, speeches, lectures, reminders or even infographics would solve this problem. 

Over the years, I have made great efforts to communicate the value of formative assessment to my students. While these efforts would make a modest impact on completion rates, I would still never obtain anything close to a 50% completion rate on formative assessments. With the introduction of the index approach, my completion rates are now well over 90% for the exact same assessments. Moreover, the overall level of achievement on associated summative assessments (ex. test at the end of the unit) has also increased. 

Putting Index Assessment Into Practice

At the moment, I have two index assessments that account, in total, for 15% of a student's overall grade in my courses. Specifically, these index assessments are the Ongoing Triangulation Index (OTI) and the Mastery Learning Lab (MLL). I have expanded an each of these forms of assessment in their own respective posts. Both of these index assessments can essentially be thought of as marks that are recorded during a given unit of study, remain observable by both the teacher and the student during the unit, and are always available to be improved upon through subsequent efforts made by the student. In other words, the student can respond to his mark in ways that can actually improve his mark during the unit.

The critical point is that an index assessment is formative during the unit, but becomes summative at the conclusion of the unit.  Naturally, it is critical that students understand this at the outset of the course. Given that index assessment is both new and somewhat unorthodox, this information needs to be communicated both verbally and in writing, repeatedly, to both students and parents. (More on communication to students and parents is explored below.)


The Strength of Index Assessment: Distributed Practice

Distributed practice refers to the long-noted beneficial effect of spacing out practice across numerous yet smaller periods of time. In other words, it is better to practice something for 15 minutes a day across four days, than to practice for one hour on one day. This effect was first studied by Hermann Ebbinghaus in 1885. Ebbinghaus discovered that he could successfully remember more material in less time if he spaced out the time that he spent on studying as opposed to concentrating for the same amount of time during fewer occasions. This effect is also known as the "spacing" effect, and it has held up very, very well over more than a century of study. In fact, I would dare say that most musicians and athletes naturally discover and take advantage of the spacing effect in their practice and training, as they come to clearly confront the fact that a learner cannot acquire skill within a limited period of time nearly as well as they can when spreading out their practice over extended periods of time. 


The Logistics of Index Assessment

Index assessments ask students to repeatedly take the same (or similar) assessments over an extended period of time. Moreover, the student is encouraged to repeat attempts with the knowledge that individual attempts will not count towards a final grade in the short-run, but that overall achievement will indeed count towards a final grade in the long-run. For such assessment to be practical or realistic, it must reside within some form of powerful educational technology that tracks the student's progress. Thus, we must use the appropriate technology, and we must set the scoring preferences in the most appropriate way. I use CourseSites for my Mastery Learning Lab. Specifically, I use the "Tests, Surveys, and Pools" feature made available under "Course Tools." I also make sure to organize my index assessments under unit titles, to manage the columns under the Grade Center so that the quizzes progress in order, and I set up an "Average Column" at the end of each unit. I set the preferences for each individual assessment so that the "highest" grade on each assessment is counted towards the overall mark within the unit - not the "average" or "most recent" grade. (The high score option locks high scores in place so that students can repeat attempts on quizzes or exercises without fear of losing a previously attained high score.)

​I also prefer to set up a "Smart View" for each section of a given course, as opposed to setting up an entirely new CourseSite for different sections of the same course. These Smart Views allow me to see each section in alphabetical order, which greatly assists when transposing the marks from CourseSites into my school's grade management system. At the end of the unit, I will then record the mark within the Unit Average column as a summative mark. Literally, this means that I will wait until I am entering the mark for the unit's culminating assessment (ex. unit test), and I will then set up a separate column entitled something like, "Unit #2, Mastery Learning Lab." 

​Other logistical advice that I would highlight include the need to collect lots of data and to clear the slate at the end of the unit. Given that indexes are based on collections of data, an index mark should be based on numerous assessments that each contribute to the overall index value over the course of a unit. An index mark should then be reset at the end of each unit, allowing a new value to be generated for each successive unit. This is achieved in different ways depending on the digital utility that one might be using to administer an index assessment. 
​

Why not just use formative assessments?

As I've mentioned elsewhere, formative assessments are great, but they're not perfect. Let's just acknowledge two elephants in the room when it comes to formative assessment: i) students often don't do them, and, ii) when they do, they don't tend to provide them their best effort.

1. Students tend not to do activities that are not associated with a summative mark. (Oh... did you think that was unique to your students? Nope! That's basically a universal law... and it doesn't matter how "interesting," "relevant," "fun," or "engaging" the work is. It's basic economics: time is scarce, so, all other things being equal, students will direct less effort toward school activities that are not associated with a summative mark.)
2. Related to the point above, even if we do collect formative data, we must remember that we are observing the results of efforts towards an assessment that the student knew would not count. Again, all other things being equal, results on formative assessments will generally reflect a lower degree of effort and diligence than we would see on equivalent summative assessments. Thus, steering  your classroom by observing formative assessment data is kind of like steering your car by looking through a foggy windshield. 

Thus, an index mark carries on as a fluctuating, formative value throughout each unit, but carries the promise of being recorded as a summative mark at the end of each unit. This provides the student with an extended period of time in a low-stress environment to master their knowledge and skills regarding a given topic, but then rewards the student's diligence and achievement with a summative mark that will actually make an impact on his overall grade.

​
How do you explain it to your students?

This is exactly what I tell my students regarding the quizzes in my MLL:

The quizzes in the Mastery Learning Lab are technically considered "formative" during the unit because they are not counted toward your mark during the unit. Moreover, these are mastery quizzes that you can take over and over again during the unit to help you develop your understanding of the topic. Finally, they will help both you and me identify areas of strength and weakness in your understanding of the topics as we move through the unit. However, at the conclusion of the unit, these quizzes will become "summative" because the overall average for a given unit will indeed be counted toward the calculation of your grade. Bear in mind that unattempted quizzes will receive marks of zero as of the conclusion of the unit. 


But... who can do all that marking?

I completely understand the skepticism that one might naturally have regarding index marking. It sounds like some airy-fairy, pie-in-the-sky initiative that only a partial load teacher could possibly pursue. I will point out, however, that I am a full-load teacher, and have been for more than 25 years. Index assessment is quite possible, but it is only made possible with appropriate technology and connectivity. 

In other articles, I have examined my index assessments in greater detail, and I would invite anyone who is curious about them to read more about the Mastery Learning Lab (MLL) and the Ongoing Triangulation Index (OTI). 


In Summary:

As might be evident from the above discussion, index assessment is inextricably tied to the idea of mastery learning. As I've mentioned before, genuine mastery learning requires unlimited opportunities to revisit material and then subject one's understanding of its content to an objective assessment until that assessment indicates that the material has been mastered. (It is all too easy for students to revisit material and then believe that they understand it, but one's sense of understanding can at times be found wanting when it is subjected to an empirical, objective test.)

In the final analysis, It's probably easiest to think of an index assessment as a summative assessment that both the student and the teacher can observe and improve upon as it develops. This provides a significant contrast to typical summative assessments because, with most summative assessments, by the time the teacher or the student sees the mark, it's too late for the teacher or the student to do anything about it. 

To be sure, it takes a while for students, teachers, and parents to get the gist of index marking. It's not quite formative, and it's not quite summative... it's a bit of both. I would like to think that it's the best of both, as I believe that index assessment allows students to enjoy the low-pressure feedback and remediation associated with formative assessments,  while also enjoying the motivation, acknowledgement, and reward associated with summative assessments.

References:

Bahrick, Harry P; Phelphs, Elizabeth. Retention of Spanish vocabulary over 8 years. Journal of Experimental Psychology: Learning, Memory, & Cognition. Vol 13(2) Apr 1987, 344-349

Bloom, Kristine C; Shuell, Thomas J. Effects of massed and distributed practice on the learning and retention of second-language vocabulary. Journal of Educational Research. Vol 74(4) Mar-Apr 1981, 245-248.

Donovan, John J; Radosevich, David J. A meta-analytic review of the distribution of practice effect: Now you see it, now you don't. Journal of Applied Psychology. Vol 84(5) Oct 1999, 795-805

Ebbinghaus, H. Memory: A contribution to experimental psychology. New York: Dover, 1964 (Originally published, 1885).

Ebbinghaus, Hermann (1885). Memory: A Contribution to Experimental Psychology.

Rea, Cornelius P; Modigliani, Vito. The effect of expanded versus massed practice on the retention of multiplication facts and spelling lists. Human Learning: Journal of Practical Research & Applications. Vol 4(1) Jan-Mar 1985, 11-18.
- See more at: http://www.aft.org/periodical/american-educator/summer-2002/ask-cognitive-scientist#sthash.g0xfsxpB.dpuf

Willingham, Daniel T. Allocating Student Study Time: "Massed" versus "Distributed" Practice.  http://www.aft.org/periodical/american-educator/summer-2002/ask-cognitive-scientist#sthash.g0xfsxpB.dpuf
​

The first test in a course is like the first goal in a hockey game: an early success can create a lot of positive momentum for the rest of the game. 

Effort directed toward ensuring that a class is ready to write a test - especially the first one - can definitely be time well spent. Here's a little idea that might just be the ticket for improving student results on your next test. 

Pre-Test Surveys

For many years now, I've made comprehensive review sheets and practice tests for my students to use while studying. I've also tended to use online surveys to see how my students are feeling about my teaching and my courses in general. Yet, it never really occurred to me until quite recently that I could use online surveys to see how my students are feeling about the material within a given unit of study.  

As soon as I got the idea, online pre-test surveys quickly became a part of my blended learning toolkit. The process associated with administering this style of survey can be broken down into four parts: i) creating, ii) distributing, iii) analyzing, and iv) responding. 

Creating the Survey:

These surveys do take some time to create. However, if you've already done some work up front to develop a review sheet, then you can use your review sheet to generate the pre-test survey fairly quickly. 

I tend to simply copy the requirements outlined within a test review sheet into a new survey. I then add a simple little introductory phrase to each requirement, such as, "Rate your ability to..." This generates a survey question that might look like: 

Rate your ability to define and explain the following concepts: elasticity of demand, "elastic" demand, "inelastic" demand, and "unitary" elastic demand.

I should point out that I do tend to lump together a few thematically related concepts into each survey question. Otherwise, these surveys might end up containing fifty or more questions and take a really long time to complete.

I then have my students choose a response from a five-point Likert Scale, ranging from "1: I'm really confused," to "5: I'm very confident." As the survey progresses, I tend to come up with increasingly outlandish polemics for each Likert Scale, such as, "I have no idea what you just said," versus "That's about as easy as opening my fridge." (I'd like to think that these humorous options tend to encourage greater student response.) 

The survey ends up looking like a collection of questions similar to the ones illustrated below. The really great thing about this style of survey is that it asks the student to engage in an extremely valuable style of assessment that we as educators rarely pursue: namely, assessment AS learning. This is the style of assessment where students are asked to reflect upon their learning. Research tells us that this style of reflection is an extremely powerful style of assessment in terms of its ability to promote learning, and it is actually one of the mechanisms that gives rise to Arthur Gates' famous "testing" effect.   

Distributing the Survey:

I tend to use Google Forms to facilitate this type of survey (as opposed to an anonymous survey tool like Survey Monkey) because I do want to collect user information on the respondents. This information allows me to provide help to specific students who might require some additional support ahead of the test.

I then send out an email to the entire class that explains the purpose of the survey and provides a hyperlink to it. I also ask students to provide their most accurate and honest response to each question. I tell them that the survey will help us both: it will help them identify any areas of weakness, and it will help me understand how to best support my students in their efforts to prepare for the test. Moreover, a pre-test survey tends to show students exactly how much material they actually do know, and that in of itself can help to calm nerves and build confidence ahead of a test.

Analyzing the Survey:

The first lesson I learned about analyzing these survey results is to not expect perfection.

In a five-point Likert Scale, where the number 5 represents perfect understanding of the requirement, and a number 1 represents complete lack of understanding, then we basically want to see responses that are skewed around the number four. Even on tests where the class average ends up in the 90% range, I have never seen a result on a pre-test survey that consists entirely of fives.

Responding to the Survey:

Now, if the results are skewed around a number two or three, then that should raise a red flag. I've definitely seen such results from time to time, and they will prompt me to do one of two possible things: i) review the material associated with such a response in class, and/or ii) send out a class-wide email addressing the issue.   

In the blended learning environment, the extra support required to address a weak area can often be facilitated with a quick email that points students to some online resources. A typical class email regarding a low-scoring survey question is illustrated below: 

I'll remind you of the fact that I do choose to use Google Forms for these surveys because they allow me to collect user information. Thus, if necessary, I can provide a little extra support to individual students who scored themselves particularly low in a given area. Generally, a quick email pointing the student to some specific online resources and offering a little extra help is enough to do the trick. A typical response to an individual student is illustrated below:

Without a doubt, pre-test surveys represent an additional time requirement for those teachers who might wish to incorporate them into their programs. In some cases, teachers may very well be pursuing some form of pre-test introspection already. The decision to use such a tool or not is invariably one that must be made based on a cost-benefit analysis. I will say this: just like most everything else related to edutech, these initiatives are often front-end loaded. Pre-test surveys will definitely take some time and effort up front, but once you have them in your kit bag, they'll pay some nice dividends down the road.  

If you've taught for long enough, you've heard all kinds of interesting and creative arguments for why a particular answer on a test question deserves a higher mark than the one you've awarded. Some of these arguments may indeed have merit, but most arguments will invariably speak to your testing philosophy. For this reason, I've found that it's a very good idea to clearly state my testing philosophy - and even post it in writing - long before my first test is ever given. 

Testing Philosophy:

I remember attending an AP (Advanced Placement) workshop where we were shown a range of answers to a selection of AP exam questions. In one case, an answer was highlighted for the fact that it would receive a mark solely because it contained a particular buzzword that was included within a pre-determined list of buzzwords - regardless of the fact that the word was not presented in any meaningful context. Fair enough: that may be part of the College Board's testing philosophy, but it's not part of mine. While I'll never dismiss or downplay the importance of knowledge, I feel strongly that test questions should be developed to examine understanding of a topic - not just memorization of facts or vocabulary. For almost twenty years now, I've articulated my testing philosophy within a statement that I include on my course websites and even at the top of my tests. I must say, I've found this to be a very useful practice because it not only tends to reduce potential challenges to poorly marked answers, it, more importantly, tends to increase the quality of written answers.  

I've shared my testing philosophy below. While the philosophy itself is of value, perhaps the thing that is of most value is simply the practice of reviewing such a philosophy with students ahead of tests. Think about your testing philosophy and how you might sum it up within a mission statement or manifesto, and then consider sharing it with your students ahead of your next test. I always enjoy hearing your feedback, so, if you decide to give this a try, let me know what kind of philosophy you develop.