What's all this about index assessment?

A basic definition of an index is "an indicator, sign, or measure of something." A more thorough analysis of the term might reveal a definition such as, "a number derived from a series of observations and used as an indicator or measure."

In either case, these definitions really do serve to describe what I mean by a new approach to assessment that I've been experimenting with in recent years. Thus, I have come to call it index marking or index assessment. As the years have gone by, I've been incorporating more and more index assessments into my assessment mix, and I've done this primarily because both technology and connectivity have made this form of assessment possible. 

Essentially, "index assessment" describes an assessment that is based on some sort of running total. That running total is based on numerous and ongoing collections of data. However, the index value is formative during a given unit of study, but becomes summative at the end of the unit. This allows both the teacher and the student to derive all of the value to be gained from formative assessment during the unit (such as low-stress check-ins, immediate feedback to students, and data to inform instructional next steps). However, students also enjoy one additional but highly critical aspect of index assessment: motivation. Knowing that an index mark will eventually become summative, those students who may be more motivated by marks will still be motivated to not only complete an index assessment, but to provide it their best effort as well. In my experience, motivation has been a perennial problem with formative assessment, and no amount of conversations, speeches, lectures, reminders or even infographics would solve this problem. 

Over the years, I have made great efforts to communicate the value of formative assessment to my students. While these efforts would make a modest impact on completion rates, I would still never obtain anything close to a 50% completion rate on formative assessments. With the introduction of the index approach, my completion rates are now well over 90% for the exact same assessments. Moreover, the overall level of achievement on associated summative assessments (ex. test at the end of the unit) has also increased. 

Putting Index Assessment Into Practice

At the moment, I have two index assessments that account, in total, for 15% of a student's overall grade in my courses. Specifically, these index assessments are the Ongoing Triangulation Index (OTI) and the Mastery Learning Lab (MLL). I have expanded an each of these forms of assessment in their own respective posts. Both of these index assessments can essentially be thought of as marks that are recorded during a given unit of study, remain observable by both the teacher and the student during the unit, and are always available to be improved upon through subsequent efforts made by the student. In other words, the student can respond to his mark in ways that can actually improve his mark during the unit.

The critical point is that an index assessment is formative during the unit, but becomes summative at the conclusion of the unit.  Naturally, it is critical that students understand this at the outset of the course. Given that index assessment is both new and somewhat unorthodox, this information needs to be communicated both verbally and in writing, repeatedly, to both students and parents. (More on communication to students and parents is explored below.)


The Strength of Index Assessment: Distributed Practice

Distributed practice refers to the long-noted beneficial effect of spacing out practice across numerous yet smaller periods of time. In other words, it is better to practice something for 15 minutes a day across four days, than to practice for one hour on one day. This effect was first studied by Hermann Ebbinghaus in 1885. Ebbinghaus discovered that he could successfully remember more material in less time if he spaced out the time that he spent on studying as opposed to concentrating for the same amount of time during fewer occasions. This effect is also known as the "spacing" effect, and it has held up very, very well over more than a century of study. In fact, I would dare say that most musicians and athletes naturally discover and take advantage of the spacing effect in their practice and training, as they come to clearly confront the fact that a learner cannot acquire skill within a limited period of time nearly as well as they can when spreading out their practice over extended periods of time. 


The Logistics of Index Assessment

Index assessments ask students to repeatedly take the same (or similar) assessments over an extended period of time. Moreover, the student is encouraged to repeat attempts with the knowledge that individual attempts will not count towards a final grade in the short-run, but that overall achievement will indeed count towards a final grade in the long-run. For such assessment to be practical or realistic, it must reside within some form of powerful educational technology that tracks the student's progress. Thus, we must use the appropriate technology, and we must set the scoring preferences in the most appropriate way. I use CourseSites for my Mastery Learning Lab. Specifically, I use the "Tests, Surveys, and Pools" feature made available under "Course Tools." I also make sure to organize my index assessments under unit titles, to manage the columns under the Grade Center so that the quizzes progress in order, and I set up an "Average Column" at the end of each unit. I set the preferences for each individual assessment so that the "highest" grade on each assessment is counted towards the overall mark within the unit - not the "average" or "most recent" grade. (The high score option locks high scores in place so that students can repeat attempts on quizzes or exercises without fear of losing a previously attained high score.)

​I also prefer to set up a "Smart View" for each section of a given course, as opposed to setting up an entirely new CourseSite for different sections of the same course. These Smart Views allow me to see each section in alphabetical order, which greatly assists when transposing the marks from CourseSites into my school's grade management system. At the end of the unit, I will then record the mark within the Unit Average column as a summative mark. Literally, this means that I will wait until I am entering the mark for the unit's culminating assessment (ex. unit test), and I will then set up a separate column entitled something like, "Unit #2, Mastery Learning Lab." 

​Other logistical advice that I would highlight include the need to collect lots of data and to clear the slate at the end of the unit. Given that indexes are based on collections of data, an index mark should be based on numerous assessments that each contribute to the overall index value over the course of a unit. An index mark should then be reset at the end of each unit, allowing a new value to be generated for each successive unit. This is achieved in different ways depending on the digital utility that one might be using to administer an index assessment. 
​

Why not just use formative assessments?

As I've mentioned elsewhere, formative assessments are great, but they're not perfect. Let's just acknowledge two elephants in the room when it comes to formative assessment: i) students often don't do them, and, ii) when they do, they don't tend to provide them their best effort.

1. Students tend not to do activities that are not associated with a summative mark. (Oh... did you think that was unique to your students? Nope! That's basically a universal law... and it doesn't matter how "interesting," "relevant," "fun," or "engaging" the work is. It's basic economics: time is scarce, so, all other things being equal, students will direct less effort toward school activities that are not associated with a summative mark.)
2. Related to the point above, even if we do collect formative data, we must remember that we are observing the results of efforts towards an assessment that the student knew would not count. Again, all other things being equal, results on formative assessments will generally reflect a lower degree of effort and diligence than we would see on equivalent summative assessments. Thus, steering  your classroom by observing formative assessment data is kind of like steering your car by looking through a foggy windshield. 

Thus, an index mark carries on as a fluctuating, formative value throughout each unit, but carries the promise of being recorded as a summative mark at the end of each unit. This provides the student with an extended period of time in a low-stress environment to master their knowledge and skills regarding a given topic, but then rewards the student's diligence and achievement with a summative mark that will actually make an impact on his overall grade.

​
How do you explain it to your students?

This is exactly what I tell my students regarding the quizzes in my MLL:

The quizzes in the Mastery Learning Lab are technically considered "formative" during the unit because they are not counted toward your mark during the unit. Moreover, these are mastery quizzes that you can take over and over again during the unit to help you develop your understanding of the topic. Finally, they will help both you and me identify areas of strength and weakness in your understanding of the topics as we move through the unit. However, at the conclusion of the unit, these quizzes will become "summative" because the overall average for a given unit will indeed be counted toward the calculation of your grade. Bear in mind that unattempted quizzes will receive marks of zero as of the conclusion of the unit. 


But... who can do all that marking?

I completely understand the skepticism that one might naturally have regarding index marking. It sounds like some airy-fairy, pie-in-the-sky initiative that only a partial load teacher could possibly pursue. I will point out, however, that I am a full-load teacher, and have been for more than 25 years. Index assessment is quite possible, but it is only made possible with appropriate technology and connectivity. 

In other articles, I have examined my index assessments in greater detail, and I would invite anyone who is curious about them to read more about the Mastery Learning Lab (MLL) and the Ongoing Triangulation Index (OTI). 


In Summary:

As might be evident from the above discussion, index assessment is inextricably tied to the idea of mastery learning. As I've mentioned before, genuine mastery learning requires unlimited opportunities to revisit material and then subject one's understanding of its content to an objective assessment until that assessment indicates that the material has been mastered. (It is all too easy for students to revisit material and then believe that they understand it, but one's sense of understanding can at times be found wanting when it is subjected to an empirical, objective test.)

In the final analysis, It's probably easiest to think of an index assessment as a summative assessment that both the student and the teacher can observe and improve upon as it develops. This provides a significant contrast to typical summative assessments because, with most summative assessments, by the time the teacher or the student sees the mark, it's too late for the teacher or the student to do anything about it. 

To be sure, it takes a while for students, teachers, and parents to get the gist of index marking. It's not quite formative, and it's not quite summative... it's a bit of both. I would like to think that it's the best of both, as I believe that index assessment allows students to enjoy the low-pressure feedback and remediation associated with formative assessments,  while also enjoying the motivation, acknowledgement, and reward associated with summative assessments.

References:

Bahrick, Harry P; Phelphs, Elizabeth. Retention of Spanish vocabulary over 8 years. Journal of Experimental Psychology: Learning, Memory, & Cognition. Vol 13(2) Apr 1987, 344-349

Bloom, Kristine C; Shuell, Thomas J. Effects of massed and distributed practice on the learning and retention of second-language vocabulary. Journal of Educational Research. Vol 74(4) Mar-Apr 1981, 245-248.

Donovan, John J; Radosevich, David J. A meta-analytic review of the distribution of practice effect: Now you see it, now you don't. Journal of Applied Psychology. Vol 84(5) Oct 1999, 795-805

Ebbinghaus, H. Memory: A contribution to experimental psychology. New York: Dover, 1964 (Originally published, 1885).

Ebbinghaus, Hermann (1885). Memory: A Contribution to Experimental Psychology.

Rea, Cornelius P; Modigliani, Vito. The effect of expanded versus massed practice on the retention of multiplication facts and spelling lists. Human Learning: Journal of Practical Research & Applications. Vol 4(1) Jan-Mar 1985, 11-18.
- See more at: http://www.aft.org/periodical/american-educator/summer-2002/ask-cognitive-scientist#sthash.g0xfsxpB.dpuf

Willingham, Daniel T. Allocating Student Study Time: "Massed" versus "Distributed" Practice.  http://www.aft.org/periodical/american-educator/summer-2002/ask-cognitive-scientist#sthash.g0xfsxpB.dpuf
​

If you're a teacher who requires your students to write essays, I highly recommend that you consider implementing a practice that I've been doing in my law course for a number of years. Essentially, I write an essay in front of my students. I call this annual custom the "80-Minute Challenge." 

The 80-Minute Challenge is a demonstration that I will do once a year during an essay-writing work period. During this challenge, I attempt to research and write a complete 800 to 1200 word essay within 80 minutes. I do not know the topic going into the challenge. Rather, the students in the class brainstorm a selection of positions on a variety of legal issues. A student volunteer writes the propositions up on the board, and then we number the propositions (Proposition 1, Proposition 2, Proposition 3, etc.). Finally, a random number generator picks one of the topics. The topic being selected, I then begin to write the essay while a timer counts down the eighty minutes. Everything I do during the 80-Minute Challenge is projected onto the screen at the front of the class, so students can turn to observe my progress throughout the class. At the end of class, the students can see how far I've progressed. By that same afternoon or evening, I share the completed essay with the class. Admittedly, over the years I have found that some essays go a little better than others, but most essay topics end up being quite a lot of fun to research and write.  

Purpose:

Too often, students put off essay projects out of a sense of fear and dread. Particularly desperate students will even resort to plagiarism in order to avoid facing the task of legitimately writing an original essay. The purpose of the 80-Minute Challenge is to demonstrate that essay writing is not just an academic skill, but a fun, stimulating, and exciting endeavour. Embarking on a good essay is actually very similar to setting out on a treasure hunt or rally. The act of searching for and locating compelling sources that support one's thesis is actually a great deal of fun, but students can only access that sense of fun after they learn their requisite skills associated with essay-writing. For too long, schools have failed to expose students to the shear fun of essay writing. Once we add in that sense of fun, the technical skills associated with essay writing become something that students are little more inspired to learn, and the actual writing of essays becomes something that students start to embrace.

The Skills:

Various points of skill, strategy, and technique are demonstrated and discussed during the 80-Minute Challenge. Students see actual demonstrations of:

• producing an effective introductory paragraph (A complete introductory paragraph includes an attention-grabbing hook, an outline of the issue to be examined, the position taken on the issue (i.e. the thesis) and the arguments forwarded in support of the essay's position.)
• source surfing (A student researcher can easily track down the source of information that might be discussed within a particular resource, such as a news article. That will lead the researcher to more authoritative resources with comprehensive bibliographies and citations, and those will unleash a tidal wave of highly relevant resources.)
  
• limiting hearsay sources (It's always better to find a study that is cited within a report, just as it is better to cite a report than a news article that discusses the report. News articles should be used primarily for the interviews they may contain.) 
• organizing online sources within bookmark folders (The snowballing of resources that occurs when students learn how to source surf can very quickly generate an overwhelming number of sources. Students should be encouraged to use sets of bookmark folders that contain a folder for each argument within each essay folder.)
  
• using online citation generators (Online citation generators can quickly assist students in overcoming one of the single most confusing and daunting aspects of essay-writing: citations and references. Consider these tools to be training wheels for essay writing. After a while, the students won't need them, but they can help a lot of students get past their essay anxiety.)
  
• incorporating and punctuating citations properly (Allowing students to watch an experienced writer incorporate and punctuate citations can help clarify one of the confusing issues associated with essay writing.)
• differentiating assertions from facts (One of the best things students can learn from watching someone write an essay is how to utilize assertions and facts. I advocate a "three-punch combo" of i) assertion, ii) fact, iii) citation.)
• utilizing and punctuating quoted material (A very good skill to demonstrate to young writers is the value of quoted material. Too often students will either fail to quote a sentiment that really should come from an authoritative source, or they will quote material that doesn't need to come from an authoritative source. At the end of the day, the student is the writer of the essay, and quotes should be used to support - not state - the student's argument, position, or perspective.)

Caveats:

• The 80-Minute Challenge is certainly not intended to encourage students to complete their essays in eighty minutes the night before they're due. Rather, the 80-Minute Challenge is intended to demonstrate that essay writing can be a fun, exciting, and rewarding process. 
  
• I do not guarantee that I will complete the essay during the 80-Minute Challenge. Nonetheless, students will certainly be able to see how far I get in the course of the eighty minutes.
• I do not claim that the work I produce within eighty minutes would net a perfect mark in a high school or university course. Nonetheless, I do attempt to write an 'A' quality paper during the challenge.
• If the essay produced during the challenge were for a real course, I would consider that essay to be a first draft. I would then augment and revise that draft over the course of several days or weeks.   
• Part of the 80-Minute Challenge process involves debriefing the experience and reviewing the first draft of the 80-Minute Challenge essay. The class discusses any inherent challenges of the topic as well as the effects of the unusual time constraint. The class then attempts to identify weaknesses in the essay and determine areas for revision.
• Finally, the 80-Minute Challenge is intended to convince students that writing a legitimate, original, well-supported and fully-cited essay can be just as quick and easy as trolling the Internet for pre-written essays or articles that can be reworked into essays.
  

In Summary:

Essay writing is one of the most critical skills we demand of our students in both high school and university, yet the fact of the matter is that essay writing is also one skill that we as educators do not actually demonstrate to our students. Contrast how we teach essay writing to almost any other skill. Do we not demonstrate everything else from physics formulas, to financial statements, to jump shots? I would argue that we continue to see both poor essay writing and high rates of plagiarism because essay writing is the one skill that we attempt to teach without actually demonstrating. 

Admittedly, writing an essay in front of our students does seem a bit extreme... possibly even eccentric, but I maintain that this is simply because we've never been able to provide such a demonstration until very recently. These days, if a teacher has i) a computer, and ii) an LCD projector in their classroom, then they have the ability to take the 80-Minute Challenge. Having said that, this final point does give rise to what may be the most important point of introspection a teacher could ask themselves on the topic: if they have the tools, and they have the time, then why wouldn't they? This question will be explored more fully in an upcoming article. 

Resources:

The Art of Argument: Exploring the foundations of essay writing. (See link below.) 

Some Past 80-Minute Challenge Essays:

• April, 2010: The Scourge of Social Media: Why Ontario Schools Should Ban Access to Facebook and Other Social Media Sites
  
  
• April, 2011: Ban a Pit Bull, Save a Life: Why the Ontario Ban against Pit Bull Terriers Should Remain in Place
  
  
• February, 2015: Does it Pay to Drive Downtown? The Benefits of Implementing a Downtown Toll System in Toronto
  

I recently came across this old article that I had posted on newlearner.com about sixteen years ago. In those days, we didn't have students walking around with an assortment of "devices" that were loaded with educational "apps." Heck, we didn't even have learning management systems. Primarily, those of us who were interested in utilizing IT in the classroom were pretty much restricted to using basic HTML editors to create web pages. In any case, as I read through this article, I was really struck by how much of my current philosophical approach had been developed within just a few short years. This article highlights the fundamental perspective that I maintain to this day on the use of educational technology: use technology to enhance teaching and learning, but don't teach and learn just to use technology. 


Originally published on www.newlearner.com, July 23, 1998

Let me share with you the first piece of advice I give to any educator who asks, "How can I use the Internet in my courses?" My first response to such a question is to ask the teacher what they would like the Internet to do for them. This may not sound like the most technically sophisticated advice in the world, but it genuinely reflects my personal attitude towards technology. Personally, I surround myself with technology - I use it every day within a variety of pursuits and contexts. Nonetheless, I am not a fan of technology. I appreciate what technology can do for me, but I do not admire the technology itself. With this attitude in mind, I will always suggest that a teacher first consider how they intend to derive benefit from the Internet before they jump into it. I believe that a teacher’s use of the Internet, or a school’s "total I.T. solution" for that matter, must always be well grounded in utility – not flair. Technology has provided educators with a plethora of capabilities that we’ve not previously had the opportunity to explore. However, just because we can do something, does not necessarily mean that we should. 


Look Before You Leap

Getting the Internet into a classroom usually requires a school to establish an Information Technology (I.T.) program. Sadly, establishing an I.T. program within a school can easily absorb hundreds of thousands of dollars. Furthermore, the annual maintenance of an I.T. program will continue to absorb 25% to 50% of the program’s set-up cost on an annual basis. This reality alone should be enough to cause educators and school administrators to pause before committing themselves toward purchasing and maintaining the latest and the greatest in I.T. gadgetry.


All That Glitters is Not Gold

We have come to associate the Internet with the cutting-edge, status, and style. This has a lot to do with the fact that the web has been so well embraced by commercial enterprises. However, commercial websites rarely offer anything beyond the same banal fluff that one might find within a television commercial or a print advertisement. For commercial enterprises this actually makes a great deal of sense. Owing to the fact that the majority of commercial purchases are inspired by emotionally driven impulses, commercial websites will generally appeal to the emotions as opposed to the mind. If a soft drink company were to place too many genuine facts on their website, they may in fact convince consumers to not buy their product at all. A motion picture studio that placed too much detail on their website might convince movie-goers to go out and buy a book. Thus, we are all quite used to observing the tremendous flair that this new medium has to offer - the “MIDI” music, the flash videos, the animated “GIFs”, and the Java Script “mouse-overs.” As a consequence, we often develop a certain irrational respect for the corporations and products associated with slick websites. More significantly, we've come to identify I.T. flash and fancy as being part and parcel of a successful image – and who doesn't want to be successful? However, we as educators cannot afford to let ourselves become distracted from the genuine value contained within this digital domain.

Educators must remember that there is a significant difference between the academic website launched by a school, and the commercial website launched by a soft-drink company. Education has utility. People pursue education based on a variety of rational, logical, and intelligent goals. A good education will stimulate one’s imagination, inspire creativity, develop skills, and endow one with knowledge of, and appreciation for, the world in which they live. Thus, any educational I.T. undertaking should, in some way, reflect these goals.


If It Ain’t Broke, Don’t Fix It!

As I was growing up, my father often said, “If it ain’t broke, don’t fix it.” This was sound advice given the environment in which I grew up. My family didn’t have a lot of money, so we certainly couldn’t afford to go around fixing or replacing things that didn’t truly need to be fixed or replaced. Our financial situation made such decisions very simple. We knew that for every purchase we made, we would forego another purchase – perhaps something more important. I developed a strong sense of “opportunity cost” at a very young age.

Schools, obviously, are in a similar predicament. As education budgets continue to shrink, the demands placed upon every education dollar will continue to increase. Naturally, school administrators are increasingly forced to prioritize their needs and spend their limited budgets accordingly. So where should the Internet fall within a list of educational priorities? Well that depends on the list. As much as I am a proponent of information technology in education, I am certainly not going to say that an I.T. program should arbitrarily be assigned to the top of any school’s list of capital acquisitions.


Let’s be Honest with Ourselves

Educators who wish to incorporate the Internet within their programs in a meaningful way must first take a good honest look at their motivation. If a teacher feels that the Internet currently offers invaluable resources that will support the curriculum of a given course, then by all means that teacher should pursue acquiring Internet access for his/her students. But if a teacher is simply jumping on a passing bandwagon, then they may be catching a ride they really don’t want to take. Teachers must remember that it is the content found on the Internet that presents exciting opportunities – not the Internet itself. Meaningful interaction with the outside world is well worth incorporating into one’s curriculum. But simply throwing money at computers, and then throwing computers into classrooms, will not in itself succeed in modernizing or improving education. Given the expense of computer hardware and software, as well as the fact that these items are some of the fastest depreciating assets on the planet, no school in the world can afford to invest in I.T. to merely keep up with the Jones’.

Holding the interest of one’s students is without a doubt the greatest challenge that educators face on a daily basis. For this reason, many teachers are quick to embrace any prop, device, or apparatus with which they can share the burden. Sadly, the mere presence of the Internet in a classroom will not hold a student’s attention for long. Otherwise, we’d all still be addicted to playing Space Invaders at the local arcade. The fascination with technology for the technology itself will always wear off. Utility is what perseveres. Activity – genuine active learning – that’s what holds a student’s attention. So, if a school is willing to invest in Internet access, then teachers must be willing to invest time into i) examining the Internet’s utility, and ii) incorporate digital resources and technology into their curriculum... both with a goal of achieving valid and meaningful objectives that would have been more difficult to achieve, if not impossible, without the technology. Rest assured, the utility is there - buried deep beneath the flash and glamour. Nevertheless, facilitating meaningful interaction with the Internet may require more time, more effort, and more money than some school’s have bargained for.