Deconstructing two approaches to triangulation: The OTI and the CWP

Art Lightstone

I know I've said it before, but teachers in Ontario are still scratching their heads over the question of how to collect evidence of "observations" and "conversations," and then integrate that evidence into a final grade for a student. 

The confusion stems from a paragraph found on page 39 of the Ministry of Education's Growing Success policy document. The paragraph states:

Evidence of student achievement for evaluation is collected over time from three different sources – observations, conversations, and student products. Using multiple sources of evidence increases the reliability and validity of the evaluation of student learning.

Sadly, teachers have not found much guidance from the Ministry on the implementation of this directive. In particular, teachers are left wondering about two critical questions:

1. How might one actually record or collect evidence of conversations and observations?
2. Were it to be collected, how would one use this data to inform a student's final grade? 

Perhaps even more troubling is the fact that teachers cannot find a great deal of consensus, even amongst themselves, about one crucial point. Namely, how do we actually differentiate between a "conversation," an "observation," and a "product"?

To be sure, in our own minds we are all quite clear on the distinction between all three, and we would scarcely imagine any particular debate in the matter. It is only when we subject our individual perceptions of these elements against the scrutiny of our colleagues that we come to realize that there is indeed a great variety of perspectives regarding products, conversations, and observations.

Invariably, most teachers probably like the spirit of triangulation, but just because we like it, doesn't mean we do it. In fact, a recent NewLearnerLab Twitter survey revealed that most teachers do not collect evidence of observations or conversations at all.

Personally, I have developed and experimented with two distinct triangulation systems that would both collect conversation and observation data, and then integrate this data with a student's final grade. I called my first system the Ongoing Triangulation Index, and my second system the Classwork Portfolio. 

I have discussed the logistics of each method in separate articles, but after spending a good deal of time implementing both systems in my program, I felt I was now in a better position to present a comparative review of both methods. 

Ongoing Triangulation Index 

Description:

The Ongoing Triangulation Index is a value that is based on recorded observations and conversations that are generated in class. The OTI depends on the ClassDojo application to capture observations and conversations and then curate these observations within a single record for each student.

Each observation or conversation is time-stamped, dated, and accompanied by a brief explanation of the observation and the curriculum associated with the observation. Within this index, students can accumulate both positive and negative observations, the sum of which will determine their overall Index value.

The OTI counts for 5% of the overall mark in a course. Thus, a number of OTI observations are recorded each term.

Strengths:

Allows the teacher to capture demonstrations of learning, understanding, appreciation, etc. that occur spontaneously within the classroom.

Generates a comprehensive record of observations and conversations for each student.

This record can contain detailed information regarding the specific learning goals associated with each observation. Example: “Demonstrated a deep understanding of how Keynesian fiscal policy could be used to fill a fiscal gap using the balanced budget multiplier.”  

The above record is updated instantly and can be made accessible to both parents and students.  

Challenges:

The OTI system uses the ClassDojo app, which lends itself to capturing evidence of observable behaviours. Thus, this system can potentially be criticized as assigning a mark for learning skills and/or work habits.

This system must record both negative and positive observations in order to generate a meaningful value that can then be translated into a mark.

While the OTI facilitated the recording of both conversations and observations, conversations never figured into the index value.

Classwork Portfolio                

Description:

The Classwork Portfolio uses Google Classroom, Kahoot, and Edvance (an online student information system and grade manager) to generate a portfolio of the various activities, exercises, and check-ins that are pursued in class on a day-to-day basis. Each activity is assessed and given a mark between zero and four based on relative quality.

For those in-class activities and exercises that require deeper qualitative assessment, a streamlined Canada and World Studies, 2015 Achievement Chart is used to assess the varying levels of achievement.

The Classwork Portfolio counts for 5% of the overall mark in a course. Thus, a number of portfolio marks are recorded for each student within each reporting period.  

Strengths:

The CWP allows the teacher to capture progress that occurs in class on a daily basis, and then record the observation of that progress in a way that impacts the student’s mark immediately.

Students come to realize that they can effect a positive influence over their mark during each and every class.

The CWP tends to focus more on the manifest result of a student’s learning than it does on student behaviour or learning skills. (Although, depending on one's perspective, this could be seen as a weakness.)

The teacher can include detailed information regarding the specific learning goals associated with each observation. Example: “Achieved the 80% threshold on today’s check-in on using Keynesian fiscal policy to fill a fiscal gap using the balanced budget multiplier. Silver medal... well done!”

As this information is recorded in an online grade manager, it is made accessible to parents and students immediately.

Regular check-ins are designed to more readily allow students to effect a positive influence on their mark, while limiting the potential for a negative impact.

Challenges:

The CWP system can generate a large quantity of micro-assessments that can rather quickly outpace a teacher’s ability to mark. Teachers are therefore advised to limit CWP items to activities and exercises that can be assessed rather quickly: preferably, during the class in which they are assigned.  

The CWP generates a rather disconnected record of observations and conversations for each student. This record is brought together, somewhat loosely, within our online grade manager.