Looking for a Practical Way to Record and Utilize Observations and Conversations? Try the Ongoing Triangulation Index!

Art  Lightstone

Teachers in Ontario have been spending a lot of time over the last couple of years discussing the use of observations and conversations in assessment. The discussion stems from approximately one paragraph of material found on page 39 of the Ministry of Education's Growing Success policy document. The paragraph begins:

Evidence of student achievement for evaluation is collected over time from three different sources – observations, conversations, and student products. Using multiple sources of evidence increases the reliability and validity of the evaluation of student learning.

Ontario teachers have not found much guidance from the Ministry on the implementation of this directive, nor have we found any real consensus amongst teachers about how one would actually record conversations and observations, how one would use this data were it to be collected, or even how one might differentiate a "conversation," from an "observation," from a "product."

Nonetheless, I like the spirit of triangulation, and so I have set out to create a practical, reliable, and useful tool that allows a teacher to both collect data regarding conversations and observations, and then actually use this data in a predictable and transparent way to inform a student's final grade. I have called this tool the Ongoing Triangulation Index.

The OTI - A Summary

The OTI is a tool, administered via ClassDojo, that records and analyzes daily in-class observations of learning, and then translates these observations into an index value that is associated with a mark. (In my courses, this mark is worth 5% of the final grade.) The Index carries on as a fluctuating, formative value throughout each unit, but is then recorded as a summative mark at the end of each unit. The Index is reset at then end of each unit, allowing a new value to be generated for each successive unit. Thus, the OTI acknowledges the learning that is demonstrated by students in class on a daily basis.

The Ongoing Triangulation Index

The Ongoing Triangulation Index is a value based on recorded observations and conversations in class. Each observation or conversation is time-stamped, dated, and accompanied by a brief explanation. Within this index, students can accumulate both positive and negative observations, the sum of which will determine their overall Index value.
​
The Index provides each student with a percentage mark, which essentially indicates the percentage of positive to negative observations. Moreover, some analysis of these observations is also provided, which can be used by students and academic advisers to identify areas of strength and weakness, and to set goals in response to these observations.

Enabled by Technology

The OTI would not be possible without the assistance of technology. While I have experimented with a number of class management apps, I have decided to use ClassDojo to track the OTI. I made this decision because ClassDojo allows me to record a given observation for several students at once. (For example, all of the students who participated in a class discussion, or the top five scorers in a Kahoot review exercise.) In addition, ClassDojo allows me to communicate my in-class observations, as well as my general class story, to my students, parents, and even academic advisers.

---

Mandate from the Ministry of Education

The mandate to implement the Ongoing Trinagulation Index comes from three interrelated documents: i) Growing Success, ii) The Ontario Curriculum, Grades 11 and 12, Canadian and World Studies, and iii) The Ontario Skills Passport.


Growing Success

Page 39 of the Growing Success policy document outlines a mandate for triangulating a student's mark based on products, observations, and conversations. Specifically, it states:

Evidence of student achievement for evaluation is collected over time from three different sources – observations, conversations, and student products. Using multiple sources of evidence increases the reliability and validity of the evaluation of student learning. 

“Student products” may be in the form of tests or exams and/or assignments for evaluation. Assignments for evaluation may include rich performance tasks, demonstrations, projects, and/or essays. To ensure equity for all students, assignments for evaluation and tests or exams are to be completed, whenever possible, under the supervision of a teacher. Assignments for evaluation must not include ongoing homework that students do in order to consolidate their knowledge and skills or to prepare for the next class. Assignments for evaluation may involve group projects as long as each student’s work within the group project is evaluated independently and assigned an individual mark, as opposed to a common group mark. 

The evaluation of student learning is the responsibility of the teacher and must not include the judgement of the student or of the student’s peers.



The Ontario Curriculum, Grades 11 and 12, Canadian and World Studies

Specific Expectations for the Grade 12 Economics Course (CIA4U) include the development of transferable skills. Page 94 of the The Ontario Curriculum, Grades 11 and 12, Canadian and World Studies states that throughout the course, students will:

A2.1 describe ways in which economic investigations can help them develop skills, including the essential skills in the Ontario Skills Passport (e.g., reading texts, writing, document use, computer use, oral communication, numeracy skills) and skills related to financial literacy, that can be transferred to postsecondary opportunities, the world of work, and everyday life.
A2.2 apply in everyday contexts skills and work habits developed through economic investigations (e.g., use skills related to budgeting, or weighing opportunity costs, to help them make responsible financial decisions; analyse trade-offs to make informed consumer decisions; analyse the meaning of statistics in a news report; apply work habits such as collaboration to help them deal with conflict and build consensus, or self-regulation to monitor their progress towards a particular financial goal).




The Ontario Skills Passport

The Essential Skills area of the Ontario Skills Passport outlines and provides a brief description the following essential skills:

• Reading Text
• Writing
• Document Use
• Computer Use
• The use of speech for a variety of purposes.
• Money Math
• Scheduling or Budgeting and Accounting
• Measurement and Calculation
• Data Analysis
• Numerical Estimation
• Job Task Planning and Organizing
• Decision Making
• Problem Solving
• Finding Information
• Critical Thinking

Moreover, the OSP outlines a separate list of skills under what it calls "Work Habits." The habits under this section are described below:

Work Habits

As it is critical that everyone in both academia and the work force possess strong work habits and skills, the Ontario Skills Passport includes a section on work habits. The work habits section of the Ontario Skills Passport includes:

• Teamwork
  • Working willingly with others
  • Showing respect for the ideas and opinions of others
  • Taking responsibility for his or her share of the work
  • Contributing to the team effort by sharing information, resources, and expertise

 

• Reliability
  • Being punctual
  • Following directions
  • Giving attention to detail
  • Using time effectively and producing work on time
  • Acting in accordance with health and safety practices

 

• Organization
  • Organizing work priorities when faced with a number of tasks
  • Devising and following a coherent plan to complete a task
  • Revising the plan when necessary to complete a task or to make improvements

 

• Working Independently
  • Accomplishing tasks independently
  • Independently selecting, evaluating, and using appropriate materials, tools, resources, and activities
  • Using prior knowledge and experience to solve problems and make decisions

 

• Initiative
  • Beginning and completing tasks with little prompting
  • Approaching new tasks with confidence and a positive attitude
  • Seeking assistance when necessary

 

• Self-advocacy
  • Asking questions and seeking clarification, where appropriate
  • Identifying and making use of appropriate resources and support when needed
  • Being proactive regarding individual rights and responsibilities, where appropriate

 

• Customer Service
  • Listening effectively to determine and meet clients' needs
  • Interacting positively with both co-workers and clients/customers
  • Endeavouring to meet and exceed expectations
  • Creating a positive impression of the company or organization

 

• Entrepreneurship
  • Recognizing and acting on opportunities
  • Showing perseverance
  • Being innovative and creative
  • Being versatile and resourceful

But... I thought we weren't allowed to mark learning skills?

Many teachers, parents, and students in Ontario are familiar with the Ontario Ministry of Education's general policy stating that the evaluation of learning skills and work habits should not be considered in the determination of a student’s grades. This directive comes from page 10 of the Growing Success policy document, under a section entitled Learning Skills and Work Habits in Grades 1 to 12. However, this particular policy is rarely cited in its full, original form. The directive actually states:

"To the extent possible, however, the evaluation of learning skills and work habits, apart from any   that may be included as part of a curriculum expectation in a subject or course, should not be considered in the determination of a student’s grades." [Emphasis added.]

After discussing a couple of specific examples where the evaluation of learning skills would in fact be associated with a mark for courses that do include learning skills within their curriculum expectations, the document goes on to state:

"In fact, achievement of the curriculum expectations in many curriculum areas is closely tied to learning skills and work habits." [Emphasis added.]

Thus, the inclusion of transferable skills across the entire social studies curriculum does not represent the policy contradiction that some people might believe. As we can see, Growing Success does indeed allow for learning skills to be considered towards the determination of a student's grade as long as those learning skills are associated with specific curriculum expectations. And, as Growing Success itself states, this actually occurs "in many curriculum areas" (page 10).