Musings on Assessment The question of whether instructors should reward participation/conscientiousness is one that has provoked a lot of debate, especially between educational experts (often academics) and those charged with teaching (hard working practitioners!). When I have talked to NorQuest instructors about this issue, they have shared several reasons for using “participation marks”:
Instead, these educators suggest making a concerted effort to focus on competence by:
0 Comments
Assessment Musings In the past few weeks our team has been asked to review assessments (mostly quizzes and tests) with an eye to clarity and fairness. Instructors were worried about question phrasing, multiple choice construction, weightings, time limits among other things. It was wonderful work and we learned much from working with our colleagues across the faculties.
In our interactions with individual instructors, we also discovered that sometimes the assessment is superficial and has only a tenuous link to the course outcomes. Sometimes it is because the instructor has been put into an awkward circumstance; they were handed a course and they are just trying to make a go of it (And might not get that course ever again!). Previous instructors made the text their curriculum and adopted it as “the source of truth”. The result is many “legacy” assessments that merely ask students to recall information. But, “List the four different kinds of organizational structure that the author identifies in the text” is not a great question. Questions like that reward “rightness” rather than comprehension. I’ll give you an example. Years ago, I was visiting a Social Studies class where students had been asked to work with terms associated with the Industrial Revolution. They were given a list of terms on one side of a page and a flowchart on the other, asking them to place the term where it might seem best. I sidled up to one student and asked him how it was going. He said “fine”, as he copied out definitions from the back of the textbook onto the paper he was given. I decided to find out “just how fine it was” going by giving him a quick quiz. I pointed to his paper and asked him to explain what capitalism was. “It is an economic system, based upon supply and demand whereby an entrepreneur controls the means of production.” He shared. “Aha”, said I, “So what is an economic system and what is an entrepreneur?” “Well, it’s got something to do with money. Not sure who this entrepreneur guy is either, but he controls the means of production!” He shrugged. I won’t bother you with the rest of the conversation. Suffice it to say, that student had a very superficial understanding of many of the terms on his sheet. However, if his instructor gave the student a short answer test asking him to define capitalism and a few other terms, he might attain full marks, if he had managed to memorize these definitions. And the instructor might be happy with this; a short answer/recall type of test would be easy to mark, provide full set of class marks and reward those students who spent time in studying (memorizing). But did real learning actually happen? I’m not so sure. That’s why our assessments should align with course GLOs and SLOs (general and specific learning outcomes). These outcomes have been crafted with an eye to essential understandings and skill development that we need to focus on in our teaching and assessment. Readings and resources, video clips and teacher lectures are used to reinforce these outcomes and not to replace them. So, our challenge is in designing assessments that truly assess the knowledge, skills, and attitudes that our students need, and that can be hard work. Related article: A Table of Contents is not a Curriculum Focus on Assessment – Again! This past Friday we had a Faculty Talking Circle on Online Assessment. It was a rich discussion and a very honest conversation! During the weekend, I did a bit more thinking about what we discussed. Here are a few thoughts. Some of you may know that our family recently added two new members to it! Edgar and Daisy (perky ears!), two precocious little border collies. Right now they are about 12 weeks old and are still learning many things - so are we. Just after we took the siblings home we arranged a visit with the veterinarian for a check-up. One of the things we mentioned was that Edgar had the habit of bolting his food while Daisy would take a mouthful, walk a bit of a distance away and carefully chew what she had before returning to get another mouthful. By that time, Edgar may have eaten all of his portion and most of hers. It didn’t seem fair or equal! We didn’t want Edgar to end up being a fat little dog, especially at Daisy’s expense, so we thought about limiting his food intake or feeding Daisy separately. The vet told us to take a deep breath and relax. Just keep adding food to the dish; puppies self-regulate and limiting Edgar’s food intake might be harmful. He is likely moving through a growth spurt and, judging from his body type, he will likely be a bigger dog. Trying to be equal wouldn’t be fair; especially to Edgar. Yet, as educators we consistently fall into the trap of “fair must be equal”. We don’t want to be seen as favoring particular students and we sometimes hesitate about providing extra resources, support, time or opportunity, because “it wouldn’t be fair to the rest of the class”. Why should a particular student get an extension on their project? They had better have some dire circumstances! How will they understand about deadlines and the “real world”? But fair isn’t equal. Fair is about giving every student the chance to thrive and succeed, and some students will need different kinds of support than others. Another example. Although I don’t often think of it as such, I have a learning accommodation. I wear eyeglasses. Without my beloved blue spectacles, I would be hard pressed to complete most tasks. If I had been taught in an environment where decisions were based solely on keeping all things equal, I should not have been permitted to wear them. Why should I be able to wear glasses while others did not? The fact that the others did not need glasses is immaterial. It’s not fair! It’s not equal! But, as educators, we think little of having students with glasses or hearing aids. They don’t get accommodations, they have aids that do little to interfere with our planning and practice. And, we do have many students who receive accommodations; and we do our best to support them. We have students who need extra time to process, students who need a scribe (graphomotor issues), students who need readers, students who need interpretation, and students who need different colour papers. We are trying to be fair by removing some of the barriers that might interfere with their learning. Their extra time, prepared notes, relaxed deadline, reader, or scribe is my glasses. Friday’s Discussion That brings me back to the Friday Talking Circle on Assessment. Originally I had intended the session to be a bit of an idea exchange. I invited a colleague from Lakeland College, Mabyn Grinde, to join us for the session, as she is working through many of the same challenges with the faculty there. Over twenty instructors were able to attend, as well as Joan Wall and Kerry Taillefer, who would help field questions! The plan was simple. Share some of the assessment principles seek input from instructors. Ask instructors to share some of their current challenges related to online assessment. Elicit a few suggestions and successes from each other that might improve our practices. Oh, and yes, maybe also talk about the new UDL 1.5 time requirement for conducting assessments. However, the 1.5 UDL (Universal Design for Learning) requirement quickly became the major focus of our discussion. The discussion became passionate and, at times, pointed. It was an honest conversation! We were concerned about what is fair, what is equal and whether students might get an unfair advantage. From my understanding the move to universal 1.5 timing for assessments was intended to allow students who may not have had learning challenges formally assessed, to be given the extra time. Since the onset of the Covid-19 pandemic, many of our students may have struggled in isolation, with barriers to achievement such as inadequate technology or Wi-Fi access, anxiety or uncertainly related to the pandemic, childcare and other familial or relational obligations, shared use of technology (one computer for a whole family to do their work on), employment responsibilities and other stressors. Students who do not have these stressors would likely not take advantage of the extra time, but those who need the extra accommodation would be able to use it. Many of the faculty in the talking circle were, for the most part, very sympathetic to the concept of time and a half. Our students are under a lot of pressure and this decision seemed to come from a caring place. Never the less, there were a lot of questions and concerns related to this move.
Compounding the issues raised by the faculty is the fact that many of the established, closed-answer assessments (especially those with right/wrong answers) no longer work in an online world where students can screen capture and instant message. Open-ended assessments require more time to ideate, build, design effective scoring guides for, and mark! Getting More Context As special guests, I invited William Hamilton (Assistive Technology) and Kerry Taillefer (Testing Centre & Learning Support) to help address some of the emergent questions that might bubble up in the talking circle. Recently, William, Kerry, Joan and I were charged with helping the faculty learn about the 1.5 UDL requirement and it has been a bit of a challenge. The decision has also affected our roles and department staffing! William unfortunately could not attend the Friday talking circle. He did send along a few messages though:
Kerry provided some excellent points to consider as we work our way through this change. She has been looking into research on this matter and found out that:
Kerry suggested that we should work with our students to prepare them for the exams, make them aware of test-taking strategies and help them avoid stressing before or over-thinking during assessments. She also suggested talking with students about on how cheating in the short term can have long-term repercussions when they are asked to apply information with an incomplete understanding. I was extremely glad to have Kerry there. A number of instructors in the circle affirmed her observations about students using or abusing extra time. At the same time, we could all agree that: “there are no magic wands” and one-size does not fit all. However, as instructional faculty and faculty support, we needed to “find ways forward”. Then we shifted gears to discuss some solutions. Finding Ways Forward
At this point Mabyn was able to share some of the ways that the faculty at Lakeland have responded to the need reality of online assessment. Some instructors at Lakeland had success with open-book and oral exams. It was a big shift away from the typical battery of 200 multiple-choice questions. The success of open book exams had much to do with the structure of them and the challenge found in them. Students were given a set block of time (a day?) to explore questions that spoke to the big questions of the course and their program. In oral exams, with some carefully crafted questions and strategic probing, instructors could find out very quickly who knew and could make connections, and who could not. Some of the instructors Mabyn worked with also had success in constructing case-study types of questions that required longer answer responses. In all three cases, making sure that the marking criteria (rubrics) are clear was essential. One instructor shared that, after conducting these new assessments, he had a much better picture of “who got it” than he did from the old multiple-choice exams. He could see how students were intersecting ideas and applying them. Some of the other suggestions shared during the session for addressing the challenge of online assessment (and even the UDL 1.5 requirement) included:
Last week I shared some handy websites from Taylor Institute (U of C). Here are a couple of others that people shared on Friday. Websites: https://learninginnovation.ca/student-cc/ (Lethbridge College Centre for Teaching and Learning) https://sites.tufts.edu/teaching/2020/05/08/integrating-inclusive-and-sustainable-assessments-in-your-online-teaching-from-beginning-to-end/ (Teaching and Learning at Tufts University) https://teaching.uwo.ca/elearning/student_assessment/alt-assessment-ideas.html (Centre for Teaching and Learning – Western Ontario) Have a wonderful week. I’m going out to play with my puppies! Jeff Looking at Assessment!
This week I wanted to talk about assessment. It is the most significant factor in encouraging and sustaining student success. Formative assessment (Assessment For and As Learning) helps to provide students and instructors with an opportunity to check on understanding, clarify learning targets, re-calibrate, and build for success. As many leading educators have pointed out, this kind of assessment needs to happen minute by minute, throughout lessons and activities; there needs to be a constant going back and forth of checking and rechecking. Summative Assessment (Assessment Of Learning) usually comes at the end of a cycle, and it helps students and instructors to measure success against clearly defined parameters. Summative assessment requires educators to establish routines and procedures that promote fairness, reliability, and validity. However, while few educators would debate the impact and role of quality assessment practices, there has been hot debate about what these practices are. A number of years ago I was involved in a district-wide assessment reform that called on K-12 teachers to re-examine their assessment practices. Questions were asked like:
That year we tackled some very prickly pears. There were parents who were very upset that their child’s progress was no longer indicated by percentile (“How can I compare my child to the rest of the class? What does “proficient” really mean? Is it a 63% or 75%? There’s a big difference!”), and teachers who bristled at the idea of no zeros and second chances (“How can I ensure that they learn their lesson about tardiness?”). Not every issue had an easy answer. Nonetheless, as a district, we learned a great deal about assessment. It was a great leap forward for all of us. The present circumstances, the transition to whole scale online learning prompted by the Covid-19 pandemic, has brought on a similar kind of college-wide assessment reform for NorQuest. This time the questions are a little different though:
Since NorQuest moved to exclusively online instruction (with some obvious exceptions), I could feel the tension mounting around just these questions (and many more). Programs that relied on 2-4 big, multiple-choice assessments to provide evidence of understanding and achievement were suddenly faced with the challenges of conducting these exams online. Other programs that required physical demonstration of situational competency, practical skills, or professional judgement were also left struggling. One recent change to how we assess at NorQuest College has been with respect to exam accommodations and to the disability intake process:
These changes are more fully explained in a couple of emails that were sent out earlier this month. If you missed the emails, have a search through your inbox, check with your chair or associate chair, or contact William Hamilton. In the next few weeks William Hamilton, Alan Jeans, Kerry Taillefer, Joan Wall and I will be meeting with departmental groups to help clarify what this means for your students and for you. With all of these assessment pressures and changes in mind, Norma Schneider met with the Deans, Chairs, and Associates during the Week of Welcome to discuss what we, as NorQuesters, see as the most important components and principles in effective online assessment. Some of the frequently mentioned terms included: authenticity, alignment, relevance, choice, reliability, rehearsal, “aha” moments, variation, growth opportunity, transparency, and ownership. Norma urged us to look past assessments that rely solely on rote and recall, and think of ways to assess that will challenge students to analyze, apply and construct based upon scenarios or problems. It reminded me of what one of my Education professors used to say to us, when I was taking my undergraduate degree many moons ago (almost 40 years!). Dr. Glenn Martin (U of A) asked us if we planned to use assessment as an affirmation to find out what students know and can do, or as a “gotcha” to find out what they missed or failed to master. All too often, in an effort to “separate the wheat from the chaff”, we build assessments that test for perfection and not for proficiency. Such a practice makes it easier to rank and sort the students, based on where they fit on the curve (normative-based). These kinds of assessments also favor those students who are very good at memorizing or at taking multiple-choice tests. Dr. Martin advocated for a range of assessment types, that might include multiple choice but also include short and long answer tests, portfolios, projects, forums, problem-based learning, role-plays and many other ways of finding out what your students know and can do. Till next week! Jeff |
AuthorJeff Kuntz Ph.D. ImagesExcept where indicated, images used in the blog posts are personal photos, images from NorQuest College or images from Pixabay. Pixabay is a vibrant community of creatives, sharing copyright free images, videos and music. https://pixabay.com/ Archives
March 2024
Categories
All
|