Assessment Musings In the past few weeks our team has been asked to review assessments (mostly quizzes and tests) with an eye to clarity and fairness. Instructors were worried about question phrasing, multiple choice construction, weightings, time limits among other things. It was wonderful work and we learned much from working with our colleagues across the faculties.
In our interactions with individual instructors, we also discovered that sometimes the assessment is superficial and has only a tenuous link to the course outcomes. Sometimes it is because the instructor has been put into an awkward circumstance; they were handed a course and they are just trying to make a go of it (And might not get that course ever again!). Previous instructors made the text their curriculum and adopted it as “the source of truth”. The result is many “legacy” assessments that merely ask students to recall information. But, “List the four different kinds of organizational structure that the author identifies in the text” is not a great question. Questions like that reward “rightness” rather than comprehension. I’ll give you an example. Years ago, I was visiting a Social Studies class where students had been asked to work with terms associated with the Industrial Revolution. They were given a list of terms on one side of a page and a flowchart on the other, asking them to place the term where it might seem best. I sidled up to one student and asked him how it was going. He said “fine”, as he copied out definitions from the back of the textbook onto the paper he was given. I decided to find out “just how fine it was” going by giving him a quick quiz. I pointed to his paper and asked him to explain what capitalism was. “It is an economic system, based upon supply and demand whereby an entrepreneur controls the means of production.” He shared. “Aha”, said I, “So what is an economic system and what is an entrepreneur?” “Well, it’s got something to do with money. Not sure who this entrepreneur guy is either, but he controls the means of production!” He shrugged. I won’t bother you with the rest of the conversation. Suffice it to say, that student had a very superficial understanding of many of the terms on his sheet. However, if his instructor gave the student a short answer test asking him to define capitalism and a few other terms, he might attain full marks, if he had managed to memorize these definitions. And the instructor might be happy with this; a short answer/recall type of test would be easy to mark, provide full set of class marks and reward those students who spent time in studying (memorizing). But did real learning actually happen? I’m not so sure. That’s why our assessments should align with course GLOs and SLOs (general and specific learning outcomes). These outcomes have been crafted with an eye to essential understandings and skill development that we need to focus on in our teaching and assessment. Readings and resources, video clips and teacher lectures are used to reinforce these outcomes and not to replace them. So, our challenge is in designing assessments that truly assess the knowledge, skills, and attitudes that our students need, and that can be hard work. Related article: A Table of Contents is not a Curriculum
0 Comments
Polling Polling can be a valuable tool for instructors to engage with their students, assess student understanding, and tailor their teaching approach accordingly. It can be used both in online and in-person settings to gauge student interest, evaluate the effectiveness of lessons, prepare students for upcoming topics, and encourage active participation. Let's explore how a college instructor teaching an environmental studies course could utilize polling for these purposes:
Online Polls In an online setting, instructors can utilize polling features within video conferencing platforms or employ dedicated polling tools (like Mentimeter, Quizlet, Polly or Poll Everywhere). They can pose questions related to the course material and ask students to respond in real-time. The results can be shared with the class, enabling students to compare their answers and discuss the reasoning behind their choices in a chat or discussion forum. Snowball Polls
In a classroom setting, students can be polled by a simple raising of hands or thumbs up/thumbs down activity. However, if you need to preserve anonymity, you might think of using a snowball war poll. A snowball war poll activity is an interactive and engaging polling exercise that involves students or participants in a playful manner. It is typically used to gather opinions, preferences, or votes on a specific topic or set of options. The activity gets its name from the concept of a snowball fight, where participants throw snowballs at each other. In a snowball war poll activity, the process involves several steps:
The snowball war poll activity encourages active participation, stimulates discussion, and provides a sense of anonymity, allowing participants to express their opinions freely. It adds an element of surprise and unpredictability, as participants are not aware of whose response they have picked up until they share it. This activity can be particularly useful when exploring subjective topics, gathering diverse opinions, or encouraging collaborative learning and critical thinking. By incorporating polling into their teaching strategies, college instructors can gain valuable insights into student interests, comprehension levels, and opinions. This information allows them to adjust their teaching methods, create engaging lessons, and foster active participation among students, ultimately enhancing the learning experience in an environmental studies course. Linking Learning Asynchronous to Synchronous Friday SoTL Canada Last Friday I took part in the first ever SoTL Canada Roadshow. Approximately 50 educational researchers from colleges and universities across Canada participated in a Zoom call to discuss how Covid-19 has affected their inquiries into teaching and learning. In this musing, I’d like to share with you just how the meeting was conducted, and not what was discussed (that’s another musing). In specific, I’d like to quickly share how we did breakout rooms. The organizers came up with nine different topics for the breakouts. Each question had a different wrinkle on how Covid-19 may or may not have affected our work. These questions were carefully laid out on a google doc that was shared with us through a link in the chat box. The google doc also explained the process and the roles we were to use while working through our discussion. The process was simple: 1) introduce yourselves and your context, 2) assign roles quickly (order of birthdays?), and 3) carry out a focused discussion. The roles were as following:
We had a very productive discussion and it is easy to see why:
High quality learning experiences take thoughtfulness and planning, and this “roadshow” was certainly well-planned. Thursday’s Talking Circle Curiously enough, what I experienced in the SoTL Canada Roadshow very much aligned with what I had been hearing from some our dedicated faculty only the day before. At least three different instructors mentioned that they use the breakout rooms in a very similar way. In the face of Covid-19 and the move to online instruction, they had found ways to make learning focused, personally relevant, engaging, and accountable. This is an encouraging aspect of the Covid-19 lockdown year. From my work with faculty, from my discussions with other faculty developers, and from the data Viola and I are going through, I can see that the pandemic has challenged educators to rethink their planning and practices, their activities and assessments, and their modes and methodologies in a very productive way. We’ve seen a bit of a “leap forward” as many teachers, instructors and professors closely examine not only what they teach, but how they might teach it to ensure engagement and proficiency. Last Thursday, we held another Faculty Talking Circle. This one asked instructors to share their favorite kinds of tools, activities and assignments when connecting asynchronous and synchronous learning. More specifically, we asked:
What follows is a bit of a summary of what I learned from my colleagues. Keeping students engaged and connected throughout the course. According to Thursday’s participants, engagement starts with quality course design. In addition to focusing on content and concepts, think about:
Facilitating Asynchronously
Asynchronous work should not feel disconnected from the rest of the course, like a modern-day iteration of the old “distance learning” that many students had to do for their high school option courses (mailing lessons off to Barrhead to be reviewed and assessed). What Covid-19 has shown us is that there are many different ways to involve students in meaningful inquiry, engage them in asynchronous discussion and collaboration, and build on concepts, skills and understandings highlighted in synchronous lessons. Thursday’s instructors urged us all to:
Facilitating Synchronously While it was always challenging to keep students following during lengthy face-to-face lecture classes, the move to online instruction has compounded this. Instructors shared that, initially, their synchronous classes were either 1) online versions of their old face-to-face PPT lectures, or lengthy and very chatty check-ins with the students that had little educational value. First attempts using breakout rooms were chaotic, staying on top of the chat and students statuses was challenging, and getting students to ask good questions or respond to teacher questions was like pulling teeth. Over time, these same instructors learned to:
Other ideas that emerged from the session:
Thanks to all the instructors who participated in either the morning or afternoon session! We will have another Faculty Talking Circle next week Friday (the 23rd). Not sure what the topic will be. Suggestions? Student Survey (a few quick observations)
As part of our research study into the transition to online teaching due to Covid-19, Viola and I sent a survey out to NorQuest students, to see if their experiences align with instructor perception and experience. One week in, we already have 400 student responses! Now properly examining and unpacking such a data set will require a considerable amount of time and focus, but I thought I might share just a few tidbits. The students considered a number of closed and open questions, but the two that were the most interesting were:
Concerning “what works”, the two most frequently mentioned aspects were 1) the ability to watch pre-recorded lectures or recordings of the synchronous classes on their own time and in their own way (25), and 2) the flexibility of online learning so they could plan their own schedules around work, family commitments and other class work (17). Quite a number of students also remarked on how the online experience has saved them time and money with no commute and a flexible schedule (13). Students also wrote that they appreciated instructors who used the tech effectively (“BBU lessons are awesome!”), especially when the instructors use the chat and microphones well, weave in games, breakouts & polls, and maintained regular communication through check-ins, forums and messaging (8). Several students remarked that “Moodle organization has improved” (6), PowerPoints and lectures are more concise (3), and that instructors are giving more clarity about expectations between lessons (3). In addition, there were students who confessed that the online experience has taught them more about tech and that they have become much more self-disciplined (4). The list of what wasn’t working for the students ran a little bit longer. Okay, a fair bit longer. Here are some of the issues that bubbled to the top of the list (so far, I’ve only got through 200 or so of the 400 responses):
Of course, it is early in the data analysis stage, but I thought I might share some of the findings. There seems to be several very different groups of students going through this change: those who are actually thriving in a flexible environment where they can watch recordings and take charge of their learning, those who feel disconnected and long for the in-person connections, and those who are challenged by the technology and demands of online connectivity. Moreover, a great deal depends on who their instructors are and just how engaging they build their asynchronous and synchronous environments. I hope we might address some of these findings in the talking circle on Friday. Focus on Assessment – Again! This past Friday we had a Faculty Talking Circle on Online Assessment. It was a rich discussion and a very honest conversation! During the weekend, I did a bit more thinking about what we discussed. Here are a few thoughts. Some of you may know that our family recently added two new members to it! Edgar and Daisy (perky ears!), two precocious little border collies. Right now they are about 12 weeks old and are still learning many things - so are we. Just after we took the siblings home we arranged a visit with the veterinarian for a check-up. One of the things we mentioned was that Edgar had the habit of bolting his food while Daisy would take a mouthful, walk a bit of a distance away and carefully chew what she had before returning to get another mouthful. By that time, Edgar may have eaten all of his portion and most of hers. It didn’t seem fair or equal! We didn’t want Edgar to end up being a fat little dog, especially at Daisy’s expense, so we thought about limiting his food intake or feeding Daisy separately. The vet told us to take a deep breath and relax. Just keep adding food to the dish; puppies self-regulate and limiting Edgar’s food intake might be harmful. He is likely moving through a growth spurt and, judging from his body type, he will likely be a bigger dog. Trying to be equal wouldn’t be fair; especially to Edgar. Yet, as educators we consistently fall into the trap of “fair must be equal”. We don’t want to be seen as favoring particular students and we sometimes hesitate about providing extra resources, support, time or opportunity, because “it wouldn’t be fair to the rest of the class”. Why should a particular student get an extension on their project? They had better have some dire circumstances! How will they understand about deadlines and the “real world”? But fair isn’t equal. Fair is about giving every student the chance to thrive and succeed, and some students will need different kinds of support than others. Another example. Although I don’t often think of it as such, I have a learning accommodation. I wear eyeglasses. Without my beloved blue spectacles, I would be hard pressed to complete most tasks. If I had been taught in an environment where decisions were based solely on keeping all things equal, I should not have been permitted to wear them. Why should I be able to wear glasses while others did not? The fact that the others did not need glasses is immaterial. It’s not fair! It’s not equal! But, as educators, we think little of having students with glasses or hearing aids. They don’t get accommodations, they have aids that do little to interfere with our planning and practice. And, we do have many students who receive accommodations; and we do our best to support them. We have students who need extra time to process, students who need a scribe (graphomotor issues), students who need readers, students who need interpretation, and students who need different colour papers. We are trying to be fair by removing some of the barriers that might interfere with their learning. Their extra time, prepared notes, relaxed deadline, reader, or scribe is my glasses. Friday’s Discussion That brings me back to the Friday Talking Circle on Assessment. Originally I had intended the session to be a bit of an idea exchange. I invited a colleague from Lakeland College, Mabyn Grinde, to join us for the session, as she is working through many of the same challenges with the faculty there. Over twenty instructors were able to attend, as well as Joan Wall and Kerry Taillefer, who would help field questions! The plan was simple. Share some of the assessment principles seek input from instructors. Ask instructors to share some of their current challenges related to online assessment. Elicit a few suggestions and successes from each other that might improve our practices. Oh, and yes, maybe also talk about the new UDL 1.5 time requirement for conducting assessments. However, the 1.5 UDL (Universal Design for Learning) requirement quickly became the major focus of our discussion. The discussion became passionate and, at times, pointed. It was an honest conversation! We were concerned about what is fair, what is equal and whether students might get an unfair advantage. From my understanding the move to universal 1.5 timing for assessments was intended to allow students who may not have had learning challenges formally assessed, to be given the extra time. Since the onset of the Covid-19 pandemic, many of our students may have struggled in isolation, with barriers to achievement such as inadequate technology or Wi-Fi access, anxiety or uncertainly related to the pandemic, childcare and other familial or relational obligations, shared use of technology (one computer for a whole family to do their work on), employment responsibilities and other stressors. Students who do not have these stressors would likely not take advantage of the extra time, but those who need the extra accommodation would be able to use it. Many of the faculty in the talking circle were, for the most part, very sympathetic to the concept of time and a half. Our students are under a lot of pressure and this decision seemed to come from a caring place. Never the less, there were a lot of questions and concerns related to this move.
Compounding the issues raised by the faculty is the fact that many of the established, closed-answer assessments (especially those with right/wrong answers) no longer work in an online world where students can screen capture and instant message. Open-ended assessments require more time to ideate, build, design effective scoring guides for, and mark! Getting More Context As special guests, I invited William Hamilton (Assistive Technology) and Kerry Taillefer (Testing Centre & Learning Support) to help address some of the emergent questions that might bubble up in the talking circle. Recently, William, Kerry, Joan and I were charged with helping the faculty learn about the 1.5 UDL requirement and it has been a bit of a challenge. The decision has also affected our roles and department staffing! William unfortunately could not attend the Friday talking circle. He did send along a few messages though:
Kerry provided some excellent points to consider as we work our way through this change. She has been looking into research on this matter and found out that:
Kerry suggested that we should work with our students to prepare them for the exams, make them aware of test-taking strategies and help them avoid stressing before or over-thinking during assessments. She also suggested talking with students about on how cheating in the short term can have long-term repercussions when they are asked to apply information with an incomplete understanding. I was extremely glad to have Kerry there. A number of instructors in the circle affirmed her observations about students using or abusing extra time. At the same time, we could all agree that: “there are no magic wands” and one-size does not fit all. However, as instructional faculty and faculty support, we needed to “find ways forward”. Then we shifted gears to discuss some solutions. Finding Ways Forward
At this point Mabyn was able to share some of the ways that the faculty at Lakeland have responded to the need reality of online assessment. Some instructors at Lakeland had success with open-book and oral exams. It was a big shift away from the typical battery of 200 multiple-choice questions. The success of open book exams had much to do with the structure of them and the challenge found in them. Students were given a set block of time (a day?) to explore questions that spoke to the big questions of the course and their program. In oral exams, with some carefully crafted questions and strategic probing, instructors could find out very quickly who knew and could make connections, and who could not. Some of the instructors Mabyn worked with also had success in constructing case-study types of questions that required longer answer responses. In all three cases, making sure that the marking criteria (rubrics) are clear was essential. One instructor shared that, after conducting these new assessments, he had a much better picture of “who got it” than he did from the old multiple-choice exams. He could see how students were intersecting ideas and applying them. Some of the other suggestions shared during the session for addressing the challenge of online assessment (and even the UDL 1.5 requirement) included:
Last week I shared some handy websites from Taylor Institute (U of C). Here are a couple of others that people shared on Friday. Websites: https://learninginnovation.ca/student-cc/ (Lethbridge College Centre for Teaching and Learning) https://sites.tufts.edu/teaching/2020/05/08/integrating-inclusive-and-sustainable-assessments-in-your-online-teaching-from-beginning-to-end/ (Teaching and Learning at Tufts University) https://teaching.uwo.ca/elearning/student_assessment/alt-assessment-ideas.html (Centre for Teaching and Learning – Western Ontario) Have a wonderful week. I’m going out to play with my puppies! Jeff |
AuthorJeff Kuntz Ph.D. ImagesExcept where indicated, images used in the blog posts are personal photos, images from NorQuest College or images from Pixabay. Pixabay is a vibrant community of creatives, sharing copyright free images, videos and music. https://pixabay.com/ Archives
March 2024
Categories
All
|