Musings on Assessment The question of whether instructors should reward participation/conscientiousness is one that has provoked a lot of debate, especially between educational experts (often academics) and those charged with teaching (hard working practitioners!). When I have talked to NorQuest instructors about this issue, they have shared several reasons for using “participation marks”:
Instead, these educators suggest making a concerted effort to focus on competence by:
0 Comments
Assessment Musings In the past few weeks our team has been asked to review assessments (mostly quizzes and tests) with an eye to clarity and fairness. Instructors were worried about question phrasing, multiple choice construction, weightings, time limits among other things. It was wonderful work and we learned much from working with our colleagues across the faculties.
In our interactions with individual instructors, we also discovered that sometimes the assessment is superficial and has only a tenuous link to the course outcomes. Sometimes it is because the instructor has been put into an awkward circumstance; they were handed a course and they are just trying to make a go of it (And might not get that course ever again!). Previous instructors made the text their curriculum and adopted it as “the source of truth”. The result is many “legacy” assessments that merely ask students to recall information. But, “List the four different kinds of organizational structure that the author identifies in the text” is not a great question. Questions like that reward “rightness” rather than comprehension. I’ll give you an example. Years ago, I was visiting a Social Studies class where students had been asked to work with terms associated with the Industrial Revolution. They were given a list of terms on one side of a page and a flowchart on the other, asking them to place the term where it might seem best. I sidled up to one student and asked him how it was going. He said “fine”, as he copied out definitions from the back of the textbook onto the paper he was given. I decided to find out “just how fine it was” going by giving him a quick quiz. I pointed to his paper and asked him to explain what capitalism was. “It is an economic system, based upon supply and demand whereby an entrepreneur controls the means of production.” He shared. “Aha”, said I, “So what is an economic system and what is an entrepreneur?” “Well, it’s got something to do with money. Not sure who this entrepreneur guy is either, but he controls the means of production!” He shrugged. I won’t bother you with the rest of the conversation. Suffice it to say, that student had a very superficial understanding of many of the terms on his sheet. However, if his instructor gave the student a short answer test asking him to define capitalism and a few other terms, he might attain full marks, if he had managed to memorize these definitions. And the instructor might be happy with this; a short answer/recall type of test would be easy to mark, provide full set of class marks and reward those students who spent time in studying (memorizing). But did real learning actually happen? I’m not so sure. That’s why our assessments should align with course GLOs and SLOs (general and specific learning outcomes). These outcomes have been crafted with an eye to essential understandings and skill development that we need to focus on in our teaching and assessment. Readings and resources, video clips and teacher lectures are used to reinforce these outcomes and not to replace them. So, our challenge is in designing assessments that truly assess the knowledge, skills, and attitudes that our students need, and that can be hard work. Related article: A Table of Contents is not a Curriculum Exit slips ask the students to reflect upon what they have learned that day. At the end of the period, with perhaps five or so minutes to go before the bell, the instructor will hand out a slip of paper with just one or two simple questions on it. A typical exit slip looks like this: Some instructors use a 3-2-1 format, “Tell me 3 things you learned, 2 questions you still have and one suggestion for next steps…” Other instructors have used “entry slips” to gauge readiness.
The slips are meant to be a very informal and anonymous assessment of the lesson and can give valuable information on whether the students have caught what is being taught – or not! They provide an avenue for students who may be too shy to partake in classroom discussions to have some input into the way the course is delivered. Instructors who effectively use exit slips use them periodically (every day is too often) to inform and refine their practice. Typically, the instructor would review the slips before the next class, note some of the dominant sentiments and questions and then start the next lesson by first addressing what was raised on the exit slips from the day before. This process helps prevent the teacher from pushing on into difficult concepts that some students might not be ready for, or from plodding on with material that the students find to be too little of a challenge. Online the “slips” can be collected through private chats to the instructor, using the feedback option in Moodle or through applications like Mentimeter or other polling apps. Quick writes provide one way for instructors and learners to find out just how much of what is being discussed and reviewed is being caught by the learner. In this strategy the instructor learners to take a moment and reflect upon a question or prompt related to what they are currently studying. The question may ask the learners to draw upon their background knowledge, make a prediction, give an analysis, relate a personal anecdote, take a stand on a particular issue, or visualize a circumstance. Along with the question you should give the learners a limited amount of time (two to five minutes) in which they free write a response.
For instance, you might ask the learners to reflect upon their last visit to a doctor and to consider all the aspects that a general practitioner is responsible for. In this case you are looking to use their background knowledge to activate this topic and give direction to the lesson. Some instructors like to use this as the first step in a think/pair/share activity. It allows individual learners to consolidate their thinking before they pair up and begin discussion. But quick writes can also be used to reflect upon on respond to new information. For instance, you might ask the learners to give their response to a video clip you presented on healthy food choices. Even just a walk through the classroom to quickly observe the student responses to this question would tell the instructor just how much learners related to what was being shown and if they caught the most important points of the presentation. These quick writes would likely also provide more questions to consider as a class. Quick writes can be done in face-to-face classrooms on paper, and in hybrid or online classrooms these writes can happen by setting up a forum in Moodle (you can choose to make it public or personal) or through many different polling or posting apps. Some instructors also use the chat for this, asking the students to flood the chat when their allotted writing time is up. The idea behind quick writes is to get the learners comfortable with writing and expressing their ideas. Quick writes can be used for assessment, but most instructors use them only for formative assessment. They provide an interesting window on your learners; you can learn much about their interests and personality through these quick and low-stress responses. But be warned, instructors who collect these “free-writes” to evaluate the learners and assign marks soon find out that the learner writing will become stiffer and less personal. Do you ever find yourself getting tired of marking the same old assignments where students almost follow a formula to crank out papers and you mindlessly score them without really engaging in them? Well some of that tedium can be avoided if only we give more choice to out students in how they might present evidence of understanding and application. One way to provide students with a choice and encourage creativity is to use a rather old assignment format called a RAFT. A RAFT (Vanderventer, 1979) allows students to choose how they want to approach an assignment letting them select their Role, intended Audience, preferred Format and particular Topic from a list of options given by you! Giving students the opportunity to write from the perspective of a small business owner, nutritionist, family member, politician, newspaper reporter or Greenpeace activist might bring some creativity into often stale assignments and stretches the students by asking them to take on perspectives and unpack assumptions that they might not usually consider. And, these assignments can also be much more fun to assess. Instead of having to read or watch 40 rather dry reports on a certain topic, instructors can now read love letters, sample travel guides, follow dialogues, react to newspaper columns, view talk show excerpts, download political speeches, and a whole host of other possibilities. For some instructors this kind of choice can be a little unnerving. “How do I mark such a diverse set of assignments?” “Aren’t the students just going to see this as an excuse to avoid serious research?” These questions illustrate the need be clear about your expectations for the assignment. The fact is, no matter which format they choose, the students need to show that they have a sound grasp of the issue they are assigned or the process they need to illustrate. In fact, working from the rather academic language of texts and resources and then converting it into another form forces the students to consolidate the learnings and then use their understanding to make more refined applications. Imagine trying explain dementia from the perspective of a close relative or care worker, rather than as a student (reciting symptoms from a resource or website). What kinds of challenges might this relative seen and experienced? What kinds of emotions have they encountered? What kinds of supports might they be asking for? These questions also show the importance of having a well-developed rubric or scoring guide for evaluating student work. The rubric should be assessing not to see if the student says all of the “right things” but to see whether the student can show understanding in how they process the task, and in how they explain, interpret, apply show perspective, extend empathy, and show self-awareness (Remember the six facets of understanding from Wiggins and McTighe?). The key to any RAFT assignment is how well it might demonstrate understanding and inquiry, so students could choose very “unreal” scenarios to illustrate some very real concerns. For instance, in a unit on “healthy choices” a student might choose to write a complaint letter written by of a choked set of lungs to the pack-a-day smoker who is subjecting them to this daily abuse. RAFTs are quite versatile. They can be simple and short checks for understanding. They can also be full-blown assignments that require careful investigation and thoughtful expression. How do you use a RAFT?
One Example Here are the beginnings of a RAFT assignment on creating a hydro-electric dam in Southern Alberta. The dam will dispossess numerous land owners, flood an historic plain of traditional hunting grounds and burial sites, bring revenue to the province and certain corporations, provide a steady flow of irrigation to farms in Southern Alberta, provide recreational opportunities(boating, sportfishing and camping), and impact countless animal populations by interfering with their habitat. Students are free to mix and match from all of the columns. Of course, the assignment above would require some more fleshing out as you weigh the kinds of tasks and work with an eye to clarity, structure, fairness, and achievement. But it gives you an idea. Assignments like this could be made for virtually any subject area. Nursing students might be asked to take on the perspective of the patient, the care-giver, the health clinic staff, a family member, or even the virus itself: I am now starting to affect Jeff’s nervous system. In the next couple of days, he will no longer be able to go to the dog park with his border collies. Soon he will have difficulty even handling the TV remote so he can change the channel to watch the Bruins games. He will have to ask his wife to hold his Dr. Pepper up to his lips so he can take a sip through one of those paper straws. It will be a slow, painful degeneration. The possibilities are endless; however, this last example is one I don’t want to explore further! Vanderventer, N (1979, Winter). RAFT: A process to structure prewriting. Highway One: A Canadian Journal of Language Experience, p.26. Wiggins, G. P., & McTighe, J. (2005). Understanding by design (Expanded 2nd). Association for Supervision and Curriculum Development. How might we support our students in preparing for exams?This past week we held two faculty talking circles. They were wonderful! Both circles were well-attended and instructors participating offered helpful suggestions on how to support and encourage students as they prepare for important and often stressful end of term assessments. The instructors were given the following prompt: We are getting to that point in the term where the students are under a lot of pressure to perform. End of module exams and end of term exams loom. And while there is a plethora of websites that suggest the best ways for students to prepare for these challenges - suggesting how to stop procrastinating, make quality notes, and move past recall to better understanding - it does not always follow that students read and apply such helpful advice. How do you support and scaffold your students in preparation for big assessments? Do you share links, develop study guides, set up review classes, schedule drop-ins, manage a chat, pair up students (study buddies), or find some other way to help and encourage? Is your Moodle environment configured to aid students in being confident and competent learners? Or, is studying and exam prep something that each student must take individual ownership of – we are not here to spoon feed or enable our learners! Several suggestions kept coming back as instructors shared their expertise and experience:
Here are many of the other suggestions shared by NorQuest faculty:
Some of the links that were shared while we were in the Talking Circle include: Student Wellness Services self-directed workshop on managing test stress: https://rise.articulate.com/share/pwSBgb-MVbc5ijad-q9s3SA5_tVNOb5L#/lessons/W0gOg6HdekbzQb-sjrT35LM2VKC5sdzR Studying 101: Study Smarter Not Harder (University of North Carolina at Chapel Hill) https://learningcenter.unc.edu/tips-and-tools/studying-101-study-smarter-not-harder/ Studying & Note-Taking Skills for College (Affordable Colleges Online) https://www.affordablecollegesonline.org/college-resource-center/study-skills/ Study Skills (Penn State) https://dus.psu.edu/academicsuccess/studyskills.html Hopefully there is a suggestion or idea that resonates with you! If you would like to go deeper into any of the suggestions above, just contact one of our Faculty Developers! Focus on Assessment – Again! This past Friday we had a Faculty Talking Circle on Online Assessment. It was a rich discussion and a very honest conversation! During the weekend, I did a bit more thinking about what we discussed. Here are a few thoughts. Some of you may know that our family recently added two new members to it! Edgar and Daisy (perky ears!), two precocious little border collies. Right now they are about 12 weeks old and are still learning many things - so are we. Just after we took the siblings home we arranged a visit with the veterinarian for a check-up. One of the things we mentioned was that Edgar had the habit of bolting his food while Daisy would take a mouthful, walk a bit of a distance away and carefully chew what she had before returning to get another mouthful. By that time, Edgar may have eaten all of his portion and most of hers. It didn’t seem fair or equal! We didn’t want Edgar to end up being a fat little dog, especially at Daisy’s expense, so we thought about limiting his food intake or feeding Daisy separately. The vet told us to take a deep breath and relax. Just keep adding food to the dish; puppies self-regulate and limiting Edgar’s food intake might be harmful. He is likely moving through a growth spurt and, judging from his body type, he will likely be a bigger dog. Trying to be equal wouldn’t be fair; especially to Edgar. Yet, as educators we consistently fall into the trap of “fair must be equal”. We don’t want to be seen as favoring particular students and we sometimes hesitate about providing extra resources, support, time or opportunity, because “it wouldn’t be fair to the rest of the class”. Why should a particular student get an extension on their project? They had better have some dire circumstances! How will they understand about deadlines and the “real world”? But fair isn’t equal. Fair is about giving every student the chance to thrive and succeed, and some students will need different kinds of support than others. Another example. Although I don’t often think of it as such, I have a learning accommodation. I wear eyeglasses. Without my beloved blue spectacles, I would be hard pressed to complete most tasks. If I had been taught in an environment where decisions were based solely on keeping all things equal, I should not have been permitted to wear them. Why should I be able to wear glasses while others did not? The fact that the others did not need glasses is immaterial. It’s not fair! It’s not equal! But, as educators, we think little of having students with glasses or hearing aids. They don’t get accommodations, they have aids that do little to interfere with our planning and practice. And, we do have many students who receive accommodations; and we do our best to support them. We have students who need extra time to process, students who need a scribe (graphomotor issues), students who need readers, students who need interpretation, and students who need different colour papers. We are trying to be fair by removing some of the barriers that might interfere with their learning. Their extra time, prepared notes, relaxed deadline, reader, or scribe is my glasses. Friday’s Discussion That brings me back to the Friday Talking Circle on Assessment. Originally I had intended the session to be a bit of an idea exchange. I invited a colleague from Lakeland College, Mabyn Grinde, to join us for the session, as she is working through many of the same challenges with the faculty there. Over twenty instructors were able to attend, as well as Joan Wall and Kerry Taillefer, who would help field questions! The plan was simple. Share some of the assessment principles seek input from instructors. Ask instructors to share some of their current challenges related to online assessment. Elicit a few suggestions and successes from each other that might improve our practices. Oh, and yes, maybe also talk about the new UDL 1.5 time requirement for conducting assessments. However, the 1.5 UDL (Universal Design for Learning) requirement quickly became the major focus of our discussion. The discussion became passionate and, at times, pointed. It was an honest conversation! We were concerned about what is fair, what is equal and whether students might get an unfair advantage. From my understanding the move to universal 1.5 timing for assessments was intended to allow students who may not have had learning challenges formally assessed, to be given the extra time. Since the onset of the Covid-19 pandemic, many of our students may have struggled in isolation, with barriers to achievement such as inadequate technology or Wi-Fi access, anxiety or uncertainly related to the pandemic, childcare and other familial or relational obligations, shared use of technology (one computer for a whole family to do their work on), employment responsibilities and other stressors. Students who do not have these stressors would likely not take advantage of the extra time, but those who need the extra accommodation would be able to use it. Many of the faculty in the talking circle were, for the most part, very sympathetic to the concept of time and a half. Our students are under a lot of pressure and this decision seemed to come from a caring place. Never the less, there were a lot of questions and concerns related to this move.
Compounding the issues raised by the faculty is the fact that many of the established, closed-answer assessments (especially those with right/wrong answers) no longer work in an online world where students can screen capture and instant message. Open-ended assessments require more time to ideate, build, design effective scoring guides for, and mark! Getting More Context As special guests, I invited William Hamilton (Assistive Technology) and Kerry Taillefer (Testing Centre & Learning Support) to help address some of the emergent questions that might bubble up in the talking circle. Recently, William, Kerry, Joan and I were charged with helping the faculty learn about the 1.5 UDL requirement and it has been a bit of a challenge. The decision has also affected our roles and department staffing! William unfortunately could not attend the Friday talking circle. He did send along a few messages though:
Kerry provided some excellent points to consider as we work our way through this change. She has been looking into research on this matter and found out that:
Kerry suggested that we should work with our students to prepare them for the exams, make them aware of test-taking strategies and help them avoid stressing before or over-thinking during assessments. She also suggested talking with students about on how cheating in the short term can have long-term repercussions when they are asked to apply information with an incomplete understanding. I was extremely glad to have Kerry there. A number of instructors in the circle affirmed her observations about students using or abusing extra time. At the same time, we could all agree that: “there are no magic wands” and one-size does not fit all. However, as instructional faculty and faculty support, we needed to “find ways forward”. Then we shifted gears to discuss some solutions. Finding Ways Forward
At this point Mabyn was able to share some of the ways that the faculty at Lakeland have responded to the need reality of online assessment. Some instructors at Lakeland had success with open-book and oral exams. It was a big shift away from the typical battery of 200 multiple-choice questions. The success of open book exams had much to do with the structure of them and the challenge found in them. Students were given a set block of time (a day?) to explore questions that spoke to the big questions of the course and their program. In oral exams, with some carefully crafted questions and strategic probing, instructors could find out very quickly who knew and could make connections, and who could not. Some of the instructors Mabyn worked with also had success in constructing case-study types of questions that required longer answer responses. In all three cases, making sure that the marking criteria (rubrics) are clear was essential. One instructor shared that, after conducting these new assessments, he had a much better picture of “who got it” than he did from the old multiple-choice exams. He could see how students were intersecting ideas and applying them. Some of the other suggestions shared during the session for addressing the challenge of online assessment (and even the UDL 1.5 requirement) included:
Last week I shared some handy websites from Taylor Institute (U of C). Here are a couple of others that people shared on Friday. Websites: https://learninginnovation.ca/student-cc/ (Lethbridge College Centre for Teaching and Learning) https://sites.tufts.edu/teaching/2020/05/08/integrating-inclusive-and-sustainable-assessments-in-your-online-teaching-from-beginning-to-end/ (Teaching and Learning at Tufts University) https://teaching.uwo.ca/elearning/student_assessment/alt-assessment-ideas.html (Centre for Teaching and Learning – Western Ontario) Have a wonderful week. I’m going out to play with my puppies! Jeff Looking at Assessment!
This week I wanted to talk about assessment. It is the most significant factor in encouraging and sustaining student success. Formative assessment (Assessment For and As Learning) helps to provide students and instructors with an opportunity to check on understanding, clarify learning targets, re-calibrate, and build for success. As many leading educators have pointed out, this kind of assessment needs to happen minute by minute, throughout lessons and activities; there needs to be a constant going back and forth of checking and rechecking. Summative Assessment (Assessment Of Learning) usually comes at the end of a cycle, and it helps students and instructors to measure success against clearly defined parameters. Summative assessment requires educators to establish routines and procedures that promote fairness, reliability, and validity. However, while few educators would debate the impact and role of quality assessment practices, there has been hot debate about what these practices are. A number of years ago I was involved in a district-wide assessment reform that called on K-12 teachers to re-examine their assessment practices. Questions were asked like:
That year we tackled some very prickly pears. There were parents who were very upset that their child’s progress was no longer indicated by percentile (“How can I compare my child to the rest of the class? What does “proficient” really mean? Is it a 63% or 75%? There’s a big difference!”), and teachers who bristled at the idea of no zeros and second chances (“How can I ensure that they learn their lesson about tardiness?”). Not every issue had an easy answer. Nonetheless, as a district, we learned a great deal about assessment. It was a great leap forward for all of us. The present circumstances, the transition to whole scale online learning prompted by the Covid-19 pandemic, has brought on a similar kind of college-wide assessment reform for NorQuest. This time the questions are a little different though:
Since NorQuest moved to exclusively online instruction (with some obvious exceptions), I could feel the tension mounting around just these questions (and many more). Programs that relied on 2-4 big, multiple-choice assessments to provide evidence of understanding and achievement were suddenly faced with the challenges of conducting these exams online. Other programs that required physical demonstration of situational competency, practical skills, or professional judgement were also left struggling. One recent change to how we assess at NorQuest College has been with respect to exam accommodations and to the disability intake process:
These changes are more fully explained in a couple of emails that were sent out earlier this month. If you missed the emails, have a search through your inbox, check with your chair or associate chair, or contact William Hamilton. In the next few weeks William Hamilton, Alan Jeans, Kerry Taillefer, Joan Wall and I will be meeting with departmental groups to help clarify what this means for your students and for you. With all of these assessment pressures and changes in mind, Norma Schneider met with the Deans, Chairs, and Associates during the Week of Welcome to discuss what we, as NorQuesters, see as the most important components and principles in effective online assessment. Some of the frequently mentioned terms included: authenticity, alignment, relevance, choice, reliability, rehearsal, “aha” moments, variation, growth opportunity, transparency, and ownership. Norma urged us to look past assessments that rely solely on rote and recall, and think of ways to assess that will challenge students to analyze, apply and construct based upon scenarios or problems. It reminded me of what one of my Education professors used to say to us, when I was taking my undergraduate degree many moons ago (almost 40 years!). Dr. Glenn Martin (U of A) asked us if we planned to use assessment as an affirmation to find out what students know and can do, or as a “gotcha” to find out what they missed or failed to master. All too often, in an effort to “separate the wheat from the chaff”, we build assessments that test for perfection and not for proficiency. Such a practice makes it easier to rank and sort the students, based on where they fit on the curve (normative-based). These kinds of assessments also favor those students who are very good at memorizing or at taking multiple-choice tests. Dr. Martin advocated for a range of assessment types, that might include multiple choice but also include short and long answer tests, portfolios, projects, forums, problem-based learning, role-plays and many other ways of finding out what your students know and can do. Till next week! Jeff |
AuthorJeff Kuntz Ph.D. ImagesExcept where indicated, images used in the blog posts are personal photos, images from NorQuest College or images from Pixabay. Pixabay is a vibrant community of creatives, sharing copyright free images, videos and music. https://pixabay.com/ Archives
March 2024
Categories
All
|