Musings on Assessment The question of whether instructors should reward participation/conscientiousness is one that has provoked a lot of debate, especially between educational experts (often academics) and those charged with teaching (hard working practitioners!). When I have talked to NorQuest instructors about this issue, they have shared several reasons for using “participation marks”:
Instead, these educators suggest making a concerted effort to focus on competence by:
0 Comments
Let's get started...Several years ago, when I was teaching face to face, I would very often start class with a “Do Now”. This meant that, after attending to classroom set-up and after connecting with the early bird students who wanted to chat, I’d share a PowerPoint slide that would ask students to consider a quote, answer a question or solve a problem. The expectation was that they were to roll up their sleeves and get to work doing a bit of heavy thinking—thinking that would springboard us into the overall theme for the day’s lesson. I’ll give you an example. Here’s a Do Now from a lesson just after the students had finished reading/viewing A Streetcar Named Desire: Do Now: censorship and values! When this play was originally written and performed it was quite controversial. When the play was adapted for film, there were some significant changes (Consider the article posted in Moodle.). How do you feel about the way in which Streetcar deals with social issues like promiscuity, spousal abuse, homosexuality, and mental health? Would the play be as controversial (and relevant) if it opened on Broadway today? Take five to ten minutes to quietly write your response in your journal and then be prepared to share your observations with an elbow partner. The students dove into the topic. They were motivated; a number had confessed that the play had made them uneasy and even angry. Obviously, it was a huge topic—not one that could be properly addressed in five minutes. In fact, as I remember, I ended up letting them chew on these questions for a half an hour or longer (writing and discussing in pairs), before I brought them back and started our collective discussion. And it was exactly these individual student feelings that I was looking to leverage as we began to tackle some of the major themes in Tennessee Williams's play. Starting the lesson off with these big questions, and asking students to reflect and discuss—before we explored the questions together—provided students with an opportunity to voice their perspective, become more invested, and take an active role in the lessons that followed. Obviously not every lesson or topic will evoke the same kind of emotional response as the one above, but it is important to try and hook our students-to create an “anticipatory set”—and Do Nows certainly help with this. Over the years I have seen instructors ask students to:
In each case, the instructors used the Do Now for a variety of reasons:
Yes but, online?Some instructors have told me that they can no longer teach in this way. They’ve said that the kinds of activities they previously did (like Do Nows), just don’t transfer into an online environment. They feel like all they can do now is talk at the students and hope that they are listening. But I’m not sure I would agree. It’s just a matter of leveraging some of the tools afforded to us and learning to frame and scaffold meaningful online reflection and discussion. For instance, if we look at the example above from A Streetcar Named Desire or any of the other ideas I shared, I could see the same kinds of prompt and activity being used:
Now I know that some instructors would say that they can’t afford to give up this much class time for such an activity. Perhaps. But our job is not to simply regurgitate the curricula and cover all the concepts. Our job is to: 1) engage the students, 2) compel them to get invested in the topics, 3) see themselves as future professionals in their field, and 4) create educational opportunities that allow them to react, respond and employ critical judgement. Besides, not every Do Now has to be as involving as the one I shared on Streetcar. Most Do Nows can be done in 10 to 15 minutes – pricking interest, soliciting feedback and sharing a few perspectives. Do Nows might also provide a bit of spark for those who are looking to break up 2-3 hour online synchronous classes and make these classes interactive and “value added” rather than passive and draining (a synchronous lecture that might be watched later at 1.5 times speed). Measuring student engagement? Quite a number of years ago, I was asked to champion a classroom walkthroughs program in a large school district. In this program school administrators would conduct “walkthroughs” as a way to assess how their school was implementing improvement goals. These administrators would get real data and a “feel” for how things were going in a quick and simple way. Principals and associate principals were to quickly pop into classrooms while teaching was taking place and, using a checklist with certain look-fors, make a five-minute assessment based on their observations of the lesson:
These and a number of other questions were all organized on a one-pager with checkboxes that administrators could quickly fill out. At the end of each day (or week) they would be expected to enter the data into an online database so they could combine all of their quick snapshots (crunch the numbers) and see “the big picture” for their school. A walkthrough mosaic, if you will. While I sympathized with the intent of the walkthroughs and was happy to get principals out of their offices and into the classrooms, I didn’t especially like the tool. It’s hard to reduce something as complex as a teaching environment into a series of “Yes I saw it!”, or “No I didn’t!” checklists. One question especially rankled me: Determine levels of class engagement:
The question was based upon a model for describing classroom engagement from the Phillip Schlechty (2002) which has five different levels of engagement: 1) Authentic Engagement, 2) Ritual Compliance, 3) Passive Compliance, 4) Retreatism, and 5) Rebellion. (for short summary of these levels check this document from Stockton University Center for Learning Design: levels) This question always bothered me. While the model professed to illustrate levels of “engagement”, there really was only one level where student were really engaged – the other levels just showed how disengaged they might be. The question elicited many questions from me. How do we know exactly what engagement looks like? Is it about eye-contact, question-asking, discussion, and busyness? Or could quiet students, who might be judged as day-dreamers, perhaps be more engaged than students who know how to “play the game”? Is a classroom of students quietly doodling as a teacher reads a story to them less engaged than those down the hall conducting a science experiment? Which group is merely compliant? I began to think about the types of engagement and considered a model of my own: 1) causing chaos, 2) confused or disconnected, 3) simple compliance, 4) making connections (personal or real world), 5) consolidating learning (building a frame for understanding), 6) challenging assumptions, and 7) creating new understandings or interpretations. Of course, most of this kind of engagement happens where we can’t observe it – inside the mind of the student. So, it wouldn’t work very well as a tool for quick observation (although it might work when reviewing student responses, written or recorded). After working with our district administrators for six months with what I thought was a flawed tool, I managed to convince the website builders to remake our walkthrough tool to expand the choices for observable engagement, and to better reflect our district improvement goals (for assessment, inquiry, and critical literacy). Then, only a year into the process, the district decided to drop the contract with the educational website designer, and walkthroughs - at least in our district - became a thing of the past. My fuss and bother about defining “student engagement” was put on hold. Student engagement as it is understood on college campuses After making the transition over ten years ago from k-12 education to post-secondary contexts, I quickly became re-acquainted with the term “student engagement”. I learned that the term “student engagement” is one that has taken on a lot of territory in and around the college scene in the past decade or so. As a result, the term itself has become rather “fuzzy”. On college and university campuses, student engagement is referred to when:
In our present post-secondary context, “student engagement” is something that is often used as a justification for many decisions made by learning institutions, but it is rarely defined and more often only fleetingly understood. More work is necessary in this field to clarify just what the students are to be engaged with and how, to help clear away the misconceptions and make decisions that most benefit our institutions, our students and our instructors. However, for the purposes of this series of blog entries, we will be focusing on only the first of those four notions of student engagement; engagement as it pertains to the kind of connection students make when involved in learning activities. Our immediate context at NorQuest At NorQuest, we currently have a faculty learning team considering how we might better engage our students in online activities, but even in this group of highly interested educators, we have had trouble pinning down what exactly we mean by student engagement. Just this past week we wrestled with a few critical questions:
Several of the instructors on the faculty learning team expressed frustration with their current online – Covid necessitated – teaching context. They lamented the loss of engagement from face-to-face contexts and the connection they always felt with their students. Now, because many students choose not to attend regularly (but might watch the recordings later), keep their cameras off, and only ask questions or use the chat box feature infrequently, these instructors feel like there is little to no engagement. And this feeling raised several more questions:
These six questions are going to become very important as we look to the future in teaching and learning at NorQuest. In next couple of blog entries, we’ll try to work though these questions with an eye to the college learning context. References: Schlechty, P.C. (2002) Working on the work: An action plan for teachers, principals, and superintendents. San Francisco, CA: Jossey-Bass Education Series. Ashwin P., McVitty D. (2015) The Meanings of Student Engagement: Implications for Policies and Practices. In: Curaj A., Matei L., Pricopie R., Salmi J., Scott P. (eds) The European Higher Education Area. Springer, Cham. https://doi.org/10.1007/978-3-319-20877-0_23 Vocabulary is Important! At NorQuest College, we have a number of courses that require a great deal of vocabulary acquisition and understanding. Knowing certain terms and their nuances is especially important for new Canadians trying to adjust to cultural norms, for budding scientists who need to clarify and identify exactly what they see, and for health care providers who must have a common lexicon for anatomy and physiology. Many of our students have coped with difficult terms by memorizing exact textbook definitions anticipating that exact definitions will be required on unit tests and quizzes. However, while this strategy may help the students to achieve slightly better marks in the short term, it actually harms the student because their knowledge of many key concepts is superficial at best. Students need to be able to do more than just define a term; they need to know why the term is significant, how it connects to other concepts and what makes it different from other terms. When assessments ask students to use terms in the right contexts, to make comparisons and analysis, and to unpack understandings, students with only a rote knowledge of key terms struggle. One of the many vocabulary strategies I like to use, to help students better understand important terminology, is a graphic organizer known as the Frayer Model. What is a Frayer Model? The Frayer Model (Frayer, Frederick, and Klausmeier, 1969) is a vocabulary strategy that actually helps learners deepen their understanding of key concepts by asking them to consider a number of facets to the terms used. This “deep understanding” comes about when students are pushed to consider a word’s essential and non-essential attributes and to refine their understanding by choosing examples and non-examples of the concept. In order to understand completely what a concept is - one must also know what it is not. How to use it: The Frayer model can be used before teaching a unit as a diagnostic tool to assess prior knowledge, during the unit to consolidate understanding as a form of note taking and after the unit as an assessment of learning.
When using such a strategy with their students, teachers need to make sure that they start with only a few key terms or concepts. Instructors also have to resist filling out the organizers for the students and simply giving the completed charts to the students as study guides. This defeat the individualized experience and accountability of working through the terms. Over the years, practicing teachers have come up with many different variations on the Frayer Model. However, the original Frayer Model focused on helping students to narrow down the meaning for each term by seeing what it is and what it is not. I’ve included a couple of modified Frayer Models I have use to help clarify the difference between more traditional and more constructivist models for teaching. Focus on Assessment – Again! This past Friday we had a Faculty Talking Circle on Online Assessment. It was a rich discussion and a very honest conversation! During the weekend, I did a bit more thinking about what we discussed. Here are a few thoughts. Some of you may know that our family recently added two new members to it! Edgar and Daisy (perky ears!), two precocious little border collies. Right now they are about 12 weeks old and are still learning many things - so are we. Just after we took the siblings home we arranged a visit with the veterinarian for a check-up. One of the things we mentioned was that Edgar had the habit of bolting his food while Daisy would take a mouthful, walk a bit of a distance away and carefully chew what she had before returning to get another mouthful. By that time, Edgar may have eaten all of his portion and most of hers. It didn’t seem fair or equal! We didn’t want Edgar to end up being a fat little dog, especially at Daisy’s expense, so we thought about limiting his food intake or feeding Daisy separately. The vet told us to take a deep breath and relax. Just keep adding food to the dish; puppies self-regulate and limiting Edgar’s food intake might be harmful. He is likely moving through a growth spurt and, judging from his body type, he will likely be a bigger dog. Trying to be equal wouldn’t be fair; especially to Edgar. Yet, as educators we consistently fall into the trap of “fair must be equal”. We don’t want to be seen as favoring particular students and we sometimes hesitate about providing extra resources, support, time or opportunity, because “it wouldn’t be fair to the rest of the class”. Why should a particular student get an extension on their project? They had better have some dire circumstances! How will they understand about deadlines and the “real world”? But fair isn’t equal. Fair is about giving every student the chance to thrive and succeed, and some students will need different kinds of support than others. Another example. Although I don’t often think of it as such, I have a learning accommodation. I wear eyeglasses. Without my beloved blue spectacles, I would be hard pressed to complete most tasks. If I had been taught in an environment where decisions were based solely on keeping all things equal, I should not have been permitted to wear them. Why should I be able to wear glasses while others did not? The fact that the others did not need glasses is immaterial. It’s not fair! It’s not equal! But, as educators, we think little of having students with glasses or hearing aids. They don’t get accommodations, they have aids that do little to interfere with our planning and practice. And, we do have many students who receive accommodations; and we do our best to support them. We have students who need extra time to process, students who need a scribe (graphomotor issues), students who need readers, students who need interpretation, and students who need different colour papers. We are trying to be fair by removing some of the barriers that might interfere with their learning. Their extra time, prepared notes, relaxed deadline, reader, or scribe is my glasses. Friday’s Discussion That brings me back to the Friday Talking Circle on Assessment. Originally I had intended the session to be a bit of an idea exchange. I invited a colleague from Lakeland College, Mabyn Grinde, to join us for the session, as she is working through many of the same challenges with the faculty there. Over twenty instructors were able to attend, as well as Joan Wall and Kerry Taillefer, who would help field questions! The plan was simple. Share some of the assessment principles seek input from instructors. Ask instructors to share some of their current challenges related to online assessment. Elicit a few suggestions and successes from each other that might improve our practices. Oh, and yes, maybe also talk about the new UDL 1.5 time requirement for conducting assessments. However, the 1.5 UDL (Universal Design for Learning) requirement quickly became the major focus of our discussion. The discussion became passionate and, at times, pointed. It was an honest conversation! We were concerned about what is fair, what is equal and whether students might get an unfair advantage. From my understanding the move to universal 1.5 timing for assessments was intended to allow students who may not have had learning challenges formally assessed, to be given the extra time. Since the onset of the Covid-19 pandemic, many of our students may have struggled in isolation, with barriers to achievement such as inadequate technology or Wi-Fi access, anxiety or uncertainly related to the pandemic, childcare and other familial or relational obligations, shared use of technology (one computer for a whole family to do their work on), employment responsibilities and other stressors. Students who do not have these stressors would likely not take advantage of the extra time, but those who need the extra accommodation would be able to use it. Many of the faculty in the talking circle were, for the most part, very sympathetic to the concept of time and a half. Our students are under a lot of pressure and this decision seemed to come from a caring place. Never the less, there were a lot of questions and concerns related to this move.
Compounding the issues raised by the faculty is the fact that many of the established, closed-answer assessments (especially those with right/wrong answers) no longer work in an online world where students can screen capture and instant message. Open-ended assessments require more time to ideate, build, design effective scoring guides for, and mark! Getting More Context As special guests, I invited William Hamilton (Assistive Technology) and Kerry Taillefer (Testing Centre & Learning Support) to help address some of the emergent questions that might bubble up in the talking circle. Recently, William, Kerry, Joan and I were charged with helping the faculty learn about the 1.5 UDL requirement and it has been a bit of a challenge. The decision has also affected our roles and department staffing! William unfortunately could not attend the Friday talking circle. He did send along a few messages though:
Kerry provided some excellent points to consider as we work our way through this change. She has been looking into research on this matter and found out that:
Kerry suggested that we should work with our students to prepare them for the exams, make them aware of test-taking strategies and help them avoid stressing before or over-thinking during assessments. She also suggested talking with students about on how cheating in the short term can have long-term repercussions when they are asked to apply information with an incomplete understanding. I was extremely glad to have Kerry there. A number of instructors in the circle affirmed her observations about students using or abusing extra time. At the same time, we could all agree that: “there are no magic wands” and one-size does not fit all. However, as instructional faculty and faculty support, we needed to “find ways forward”. Then we shifted gears to discuss some solutions. Finding Ways Forward
At this point Mabyn was able to share some of the ways that the faculty at Lakeland have responded to the need reality of online assessment. Some instructors at Lakeland had success with open-book and oral exams. It was a big shift away from the typical battery of 200 multiple-choice questions. The success of open book exams had much to do with the structure of them and the challenge found in them. Students were given a set block of time (a day?) to explore questions that spoke to the big questions of the course and their program. In oral exams, with some carefully crafted questions and strategic probing, instructors could find out very quickly who knew and could make connections, and who could not. Some of the instructors Mabyn worked with also had success in constructing case-study types of questions that required longer answer responses. In all three cases, making sure that the marking criteria (rubrics) are clear was essential. One instructor shared that, after conducting these new assessments, he had a much better picture of “who got it” than he did from the old multiple-choice exams. He could see how students were intersecting ideas and applying them. Some of the other suggestions shared during the session for addressing the challenge of online assessment (and even the UDL 1.5 requirement) included:
Last week I shared some handy websites from Taylor Institute (U of C). Here are a couple of others that people shared on Friday. Websites: https://learninginnovation.ca/student-cc/ (Lethbridge College Centre for Teaching and Learning) https://sites.tufts.edu/teaching/2020/05/08/integrating-inclusive-and-sustainable-assessments-in-your-online-teaching-from-beginning-to-end/ (Teaching and Learning at Tufts University) https://teaching.uwo.ca/elearning/student_assessment/alt-assessment-ideas.html (Centre for Teaching and Learning – Western Ontario) Have a wonderful week. I’m going out to play with my puppies! Jeff Feedback
I hope you all had a wonderful and relaxing long weekend. I spent mine doing a lot of yard work. It was great to get outside and away from this computer! The first thing we did was tear down our old garage. We had already made a new larger garage on the same property, so the old beast had to go! Saturday morning at 8, my nephew and his friend showed up with a very large trailer and, with the help of my son and son-in-law, they attacked the old structure. Within two hours, the old garage was gone. Four guys, a couple of power tools, several crowbars and sledge hammers and a structure that had lasted almost sixty years was reduced to a heap of rubble and on its way to the dump. The boys had a bit of motivation; they wanted to go dirt biking. The old garage was not going to interfere with that! The demolition made me think of one of the recent sessions I had given on feedback and on how easy and quick it is to tear something (or someone) down. I know the previous owner had spent quite some time building that old garage, and I know how long it took to build our new garage, but all it takes is a few motivated young men and some sledge hammers. It made me think of the many times that I had used my words like a sledgehammer when I was providing feedback to my students on their work. I could reduce their essays and their confidence to rubble with a few well-placed slams. It’s not something I’m proud of. When providing feedback to your students there’s a few things we should keep in mind:
As Dylan Wiliam and Paul Black proved over 20 years ago, timely and meaningful feedback is critical in raising student achievement. But there is a difference between feedback that encourages and feeds forward and feedback that can only tear down. (Black, P & Wiliam, D, 1998, Inside the Black Box: Raising standards through classroom assessment, School of Education, King's College, London, United Kingdom.) |
AuthorJeff Kuntz Ph.D. ImagesExcept where indicated, images used in the blog posts are personal photos, images from NorQuest College or images from Pixabay. Pixabay is a vibrant community of creatives, sharing copyright free images, videos and music. https://pixabay.com/ Archives
March 2024
Categories
All
|