Assessment Musings In the past few weeks our team has been asked to review assessments (mostly quizzes and tests) with an eye to clarity and fairness. Instructors were worried about question phrasing, multiple choice construction, weightings, time limits among other things. It was wonderful work and we learned much from working with our colleagues across the faculties.
In our interactions with individual instructors, we also discovered that sometimes the assessment is superficial and has only a tenuous link to the course outcomes. Sometimes it is because the instructor has been put into an awkward circumstance; they were handed a course and they are just trying to make a go of it (And might not get that course ever again!). Previous instructors made the text their curriculum and adopted it as “the source of truth”. The result is many “legacy” assessments that merely ask students to recall information. But, “List the four different kinds of organizational structure that the author identifies in the text” is not a great question. Questions like that reward “rightness” rather than comprehension. I’ll give you an example. Years ago, I was visiting a Social Studies class where students had been asked to work with terms associated with the Industrial Revolution. They were given a list of terms on one side of a page and a flowchart on the other, asking them to place the term where it might seem best. I sidled up to one student and asked him how it was going. He said “fine”, as he copied out definitions from the back of the textbook onto the paper he was given. I decided to find out “just how fine it was” going by giving him a quick quiz. I pointed to his paper and asked him to explain what capitalism was. “It is an economic system, based upon supply and demand whereby an entrepreneur controls the means of production.” He shared. “Aha”, said I, “So what is an economic system and what is an entrepreneur?” “Well, it’s got something to do with money. Not sure who this entrepreneur guy is either, but he controls the means of production!” He shrugged. I won’t bother you with the rest of the conversation. Suffice it to say, that student had a very superficial understanding of many of the terms on his sheet. However, if his instructor gave the student a short answer test asking him to define capitalism and a few other terms, he might attain full marks, if he had managed to memorize these definitions. And the instructor might be happy with this; a short answer/recall type of test would be easy to mark, provide full set of class marks and reward those students who spent time in studying (memorizing). But did real learning actually happen? I’m not so sure. That’s why our assessments should align with course GLOs and SLOs (general and specific learning outcomes). These outcomes have been crafted with an eye to essential understandings and skill development that we need to focus on in our teaching and assessment. Readings and resources, video clips and teacher lectures are used to reinforce these outcomes and not to replace them. So, our challenge is in designing assessments that truly assess the knowledge, skills, and attitudes that our students need, and that can be hard work. Related article: A Table of Contents is not a Curriculum
0 Comments
In the first blog entry on this topic, I shared how the term “student engagement” is often used in very different contexts in and around post-secondary learning (active learning, curricula development, quality assurance, and institutional governance – to name a few). And even when we narrow down our focus to student learning and the connections made during this process, it seems as if it is easier to identify when engagement doesn’t happen, than to unpack what engagement actually is. This challenge gave rise to a set of three questions:
1. What is our definition of “student engagement”?In our NorQuest Online Engagement Learning Team (a small faculty community of practice), we are continuing to shape and refine our understanding of student engagement. In a chapter entitled: “The Meanings of Student Engagement: Implications for Policies and Practices”, Ashwin and McVitty share one definition that has promise: Within the community of academic practitioners, engagement by students is most commonly interpreted in relation to the psychology of individual learning: the degree at which students engage with their studies in terms of motivation, the depth of their intellectual perception or simply studiousness. Engaged students are viewed as taking ownership for their own learning, working together with staff on ensuring academic success and accepting the role of engaged and willing apprentice to an academic master (Velden 2013, p. 78). This definition makes me think of terms like: empowerment, inquiry, self-efficacy, perseverance, and personal accountability. As such it deals with the notion of engagement as something that may develop and grow as students partake in learning opportunities, and much of it will depend upon the students themselves. However, our learning team was looking more closely at engagement in direct response to specific learning activities, as a response to the lesson or course design of the instructors and the interactions and thinking that happens while such learning occurs. Another common definition of student engagement is “being in the zone” - to become so preoccupied with a task or challenge that time and other constraints fade to the background. I like this definition for its simplicity. This definition focuses on engagement from moment to moment, not over a semester or a year. I think it is every instructor’s hope that we can get our students so wrapped up in their activities that time seems to fly by. And it seems to me, that time flies by, not when students are simply trying to listen and remember, but when they are acting upon the content, skills and professional judgments that are the focus of their asynchronous or synchronous learning activities and assignments. However, if we accept this student engagement definition of “being in the zone”, we also have to ask, just how do we get our students to enter this zone? Experienced educators will tell us that it requires an artful mix of intriguing questions, humor, clear demonstrations, choice, opportunities to collaborate, challenge, feedback, personal connections, and a host of other classroom conditions and influences. Such artful teaching challenges our students to: ask questions, make inferences, compare and contrast, synthesize, role play, problem solve, build new frames of understanding for themselves and reflect (and many more - higher order - thinking skills). 2. How do we cultivate “in the zone” engagement?In an article published in 2011, Jim Parsons and Leah Taylor suggest that student engagement is directly related to how educators build learning experiences with sic different six different factors in mind:
While the work of Parsons and Taylor is perhaps a bit dated now, especially in light of how technology has transformed learning environments in the past couple of years, I still like this list. It points us away from the superficial (but often energy providing) things that we consider indicators of engagement; it is more than just eye-contact, body posture, question-asking and quick collaboration (in-class) or cameras, on clicks of the mouse and chat messages (online). Engagement is related to how a student connects to the learning outcomes, how they personalize learning, how they apply their gifts and abilities, and how we, as instructional designers, set up a context that allows them to do all of this. 3. Is there a difference between engagement in synchronous online environments compared to asynchronous work?In a word, no! And then again, yes! Engagement, as we’re coming to understand in our little learning team, is really more about thinking than it is about interactivity. Good questions, challenging tasks, and interesting readings and video recordings all contribute to student engagement – it matters not if these are situated through asynchronous or synchronous means. However, as Viola and I very quickly discovered when we started our research study on the rapid move to online instruction, asynchronous online instruction (which has a significant research base) is quite different from what most colleges, institutes and universities are now offering in the wake of the Covid 19 pandemic. Tried and true asynchronous online instruction is dependent upon well-laid out course maps, where the activities, assignments and assessments are established well before the students take the course. Such courses attract students who welcome the flexibility offered and these students feel that they have the drive and self-discipline to engage in the learning and complete the work. While this learning can be enhanced by creating a community of inquiry (see CoI Framework AthabascaU), the success of individual students depends more upon the instructor’s selection and organization of the learning tasks than upon the instructor’s connection with the learners. What most of us are offering now is not that kind of online learning. Leading educators have said that our current Covid reality has pushed us into a “remote learning” model, where instructors are trying – to various levels of success - to emulate face-to-face learning in an online environment. Some have even gone so far as to call this “emergency remote teaching”, thereby conveying the notion that our efforts are reactive and responsive, rather than carefully mapped or based upon proven practices. (See excellent article at Educause: emergency-remote-teaching-and-online-learning ) We are learning as we go. And one year in, we are getting better at it. But many instructors are still struggling with striking the right balance and settling on practices that create interest, promote community/communication and provide enough support without frustrating or burning out the student or the teacher. So, using the frames and models that others have developed to describe engagement in online instruction to gauge and unpack what is happening now, has its limitations. We can apply some of the findings and principles from this scholarship in online learning to our new context, but we can only go so far. Most of our students at NorQuest, did not sign up for asynchronous online instruction and they certainly didn’t sign up emergency remote instruction. And while some students have thrived in this new, more flexible way of learning (they could more easily accommodate work responsibilities and could eliminate the commute), many more students lament the loss of the feeling that they are part of a classroom community with its shared pacing of learning activities, deadlines and camaraderie. And many of our instructors are missing this shared experience just as much as the students. In next week’s blog, part 3 of engagement, I hope to more closely explore the concepts of student engagement, interactivity, and student and instructor experience in our current Covid (remote learning) context and make some practical suggestions. References: Anderson, T. (2011). The theory and practice of online learning. (2nd Edition). Edmonton, AB: AU Press. Ashwin P., McVitty D. (2015) The Meanings of Student Engagement: Implications for Policies and Practices. In: Curaj A., Matei L., Pricopie R., Salmi J., Scott P. (eds) The European Higher Education Area. Springer, Cham. https://doi.org/10.1007/978-3-319-20877-0_23 Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education model. The Internet and Higher Education, 2(2-3), 87-105. Taylor, L. & Parsons, J. (2011). Improving Student Engagement. Current Issues in Education, 14(1). Well it has been 7 weeks since my last Monday Morning Musing. I went on vacation at the beginning of July, and when I came back to work, things were quiet – except for the instructors in language instruction and LINC. They just kept motoring along! What I learned on my summer vacation… On my “staycation” I managed to do a lot of work on my patio. Cynthia and I spent most of our quality time doing intense, manual labor. I prefer to think of it as my “2020 Covid-19 fitness workout”. Now, nearing the end of our project, we are thrilled with how it all looks. However, there were some challenging moments. I had a feeling that the project would not be an easy one. However, as most do-it-yourselfers soon find out, I just had no clear idea of just how much planning, dumb labor, fine motor skill, perseverance, dumb labor, compromise, new learning, and dumb labor it would require. And don’t get me started on the number of times I ended up going to the hardware store or the rental shop to buy more materials or rent a tool for the day! I think the most stressful part was in operating the gas powered concrete saw to make cuts in the forty-pound pavers. I did not want to make any mistakes (I did make a few though), and I did not want to lose any fingers (I still have all ten)! Now, as I turn my focus back to supporting faculty with the world of online teaching, I can see a few parallels between planning and building our patio and planning and facilitating an online course.
A Table of Contents is not a Curriculum In 1983, I was the new Social Studies, English and French teacher in a small Christian school in Chatham Ontario, which seemed like a universe away from my experiences in Edmonton at the big public schools and the big public university. It was August 1, my first year of teaching, and my principal was asking me to submit “year plans” for every course I was supposed to teach in the coming year by the end of August. Yikes? Where to start? How do you know what you don’t know - if you don’t know what you don’t know? Then he left for vacation. I was left to figure out what a year plan was and how I would create one for the 12 different courses I would be teaching that year. So I spent the next couple of weeks haunting that little rural school. Every other teacher was gone; it was just me, the humidity of Southern Ontario and the thousands of flies that came into the school from the adjoining corn fields and buzzed around my sweating pate. I dolefully went through the filing cabinets in my classroom, I found some archaic readers and texts in the book room and eventually I cobbled together a year plan for each of my courses. It wasn’t easy for a 21 year old kid on his first teaching assignment. But, I made them: each year plan carefully laid out the units, objectives and time allotments for the ten coming months, often based upon units or chapters I found in some of the texts that looked promising. Then he came back. After I submitted my plans and the principal dutifully went through them all making copious comments with his red pen, he handed the plans back to me and asked why I hadn’t looked at the year plans from the teachers before me, or why I hadn’t referenced the provincial curricula for the various courses I was about to teach. Apparently all those documents were right there in his office, in the filing cabinet next to his desk. It would have saved me a lot of time and re-writing. When I shared that I thought his office was definitely “out of bounds” and that he never once said that I might find the materials in there, he just took another drag on his pipe (you could smoke in the schools at that time), shrugged his shoulders and said: “Oh well then - that explains it!” And then I started rewriting my year plans. I learned a couple of valuable lessons that summer. One lesson was that you need to be a self-advocate and ask as many questions as you can, especially when you’ve been given the task of roughing out a plan for the year. The second lesson was a little more complex; I learned about the complex relationship between curricula, year plans (syllabi), resources and textbooks, and the freedom to teach what I thought might be important. So why the story, Jeff? At this moment, all around the college, NorQuest instructors are learning the very same lessons. I know that a number of departments are going through a major curricular review and that certain instructors have been given the responsibility of making a “curriculum map” for particular courses. It’s not an easy task. It may be tempting to simply find a suitable text, copy down a few chapter headings and extrapolate some key learning outcomes and figure all is OK, but the table of contents in a textbook is not a curriculum map. Now this is not a small topic, and certainly not something that can be sufficiently addressed through a Monday Morning Musings, but here are a few things to keep in mind when looking at the big picture of curriculum planning:
So where does academic freedom fit it? Are we teaching robots? No, we have been given a great deal of liberty, provided we meet the learning outcomes, program and government requirements, and accreditation requirements. (See the NQ Academic Freedom Policy) You also need to consider the program and college established curriculum, assessments, processes, and practices Each department may have specific requirements and concerns that can influence or impact your plans. (See again, the NQ Academic Freedom Policy). Think of curriculum maps like a road map; you know that you need to visit Red Deer, Calgary, Cardston, Brooks, Drumheller, Jasper and Banff. There is nothing that says how you get there (bus, plane, moped, skateboard, Harley, or minivan), or when (zip out and come back, plan a round trip, cluster certain municipalities according to festivals, plan according to COVID release times?), or even what else you may look at along the way. However, you do need to visit these specific places within a certain amount of time (by the end of term) and there are things you need to see/explore/do/appreciate in each of the places. In the end though, your students will eventually be assessed on their knowledge, navigational skills and cultural appreciation of Alberta, and hopefully your plans (route), activities and assessments will have helped to prepare them for this. If you want to discuss this further or if you have any specific questions, don’t hesitate to contact one of curriculum consultants or faculty developers at the college. Most of us love talking about this stuff (yes, we are nerds!), and see it as important, challenging and kind of fun. Planning with the End In Mind! This past week I gave a session on “Powerful Planning”. It was based upon the principles of backwards design and followed a model established by Grant Wiggins and Jay McTighe in their seminal work: Understanding by Design. Wiggins and McTighe suggest that, if we want to take our student on a learning journey, we really need to think about our destination and then, plan our trip backwards! Furthermore, we need to decide upon the most important sights or stop-overs. In every course of study there are big ideas or “enduring understandings” that students need to grasp in order to become fully competent in that areas. These big ideas could be important conceptualizations, skills or attitudes. Often such ideas can be found in the GLOs found in our curriculum guides or curriculum maps for the courses we teach – but not always. Sometimes we have to dig into professional standards, consider pre-requisites for follow-up courses, or reflect on our own professional experience to determine what is “enduring” or “essential”. Unfortunately many of us are just too busy to think about the “big ideas” in our coursework until we get quite far along into the term. Often the course is assigned shortly before we start teaching it, and we begin a frenzied cycle of familiarizing ourselves with the content, preparing lessons (often content laden PowerPoint slides) and delivering lessons, in an effort to stay one step ahead of our students. Engagement, support and assessment can become after-thoughts. It is an unfortunate reality of post-secondary teaching. However, Wiggins and McTighe would have us step back to see the big picture and, at the outset of planning a course or a unit, follow three important steps:
An integral part of “Understanding by Design” is the generation of essential questions. Big Ideas can often be explored more fully when we take our students on an inquiry process, interrogating the important ideas and their underpinnings together with our students, rather than just summarizing a few key ideas for them. So getting a handle on the content of our courses is simply not enough. If we truly want our students to connect with the content and make sense of it, we need to begin with the end in mind, and think about not just the “what” but the “how” and “why”. And if we identify the big ideas, enduring understandings and essential questions, we will be able to provide a clear framework with touchpoints that students can come back to again and again. (If you are interested in learning more about the backwards learning process, don’t hesitate to contact the PD team or the Curriculum team – we would be more than happy to share our resources and insights with you.) Wiggins, G. P., & McTighe, J. (2005). Understanding by design (Expanded 2nd). Association for Supervision and Curriculum Development. |
AuthorJeff Kuntz Ph.D. ImagesExcept where indicated, images used in the blog posts are personal photos, images from NorQuest College or images from Pixabay. Pixabay is a vibrant community of creatives, sharing copyright free images, videos and music. https://pixabay.com/ Archives
March 2024
Categories
All
|