Category Archives: Formative Assessment

Developing a pedagogy for online delivery

After the initial excitement about so-called “online learning” and “blended learning”, educators are now coming to the realization that these terms are essentially meaningless. There is just learning, and the delivery medium (online, blended or face-to-face) is pedagogically neutral. This is an important point because it puts the emphasis back on the pedagogy and away from the technology.

In saying that technology is pedagogically neutral it does not, however,  follow that technology is unimportant. Technology increases the extraneous cognitive load for teachers and learners, at least in the short term, and this needs to be planned for and managed. Secondly, technology favours some pedagogies by making them more efficient  and effective. This is the aspect I would like to explore further now.

So where are we currently at with online and blended delivery? Many (or perhaps most?) purely online courses and MOOCs are still based on a didactic pedagogy. These courses do provide administrative and delivery efficiencies. The use of multimedia and VR technology can also make delivery more engaging and effective. On the other hand, however, the lack of personal contact with a teacher probably reduces the depth of learning. It is also very difficult to establish an experience of “belonging” for the students, resulting in reduced intrinsic motivation and a high dropout rate. (See my previous posts on Self Determination Theory.) Blended delivery is an improvement, primarily because it is a technological augmentation of the classroom experience, and therefore retains some of the advantages of face-to-face and online delivery methods.

The purpose of this discussion, however, is to focus on pedagogy rather than delivery. Using a course I developed for my students as an example I would like to explore new pedagogies, which take advantage of new technology by moving away from the didactic approach.

I teach a design class and I wanted to develop an online course which  would support the design projects that my students were working on. I called the course the “Design Canvas” and the aim was to provide scaffolding to guide students through the design process and complete their projects. The course provided students with autonomy, collective responsibility and support, and support for their competence. In this sense the Design Canvas is an application of Self Determination Theory.

The Design Canvas applies a pedagogy that is not easily applied in a traditional classroom. The pedagogy is based on a concept called ‘participatory pedagogy’, while the assessment is based on the principles of ‘assessment for learning’.

The Design Canvas was developed in a VLE package called “Canvas  LMS” by Instructure. This was chosen as it has a powerful peer assessment feature, including rubrics. These are an import aspect of the pedagogy the Design Canvas was designed around.

In the Design Canvas students are provided with very little information. Instead they are guided through the stages in the design process:

  • Design Brief
  • Research
  • Ideation
  • Implementation
  • Evaluation

This is applied to a design project of their own choice. The course explains these stages in turn and provides scaffolding so they can move forward through their project to completion.

There are also exemplar completed projects and a recreational area where they can play online games design to stimulate their thinking and creativity. (e.g. Electric Box, Machinarium )

Participatory Pedagogy

The Design Canvas is based on Participatory Pedagogy (PP).

PP uses a collaborative and reflective process to formalize knowledge. It aims to develop transformative learning (changes in how we know) rather than informative knowledge (changes in what we know).
Successful PP contains three main elements:
1. providing choice and flexibility in learning activities and assignment work.
2. navigating the balance between challenge and risk.
3. creating contexts for critical reflection.

Students will have had little experience in learning like this. Most of their experience will be with didactic learning. (Given information, learn information, reproduce information.)

Students familiar with the PP approach report that a teacher using the participatory pedagogy approach needs to exhibit a range of qualities (quote from reference):

  • Be open to the interaction thereby enabling the students to have a voice.
  • Be willing to commit to the style and be an active participant yourself.
  • Have courage and be willing to go outside of your teaching safety zone into new unexplored domains.
  • Be frank, up front with the format, and provide encouragement to the students so they feel supported during this new learning format.
  • Plan learning based on student interests and choice, and do so by collaborating with and guiding learners as opposed to informing them.
  • Create an atmosphere of learning where expectation of learner action is high and modify the activities/plans to meet their needs.
  • Be open, willing, and supportive to students if you are trying creative adventures because it can be a risky thing for adult learners to engage in.
  • Be very comfortable with awkward pauses and strange looks, and be willing to walk students through their discomfort.
  • Be humble but have a good depth of experience, both human and professional.

To be successful, therefore, participatory pedagogy needs to be well supported by a mentor-style teacher who demonstrates these qualities.

Choice and Flexibility

In the Design Canvas students will choose the design task they wish to embrace. This provides for choice and flexibility. As a result:

  • They will have ownership of the task
  • The task is in a context they are interested in
  • Within the design process they are able to work at their own pace
  • Can go into as much depth as they wish.
  • Assessment/success is based on reflection and collaboration rather than comparison with external standard

Other courses using PP will need to find their own way to provide for choice and flexibility, but allowing (or forcing) students to set their own goals or topics will be one way of doing that.

Challenge and Risk

In the Design Canvas students will set their design brief/task before they embark on the solution. This has inherent challenge and risk.

  • Can the task be completed?
  • Do they have the skills or ability to effect a solution?
  • Resilience is needed to overcome problems and stalled development.

Students share their work for comment by others and comment on the work of others.

  • This is a risk and a challenge for students.
  • What do they say, how do they evaluate the work of others?
  • This evaluation process is closer to that which they will face in the real world. Students need to feel comfortable with putting their work and opinions out there for others to see and comment on.
  • Risk involved in not knowing what basis others will evaluate the work.

The higher the challenge the higher the risk. Structure is provided by the design process  for managing the development of the solution to the design task. It provides an intellectual safety net to allow for a higher degree of risk than normally attempted.

A mentoring teacher also provides support managing challenge and risk.

Critical Reflection

Students share their work for comment by others and comment on the work of others. This is a powerful driver for critical reflection.

  • What do they say, how do they evaluate the work of others?
  • Is their work ready for criticism?
  • How do they respond to the criticism?
  • This evaluation process is closer to that which they will face in the real world.

This process is close to a ‘real world’ process and the structure encourages the students to reflect on their own work and provides the tools for them to do that. They begin to see their work through the eyes of others.

The use of the peer assessment features and rubrics in Canvas LMS are therefore central to this pedagogy.

Assessment

Assessment for Learning

Assessment can be formative or summative (or even both). Formative assessment can best be defined as an assessment which moves learning forward, while summative assessment measures learning.

The assessment used in the Design Canvas is intended to be strongly formative and is developed from principles collectively referred to as “Assessment for Learning”.

When considering formative assessment we need to think about where the learner is going, where they are now and how they will get there. This needs to be considered along with the joint responsibilities of the teacher, peers and the learner.

Leahy et al. [2] uses this framework to identify 5 key strategies of formative assessment.

Capture

The teacher in traditional classrooms spends a significant amount of time on content delivery, lesson planning and assessment. When a course is delivered through a VLE these roles are filled by the VLE. As a result a teacher has more time to spend in a mentoring role.

The teacher/mentor does not need to be expert in design or the skills the students require, but they do need to fulfill some of the roles of the teacher in the formative assessment structure described above. The mentor is responsible for providing encouragement, support and advice. Clarifying the goals of the course and monitoring progress through the course. The mentor needs to be prepared to intervene as necessary to keep the learning moving forward.

The roles of peer and learner are also key in the Design Canvas. In a traditional classroom it is not easy to effectively activate peers and learners as owners and resources of learning. On the other hand educators agree that these are important steps towards students being independent learners and therefore life-long-learners. The Design Canvas makes use of the online forums and peer assessment throughout the course to activate students in this respect and it is important for the mentor to monitor this activity and intervene as needed to ensure that those processes are working effectively.

So what is the assessment for the Design Canvas? In a typical course there are defined outcomes which students are assessed against. But Wiliam and Leahy [3] point out that often specific learning outcomes cannot be identified, rather a horizon of outcomes exist and

“…sometimes it is appropriate to do things not because they are guaranteed to result in specific learning outcomes but because they are important experiences for students”

The Design Canvas fits into this category of learning experience. Every student will be working on something different and taking a slightly different path through the course. The value of the course is the experience of designing and collaborating, so success is measured in terms of the degree to which they have engaged in that process.

For this reason, while some teachers may decide to assess the final design, it is intended that assessment focuses on completion of the 5 modules and effective engagement with other students through the peer assessment process.

Summary

Through this discussion I have used a specific course as an example, but the principles can be applied to any online learning environment.

Technology provides for new and better ways of doing things, but in education many (most?) teaching is essentially the same process when moved online. This discussion is an attempt to describe a new pedagogy which takes advantage of the online environment.

[1] http://celt.uwindsor.ca/ojs/leddy/index.php/CELT/article/view/3278

[2] Leahy, S., Lyon, C., Thompson, M., & Wiliam, D., (2005). Classroom assessment: Minute-by-minute and day-by-day. Educational Leadership, 63(3), 18-24

[3] Wiliam, D., Leahy, S., (2015). Embedding Formative Assessment: Practical techniques for K-12 classrooms, pg. 29, Learning Sciences International, USA

Advertisements

Assessment for Learning in an online environment

At Don College we have made assessment for learning as defined by Dylan Wiliam a PD priority this year. As a teacher who makes extensive use of online content I found that many of the techniques described were not easy to apply directly to my classroom (which is a computer lab). With that in mind I have addressed the task of devloping some AFL strategies which can be used in my classroom.

What is assessment for learning (AFL)?

AFL has been developed and promoted by Dylan Wiliam. In a nutshell AFL encompasses 5 principles for embedding formative assessment into our pedagogy.

  1. Clarify learning intentions and ensure that teachers, learners and peers share that understanding
  2. Engineer effective classroom discussions, activities and tasks that elicit evidence of learning
  3. Provide feedback that moves students forward
  4. Activating students as instructional resources for one another
  5. Activating students as the owners of their own learning

(Teachers to Schools: Scaling up professional development for formative assessment, Siobhan Leahy & Dylan Wiliam, http://www.dylanwilliam.org/Dylan_Wiliams_website/Papers.html) These 5 principles have been applied by Wiliam to develop a (large) collection of strategies for teachers to employ in the classroom and if you are not familiar with Wiliam’s work I encourage you to investigate further. Wiliam does not expect teachers to use all the strategies, and he is specific in saying that teachers should use those strategies which best suit their class and teaching style. The important thing is to ensure that whatever the teacher does is true to the 5 principles above.

In general the techniques published by Wiliam are designed for a typical classroom. As a teacher of Graphic Design and Computer Science I spend most of my teaching time in a computer lab, so I have been thinking about ways of applying those 5 strategies in an online environment by adding to and adapting existing AFL techniques.

Use of online forums

All VLE environments have a forum feature. There are also free online services that provide a forum platform. (I use a service at http://www.wikidot.com which provides a free forum service if your school does not have a suitable in-house solution) An online forum can be used to support AFL in a virtual environment. Dylan Wiliam describes a technique called the ‘double deadline’.

  • When the students arrive at class they hand their work in with a colour coding. Red indicates that they are struggling, yellow indicates that they have done the work but are not confident, green indicates that they believe they have a full grasp of the work.
  • The teacher then uses those categorizations to pair the students up. Strugglers with confident students, and the teacher puts the yellow students together and helps them.
  • The students review the assignment in pairs according to a check list of expectations.
  • After the discussion the students take their work away and refine it before submitting it at the next lesson for the teacher to assess.

Another technique is called ‘2 stars and a wish’.

  • A student presents their work to the class (or a group) and each class member then reviews the presentation.
  • The review follows a standard format of 2 stars (2 things done well) and a wish (one suggestion for improvement)
  • It is important for the teacher to make clear what the students are looking for so that their stars and wishes are helpful.
  • There needs to be some preparation so that students understand how to give feedback in a positive way. In other words suggest improvements rather than just indicate what was not done well.

I am using online forums as a way of adapting those two techniques to an online environment. When students get their work to a state that is just short of complete they submit it to a class forum. To do this the students upload their work in a cloud service (I use OneDrive) and share the link on the forum. All students check the forum and respond to the posts of their peers using the 2 stars and a wish formula. Students can then take the feedback and modify their work before submitting it for assessment.

  • This technique allows the teacher to monitor the communication between students and step in privately if assistance is needed (for instance if a student is not getting suitable feedback, or enough feedback). Which is something that the classroom techniques don’t allow.
  • Students also learn the best time to ask for feedback. In general students are used to submitting complete work, while it is actually more powerful to submit incomplete work and ask for guidance in completing it. This is a big shift for students, and it takes some getting used to.
  • This technique also provides for students to submit their work at different times, so they submit when they are ready, not when the class timetable dictates.
  • Not only do students benefit from their own feedback, but they see the work and the feedback of all the students in the class. Indeed some students that are having difficulty getting started will be inspired by the early starters.
  • Students experience the difference between plagiarism (stealing work) and collaboration (drawing on work). By not working in isolation they are modelling the real world and learning an important 21st Century skill.
  • I can monitor the forum and assess the final submissions and the quality of feedback each student is giving.

This use of a forum in this way clearly applies all 5 of the AFL principles listed above.

Use a class wiki

This is not a direct adaptation from an assessment for learning technique (at least the ones I know) but it does implement the principles of AFL. We are all familiar with Wikipedia. In my design class the students have developed their own wiki on graphic design, focussing on the curriculum content of their course. You can view this wiki at http://dcdesign.wikidot.com. (Another service available is Wikispaces, or some schools have an intranet that can house a wiki.) At random I allocate each student a topic or key word related to the curriculum . They then check the wiki and if a page exists they read through it and make modifications, additions and corrections as necessary. If the page doesn’t exists they get one started. I don’t make this a long task and typically only commit 20-30 minutes to work on the wiki each week through a school term.

Each year we have a mid year exam. I allow my students free access to the class wiki as they complete the exam. This has the effect of adding a strong formative element to the exam. It also gives the development of the wiki an added purpose.

  • The wiki structure  encourages students to see and form the connections between information, which is the way experienced learners operate.
  • The activity is well differentiated in that some students can add in a picture or correct some typing, while stronger students make more substantial contributions. All contributions are valued.
  • The development of the wiki actually models effective exam preparation. It is commonly known that writing and answering their own sample questions is a highly effective study technique. Writing wiki pages is, I believe, similar in nature and equally as effective.
  • Students are reviewing and improving each others’ work. They are also able to see what other students have done with their work.
  • Students experience an authentic 21st Century collaborative environment and experience the difference between copying and using information.
  • Using the wiki in the exam models 21st Century problem solving and also turns the exam into a formative assessment exercise. Some might think that since they can look up the answers all students would get full marks. My experience is that they don’t as the questions are not simple fact recall, but the wiki does help students to think more about the questions in the exam and not give up if the question looks unfamiliar. I believe I actually get better information about the competence of my students from this ‘open wiki’ exam than I do from a closed exam.
  • The wiki platform allows me to see past versions of each page and identify the edits each student has made. This allows me as a teacher to do an assess the contributions they are making if I wish.
  • The wiki has been running for a few years now, so students are also collaborating indirectly with past and future students.

This learning activity supports all 5 of the principles of assessment for learning, but it is especially powerful at activating students as instructional resources for one another and activating students as the owners of their own learning.

Conclusion

Assessment for learning provides principles and a range of techniques which improve learning. I found that applying the techniques as described by Dylan Wiliam and others were not directly applicable to my ICT based learning environment. I have however, found that activities can be planned using  online forums and wikis that effectively implement the underlying principles of assessment for learning.