Category Archives: Learning Design

Assessment for Learning in an online environment

At Don College we have made assessment for learning as defined by Dylan Wiliam a PD priority this year. As a teacher who makes extensive use of online content I found that many of the techniques described were not easy to apply directly to my classroom (which is a computer lab). With that in mind I have addressed the task of devloping some AFL strategies which can be used in my classroom.

What is assessment for learning (AFL)?

AFL has been developed and promoted by Dylan Wiliam. In a nutshell AFL encompasses 5 principles for embedding formative assessment into our pedagogy.

  1. Clarify learning intentions and ensure that teachers, learners and peers share that understanding
  2. Engineer effective classroom discussions, activities and tasks that elicit evidence of learning
  3. Provide feedback that moves students forward
  4. Activating students as instructional resources for one another
  5. Activating students as the owners of their own learning

(Teachers to Schools: Scaling up professional development for formative assessment, Siobhan Leahy & Dylan Wiliam, http://www.dylanwilliam.org/Dylan_Wiliams_website/Papers.html) These 5 principles have been applied by Wiliam to develop a (large) collection of strategies for teachers to employ in the classroom and if you are not familiar with Wiliam’s work I encourage you to investigate further. Wiliam does not expect teachers to use all the strategies, and he is specific in saying that teachers should use those strategies which best suit their class and teaching style. The important thing is to ensure that whatever the teacher does is true to the 5 principles above.

In general the techniques published by Wiliam are designed for a typical classroom. As a teacher of Graphic Design and Computer Science I spend most of my teaching time in a computer lab, so I have been thinking about ways of applying those 5 strategies in an online environment by adding to and adapting existing AFL techniques.

Use of online forums

All VLE environments have a forum feature. There are also free online services that provide a forum platform. (I use a service at http://www.wikidot.com which provides a free forum service if your school does not have a suitable in-house solution) An online forum can be used to support AFL in a virtual environment. Dylan Wiliam describes a technique called the ‘double deadline’.

  • When the students arrive at class they hand their work in with a colour coding. Red indicates that they are struggling, yellow indicates that they have done the work but are not confident, green indicates that they believe they have a full grasp of the work.
  • The teacher then uses those categorizations to pair the students up. Strugglers with confident students, and the teacher puts the yellow students together and helps them.
  • The students review the assignment in pairs according to a check list of expectations.
  • After the discussion the students take their work away and refine it before submitting it at the next lesson for the teacher to assess.

Another technique is called ‘2 stars and a wish’.

  • A student presents their work to the class (or a group) and each class member then reviews the presentation.
  • The review follows a standard format of 2 stars (2 things done well) and a wish (one suggestion for improvement)
  • It is important for the teacher to make clear what the students are looking for so that their stars and wishes are helpful.
  • There needs to be some preparation so that students understand how to give feedback in a positive way. In other words suggest improvements rather than just indicate what was not done well.

I am using online forums as a way of adapting those two techniques to an online environment. When students get their work to a state that is just short of complete they submit it to a class forum. To do this the students upload their work in a cloud service (I use OneDrive) and share the link on the forum. All students check the forum and respond to the posts of their peers using the 2 stars and a wish formula. Students can then take the feedback and modify their work before submitting it for assessment.

  • This technique allows the teacher to monitor the communication between students and step in privately if assistance is needed (for instance if a student is not getting suitable feedback, or enough feedback). Which is something that the classroom techniques don’t allow.
  • Students also learn the best time to ask for feedback. In general students are used to submitting complete work, while it is actually more powerful to submit incomplete work and ask for guidance in completing it. This is a big shift for students, and it takes some getting used to.
  • This technique also provides for students to submit their work at different times, so they submit when they are ready, not when the class timetable dictates.
  • Not only do students benefit from their own feedback, but they see the work and the feedback of all the students in the class. Indeed some students that are having difficulty getting started will be inspired by the early starters.
  • Students experience the difference between plagiarism (stealing work) and collaboration (drawing on work). By not working in isolation they are modelling the real world and learning an important 21st Century skill.
  • I can monitor the forum and assess the final submissions and the quality of feedback each student is giving.

This use of a forum in this way clearly applies all 5 of the AFL principles listed above.

Use a class wiki

This is not a direct adaptation from an assessment for learning technique (at least the ones I know) but it does implement the principles of AFL. We are all familiar with Wikipedia. In my design class the students have developed their own wiki on graphic design, focussing on the curriculum content of their course. You can view this wiki at http://dcdesign.wikidot.com. (Another service available is Wikispaces, or some schools have an intranet that can house a wiki.) At random I allocate each student a topic or key word related to the curriculum . They then check the wiki and if a page exists they read through it and make modifications, additions and corrections as necessary. If the page doesn’t exists they get one started. I don’t make this a long task and typically only commit 20-30 minutes to work on the wiki each week through a school term.

Each year we have a mid year exam. I allow my students free access to the class wiki as they complete the exam. This has the effect of adding a strong formative element to the exam. It also gives the development of the wiki an added purpose.

  • The wiki structure  encourages students to see and form the connections between information, which is the way experienced learners operate.
  • The activity is well differentiated in that some students can add in a picture or correct some typing, while stronger students make more substantial contributions. All contributions are valued.
  • The development of the wiki actually models effective exam preparation. It is commonly known that writing and answering their own sample questions is a highly effective study technique. Writing wiki pages is, I believe, similar in nature and equally as effective.
  • Students are reviewing and improving each others’ work. They are also able to see what other students have done with their work.
  • Students experience an authentic 21st Century collaborative environment and experience the difference between copying and using information.
  • Using the wiki in the exam models 21st Century problem solving and also turns the exam into a formative assessment exercise. Some might think that since they can look up the answers all students would get full marks. My experience is that they don’t as the questions are not simple fact recall, but the wiki does help students to think more about the questions in the exam and not give up if the question looks unfamiliar. I believe I actually get better information about the competence of my students from this ‘open wiki’ exam than I do from a closed exam.
  • The wiki platform allows me to see past versions of each page and identify the edits each student has made. This allows me as a teacher to do an assess the contributions they are making if I wish.
  • The wiki has been running for a few years now, so students are also collaborating indirectly with past and future students.

This learning activity supports all 5 of the principles of assessment for learning, but it is especially powerful at activating students as instructional resources for one another and activating students as the owners of their own learning.

Conclusion

Assessment for learning provides principles and a range of techniques which improve learning. I found that applying the techniques as described by Dylan Wiliam and others were not directly applicable to my ICT based learning environment. I have however, found that activities can be planned using  online forums and wikis that effectively implement the underlying principles of assessment for learning.

Design for Learning – Handsonict MOOC

I have just competed a rather intensive MOOC on designing ICT rich learning activities called Hands- on ICT.

In the words of the MOOC developers it:

… [focuses] on the Learning Design Studio (LDS) approach, a design process to help educators design courses and learning activities.

A colleague has posted in detail his overall impressions of the MOOC at http://ianinsheffield.wordpress.com/2014/12/02/definitely-handson/ and rather than cover that ground again, I will endorse his comments and suggest you read his contribution yourself.

Since the MOOC finishes tomorrow, and may not be offered again (let’s hope it can be) I would like to preserve the essentials of the LDS approach as it was presented through the MOOC.

In a nutshell the Learning Design Studio (LDS) approach to educational design is an adaptation of the standard design process as taught in my design classes. The steps in the LDS process are adaptations from the object design process, taking into account the learning context for the design. In other words the result of the LDS process is a fluid activity involving interactions between learners and teachers, rather than a static artifact and the process reflects that.

The Dream

It is usual in design to start with a design brief. The design brief sets out the design problem and the context, constraints and considerations, for the design.

In the LDS based approach the brief is replaced with a dream. The dream consists of:

  • Situation or context
  • The change we would like to see (challenge)
  • The proposed solution
  • Measures of success

So the dream fulfills the role of the design brief, but it is subtly different. While a design brief begins with a problem and outlines the parameters around that problem, the dream involves looking at how things are in our classroom, imagining how it might be better and describing what that ‘something better’  might look like.

The dream is the driver for the learning design process and much of what follows involves adding specifics and details to the initial dream.

Starting with a dream colours the entire design process with an optimism. In some ways traditional object designers find themselves always referring back to a problem, while in this learning design process the designer is always referring back to a solution, a vision of something better.

Persona

The second stage is to develop persona(e) for the participants in the learning activity. The persona is an extension from the context in the dream. This is an important stage, as learning is a human activity and it is important to consider those personae in the learning design. The personae consist of things such as:

  • Education and experience
  • Role and responsibilities
  • Technical skills
  • Subject domain skills and knowledge
  • Motivation and desires
  • Goals and expectations
  • Obstacles to their success
  • Unique Assets

All these things have an impact on the teaching and learning and need to be considered in the learning design.  For instance pitching an activity at the wrong level of technical skill for the participants can limit the effectiveness of the activity. Failure to align the activity with the Goals of the students can also create issues etc.

Context, Factors and Concerns

Based on the personae and the dream, the next step in the process is to identify specific factors and concerns. These might involve the availability of suitable resources, or factors arising out of the personae such as inappropriate motivations or goals, or limited background skills.

Once these things are documented they can be taken into account in the final learning design.

Heuristics

Heuristic evaluation is a technique borrowed from usability research, where a group of experts is asked to assess a particular design using a given rubric (set of heuristics). It offers a low-cost rapid evaluation which often uncovers design flaws at an early stage.

It is fair to say that this step in the LDS process caused the most difficulty for the participants in the MOOC. The heuristics consist of a series of evaluative questions or considerations used to judge the effectiveness of the activity. In my case the activity was an online learning package and my heuristics evaluated content, layout, navigation, platform and pedagogy. For example my content heuristics included:

  1. Structure and advanced organizers
    1. Are there summaries which assist communication?
    2. Is the content logically organized through the units and pages?
    3. Are the rubrics clear and effective in assisting peer review?
  2. Language
    1. Is the language appropriate to 16 – 18 year old students?
    2. Are the ideas clearly explained?

These questions can be used to evaluate and improve the activity.

Based on the forum discussion it seemed that a lot of the participants had difficulty envisioning heuristics for an activity that was yet to be written. It is important to write the heuristics first, however, so that they can guide the development of the activity. Heuristics written later in the process are likely to be limited by the designer’s perception of the work. In effect there is the danger that the heuristics share the designer’s blindness to the weaknesses in the activity. In other words the heuristics need to guide the design process, not be shaped by the design process.

Research existing activities

It is a standard part of any design activity to look for comparable work. This can act as an inspiration while also avoiding ‘reinventing the wheel’.

In my case I found a number of examples amongst the references given which assisted in developing my activity.

Scenario

At this stage the designer returns again the the dream to write a scenario about the learning activity. The scenario is really just a story about the activity in operation. It is important because it focuses the dream and assists the designer to fully envision how the activity will work from the student’s point of view. This slide share (also provided as part of the MOOC resources) describes scenarios well:

Prototype

At this stage a prototype, or rough version, of the activity is developed. This is a further opportunity to envision the learning activity as a whole. In object design a prototype would consist of a scale model, but in learning design it is more likely to take the form of a part of the activity or an overview of the activity. Participants in the handson MOOC favoured Prezi presentations, or mind maps for their prototypes. These are suitable for providing an overview, but in my case it was difficult to apply the heuristics to properly test the prototype. Sample content or an abbreviated activity is more effective at this stage in my (retrospective) opinion.

The Activity – Evaluation and Review

After all the work it is with some relief that the activity can be developed. And of course evaluated and reviewed using the heuristics developed earlier. (In the design world nothing is ever finished.)

It is tempting to wonder why this involved process in developing a learning activity is important. At times I questioned some of the steps as I completed them, but towards the end of the process it became obvious to me that I had learned things along the way that significantly informed the activity I developed. It is true that as teachers we don’t have the time to replicate this process for every learning activity we develop, but it is certainly valuable to apply this process to some activities. It is also true that having been through this process once, much of it can be reused to develop further activities. For instance the personae, context, heuristics etc. once developed will be relevant to further learning design situations.

I think that as we come under increasing pressure to develop learning activities that are relevant to students and provide rich opportunities to develop 21st Century skills, this design process will be an important aspect of our professional practice. I was not taught to approach learning design this way during my teacher training, but I am sure that I would have become a better teacher more quickly if I had.

For more information on LDS I refer you to this blog. http://www.somolearn.org/lds