The impossibility of Teacher Assessment

I’ve said for a fair while now that I’d like to see the end of statutory Teacher Assessment. It’s becoming a less unpopular thing to say, but I still don’t think it’s quite reached the point of popularity yet. But let me try, once again, to persuade you.

The current focus of my ire is the KS2 Writing assessment, partly because it’s the one I am most directly involved in (doing as a teacher, not designing the monstrosity!), and partly because it is the one with the highest stakes. But the issues are the same at KS1.

Firstly, let me be frank about this year’s KS2 Writing results: they’re nonsense! Almost to a man we all agreed last year that the expectations were too high; that the threshold was something closer to a Level 5 than a 4b; that the requirements for excessive grammatical features would lead to a negative impact on the quality of writing. And then somehow we ended up with 74% of children at the expected standard, more than in any other subject. It’s poppycock.

Some of that will be a result of intensive drilling, which won’t have improved writing that much. Some of it will be a result of a poor understanding of the frameworks, or accidental misuse of them. Some of it will be because of cheating. The real worry is that we hardly know which is which. And guidance released this year which is meant to make things clearer barely helps.

I carried out a poll over the last week asking people to consider various sets of success criteria and to decide whether they would be permitted under the new rules which state that

independent

So we need to decide what constitutes “over-aiding” pupils. At either end of the scale, that seems quite simple.Just short of 90% of responses (of 824) said that the following broad guidance would be fine:

1.png

Simplest criteria

Similarly, at the other extreme, 92% felt that the following ‘slow-writing’ type model would not fit within the definition of ‘independent’:

8

Slow writing approach

This is all very well, but in reality, few of us would use such criteria for assessed work. The grey area in the middle is where it becomes problematic. Take the following example:

5

The disputed middle ground

In this case results are a long way from agreement. 45% of responses said that it would be acceptable, 55% not. If half of schools eschew this level of detail and it is actually permitted, then their outcomes are bound to suffer. By contrast, if nearly half use it but it ought not be allowed, then perhaps their results will be inflated. Of course, a quarter of those schools maybe moderated which could lead to even those schools with over-generous interpretations of the rules suffering. There is no consistency here at all.

The STA will do their best to temper these issues, but I really think they are insurmountable. At last week’s Rising Stars conference on the tests, John McRoberts of the STA was quoted as explaining where the line should be drawn:

That advice does appear to clarify things (such that it seems the 45% were probably right in the example above), but it is far from solving the problem. For the guidance is full of such vague statements. It’s clear that I ought not to be telling children to use the word “anxiously”, but is it okay to tell them to open with an adverb while also having a display on the wall listing appropriate adverbs – including anxiously? After all, the guidance does say that:

guidance.png

Would that count as independent? What if my classroom display contained useful phrases for opening sentences for the particular genre we were writing? Would that still be independent?

The same problems apply in many contexts. For spelling children are meant to be able to spell words from the Y5/6 list. Is it still okay if they have the list permanently printed on their desks? If they’re trained to use the words in every piece?

What about peer-editing, which is also permitted? Is it okay if I send my brightest speller around the room to edit children’s work with them. Is that ‘independent’?

For an assessment to be a fair comparison of pupils across the country, the conditions under which work is produced must be as close to identical as possible, yet this is clearly impossible in this case.

Moderation isn’t a solution

The temptation is to say that Teacher Assessment can be robust if combined with moderation. But again, the flaws are too obvious. For a start, the cost of moderating all schools is likely to be prohibitive. But even if it were possible, it’s clear that a moderator cannot tell everything about how a piece of work was produced. Of course moderators will be able to see if all pupils use the same structure or sentence openers. But they won’t know what was on my classroom displays while the children were writing the work. They won’t know how much time was spent on peer-editing work before it made the final book version. They won’t be able to see whether or not teachers have pointed out the need for corrections, or whether each child had been given their own key phrases to learn by heart. Moderation is only any good at comparing judgements of the work in front of you, not of the conditions in which it was produced.

That’s not to imply that cheating is widespread. Far from it: I’ve already demonstrated that a good proportion of people will be wrong in their interpretations of the guidance in good faith. The system is almost impossible to be any other way.

The stakes are too high now. Too much rests on those few precious numbers. And while in an ideal world that wouldn’t be the case, we cannot expect teachers to provide accurate, meaningful and fair comparisons, while also judging them and their schools on the numbers they produce in the process.

Surely it’s madness to think otherwise?


For the results of all eight samples of success criteria, see this document.

 

Advertisements

Tagged:

8 thoughts on “The impossibility of Teacher Assessment

  1. Teachercath123 8 December 2016 at 5:05 pm Reply

    I am with you! In our school last year we followed the framework diligently and were rigorously moderated by the LA. Our results reflected the impact of the higher standards. At local cluster meetings earlier this term, it became apparent that moderation in the area had been inconsistent and application of the frameworks similarly inconsistent, making the writing assessment data useless and irrelevant.

  2. Tom Burkard 8 December 2016 at 5:51 pm Reply

    The ability to write well is one of the highest-order skills of all, and the formal assessment of pupils’ writing at KS2 is a joke. Yes, we can and should teach subskills like spelling and grammar, and errors in pupils’ work should be marked as a matter of course. However, we’re dreaming if we think that formulaic application of the ‘success critieria’ will improve children’s writing.

    Any kind of subjective assessment is highly problematic both in terms of validity and reliability. It is also extremely time consuming both in terms of preparing children for tests and marking. The need to demonstrate progress leads to such wildly inflated assessments that secondary schools routinely re-assess their intakes. They in turn find their teacher assessments are wildly optimistic by the time they reach KS4.

    .

  3. Mrs A 8 December 2016 at 8:12 pm Reply

    As a moderator for writing and year 6 teacher, I completely agree with you. Last year I signed up to be part of the moderating team for my area under the thought, ‘If you can’t beat them join them’. I thought if I was on the inside, I might have a clue what was going on and be better prepared and more confident in my own teacher assessment.
    Schools all interpreted the interim assessment guidelines differently and one school was unware that there was even any exemplification material.
    The type of work I had to look at from children was prescriptive and boring. I was shown books that had beautifully presented, polished pieces of writing that all seemed rather formulaic and rehearsed. I was told by one school that draft work wasn’t available as it was done on bits of paper and hadn’t been saved and by another, that the children all used erasable pens in their English books: polishing pens! I had no way of telling what was really independent work.
    All schools agreed that they were unsure how to define independent work and seemed to look to me to have a definitive answer for them. But as your own research and survey shows – there is no answer.
    I came away from my moderating experience feeling angry for having signed up for the role (unpaid) and embarrassed for having put my name to the sham that is Interim Assessment. Today I received a request to sign up again for this year!

  4. Julie McCulloch 9 December 2016 at 9:27 am Reply

    Completely agree, Michael. What are your thoughts on whether/how we should assess writing?

  5. Richard 14 December 2016 at 5:17 pm Reply

    I am relatively new to teaching, having previously had a career in management consulting. Back in the day, consulting would get a bad rap for its methodologies leading to specious conclusions used to direct a firm’s management. But let me tell you, nothing I ever saw or did would come even close to the KS2 writing assessment.

    I mean, what’s the point of a writing assessment? To see if schools are teaching their children to write well. What’s the KS2 writing assessment data telling us? Absolutely nothing. Hundred of thousands of man hours to produce a dataset that is total garbage. Hundred of thousands of teacher hours that could have been better spent helping children to write well.

    You’d think there would be a strong correlation at school level between the reading, SPAG and writing performances (and maths, to be honest), but writing is all over the place. Something’s up. Schools with low SPAG and reading scores with high writing scores – I just don’t buy it. 74% at expected standard for writing – utterly unbelievable if you have a look at the criteria for meeting standards.

    I have asked many KS2 teachers why the KS2 writing test was abandoned. I was surprised that many disliked it saying in their defence that children could have an off day and so get a lower score than expected. At school level (if we ignore the small classes of rural schools) it would all average out, surely?

    The test is really there to measure the school, not the child. Whether a child passes or fails a test aged 11 has no effect on their outcome aged 16 or 18. I’ve looked at a few old KS2 writing tests and they look fine to me. I’d would have been happy for any of my year 6s (when I taught that year) to take them and show what they could do.

    I like teaching year 6 children – so much talent and enthusiasm in the room – but the KS2 writing assessment is enough to put me off doing it any time soon. And to be honest, the mere existence of the KS2 writing assessment makes me wonder about the professionalism in the management of education in this country. It should never have been allowed to get out of the blocks.

  6. teachingbattleground 15 January 2017 at 7:50 am Reply

    Reblogged this on The Echo Chamber.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: