The new primary progress measure is a Good Thing

I do my best to be fair to the Department for Education, even when I find things frustrating. (I honestly do!) I’ve given the department it’s fair share of stick, but the civil servants who work there have a difficult job and almost certainly deserve more credit than I give. And despite the complications and delays and various other issues relating to primary assessment, I think when it comes to the new progress measures for primary school, the department has got it right.

The calculations are complex, but I shall try to illustrate here why I think the new method is an improvement on the old. It is fairly closely based on the existing Value Added measure, but linked to the new scaled scores in KS2. I provide this video to illustrate my understanding of how it works… and hope that if I’ve made an error that someone at the DfE will correct me.

Importantly, the measure is based on individuals’ progress. Each child’s progress is compared to that of other children with similar starting points (if you consider their KS1 assessments to be a starting point!), so even if they don’t reach the new expected standard, their progress can be recognised.

The reason this is important is that reduction of the cliff-edge measures – something best illustrated by the extremes. Many teachers will be familiar with the child who was very close to level 3 at KS1 and just scraped a L4 at KS2 – they are counted as having achieved two levels progress, despite having barely moved more an a single level. Contrast them to the child who barely scraped a Level 2 on teacher assessment at KS1 and then misses Level 4 in KS2 by a single mark. Despite having very nearly made the full 2 levels, and more than the first pupil, they are counted as having failed to achieve the 2 levels measure.

2levels

Under the new system, the measures of progress are much finer, and so not so much rests on specific thresholds. That’s got to be an improvement. Instead of everything hanging on the luck of the thresholds for Level 4, or the poor child who misses the key point by a single mark, the progress of all pupils is counted towards the final measure for schools.

It also helps to improve one of the other issues of the system. We know that children with higher prior attainment are more likely to make rapid progress. By comparing individual pupils to peers with similar starting points, that likelihood is less problematic. All of those pupils with low attainment in KS1 will be compared to others with similar attainment; schools with high proportions of lower prior attainment won’t be battling to match the progress figures of high attaining schools in very different areas, because the measure will be based on the sum of the work they’ve done with all their children, not just the few who happen to tip over the next threshold by May.

It won’t solve everything. There are still going to be negative effects. But it’s a much better system than the one we’ve got, and the department should be proud to announce it.

(Although if you’re thinking about it, DfE: judging by how hard I’ve found it to write this post and make the accompanying video, can I suggest that you make something even better to clearly illustrate the numerical frippery?)

Tagged: , ,

31 thoughts on “The new primary progress measure is a Good Thing

  1. cazzypot2013 13 November 2015 at 5:30 pm Reply

    Reblogged this on The Echo Chamber.

  2. northnibleyhead 13 November 2015 at 6:42 pm Reply

    Hi – and thank you. Fascinating. Do we know if another child with an identical KS1 average point score to ‘Abigail’, but attained with (for example) higher Maths and lower Reading, is part of the same group? Does the mix of L2C’s and L1’s make a difference?
    Thanks, Paul. PS – Can I use this with my Governors next week?

    • Michael Tidd 13 November 2015 at 6:47 pm Reply

      The mix of results doesn’t matter apparently. The chap I asked from the DfE said that they’d found that APS is actually a more reliable predictor than separate levels.
      And yes, please feel free to use it!

  3. northnibleyhead 13 November 2015 at 9:20 pm Reply

    Many thanks. Does it therefore follow that the separate progress scores at KS2 will get combined (EYFS style) to determine a “Good Level of Development / Progress”. If at KS1 the APS is king, this would make KS2 the odd one out if no combined measure is created. Just thinking! See you in Gloucestershire in January. Paul.

    • Michael Tidd 14 November 2015 at 10:45 am Reply

      I don’t think there’s any need for a combined figure like that, which would be an attainment score really.

  4. Rut 14 November 2015 at 8:27 am Reply

    Hmmm….. Sort of. The positives you are explained only to be replaced with a whole host of more negatives. Individual children against individual children and the potential for this to be reported in a multitude of ways. Like I said hmmmmm!!

  5. Hayley Earl 14 November 2015 at 8:44 am Reply

    Reblogged this on The Musings Of A Teacher and commented:
    As ever,Michael Tidd provides a clear and simple explanation of what many may find difficult to understand.

  6. Hayley Earl 14 November 2015 at 8:45 am Reply
  7. Alan Phair 16 November 2015 at 8:05 am Reply

    Just a couple of points to add to a great piece of work you’ve done. Doesn’t this replicate what currently happens with the VA scores on RAISE?
    Secondly, I must pick you up on the statement “We know that children with higher prior attainment are more likely to make rapid progress”. My experience tells me that many high attainers in fact plateau and many low attainers were not really low attainers as such, they were children who hadn’t yet reached their stride (and I am not talking about children with SEN/D). Also, to kind of counter your argument, the percentage of low attainers (ie. reaching Level 1 at Year 2) who go on to exceed 2 levels of progress by end of KS2 in reading, writing and maths is 62%, 57% and 43% respectively. The percentage of high attainers (Level 3 at end of Year 2) who exceed 2 levels of progress is 1%, 13% and 34%. There is a sort of glass ceiling for progress for these high attainers, certainly in reading and writing, that the new system may remove – perhaps.
    Would you agree?

    • Michael Tidd 16 November 2015 at 5:09 pm Reply

      I think you raise a good point. It’s interesting the differences you highlight in the 3LP bands. I wonder how much of that is currently influenced by the all-important Level 4 threshold. For while in theory getting a L1 child to L4 is worth as much as getting an L3 child to L6, in reality the high stakes of the L4 assessments means that there is a far higher incentive to achieve the former than the latter.
      I’d also raise some queries about the L6 tests in KS2 anyway, but that’s a whole other story!

  8. Simon 16 November 2015 at 12:13 pm Reply

    Hi,

    I’ve worked in year 6 for years and I thought this is what currently happened with VA scores on Raise? That ‘Each child’s progress is compared to that of other children with similar starting points’ wasn’t new and was the status quo?

    So a 2c child achieving 3a, although can only go down as 1 level progress which I agree is wrong, they were still compared to other 2c children and their 3a was taken into account for their individual progress then combined for VA?

    • Michael Tidd 16 November 2015 at 5:07 pm Reply

      You’re quite right that it is what currently happens with the VA measure (or very similar, anyway). The difference is that this will now become the main (only?) measure of progress that is used to judge schools.

      • Simon 18 November 2015 at 8:47 am Reply

        Ah ok, you mean because the ‘2 levels progress’ measure has gone? I agree this was far too broad a measure.

        The video is great and a shame we have to come up with these things ourselves on top of our day job, so thanks! One more query though, I would have thought they’d do this comparison with individual subjects’ points scores for their progress measure in each subject rather than combing reading, writing and maths to find an overall measure. This will then give the child a + or – progress measure for each subject and be much more helpful and tell a better, more detailed story of the school don’t you think?

        • Michael Tidd 18 November 2015 at 8:57 am

          Yes – it’s a commonly-held view. The DfE say that they have looked in detail at previous years and they feel that the combined APS score is a better (fairer?) predictor of attainment at the end of KS2. It does make some sense as, for example in Maths, the ability to read is definitely a factor in attainment on the test.
          I can only presume that they’ve looked in some depth at the figures and found stronger correlation between APS scores and KS2 results than for individual subjects.
          Which is another nail in the coffin of 2LP!

    • northnibleyhead 18 November 2015 at 10:33 am Reply

      If I have understood correctly Jamie Pemboke’s excellent post here – http://sigplus.blogspot.co.uk/2015/11/the-value-added-tax.html – then schools where Maths was traditionally a greater strength at KS1 had higher fine-grades points to reach before positive value-added was achieved. The new combined APS ‘predictor’ should favour those schools since it lowers the +VA threshold. This might mitigate some of Alan Phair concerns above – unless of course your school has stronger Reading and Writing, In which case it’s worse!

  9. Kate Cameron 16 November 2015 at 1:39 pm Reply

    Interesting… I think it would actually be helpful to know individual’s progress measures, the way we currently do about the 2 or 3 levels progress. We need to unpick the story and if we get a lower (or higher) than expected (or previous) overall VA be able to see why. Which pupils have pulled the score down/up? Do they have anything in common, either as a pupil attribute or as a provision the school has put in place?

    The bit that worries me is that all this is predicated on test scores – one or two hours on one or two days in May for pupils at a fairly tender age of 10 or 11. But that’s nothing new!

  10. Greg 18 November 2015 at 11:27 pm Reply

    This was really clear, but when I’ve watched it through a few more times, I now feel a little confused! If ‘John’ scored a level 3 at maths (21) in KS1, a level 1 in writing (9) and a level 1 in reading (9) = 39 divided by 3 which equals 13. He’s then put into a group of children who had that same average. As in your example on the video, you compared an average of all 3 subjects in KS1 and compared it to one subject (maths in your example) in KS2. If you compared an average of the 3 in KS1 v an average of the 3 in KS2, then that would make sense. Surely there should be a direct comparison between subjects!

  11. David G 21 November 2015 at 5:02 pm Reply

    The video is a great way to try and understand the new system … however, I think I may have misinterpreted it ! I have just used this system to work out the progress of our current Y6s based on their KS1 APS and their scaled scores in a recent standardised test. Each group (and therefore the class as a whole) had an average progress score of 0. At first I thought it was a fluke, so I played around with different numbers and always came up with a score of 0.

    If I demonstrate with one group – all these children had an APS of 21 at KS1.
    In the test, their scaled scores were as follows:

    Child A 129
    Child B 127
    Child C 117
    Child D 126
    Child E 118
    Child F 121

    The average score is therefore 123

    Progress scores for each child are:

    Child A +6
    Child B +4
    Child C -6
    Child D +3
    Child E -5
    Child F -2

    Total progress score of 0 and therefore an average of 0

    As I said earlier, this happens with every group … am I going wrong somewhere ?! Data and statistics has never been my strong point !

    • Michael Tidd 21 November 2015 at 6:00 pm Reply

      An entirely understandable misunderstanding, based on my lack of clarity in the commentary. As you rightly say, comparing within your cohort you’ll end up with a progress score of 0. However, when each child is put into a group of others of the same starting point, that is a *national* group

      • David G 21 November 2015 at 6:03 pm Reply

        That makes more sense !
        Thank you for clarifying that !

      • Tom Bishop 26 November 2015 at 11:05 am Reply

        I’m very pleased and impressed with your work Michael – thank you. I’m wondering 2 things:

        1. How did David G calculate his scaled scores (or were these simply examples of what could be)?

        2. Where did you get the information about the Primary Progress measure from originally?

        Thank you for your help.

        • Michael Tidd 26 November 2015 at 11:10 am

          I think David’s scaled scores were just ‘mock data’
          Original information is only available so far from having been at conferences with DfE speakers!

        • David G 26 November 2015 at 11:23 am

          Our scaled scores were “standardised” scores from tests we’ve just done although we’re not entirely sure if a scaled score and standardised score are the same !

        • Michael Tidd 26 November 2015 at 11:27 am

          Ah, they’re definitely not. 100 on a standardised score means average (compared to the large sample that were used for setting the standard), whereas – in theory – 100 on the scaled score should represent anyone in the top 85% , very roughly (i.e.children who would have got a 4b+ in the past)
          One presumes, therefore, that a standardised score of 100 would lead to a scaled score *higher* than 100. But it’s a big presumption!

        • David G 26 November 2015 at 12:00 pm

          Thank you !!

  12. Tom Bishop 26 November 2015 at 2:38 pm Reply

    So I wonder… to play the progress game would it be best to rank the current Year 6 in terms of KS1 APS and then with current data (however this is collated – sample test scores for example), then provide interventions for those who fall below in the current rankings compared to KS1 APS rankings…

    • Michael Tidd 26 November 2015 at 2:41 pm Reply

      Much like people used to do with children falling behind their 3 points per year!

      • Tom Bishop 26 November 2015 at 2:47 pm Reply

        I guess those children who had a 2C to defend had a big part to play in schools stretching that 3 points to 4.

  13. Ed Crocombe 31 December 2015 at 11:57 am Reply

    Thanks for all these comments – I am still left wondering how the scaled score will be calculated from the KS2 raw score. Having (after many emails) managed to extract some sort of response from the DfE, it seems they will scale scores using ‘item response theory’. However, this can only be done when they can see how pupils have responded to each question. Do you have any further detail on how the KS2 raw score will be converted into a scaled score?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: