After 3 general elections with severe polling errors, the UK opinion pollsters redeemed themselves in the 2019 UK General Election with their most accurate performance since 1955. I base this statement on data provided by Mark Pack who has systematically recorded every opinion poll published since 1945. The challenge now for the industry is to maintain this level of performance for the next election which may be easier said than done given that 5 out of the last 8 elections have experienced a major polling error.
My analysis looks at only the polls that took place in the week before every general election between 1950 & 2019. Note this is based on the fieldwork dates, not the publication date which can be a few days later. For the elections in the 1950’s, Gallup were the only pollster so instead of using the week before, I used the month before.
For each election, I calculated the average vote share recorded across all pollsters for the Conservatives, Labour and Liberal Democrats (Liberals 1950 to 1979, Alliance 1983 & 1987). The polling error for each party can then be calculated as the actual election result for Great Britain minus the average vote share from the polls. I use the figures for Great Britain rather than the United Kingdom since nearly all polls do not survey in Northern Ireland.
The results can be seen in the chart below. The black line is the number of polling companies that polled in the run up to the election.
After 3 general elections in a row with major polling errors, the polling industry breathed a sigh of relief in 2019 with all 3 parties predicted to within 2 percentage points. The last time this happened was in 1979. In fact, the pollsters performance is better than 1979 if you measure the RMSE (Root Mean Squared Error) based on the 3 parties. This is calculated by squaring the errors, finding the average of the squared errors and then taking the square root of the average. In 2019, the RMSE was 1.1% which is the lowest seen since 1955 and the second lowest on record.
Has the polling industry overcome the issues it has had in the 3 elections prior to 2019? History tells us that RMSE this low are rare and higher errors can occur. During the election, I kept pointing out that 5 of the last 7 (now 8) elections saw at least one party experiencing a polling error of at least 4 percentage points, namely –
- 1992 – An underestimate of 5.2% for the Conservatives.
- 1997 – An overestimate of 4.0% for Labour.
- 2010 – An overestimate of 4.1% for the Liberal Democrats.
- 2015 – An underestimate of 4.1% for the Conservatives.
- 2017 – An underestimate of 4.5% for Labour.
Of the 3 elections that didn’t see such errors, 2001 still had a 3.5% overestimate for Labour and 1992 also had a 3.9% overestimate for Labour. Prior to 1992, only two elections experienced such large errors which were 1970 (Labour overestimated by 4%) and 1951 (Labour underestimated by 4.3%). Such a shift in error rate from 2 out of 12 to 5 out of 8 is statistically significant. So whilst I hope that the industry has made progress, the worst thing they can do is be complacent and assume they have the issues licked.
One of the things that strikes me about the first chart above is how the errors for Labour and the Liberal Democrats are inversely correlated with each other with a correlation coefficient of -0.61. This makes sense in today’s environment where there is a lot of talk of a progressive alliance between Labour, Lib Dems and the Greens and one easily imagine a scenario where tactical voting means the polls overestimate Labour and underestimate the Lib Dems. However it would seem that this has been a dynamic in British elections for a very long period of time. The equivalent correlation between the Conservatives and Lib Dems is only -0.13.
Given this, I have redone the chart by combining the Conservatives & UKIP & Brexit Party into one group and Labour, Lib Dems & Greens into another group. In practice, the UKIP, Brexit Party & Green errors are only known for the last 3 elections as those have been the only elections where pollsters have recorded votes for these parties separately rather than putting lumping them into Others. So for the most part the chart below is comparing the Conservative poll error with the combined Lab/LD error.
The lines represent centred 5-election moving averages and currently sit at a 1.1% underestimate for the CON/UKIP/BXP error and a 1.3% overestimate for the LAB/LD/GRN error.
This revised chart makes things a lot clearer and also emphasises the exceptional nature of 2017. In 2010, the Lib Dems were badly overestimated but this was partly compensated by an underestimate in the Labour vote. In 2017, the underestimate in the Labour vote was only slightly compensated by an error in the Lib Dem & Green vote. The CON+UKIP vote share was overestimated by 2.4% but this was almost entirely due to the UKIP vote being overestimated since the Conservative vote share was more or less spot on.
In effect, 2017 was a repeat of the 1983 & 1951 elections but on a larger scale. History shows that errors like these favouring Labour are exceptional and the norm has been errors (usually significant errors) that favour the Conservatives. The chart above shows only 6 elections out of 20 with errors favouring the “progressive alliance” with 4 of these taking place before 1966. Conversely, the 7 elections prior to 2017 all had errors favouring the Conservatives.
So far I have been concentrating on the expected vote share for each party or combination of parties. In practice, when it comes to predicting the outcome of an election that uses First Past the Post as its election system, the more important prediction is the Conservative lead over Labour. These parties have always been expected to take the top two places nationally so I have calculated the expected lead from the polls and compared it with the actual lead to produce the following chart.
We see again how 2017 reflects 1983 though the error is larger than 1983 but not as large as 1951. The 5-election centred moving average still shows an underestimate of 1.5% in the Conservative lead over Labour and indeed this appears to have been the long run average since 1964. It is very tempting (and no doubt many people will try) to put reasons on this chart but treated as a time series in its own right, I have to say that I do not see any explanatory patterns apparent.
If we define a SIGNIFICANT error as being one where the CON-LAB lead is out by 3%, then we can make the following observations about the 19 elections since 1950:-
- 8 out of 20 elections did not experience a significant polling error (1955, 1959, 1964, 1974F, 1979, 1987, 2010, 2019)
- 8 out of 20 elections experienced a significant polling error favouring the Conservatives. (1966, 1970, 1974O, 1992, 1997, 2001, 2005, 2015)
- 4 out of 20 elections experienced a significant polling error favouring Labour (1950, 1951, 1983 & 2017)
- The average polling error (in CON-LAB lead) is +1.5% and the standard deviation is 4.2%.
- If our null hypothesis is that the average polling error is 0%, then our t-statistic is +1.26 and the p-value (using 2-tailed t-test) is 23%.
- Nothing greatly changes if we confine our analysis to 1974 onwards i.e. from when the Northern Ireland parties and the Nationalists arrived on the political scene and the CON+LAB vote share saw a significant shift downwards.
I chose 3% as a definition of a significant error as my experience shows that an error on this scale will mislead election forecasters as happened for the most part in 2017.
Ahead of the 2019 election, I stated that I would calculate 3 scenarios as below:
- A 4% underestimate in the CON-LAB lead as stated by the polls which favours the Conservatives.
- No error in the CON-LAB lead as stated by the polls.
- A 4% overestimate in the CON-LAB lead as stated by the polls which favours Labour.
I also stated that I would use a 2:1:1 ratio of these scenarios as my official forecast i.e. I would give scenario 1 50% weight, scenario 2 25% weight and scenario 3 25% weight. This is what I ended up doing and I see no reason to change this for the next election though based on the numbers listed above, an alternative weighting would be 2:2:1.
