Nigel Marriott's Blog

An independent statistician using data to understand our world and to predict the future

You are here: Home / Elections / UK General Elections #8 – How accurate are voting intention polls? – updated with GE2024

UK General Elections #8 – How accurate are voting intention polls? – updated with GE2024

By

After their 2nd best ever performance in the 2019 UK General Election, the UK polling industry flipped in the 2024 UK General Election to their 2nd worst ever performance after 1992.  6 of the last 9 elections have seen at least one party experience a major polling error.  It would appear the move to web polling and lower barriers of entry has led to poorer quality polls.

Data used in this article

I base my conclusion on an analysis using data provided by Mark Pack who has systematically recorded every opinion poll published since 1945.  I’ve used only the polls that took place in the week before each of the 21 General Elections between 1950 & 2024 based on the fieldwork dates, not the publication date which can be a few days later.  For the elections in the 1950’s, Gallup were the only pollster so instead of using the week before, I used the month before.  You can download a spreadsheet with the data I’ve used here.

GB General Election Data 1918-2024 – votes and seats

For each election, I calculated the Polling Vote Share for each party as the average vote share recorded across all polls for the Conservatives, Labour and Liberal Democrats (Liberals 1950 to 1979, Alliance 1983 & 1987) in the week/month before the election.  The Polling Error for each party can then be calculated in two different ways –

  1. National Poll Error – the National Vote Share for Great Britain minus the Polling Vote Share.
  2. Average Poll Error – the Average Vote Share per Seat across Great Britain minus the Polling Vote Share.

For a more detailed explanation of these two methods of calculating polling error, please read this article.

Note I use vote shares for Great Britain rather than the United Kingdom since nearly all polls do not survey in Northern Ireland.

This article will begin with National Poll Errors since that is how most commentators look at polling error.  However, I will then finish with Average Poll Errors since I consider that to be more useful for election forecasting.

National Poll Errors by Party – 1950 to 2024

The errors for the three main parties can be seen in the chart below.  The black line is the number of polling companies that published polls in the run up to the election.

2024 saw more pollsters publish polls than ever before and the outcome was the industry’s 2nd worst ever performance since 1992.  The error in Labour’s national vote share was the 2nd largest ever after 2017, whilst the error in the Conservative’s was the 3rd largest after 1992 and 2015.  After recording their 2nd lowest RMSE (Root Mean Squared Error) based on the main 3 parties in 2019, the industry reversed this in 2024 with their 2nd highest RMSE after 1992.

Note – RMSE is calculated by squaring the national poll errors for the Conservatives, Labour and Liberal Democrats, finding the average of the squared errors for these 3 parties and then taking the square root of the average.  In 2024, the RMSE was 3.2% (= square root of average of 3.0%^2 (CON), –4.4%^2 (LAB) and 1.5%^2 (LD)).

After the 2019 election, I warned the industry not to be complacent about their apparent success.  I pointed out that Major Polling Errors, defined to be an error of 4 percentage points or more for a particular party, were common.  Including 2024, the record of major national polling errors in the 9 elections between 1992 & 2024 now reads as –

  1. 1992 – An underestimate of 5.2% for the Conservatives.
  2. 1997 – An overestimate of 4.0% for Labour.
  3. 2010 – An overestimate of 4.1% for the Liberal Democrats.
  4. 2015 – An underestimate of 4.1% for the Conservatives.
  5. 2017 – An underestimate of 4.5% for Labour.
  6. 2024 – An overestimate of 4,4% for Labour.

Of the 3 elections that didn’t see such errors, 2001 still had a 3.5% overestimate for Labour and 1992 also had a 3.9% overestimate for Labour.  In the 12 elections prior to 1992, only 2 elections experienced major polling errors which were 1970 (Labour overestimated by 4%) and 1951 (Labour underestimated by 4.3%).  Such a shift in error rate from 2 out of 12 to 6 out of 9 is statistically significant.  Whilst I have not examined the reasons in depth, an obvious explanation on the face of it is that the move to web based polling has reduced barriers to entry for new players but at the cost of quality.

National Polling Errors by Wings – 1950 to 2024

One of the things that strikes me about the first chart above is how the errors for Labour and the Liberal Democrats are inversely correlated with each other with a correlation coefficient of -0.63.  This makes sense in today’s environment where there is a lot of talk of a progressive alliance between Labour, Lib Dems and the Greens and one can easily imagine a scenario where tactical voting means the polls overestimate Labour and underestimate the Lib Dems.  However it would seem that this has been a dynamic in British elections for a longer period of time than is realised.  The equivalent correlation between the Conservatives and Lib Dems is only -0.08.

Given this, I have redone the chart by combining the Conservatives & Reform/UKIP/Brexit Party into one group and Labour, Lib Dems & Greens into another group.  In practice, the Reform/UKIP/Brexit Party & Green errors are only known for the last 4 elections as those have been the only elections where pollsters have recorded votes for these parties separately rather than putting lumping them into Others.  So for the most part the chart below is comparing the Conservative poll error with the combined Lab/LD error.

The lines represent centred 5-election moving averages and currently sit at a 1.1% underestimate for the CON/REF/UKIP/BXP error and a 1.2% overestimate for the LAB/LD/GRN error.

The chart above shows only 6 elections out of 21 had combined errors favouring the “progressive alliance” with 4 of these taking place before 1966.  Conversely, the 7 elections between 1983 & 2017 all had errors favouring the Conservatives.

National Polling Error in CON lead over LAB – 1950 to 2024

So far I have been concentrating on the expected vote share for each party or combination of parties.  In practice, when it comes to predicting the outcome of an election that uses First Past the Post (FPTP) as its election system, the more important prediction is the Conservative lead over Labour.  These parties have always been expected to take the top two places nationally so I have calculated the expected lead from the polls and compared it with the actual lead to produce the following chart.

The latest 5-election centred moving average still shows an underestimate of 2.1% in the Conservative lead over Labour and this is close to the long run average of +1.8% since 1964.

If I define a Major Polling Error as being one where the CON–LAB lead is out by 4%, then we can make the following observations about the 21 elections since 1950:-

  1. 10 out of 21 elections did not experience a major polling error (1955, 1959, 1964, 1974F, 1979, 1983, 1987, 2005, 2010, 2019)
  2. 8 out of 21 elections experienced a major polling error favouring the Conservatives. (1966, 1970, 1974O, 1992, 1997, 2001, 2015,2024)
  3. 3 out of 21 elections experienced a major polling error favouring Labour (1950, 1951, 2017)
  4. The average polling error in CON–LAB lead is +1.8% and the standard deviation is 4.3%.
  5. If our null hypothesis is that the average polling error in CON–LAB lead is 0%, then our t-statistic is +1.91 and the p-value (using 2-tailed t-test) is 7%.
  6. If I repeat for 1974 onwards i.e. from when the Northern Ireland parties and the Nationalists arrived on the political scene and the CON+LAB vote share saw a significant shift downwards, the t-statistic is +2.36 and the p-value is 4%.

Ahead of the 2019 election, I stated that I would calculate 3 scenarios as below:

  1. A 4% underestimate in the CON–LAB lead as stated by the polls which favours the Conservatives.
  2. No error in the CON–LAB lead as stated by the polls.
  3. A 4% overestimate in the CON–LAB lead as stated by the polls which favours Labour.

I also stated that I would use a 2:1:1 ratio of these scenarios as my official forecast i.e. I would give scenario 1 50% weight, scenario 2 25% weight and scenario 3 25% weight.  This is what I ended up doing for the 2019 election and I see no reason to change this for future elections.  In the event though, my election model for 2024 was based on average vote share per seat as explained in the next section and so I did not apply the above.  However, it could be at the next election, I will revert to national vote shares in which case the above scenarios will be used.

Average Polling Error in CON lead over LAB – 1950 to 2024

Prior to the 2024 general election, I realised polls were better predictors of the CON–LAB lead based on average vote share per seat rather than national vote share.  As shown in the chart below, the average error in this estimate is essentially zero.

Whilst a better predictor in theory, that did not stop the 2024 general election experiencing the 3rd largest average polling error for CON–LAB lead with an underestimate of 5.7%.  The two largest errors were in 1992 (7% underestimate) and 2017 (6.5% overestimate).

Altogether, the standard deviation of the errors in the 12 elections before 1992 was 3.1% but for the 9 elections since 1992, it is now 4.2%.  The increase is not statistically significant but it is reasonable to proceed on the basis that major polling errors are more likely in future.

As described in my election forecasting model for 2024, I also needed estimates of the combined average vote share for CON+LAB.  For the 4 elections prior to 2024, this had been underestimated by the polls but in 2024, the polls correctly predicted the lowest ever CON+LAB average vote share.

 

— Would you like to comment on this article? —-

Please do leave your comments on this X/Twitter and/or this LinkedIn threads.

—- Want to read more about polls and polling errors? —

  1. Do pollsters show forecasting skill?
  2. Who is the most accurate pollster?
  3. The final set of polls for GE2019.

More articles on polling can be found by clicking on the Polling tab at the top of your screen.

— Subscribe to my newsletter to receive more articles like this one! —-

If you would like to receive notifications from me of news, articles and offers relating to Polling & Elections, please click here to go to my Newsletter Subscription page and tick the Surveys and/or Elections category and other categories that may be of interest to you.  You will be able to unsubscribe at anytime.