Labour have retained the West of England Combined Authority (WECA) Mayoralty As I predicted, the contest was won with less than a quarter of the vote and with a small majority (less than 3%). Unfortunately, I had the Greens winning the mayoralty so on the face of it, this is a forecasting error for me. However, when voters are as fragmented as they are today, measuring forecasting skill is not as straightforward as it might seem.
How did my forecast perform?
“…This is a genuine 5-way marginal where all five parties can have reasons to be optimistic even if the Greens are favourites.“. This quote is from my second forecast published two weeks apart (Part 1 here, Part 2 here) and I will reuse terminology from those articles.
My forecast was a simple average of 4 models, two based on changes in the polls since the 2021 mayoral election
and two based on changes in polls since the general election last year. My average predicted the Greens to win with 24.7% of the vote with Labour in 2nd place with 21.4% of the vote. In the event, Labour won with 24.97% of the vote, ahead of Reform on 22.07%.
When it comes to the winner, I clearly got it wrong. However, I did say in my final forecast “This is a true 5-way marginal which is incredibly difficult to forecast. The parties get-out-the-vote efforts will have a big say on the final outcome and could be decisive.” So even I had called it correctly (and my GE24-Diff-Excl model did), I would have had to recognise that luck could play a bigger part than forecasting skill. This is why I want to spend time exploring how one can measure forecasting skill when close contests like this one are more likely these days as is evident from the recent local election results (link opens a table of all ward level votes). I will then look at how each party performed in each of the three local authorities which make up WECA.
Measuring forecasting skill – RMSE & MAE
A commonly used metric is RMSE (Root Mean Squared Error) as used in the table above. This is calculated by first noting the difference in vote share for each party between their actual vote share and the expected vote share, squaring that difference, finding the average squared difference across the six parties and then taking the square root of the average squared difference to get the Root Mean Squared Error. The RMSE for my 2025 WECA Mayor forecast was 3.9%. For my best performing model GE24-Diff-Excl, it was 3.0% and for my worst performing model WM21-Rat-Incl it was 7.6%.
A variant you may see elsewhere is MAE (Mean Absolute Error) where instead of squaring the difference between forecast and actual vote share, you calculate the absolute difference instead which ignores whether the difference is positive of negative. You then calculate the average of the absolute differences to get the Mean Absolute Error. The MAE for my 2025 WECA Mayor forecast was 3.8%. For my best performing model GE24-Diff-Excl, it was 2.5% and for my worst performing model WM21-Rat-Incl it was 7.0%.
I have a preference for RMSE and will focus on it going forward but there is nothing wrong with MAE and most people will find it easier to calculate. Both give similar answers for the 2025 WECA Mayor election but what conclusions can I draw from them?
My definitions of Success, Minor Error and Major Error
In my forecasting track record page, I define Success for UK general elections to be when the difference between actual number of seats won and expected number seats won is less than 2%. I define a Major Error to be when this difference is greater than 4% and a Minor Error when this is between 2% and 4%. Whilst I did not explicitly restate these criteria in my 2025 WECA Mayor forecasts, I am going to reuse these limits for assessing my RMSE of 3.9%. On that basis my forecast was officially a Minor Error. Yes you might argue I am close to being a Major Error but I have track record for respecting my stated thresholds (see my 2023 Australian Voice Referendum review where my success criteria was an error of 3.0% or less and the actual error was 3.06% and thus officially an error!).
Why do I use these thresholds? The answer is I do not have access to private information held by the political parties and larger pollsters. My forecasts are based on data in the public domain, mostly voting intention polls published by pollsters. These typically have a stated 95% confidence interval of 2% to 3% for each party but history tells us polls can be biased and the historical error for some statistics (such as the Conservative lead over Labour) can be larger than this. It is also my opinion that errors on this scale changes the narratives of elections.
Issues with RMSE – Are there alternatives?
Changing narratives leads me onto an issue with RMSE. It is calculated using the errors of all parties standing in a election. Suppose for WECA in 2025, my estimated vote shares for each of the top three parties (LAB, REF, GRN) had been spot on with zero error but my estimates for the other three parties (CON, LD, IND) were +3%, +5% and -8%. Then the RMSE would be 4.0% officially making it a Major Error but I would have predicted the top three precisely and the narrative would have been the same.
This demonstrates the issue with RMSE. All parties are treated equally but narratives usually focus on the winners (Labour), runner ups (Reform) & those who over/underperformed (Greens). So perhaps an alternative in close elections is to say a forecast is a success if I predict the top three parties whose vote share will be within 5 percentage points of each other? On that basis, my forecast was a success because I correctly predicted the winner would get less than 25% of the vote (Labour got 24.97%) and the top three parties would be Greens, Labour & Reform which is what happened, just not in that order. The final gap between first place and third place was 4.93% (=24.97% – 20.04%).
In my next article, I will look use this metric (among others) when I compare my forecast with four other predictions of the 2025 WECA Mayor election. A problem with this metric though is that deciding which parties should be included and excluded from the RMSE calculation can be arbitrary such as choosing top three parties as above. What is needed is a more objective rule for including and excluding parties in an RMSE (and/or MAE) calculation.
The 1/N Rule of First Past The Post
Here’s my proposed rule. Include only those parties who in theory could have won a close election and exclude those who had no chance of winning. The 1/N rule comes from studying the mathematics of FPTP (First Past The Post) which I first wrote about 10 years ago in this article for Significance magazine.
What is the minimum number of votes you need to win an FPTP election. Suppose you have 1000 voters and 10 candidates. If candidate A gets 99 votes then there is no way they can win because there will be at least one candidate with 100+ votes. You know this because the most equal outcome is every candidate getting 100 (=1000/100) votes, a 10-way tie. To break the tie, one candidate needs 101 votes which then means one of the other nine candidates is getting 99 votes and is therefore out of the running. On the other hand, if you get 101 votes, you at least have a theoretical chance of winning which will be small if there are two candidates well ahead but your chances are higher if it is a close contest like it was with the WECA Mayor vote.
This is the 1/N rule or 100%/N rule if you prefer. N is the number of candidates standing and so in the case of WECA, the minimum vote share needed to have a chance of winning was 1/6 of the vote or 16.667%. In my forecast, I expected 4 parties (GRN, LAB, REF & LD) to be over this threshold, so perhaps these are the parties whose performance I should be evaluated on? The alternative is to look at the result instead and say only 3 parties (GRN, LAB & REF) were over this threshold and evaluate my forecast for these three parties only. You will see how this works out in my next article where I compare my forecast with four other predictions.
There are variants on this rule. One is the (1-d)/R rule where R is the number of candidates who Retain their deposits (vote share of 5% or more) and d is the collective vote share of those losing their deposits. The rationale behind this alternative is sometimes you have a lot of fringe and joke candidates who are there for the publicity e.g. Rishi Sunak’s seat at the last general election where N=13 candidates stood but only R=4 retained their deposit with the other 9 losing their deposits accounting for a total of d=0.065 or 6.5% of the vote. In WECA this year, the Independent candidate was the only one to lose his deposit so the revised threshold would be (1 – 0.0228)/5 = 19.54%. Had I used this threshold, then my forecast would have only predicted two possible winners of a close contest, Greens and Labour.
How did my forecast perform by Council?
WECA is a Combined Authority covering three local authorities, B&NES, Bristol & S.Glocs. 
Here are the results by council followed by the errors in my forecast.
One thing I did get right was the turnout in B&NES and S.Glocs. For both councils, I assumed the 2025 turnout as a percentage of the 2024 general election turnout would be equal to the 2021 turnout as a percentage of the 2019 general election turnout. This percentage was 44% in both councils in 2021.
For Bristol, I did not do this because 2021 turnout was inflated by city council elections taking place at the same time. As can be seen from the error table, my predicted turnout in Bristol was too high. I had assumed 2025 turnout would be 54% of the 2024 general election but had I instead used the 2017 turnout, which was 47% of the 2015 general election turnout, I would have been much closer.
From the errors shown in the table, can I now draw conclusions about which parties under or over performed? The answer is no because my model was close to being a major error. If I had a better forecast I could do this.
Was there a better model I could have used?
Yes there was; GE24-Diff-Incl applied to each local authority combined with the assumption that Bristol’s turnout would be similar to 2017 rather than 2021. This is in fact the simplest model I could have come up as all it does is add the change in vote share as measured by the latest polls for each party since the general election last year for each council separately before combining the results. The RMSE across all six parties is 2.3% compared to 3.9% for my actual forecast. All parties bar Reform are predicted to be within 2 percentage points. The full set of tables by local authority detailing 2021 & 2025 actuals, 2025 forecast and errors are shown below.
I think the takeout from these tables is fascinating. In part 1 of my forecast, I stated I lacked on the ground information as to how voters saw the Greens given they had been running Bristol council for the last year. Had this put Bristol voters off from voting Green? The answer seems to be yes since the Greens were 1 percentage point above the best model forecast in B&NES and S.Glocs but 4 percentage points behind expectations in Bristol. That adds up to the Greens underperforming by 5 points as a result of Bristol voters experiencing life under the Greens for a year.
The incumbency disillusionment seems to have repeated itself with the Lib Dems in B&NES and S.Glocs. In both authorities, they were 7 points under the best model forecast but in Bristol they were 3 points above. Since 2023, the Lib Dems have had majority control of B&NES and are in coalition with Labour in S. Glocs. Again, it would appear life under the Lib Dems for two years is not conducive to voting for them.
The best model gets Labour spot on in Bristol & S.Glocs and they seem to be the beneficiaries of the Lib Dems underperformance in B&NES. It is the reverse for Reform where Aaron Banks appears to have performed extremely well in S.Glocs which is his home council. It is worth noting all four parties who stood in 2021 (CON, LAB, LD, GRN) ended up with lower vote shares in 2025. That is the effect of Reform‘s entry into the race. WECA is not natural Reform territory but they came with 3 percentage points of winning it.
My learnings from my 2025 WECA Mayor forecast
I was always clear that forecasting a close election like this one was going to be hard and so it proved. However, it is surprising the best model turned out to be the simplest possible model that was available to me. I take that is a warning I may have overcomplicated my thinking. In my defence there were a number of factors I had to disentangle such as the change of voting system and the collapse of the Conservatives last year.
Close results like this look like staying with us for the next few years. That makes it harder to judge forecasting skill on the basis of how many times you identify the winner under FPTP. This forced me to rethink how to measure forecasting skill and my recommendation is to take into account the 1/N rule of FPTP when deciding on which parties to evaluate when assessing forecasting accuracy. My next article will demonstrate this by looking at my forecast of the 2025 WECA Mayor election alongside four other forecasts.
— Would you like to comment on this article? —-
Please do leave your comments on this X/Twitter thread.
You can find the X/Twitter thread for my earlier forecast here and the X/Twitter thread for my final forecast here.
— Subscribe to my newsletter to receive more articles like this one! —-
If you would like to receive notifications from me of news, articles and offers relating to Elections & Polling, please click here to go to my Newsletter Subscription page and tick the Elections and/or Surveys category and other categories that may be of interest to you. You will be able to unsubscribe at anytime.
— Read some of my other blog posts on elections —
More posts can be found by clicking on the Elections tab at the top of your screen.
