{"id":6255,"date":"2025-05-06T23:28:43","date_gmt":"2025-05-06T22:28:43","guid":{"rendered":"https:\/\/marriott-stats.com\/nigels-blog\/?p=6255"},"modified":"2025-05-10T18:52:27","modified_gmt":"2025-05-10T17:52:27","slug":"weca-mayor-2025-election-forecast-review","status":"publish","type":"post","link":"https:\/\/marriott-stats.com\/nigels-blog\/weca-mayor-2025-election-forecast-review\/","title":{"rendered":"UK Local Elections #2C – 2025 West England Mayoral Election – Review of My Forecast"},"content":{"rendered":"

Labour have retained the West of England Combined Authority (WECA<\/strong>) Mayoralty<\/a>\u00a0 As I predicted, the contest was won with less than a quarter of the vote and with a small majority (less than 3%).\u00a0 Unfortunately, I had the Greens<\/strong> <\/span>winning the mayoralty so on the face of it, this is a forecasting error for me.\u00a0 However, when voters are as fragmented as they are today, measuring forecasting skill is not as straightforward as it might seem.<\/p>\n

 <\/p>\n

<\/p>\n

How did my forecast perform?<\/strong><\/span><\/h5>\n

“…This is a genuine 5-way marginal where all five parties can have reasons to be optimistic even if the Greens<\/span><\/strong> are favourites.<\/em><\/span>“.\u00a0 This quote is from my second forecast published two weeks apart (Part 1 here<\/a>, Part 2 here<\/a>)<\/strong> and I will reuse terminology from those articles.<\/p>\n

My forecast was a simple average of 4<\/strong> models, two based on changes in the polls since the 2021 mayoral election\"\" and two based on changes in polls since the general election last year.\u00a0 My average predicted the Greens<\/strong><\/span> to win with 24.7%<\/strong><\/span> of the vote with Labour<\/strong><\/span> in 2nd place with 21.4%<\/strong><\/span> of the vote.\u00a0 In the event, Labour<\/span><\/strong> won with 24.97%<\/span><\/strong> of the vote, ahead of Reform<\/strong><\/span> on 22.07%<\/strong><\/span>.<\/p>\n

When it comes to the winner, I clearly got it wrong.\u00a0 However, I did say in my final forecast “This is a true 5-way marginal which is incredibly difficult to forecast.\u00a0 The parties get-out-the-vote efforts will have a big say on the final outcome and could be decisive.<\/em><\/span>”\u00a0 So even I had called it correctly (and my GE24-Diff-Excl<\/strong><\/span> model did), I would have had to recognise that luck could play a bigger part than forecasting skill.\u00a0 This is why I want to spend time exploring how one can measure forecasting skill when close contests like this one are more likely these days as is evident from the recent local election results (link opens a table of all ward level votes)<\/a>.\u00a0 I will then look at how each party performed in each of the three local authorities which make up WECA.<\/p>\n

 <\/p>\n

Measuring forecasting skill – RMSE & MAE<\/strong><\/span><\/h5>\n

A commonly used metric is RMSE<\/strong> (Root Mean Squared Error) as used in the table above.\u00a0 This is calculated by first noting the difference in vote share for each party between their actual vote share and the expected vote share, squaring that difference, finding the average squared difference across the six parties and then taking the square root of the average squared difference to get the Root Mean Squared Error.\u00a0 The RMSE<\/strong> for my 2025 WECA<\/strong> Mayor forecast was 3.9%.\u00a0 <\/strong>For my best performing model GE24-Diff-Excl<\/strong><\/span>, it was 3<\/strong>.0%<\/strong><\/span> and for my worst performing model WM21-Rat-Incl<\/strong><\/span> it was 7.6%<\/span>.<\/strong><\/p>\n

A variant you may see elsewhere is MAE<\/strong> (Mean Absolute Error) where instead of squaring the difference between forecast and actual vote share, you calculate the absolute difference instead which ignores whether the difference is positive of negative.\u00a0 You then calculate the average of the absolute differences to get the Mean Absolute Error.\u00a0 The MAE<\/strong> for my 2025 WECA Mayor forecast was 3.8%.\u00a0 <\/strong>For my best performing model GE24-Diff-Excl<\/span><\/strong>, it was 2.5%<\/span><\/strong> and for my worst performing model WM21-Rat-Incl<\/span><\/strong> it was 7.0%<\/span><\/strong>.<\/p>\n

I have a preference for RMSE<\/strong> and will focus on it going forward but there is nothing wrong with MAE<\/strong> and most people will find it easier to calculate.\u00a0 Both give similar answers for the 2025 WECA<\/strong> Mayor election but what conclusions can I draw from them?<\/p>\n

 <\/p>\n

My definitions of Success, Minor Error and Major Error<\/strong><\/span><\/h5>\n

In my forecasting track record page<\/a>, I define Success<\/strong> for UK general elections to be when the difference between actual number of seats won and expected number seats won is less than 2%<\/strong>. I define a Major Error<\/span><\/strong> to be when this difference is greater than 4%<\/strong><\/span> and a Minor Error<\/span><\/strong> when this is between 2% and 4%<\/strong><\/span>.\u00a0 Whilst I did not explicitly restate these criteria in my 2025 WECA<\/strong> Mayor forecasts, I am going to reuse these limits for assessing my RMSE<\/strong> of 3.9%<\/strong>.\u00a0 On that basis my forecast was officially a Minor Error<\/strong>.<\/span>\u00a0 Yes you might argue I am close to being a Major Error<\/strong> <\/span>but I have track record for respecting my stated thresholds (see my 2023 Australian Voice Referendum review<\/a> where my success criteria was an error of 3.0%<\/strong> or less and the actual error was 3.06%<\/strong> and thus officially an error!).<\/p>\n

Why do I use these thresholds?\u00a0 The answer is I do not have access to private information held by the political parties and larger pollsters.\u00a0 My forecasts are based on data in the public domain, mostly voting intention polls published by pollsters.\u00a0 These typically have a stated 95% confidence interval of 2% to 3%<\/strong> for each party but history tells us polls can be biased and the historical error for some statistics (such as the Conservative lead over Labour<\/a>) can be larger than this.\u00a0 It is also my opinion that errors on this scale changes the narratives of elections.<\/p>\n

 <\/p>\n

Issues with RMSE – Are there alternatives?<\/strong><\/span><\/h5>\n

Changing narratives leads me onto an issue with RMSE<\/strong>.\u00a0 It is calculated using the errors of all parties standing in a election.\u00a0 Suppose for WECA<\/strong> in 2025, my estimated vote shares for each of the top three parties (LAB<\/strong><\/span>, REF<\/strong><\/span>, GRN<\/strong><\/span>) had been spot on with zero error<\/strong> but my estimates for the other three parties (CON<\/strong><\/span>, LD<\/span><\/strong>, IND<\/strong>) were +3%<\/strong>, +5%<\/strong> and -8%<\/strong>.\u00a0 Then the RMSE would be 4.0%<\/strong> officially making it a Major Error<\/strong><\/span> but I would have predicted the top three precisely and the narrative would have been the same.<\/p>\n

This demonstrates the issue with RMSE.<\/strong>\u00a0 All parties are treated equally but narratives usually focus on the winners (Labour<\/strong><\/span>), runner ups (Reform<\/strong><\/span>) & those who over\/underperformed (Greens<\/strong><\/span>).\u00a0 So perhaps an alternative in close elections is to say a forecast is a success if I predict the top three parties whose vote share will be within 5<\/strong> percentage points of each other?\u00a0 On that basis, my forecast was a success because I correctly predicted the winner would get less than 25%<\/strong> of the vote (Labour<\/strong> <\/span>got 24.97%<\/strong><\/span>) and the top three parties would be Greens<\/strong><\/span>, Labour<\/strong><\/span>\u00a0& Reform<\/strong><\/span> which is what happened, just not in that order.\u00a0 The final gap between first place and third place was 4.93%<\/strong> (=24.97% – 20.04%).<\/p>\n

In my next article, I will look use this metric (among others) when I compare my forecast with four other predictions of the 2025 WECA<\/strong> Mayor election.\u00a0 A problem with this metric though is that deciding which parties should be included and excluded from the RMSE<\/strong> calculation can be arbitrary such as choosing top three parties as above.\u00a0 What is needed is a more objective rule for including and excluding parties in an RMSE<\/strong> (and\/or MAE<\/strong>) calculation.<\/p>\n

 <\/p>\n

The 1\/N Rule of First Past The Post<\/span><\/strong><\/h5>\n

Here’s my proposed rule.\u00a0 Include only those parties who in theory could have won a close election and exclude those who had no chance of winning.\u00a0 The 1\/N<\/strong> rule comes from studying the mathematics of FPTP<\/strong> (First Past The Post) which I first wrote about 10 years ago in this article for Significance magazine<\/a>.<\/p>\n

What is the minimum number of votes you need to win an FPTP<\/strong> election.\u00a0 Suppose you have 1000<\/strong> voters and 10<\/strong> candidates.\u00a0 If candidate A gets 99<\/strong> votes then there is no way they can win because there will be at least one candidate with 100+<\/strong> votes.\u00a0 You know this because the most equal outcome is every candidate getting 100<\/strong> (=1000\/100) votes, a 10-way tie.\u00a0 To break the tie, one candidate needs 101<\/strong> votes which then means one of the other nine candidates is getting 99<\/strong> votes and is therefore out of the running.\u00a0 On the other hand, if you get 101<\/strong> votes, you at least have a theoretical chance of winning which will be small if there are two candidates well ahead but your chances are higher if it is a close contest like it was with the WECA<\/strong> Mayor vote.<\/p>\n

This is the 1\/N<\/strong> rule or 100%\/N<\/strong> rule if you prefer.\u00a0 N<\/strong> is the number of candidates standing and so in the case of WECA<\/strong>, the minimum vote\u00a0share needed to have a chance of winning was 1\/6<\/strong> of the vote or 16.667%<\/strong>.\u00a0 In my forecast, I expected 4<\/strong> parties (GRN<\/span>, LAB<\/span>, REF<\/span> & LD<\/span><\/strong>) to be over this threshold, so perhaps these are the parties whose performance I should be evaluated on?\u00a0 The alternative is to look at the result instead and say only 3<\/strong> parties (GRN<\/span>, LAB<\/span> & REF<\/span><\/strong>) were over this threshold and evaluate my forecast for these three parties only.\u00a0 You will see how this works out in my next article where I compare my forecast with four other predictions.<\/p>\n

\"\"<\/p>\n

There are variants on this rule.\u00a0 One is the (1-d)\/R<\/strong> rule where R<\/strong> is the number of candidates who Retain<\/strong> their deposits (vote share of 5%<\/strong> or more) and d<\/strong> is the collective vote share of those losing their d<\/strong>eposits.\u00a0 The rationale behind this alternative is sometimes you have a lot of fringe and joke candidates who are there for the publicity e.g. Rishi Sunak’s seat at the last general election where N=13<\/strong> candidates stood but only R=4<\/strong> retained their deposit with the other 9<\/strong> losing their deposits accounting for a total of d=0.065<\/strong> or 6.5% of the vote.\u00a0 In WECA<\/strong> this year, the Independent<\/strong> candidate was the only one to lose his deposit so the revised threshold would be (1<\/strong> – 0.0228<\/strong>)\/5<\/strong> = 19.54%<\/strong>.\u00a0 Had I used this threshold, then my forecast would have only predicted two possible winners of a close contest, Greens<\/strong> <\/span>and Labour<\/span><\/strong>.<\/p>\n

 <\/p>\n

How did my forecast perform by Council?<\/strong><\/span><\/h5>\n

WECA<\/strong> is a Combined Authority covering three local authorities, B&NES, Bristol<\/strong> & S.Glocs<\/strong>.\u00a0 \"\"Here are the results by council followed by the errors in my forecast.<\/p>\n

One thing I did get right was the turnout in B&NES and S.Glocs.\u00a0 For both councils, I assumed the 2025 turnout as a percentage of the 2024 general election turnout would be equal to the 2021 turnout as a percentage of the 2019 general election turnout.\u00a0 This percentage was 44%<\/strong> in both councils in 2021.<\/p>\n

For Bristol, I did not do this because 2021 turnout was inflated by city council elections taking place at the same time.\u00a0 As can be seen from the error table, my predicted turnout in Bristol was too high.\u00a0 I had assumed 2025 turnout would be 54%<\/strong> of the 2024 general election but had I instead used the 2017 turnout, which was 47%<\/strong> of the 2015 general election turnout, I would have been much closer.<\/p>\n

From the errors shown in the table, can I now draw conclusions about which parties under or over performed?\u00a0 The answer is no because my model was close to being a major error.\u00a0 If I had a better forecast I could do this.<\/p>\n

 <\/p>\n

Was there a better model I could have used?<\/strong><\/span><\/h5>\n

Yes there was; GE24-Diff-Incl<\/strong> <\/span>applied to each local authority combined with the assumption that Bristol’s turnout would be similar to 2017 rather than 2021.\u00a0 This is in fact the simplest model I could have come up as all it does is add the change in vote share as measured by the latest polls for each party since the general election last year for each council separately before combining the results.\u00a0 The RMSE<\/strong> across all six parties is 2.3%<\/strong> compared to 3.9%<\/strong> for my actual forecast.\u00a0 All parties bar Reform<\/span><\/strong> are predicted to be within 2<\/strong> percentage points.\u00a0 The full set of tables by local authority detailing 2021 & 2025 actuals, 2025 forecast and errors are shown below.<\/p>\n

\"\"<\/p>\n

I think the takeout from these tables is fascinating.\u00a0 In part 1 of my forecast, I stated I lacked on the ground information as to how voters saw the Greens<\/strong><\/span> given they had been running Bristol council for the last year.\u00a0 Had this put Bristol voters off from voting Green?<\/strong><\/span>\u00a0 The answer seems to be yes since the Greens<\/strong> were 1<\/strong> percentage point above the best model forecast in B&NES and S.Glocs but 4<\/strong> percentage points behind expectations in Bristol.\u00a0 That adds up to the Greens<\/strong> underperforming by 5 points<\/strong> as a result of Bristol voters experiencing life under the Greens for a year.<\/p>\n

The incumbency disillusionment seems to have repeated itself with the Lib Dems<\/strong><\/span> in B&NES and S.Glocs.\u00a0 In both authorities, they were 7<\/strong> points under the best model forecast but in Bristol they were 3<\/strong> points above.\u00a0 Since 2023, the Lib Dems have had majority control of B&NES and are in coalition with Labour in S. Glocs.\u00a0 Again, it would appear life under the Lib Dems for two years is not conducive to voting for them.<\/p>\n

The best model gets Labour<\/strong><\/span> spot on in Bristol & S.Glocs and they seem to be the beneficiaries of the Lib Dems<\/strong><\/span> underperformance in B&NES.\u00a0 It is the reverse for Reform<\/strong><\/span> where Aaron Banks appears to have performed extremely well in S.Glocs which is his home council.\u00a0 It is worth noting all four parties who stood in 2021 (CON<\/strong><\/span>, LAB<\/span><\/strong>, LD<\/strong><\/span>, GRN<\/strong><\/span>) ended up with lower vote shares in 2025.\u00a0 That is the effect of Reform<\/strong><\/span>‘s entry into the race.\u00a0 WECA<\/strong><\/span> is not natural Reform<\/span> territory but they came with 3<\/strong> percentage points of winning it.<\/p>\n

 <\/p>\n

My learnings from my 2025 WECA Mayor forecast<\/strong><\/span><\/h5>\n

I was always clear that forecasting a close election like this one was going to be hard and so it proved.\u00a0 However, it is surprising the best model turned out to be the simplest possible model that was available to me.\u00a0 I take that is a warning I may have overcomplicated my thinking.\u00a0 In my defence there were a number of factors I had to disentangle such as the change of voting system and the collapse of the Conservatives<\/strong><\/span> last year.<\/p>\n

Close results like this look like staying with us for the next few years.\u00a0 That makes it harder to judge forecasting skill on the basis of how many times you identify the winner under FPTP<\/strong>.\u00a0 This forced me to rethink how to measure forecasting skill and my recommendation is to take into account the 1\/N<\/strong> rule of FPTP<\/strong> when deciding on which parties to evaluate when assessing forecasting accuracy.\u00a0 My next article will demonstrate this by looking at my forecast of the 2025 WECA<\/strong> Mayor election alongside four other forecasts.<\/p>\n

 <\/p>\n

— Would you like to comment on this article? —-<\/span><\/strong><\/h5>\n

Please do leave your comments on this X\/Twitter thread<\/strong><\/span><\/a>.<\/p>\n

You can find the X\/Twitter<\/strong><\/span> thread for my earlier forecast here<\/a> and the X\/Twitter<\/strong><\/span> thread for my final forecast here<\/a>.<\/p>\n

— Subscribe to my newsletter to receive more articles like this one! —-<\/span><\/strong><\/h5>\n

If you would like to receive notifications from me of news, articles and offers relating to Elections & Polling, please click here to go to my Newsletter Subscription page<\/a><\/strong><\/span> and tick the Elections and\/or Surveys category and other categories that may be of interest to you.\u00a0 You will be able to unsubscribe at anytime.<\/p>\n

— Read some of my other blog posts on elections —<\/span><\/strong><\/h5>\n

More posts can be found by clicking on the Elections<\/a> tab at the top of your screen.<\/p>\n","protected":false},"excerpt":{"rendered":"

Labour have retained the West of England Combined Authority (WECA) Mayoralty\u00a0 As I predicted, the contest was won with less than a quarter of the vote and with a small majority (less than 3%).\u00a0 Unfortunately, I had the Greens winning the mayoralty so on the face of it, this is a forecasting error for me.\u00a0 […]<\/p>\n","protected":false},"author":3,"featured_media":6271,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_genesis_hide_title":false,"_genesis_hide_breadcrumbs":false,"_genesis_hide_singular_image":false,"_genesis_hide_footer_widgets":false,"_genesis_custom_body_class":"","_genesis_custom_post_class":"","_genesis_layout":"","footnotes":""},"categories":[2,6,3],"tags":[232,229,228,214,21,338,225,213,218,224,210,231,230,233,227,226],"class_list":{"0":"post-6255","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-elections","8":"category-forecasting","9":"category-polling","10":"tag-bnes","11":"tag-bath","12":"tag-bristol","13":"tag-conservative","14":"tag-election-forecasting","15":"tag-fptp","16":"tag-greens","17":"tag-labour","18":"tag-lib-dems","19":"tag-local-election","20":"tag-majority","21":"tag-metro-mayor","22":"tag-south-gloucs","23":"tag-supplementary-vote","24":"tag-weca","25":"tag-west-england","26":"entry","27":"override"},"_links":{"self":[{"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/posts\/6255","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/comments?post=6255"}],"version-history":[{"count":18,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/posts\/6255\/revisions"}],"predecessor-version":[{"id":6284,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/posts\/6255\/revisions\/6284"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/media\/6271"}],"wp:attachment":[{"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/media?parent=6255"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/categories?post=6255"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/tags?post=6255"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}