{"id":1004,"date":"2018-03-31T21:53:40","date_gmt":"2018-03-31T20:53:40","guid":{"rendered":"https:\/\/marriott-stats.com\/nigels-blog\/?p=1004"},"modified":"2018-03-31T21:53:40","modified_gmt":"2018-03-31T20:53:40","slug":"uk-general-elections-3-who-is-the-most-accurate-pollster","status":"publish","type":"post","link":"https:\/\/marriott-stats.com\/nigels-blog\/uk-general-elections-3-who-is-the-most-accurate-pollster\/","title":{"rendered":"UK General Elections #3 &#8211; Who is the most accurate pollster?"},"content":{"rendered":"<p>The last 3 general elections have seen some significant polling errors.\u00a0 In 2010, the Lib Dems were significantly overestimated, in 2015 the Conservatives were underestimated and last year saw <a href=\"https:\/\/marriott-stats.com\/nigels-blog\/uk-general-elections-2-how-accurate-are-the-opinion-poll-updated-with-ge17\/\" target=\"_blank\" rel=\"noopener\">the largest ever underestimate in the Labour vote<\/a>.\u00a0 Whilst these errors suggest that the polling industry is struggling with general elections these days, a natural question to ask is &#8220;are all pollsters equally bad or are some better than others?&#8221;<\/p>\n<p><!--more-->My answer to this question is to look at all polls undertaken in the week before the 2010, 2015 and 2017 UK general elections using polling data provided\u00a0by <a href=\"http:\/\/www.markpack.org.uk\/opinion-polls\/\" target=\"_blank\" rel=\"noopener noreferrer\">Mark Pack who has systematically recorded every opinion poll published since 1945<\/a>. \u00a0 A total of 50 polls were used with fieldwork dates in the following date ranges:<\/p>\n<ul>\n<li>2010 &#8211; 13 polls between 1st &amp; 5th May.<\/li>\n<li>2015 &#8211; 18 polls between 1st &amp; 7th May<\/li>\n<li>2017 &#8211; 19 polls between 1st &amp; 7th June<\/li>\n<\/ul>\n<p>For 2010, I used the last 5 days worth of polls because there was a noticeable shift in the last few days with the Lib Dems coming off their highs in the 2010 campaign due to &#8220;Cleggmania&#8221;.\u00a0 There were no noticeable shifts in the last week of the 2015 &amp; 2017 campaigns.<\/p>\n<p>Obviously I need to devise a way to measure the accuracy of the polls and a common metric is the root mean square error.\u00a0 This is calculated by taking the difference between the estimated and actual vote shares i.e. the ERROR, SQUARE it, calculate the MEAN squared error across the parties taking part and then take the square ROOT of the mean squared error.\u00a0 For example, the last poll in the 2010 campaign was undertaken by IpsosMORI and they had errors of -1%, -1% and +3% for the Conservatives, Labour and Liberal Democrats respectively i.e. they underestimated the Conservative and Labour vote and overestimated the Lib Dem vote.\u00a0 The sum of the squared errors would be 0.01^2 + 0.01^2 + 0.03^2 and the root mean squared error is 0.021 or 2.1%.<\/p>\n<p>This might be the obvious way to do it but there is a flaw which can be illustrated using 2015 as an example where no pollster was close.\u00a0 The final vote share in Great Britain (pollsters do not survey Northern Ireland) was 38% for the Conservatives and 31% for Labour, a lead of 7 points.\u00a0 Suppose we have two pollsters A &amp; B as follows:<\/p>\n<ul>\n<li>Pollster A &#8211; CON 35 LAB 34<\/li>\n<li>Pollster B &#8211; CON 35 LAB 28<\/li>\n<\/ul>\n<p>Both pollsters have a 3 point underestimate in the CON vote and 3 point errors in the LAB vote but pollster A has a 3 point overestimate whilst pollster B has a 3 point underestimate.\u00a0 Using RMSE, both pollsters would have the same value of 3% but there is no question that they are giving different narratives to the public.\u00a0 Pollster A says the parties are neck and neck which would imply a hung parliament whilst pollster B says the Conservatives have a 7 point lead which could result in a small majority which is exactly what happened in 2015.\u00a0 Under the First Past the Post voting system it is the lead between the two main parties that determines the number of seats and so there is no question that pollster B is more accurate.\u00a0 So the correct way to look at the poll estimates is to look at these values instead<\/p>\n<ul>\n<li>Pollster A &#8211; CON 35, CON-LAB lead +1<\/li>\n<li>Pollster B &#8211; CON 35, CON-LAB lead +7<\/li>\n<\/ul>\n<p>which give RMSE of 2.1% for pollster B and 4.7% for pollster A.<\/p>\n<p>With this logic in mind, I have chosen to calculate the RMSE for each pollster by using the following values:<\/p>\n<ol>\n<li>Error in the Conservative vote share<\/li>\n<li>Error in the Conservative lead over Labour<\/li>\n<li>Error in the Labour lead over the Lib Dems<\/li>\n<li>Error in the Labour lead over UKIP (for 2015 only)<\/li>\n<\/ol>\n<p>So RMSE was based on 3 values in 2010 &amp; 2017 and 4 values for 2015.\u00a0 I feel these values determine the na<img loading=\"lazy\" decoding=\"async\" class=\"alignright wp-image-1006\" style=\"margin: 8px 0px 8px 16px;height: 384px;text-align: left;color: #333333;text-indent: 0px;letter-spacing: normal;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;text-decoration: none;max-width: 920.57px;float: right;background-color: transparent\" src=\"https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2018\/03\/PollErrors4-257x300.png\" alt=\"\" width=\"329\" height=\"300\" \/>rratives of the campaign and the best pollster will be the one with the lowest RMSE.\u00a0 Using these, can we now say who is the best pollster?<\/p>\n<p>First, I decided to list the top and bottom 3 individual polls based on RMSE as shown in the table.\u00a0 Some pollsters carried out more than one poll in the last week of the campaign so it is possible for them to appear more than once.\u00a0 Indeed in 2010, YouGov had two of the worst polls and the best poll whilst Comres had two of the best polls and one of the worst polls.\u00a0 Note that I have only included in this table pollsters who have undertaken polls in at least two elections.<\/p>\n<p>What does the table show us?\u00a0 The most striking fact for me is that Survation had the most accurate poll in 2017 and the most inaccurate poll in 2015.\u00a0 I think this point is not widely understood.\u00a0 Just because a pollster happens to be the most accurate in one election, does not mean they will be the best next time around.\u00a0 Indeed Comres who came closest in 2015 had one of the worst polls in 2017.<\/p>\n<p>Not all errors are equal though.\u00a0 You can see that Survations&#8217; RMSE in 2015 was +5.5% whereas YouGov had the worst poll in 2010 but only with an RMSE of 1.9%.\u00a0 Pollsters often tell us that they aim for errors of less than 3 points so I have chosen to use that as my criteria.\u00a0 Consequently any poll with an RMSE over 3% is coloured red in the WORST column.\u00a0 I should have done the same with the BEST column since the 2nd and 3rd best polls in 2015 actually had RMSEs over 3% which just goes to show how bad the 2015 performance was.<\/p>\n<p>These are of course individual polls and if we want to identify the best pollster, we really should look at all the polls they published in the last week of the campaigns.\u00a0 The next table shows the number of polls undertaken by each pollster in the last week of each election and the average RMSE across those polls.\u00a0 The last column is a straight average of the RMSE for each election i.e. I have not taken into account how many polls were undertaken in each <img loading=\"lazy\" decoding=\"async\" class=\" wp-image-1007 alignleft\" src=\"https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2018\/03\/PollErrors5-300x239.png\" alt=\"\" width=\"413\" height=\"329\" srcset=\"https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2018\/03\/PollErrors5-300x239.png 300w, https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2018\/03\/PollErrors5-439x350.png 439w, https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2018\/03\/PollErrors5.png 497w\" sizes=\"auto, (max-width: 413px) 100vw, 413px\" \/>campaign which I feel is fairer as a pollster who did 4 polls in 2015 and 1 in 2010 would be treated more favourably than a pollster who did 4 in 2010 and 1 in 2015.<\/p>\n<p>The pollsters have been split into two groups; those who polled in all 3 elections and those who polled in only 2 elections such as Survation.\u00a0 Those who only did 1 election are not shown here but they are included in the overall average in the bottom row.\u00a0 I should point out that some pollsters may have changed their names or been bought out so it is possible that two apparently different pollsters are in fact the same company.<\/p>\n<p>You can see that I have used a colour coding system for the RMSE.\u00a0 Interestingly, it appears that the established companies who did all 3 elections are better on average than the newcomers.\u00a0 Comres come out on top and none of their RMSE exceed my criteria of 3% though one might notice an apparent trend for the worse and indeed one of their polls was one of the worst in 2017.\u00a0 ICM struggled with 2015 in particular and whilst 2017 was better, the RMSE was still large.<\/p>\n<p>From the newcomers, Survation took the plaudits in 2017 but they were a particularly poor performer in 2015.\u00a0 This raises the question, did they genuinely learn from their mistakes and make the correct improvements or were they simply lucky?\u00a0 We will find out in the next general election!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The last 3 general elections have seen some significant polling errors.\u00a0 In 2010, the Lib Dems were significantly overestimated, in 2015 the Conservatives were underestimated and last year saw the largest ever underestimate in the Labour vote.\u00a0 Whilst these errors suggest that the polling industry is struggling with general elections these days, a natural question [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":1006,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_genesis_hide_title":false,"_genesis_hide_breadcrumbs":false,"_genesis_hide_singular_image":false,"_genesis_hide_footer_widgets":false,"_genesis_custom_body_class":"","_genesis_custom_post_class":"","_genesis_layout":"","footnotes":""},"categories":[2,6,3],"tags":[19,43,33],"class_list":{"0":"post-1004","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-elections","8":"category-forecasting","9":"category-polling","10":"tag-elections","11":"tag-general-election-2017","12":"tag-opinion-polls","13":"entry","14":"override"},"_links":{"self":[{"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/posts\/1004","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/comments?post=1004"}],"version-history":[{"count":3,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/posts\/1004\/revisions"}],"predecessor-version":[{"id":1011,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/posts\/1004\/revisions\/1011"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/media\/1006"}],"wp:attachment":[{"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/media?parent=1004"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/categories?post=1004"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/tags?post=1004"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}