{"id":680,"date":"2017-07-02T14:19:12","date_gmt":"2017-07-02T13:19:12","guid":{"rendered":"https:\/\/marriott-stats.com\/nigels-blog\/?p=680"},"modified":"2017-07-02T18:05:09","modified_gmt":"2017-07-02T17:05:09","slug":"uk-general-elections-2-how-accurate-are-the-opinion-poll-updated-with-ge17","status":"publish","type":"post","link":"https:\/\/marriott-stats.com\/nigels-blog\/uk-general-elections-2-how-accurate-are-the-opinion-poll-updated-with-ge17\/","title":{"rendered":"UK General Elections #2 &#8211; How accurate are the opinion polls? &#8211; updated with GE17"},"content":{"rendered":"<p>Ahead of the 2017 general election, I\u00a0predicted that the\u00a0opinion polls would be wrong again and that<a href=\"https:\/\/marriott-stats.com\/nigels-blog\/uk-general-elections-1-how-accurate-are-the-opinion-polls\/\"> the Conservatives lead over Labour would be underestimated by 2.6%<\/a>.\u00a0 \u00a0I\u00a0based this on\u00a0data provided\u00a0by <a href=\"http:\/\/www.markpack.org.uk\/opinion-polls\/\" target=\"_blank\" rel=\"noopener noreferrer\">Mark Pack who has systematically recorded every opinion poll published since 1945<\/a>.\u00a0 In the event, I was right that the polls would be wrong but instead of an error favouring the Conservatives, the polls recorded the largest ever underestimate of the Labour vote.\u00a0 As a result, election forecasters were\u00a0blindsided yet again and the result was a hung parliament which few saw coming.<\/p>\n<p><!--more--><\/p>\n<p>This post analyses only the polls that\u00a0took place in the week before every general election between 1950 &amp; 2017.\u00a0 Note this is based on the fieldwork dates, not the publication date which can be a few days later.\u00a0 For the elections in the 1950&#8217;s, Gallup were the only pollster so instead of using the week before, I used the month before.<\/p>\n<p>For each election, I calculated the average vote share recorded across all pollsters for the Conservatives, Labour and Liberal Democrats (Liberals 1950 to 1979, Alliance 1983 &amp; 1987).\u00a0 The polling error for each party can then be calculated as the actual election result for Great Britain minus the average vote share from the polls.\u00a0 I use the figures for Great Britain rather than the United Kingdom since nearly all polls do not survey in Northern Ireland.<\/p>\n<p>The results can be seen in the chart below.\u00a0 The black line is the number of polling companies that polled in the run up to the election.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-684 alignnone\" src=\"https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2017\/07\/UpdatedPollErrors2-300x141.png\" alt=\"\" width=\"732\" height=\"344\" srcset=\"https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2017\/07\/UpdatedPollErrors2-300x141.png 300w, https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2017\/07\/UpdatedPollErrors2-768x361.png 768w, https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2017\/07\/UpdatedPollErrors2-450x212.png 450w, https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2017\/07\/UpdatedPollErrors2.png 1002w\" sizes=\"auto, (max-width: 732px) 100vw, 732px\" \/><\/p>\n<p>The 2017 general election saw the highest ever underestimate of the Labour vote.\u00a0 At 4.5% this exceeded the 4.3% underestimate in the 1951 general election when Gallup was effectively the only pollster.\u00a0 The Conservative and Liberal Democrat vote shares were spot on.\u00a0 Unlike 2015 when all pollsters got it wrong, some pollsters did get Labour right in 2017 notably Survation.<\/p>\n<p>Looking back, I am particularly struck by what has happened in the last 3 general elections, specifically &#8211;<\/p>\n<ul>\n<li>An overestimate of 4.1% for the Liberal Democrats in 2010.<\/li>\n<li>An underestimate of 4.1% for the Conservatives in 2015.<\/li>\n<li>An underestimate of 4.5% for Labour in 2017.<\/li>\n<li>An increase in the number of pollsters to 11+ compared to 5 to 7 between 1970 &amp; 2005.<\/li>\n<\/ul>\n<p>The 3 significant errors listed are in fact 3 of the\u00a05 largest errors ever seen since 1950 with the Labour underestimate of 4.3% in 1951 and the Conservative underestimate of 5.2% in 1992 being the other two.\u00a0 One could make the argument that the lower cost of internet polling has encouraged more pollsters to join the party but at the cost of quality.\u00a0 There might be something in that argument but that would be to ignore the poor record of the pollsters in other elections such as the period 1992 to 2001.\u00a0 1992 saw the largest ever error for a party which was also accompanied by a 4% overestimate for Labour.\u00a0 This polling miss is well known but what is not so well known is that\u00a0the pollsters continued to get the Labour vote wrong by a similar order of magnitude in 1997 &amp; 2001.\u00a0 Those errors passed unnoticed because the pollsters did get the outcome right i.e. a Labour landslide, but I would contend that these have to be recorded as significant polling errors.<\/p>\n<p>One of the things that strikes me about this chart is how the errors for Labour and the Liberal Democrats are inversely correlated with each other with a correlation coefficient of -0.61.\u00a0 This makes sense in today&#8217;s environment where there is a lot of talk of a progressive alliance between Labour, Lib Dems and the Greens and one easily imagine a scenario where tactical voting means the polls overestimate Labour and underestimate the Lib Dems.\u00a0 However it would seem that this has been a dynamic in British elections for a very long period of time.\u00a0 The equivalent correlation between the Conservatives and Lib Dems is only -0.13.<\/p>\n<p>Given this, I have redone the chart by combining the Conservatives &amp; UKIP into one group and Labour, Lib Dems &amp; Greens into another group.\u00a0 In practice, the UKIP &amp; Green errors are only known for 2015 &amp; 2017\u00a0as those have been the only elections where pollsters have recorded votes for these parties separately rather than putting lumping them into Others.\u00a0 So for the most part the\u00a0chart below\u00a0is comparing the Conservative poll error with the combined Lab\/LD error.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-683 alignnone\" src=\"https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2017\/07\/UpdatedPollErrors1-300x141.png\" alt=\"\" width=\"740\" height=\"348\" srcset=\"https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2017\/07\/UpdatedPollErrors1-300x141.png 300w, https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2017\/07\/UpdatedPollErrors1-768x360.png 768w, https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2017\/07\/UpdatedPollErrors1-450x211.png 450w, https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2017\/07\/UpdatedPollErrors1.png 1002w\" sizes=\"auto, (max-width: 740px) 100vw, 740px\" \/><\/p>\n<p>The lines represent centred 5-election moving averages and currently sit at a 1.3% underestimate for the CON\/UKIP error and a 1.4% overestimate for the LAB\/LD\/GRN error.<\/p>\n<p>This revised chart makes things a lot clearer and also emphasises the exceptional nature of 2017.\u00a0 In 2010, the Lib Dems were badly overestimated but this was partly compensated by an underestimate in the Labour vote.\u00a0 In 2017, the underestimate in the Labour vote was only slightly compensated by an error in the Lib Dem &amp; Green vote.\u00a0 The CON+UKIP vote share was overestimated by 2.4% but this was almost entirely due to the UKIP vote being overestimated since the Conservative vote share was more or less spot on.<\/p>\n<p>In effect, 2017 was a repeat of the 1983 &amp; 1951 elections but on a larger scale.\u00a0 History shows that errors like these favouring Labour are exceptional and the norm has been errors (usually significant errors) that favour the Conservatives.\u00a0 The chart above shows only 6 elections out of 19 with errors favouring the &#8220;progressive alliance&#8221;\u00a0with 4 of these taking place before 1966.\u00a0\u00a0Conversely, the 7 elections prior to 2017 all had errors favouring the Conservatives.\u00a0 <a href=\"https:\/\/marriott-stats.com\/nigels-blog\/uk-general-elections-1-how-accurate-are-the-opinion-polls\/\">In my earlier post on polling err<\/a>ors, I stated that a repeat of 1983 was a distinct possibility for a number of reasons and I put the probability of an error (of any magnitude) favouring Labour at 25% so my warnings were prescient.\u00a0 At the same time, I stated that I was likely to go with a final forecast that assumed an error that favoured the Conservatives whilst\u00a0considering a scenario where the errors favoured Labour.\u00a0 This is what I did\u00a0in the end but I went about this in a different way as I explained in my post &#8220;<a href=\"https:\/\/marriott-stats.com\/nigels-blog\/uk-general-election-2017-forecast-5-5-steps-to-making-sense-of-the-latest-polls\/\">5 steps to making sense of the latest polls<\/a>.&#8221;\u00a0The difference between my original intentions and\u00a0my actual method was only 0.5% in the Conservative lead over Labour so I am satisfied that I fulfilled the spirit of what I intended.<\/p>\n<p>&nbsp;<\/p>\n<p>So far I have been concentrating on the expected vote share for each party or combination of parties.\u00a0 In practice, when it comes to predicting the outcome of\u00a0an election that uses First Past the Post as its election system, the more important prediction is the Conservative lead over Labour.\u00a0 These parties have always been expected to take the top two places nationally so I have calculated the expected lead from the polls and compared it with the actual lead to produce the following chart.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-685 alignnone\" src=\"https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2017\/07\/UpdatedPollErrors3-300x141.png\" alt=\"\" width=\"757\" height=\"356\" srcset=\"https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2017\/07\/UpdatedPollErrors3-300x141.png 300w, https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2017\/07\/UpdatedPollErrors3-768x361.png 768w, https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2017\/07\/UpdatedPollErrors3-450x211.png 450w, https:\/\/marriott-stats.com\/nigels-blog\/wp-content\/uploads\/2017\/07\/UpdatedPollErrors3.png 1001w\" sizes=\"auto, (max-width: 757px) 100vw, 757px\" \/><\/p>\n<p>We see again how 2017 reflects 1983 though the error is larger than 1983 but\u00a0not as large as 1951.\u00a0 The 5-election centred moving average still shows an underestimate of 2% in the Conservative lead over Labour and indeed this appears to have been the long run average since 1964.\u00a0 It is very tempting (and no doubt many people will try) to put reasons on this chart but treated as a time series in its own right, I have to say that I do not see any explanatory patterns apparent.<\/p>\n<p>If we define a SIGNIFICANT error as being one where the CON-LAB lead is out by 2%, then we can make the following observations about the 19 elections since 1950:-<\/p>\n<ol>\n<li>Only 4 out of 19 elections did not experience a significant polling error (1955, 1964, 1979, 2010)<\/li>\n<li>10 out of 19 elections experienced a significant polling error favouring the Conservatives.<\/li>\n<li>5 out of 19 elections experienced a significant polling error favouring Labour.<\/li>\n<li>The average polling error (in CON-LAB lead) is +1.5% and the standard deviation is 4.3%.<\/li>\n<li>If our null hypothesis is that the average polling error is 0%, then our t-statistic is +1.5 and the p-value (using 2-tailed t-test) is 15%.<\/li>\n<li>Nothing greatly changes if we confine our analysis to 1974 onwards i.e. from when the Northern Ireland parties and the Nationalists arrived on the political scene and the CON+LAB vote share saw a significant shift downwards.<\/li>\n<\/ol>\n<p>I chose 2% as a definition of a significant error as my experience shows that an error on this scale will mislead election forecasters as happened for the most part in 2017.\u00a0 If another election is called, I would now seek to calculate 3 scenarios:<\/p>\n<ol>\n<li>A 4% underestimate in the CON-LAB lead as stated by the polls\u00a0which favours the Conservatives.<\/li>\n<li>No error in the CON-LAB lead as stated by the polls.<\/li>\n<li>A 4% overestimate in the CON-LAB lead as stated by the polls which favours Labour.<\/li>\n<\/ol>\n<p>The question which then has to be answered is what is the relative likelihood of each scenario?\u00a0 Whilst the statistical significance of the historical errors is not overwhelming, I would still be happy to use a 2:1:1 ratio i.e. I would give scenario 1 50% weight, scenario 2 25% weight and scenario 3 25% weight.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Ahead of the 2017 general election, I\u00a0predicted that the\u00a0opinion polls would be wrong again and that the Conservatives lead over Labour would be underestimated by 2.6%.\u00a0 \u00a0I\u00a0based this on\u00a0data provided\u00a0by Mark Pack who has systematically recorded every opinion poll published since 1945.\u00a0 In the event, I was right that the polls would be wrong but [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_genesis_hide_title":false,"_genesis_hide_breadcrumbs":false,"_genesis_hide_singular_image":false,"_genesis_hide_footer_widgets":false,"_genesis_custom_body_class":"","_genesis_custom_post_class":"","_genesis_layout":"","footnotes":""},"categories":[2,6,3],"tags":[21,19,33],"class_list":{"0":"post-680","1":"post","2":"type-post","3":"status-publish","4":"format-standard","6":"category-elections","7":"category-forecasting","8":"category-polling","9":"tag-election-forecasting","10":"tag-elections","11":"tag-opinion-polls","12":"entry","13":"override"},"_links":{"self":[{"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/posts\/680","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/comments?post=680"}],"version-history":[{"count":4,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/posts\/680\/revisions"}],"predecessor-version":[{"id":687,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/posts\/680\/revisions\/687"}],"wp:attachment":[{"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/media?parent=680"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/categories?post=680"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/tags?post=680"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}