{"id":1098,"date":"2018-05-12T17:37:22","date_gmt":"2018-05-12T16:37:22","guid":{"rendered":"https:\/\/marriott-stats.com\/nigels-blog\/?p=1098"},"modified":"2019-04-09T15:11:14","modified_gmt":"2019-04-09T14:11:14","slug":"1-in-10-orgs-published-incorrect-gender-pay-gap-data","status":"publish","type":"post","link":"https:\/\/marriott-stats.com\/nigels-blog\/1-in-10-orgs-published-incorrect-gender-pay-gap-data\/","title":{"rendered":"Pay Gaps #2 – 1 in 10 organisations have published incorrect gender pay gap data. Are you one of them?"},"content":{"rendered":"

The first round of gender pay gap data is now available for all organisations employing more than 250 employees.<\/a>\u00a0 The UK now faces the challenge of interpreting this data and using it properly which will not be easy given “the 7 ways to misuse gender pay gap data<\/a><\/em>” I recently wrote about.\u00a0 In the short term, there is an immediate need to resolve errors in the data published and I will demonstrate that potentially 10-15% of organisations have entered incorrect data into the government’s database.<\/p>\n

I have downloaded the gender pay gap data for all 10,504 organisations<\/a> who have submitted their results.\u00a0 My analysis of this data has uncovered 3 distinct type of errors.<\/p>\n

    \n
  1. Incorrect data entered<\/strong><\/li>\n
  2. Income quartiles entered the wrong way around<\/strong><\/li>\n
  3. Claiming no gender pay gap when income quartile gender splits clearly show there is a gap<\/strong><\/li>\n<\/ol>\n

    I will explore each in turn and explain how the government should have prevented these errors through simple sanity check code on their website.<\/p>\n

    Error 1 – Incorrect data entry<\/strong><\/span><\/h4>\n

    The following two companies say their gender pay gap is greater than 100%.\u00a0 This is mathematically impossible.<\/p>\n

    \"\"<\/p>\n

    In statement 1 of my “7 ways to misuse gender pay gap data<\/em>” post<\/a>, I explain that the median gender pay gap is calculated as the difference between the median man and the median woman in terms of their hourly earnings (including bonuses) expressed as a percentage of the median man’s figure.\u00a0 So if the median man in a company is paid \u00a325 per hour and the median woman is paid \u00a320 per hour, then the difference is \u00a325-\u00a320=\u00a35 which is 20% of the median man’s figure of \u00a325.\u00a0 The convention (which I disagree with) is that if men are paid more than woman then the gender pay gap is positive and if woman are paid more than men, it is negative.<\/p>\n

    In the two examples above, gender pay gaps of 121% and 320% were recorded.\u00a0 Since the convention is that a positive number means that median man earns more than the median woman, this means the median woman is earning 121% LESS than the median man in the first chart and 320% less in the second chart.\u00a0 But the maximum positive value the median gender pay gap can be has to be less than 100% i.e. a situation where woman are paid nothing which obviously cannot happen.\u00a0 So what has gone wrong here?<\/p>\n

    All companies have the option of uploading their own report to the government database.\u00a0 Partnering Health Ltd chose not to do so but Shrewsbury Academies Trust did so<\/a> and they have reported the same number.\u00a0 At this point in time, I cannot work out what error they made so they need to look at this again.\u00a0 It is possible they have made a variant on Error 3 described below.<\/p>\n

    Obviously it is a simple matter for the government to prevent such errors occurring.\u00a0 All that is needed a piece of code that rejects any pay gap (whether median or mean) that exceeds 100%.\u00a0 One possible reason why people might make this error is that instead of expressing the difference between the median man and the median woman as a percentage of the median man, they instead express it as a percentage of the median woman.\u00a0 Then if the difference between the median man & woman is \u00a320 per hour and the median woman earns \u00a310 per hour, then the difference is 200%.<\/p>\n

    When the median woman earns more than the median man, then it is possible for the gender pay gap to exceed 100% but in the negative direction according to the con\"\"vention.\u00a0 An example is Randstad HR as shown.\u00a0 I found this company by looking at who was paying women the most relative to men and at first I was curious why Randstad should be so favourable to women.\u00a0 Then I noticed a variant of Error 3 below had occurred and clicked on their own analysis of their gender pay gap<\/a>.\u00a0 If you read this you will see that they are in fact reporting a gender pay gap of 18% not -104% as claimed by the chart.\u00a0 I am not sure how this error has been made but again any claim to pay woman twice as much as men should be challenged by the government’s website if someone enters such data.\u00a0 Randstad in fact are not the largest such claimant.\u00a0 That distinction goes to Bar 2010 Ltd<\/a> though in that instance, the difference is understandable due to the fact the company is practically all male.<\/p>\n

    How many companies are making errors like these?\u00a0 The answer is I don’t know but it could be mitigated with better sanity checks on the government website.<\/p>\n

    Error 2 – Income Quartiles entered the wrong way around<\/strong><\/span><\/h4>\n

    In statement 3 of my <\/span>“7 ways to misuse gender pay gap data<\/em>” post<\/a>, I explain the idea of income quartiles whereby the organisation is split into 4 equal sized chunks (known as quartiles) of employees based on their hourly earnings (including bonuses) and then within each quartile, the gender split between men and women is recorded.\u00a0 I stated in my other post that it is the quartile data that potentially offers the greatest value to understanding why women are paid less than men overall but it also offers a straightforward sanity check on the median gender pay gap.<\/span><\/p>\n

    I will use Worcestershire County Council as my example here.\"\"\u00a0 Their median gender pay gap is completely in line with the national median of 9%.\u00a0 Yet based on the quartile data submitted, it is possible they have made a mistake and should have submitted -9% i.e. woman paid more than men.\u00a0 A little thought as I will explain shows why.<\/p>\n

    Suppose exactly 400 people work for the council.\u00a0 When these people are allocated to income quartiles, each quartile will have exactly 100 people.\u00a0 Then within the lowest income quartile, according to the chart, 66 are woman and 34 are men.\u00a0 Conversely, in the upper quartile, 75 are woman and 25 are men.\u00a0 If you add up the number of woman and men in the council as implied by the chart, you will see that there are 293 women and 107 men working for the council.<\/p>\n

    To find the median woman, the 293 women have to stand in a line in order of their hourly earnings.\u00a0 The 147th woman will be in the middle of the line and becomes the median woman.\u00a0 We repeat with the 107 men and 54th man in the line becomes the median man.<\/p>\n

    Which income quartiles do the median man and median woman come from?\u00a0 We know 138 (=72+66) women are in the bottom two income quartiles but we are looking for the 147th woman.\u00a0 Therefore she must be in the upper middle income quartile.\u00a0 For the median man, the bottom two income quartiles contain 62 (=34+28) men and we are looking for the 54th man so he must be in the lower middle income quartile.\u00a0 Therefore the median man is earning less than the median woman and the reported gender pay gap should be negative not positive as shown in the chart.<\/p>\n

    Something is clearly not right here.\u00a0 Either the median gender pay gap is incorrect or more likely, the gender splits by income quartile have been entered in the government database the wrong way round.\u00a0 According to the chart, 75-80% of the higher paid roles (presumably managers and directors) are women whereas 65-70% of lower paid roles (presumably front line staff) are women.\u00a0 It is a truism that senior roles are more likely to be men and even though this is a public sector organisation which tends to be more aware of gender gaps than the private sector, I suspect this data has been entered the wrong way around in this instance.\u00a0 A week ago, when I was working with an earlier version of the database, I noticed that Arriva Plc had made this mistake since the database figures were the reverse of what was published in their own report but they spotted this themselves and the latest version of the database shows the correct figures.<\/p>\n

    Such errors can be captured by coding the thought process I describe above and prompting the person who is entering the data to check their data if this situation occurs.\u00a0 It is not difficult to work out that if a company is reporting a POSITIVE median (not mean) gender pay gap then this must imply that the sum of male %’s in the upper income & upper middle in\"\"come quartiles should be HIGHER than the sum of the male %’s in the lower middle and lower income quartiles.\u00a0 So in the case of Worcs CC, since they claim a gender pay gap of +9% then the inequality 25%+20% > 28%+34% must be true but it isn’t in this case.\u00a0 Conversely if an organisation claims a\u00a0negative gender pay gap of -13% such as Eastleigh Borough Council, then the inequality of 75%+47% < 45% + 56% should be true but it clearly isn’t in this case.\u00a0 Following up through Eastleigh’s own report shows that the reason for this error is that they do not understand what a quartile is<\/a> and therefore their calculations are completely wrong.<\/p>\n

    Error 3 – Claim of no pay gap conflicts with the male quartile gap<\/strong><\/span><\/h4>\n

    The sanity check I explain in error 2 can be coded as follows.<\/p>\n

      \n
    1. Calculate the Male Quartile Gap<\/strong> as the sum of the male %’s in the upper & upper middle income quartiles minus the sum of the male %s in the lower and lower middle income quartiles.<\/span><\/li>\n
    2. If gender pay gap is positive then the male quartile gap must be positive.<\/li>\n
    3. If gender pay gap is negative then the male quartile gap must be negative.<\/span><\/li>\n<\/ol>\n

      If these sanity checks are violated the data published is wrong and an error has been made somewhere.\u00a0 But what about companies that claim a median gender pay gap of zero?\u00a0 937 of the 10,504 organisations claim the median man is paid the same as the median woman.\u00a0 Based on the sanity check, it should be obvious that this can only be correct if the male quartile gap is zero and only 48 of the 937 organisations meet this criteria.<\/p>\n

      Actually, it is mathematically possible for a small violation of this sanity check to occur and still have a legitimate median gender pay gap of zero.\u00a0 Chelsea football club<\/a> is an example where the male quartile gap is 1% even though their median gender pay gap is 0%.\u00a0 Of the 889 organisations reporting zero median gender pay gaps but non-zero male quartile gaps, 325 have male quartile gaps between -5% and +5%.\u00a0 Some of these 325 will be wrong but some could be right.\u00a0 But this still leaves 564 organisations which must have made a mistake somewhere and a number of football clubs, including my own club of Newcastle Utd, appear to be making error 3 as shown below.<\/p>\n

      \"\"<\/p>\n

      In addition to these three clubs, I saw the same error with Millwall, Notts County, Wigan, Rangers, Hearts and Leicester City.<\/p>\n

      Why has this happened?\u00a0 As we saw for Eastleigh Borough Council, they might simply not know how to do the calculations properly.\u00a0 I therefore decided to read the government guidance on how to calculate the median (and the quartiles)<\/a> and I was shocked by what I saw.\u00a0 Here I quote the calculation of the median in red italics:-<\/p>\n

      Median gender pay gap in hourly pay: how to calculate<\/em><\/span><\/h3>\n
        \n
      1. Arrange the hourly pay rates of all male full-pay relevant employees from highest to lowest<\/em><\/span><\/li>\n
      2. Find the hourly pay rate that is in the middle of the range \u2013 this gives you the median hourly rate of pay for men<\/em><\/span><\/strong><\/li>\n
      3. Arrange the hourly pay rates of all female full-pay relevant employees from highest to lowest<\/em><\/span><\/li>\n
      4. Find the hourly pay rate that is in the middle of the range \u2013 this gives you the median hourly rate of pay for women<\/em><\/span><\/strong><\/li>\n
      5. Subtract the median hourly pay rate for women from the median hourly pay rate for men<\/em><\/span><\/li>\n
      6. Divide the result by the median hourly pay rate for men<\/em><\/span><\/li>\n
      7. Multiply the result by 100 \u2013 this gives you the median gender pay gap in hourly pay as a percentage of mens\u2019 pay<\/em><\/span><\/li>\n<\/ol>\n

        Steps 1, 3, 5, 6 & 7 are OK but steps 2 & 4 are plain wrong.\u00a0 A non-mathematical person will interpret this as meaning the middle of a numerical range.\u00a0 So if hourly earnings of woman vary between \u00a310 & \u00a390 per hour, the middle of the range will be \u00a350 per hour.\u00a0 But there is absolutely no guarantee that this will be the earnings of the median woman who is the in the middle of the LINE not the middle of the range!\u00a0 I run basic statistical training courses<\/a> and this misconception of the median comes up all the time so I am certain that many organisations have read these instructions literally.<\/p>\n

        By the way, the ACAS guide is no better<\/a> and makes the same mistake regarding the median.<\/p>\n

        The information on calculating quartiles starts off OK but then gets into a muddle when dealing with numbers that don’t divide evenly into 4 whole numbers.\u00a0 However, for many people who think they know that the median is the middle of the range rather than the line, I would not be at all surprised if they then think that quartiles is just the range divided into 4 equal sized chunks e.g. \u00a310-30, \u00a330-50, \u00a350-70 & \u00a370-90 which of course it isn’t.<\/p>\n

        How many organisations are making errors?<\/strong><\/h4>\n

        Based solely on the comparison of the reported median gender pay gap versus the calculated male quartile gap, I estimate that 9% of organisations have definitely made an error and a further 8% may have made an error as shown in the table.\u00a0 Note the sanity check of comparing male quartile gaps with reported gender gaps only captures errors 2 & 3.\u00a0\"\" Error type 1 can still occur and not create a conflict between the reported median gender pay gap and the calculated male quartile gap so the true error rate may yet be higher.\u00a0 I urge the government to revamp its database and include sanity checks as described in this post as soon as possible.\u00a0 At the same time, their guidance on calculating the median and quartiles must be rewritten by a statistician!<\/p>\n","protected":false},"excerpt":{"rendered":"

        The first round of gender pay gap data is now available for all organisations employing more than 250 employees.\u00a0 The UK now faces the challenge of interpreting this data and using it properly which will not be easy given “the 7 ways to misuse gender pay gap data” I recently wrote about.\u00a0 In the short […]<\/p>\n","protected":false},"author":3,"featured_media":1107,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_genesis_hide_title":false,"_genesis_hide_breadcrumbs":false,"_genesis_hide_singular_image":false,"_genesis_hide_footer_widgets":false,"_genesis_custom_body_class":"","_genesis_custom_post_class":"","_genesis_layout":"","footnotes":""},"categories":[64],"tags":[65,66,37,63],"class_list":{"0":"post-1098","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-diversity","8":"tag-checking-for-errors","9":"tag-data-errors","10":"tag-data-journalism","11":"tag-gender-pay-gap","12":"entry","13":"override"},"_links":{"self":[{"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/posts\/1098","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/comments?post=1098"}],"version-history":[{"count":10,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/posts\/1098\/revisions"}],"predecessor-version":[{"id":1688,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/posts\/1098\/revisions\/1688"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/media\/1107"}],"wp:attachment":[{"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/media?parent=1098"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/categories?post=1098"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/marriott-stats.com\/nigels-blog\/wp-json\/wp\/v2\/tags?post=1098"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}